.list.index_of_in() architectural review PR

[itamarst]

This PR implements a sketch of .list.index_of_in() as described in #20626

The questions I would definitely like feedback on at the moment are:

1. Is using amortized_iter() the right approach?
2. Is using needles.iter() the right approach?

[itamarst]

It's also possible the casting logic in Polars will need to be extended; CastingRules::FirstArgLossless isn't what we want since it casts the needles e.g. to List[Int64], we'd need some new option like CastingRules::FirstArgInnerLossless or something that casts to the inner type of the first argument.

[codecov]

Codecov Report

Attention: Patch coverage is 93.24324% with 5 lines in your changes missing coverage. Please review.

Project coverage is 79.70%. Comparing base (958d00f) to head (9fc4d5f).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...s/polars-ops/src/chunked_array/list/index_of_in.rs	91.83%	4 Missing ⚠️
crates/polars-plan/src/dsl/function_expr/list.rs	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #20733      +/-   ##
==========================================
- Coverage   79.75%   79.70%   -0.05%     
==========================================
  Files        1561     1562       +1     
  Lines      221785   222004     +219     
  Branches     2530     2530              
==========================================
+ Hits       176885   176955      +70     
- Misses      44318    44467     +149     
  Partials      582      582              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ritchie46]

I am not sure we want this. The compilation bloat of this seems very high to me and we've got list.eval, so it is able to get what you want.

[itamarst]

So the code in the PR doesn't have much in the way of generics or templates, at least at the moment, so it doesn't seem like it adds much to compilation? Or is it that you're worried about the cumulative costs of lots of small features adding up?

[itamarst]

AFAICT list.eval() doesn't help with the case where you want to match up across columns, per row:

df = pl.DataFrame({"lists": [[3, 1], [2, 4], [5, 3, 1]], "values": [1, 2, 6]})
result = df.select(pl.col("lists").list.index_of_in(pl.col("values")))

because you can't use pl.col("values") inside of .list.eval() expressions (case 1B in the original issue).

@orlp had a pure-Python suggestion in the issue, but it was certainly beyond my admittedly very basic Polars query-writing skills:

result2 = df.with_row_index().select(
    pl.col("lists").explode().index_of(pl.col("values").first()).over("index")
)
print(result2)

If this technique is acceptable another option is a pure Python .list.index_of_in() so it's easier for users.

[itamarst]

Actually, not clear you can actually apply that technique with index inside expression function, so that might not be a viable approach.