list_concat([list<T>, list<T>]) gives list<T>, not list<list<T>>

[NickCrews]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

pl.select(pl.lit([1]))
# Gives [1], of type list[i64], as expected

pl.select(pl.concat_list([pl.lit([1]), pl.lit([2])]))
# Gives [1, 2], of type list[i64]
# I would expect it to be [[1], [2]], of type list[list[i64]]

Log output

<nothing came out>

Issue description

see above

Expected behavior

see above

Installed versions

--------Version info---------
Polars:               1.0.0-rc.2
Index type:           UInt32
Platform:             macOS-14.5-arm64-arm-64bit
Python:               3.11.6 (main, Oct 23 2023, 09:04:08) [Clang 14.0.0 (clang-1400.0.29.202)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
great_tables:         <not installed>
hvplot:               <not installed>
matplotlib:           <not installed>
nest_asyncio:         1.6.0
numpy:                <not installed>
openpyxl:             <not installed>
pandas:               <not installed>
pyarrow:              <not installed>
pydantic:             <not installed>
pyiceberg:            <not installed>
sqlalchemy:           <not installed>
torch:                <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

[ritchie46]

It is a concat list operation. It concats the lists. If you want to create an extra level of nesting you need to implode() the arguments.

[mcrumiller]

@ritchie46 I think this is a valid issue. Why do the following two produce the same result?

pl.DataFrame({"a": [1], "b": [2]}).select(
    pl.concat_list("a", "b")
)
# shape: (1, 1)
# ┌───────────┐
# │ a         │
# │ ---       │
# │ list[i64] │
# ╞═══════════╡
# │ [1, 2]    │
# └───────────┘

pl.DataFrame({"a": [[1]], "b": [[2]]}).select(  # note the [[1]]
    pl.concat_list("a", "b")
)
# shape: (1, 1)
# ┌───────────┐
# │ a         │
# │ ---       │
# │ list[i64] │
# ╞═══════════╡
# │ [1, 2]    │
# └───────────┘

The documentation says "Horizontally concatenate columns into a single list column." If the columns are lists, then they should form a list of lists.

Edit: I just noticed there is more to the documentation in 1.0 which isn't on the current 0.2 documentation, that clarifies.

[mcrumiller]

FYI I feel that pl.list would be a better name if the intended behavior is what's written in the description. I agree that the word "concat" implies that you take existing lists and concatenate them together, but I believe the intent is to take existing columns and concanate them into a list, in which case the inner dtype of the list should be common supertype of the columns.

[ritchie46]

The initial implosion is because there isn't any list yet. So in order to concat them, they are imploded.

I think we should improve the description here.

[mcrumiller]

@ritchie46 the description's improved in the 1.0 docs, I had missed that.

[NickCrews]

@mcrumiller pinpoints my exact confusion. I agree there should be two functions, one that is a constructor as in
Iterable[T] -> List[T] , and another which concatenates Iterable[List[T]] -> List[T]

[NickCrews]

I'm looking for the constructor. What could I use for that currently?

[mcrumiller]

Hi @NickCrews, can you be a little more specific about what you want to do?

[NickCrews]

Yes, but I'll get to it in an hour or two once I can get back to my computer

[NickCrews]

Actually, I think we should move all discussion to #8510.

Specifically, I responded in this comment in that issue.