Issue with UDF Return Type Change in Latest Polars Version

[tfiasco]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

lf = pl.DataFrame(
    {
        "a": [1.0, 2, 3, 4, 5],
    }
).lazy()

def np_cumsum(x1):
    x1 = x1.to_numpy()
    return {"a": pl.Series(x1.cumsum())}

result = lf.select(
    z=pl.col("a").map_batches(np_cumsum, return_dtype=pl.Struct({"a": pl.List(pl.Float64())})),
).collect()

print(result)

Log output

polars.exceptions.ComputeError: expected a something that could convert to a `Series` but got: <class 'dict'>

Issue description

I recently upgraded from Polars 1.8.0 to the latest version and encountered an error with my existing code.

Previously, I could return dict[str, pl.Series[float]] from the UDF without any issues. However, after upgrading, this code throws an error, and I now need to return dict[str, list[float]].

This change requires using {"a": list(x1.cumsum())}, which involves a memory copy, whereas pl.Series did not, resulting in a significant performance difference.

Is this a bug or an intended feature? If this is the new expected behavior, how should I modify my code accordingly?

Expected behavior

output:

shape: (1, 1)
┌──────────────────────┐
│ z                    │
│ ---                  │
│ struct[1]            │
╞══════════════════════╡
│ {[1.0, 3.0, … 15.0]} │
└──────────────────────┘

Installed versions

--------Version info---------
Polars:              1.19.0
Index type:          UInt32
Platform:            Linux-5.10.0-30-amd64-x86_64-with-glibc2.31
Python:              3.10.11 (main, May 16 2023, 00:28:57) [GCC 11.2.0]
LTS CPU:             False

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               <not installed>
azure.identity       <not installed>
boto3                <not installed>
cloudpickle          <not installed>
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               <not installed>
gevent               <not installed>
google.auth          <not installed>
great_tables         <not installed>
matplotlib           <not installed>
nest_asyncio         1.6.0
numpy                2.1.3
openpyxl             <not installed>
pandas               2.2.3
pyarrow              18.0.0
pydantic             <not installed>
pyiceberg            <not installed>
sqlalchemy           <not installed>
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           <not installed>```

</details>