Would Zarr be a possible format option for pyglider?

[jklymak]

@truedichotomy asked:

    Would Zarr be a possible format option for pyglider?

https://medium.com/pangeo/cloud-performant-reading-of-netcdf4-hdf5-data-using-the-zarr-library-1a95c5c92314

https://www.azavea.com/blog/2022/09/22/benchmarking-zarr-and-parquet-data-retrieval-using-the-national-water-model-nwm-in-a-cloud-native-environment/

Originally posted by @truedichotomy in #120 (comment)

[jklymak]

I'm not sure - do you mean for the final output or the intermediate? I don't think polars can use zarr, so that won't fix @callumrollo 's issue with performance. For the final output I think thats possible, particularly if folks want to chunk the output. However, I think that netcdf can chunk output as well?

I'm a little confused if we are having problems with data sets being out of memory or not. If so, that may require better engineering of the whole stack here.