cosima-cookbook can't load 3D variables in conda-env:analysis3-24.04 and later

[taimoorsohail]

Hello,

I hit a Memory Error if I try to load a 3D variable (e.g., temperature) using cosima cookbook for analysis3-24.04 and greater.

I have isolated an example code that demonstrates the issue:

from dask.distributed import Client

client = Client() # n_workers=4
client

session = cc.database.create_session()
# Define experiment in database
expt = '01deg_jra55v13_ryf9091'

start_time, end_time = '2170-01-01', '2180-12-31'

temp = cc.querying.getvar(expt, 'temp', session, start_time=start_time, end_time=end_time,  frequency='1 monthly')

The error is:

MemoryError: Task 'getattr-d071745a-50f5-49b1-bda9-f8a50aeb807d' has 197.16 GiB worth of input dependencies, but worker tcp://127.0.0.1:37893 has memory_limit set to 35.88 GiB.

@angus-g??

[navidcy]

cc'ing @dougiesquire here as well

[navidcy]

so this is not an issue if you use conda envs 24.01 or earlier?

[angus-g]

This is discussed on the Hive: https://forum.access-hive.org.au/t/xarray-warnings-while-loading-data-using-cosima-cookbook/2169/14

[taimoorsohail]

so this is not an issue if you use conda envs 24.01 or earlier?

Correct

[navidcy]

@taimoorsohail does the threads_per_worker=1 kwarg in the dask Client solve this issue?

[navidcy]

I believe this is a duplicate of #409. The PR #488 should make this work for now.