Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SVD error when using eof solver on dask array #122

Open
emmomp opened this issue Jun 7, 2021 · 1 comment
Open

SVD error when using eof solver on dask array #122

emmomp opened this issue Jun 7, 2021 · 1 comment
Labels

Comments

@emmomp
Copy link

emmomp commented Jun 7, 2021

I have been getting an SVD error when trying to call the xarray eof solver on a daskarray.

The error is due to the following line in eofs.standard:
nonMissingIndex = np.where(np.logical_not(np.isnan(self._data[0])))[0]

np.where always fails and gives nans for dask arrays (see e.g. https://stackoverflow.com/questions/59957541/what-is-the-dask-equivalent-of-numpy-where)

Possibly related to #115 (although I can no longer see those notebooks).

Solved by calling .load() before calling the solver, but loses the advantages of dask.

@lfery
Copy link

lfery commented Apr 14, 2022

I had the same problem and I was able to solve it by replacing this line in standard.py
nonMissingIndex = np.where(np.logical_not(np.isnan(self._data[0])))[0]
by

nonMissingIndex = dask.array.where(np.logical_not(np.isnan(self._data[0])))[0]
nonMissingIndex.compute_chunk_sizes()

I think it doesn't lose the advantages of dask as I have a dataset that doesn't fit in memory but the code worked even so.
I hope that can be useful to anyone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants