Implement dataless cubes #6253

ESadek-MO · 2024-12-11T11:36:05Z

Closes #4447.

I plan to do this in four stages:

Ensure that cubes can handle data is None when handed a shape value, and can essentially round-trip removing and adding data. This shouldn't break existing tests. 71c7ae8
Create new tests for the above. b2fb2f8
Ensure that all DataManager methods all make sense and work with None data. DataManager.copy() is an example of a method that won't make sense with None data.
Create new tests for the above.

for more information, see https://pre-commit.ci

codecov · 2024-12-11T17:26:59Z

Codecov Report

Attention: Patch coverage is 66.00000% with 34 lines in your changes missing coverage. Please review.

Project coverage is 89.73%. Comparing base (d135ebb) to head (9221917).
Report is 16 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/iris/cube.py	17.39%	9 Missing and 10 partials ⚠️
lib/iris/_data_manager.py	91.52%	3 Missing and 2 partials ⚠️
lib/iris/_merge.py	0.00%	2 Missing and 2 partials ⚠️
lib/iris/_concatenate.py	60.00%	1 Missing and 1 partial ⚠️
lib/iris/_lazy_data.py	50.00%	1 Missing and 1 partial ⚠️
lib/iris/exceptions.py	50.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6253      +/-   ##
==========================================
- Coverage   89.83%   89.73%   -0.10%     
==========================================
  Files          88       88              
  Lines       23347    23451     +104     
  Branches     4344     4383      +39     
==========================================
+ Hits        20974    21044      +70     
- Misses       1646     1664      +18     
- Partials      727      743      +16

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This reverts commit 6ed270d.

bjlittle

@ESadek-MO I'm only partially through the review, but I thought I'd push these review comments early so that you can see them and address them ASAP.

Note that you also need to refactor the following DataManager methods:

__equal__ to account for dataless cubes, particularly when dataless cubes are involved in the operation i.e., for the new use cases between data and dataless, and dataless and dataless
lazy_data to deal with the dataless case
akin to the lazy_data method, you also have to deal with core_data for the dataless case
__repr__ requires to cope with the dataless case i.e., provide the shape

bjlittle · 2024-12-18T13:31:29Z

lib/iris/__init__.py

@@ -832,3 +841,6 @@ def use_plugin(plugin_name):
    significance of the import statement and warn that it is an unused import.
    """
    importlib.import_module(f"iris.plugins.{plugin_name}")
+
+
+DATALESS_COPY = "NONE"


@ESadek-MO It's better to have such constants defined at the top of the module, plus with a comment please 👍

Around about line+142 (after the constraint definition conveniences seems about right)

bjlittle · 2024-12-18T13:37:33Z

lib/iris/_data_manager.py

+            managed. If a value of None is given, the data manager will be
+            considered dataless.
+
+        shape :


@ESadek-MO We've been adopting the following numpydoc standard for specifying the type of parameters i.e., shape : tuple, optional

Same standard applies to the data parameter 👍

bjlittle · 2024-12-18T13:44:15Z

lib/iris/_data_manager.py

+
+        shape :
+            A tuple, representing the shape of the data manager. This can only
+            be used in the case of `data=None`, and will render the data manager


Use double-ticks, see here

i.e., ``data=None``

bjlittle · 2024-12-18T13:44:49Z

lib/iris/_data_manager.py


        """
+        if (shape is not None) and (data is not None):
+            msg = "`shape` should only be provided if `data is None`"


Suggested change

msg = "`shape` should only be provided if `data is None`"

msg = '"shape" should only be provided if "data" is None'

bjlittle · 2024-12-18T13:57:00Z

lib/iris/_data_manager.py

        # Initialise the instance.
        self._lazy_array = None
        self._real_array = None

        # Assign the data payload to be managed.
+        self._shape = shape


@ESadek-MO The comment on line+45 applies to self.data = data on line+47.

Could you move self._shape = shape to the above # Initialise the instance. block 👍

bjlittle · 2024-12-18T15:21:47Z

lib/iris/_data_manager.py

+                if not dataless:
+                    data = np.asarray(data)


@ESadek-MO We shouldn't need this defensive code for the dataless case i.e., we shouldn't get here.

bjlittle · 2024-12-18T15:23:04Z

lib/iris/_data_manager.py

+            result = self.core_data().shape
+        return result
+
+    def is_dataless(self):


Suggested change

def is_dataless(self):

def is_dataless(self) -> bool:

bjlittle · 2024-12-18T15:24:49Z

lib/iris/_data_manager.py

+        return result
+
+    def is_dataless(self):
+        """Determine whether the cube is dataless.


@ESadek-MO Perhaps it's best to not use dataless to describle is_dataless e.g., maybe something like Determine whether the cube has no data. instead?

bjlittle · 2024-12-18T15:28:34Z

lib/iris/_data_manager.py

+        if self.core_data() is None:
+            result = self._shape
+        else:
+            result = self.core_data().shape
+        return result


Suggested change

if self.core_data() is None:

result = self._shape

else:

result = self.core_data().shape

return result

return self._shape if self._shape else self.core_data().shape

bjlittle · 2024-12-18T15:34:09Z

lib/iris/_data_manager.py

+        bool
+
+        """
+        return (self.core_data() is None) and (self.shape is not None)


@ESadek-MO Given our axiom (and if this isn't true then something is wrong) it must always be the case that:

self._shape = None and (self._lazy_array is not None or self._real_array is not None)

self._shape is not None and (self._lazy_array is None and self._real_array is None)

Therefore, is_dataless should be defined simply as return self._shape is not None, right?

for more information, see https://pre-commit.ci

ESadek-MO · 2024-12-18T18:24:38Z

Something is wrong with the _assert_axiom code now. I need to catch a bus, so will have to figure out why tomorrow.

for more information, see https://pre-commit.ci

ESadek-MO · 2024-12-23T18:02:07Z

Although the tests are passing, next time I look at this, I need to investigate the data setter. Currently, line 256 causes issues, as self._shape should be changable if self._shape = (), and it won't be.

That is to say, it works as expected, but I believe it hasn't been written in such a way as it SHOULD be.

ESadek-MO · 2024-12-24T19:23:22Z

lib/iris/_data_manager.py


    @property
    def shape(self):
        """The shape of the data being managed."""
-        return self.core_data().shape
+        return self._shape if self._shape else self.core_data().shape


This is defensive; it wouldn't cause any issues that I can think of to just have return self._shape, but this should protect us in case I've missed something.

I've actually removed this now.

() is falsy. I could have easily done ...if self._shape is not None, but I decided there's no point in adding redundant code. Easy undo if my reviewer disagrees!

for more information, see https://pre-commit.ci

stephenworsley · 2025-01-02T14:56:52Z

lib/iris/_data_manager.py

-        state = is_lazy ^ is_real
-        assert state, emsg.format("" if is_lazy else "no ", "" if is_real else "no ")
+
+        if not (is_lazy ^ is_real):


This might be clearer as if is_lazy == is_real.

ESadek-MO and others added 2 commits December 11, 2024 11:34

Init commit, part way through ensuring Nones aren't wrapped in np.array

96a3f2d

Merge branch 'main' into dataless

a2e0da9

ESadek-MO marked this pull request as draft December 11, 2024 11:36

ESadek-MO requested a review from trexfeathers December 11, 2024 11:36

pre-commit-ci bot and others added 13 commits December 11, 2024 11:38

[pre-commit.ci] auto fixes from pre-commit.com hooks

7114f0f

for more information, see https://pre-commit.ci

None types are no longer wrapped

84ea908

merge conflicts

44afb25

[pre-commit.ci] auto fixes from pre-commit.com hooks

cdc51d5

for more information, see https://pre-commit.ci

clarified axiom check

7b402e0

moved shape order in Cube

bc1ee6f

merge conflicts

17e0b30

[pre-commit.ci] auto fixes from pre-commit.com hooks

16779aa

for more information, see https://pre-commit.ci

replace call to self.data with self.core_data()

f42c19d

Merge branch 'dataless' of github.com:ESadek-MO/iris into dataless

b4ffc31

[pre-commit.ci] auto fixes from pre-commit.com hooks

6d62c7d

for more information, see https://pre-commit.ci

fixed test regex

cc13e6d

precommit

71c7ae8

ESadek-MO added 12 commits December 12, 2024 11:16

written tests, and refactored redundant checks

e59d5c9

refactored tests

95c09e1

removed shape asserts within data tests

b2fb2f8

copy cube now has FUTURE behaviour

c6510a5

pre-commit

af7d727

written tests for cube.copy

144e164

fixed cbe.copy failure in dim coords

ba84553

experimenting with 4d cube with everything; doesn't run locally

ea3c150

pre-c

0cec4ca

edited Coord.copy

6ed270d

Revert "edited Coord.copy"

d79ad27

This reverts commit 6ed270d.

tried tearingdown FUTUREFLAG

2bb6e61

ESadek-MO added 3 commits December 17, 2024 16:17

fixed copying from dataless

a042451

Merge branch 'dataless' of github.com:ESadek-MO/iris into dataless

088bb94

added in exceptions, not tested

21489d5

ESadek-MO requested a review from bjlittle December 18, 2024 12:49

ESadek-MO marked this pull request as ready for review December 18, 2024 12:50

ESadek-MO added 4 commits December 18, 2024 13:54

renamed DATALESS_COPY to DATALESS, and added comment

a059c5e

fixed merge conflicts

19e956d

made DATALESS docstring a docstring, rather than a comment

8178809

added whatsnew

2123b8f

bjlittle requested changes Dec 18, 2024

View reviewed changes

ESadek-MO and others added 7 commits December 18, 2024 16:45

added cubelist errors

703bbb6

[pre-commit.ci] auto fixes from pre-commit.com hooks

33649c1

for more information, see https://pre-commit.ci

review comments

021bd31

Merge branch 'dataless' of github.com:ESadek-MO/iris into dataless

f1bfe7b

[pre-commit.ci] auto fixes from pre-commit.com hooks

abb5a57

for more information, see https://pre-commit.ci

fixed broken review suggestion

94d02ea

Merge branch 'dataless' of github.com:ESadek-MO/iris into dataless

5be6a1f

ESadek-MO and others added 5 commits December 20, 2024 19:23

fixed some problems, and ensured self._shape is always set

5c5bbf8

[pre-commit.ci] auto fixes from pre-commit.com hooks

de348f9

for more information, see https://pre-commit.ci

fixed __eq__ and __repr__, and maybe dtype of dataless equal None

7c74460

Merge branch 'dataless' of github.com:ESadek-MO/iris into dataless

0ceed69

included self.core_data().shape when shape is none, for Coord cases

8bddec7

made sure _shape is set in case of prexisting _shape of ()

2a67050

ESadek-MO commented Dec 24, 2024

View reviewed changes

ESadek-MO and others added 3 commits December 24, 2024 20:21

added eq and repr tests, and fixed shape of () not being valid

f8c2dad

[pre-commit.ci] auto fixes from pre-commit.com hooks

ed75485

for more information, see https://pre-commit.ci

fixed test now that self._shape is always set

9221917

stephenworsley reviewed Jan 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement dataless cubes #6253

Implement dataless cubes #6253

ESadek-MO commented Dec 11, 2024 •

edited

Loading

codecov bot commented Dec 11, 2024 •

edited

Loading

bjlittle left a comment •

edited

Loading

bjlittle Dec 18, 2024

bjlittle Dec 18, 2024

bjlittle Dec 18, 2024

bjlittle Dec 18, 2024

bjlittle Dec 18, 2024

bjlittle Dec 18, 2024

bjlittle Dec 18, 2024

bjlittle Dec 18, 2024

bjlittle Dec 18, 2024

bjlittle Dec 18, 2024

ESadek-MO commented Dec 18, 2024

ESadek-MO commented Dec 23, 2024 •

edited

Loading

ESadek-MO Dec 24, 2024

ESadek-MO Dec 24, 2024

stephenworsley Jan 2, 2025

	msg = "`shape` should only be provided if `data is None`"
	msg = '"shape" should only be provided if "data" is None'

Implement dataless cubes #6253

Are you sure you want to change the base?

Implement dataless cubes #6253

Conversation

ESadek-MO commented Dec 11, 2024 • edited Loading

codecov bot commented Dec 11, 2024 • edited Loading

Codecov Report

bjlittle left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ESadek-MO commented Dec 18, 2024

ESadek-MO commented Dec 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ESadek-MO commented Dec 11, 2024 •

edited

Loading

codecov bot commented Dec 11, 2024 •

edited

Loading

bjlittle left a comment •

edited

Loading

ESadek-MO commented Dec 23, 2024 •

edited

Loading