cloudpickle fails on Python 3.14: pickle regression?

[vstinner]

test_empty_file() of cloudpickle fails on Python 3.14: cloudpipe/cloudpickle#544

It fails since commit 1bb955a:

commit 1bb955a2fe0237721c141fdfe520fd3ba46db11e
Author: Serhiy Storchaka <[email protected]>
Date:   Mon Aug 5 16:21:32 2024 +0300

    gh-122459: Optimize pickling by name objects without __module__ (GH-122460)

• Old behavior (ok): reducer_override() is only called once on a function: _file_reconstructor().
• New behavior (bug?): reducer_override() is called on _file_reconstructor() function, and then it is called in a loop on the function _make_function().

[vstinner]

cc @serhiy-storchaka

[vstinner]

Try reproducer.py:

# OK
$ python3.13 -c 'import reproducer'

# ERROR
$ python3.14 -c 'import reproducer'
Traceback (most recent call last):
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 253, in dump
    return super().dump(obj)
           ~~~~~~~~~~~~^^^^^
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 280, in reducer_override
    return self._function_reduce(obj)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 232, in _function_reduce
    return self._dynamic_function_reduce(obj)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 225, in _dynamic_function_reduce
    state = _function_getstate(func)
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 124, in _function_getstate
    f_globals_ref = _extract_code_globals(func.__code__)
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 99, in _extract_code_globals
    out_names = _extract_code_globals_cache.get(co)
  File "/usr/lib64/python3.14/weakref.py", line 357, in get
    return self.data.get(ref(key),default)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded in comparison
when serializing function reconstructor
when serializing function object
when serializing function reconstructor
when serializing function object
(...)
when serializing _io.TextIOWrapper reconstructor
when serializing _io.TextIOWrapper object

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import reproducer
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 292, in <module>
    serialized = dumps(f)
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 287, in dumps
    cp.dump(obj)
    ~~~~~~~^^^^^
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 257, in dump
    raise pickle.PicklingError(msg) from e
_pickle.PicklingError: Could not pickle object as excessively deep recursion required.

[vstinner]

What I don't understand is that the reproducer fails on Python 3.13 if run directly, without import:

$ python3.13 reproducer.py
Traceback (most recent call last):
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 253, in dump
    return super().dump(obj)
           ~~~~~~~~~~~~^^^^^
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 280, in reducer_override
    return self._function_reduce(obj)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 232, in _function_reduce
    return self._dynamic_function_reduce(obj)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 225, in _dynamic_function_reduce
    state = _function_getstate(func)
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 124, in _function_getstate
    f_globals_ref = _extract_code_globals(func.__code__)
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 99, in _extract_code_globals
    out_names = _extract_code_globals_cache.get(co)
  File "/usr/lib64/python3.13/weakref.py", line 452, in get
    return self.data.get(ref(key),default)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded in comparison

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 292, in <module>
    serialized = dumps(f)
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 287, in dumps
    cp.dump(obj)
    ~~~~~~~^^^^^
  File "/home/vstinner/dev/cloudpickle/reproducer.py", line 257, in dump
    raise pickle.PicklingError(msg) from e
_pickle.PicklingError: Could not pickle object as excessively deep recursion required.

[vstinner]

False alarm: it's not a Python regression.

It fails since commit 1bb955a

This commit changes _getattribute() API and cloudpickle uses it. I wrote cloudpipe/cloudpickle#545 to port cloudpickle to Python 3.14.