"maximum recursion depth exceeded " when serializing enum

[rodrigob]

After pip2.7 install enum34 dill running the following script with python2.7

import dill
import enum

class SomeEnumType(enum.Enum):
  VALUE_A = "value_a"
  VALUE_B = "value_b"

dill.dumps(SomeEnumType.VALUE_A)

fails with
RuntimeError: maximum recursion depth exceeded while calling a Python object

the same script under python3 works fine.

[sam-habitat]

I can reproduce this in Python 3.7 and 3.8

[rodrigob]

@sam-habitat you mean that in python3.7 and 3.8 you see or do not see the RuntimeError ?

[rodrigob]

Just tried with Python 3.7.5rc1 and dill-0.3.1.1 and got the RuntimeError.
:(

[rodrigob]

Using byref seems to be a work-around; but unclear to me what are the side effects of this.

>>> dill.settings['byref'] = True
>>> dill.dumps(SomeEnumType.VALUE_A)
b'\x80\x03c__main__\nSomeEnumType\nq\x00X\x07\x00\x00\x00value_aq\x01\x85q\x02Rq\x03.'

[spenceryee]

Using byref allows you to dump it, but if you start a new session you won't be able to load it. I don't fully understand what byref does - perhaps this is intended - but it is a pretty big limitation.

>>> dill.loads(b'\x80\x03c__main__\nSomeEnumType\nq\x00X\x07\x00\x00\x00value_aq\x01\x85q\x02Rq\x03.')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 275, in loads
    return load(file, ignore, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 270, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 473, in load
    obj = StockUnpickler.load(self)
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 463, in find_class
    return StockUnpickler.find_class(self, module, name)
AttributeError: Can't get attribute 'SomeEnumType' on <module '__main__' (built-in)>

[fteicht]

Hi all,

This bug has been opened nearly 2.5 years ago and I am still experiencing it both on MacOs and on Windows with python 3.7.
Here is a minimal code to reproduce the bug:

from enum import Enum
import dill as serializer
# import pickle as serializer

class MyEnum(Enum):
    zero = 0
    one = 1

if __name__ == '__main__':
    fw = open("mintest.ses", "wb")
    serializer.dump(MyEnum, fw)
    fw.close()
    fr = open("mintest.ses", "rb")
    myenum = serializer.load(fr)
    fr.close()

Note that commenting out Line 2 instead of Line 3 in the code above works, which shows that the standard python's pickle can do the job but not dill.

Any progress on solving this bug please?

Thanks,
Florent

[matthewgdv]

I can confirm that on Python 3.9 attempting to pickle an enum with dill causes an error. Crucially, python's standard library pickle module works just fine.

Could we get this assigned please? It's pretty important that at the very least switching to dill doesn't cause pickle functionality to be lost.

[kibergus]

Fix for this problem is very simple, a custom reducer needs to be defined for enums. Following Pickler would behave exactly like dill default one, but would handle enums correctly:

class Pickler(dill.Pickler):
  def reducer_override(self, obj):
    if isinstance(obj, enum.Enum):
      return pickle.loads, (pickle.dumps(obj),)
    return NotImplemented

instead of dill.dump(obj, file) one would need to write Pickler(file).dump(obj). The result can be unpickled by standard dill Unpickler.

I would appreciate if this fix could be integrated in-to dill.

[mmckerns]

@matthewgdv @kibergus: this is not an easy fix. The Enum class refers back to the enum meta class, which then references that enum instances... and this cycle is needed to construct the enum properly. It's handled in pickle in certain cases because pickle is serializing the classes by reference... hence, if the enum class can't be imported (or is present in __main__), it won't serialize with pickle. Thus, in many cases, pickle will still fail. The fix involves creating a reducer for an Enum, and some other additional machinery to support the reducer. I've spec'd out the problem, and have a plan of attack. It's on my todo list.

[mmckerns]

This does not seem to be closed by #577

[anivegesana]

Yes. That is intended. The support for enums was in the other half of #450. Sorry about the delay in putting out the remaining two/three PRs for enums. I got busy and forgot about it. I will need to reorient myself to the new changes in the repo in the last year and see if they conflict with the changes. I hope to get to it soon.

[akiss-ic]

Any progress on this? I am using dill to support session saving and loading in Jupyter notebooks for expensive computations and the use of Enum is very helpful for post-processing and general code cleanliness. Unfortunately I have just discovered I cannot use dill.dump_session() when custom enums are defined...