Pool objects fail to adhere to dill settings during asynchronous processing.

[cjayross]

The problem here is that setting dill.settings['recurse'] = True doesn't seem to prevent the same issues that are present in Issue #155 of dill while using multiprocess.

To give a minimal test code, running the code below,

import dill
from sympy import Symbol, lambdify
dill.detect.trace(False) # Let me know if you'd like to see this output.
dill.settings['recurse'] = True

x = Symbol('x', real=True)
H = lambdify(x, x**2, 'numpy')

print(dill.loads(dill.dumps(H)))

yields an expected result of printing

<function _lambdifygenerated at 0x7f81f0e79510>

While the following code

import multiprocess as mp
import dill
from sympy import Symbol, lambdify
dill.detect.trace(True)
dill.settings['recurse'] = True

x = Symbol('x', real=True)
H = lambdify(x, x**2, 'numpy')

pool = mp.Pool(mp.cpu_count())
res = pool.map(H, (2,3,))
print(res.get())

throws the exception:

  File "/usr/lib/python3.7/site-packages/dill/_dill.py", line 1321, in save_type
    StockPickler.save_global(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 962, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle <class 'numpy.int64'>: it's not the same object as numpy.int64

with the full output posted here.

I'm running Python 3.7.1 with

>>> dill.__version__
'0.2.8.2'
>>> mp.__version__
'0.70.5'

Oh, and running the first script with

dill.settings['recurse'] = False

yields the same error as in the multiprocess script.

[mmckerns]

@cjayross: Sorry for the slow response. Yep, I know -- multiprocess cannot yet adhere to dill.settings outside of the defaults. It's a planned feature.

[mmckerns]

recent changes to dill now enable dill.settings to have an effect on serialization used by multiprocess. Closing this. Please reopen if the issue isn't resolved for you.