cloudpickle 1.5.0 breaks on python 3.6+7 without pickle5

[yannikschaelte]

cloudpickle >=1.5.0 fails to dump a complex object using python 3.6+7. Works with 1.4.1. After installing pickle5, it works (related to #370 ?). All works just fine on python 3.8.

Unfortunately, I currently cannot create a minimum test case, as I am not sure what the problem is (tests in an own project fail here https://github.com/ICB-DCM/pyABC/runs/1463556418?check_suite_focus=true, in particular pytest test/base/test_samplers.py). For simple things with local variables etc. I could not reproduce the error. Any hints are welcome, but for the moment installing pickle5 or downgrading to 1.4.0 (or upgrading to python 3.8 ;)) solves the problem for me.

Error message:

pyabc/sampler/multicore_evaluation_parallel.py:104: in sample_until_n_accepted
 simulate_one = pickle.dumps(simulate_one)
../anaconda3/envs/python37/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py:102: in dumps
 cp.dump(obj)
../anaconda3/envs/python37/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py:563: in dump
 return Pickler.dump(self, obj)
../anaconda3/envs/python37/lib/python3.7/pickle.py:437: in dump
 self.save(obj)
../anaconda3/envs/python37/lib/python3.7/pickle.py:504: in save
 f(self, obj) # Call unbound method with explicit self
../anaconda3/envs/python37/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py:745: in save_function
 *self._dynamic_function_reduce(obj), obj=obj
../anaconda3/envs/python37/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py:682: in _save_reduce_pickle5
 dictitems=dictitems, obj=obj
../anaconda3/envs/python37/lib/python3.7/pickle.py:638: in save_reduce
 save(args)
../anaconda3/envs/python37/lib/python3.7/pickle.py:504: in save
 f(self, obj) # Call unbound method with explicit self
../anaconda3/envs/python37/lib/python3.7/pickle.py:789: in save_tuple
 save(element)
../anaconda3/envs/python37/lib/python3.7/pickle.py:504: in save
 f(self, obj) # Call unbound method with explicit self
../anaconda3/envs/python37/lib/python3.7/pickle.py:789: in save_tuple
 save(element)
../anaconda3/envs/python37/lib/python3.7/pickle.py:504: in save
 f(self, obj) # Call unbound method with explicit self
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

pickler = <cloudpickle.cloudpickle_fast.CloudPickler object at 0x7f96ee0095d0>
obj = <cell at 0x7f96ee009510: empty>

 @register(CellType)
 def save_cell(pickler, obj):
 log.info("Ce: %s" % obj)
> f = obj.cell_contents
E ValueError: Cell is empty

[pierreglaser]

Hi,

The error does originate from cloudpickle. The lines responsible for raising the ValueError (as shown on your traceback) are

@register(CellType)
def save_cell(pickler, obj):
 log.info("Ce: %s" % obj)
 f = obj.cell_contents
 pickler.save_reduce(_create_cell, (f,), obj=obj)
 log.info("# Ce")
 return

Which are present in dill, another Python serialization library (see https://github.com/uqfoundation/dill/blob/615eee9560be64a451bc9ea276f4706d287f8d38/dill/_dill.py#L1174-L1180).

It is known that when being imported, dill generates some global side effects which will interfere with cloudpickle, see
#393. So I suspect that somewhere either in your code, or in your imported dependencies, dill is imported. Depending on what is possible/best for you, you can fix your original problem by either not importing dill,or using the workaround suggested in
#393 (comment)

[yannikschaelte]

Hi @pierreglaser , thanks for the fast response! Indeed dill is imported in another (here unused) part of the package. Delaying its import indeed fixed the error and will suffice here as both won't be used at the same time, thanks!