Dill breaks cloudpickle > 1.3.0 on inner functions with closure

[davidsmf]

Cloudpickle 1.5.0, dill 0.3.2 or 0.3.1.1:

import dill
import cloudpickle


class Foo:

 def bar(self, param: int):
 hello = "Hello"
 def baz() -> None:
 print(hello, param)

 cloudpickle.loads(cloudpickle.dumps(baz))()


if __name__ == '__main__':
 Foo().bar(param=2)

This fails with ValueError: Cell is empty.

Without the dill import, or with the dill import after the cloudpickle import, and it's all fine.

If the inner function is just

 def baz() -> None:
 print("Hello world")

then it's also fine.

The problem seems to be if the inner function captures anything in its closure.

This was fine with cloudpickle 1.3.0

Should I bring this up with dill?

[pierreglaser]

Hi, thanks for the report, I can reproduce locally. This is due to a recent refactoring in cloudpickle that decreased the priority of some cloudpickle reducers, including the one for cell objects. The consequence is that cloudpickle can in some case give the priority to other registered reducers. In this precise case, importing dill triggers a global side effect which add a high-priority cell reducer that is not compatible with cloudpickle's way of reducing cells.

The main problem is that importing dill triggers a global side effect. Ideally, this should not happen - so I suggest you open a bug in dill. Feel free to tag me in it in case the maintainers need more detail.

[mmckerns]

You can just turn off the injection (temporarily, if you like) with dill.extend(False). This provides a toggle for the injection of dill types into the pickle registry, so you can turn it back on with dill.extend(True). It's not a dill bug, it's that the two packages make different choices, some of which are incompatible -- and with the registry injection from dill, this actually enables both serializers to work together often enough. You might also want to use dill.settings['recurse'] = True either globally, or on a per-dump basis, to tell dill to use serialization choices that are closer to the choices that cloudpickle also makes.

[davidsmf]

Thanks, this helps. I think I'm now back to a bug in cloudpickle with custom metaclasses.

import cloudpickle
import pydantic

class Bar(pydantic.BaseModel):
 pass

if __name__ == '__main__':
 cloudpickle.loads(cloudpickle.dumps(Bar()))

this works with cloudpickle==1.3.0, but not with cloudpickle==1.5.0. Is this a bug? Am I doing something wrong?

I'm on python 3.6. It works on python 3.8.

[marco-neumann-by]

So this works:

import dill
dill.extend(False)
import cloudpickle
dill.extend(True)


class Foo:

 def bar(self, param: int):
 hello = "Hello"
 def baz() -> None:
 print(hello, param)

 cloudpickle.loads(cloudpickle.dumps(baz))()


if __name__ == '__main__':
 Foo().bar(param=2)

This however does NOT:

import dill
dill.settings['recurse'] = True
import cloudpickle


class Foo:

 def bar(self, param: int):
 hello = "Hello"
 def baz() -> None:
 print(hello, param)

 cloudpickle.loads(cloudpickle.dumps(baz))()


if __name__ == '__main__':
 Foo().bar(param=2)

The issue now is that the workaround is very difficult to implement in practice, since it would require to be very careful import import orders and how other libs load dill and cloudpickle. Could cloudpickle do something like this?

try:
 import dill
 dill.extend(False)
except ImportError:
 dill = None

# ... setup cloudpickle

if dill is not None:
 dill.extend(True)

[marco-neumann-by]

Please note that there are two issues here:

1. dill interaction: See analysis in this comment. This btw. can also be fixed by installing pickle5 since cloudpickle prefers using it as a base implementation and dill doesn't seem to have a side-effect on it.
2. pydantic issue: See example in this comment. Even if dill is not involved, the pydantic issue still remains, at least under python 3.6.

[marco-neumann-by]

For pydantic, the issue is the interaction between:

https://github.com/samuelcolvin/pydantic/blob/de0657e4a5495de9378db9fe15a17334c2b0fae5/pydantic/main.py#L865-L868

and:

cloudpickle/cloudpickle/cloudpickle.py

Line 460 in cdc704d

is_litteral = getattr(obj, '__values__', None) is not None

[pierreglaser]

I'm wondering: are the two issues related? If not, can you please open a separate issue for the pydantic interaction?