Class-level attributes dropped during serialization across processes

[jrbourbeau]

I came across a use case where class-level attributes don't seem to survive a roundtrip serialization / deserialization trip where serialization and deserialization happen in different processes.

For example, if I define a Foo class in a test_module.py Python module that keeps track of state through a class-level bar attribute:

# test_module.py

class Foo:
 bar = []

 def append(self, value):
 self.bar.append(value)

and then, in a separate script, create a Foo instance (named foo), add some data to the class-level bar list, and then send the foo instance to a separate subprocess, the state stored in foo.bar isn't included in the serialized form of foo:

import os
import subprocess
import sys

import cloudpickle
from test_module import Foo

# Create Foo instance, add a value to the Foo.bar class-level
# attribute, and then display the value
foo = Foo()
foo.append(123)
print(f"(main process) {foo.bar = }")

# Serialize foo and display foo.bar in a subprocess
s = cloudpickle.dumps(foo)
command = ("import cloudpickle; "
 f"result = cloudpickle.loads({s}); "
 "print(f'(subprocess) foo.bar = {result.bar}')")
subprocess.call([sys.executable, '-c', command])

outputs

(main process) foo.bar = [123]
(subprocess) foo.bar = []

Though I would have expected the output to be:

(main process) foo.bar = [123]
(subprocess) foo.bar = [123]

For reference, this issue originated over in distributed (xref dask/distributed#4233). Also the above was run using cloudpickle 1.6.0.

[pierreglaser]

Thank you for the reproducer @jrbourbeau.

The only way to serialize classes using the standard-library pickle implementation (not cloudpickle) is to serialize them by reference, which means that pickle can only serialize importable/top-level classes. cloudpickle implements extra-logic to serialize dynamically defined classes- such classes are serialized "by value", meaning their state and methods are fully serialized so that the classes can be reconstructed back in the receiving process. However, cloudpickle mimics pickle's behavior when serializing importable classes, e.g serializes them by reference (for compatibility purposes). Because in your case, Foo is importable, it is serialized by reference, and no class-level state is passed during serialization.

#391 Proposes to allow serialization by value for arbitrary classes. I need to find the time for reviewing it, but I'm in favor of adding this feature. As a temporary.

As a temporary solution, you can trick cloudpickle into treating your class as dynamic by changing Foo.__module__ to __main__. But it may imply some side effects coming from your specific application that I am not aware of.

[jrbourbeau]

Thank you for the detailed explanation @pierreglaser! This additional context helps a lot and I'll make sure to pass the temporary workaround to the user over in dask/distributed#4233.

I'll close this issue as a duplicate of #206 (which I missed when looking through issues earlier). Thanks again