Fix cloudpickle incompatibilities on early Python 3.5 versions

[pierreglaser]

Closes#360 . cloudpickle 1.4.0 is not compatible with early Python 3.5 versions.
This should fix it.

Note that I did not set up any CI for Python 3.5.[0-2], I simply tested it on my machine using fresh conda envs.

@vedran If you have some time, could you tell me if this branch fixes the problems that made you create #360?

I would be tempted to release a bugfix version by tonight since this bug completely breaks cloudpickle on Python 3.5.

[codecov]

Codecov Report

Merging #361 into master will not change coverage.
The diff coverage is n/a.

@@ Coverage Diff @@## master #361 +/- ##
=======================================
Coverage 92.95% 92.95% =======================================
Files 2 2 Lines 809 809 Branches 164 164 =======================================
Hits 752 752 Misses 29 29 Partials 28 28 

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e58c34b...e58c34b. Read the comment docs.

[ogrisel]

First batch of reviews:

[pierreglaser]

rebased.

[ogrisel]

Maybe it's not worth fighting with the CI too much to test Python 3.5.0... I will test this PR one more time locally.

[pierreglaser]

Cool.

[ogrisel]

How did you install Python 3.5.0 to run your tests? From source? I cannot find it with conda.

[pierreglaser]

let me know when if it works locally - if so, I'll clean up the CI additions in this PR.

[pierreglaser]

How did you install Python 3.5.0 to run your tests? From source? I cannot find it with conda.

Using miniconda2. But actually miniconda3 works too:

conda create -y -n c3tmp35 python=3.5.0

[ogrisel]

Ok I managed to run the tests of this PR successfully using python:3.5.0 from docker.

[ogrisel]

From your last CI run:

ERROR: Package 'cloudpickle' requires a different Python: 2.7.17 not in '>=3.5'

That's an interesting error message ;) I am not sure what it means.

Anyways, feel free to rollback the CI changes and merge this PR to make the release.

[pierreglaser]

Great. I'm quickly testing 3.5.1 and 3.5.2 while I'm at it, and i'll merge.

[ogrisel]

Actually I made a mistake when testing with Python 3.5.0. I had forgotten to install the typing_extensions package. If I do so I get the following failure:

___________________________________________________________________ CloudPickleTest.test_generic_extensions ____________________________________________________________________
self = <tests.cloudpickle_test.CloudPickleTest testMethod=test_generic_extensions>
deftest_generic_extensions(self):
typing_extensions = pytest.importorskip('typing_extensions')
objs = [
typing_extensions.Literal,
typing_extensions.Final,
typing_extensions.Literal['a'],
typing_extensions.Final[int],
]
for obj in objs:
depickled_obj = pickle_depickle(obj, protocol=self.protocol)
> assert depickled_obj == obj
tests/cloudpickle_test.py:2137: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/local/lib/python3.5/site-packages/typing_extensions.py:653: in __eq__
ifnotisinstance(other, Literal):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = typing_extensions.Literal, obj = typing_extensions.Literal
def__instancecheck__(self, obj):
> raise TypeError("Literal cannot be used with isinstance().")
E TypeError: Literal cannot be used with isinstance().
/usr/local/lib/python3.5/site-packages/typing_extensions.py:624: TypeError

[pierreglaser]

What kind of can of worms is this.....

[pierreglaser]

The test suite passes locally on cloudpickle 3.5.[0-6] for me. 3.5.9 (the one used by the CI) passes too. miniconda does not seem to have 3.5.[7-8] so it's harder for me to test those two versions.

Let's sleep on it until tomorrow morning and then release.

[ogrisel]

I confirm the tests now pass on my local 3.5.0. I will test each other 3.5.x version with docker and then review the diff.

• 3.5.0
• 3.5.1
• 3.5.2
• 3.5.3
• 3.5.4
• 3.5.5
• 3.5.6
• 3.5.7
• 3.5.8
• 3.5.9

[ogrisel]

For your information, here is the command line I use to run the tests with docker:

docker run --rm -v `pwd`:/io -ti python:3.5.2 \
bash -c "cd /io/ && pip install -e . psutil pytest typing_extensions && pytest -v"

[ogrisel]

I think we should restore TypeVar tracking, at least for recent Python versions (>= 3.6).

[ogrisel]

Why don't you track TypeVar defintions anymore? This seems unrelated to the change to support early 3.5.x.

Doesn't this change break ant test?

Edit: I see you removed test_pickle_dynamic_typevar_tracking.

[ogrisel]

Also won't this break unpickling objects pickled with cloudpickle 1.4.0?

[pierreglaser]

TyperVar instances are not weakreferable in Python 3.5.3..

[pierreglaser]

Also won't this break unpickling objects pickled with cloudpickle 1.4.0?

We can always restore the class_tracker_id for backward compat.

[ogrisel]

TyperVar instances are not weakreferable in Python 3.5.3..

I re-ran the tests with old Python 3.5.x and they pass...

[ogrisel]

Hum you are right they fail just for 3.5.3 ...

[ogrisel]

ok let me do a workaround for 3.5.3.

[ogrisel]

Ok I re-enabled dynamic TypeVar tracking and the tests still pass for all the 3 oldest versions of 3.5.x. Will try others in the 3.5.3-3.5.9 range at random.

[pierreglaser]

Ok I re-enabled dynamic TypeVar tracking and the tests still pass for all the 3 oldest versions of 3.5.x

I'm not sure we can. See #361 (comment)

[ogrisel]

Some more comments:

[ogrisel]

LGTM now! Thanks very much @pierreglaser.