Try to use pickle5 when available

[jakirkham]

As there are some improvements for buffer serialization in pickle protocol 5 and these can be opted in using a backports package on older Python versions. Try to leverage the backport package when available and fallback to the builtin pickle otherwise.

cc @quasiben@pentschev

[mrocklin]

When using protocol 5 I would expect us to select out the frames from the pickling process and send them along separately. Does using pickle5 provide a speed boost if we just use dumps/loads naively?

cc @pitrou@ogrisel

[mrocklin]

Closing this soonish if there are no further comments.

[pitrou]

When using protocol 5 I would expect us to select out the frames from the pickling process and send them along separately.

That would be the best reason to use it indeed.

Does using pickle5 provide a speed boost if we just use dumps/loads naively?

It may, depending exactly how pickle is invoked (some copies may be avoided, though not all). The best is if you can do some measurements and post them here :-)

[pitrou]

PS: are you still Python 2-compatible?

[mrocklin]

No, we dropped Python 2 a little while ago.

[jakirkham]

Does using pickle5 provide a speed boost if we just use dumps/loads naively?

This is what I was finding.

When using protocol 5 I would expect us to select out the frames from the pickling process and send them along separately.

That would certainly be better. Though it would be nice if we didn't have to maintain non-protocol 5 support as well.

[jakirkham]

Replacing with PR ( #3784 ), which actually handles out-of-band buffers. Admittedly it doesn't use pickle5 on older Python versions. However we would need cloudpickle to support to support pickle5 for that to work correctly ( cloudpipe/cloudpickle#179 ) anyways.

[jakirkham]

Also this has been extended to Python versions pre-3.8 with pickle5 with PR ( #3849 ).