[SPARK-6883] Rely on cloudpickle in pyspark

[rgbkrk]

This package was based on the version that was in pyspark, with some bug fixes contributed by folks as well as more comprehensive tests. I'm opening this issue to track the upstream work in PySpark.

[ogrisel]

Here is the link: https://issues.apache.org/jira/browse/SPARK-6883

[Peque]

@JoshRosen is there any update on this matter? 😊

[rgbkrk]

@Peque I brought over cloudpickle as a pseudo-vendored library in apache/spark#18282 then realized that I needed to update tests in pyspark (or fix things). I didn't have time to complete so I've dropped it and hope someone else can pick it up.

/cc @holdenk

[Peque]

@holdenk Friendly ping. 😇

[HyukjinKwon]

At least, Spark keeps matching it to specific version of cloudpickle for now. Now it's 0.8.0 as of apache/spark#23904

The problem is that we gotta make sure cloudpickle is installed on each nodes in a distributed environment but Spark currently has some holes about dependency management in PySpark side. Once that's fixed, Spark could relatively easily switch the fork to the proper dependency.

[Peque]

@HyukjinKwon Is there an issue to track those holes about dependency management in PySpark side?

[HyukjinKwon]

It's a general issue. So, there are multiple JIRAs to fix it. For instance, SPARK-13587. So, once we have such mechanism, we could internally use this and install cloudpickle out of box.

There might be other ways to fix it but this is my current thought :D.