Uh oh!
There was an error while loading. Please reload this page.
[SPARK-29536][PYTHON] Upgrade cloudpickle to 1.1.1 to support Python 3.8 - #26194
[SPARK-29536][PYTHON] Upgrade cloudpickle to 1.1.1 to support Python 3.8#26194HyukjinKwon wants to merge 8 commits into
Conversation
HyukjinKwon
commented
Oct 21, 2019
SparkQA
commented
Oct 21, 2019
Test build #112392 has finished for PR 26194 at commit
|
HyukjinKwon
commented
Oct 21, 2019
Hm .. let me investigate test failures further. |
BryanCutler
left a comment
There was a problem hiding this comment.
Sounds like a good reason to upgrade. Would now be a good time to stop including the file and use the official package or as a zip file in pyspark/lib?
viirya
left a comment
There was a problem hiding this comment.
Should be fine. I've tried this locally with 3.8 beta before.
@BryanCutler, yea, I think we should. I will take a look separately if you don't mind. It will need a fix here and there. One side concern is that it's pretty difficult to use it as the official package. We can try it with a zip I suspect .. |
HyukjinKwon
commented
Oct 22, 2019
retest this please |
SparkQA
commented
Oct 22, 2019
Test build #112421 has finished for PR 26194 at commit
|
SparkQA
commented
Oct 22, 2019
Test build #112424 has finished for PR 26194 at commit
|
SparkQA
commented
Oct 22, 2019
Test build #112429 has finished for PR 26194 at commit
|
SparkQA
commented
Oct 22, 2019
Test build #112432 has finished for PR 26194 at commit
|
SparkQA
commented
Oct 22, 2019
Test build #112438 has finished for PR 26194 at commit
|
The test failure is by cloudpipe/cloudpickle#278 . Let me stick to 1.1.1 for now since we didn't drop Python 2 support yet. |
SparkQA
commented
Oct 22, 2019
Test build #112441 has finished for PR 26194 at commit
|
SparkQA
commented
Oct 22, 2019
Test build #112443 has finished for PR 26194 at commit
|
HyukjinKwon
commented
Oct 22, 2019
Last commit is just adding metadata which does not affect pip check. I double checked the Python linter pass. Merged to master. Thanks, @srowen, @BryanCutler and @viirya |
jackhhh
commented
Jul 5, 2020
So, does it mean that I have to upgrade Spark and PySpark to 3.0 to use Python3.8? I tried to run with Spark 2.3.2 and PySpark 3.0.0 but it failed. |
HyukjinKwon
commented
Jul 6, 2020
Yup. |
dongjoon-hyun
commented
Oct 14, 2020
I updated SPARK-29536 by adding |
HyukjinKwon
commented
Oct 14, 2020
Thanks @dongjoon-hyun. |
gliptak
commented
Nov 28, 2020
Python 2 has reached EOL last year and should not be used anymore. This commit replaces all references to the "python" binary with the more explicit "python3" binary. If desired, the build can still be performed for Python 2 by settings the "PYTHON_EXECUTABLE" environment variable to an appropriate version. Additionally, python wheels are the preferred way to distribute python code (see https://packaging.python.org/discussions/wheel-vs-egg/). This commit additionally builds the job-server-python wheel. Spark-2.4 does not support python >= 3.8 (see apache/spark#26194) leading to failed test cases (TypeError: an integer is required (got type bytes)). If you encounter these issues try to state a python executable < 3.8 explicitly.
Python 2 has reached EOL last year and should not be used anymore. This commit replaces all references to the "python" binary with the more explicit "python3" binary. If desired, the build can still be performed for Python 2 by settings the "PYTHON_EXECUTABLE" environment variable to an appropriate version. Additionally, python wheels are the preferred way to distribute python code (see https://packaging.python.org/discussions/wheel-vs-egg/). This commit additionally builds the job-server-python wheel. Spark-2.4 does not support python >= 3.8 (see apache/spark#26194) leading to failed test cases (TypeError: an integer is required (got type bytes)). If you encounter these issues try to state a python executable < 3.8 explicitly.
Python 2 has reached EOL last year and should not be used anymore. This commit replaces all references to the "python" binary with the more explicit "python3" binary. If desired, the build can still be performed for Python 2 by settings the "PYTHON_EXECUTABLE" environment variable to an appropriate version. Additionally, python wheels are the preferred way to distribute python code (see https://packaging.python.org/discussions/wheel-vs-egg/). This commit additionally builds the job-server-python wheel. Spark-2.4 does not support python >= 3.8 (see apache/spark#26194) leading to failed test cases (TypeError: an integer is required (got type bytes)). If you encounter these issues try to state a python executable < 3.8 explicitly.
Python 2 has reached EOL last year and should not be used anymore. This commit replaces all references to the "python" binary with the more explicit "python3" binary. If desired, the build can still be performed for Python 2 by settings the "PYTHON_EXECUTABLE" environment variable to an appropriate version. Additionally, python wheels are the preferred way to distribute python code (see https://packaging.python.org/discussions/wheel-vs-egg/). This commit additionally builds the job-server-python wheel. Spark-2.4 does not support python >= 3.8 (see apache/spark#26194) leading to failed test cases (TypeError: an integer is required (got type bytes)). If you encounter these issues try to state a python executable < 3.8 explicitly.
Python 2 has reached EOL last year and should not be used anymore. This commit replaces all references to the "python" binary with the more explicit "python3" binary. If desired, the build can still be performed for Python 2 by settings the "PYTHON_EXECUTABLE" environment variable to an appropriate version. Additionally, python wheels are the preferred way to distribute python code (see https://packaging.python.org/discussions/wheel-vs-egg/). This commit additionally builds the job-server-python wheel. Spark-2.4 does not support python >= 3.8 (see apache/spark#26194) leading to failed test cases (TypeError: an integer is required (got type bytes)). If you encounter these issues try to state a python executable < 3.8 explicitly.
Python 2 has reached EOL last year and should not be used anymore. This commit replaces all references to the "python" binary with the more explicit "python3" binary. If desired, the build can still be performed for Python 2 by settings the "PYTHON_EXECUTABLE" environment variable to an appropriate version. Additionally, python wheels are the preferred way to distribute python code (see https://packaging.python.org/discussions/wheel-vs-egg/). This commit additionally builds the job-server-python wheel. Spark-2.4 does not support python >= 3.8 (see apache/spark#26194) leading to failed test cases (TypeError: an integer is required (got type bytes)). If you encounter these issues try to state a python executable < 3.8 explicitly.
Python 2 has reached EOL last year and should not be used anymore. This commit replaces all references to the "python" binary with the more explicit "python3" binary. If desired, the build can still be performed for Python 2 by settings the "PYTHON_EXECUTABLE" environment variable to an appropriate version. Additionally, python wheels are the preferred way to distribute python code (see https://packaging.python.org/discussions/wheel-vs-egg/). This commit additionally builds the job-server-python wheel. Spark-2.4 does not support python >= 3.8 (see apache/spark#26194) leading to failed test cases (TypeError: an integer is required (got type bytes)). If you encounter these issues try to state a python executable < 3.8 explicitly.
Python 2 has reached EOL last year and should not be used anymore. This commit replaces all references to the "python" binary with the more explicit "python3" binary. If desired, the build can still be performed for Python 2 by settings the "PYTHON_EXECUTABLE" environment variable to an appropriate version. Additionally, python wheels are the preferred way to distribute python code (see https://packaging.python.org/discussions/wheel-vs-egg/). This commit additionally builds the job-server-python wheel. Spark-2.4 does not support python >= 3.8 (see apache/spark#26194) leading to failed test cases (TypeError: an integer is required (got type bytes)). If you encounter these issues try to state a python executable < 3.8 explicitly.
Neeraj9697
commented
Mar 5, 2021
@HyukjinKwon But Getting below error message while initializing pyspark in windows |
dongjoon-hyun
commented
Mar 5, 2021
Hi, @Neeraj9697 . The following is the result of Apache Spark 3.1.1 on Python 3.9.1 on Mac.
|
dongjoon-hyun
commented
Mar 5, 2021
Could you file a JIRA issue with your specific information please, @Neeraj9697 ? |
Like the following, we officially documented that some features (like
|
HyukjinKwon
commented
Mar 5, 2021
Assuming from the error message, looks like you have multiple PySpark installed in your local and the paths messed up for some reasons. pyspark.cloudpickle is a package now but your error message said that this is a module (before this fix). |
Neeraj9697
commented
Mar 6, 2021
Thanks @HyukjinKwon and @dongjoon-hyun https://gist.github.com/Neeraj9697/f7aa1c6951bd3021eb48ccb919cbfd57 |
Great! Thank you for confirmation, @Neeraj9697 . For your new issue, please install
|
Python 2 has reached EOL last year and should not be used anymore. This commit replaces all references to the "python" binary with the more explicit "python3" binary. If desired, the build can still be performed for Python 2 by settings the "PYTHON_EXECUTABLE" environment variable to an appropriate version. Additionally, python wheels are the preferred way to distribute python code (see https://packaging.python.org/discussions/wheel-vs-egg/). This commit additionally builds the job-server-python wheel. Spark-2.4 does not support python >= 3.8 (see apache/spark#26194) leading to failed test cases (TypeError: an integer is required (got type bytes)). If you encounter these issues try to state a python executable < 3.8 explicitly.
What changes were proposed in this pull request?
Inline cloudpickle in PySpark to cloudpickle 1.1.1. See https://github.com/cloudpipe/cloudpickle/blob/v1.1.1/cloudpickle/cloudpickle.py
cloudpipe/cloudpickle#269 was added for Python 3.8 support (fixed from 1.1.0). Using 1.2.2 seems breaking PyPy 2 due to cloudpipe/cloudpickle#278 so this PR currently uses 1.1.1.
Once we drop Python 2, we can switch to the highest version.
Why are the changes needed?
positional-only arguments was newly introduced from Python 3.8 (see https://docs.python.org/3/whatsnew/3.8.html#positional-only-parameters)
Particularly the newly added argument to
types.CodeTypewas the problem (https://docs.python.org/3/whatsnew/3.8.html#changes-in-the-python-api):Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manually tested. Note that the optional dependency PyArrow looks not yet supporting Python 3.8; therefore, it was not tested. See "Details" below.
Details
cd python ./run-tests --python-executables=python3.8