[SPARK-25079][python][branch-2.4] update python3 executable to 3.6.x by shaneknapp · Pull Request #24379 · apache/spark

shaneknapp · 2019-04-15T20:39:21Z

What changes were proposed in this pull request?

have jenkins test against python3.6 (instead of 3.4).

How was this patch tested?

extensive testing on both the centos and ubuntu jenkins workers revealed that 2.4 doesn't like python 3.6... :(

NOTE: this is just for branch-2.4

PLEASE DO NOT MERGE

shaneknapp · 2019-04-15T23:04:58Z

@BryanCutler the test failures should be epic. ;)

BryanCutler · 2019-04-15T23:47:18Z

If it's gonna fail, might as well be epic \m/

shaneknapp · 2019-04-16T00:46:25Z

If it's gonna fail, might as well be epic \m/

hopefully it's not as epic as this:
https://www.usenix.org/conference/lisa16/conference-program/presentation/kuroda

SparkQA · 2019-04-16T00:50:23Z

Test build #104601 has finished for PR 24379 at commit a35a2cf.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

BryanCutler · 2019-04-16T00:54:39Z

Well, it's not too bad.. at least it didn't explode. Was the Pandas version upgraded also?

shaneknapp · 2019-04-16T01:59:24Z

Well, it's not too bad.. at least it didn't explode. Was the Pandas version upgraded also?

in this particular env, pandas is @ 0.24.2

BryanCutler · 2019-04-16T04:25:53Z

Is it possible to keep pandas and pyarrow versions the same as before (0.19.2 and 0.8.0) for envs of branches 2.3/2.4 or do they need to share the same env as master?

The failures here have been fixed in master from various PRs, but not backported. It's possible to apply them, but it would take some time and could be a bit risky..

shaneknapp · 2019-04-16T17:16:01Z

Is it possible to keep pandas and pyarrow versions the same as before (0.19.2 and 0.8.0) for envs of branches 2.3/2.4 or do they need to share the same env as master?

they need to share the same env as master (or we change all of the testing framework for all branches to create temporary python envs for each branch.... which isn't actually a horrible idea but a much bigger project).

regarding pandas 0.19.2, it seems that pandas 0.24.2 is the minimum according to conda forge?

(output below trimmed for readability)

$ conda install -c conda-forge pyarrow=0.12.1
<snip>
added / updated specs:
- pyarrow=0.12.1
<snip>
The following NEW packages will be INSTALLED:
arrow-cpp conda-forge/linux-64::arrow-cpp-0.12.1-py36h0e61e49_0
mkl_fft conda-forge/linux-64::mkl_fft-1.0.11-py36h14c3975_0
mkl_random conda-forge/linux-64::mkl_random-1.0.2-py36h637b7d7_2
parquet-cpp conda-forge/noarch::parquet-cpp-1.5.1-4
pyarrow conda-forge/linux-64::pyarrow-0.12.1-py36hbbcf98d_0
The following packages will be UPDATED:
numpy 1.11.3-py36h7e9f1db_12 --> 1.16.2-py36h7e9f1db_0
numpy-base 1.11.3-py36hde5b4d6_12 --> 1.16.2-py36hde5b4d6_0
pandas 0.19.2-np111py36_1 --> 0.24.2-py36hf484d3e_0 <----- NOOOOO!

i'll try and see if i can get pandas to 0.19.2, but it's looking to be kinda difficult. i hacked a conda spec file and manually set pandas to 0.19.2 and will run the tests against it and see what happens.

The failures here have been fixed in master from various PRs, but not backported. It's possible to apply them, but it would take some time and could be a bit risky..

i'll take time/risk vs not having any python test coverage for 2.3 and 2.4... but we'll need a commitment from dev@ to help make this stuff work.

shaneknapp · 2019-04-16T17:35:35Z

obligatory xkcd comic:

(i actually have this printed out and hanging in my exterior-facing office window)

shaneknapp · 2019-04-16T20:08:32Z

ok... i got py36 + pandas 0.19.2 + pyarrow 0.12.1 to happily install in a test conda env, then i built a spark 2.4.2 dist and ran python/run-tests against the tarball.

pyspark.sql.tests + py36 + pandas 0.19.2 + pyarrow 0.12.1: 346 tests, errors=21
pyspark.sql.tests + py36 + pandas 0.24.2 + pyarrow 0.12.1: 346 tests, errors=24

the tests that passed w/pandas 0.19.2 (and failed w/0.24.2) are:

test_column_order (pyspark.sql.tests.GroupedMapPandasUDFTests)
test_complex_groupby (pyspark.sql.tests.GroupedMapPandasUDFTests)
test_udf_with_key (pyspark.sql.tests.GroupedMapPandasUDFTests)

the tests that failed for both versions of pandas:

test_createDataFrame_column_name_encoding (pyspark.sql.tests.EncryptionArrowTests)
test_createDataFrame_does_not_modify_input (pyspark.sql.tests.EncryptionArrowTests)
test_createDataFrame_respect_session_timezone (pyspark.sql.tests.EncryptionArrowTests)
test_createDataFrame_toggle (pyspark.sql.tests.EncryptionArrowTests)
test_createDataFrame_with_array_type (pyspark.sql.tests.EncryptionArrowTests)
test_createDataFrame_with_incorrect_schema (pyspark.sql.tests.ArrowTests)
test_createDataFrame_with_incorrect_schema (pyspark.sql.tests.EncryptionArrowTests)
test_createDataFrame_with_int_col_names (pyspark.sql.tests.EncryptionArrowTests)
test_createDataFrame_with_names (pyspark.sql.tests.EncryptionArrowTests)
test_createDataFrame_with_schema (pyspark.sql.tests.ArrowTests)
test_createDataFrame_with_schema (pyspark.sql.tests.EncryptionArrowTests)
test_null_conversion (pyspark.sql.tests.ArrowTests)
test_null_conversion (pyspark.sql.tests.EncryptionArrowTests)
test_pandas_round_trip (pyspark.sql.tests.ArrowTests)
test_pandas_round_trip (pyspark.sql.tests.EncryptionArrowTests)
test_timestamp_dst (pyspark.sql.tests.EncryptionArrowTests)
test_toPandas_arrow_toggle (pyspark.sql.tests.ArrowTests)
test_toPandas_arrow_toggle (pyspark.sql.tests.EncryptionArrowTests)
test_toPandas_respect_session_timezone (pyspark.sql.tests.ArrowTests)
test_toPandas_respect_session_timezone (pyspark.sql.tests.EncryptionArrowTests)
test_vectorized_udf_dates (pyspark.sql.tests.ScalarPandasUDFTests)

BryanCutler · 2019-04-16T20:53:05Z

I think the 2.3/2.4 branches will have to stay with pyarrow 0.8.0 and pandas 0.19.2 for the tests to pass. There were a few fixes that were done in master to make later versions of pyarrow work. These could be applied to the branches, but would have to be done manually. The risk is that we could end up breaking support for pyarrow 0.8.0, which would be pretty bad.

@shaneknapp do you think you can try pyspark.sql.tests + py36 + pandas 0.19.2 + pyarrow 0.8.0 for the 2.4 branch? I think that should work. Sorry, I should have explained that better before..

shaneknapp · 2019-04-16T21:39:43Z

I think the 2.3/2.4 branches will have to stay with pyarrow 0.8.0 and pandas 0.19.2 for the tests to pass. There were a few fixes that were done in master to make later versions of pyarrow work. These could be applied to the branches, but would have to be done manually. The risk is that we could end up breaking support for pyarrow 0.8.0, which would be pretty bad.

edit: i guess py36 does support pyarrow 0.8.0!

https://conda.anaconda.org/conda-forge/linux-64/pyarrow-0.8.0-py36_0.tar.bz2

@shaneknapp do you think you can try pyspark.sql.tests + py36 + pandas 0.19.2 + pyarrow 0.8.0 for the 2.4 branch? I think that should work. Sorry, I should have explained that better before..

sure. give me a couple of hours to make the env/run tests.

shaneknapp · 2019-04-16T21:50:20Z

I think the 2.3/2.4 branches will have to stay with pyarrow 0.8.0 and pandas 0.19.2 for the tests to pass. There were a few fixes that were done in master to make later versions of pyarrow work. These could be applied to the branches, but would have to be done manually. The risk is that we could end up breaking support for pyarrow 0.8.0, which would be pretty bad.
@shaneknapp do you think you can try pyspark.sql.tests + py36 + pandas 0.19.2 + pyarrow 0.8.0 for the 2.4 branch? I think that should work. Sorry, I should have explained that better before..

also, this means we'll need 3 python envs to test everything.

py27: master, 2.3, 2.4
py36 + pandas 0.19.2 + pyarrow 0.8.0: 2.3, 2.4
py36 + pandas 0.24.2 (or 0.19.2) + pyarrow 0.12.1: master

i'm not completely opposed, as this will most likely be easier than backporting and fixing the bugs in 2.3 and 2.4...

...but i would really like to stop at 3 python envs. it's already annoying enough as-is managing this stuff. :)

BryanCutler · 2019-04-16T22:04:17Z

sure. give me a couple of hours to make the env/run tests.

Cool, thanks!

py36 + pandas 0.19.2 + pyarrow 0.8.0: 2.3, 2.4

This will be be best to avoid making patches and should work as is (fingers crossed)

py36 + pandas 0.24.2 (or 0.19.2) + pyarrow 0.12.1: master

I'm not sure how nice pandas 0.19.2 will play with others, so it might be a good idea to use 0.24.2 but maybe we should post to the dev list about this

shaneknapp · 2019-04-16T22:05:08Z

I'm not sure how nice pandas 0.19.2 will play with others, so it might be a good idea to use 0.24.2 but maybe we should post to the dev list about this

tbh, i'm totally fine to go w/0.24.2 on master as all the tests are passing.

shaneknapp · 2019-04-16T22:28:46Z

sure. give me a couple of hours to make the env/run tests.
Cool, thanks!

tests running now... once 2.4 is done, i'll re-build and test this against 2.3.

shaneknapp · 2019-04-16T22:42:00Z

woot!

Tests passed in 1275 seconds

now on to 2.3

shaneknapp · 2019-04-16T23:51:15Z

2.3 passed!

Tests passed in 1148 seconds

i'll get these new envs created on the workers and update both this PR and #24380 by EOW.

BryanCutler · 2019-04-17T01:12:43Z

Woohoo! Thanks Shane!!

…

On Tue, Apr 16, 2019, 4:52 PM shane ***@***.***> wrote: 2.3 passed! Tests passed in 1148 seconds i'll get these new envs created on the workers and update both this PR and #24380 <#24380> by EOW. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24379 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEUwdW0wZ1tgkjlMcGnaf9K6D3WdyRvhks5vhmI4gaJpZM4cw1eh> .

SparkQA · 2019-04-17T17:46:01Z

Test build #104673 has started for PR 24379 at commit 6a3b3c6.

shaneknapp · 2019-04-17T17:48:19Z

test this please

SparkQA · 2019-04-17T22:24:32Z

Test build #104676 has finished for PR 24379 at commit d17e8fc.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2019-04-17T23:06:16Z

test this please

SparkQA · 2019-04-18T03:44:24Z

Test build #104683 has finished for PR 24379 at commit d17e8fc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2019-04-18T15:38:51Z

test this please

SparkQA · 2019-04-18T21:05:34Z

Test build #104709 has finished for PR 24379 at commit d17e8fc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-04-19T01:05:15Z

I added a prefix in the title cus I was about to merge .. :D

shaneknapp · 2019-04-19T02:44:26Z

i was panicking as i was opening this PR, hoping that you didn't merge it (and the other one). ;)

## What changes were proposed in this pull request? have jenkins test against python3.6 (instead of 3.4). ## How was this patch tested? extensive testing on both the centos and ubuntu jenkins workers revealed that 2.4 doesn't like python 3.6... :( NOTE: this is just for branch-2.4 PLEASE DO NOT MERGE Closes#24379 from shaneknapp/update-python-executable. Authored-by: shane knapp <incomplete@gmail.com> Signed-off-by: shane knapp <incomplete@gmail.com>

AmplabJenkins · 2019-04-19T16:53:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10065/
Test FAILed.

shaneknapp · 2019-04-19T16:54:41Z

ignore the failed build -- to be expected

## What changes were proposed in this pull request? have jenkins test against python3.6 (instead of 3.4). ## How was this patch tested? extensive testing on both the centos and ubuntu jenkins workers revealed that 2.4 doesn't like python 3.6... :( NOTE: this is just for branch-2.4 PLEASE DO NOT MERGE Closesapache#24379 from shaneknapp/update-python-executable. Authored-by: shane knapp <incomplete@gmail.com> Signed-off-by: shane knapp <incomplete@gmail.com>

update python3 executable to 3.6.x
a35a2cf

updated path for testing
6a3b3c6

s/36/3k
d17e8fc

HyukjinKwon changed the title ~~[SPARK-25079][python][branch-2.4] update python3 executable to 3.6.x~~[DO-NOT-MERGE][SPARK-25079][python][branch-2.4] update python3 executable to 3.6.xApr 19, 2019

dongjoon-hyun mentioned this pull request Apr 19, 2019
[SPARK-25079][python] update python3 executable to 3.6.x #24266
Closed

remove testing path
a7bd7bf

shaneknapp changed the title ~~[DO-NOT-MERGE][SPARK-25079][python][branch-2.4] update python3 executable to 3.6.x~~[SPARK-25079][python][branch-2.4] update python3 executable to 3.6.xApr 19, 2019

shaneknapp closed this Apr 19, 2019

shaneknapp deleted the update-python-executable branch April 19, 2019 16:47

Uh oh!

Conversation

shaneknapp commented Apr 15, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

shaneknapp commented Apr 15, 2019

Uh oh!

BryanCutler commented Apr 15, 2019

Uh oh!

shaneknapp commented Apr 16, 2019

Uh oh!

SparkQA commented Apr 16, 2019

Uh oh!

BryanCutler commented Apr 16, 2019

Uh oh!

shaneknapp commented Apr 16, 2019

Uh oh!

BryanCutler commented Apr 16, 2019

Uh oh!

shaneknapp commented Apr 16, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shaneknapp commented Apr 16, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shaneknapp commented Apr 16, 2019

Uh oh!

BryanCutler commented Apr 16, 2019

Uh oh!

shaneknapp commented Apr 16, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shaneknapp commented Apr 16, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BryanCutler commented Apr 16, 2019

Uh oh!

shaneknapp commented Apr 16, 2019

Uh oh!

shaneknapp commented Apr 16, 2019

Uh oh!

shaneknapp commented Apr 16, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shaneknapp commented Apr 16, 2019

Uh oh!

BryanCutler commented Apr 17, 2019 via email

Uh oh!

SparkQA commented Apr 17, 2019

Uh oh!

shaneknapp commented Apr 17, 2019

Uh oh!

SparkQA commented Apr 17, 2019

Uh oh!

shaneknapp commented Apr 17, 2019

Uh oh!

SparkQA commented Apr 18, 2019

Uh oh!

shaneknapp commented Apr 18, 2019

Uh oh!

SparkQA commented Apr 18, 2019

Uh oh!

HyukjinKwon commented Apr 19, 2019

Uh oh!

shaneknapp commented Apr 19, 2019

Uh oh!

AmplabJenkins commented Apr 19, 2019

Uh oh!

shaneknapp commented Apr 19, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

shaneknapp commented Apr 15, 2019•
edited
Loading

shaneknapp commented Apr 16, 2019•
edited
Loading

shaneknapp commented Apr 16, 2019•
edited
Loading

shaneknapp commented Apr 16, 2019•
edited
Loading

shaneknapp commented Apr 16, 2019•
edited
Loading

shaneknapp commented Apr 16, 2019•
edited
Loading