[SPARK-25079][python] update python3 executable to 3.6.x by shaneknapp · Pull Request #24266 · apache/spark

shaneknapp · 2019-04-01T22:50:04Z

What changes were proposed in this pull request?

have jenkins test against python3.6 (instead of 3.4).

How was this patch tested?

extensive testing on both the centos and ubuntu jenkins workers.

NOTE: this will need to be backported to all active branches.

HyukjinKwon · 2019-04-01T22:53:04Z

Thanks for taking a look for this @shaneknapp!

SparkQA · 2019-04-02T03:45:40Z

Test build #104173 has finished for PR 24266 at commit 1d14800.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2019-04-02T04:19:45Z

huh. looks like pandas got downgraded for python2.7... i'll check the versions installed on the jenkins workers tomorrow.

we've never supported pandas for pypy, so those skipped tests are expected.

however, all of the 3.6 tests passed! woo!

HyukjinKwon · 2019-04-02T04:27:16Z

Yea, that sounds all good. If I haven't missed something, Pandas of Python 2.7 was already low version too. Nice to see Python 3.6 got all passed. We have picked one version of Python 3.x so far so I don't think it matters the test coverage by this change - should be fine.

felixcheung

LGTM

shaneknapp · 2019-04-02T18:11:16Z

ok, fixed the centos workers' python2.7/pandas versioning:

$ pssh -h centos_workers.txt -t 0 -i "export PATH=/home/anaconda/bin:$PATH; python2.7 -c 'import pandas; print pandas.__version__'"
[1] 11:10:37 [SUCCESS] amp-jenkins-worker-03
0.19.2
[2] 11:10:37 [SUCCESS] amp-jenkins-worker-02
0.19.2
[3] 11:10:37 [SUCCESS] amp-jenkins-worker-05
0.19.2
[4] 11:10:37 [SUCCESS] amp-jenkins-worker-04
0.19.2
[5] 11:10:37 [SUCCESS] amp-jenkins-worker-06
0.19.2
[6] 11:10:37 [SUCCESS] amp-jenkins-worker-01
0.19.2

will retrigger a test now.

shaneknapp · 2019-04-02T18:11:24Z

test this please

HyukjinKwon

LGTM

SparkQA · 2019-04-02T22:17:41Z

Test build #104220 has finished for PR 24266 at commit 1d14800.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2019-04-02T22:21:43Z

argh... spurious non-python failure.

shaneknapp · 2019-04-02T22:21:50Z

test this please

srowen · 2019-04-02T23:37:30Z

 test_env = "amplab_jenkins"
 # add path for Python3 in Jenkins if we're calling from a Jenkins machine
- os.environ["PATH"] = "/home/anaconda/envs/py3k/bin:" + os.environ.get("PATH")
+ # TODO(sknapp): s/py36/py3k before merging!


OK by me; does this need to change before you merge?

yeah... this is so i can test against the py36 conda environment on the jenkins workers. right before i merge, i'll pause jenkins, update the conda env name, push my revert and then let jenkins build again.
folks will need to update their PRs post-merge as well.
i'm waiting on @BryanCutler and the arrow 0.12.1 release before doing any of this, however. probably early to mid next week.
there will be a detailed email going out to dev@ about this on monday.

this also reminds me that one day, i would love to go through everything and remove all mention of amplab. that project has been over since 2016! :)

quick update: py36 will actually remain the path for this python env on master...

SparkQA · 2019-04-03T02:40:54Z

Test build #104223 has finished for PR 24266 at commit 1d14800.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-03T03:04:31Z

Test build #104222 has finished for PR 24266 at commit 1d14800.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-04-03T16:20:23Z

Hi, @shaneknapp . Looks nice~

Finished test(python3.6): pyspark.sql.dataframe (50s)
Finished test(python3.6): pyspark.sql.functions (47s)
Tests passed in 1088 seconds

BTW, from the log, I'm wondering if we can change pip installation test together. It's using python 3.5. If it's beyond of the scope of this PR, it's okay for me.

...
Testing pip installation with python 3.5
Using /tmp/tmp.yP79vT0xxQ for virtualenv
Fetching package metadata ...........
Solving package specifications: .
Package plan for installation in environment /tmp/tmp.yP79vT0xxQ/3.5:

shaneknapp · 2019-04-03T16:47:26Z

Hi, @shaneknapp . Looks nice~
Finished test(python3.6): pyspark.sql.dataframe (50s)
Finished test(python3.6): pyspark.sql.functions (47s)
Tests passed in 1088 seconds
BTW, from the log, I'm wondering if we can change pip installation test together. It's using python 3.5. If it's beyond of the scope of this PR, it's okay for me.

yep, definitely beyond the scope of this PR.

shaneknapp · 2019-04-03T16:49:50Z

Test build #104223 has finished for PR 24266 at commit 1d14800.
This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

i'm really curious why this is suddenly popping up (from the console output of this build):

UnknownTimeZoneError: 'US/Pacific-New'

i noticed this yesterday, and went through and made sure that pytz was installed and at the same version across all workers... which it is. this failed on amp-jenkins-worker-05, but the next build passed on amp-jenkins-worker-02.

i'll poke around a bit more and see if i can't get to the bottom of things today... if not today, then next week as i'm out thursday-sunday.

shaneknapp · 2019-04-03T16:51:28Z

test this please

dongjoon-hyun · 2019-04-03T16:55:16Z

Oh, the test failures are on the master branch. I agree that it's pytz issue.

 File "/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py", line 178, in timezone
raise UnknownTimeZoneError(zone)
UnknownTimeZoneError: 'US/Pacific-New'

Locally, it works in my Mac (Python 2.7.10).

>>> from pytz import timezone
>>> import pytz
>>> timezone('US/Pacific-New')
<DstTzInfo 'US/Pacific-New' PST-1 day, 16:00:00 STD>

@HyukjinKwon , @BryanCutler . Could you take a look on the master branch Python UT failures (5692 ~ 5696) to unblock this PR?

dongjoon-hyun · 2019-04-03T17:58:16Z

Oh, it fails on branch-2.4, too.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.4-test-sbt-hadoop-2.7/356/console

SparkQA · 2019-04-03T21:48:25Z

Test build #104256 has finished for PR 24266 at commit 1d14800.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-04-04T13:40:49Z

So, hm, are all the tests failing already due to a Python update? I don't necessarily want to roll that back but is it a matter of updating pytz too everywhere? does it need to be updated separately for Python 2 vs 3?

squito · 2019-04-05T15:58:44Z

btw I filed a separate jira on the US/Pacific-New thing: https://issues.apache.org/jira/browse/SPARK-27389. I couldn't figure out what was going on either ... It does seem like US/Pacific-New is some weird TZ that should never have really existed and isn't supported everywhere, but I can't figure out what has changed that is causing it to suddenly fail this.

dongjoon-hyun · 2019-04-07T00:54:23Z

Retest this please.

SparkQA · 2019-04-07T05:08:20Z

Test build #104349 has finished for PR 24266 at commit 1d14800.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-04-10T21:59:00Z

Test build #104490 has finished for PR 24266 at commit 1d14800.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2019-04-10T22:01:00Z

omfg lol:

 File "/home/anaconda/envs/py36/lib/python3.6/site-packages/pytz/__init__.py", line 178, in timezone
raise UnknownTimeZoneError(zone)
pytz.exceptions.UnknownTimeZoneError: 'US/Pacific-New'

(˚Õ˚)ر ~~~~╚╩╩╝

BryanCutler · 2019-04-11T16:27:12Z

did this worker revolt against the fix?! I guess lets try this again

BryanCutler · 2019-04-11T16:27:21Z

retest this please

shaneknapp · 2019-04-11T16:30:07Z

python3.6 revolted. this is a "new" failure... did something change in the tests?

BryanCutler · 2019-04-11T16:39:04Z

did something change in the tests?

Only #24306 but it has nothing to do with timestamps or timezones..

shaneknapp · 2019-04-11T17:55:54Z

did something change in the tests?
Only #24306 but it has nothing to do with timestamps or timezones..

the tests pass when i run them manually on the ubuntu workers... i can't wait to reimage the centos boxes.

anyways, i'll patch up the centos workers later today and this should go away.

SparkQA · 2019-04-11T21:32:14Z

Test build #104521 has finished for PR 24266 at commit 1d14800.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2019-04-12T17:43:56Z

Test build #104521 has finished for PR 24266 at commit 1d14800.

so, this build passed on amp-jenkins-worker-02, and failed on -05 (which was one of the workers that initially had this issue).

i just finished manually running these tests on both of these workers, as well as my ubuntu testing worker, and both the ubuntu box and -02 passed, while -05 failed (as expected).

the python 3.6 envs on all three machines are identical, which i confirmed by checking conda list --explicit and pip list.

i'm now resorting to strace to see if i can spot any system-level differences. :(

dongjoon-hyun · 2019-04-12T17:53:33Z

Thank you so much for spending much time on this, @shaneknapp !

shaneknapp · 2019-04-12T19:15:39Z

well, i got the tests to pass on worker-05:

sknapp@amp-jenkins-worker-05 US]$ sudo mv /usr/share/zoneinfo/US/Pacific-New /tmp

however, that tzdata file is present and in the same location on the other workers that AREN'T failing tests.

¯\_(ツ)_/¯

shaneknapp · 2019-04-12T20:29:18Z

well, i got the tests to pass on worker-05:
sknapp@amp-jenkins-worker-05 US]$ sudo mv /usr/share/zoneinfo/US/Pacific-New /tmp
however, that tzdata file is present and in the same location on the other workers that AREN'T failing tests.
¯\_(ツ)_/¯

so, to test whether or not (re)moving the Pacific-New timezone data file to /tmp actually works on all machines, i did the same thing on -03:

sknapp@amp-jenkins-worker-03 US]$ sudo mv /usr/share/zoneinfo/US/Pacific-New /tmp

i then ran the python tests, and they failed.

seriously, i think my brain is about to (im|ex)plode.

edit:

to get this to run successfully on -03, i had to:

manually edit anaconda/envs/py36/lib/python3.6/site-packages/pytz/__init__.py and add US/Pacific-New to both all_timezones and common_timezones
recompile the package (python3.6 -m compileall .)
created a symlink for Pacific-New to point at Pacific in anaconda/envs/py36/lib/python3.6/site-packages/pytz/zoneinfo/US.

for now i remain, and most likely for a very long time, completely confused.

shaneknapp · 2019-04-12T20:39:37Z

test this please

SparkQA · 2019-04-13T01:28:11Z

Test build #104561 has finished for PR 24266 at commit 1d14800.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2019-04-13T18:47:40Z

ok, i think this is ready to be merge/deployed next week. i'll send out an email to dev@ and get this scheduled.

…

On Fri, Apr 12, 2019 at 6:30 PM UCB AMPLab ***@***.***> wrote: Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104561/ Test PASSed. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24266 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABiDrJ8WxD9o9CqWeMUkCJFMUD_ey-Dxks5vgTMugaJpZM4cWyjH> .

SparkQA · 2019-04-17T21:38:09Z

Test build #104671 has finished for PR 24266 at commit ba77b41.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2019-04-17T22:17:03Z

test this please

SparkQA · 2019-04-18T03:05:13Z

Test build #104680 has finished for PR 24266 at commit ba77b41.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-04-18T12:14:09Z

retest this please

HyukjinKwon · 2019-04-18T12:14:38Z

I think it's good to go

shaneknapp · 2019-04-18T15:38:04Z

I think it's good to go

yep, i think so too! let's wait for the build to finish...

SparkQA · 2019-04-18T18:17:58Z

Test build #104700 has finished for PR 24266 at commit ba77b41.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-04-19T01:02:33Z

Merged to master.

Thank you so much, @shaneknapp

dongjoon-hyun · 2019-04-19T15:48:27Z

Thank you, @shaneknapp and @HyukjinKwon .

Since this is an official Python 3.6 community support, can we have this on branch-2.4/2.3 too?
I monitored #24379 and #24380 , but I'm not sure which one will land on branch-2.4/2.3.

cc @dbtsai and @mattf-apache

shaneknapp · 2019-04-19T16:24:59Z

Thank you, @shaneknapp and @HyukjinKwon .
Since this is an official Python 3.6 community support, can we have this on branch-2.4/2.3 too?
I monitored #24379 and #24380 , but I'm not sure which one will land on branch-2.4/2.3.
cc @dbtsai and @mattf-apache

since @HyukjinKwon jumped the gun a little bit and merged this to master, i'm going to try and accelerate the schedule and deploy the new python envs for these two branches today. if not today, then monday.

dongjoon-hyun · 2019-04-19T17:43:05Z

Thank you!

update python3 executable to 3.6.x
1d14800

shaneknapp mentioned this pull request Apr 1, 2019
[SPARK-25079][PYTHON] preparing for python 3.5 bump #22061
Closed

felixcheung reviewed Apr 2, 2019
View reviewed changes

BryanCutler approved these changes Apr 2, 2019
View reviewed changes

HyukjinKwon approved these changes Apr 2, 2019
View reviewed changes

srowen reviewed Apr 2, 2019
View reviewed changes

dongjoon-hyun approved these changes Apr 3, 2019
View reviewed changes

srowen mentioned this pull request Apr 4, 2019
[SPARK-27328][SQL] Add 'deprecated' in ExpressionDescription for extended usage and SQL doc #24259
Closed

update comment for future shane and new jenkins workers
ba77b41

HyukjinKwon closed this in e1ece6aApr 19, 2019

shaneknapp deleted the updating-python3-executable branch April 19, 2019 16:47

Uh oh!

Conversation

shaneknapp commented Apr 1, 2019• edited by dongjoon-hyun Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

HyukjinKwon commented Apr 1, 2019

Uh oh!

SparkQA commented Apr 2, 2019

Uh oh!

shaneknapp commented Apr 2, 2019

Uh oh!

HyukjinKwon commented Apr 2, 2019

Uh oh!

felixcheung left a comment

Choose a reason for hiding this comment

Uh oh!

shaneknapp commented Apr 2, 2019

Uh oh!

shaneknapp commented Apr 2, 2019

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 2, 2019

Uh oh!

shaneknapp commented Apr 2, 2019

Uh oh!

shaneknapp commented Apr 2, 2019

Uh oh!

srowenApr 2, 2019

Choose a reason for hiding this comment

Uh oh!

shaneknappApr 2, 2019

Choose a reason for hiding this comment

Uh oh!

shaneknappApr 3, 2019

Choose a reason for hiding this comment

Uh oh!

shaneknappApr 17, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 3, 2019

Uh oh!

SparkQA commented Apr 3, 2019

Uh oh!

dongjoon-hyun commented Apr 3, 2019

Uh oh!

shaneknapp commented Apr 3, 2019

Uh oh!

shaneknapp commented Apr 3, 2019

Uh oh!

shaneknapp commented Apr 3, 2019

Uh oh!

dongjoon-hyun commented Apr 3, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Apr 3, 2019

Uh oh!

SparkQA commented Apr 3, 2019

Uh oh!

srowen commented Apr 4, 2019

Uh oh!

squito commented Apr 5, 2019

Uh oh!

dongjoon-hyun commented Apr 7, 2019

Uh oh!

SparkQA commented Apr 7, 2019

Uh oh!

SparkQA commented Apr 10, 2019

Uh oh!

shaneknapp commented Apr 10, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BryanCutler commented Apr 11, 2019

Uh oh!

BryanCutler commented Apr 11, 2019

Uh oh!

shaneknapp commented Apr 11, 2019

shaneknapp commented Apr 1, 2019•
edited by dongjoon-hyun
Loading

dongjoon-hyun commented Apr 3, 2019•
edited
Loading

shaneknapp commented Apr 10, 2019•
edited
Loading

shaneknapp commented Apr 12, 2019•
edited
Loading

shaneknapp commented Apr 12, 2019•
edited
Loading

shaneknapp commented Apr 12, 2019•
edited
Loading