[SPARK-5482][PySpark] Allow individual test suites in python/run-tests by potix2 · Pull Request #4269 · apache/spark

potix2 · 2015-01-29T13:40:08Z

Add options to run individual test suites in python/run-tests.

TODO

split the script into two, one that tests with a particular python version and another that loops over the versions and invokes that script.
~~rewrite in python~~

AmplabJenkins · 2015-01-29T13:42:10Z

Can one of the admins verify this patch?

AmplabJenkins · 2015-04-27T18:22:06Z

Can one of the admins verify this patch?

JoshRosen · 2015-05-31T19:31:24Z

Another useful refactoring would be to let us run the tests for only one of the supported Python versions instead of all of them. This is useful when augmenting the test suite to collect coverage metrics. To do this, maybe we should split the script into two, one that tests with a particular python version and another that loops over the versions and invokes that script.

potix2 · 2015-06-02T14:51:04Z

ok, I understand it. I try to split run-tests and rebase this branch onto master.

potix2 · 2015-06-03T02:23:35Z

@JoshRosen Should we define the list of supported python versions in a test script?
I found the below versions in a current python/run-tests.

2.6
3.4
pypy

JoshRosen · 2015-06-03T02:33:21Z

@potix2, yep, that's the right set of versions to test against. In the top-level test runner script, I think that we can check whether those versions are installed (using which) then build up a list of Python executables that we'll run against. So maybe by default we'll run against whatever subset of those are installed, but will have an option to manually specify just one version to run for manual testing / debugging.

By the way, if it's easier then I'd be fine with rewriting this file in Python rather than keeping it as bash.

potix2 · 2015-06-03T05:05:11Z

Thank you, @JoshRosen. I got it. There is no problem to rewrite in Python.

…lar python version

JoshRosen · 2015-06-18T20:28:26Z

Jenkins, this is ok to test.

AmplabJenkins · 2015-06-18T20:32:15Z

Merged build triggered.

AmplabJenkins · 2015-06-18T20:32:23Z

Merged build started.

SparkQA · 2015-06-18T20:33:00Z

Test build #35164 has started for PR 4269 at commit 5aa5aa5.

JoshRosen · 2015-06-18T21:21:39Z

Hey @potix2, thanks for updating this. I'm going to take a look at this today because I have an outstanding PR to improve dev/run-tests's support for conditionally skipping certain tests in the pull request builder (see #6866). Once we get this pull request merged, then I'll be able to update my PR to perform its optimizations for Python tests as well.

In the long run, it might be nice to rewrite this in Python, but I don't feel that such as rewrite necessarily has to be a blocker to merging this. If you're already working on a Python port, though, then just let me know and I'll be happy to wait for it.

JoshRosen · 2015-06-18T21:22:44Z

It looks like this pypy.sh file isn't invoked from anywhere; do we need it?

Also, we might want to remove the # -*- coding: utf-8 -*- lines in case Apache RAT complains about them.

ok, I remove them.

SparkQA · 2015-06-18T21:49:14Z

Test build #35164 has finished for PR 4269 at commit 5aa5aa5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-18T21:49:27Z

Merged build finished. Test FAILed.

JoshRosen · 2015-06-18T22:01:47Z

Jenkins, retest this please.

AmplabJenkins · 2015-06-18T22:02:16Z

Merged build triggered.

AmplabJenkins · 2015-06-18T22:02:24Z

Merged build started.

SparkQA · 2015-06-18T22:02:42Z

Test build #35179 has started for PR 4269 at commit 5aa5aa5.

SparkQA · 2015-06-18T23:50:06Z

Test build #35179 has finished for PR 4269 at commit 5aa5aa5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-18T23:50:22Z

Merged build finished. Test PASSed.

potix2 · 2015-06-19T00:55:53Z

@JoshRosen, I haven't started rewriting in python yet, so it's no problem that I make it to be out of scope for this PR.

AmplabJenkins · 2015-06-19T01:02:13Z

Merged build triggered.

AmplabJenkins · 2015-06-19T01:02:22Z

Merged build started.

SparkQA · 2015-06-19T01:05:35Z

Test build #35202 has started for PR 4269 at commit 36153f6.

SparkQA · 2015-06-19T01:05:42Z

Test build #35202 has finished for PR 4269 at commit 36153f6.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-19T01:05:43Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-06-19T01:22:15Z

Merged build triggered.

AmplabJenkins · 2015-06-19T01:22:22Z

Merged build started.

SparkQA · 2015-06-19T01:22:53Z

Test build #35206 has started for PR 4269 at commit 6dec246.

AmplabJenkins · 2015-06-19T01:32:12Z

Merged build triggered.

AmplabJenkins · 2015-06-19T01:32:22Z

Merged build started.

SparkQA · 2015-06-19T01:35:33Z

Test build #35208 has started for PR 4269 at commit e08f602.

SparkQA · 2015-06-19T02:38:00Z

Test build #35206 has finished for PR 4269 at commit 6dec246.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-19T02:38:13Z

Merged build finished. Test FAILed.

SparkQA · 2015-06-19T03:37:59Z

Test build #35208 has finished for PR 4269 at commit e08f602.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-19T03:38:15Z

Merged build finished. Test PASSed.

JoshRosen · 2015-06-19T06:35:51Z

Do we need to double-quote $1?

By the way, this prints an error when I run it on my Mac:
[joshrosen python (2c6a7d4...)]$ ./run-tests ./run-tests: line 33: [: too many arguments

JoshRosen · 2015-06-19T06:39:17Z

Actually, I find this bash code too difficult to understand / review and would rather just block on a complete Python rewrite. I don't want to spend the time to understand the bash-isms here if we're just going to rewrite anyways.

potix2 · 2015-06-19T22:27:24Z

Sorry to confuse you, I agree with you. As a first step, we should rewrite run-tests in Python, then append new features.

I took a look at #6866, I think it has some useful functions to rewrite bash code into Python. If you don't mind, I want to wait to merge it.

JoshRosen · 2015-06-22T18:08:44Z

My other PR (#6866) was merged, so I think this should be unblocked now. Which functions were you hoping to borrow from that script? If we're going to use some of those functions in two places, then I guess we can either duplicate them or move them to a common location and import them from there. If we do choose to extract them into their own module, I'd probably opt to create a new Python module inside of the dev folder (maybe dev/sparktestsupport) and place the new utility functions file under there.

JoshRosen · 2015-06-24T01:35:24Z

Hey @potix2, I've gone ahead and rewritten python/run-tests in Python in #6967. I haven't added any command-line flags for easily running individual suites, though. If you don't mind, can we wait until my PR is merged and then add those flags as part of this PR?

potix2 · 2015-06-24T03:00:39Z

Hi @JoshRosen, no problem. Thank you for considering this PR. I try to revise my code.

JoshRosen · 2015-06-24T20:35:56Z

I ended up incorporating this feature into my pull request because I needed something like this in order to trigger a subset of the Python tests in the pull request builder. Therefore, I recommend that we close this PR and continue discussion on mine. Thanks!

potix2 · 2015-06-24T22:30:38Z

All right. Thanks!

JoshRosen mentioned this pull request May 31, 2015
[MINOR] Enable PySpark SQL readerwriter and window tests #6542
Closed

potix2 added 3 commits June 14, 2015 18:39

[SPARK-5482][PySpark] Allow individual test suites in python/run-tests
5f6ee2b

Move conditional check into the test functions
0421f60

Split run-tests into the main script and runner scripts for a particu…
5aa5aa5
…lar python version

potix2 force-pushed the spark-5482 branch from 3e8ff43 to 5aa5aa5Compare June 17, 2015 11:41

potix2 changed the title ~~[SPARK-5482][PySpark] Allow individual test suites in python/run-tests~~[WIP][SPARK-5482][PySpark] Allow individual test suites in python/run-testsJun 17, 2015

JoshRosen reviewed Jun 18, 2015
View reviewed changes

Remove unnecessary comments
36153f6

Add license headers.
6dec246

potix2 changed the title ~~[WIP][SPARK-5482][PySpark] Allow individual test suites in python/run-tests~~[SPARK-5482][PySpark] Allow individual test suites in python/run-testsJun 19, 2015

Remove redundant shift
e08f602

JoshRosen reviewed Jun 19, 2015
View reviewed changes

JoshRosen mentioned this pull request Jun 24, 2015
[SPARK-8583] [SPARK-5482] [BUILD] Refactor python/run-tests to integrate with dev/run-tests module system #6967
Closed

potix2 closed this Jun 24, 2015

Uh oh!

Conversation

potix2 commented Jan 29, 2015

TODO

Uh oh!

AmplabJenkins commented Jan 29, 2015

Uh oh!

AmplabJenkins commented Apr 27, 2015

Uh oh!

JoshRosen commented May 31, 2015

Uh oh!

potix2 commented Jun 2, 2015

Uh oh!

potix2 commented Jun 3, 2015

Uh oh!

JoshRosen commented Jun 3, 2015

Uh oh!

potix2 commented Jun 3, 2015

Uh oh!

JoshRosen commented Jun 18, 2015

Uh oh!

AmplabJenkins commented Jun 18, 2015

Uh oh!

AmplabJenkins commented Jun 18, 2015

Uh oh!

SparkQA commented Jun 18, 2015

Uh oh!

JoshRosen commented Jun 18, 2015

Uh oh!

JoshRosenJun 18, 2015

Choose a reason for hiding this comment

Uh oh!

JoshRosenJun 18, 2015

Choose a reason for hiding this comment

Uh oh!

potix2Jun 18, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 18, 2015

Uh oh!

AmplabJenkins commented Jun 18, 2015

Uh oh!

JoshRosen commented Jun 18, 2015

Uh oh!

AmplabJenkins commented Jun 18, 2015

Uh oh!

AmplabJenkins commented Jun 18, 2015

Uh oh!

SparkQA commented Jun 18, 2015

Uh oh!

SparkQA commented Jun 18, 2015

Uh oh!

AmplabJenkins commented Jun 18, 2015

Uh oh!

potix2 commented Jun 19, 2015

Uh oh!

AmplabJenkins commented Jun 19, 2015

Uh oh!

AmplabJenkins commented Jun 19, 2015

Uh oh!

SparkQA commented Jun 19, 2015

Uh oh!

SparkQA commented Jun 19, 2015

Uh oh!

AmplabJenkins commented Jun 19, 2015

Uh oh!

AmplabJenkins commented Jun 19, 2015

Uh oh!

AmplabJenkins commented Jun 19, 2015

Uh oh!

SparkQA commented Jun 19, 2015

Uh oh!

AmplabJenkins commented Jun 19, 2015

Uh oh!

AmplabJenkins commented Jun 19, 2015

Uh oh!

SparkQA commented Jun 19, 2015

Uh oh!

SparkQA commented Jun 19, 2015

Uh oh!