[SPARK-6953] [PySpark] speed up python tests by davies · Pull Request #5427 · apache/spark

davies · 2015-04-08T19:46:54Z

This PR try to speed up some python tests:

tests.py 144s -> 103s -41s
mllib/classification.py 24s -> 17s -7s
mllib/regression.py 27s -> 15s -12s
mllib/tree.py 27s -> 13s -14s
mllib/tests.py 64s -> 31s -33s
streaming/tests.py 185s -> 84s -101s

Considering python3, the total saving will be 558s (almost 10 minutes) (core, and streaming run three times, mllib runs twice).

During testing, it will show used time for each test file:

Run core tests ...
Running test: pyspark/rdd.py ... ok (22s)
Running test: pyspark/context.py ... ok (16s)
Running test: pyspark/conf.py ... ok (4s)
Running test: pyspark/broadcast.py ... ok (4s)
Running test: pyspark/accumulators.py ... ok (4s)
Running test: pyspark/serializers.py ... ok (6s)
Running test: pyspark/profiler.py ... ok (5s)
Running test: pyspark/shuffle.py ... ok (1s)
Running test: pyspark/tests.py ... ok (103s) 144s

SparkQA · 2015-04-08T19:49:28Z

Test build #29883 has finished for PR 5427 at commit be23e1d.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-08T22:49:16Z

Test build #29893 has finished for PR 5427 at commit 0c49785.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Conflicts: python/pyspark/streaming/tests.py

SparkQA · 2015-04-16T01:39:20Z

Test build #30386 has finished for PR 5427 at commit 945a2b5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-16T01:46:14Z

Test build #30385 has finished for PR 5427 at commit fec2da2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAttribute(nameParts: Seq[String])
- trait CaseConversionExpression
- final class UTF8String extends Ordered[UTF8String] with Serializable
- case class Exchange(
- case class SortMergeJoin(
This patch does not change any dependencies.

mengxr · 2015-04-16T07:27:30Z

@davies Did you measure the speedup on each component?

davies · 2015-04-16T17:46:46Z

@mengxr I had update the difference in the description.

SparkQA · 2015-04-16T19:10:10Z

Test build #30427 has finished for PR 5427 at commit 55bb451.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Conflicts: python/pyspark/mllib/tests.py python/pyspark/mllib/tree.py python/pyspark/tests.py

SparkQA · 2015-04-17T00:47:50Z

Test build #30444 has finished for PR 5427 at commit aa39c55.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-17T07:17:53Z

Test build #30458 has finished for PR 5427 at commit 2654bfd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-18T09:15:35Z

Test build #30516 has finished for PR 5427 at commit 2654bfd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

mengxr · 2015-04-21T07:10:31Z

MLlib changes look good to me.

[SPARK-6953] [PySpark] speed up python tests Conflicts: python/pyspark/streaming/tests.py

[SPARK-6953] [PySpark] speed up python tests Signed-off-by: Reynold Xin <rxin@databricks.com> Conflicts: python/pyspark/streaming/tests.py (cherry picked from commit 21b15f5) Signed-off-by: Reynold Xin <rxin@databricks.com>

rxin · 2015-04-21T07:13:56Z

I brought this up to date at #5605

This PR try to speed up some python tests: ``` tests.py 144s -> 103s -41s mllib/classification.py 24s -> 17s -7s mllib/regression.py 27s -> 15s -12s mllib/tree.py 27s -> 13s -14s mllib/tests.py 64s -> 31s -33s streaming/tests.py 185s -> 84s -101s ``` Considering python3, the total saving will be 558s (almost 10 minutes) (core, and streaming run three times, mllib runs twice). During testing, it will show used time for each test file: ``` Run core tests ... Running test: pyspark/rdd.py ... ok (22s) Running test: pyspark/context.py ... ok (16s) Running test: pyspark/conf.py ... ok (4s) Running test: pyspark/broadcast.py ... ok (4s) Running test: pyspark/accumulators.py ... ok (4s) Running test: pyspark/serializers.py ... ok (6s) Running test: pyspark/profiler.py ... ok (5s) Running test: pyspark/shuffle.py ... ok (1s) Running test: pyspark/tests.py ... ok (103s) 144s ``` Author: Reynold Xin <rxin@databricks.com> Author: Xiangrui Meng <meng@databricks.com> Closes#5605 from rxin/python-tests-speed and squashes the following commits: d08542d [Reynold Xin] Merge pull request #14 from mengxr/SPARK-6953 89321ee [Xiangrui Meng] fix seed in tests 3ad2387 [Reynold Xin] Merge pull request #5427 from davies/python_tests

This PR try to speed up some python tests: ``` tests.py 144s -> 103s -41s mllib/classification.py 24s -> 17s -7s mllib/regression.py 27s -> 15s -12s mllib/tree.py 27s -> 13s -14s mllib/tests.py 64s -> 31s -33s streaming/tests.py 185s -> 84s -101s ``` Considering python3, the total saving will be 558s (almost 10 minutes) (core, and streaming run three times, mllib runs twice). During testing, it will show used time for each test file: ``` Run core tests ... Running test: pyspark/rdd.py ... ok (22s) Running test: pyspark/context.py ... ok (16s) Running test: pyspark/conf.py ... ok (4s) Running test: pyspark/broadcast.py ... ok (4s) Running test: pyspark/accumulators.py ... ok (4s) Running test: pyspark/serializers.py ... ok (6s) Running test: pyspark/profiler.py ... ok (5s) Running test: pyspark/shuffle.py ... ok (1s) Running test: pyspark/tests.py ... ok (103s) 144s ``` Author: Reynold Xin <rxin@databricks.com> Author: Xiangrui Meng <meng@databricks.com> Closesapache#5605 from rxin/python-tests-speed and squashes the following commits: d08542d [Reynold Xin] Merge pull request apache#14 from mengxr/SPARK-6953 89321ee [Xiangrui Meng] fix seed in tests 3ad2387 [Reynold Xin] Merge pull request apache#5427 from davies/python_tests

speed up python tests
be23e1d

Update tests.py
0c49785

Davies Liu added 4 commits April 13, 2015 16:22

Merge branch 'master' of github.com:apache/spark into python_tests
5fb17db
Conflicts: python/pyspark/streaming/tests.py

Merge branch 'master' of github.com:apache/spark into python_tests
5ab44f6

speed tests.py, and reduce logging
efa9c39

Merge branch 'python_tests' of github.com:davies/spark into python_tests
fec2da2

davies changed the title ~~[WIP] [PySpark] speed up python tests~~[PySpark] speed up python testsApr 15, 2015

davies changed the title ~~[PySpark] speed up python tests~~[SPARK-6953] [PySpark] speed up python testsApr 16, 2015

show total time
945a2b5

Davies Liu added 2 commits April 16, 2015 10:01

Merge branch 'master' of github.com:apache/spark into python_tests
28fde53

increase the timeout for kafka test
55bb451

Merge branch 'master' of github.com:apache/spark into python_tests
aa39c55
Conflicts: python/pyspark/mllib/tests.py python/pyspark/mllib/tree.py python/pyspark/tests.py

fix tests in python3
2654bfd

rxin added a commit to rxin/spark that referenced this pull request Apr 21, 2015
Merge pull requestapache#5427from davies/python_tests
21b15f5
[SPARK-6953] [PySpark] speed up python tests Conflicts: python/pyspark/streaming/tests.py

rxin mentioned this pull request Apr 21, 2015
[SPARK-6953] [PySpark] speed up python tests #5605
Closed

asfgit closed this in 41ef78aApr 22, 2015

Navigation Menu

Uh oh!

[SPARK-6953] [PySpark] speed up python tests - #5427

[SPARK-6953] [PySpark] speed up python tests#5427
davies wants to merge 11 commits into
apache:masterfrom
davies:python_tests

davies commented Apr 8, 2015

Uh oh!

SparkQA commented Apr 8, 2015

Uh oh!

SparkQA commented Apr 8, 2015

Uh oh!

SparkQA commented Apr 16, 2015

Uh oh!

SparkQA commented Apr 16, 2015

Uh oh!

mengxr commented Apr 16, 2015

Uh oh!

davies commented Apr 16, 2015

Uh oh!

SparkQA commented Apr 16, 2015

Uh oh!

SparkQA commented Apr 17, 2015

Uh oh!

SparkQA commented Apr 17, 2015

Uh oh!

SparkQA commented Apr 18, 2015

Uh oh!

mengxr commented Apr 21, 2015

Uh oh!

rxin commented Apr 21, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

davies commented Apr 8, 2015

Uh oh!

SparkQA commented Apr 8, 2015

Uh oh!

SparkQA commented Apr 8, 2015

Uh oh!

SparkQA commented Apr 16, 2015

Uh oh!

SparkQA commented Apr 16, 2015

Uh oh!

mengxr commented Apr 16, 2015

Uh oh!

davies commented Apr 16, 2015

Uh oh!

SparkQA commented Apr 16, 2015

Uh oh!

SparkQA commented Apr 17, 2015

Uh oh!

SparkQA commented Apr 17, 2015

Uh oh!

SparkQA commented Apr 18, 2015

Uh oh!

mengxr commented Apr 21, 2015

Uh oh!

rxin commented Apr 21, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants