[SPARK-2871] [PySpark] Add missing API by davies · Pull Request #1791 · apache/spark

davies · 2014-08-05T17:51:13Z

Add the following APIs:

SparkContext.conf
SparkContext.isLocal
SparkContext.startTime

SparkQA · 2014-08-05T22:33:29Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17953/consoleFull

SparkQA · 2014-08-05T23:33:53Z

QA results for PR 1791:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17953/consoleFull

SparkQA · 2014-08-05T23:54:23Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17958/consoleFull

SparkQA · 2014-08-05T23:58:22Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17959/consoleFull

SparkQA · 2014-08-06T00:48:23Z

QA results for PR 1791:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17958/consoleFull

zzl0 · 2014-08-06T05:20:44Z

Convert a RDD of Java objects to and RDD of serialized Python objects
=>
Convert an RDD of Java objects to an RDD of serialized Python objects ?

SparkQA · 2014-08-06T05:44:27Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18001/consoleFull

davies · 2014-08-06T06:19:17Z

The histogram() had been implemented in pure Python, it will support integer better, also it will support RDD of strings and other comparable objects.

This was inspired by #1783 et, and much improved.

SparkQA · 2014-08-06T06:48:51Z

QA results for PR 1791:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18001/consoleFull

nrchandan · 2014-08-06T06:50:12Z

This was smart!

SparkQA · 2014-08-06T18:04:21Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18032/consoleFull

SparkQA · 2014-08-06T18:53:01Z

QA results for PR 1791:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18032/consoleFull

meanApprox() and sumApprox()

SparkQA · 2014-08-06T21:39:29Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18055/consoleFull

SparkQA · 2014-08-06T21:40:27Z

QA results for PR 1791:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class BoundedFloat(float):

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18055/consoleFull

SparkQA · 2014-08-06T21:48:33Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18058/consoleFull

davies · 2014-08-14T00:17:54Z

Jenkins, test this please

SparkQA · 2014-08-14T00:20:33Z

QA tests have started for PR 1791. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18498/consoleFull

SparkQA · 2014-08-14T01:11:40Z

QA results for PR 1791:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class BoundedFloat(float):

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18498/consoleFull

mateiz · 2014-08-14T21:08:58Z

Maybe explain what "comp" is in the doc comment

mateiz · 2014-08-14T21:25:31Z

@davies I looked over all of this now and made some comments, but you should have Josh check too. Just to be clear though, I don't think this can make it into 1.1, so we can hold off on it for a while while we fix issues for 1.1. But these are great APIs to have.

1. implement lookup(), similar to that in Scala 2. handle None, nan, inf in histogram, add many tests 3. remove collectPartitions() 4. improve docs

davies · 2014-08-14T23:03:48Z

@mateiz thanks for review this, I had addressed all you comments.

@JoshRosen could you take a look a this again?

davies · 2014-08-14T23:25:28Z

The description had been updated to list all the added APIs.

Conflicts: python/pyspark/rdd.py python/pyspark/tests.py

davies · 2014-08-22T05:25:17Z

@mateiz @JoshRosen some APIs has been splitted out as separated PRs: #2091, #2092, #2093, #2094, #2095

Conflicts: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala python/pyspark/rdd.py

SparkQA · 2014-08-26T18:20:59Z

QA tests have started for PR 1791 at commit 28fd368.

This patch merges cleanly.

SparkQA · 2014-08-26T18:31:02Z

QA tests have started for PR 1791 at commit 1ac98d6.

This patch merges cleanly.

SparkQA · 2014-08-26T19:11:06Z

QA tests have started for PR 1791 at commit 657a09b.

This patch merges cleanly.

SparkQA · 2014-08-26T19:12:09Z

QA tests have finished for PR 1791 at commit 28fd368.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BoundedFloat(float):

SparkQA · 2014-08-26T19:22:45Z

QA tests have finished for PR 1791 at commit 1ac98d6.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-08-26T19:57:10Z

QA tests have finished for PR 1791 at commit 657a09b.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2014-08-27T21:58:13Z

Most of useful parts have been merged separately, so close this.

davies added 4 commits August 4, 2014 23:55

add missing API in SparkContext
ff2cbe3

add histogram()
e0b3d30

change histogram API
5d5be95

add zipWithIndex and zipWithUniqueId
a95eca0

collectPartitions()
4ffae00

update docs of histogram
7a9ea0a

davies changed the title ~~[WIP] [PySpark] Add missing API~~[PySpark] Add missing APIAug 5, 2014

davies changed the title ~~[PySpark] Add missing API~~[SPARK-2871] [PySpark] Add missing APIAug 6, 2014

zzl0 reviewed Aug 6, 2014
View reviewed changes

davies added 2 commits August 5, 2014 22:38

histogram() in pure Python, better support for int
53640be

fix docs
9a01ac3

nrchandan mentioned this pull request Aug 6, 2014
[SPARK-1170] Add histogram method to Python's RDD API #1783
Closed

nrchandan reviewed Aug 6, 2014
View reviewed changes

Comment threadpython/pyspark/rdd.py Outdated
nrchandanAug 6, 2014
Copy link
Copy Markdown
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was smart!

refactor
7ba5f88

davies added 2 commits August 6, 2014 14:33

fix bug of countApproxDistinct
a25c34e

add countApprox and countApproxDistinct
1218b3b
meanApprox() and sumApprox()

Merge branch 'master' into api
034124f

mateiz reviewed Aug 14, 2014
View reviewed changes

Comment threadpython/pyspark/rdd.py Outdated
mateizAug 14, 2014
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe explain what "comp" is in the doc comment

address all the comments:
63c013d
1. implement lookup(), similar to that in Scala 2. handle None, nan, inf in histogram, add many tests 3. remove collectPartitions() 4. improve docs

Merge branch 'master' into api
1213aca
Conflicts: python/pyspark/rdd.py python/pyspark/tests.py

Merge branch 'master' into api
28fd368
Conflicts: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala python/pyspark/rdd.py

remove splitted changes
1ac98d6

remove countApproxDistinct()
657a09b

davies closed this Aug 27, 2014

davies mentioned this pull request Aug 27, 2014
[SPARK-2871] [PySpark] add countApproxDistinct() API #2142
Closed

davies deleted the api branch September 15, 2014 22:15

Uh oh!

Conversation

davies commented Aug 5, 2014

Uh oh!

SparkQA commented Aug 5, 2014

Uh oh!

SparkQA commented Aug 5, 2014

Uh oh!

SparkQA commented Aug 5, 2014

Uh oh!

SparkQA commented Aug 5, 2014

Uh oh!

SparkQA commented Aug 6, 2014

Uh oh!

zzl0Aug 6, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 6, 2014

Uh oh!

davies commented Aug 6, 2014

Uh oh!

SparkQA commented Aug 6, 2014

Uh oh!

nrchandanAug 6, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 6, 2014

Uh oh!

SparkQA commented Aug 6, 2014

Uh oh!

SparkQA commented Aug 6, 2014

Uh oh!

SparkQA commented Aug 6, 2014

Uh oh!

SparkQA commented Aug 6, 2014

Uh oh!

davies commented Aug 14, 2014

Uh oh!

SparkQA commented Aug 14, 2014

Uh oh!

SparkQA commented Aug 14, 2014

Uh oh!

mateizAug 14, 2014

Choose a reason for hiding this comment

Uh oh!

mateiz commented Aug 14, 2014

Uh oh!

davies commented Aug 14, 2014

Uh oh!

davies commented Aug 14, 2014

Uh oh!

davies commented Aug 22, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

davies commented Aug 27, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants