Skip to content

[SPARK-1690] Tolerating empty elements when saving Python RDD to text files - #644

Closed
kanzhang wants to merge 2 commits into
apache:masterfrom
kanzhang:SPARK-1690
Closed

[SPARK-1690] Tolerating empty elements when saving Python RDD to text files#644
kanzhang wants to merge 2 commits into
apache:masterfrom
kanzhang:SPARK-1690

Conversation

@kanzhang

@kanzhangkanzhang commented May 5, 2014

Copy link
Copy Markdown
Contributor

Tolerate empty strings in PythonRDD

@AmplabJenkins

AmplabJenkins commented May 5, 2014

Copy link
Copy Markdown

Can one of the admins verify this patch?

@kanzhang

kanzhang commented May 5, 2014

Copy link
Copy Markdown
ContributorAuthor

Manually verified the patch on file with empty lines in the beginning, middle or end of file. Also tested empty file and file with only empty lines.

@mateiz

mateiz commented May 5, 2014

Copy link
Copy Markdown
Contributor

Can you add a test case for this? What file was it breaking on?

@kanzhang

kanzhang commented May 5, 2014

Copy link
Copy Markdown
ContributorAuthor

Any text file with empty lines in it will break, like Glenn reported in the JIRA - a file consists of ["foo", "", "bar"]. Unfortunately, I don't see an easy way to separate out the stdoutIterator logic and test it since it references readerException in the enclosing scope. Ideas welcome.

@kanzhang

kanzhang commented May 8, 2014

Copy link
Copy Markdown
ContributorAuthor

@mateiz just realized I could test it from Python side. Added a doctest. This makes Python API behave identical to Scala API.

@kanzhangkanzhang changed the title [SPARK-1690] Allow empty lines in PythonRDD[SPARK-1690] Tolerating empty elements when saving Python RDD to text filesMay 8, 2014
@kanzhangkanzhang mentioned this pull request May 8, 2014
Closed
@pwendell

pwendell commented May 10, 2014

Copy link
Copy Markdown
Contributor

Jenkins, test this please.

@AmplabJenkins

AmplabJenkins commented May 10, 2014

Copy link
Copy Markdown

Merged build triggered.

@AmplabJenkins

AmplabJenkins commented May 10, 2014

Copy link
Copy Markdown

Merged build started.

mateiz
mateiz reviewed May 10, 2014
View reviewed changes
Comment threadcore/src/main/scala/org/apache/spark/api/python/PythonRDD.scala

@mateizmateizMay 10, 2014

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, though you could probably just change the if length > 0 to if length >= 0 above

@AmplabJenkins

AmplabJenkins commented May 10, 2014

Copy link
Copy Markdown

Merged build finished. All automated tests passed.

@AmplabJenkins

AmplabJenkins commented May 10, 2014

Copy link
Copy Markdown

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14871/

@pwendell

pwendell commented May 10, 2014

Copy link
Copy Markdown
Contributor

Okay I'll pull this in. Thanks!

asfgit pushed a commit that referenced this pull request May 10, 2014
@kanzhang@pwendell
[SPARK-1690] Tolerating empty elements when saving Python RDD to text…
ac86af8
… files
Tolerate empty strings in PythonRDD
Author: Kan Zhang <kzhang@apache.org>
Closes#644 from kanzhang/SPARK-1690 and squashes the following commits:
c62ad33 [Kan Zhang] Adding Python doctest
473ec4b [Kan Zhang] [SPARK-1690] Tolerating empty elements when saving Python RDD to text files
(cherry picked from commit 6c2691d)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
@kanzhang
kanzhang deleted the SPARK-1690 branch May 10, 2014 23:37
pdeyhim pushed a commit to pdeyhim/spark-1 that referenced this pull request Jun 25, 2014
@kanzhang@pdeyhim
[SPARK-1690] Tolerating empty elements when saving Python RDD to text…
762ba0b
… files
Tolerate empty strings in PythonRDD
Author: Kan Zhang <kzhang@apache.org>
Closesapache#644 from kanzhang/SPARK-1690 and squashes the following commits:
c62ad33 [Kan Zhang] Adding Python doctest
473ec4b [Kan Zhang] [SPARK-1690] Tolerating empty elements when saving Python RDD to text files
andrewor14 pushed a commit to andrewor14/spark that referenced this pull request Jan 8, 2015
@bouk@JoshRosen
SPARK-1115: Catch depickling errors
886a466
This surroungs the complete worker code in a try/except block so we catch any error that arrives. An example would be the depickling failing for some reason
@JoshRosen
Author: Bouke van der Bijl <boukevanderbijl@gmail.com>
Closesapache#644 from bouk/catch-depickling-errors and squashes the following commits:
f0f67cc [Bouke van der Bijl] Lol indentation
0e4d504 [Bouke van der Bijl] Surround the complete python worker with the try block
(cherry picked from commit 12738c1)
Signed-off-by: Josh Rosen <joshrosen@apache.org>
rshkv pushed a commit to rshkv/spark that referenced this pull request Feb 27, 2020
@LorenzoMartini
Not print conda environment when executing conda process (apache#644)
92fdc23
Not print conda environment as spark doesn't have arg logging and the env can therefore contain unsafe information.
agirish pushed a commit to HPEEzmeral/apache-spark that referenced this pull request May 5, 2022
Egor Krivokon
MapR [SPARK-705] Spark streaming fail with NoSuchMethodError: org.apa…
7ef5071
…che.kafka.clients.producer.ProducerConfig.getBoolean(Ljava/lang/String;)Ljava/lang/Boolean; (apache#644)
udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024
Egor Krivokon@ekrivokonmapr
MapR [SPARK-705] Spark streaming fail with NoSuchMethodError: org.apa…
e9dc2cb
…che.kafka.clients.producer.ProducerConfig.getBoolean(Ljava/lang/String;)Ljava/lang/Boolean; (apache#644)
mapr-devops pushed a commit to mapr/spark that referenced this pull request May 8, 2025
Egor Krivokon@ekrivokonmapr
MapR [SPARK-705] Spark streaming fail with NoSuchMethodError: org.apa…
1b93f1d
…che.kafka.clients.producer.ProducerConfig.getBoolean(Ljava/lang/String;)Ljava/lang/Boolean; (apache#644)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kanzhang@AmplabJenkins@mateiz@pwendell