Skip to content

[SPARK-5363] fix bug: remove() inside iterator is not safe - #4776

Closed
davies wants to merge 1 commit into
apache:masterfrom
davies:fix_hang
Closed

[SPARK-5363] fix bug: remove() inside iterator is not safe#4776
davies wants to merge 1 commit into
apache:masterfrom
davies:fix_hang

Conversation

@davies

@daviesdavies commented Feb 26, 2015

Copy link
Copy Markdown
Contributor

During iterating, it's not safe to remove item.

@davies

davies commented Feb 26, 2015

Copy link
Copy Markdown
ContributorAuthor

cc @JoshRosen

@SparkQA

SparkQA commented Feb 26, 2015

Copy link
Copy Markdown

Test build #27969 has started for PR 4776 at commit a4384a5.

  • This patch merges cleanly.

@JoshRosen

JoshRosen commented Feb 26, 2015

Copy link
Copy Markdown
Contributor

LGTM pending Jenkins.

@JoshRosen

JoshRosen commented Feb 26, 2015

Copy link
Copy Markdown
Contributor

Context for other reviewers: removing elements from a mutable HashSet while iterating over it can cause the iteration to skip over entries that weren't removed. In this case, this would cause PythonRDD to write fewer than cnt broadcasts, which caused the Python worker to hang while expecting to read cnt total broadcasts.

@SparkQA

SparkQA commented Feb 26, 2015

Copy link
Copy Markdown

Test build #27969 has finished for PR 4776 at commit a4384a5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Feb 26, 2015

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27969/
Test PASSed.

@JoshRosen

JoshRosen commented Feb 26, 2015

Copy link
Copy Markdown
Contributor

I'm going to merge this into master (1.4.0), branch-1.3 (1.3.0), and branch-1.2 (1.2.2). Thanks!

asfgit pushed a commit that referenced this pull request Feb 26, 2015
Davies Liu@JoshRosen
[SPARK-5363] Fix bug in PythonRDD: remove() inside iterator is not safe
5d309ad
Removing elements from a mutable HashSet while iterating over it can cause the
iteration to incorrectly skip over entries that were not removed. If this
happened, PythonRDD would write fewer broadcast variables than the Python
worker was expecting to read, which would cause the Python worker to hang
indefinitely.
Author: Davies Liu <davies@databricks.com>
Closes#4776 from davies/fix_hang and squashes the following commits:
a4384a5 [Davies Liu] fix bug: remvoe() inside iterator is not safe
(cherry picked from commit 7fa960e)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
asfgit pushed a commit that referenced this pull request Feb 26, 2015
Davies Liu@JoshRosen
[SPARK-5363] Fix bug in PythonRDD: remove() inside iterator is not safe
cc7313d
Removing elements from a mutable HashSet while iterating over it can cause the
iteration to incorrectly skip over entries that were not removed. If this
happened, PythonRDD would write fewer broadcast variables than the Python
worker was expecting to read, which would cause the Python worker to hang
indefinitely.
Author: Davies Liu <davies@databricks.com>
Closes#4776 from davies/fix_hang and squashes the following commits:
a4384a5 [Davies Liu] fix bug: remvoe() inside iterator is not safe
(cherry picked from commit 7fa960e)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@davies@SparkQA@JoshRosen@AmplabJenkins