Skip to content

[SPARK-23040][CORE][FOLLOW-UP] Avoid double wrap result Iterator. - #20920

Closed
jiangxb1987 wants to merge 1 commit into
apache:masterfrom
jiangxb1987:SPARK-23040
Closed

[SPARK-23040][CORE][FOLLOW-UP] Avoid double wrap result Iterator.#20920
jiangxb1987 wants to merge 1 commit into
apache:masterfrom
jiangxb1987:SPARK-23040

Conversation

@jiangxb1987

@jiangxb1987jiangxb1987 commented Mar 28, 2018

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Address #20449 (comment), If resultIter is already a InterruptibleIterator, don't double wrap it.

How was this patch tested?

Existing tests.

@jiangxb1987

jiangxb1987 commented Mar 28, 2018

Copy link
Copy Markdown
ContributorAuthor

cc @cloud-fan@advancedxy

advancedxy
advancedxy approved these changes Mar 28, 2018
View reviewed changes

@advancedxyadvancedxy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Sorry for didn't send this by myself. I am quite occupied.

@SparkQA

SparkQA commented Mar 28, 2018

Copy link
Copy Markdown

Test build #88664 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987

jiangxb1987 commented Mar 29, 2018

Copy link
Copy Markdown
ContributorAuthor

retest this please

@SparkQA

SparkQA commented Mar 29, 2018

Copy link
Copy Markdown

Test build #88685 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987

jiangxb1987 commented Mar 29, 2018

Copy link
Copy Markdown
ContributorAuthor

retest this please

@cloud-fan

cloud-fan commented Mar 29, 2018

Copy link
Copy Markdown
Contributor

LGTM

@jerryshao

jerryshao commented Mar 29, 2018

Copy link
Copy Markdown
Contributor

LGTM.

@SparkQA

SparkQA commented Mar 29, 2018

Copy link
Copy Markdown

Test build #88691 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jerryshao

jerryshao commented Mar 29, 2018

Copy link
Copy Markdown
Contributor

Jenkins, retest this please.

@SparkQA

SparkQA commented Mar 29, 2018

Copy link
Copy Markdown

Test build #88694 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails from timeout after a configured wait of `300m`.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987

jiangxb1987 commented Mar 29, 2018

Copy link
Copy Markdown
ContributorAuthor

retest this please

@SparkQA

SparkQA commented Mar 29, 2018

Copy link
Copy Markdown

Test build #88707 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

cloud-fan commented Mar 30, 2018

Copy link
Copy Markdown
Contributor

retest this please

@SparkQA

SparkQA commented Mar 30, 2018

Copy link
Copy Markdown

Test build #88735 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jerryshao

jerryshao commented Mar 30, 2018

Copy link
Copy Markdown
Contributor

Jenkins, retest this please.

@SparkQA

SparkQA commented Mar 30, 2018

Copy link
Copy Markdown

Test build #88744 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987

jiangxb1987 commented Mar 30, 2018

Copy link
Copy Markdown
ContributorAuthor

retest this please

@jiangxb1987

jiangxb1987 commented Mar 30, 2018

Copy link
Copy Markdown
ContributorAuthor

Jenkins is pretty unstable recently.

@SparkQA

SparkQA commented Mar 30, 2018

Copy link
Copy Markdown

Test build #88747 has finished for PR 20920 at commit 35ecbf9.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987

jiangxb1987 commented Mar 30, 2018

Copy link
Copy Markdown
ContributorAuthor

retest this please

@SparkQA

SparkQA commented Mar 30, 2018

Copy link
Copy Markdown

Test build #88754 has finished for PR 20920 at commit 35ecbf9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

cloud-fan commented Mar 31, 2018

Copy link
Copy Markdown
Contributor

thanks, merging to master!

jiangxb1987 added a commit to jiangxb1987/spark that referenced this pull request Mar 31, 2018
@jiangxb1987
[SPARK-23040][CORE][FOLLOW-UP] Avoid double wrap result Iterator.
e9a661f
## What changes were proposed in this pull request?
Address apache#20449 (comment), If `resultIter` is already a `InterruptibleIterator`, don't double wrap it.
## How was this patch tested?
Existing tests.
Author: Xingbo Jiang <xingbo.jiang@databricks.com>
Closesapache#20920 from jiangxb1987/SPARK-23040.
@jiangxb1987jiangxb1987 mentioned this pull request Mar 31, 2018
Closed
asfgit pushed a commit that referenced this pull request Apr 1, 2018
@advancedxy@cloud-fan
[SPARK-23040][BACKPORT][CORE] Returns interruptible iterator for shuf…
f1f10da
…fle reader
Backport #20449 and #20920 to branch-2.3
---
## What changes were proposed in this pull request?
Before this commit, a non-interruptible iterator is returned if aggregator or ordering is specified.
This commit also ensures that sorter is closed even when task is cancelled(killed) in the middle of sorting.
## How was this patch tested?
Add a unit test in JobCancellationSuite
Author: Xianjin YE <advancedxy@gmail.com>
Author: Xingbo Jiang <xingbo.jiang@databricks.com>
Closes#20954 from jiangxb1987/SPARK-23040-2.3.
mshtelma pushed a commit to mshtelma/spark that referenced this pull request Apr 5, 2018
@jiangxb1987Mykhailo Shtelma
[SPARK-23040][CORE][FOLLOW-UP] Avoid double wrap result Iterator.
5152b3c
## What changes were proposed in this pull request?
Address apache#20449 (comment), If `resultIter` is already a `InterruptibleIterator`, don't double wrap it.
## How was this patch tested?
Existing tests.
Author: Xingbo Jiang <xingbo.jiang@databricks.com>
Closesapache#20920 from jiangxb1987/SPARK-23040.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@advancedxyadvancedxyadvancedxy approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@jiangxb1987@SparkQA@cloud-fan@jerryshao@advancedxy