Skip to content

[SPARK-24795][Core][FOLLOWUP] Kill all running tasks when a task in a barrier stage fail - #21943

Closed
jiangxb1987 wants to merge 5 commits into
apache:masterfrom
jiangxb1987:killAllTasks
Closed

[SPARK-24795][Core][FOLLOWUP] Kill all running tasks when a task in a barrier stage fail#21943
jiangxb1987 wants to merge 5 commits into
apache:masterfrom
jiangxb1987:killAllTasks

Conversation

@jiangxb1987

@jiangxb1987jiangxb1987 commented Aug 1, 2018•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Kill all running tasks when a task in a barrier stage fail in the middle. TaskScheduler.cancelTasks() will also fail the job, so we implemented a new method killAllTaskAttempts() to just kill all running tasks of a stage without cancel the stage/job.

How was this patch tested?

Add new test cases in TaskSchedulerImplSuite.

@holdensmagicalunicorn

holdensmagicalunicorn commented Aug 1, 2018

Copy link
Copy Markdown

@jiangxb1987, thanks! I am a bot who has found some folks who might be able to help with the review:@gatorsmile, @mateiz and @kayousterhout

@jiangxb1987jiangxb1987 mentioned this pull request Aug 1, 2018
Closed
mengxr
mengxr reviewed Aug 1, 2018
View reviewed changes
Comment threadcore/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
interruptThread: Boolean,
reason: String): Unit = synchronized {
logInfo(s"Killing all running tasks in stage $stageId: $reason")
taskSetsByStageIdAndAttempt.get(stageId).foreach { attempts =>

@mengxrmengxrAug 1, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is some dup code and we dropped the useful comments from cancelTasks. It would be great if we move the common code here with comment and let cancelTasks call this method.

@mengxr

mengxr commented Aug 1, 2018

Copy link
Copy Markdown
Contributor

@jiangxb1987 Could you add a test to the new method?

@SparkQA

SparkQA commented Aug 1, 2018

Copy link
Copy Markdown

Test build #93882 has finished for PR 21943 at commit 97bfba8.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Aug 1, 2018

Copy link
Copy Markdown

Test build #93893 has finished for PR 21943 at commit b5fca7a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987

jiangxb1987 commented Aug 2, 2018

Copy link
Copy Markdown
ContributorAuthor

@mengxr Updated and added test cases, PTAL!

cloud-fan
cloud-fan reviewed Aug 2, 2018
View reviewed changes
Comment threadcore/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala
def submitTasks(taskSet: TaskSet): Unit

// Cancel a stage.
// Kill all the tasks in a stage and fail the stage and all the jobs that depend on the stage.

@cloud-fancloud-fanAug 2, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it guaranteed to work for any backend like YARN, Mesos, K8s?

@jiangxb1987jiangxb1987Aug 2, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated comment to note that if the backend doesn't support kill a task then the method shall throw UnsupportedOperationException.

@mengxr

mengxr commented Aug 2, 2018

Copy link
Copy Markdown
Contributor

LGTM

@SparkQA

SparkQA commented Aug 2, 2018

Copy link
Copy Markdown

Test build #93951 has finished for PR 21943 at commit 7cca33f.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Aug 2, 2018

Copy link
Copy Markdown

Test build #93945 has finished for PR 21943 at commit bc77a8e.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987

jiangxb1987 commented Aug 2, 2018

Copy link
Copy Markdown
ContributorAuthor

retest this please

@cloud-fan

cloud-fan commented Aug 2, 2018

Copy link
Copy Markdown
Contributor

LGTM

@SparkQA

SparkQA commented Aug 2, 2018

Copy link
Copy Markdown

Test build #93969 has finished for PR 21943 at commit 7cca33f.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

cloud-fan commented Aug 2, 2018

Copy link
Copy Markdown
Contributor

retest this please

@SparkQA

SparkQA commented Aug 2, 2018

Copy link
Copy Markdown

Test build #93968 has finished for PR 21943 at commit 7cca33f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

cloud-fan commented Aug 2, 2018

Copy link
Copy Markdown
Contributor

thanks, merging to master!

@asfgitasfgit closed this in 2754157Aug 2, 2018
@jiangxb1987
jiangxb1987 deleted the killAllTasks branch August 2, 2018 13:01
@SparkQA

SparkQA commented Aug 2, 2018

Copy link
Copy Markdown

Test build #93996 has finished for PR 21943 at commit 7cca33f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mengxrmengxrmengxr left review comments
@cloud-fancloud-fancloud-fan left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@jiangxb1987@holdensmagicalunicorn@mengxr@SparkQA@cloud-fan