Skip to content

[SPARK-16709][CORE] Kill the running task if stage failed - #14557

Closed
shenh062326 wants to merge 4 commits into
apache:masterfrom
shenh062326:SPARK-16709
Closed

[SPARK-16709][CORE] Kill the running task if stage failed#14557
shenh062326 wants to merge 4 commits into
apache:masterfrom
shenh062326:SPARK-16709

Conversation

@shenh062326

@shenh062326shenh062326 commented Aug 9, 2016•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

At SPARK-16709, when a stage failed, but the running task is still running, the retry stage will rerun the running task, it could cause TaskCommitDeniedException and task retry forever.

Here is the log:

16/07/28 05:22:15 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 175, 10.215.146.81, partition 1,PROCESS_LOCAL, 1930 bytes)
16/07/28 05:28:35 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.1 (TID 207, 10.196.147.232, partition 1,PROCESS_LOCAL, 1930 bytes)
16/07/28 05:28:48 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 175) in 393261 ms on 10.215.146.81 (3/50)
16/07/28 05:34:11 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 1.1 (TID 207, 10.196.147.232): TaskCommitDenied (Driver denied task commit) for job: 1, partition: 1, attemptNumber: 207
16/07/28 05:34:40 INFO scheduler.TaskSetManager: Starting task 1.1 in stage 1.1 (TID 227, 10.196.129.22, partition 1,PROCESS_LOCAL, 1930 bytes)
16/07/28 05:40:08 WARN scheduler.TaskSetManager: Lost task 1.1 in stage 1.1 (TID 227, 10.196.129.22): TaskCommitDenied (Driver denied task commit) for job: 1, partition: 1, attemptNumber: 227
16/07/28 05:42:28 INFO scheduler.TaskSetManager: Starting task 1.2 in stage 1.1 (TID 250, 10.215.146.82, partition 1,PROCESS_LOCAL, 1930 bytes)
...

1 task 1.0 in stage1.0 start
2 stage1.0 failed, start stage1.1.
3 task 1.0 in stage1.1 start
4 task 1.0 in stage1.0 finished.
5 task 1.0 in stage1.1 failed with TaskCommitDenied Exception, then retry forever.

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

@SparkQA

SparkQA commented Aug 9, 2016

Copy link
Copy Markdown

Test build #63421 has finished for PR 14557 at commit 1a1ea2f.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

jerryshao
jerryshao reviewed Aug 9, 2016
View reviewed changes
Comment threadcore/src/main/scala/org/apache/spark/SparkContext.scala Outdated
}
}

def killTasks(tasks: HashSet[Long], taskInfo: HashMap[Long, TaskInfo]): Boolean = {

@jerryshaojerryshaoAug 9, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not suitable to add a public method here in SparkContext, SparkContext is a public entry point, any method adds to here should be considered carefully. In your case looks like only Spark internally will use this method, why not directly change the TaskSetManager?

@shenh062326shenh062326Aug 9, 2016

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jerryshao, Thanks for your prompt. I will move the method to TaskSetManager.

@SparkQA

SparkQA commented Aug 9, 2016

Copy link
Copy Markdown

Test build #63424 has finished for PR 14557 at commit 9263678.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Aug 9, 2016

Copy link
Copy Markdown

Test build #63434 has finished for PR 14557 at commit 9ea08e8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

jaceklaskowski
jaceklaskowski reviewed Aug 13, 2016
View reviewed changes
Comment threadcore/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala Outdated
maybeFinishTaskSet()

// kill running task if stage failed
if(reason.isInstanceOf[FetchFailed]) {

@jaceklaskowskijaceklaskowskiAug 13, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A space between if and (.

@jaceklaskowski

jaceklaskowski commented Aug 15, 2016

Copy link
Copy Markdown
Contributor

LGTM

@SparkQA

SparkQA commented Aug 15, 2016

Copy link
Copy Markdown

Test build #63768 has finished for PR 14557 at commit fbe31eb.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@markhamstra

markhamstra commented Aug 15, 2016

Copy link
Copy Markdown
Contributor

There are multiple issues with this PR. Some are at a more stylistic level, but some include deeper issues -- e.g. see SPARK-17064. Most fundamentally, this PR is the wrong solution at least in the sense that it does not implement a minimal fix without other side effects. The problem is that TaskCommitDenied is not being handled properly when a duplicate Task tries to commit a result that has already been successfully committed by another attempt of this Task. The proper fix needs to be at that point of committing duplicate results, not by making the larger, unnecessary change in how we handle cancellation/interruption of other Tasks in a TaskSet when one of them produces a FetchFailed.

@HyukjinKwon

HyukjinKwon commented May 11, 2017

Copy link
Copy Markdown
Member

@shenh062326 I would rather like to propose to close this if there is no argument against ^.

@HyukjinKwonHyukjinKwon mentioned this pull request May 17, 2017
Closed
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
@HyukjinKwon@zifeif2
[INFRA] Close stale PRs
47a2872
## What changes were proposed in this pull request?
This PR proposes to close PRs ...
- inactive to the review comments more than a month
- WIP and inactive more than a month
- with Jenkins build failure but inactive more than a month
- suggested to be closed and no comment against that
- obviously looking inappropriate (e.g., Branch 0.5)
To make sure, I left a comment for each PR about a week ago and I could not have a response back from the author in these PRs below:
Closesapache#11129Closesapache#12085Closesapache#12162Closesapache#12419Closesapache#12420Closesapache#12491Closesapache#13762Closesapache#13837Closesapache#13851Closesapache#13881Closesapache#13891Closesapache#13959Closesapache#14091Closesapache#14481Closesapache#14547Closesapache#14557Closesapache#14686Closesapache#15594Closesapache#15652Closesapache#15850Closesapache#15914Closesapache#15918Closesapache#16285Closesapache#16389Closesapache#16652Closesapache#16743Closesapache#16893Closesapache#16975Closesapache#17001Closesapache#17088Closesapache#17119Closesapache#17272Closesapache#17971
Added:
Closesapache#17778Closesapache#17303Closesapache#17872
## How was this patch tested?
N/A
Author: hyukjinkwon <gurwls223@gmail.com>
Closesapache#18017 from HyukjinKwon/close-inactive-prs.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@shenh062326@SparkQA@jaceklaskowski@markhamstra@HyukjinKwon@jerryshao