Uh oh!
There was an error while loading. Please reload this page.
[SPARK-18981][Core]The job hang problem when speculation is on - #16389
[SPARK-18981][Core]The job hang problem when speculation is on#16389zhaorongsheng wants to merge 2 commits into
Conversation
mridulm
left a comment
There was a problem hiding this comment.
While I dont think the change itself might be incorrect : did you see
'No stages are running, but numRunningTasks != 0' in the logs ?
| val taskInfo = createTaskInfo(1, 1, "executor-1") | ||
| val speculatedTaskInfo = createTaskInfo(2, 1, "executor-1") | ||
| sc.listenerBus.postToAll(SparkListenerTaskStart(0, 0, taskInfo)) | ||
| assert(maxNumExecutorsNeeded(manager) === 1) |
There was a problem hiding this comment.
This tests looks wrong - taskIndex is higher than numTasks ?
Would be better for the test to :
- Launch stage with 1 task.
- Launch a normal task and 1 speculative task - with same taskIndex, but different taskId's
- Finish normal task.
- Ensure stage is completed.
- Now finish speculative task and check if bug is not reproduced (it should be reproduced without this fix).
There was a problem hiding this comment.
Yes, the warning info 'No stages are running, but numRunningTasks != 0' is printed and at that time the #numRunningTasks is set to 0. But after that the speculated task end event is arrived and the #numRunningTasks will plus 1.
The tests are wrong, I will fix it.
zhaorongsheng
commented
Dec 24, 2016
Hi @mridulm . I have modified the tests. Please check it. |
zhaorongsheng
commented
Dec 24, 2016
Jenkins, retest this please |
Does it fail in master without the fix ? |
zhaorongsheng
commented
Dec 24, 2016
Yes, I have checked it. |
zhaorongsheng
commented
Dec 25, 2016
@mridulm Please check it. Thanks~ |
Hi, can anyone check this PR? |
zsxwing
commented
Dec 29, 2016
Can we just not reset |
zhaorongsheng
commented
Dec 31, 2016
@zsxwing I think it may cause some other problem. |
zhaorongsheng
commented
Jan 24, 2017
jinxing64
commented
Mar 21, 2017
@zhaorongsheng |
HyukjinKwon
commented
May 11, 2017
@zhaorongsheng, is this still active and any opinion on ^? |
## What changes were proposed in this pull request? This PR proposes to close PRs ... - inactive to the review comments more than a month - WIP and inactive more than a month - with Jenkins build failure but inactive more than a month - suggested to be closed and no comment against that - obviously looking inappropriate (e.g., Branch 0.5) To make sure, I left a comment for each PR about a week ago and I could not have a response back from the author in these PRs below: Closesapache#11129Closesapache#12085Closesapache#12162Closesapache#12419Closesapache#12420Closesapache#12491Closesapache#13762Closesapache#13837Closesapache#13851Closesapache#13881Closesapache#13891Closesapache#13959Closesapache#14091Closesapache#14481Closesapache#14547Closesapache#14557Closesapache#14686Closesapache#15594Closesapache#15652Closesapache#15850Closesapache#15914Closesapache#15918Closesapache#16285Closesapache#16389Closesapache#16652Closesapache#16743Closesapache#16893Closesapache#16975Closesapache#17001Closesapache#17088Closesapache#17119Closesapache#17272Closesapache#17971 Added: Closesapache#17778Closesapache#17303Closesapache#17872 ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#18017 from HyukjinKwon/close-inactive-prs.
What changes were proposed in this pull request?
The root cause of this issue is that
ExecutorAllocationListenergets the speculated task end info after the stage end event handling which letnumRunningTasks = 0. Then it letnumRunningTasks -= 1so the #numRunningTasks is negative. When calculate #maxNeeded in methodmaxNumExecutorsNeeded(), the value may be 0 or negative. SoExecutorAllocationManagerdoes not request container and the job will be hung.This PR changes the method
onTaskEnd()inExecutorAllocationListener. WhenstageIdToNumTaskscontains the taskEnd's stageId, let #numRunningTasks minus 1.How was this patch tested?
This patch was tested in the method
test("SPARK-18981...)of ExecutorAllocationManagerSuite.scala.Create two taskInfos and one of them is speculated task. After the stage ending event, the speculated task ending event is posted to listener.