[SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost by weixiuli · Pull Request #26206 · apache/spark

weixiuli · 2019-10-22T11:16:07Z

What changes were proposed in this pull request?

When an executor lost with some reason and some things (e.g. the external shuffle service or host lost on the executor's host.) happened, and the executor loses time happens to be reduce stage fetch failed from it which is really bad, the previous only call mapOutputTracker.unregisterMapOutput(shuffleId, mapIndex, bmAddress) to mark clear shuffle status for the mapper at this time , but the other mappers shuffle status on the executor are also unnavailable and the DagScheduler does Not know that, so the reduce stages will fetch failed again when fetch them, the unavailable shuffle status can only be resubmitted by a nest retry stage which is the regression.

As we all know that the previous will call mapOutputTracker.removeOutputsOnHost(host) to mark clear shuffle status on the host or mapOutputTracker.removeOutputsOnExecutor(execId) to mark clear shuffle status on the executor when reduce stage fetches failed and the executor is active, while it does NOT nothing when the executor lost happened, which is really bad .

So we should distinguish the failedEpoch of 'executor lost' from the fetchFailedEpoch of 'fetch failed' to solve the above problem.

Why are the changes needed?

The regression has been described above.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Add unittests.

weixiuli · 2019-10-22T11:19:47Z

@cloud-fan @gatorsmile Kindly review, thanks.

AmplabJenkins · 2019-10-22T11:22:56Z

Can one of the admins verify this patch?

cloud-fan · 2019-10-22T12:07:26Z

Can you explain what is the regression? I have no idea what is the problem you are trying to fix.

weixiuli · 2019-10-23T02:21:21Z

When an executor lost with some reason and some things (eg:. the external shuffle service or host lost on the executor's host.) happened, and the executor loses time happens to be reduce stage fetch failed from it which is really bad, the previous only call mapOutputTracker.unregisterMapOutput(shuffleId, mapIndex, bmAddress) to mark one map as broken in the map stage at this time , but other maps on the executor are also not available which can only be resubmitted by a nest retry stage which is the regression.

As we all know that the previous will call mapOutputTracker.removeOutputsOnHost(host) or
mapOutputTracker.removeOutputsOnExecutor(execId) when reduce stage fetches failed and the executor is active, while it does NOT for the above problems.

So we should distinguish the failedEpoch of 'executor lost' from the fetchFailedEpoch of 'fetch failed' to solve the above problem.

@cloud-fan

…lost - There will be a regression when the executor lost and then causes 'fetch failed'. - Add fetchFailedEpoch to solve the above problem.

weixiuli · 2019-10-23T09:28:19Z

@cloud-fan @gatorsmile @dongjoon-hyun PTAL.

tgravescs · 2019-10-23T13:22:02Z

can you please add the description to the first message. Please change the title to be what is the description of change. Please specify if this is a regression from spark 2.4 or if you are proposing a new feature. You talk about the specific code in mapoutputtracker but can you please say what the user sees - does the job fail, does it hang, etc.,. fetch failed is a large category.

cloud-fan · 2019-10-23T17:02:27Z

Yea, please describe the problem from an end-user's perspective.

gatorsmile · 2019-10-23T17:29:10Z

cc @jiangxb1987 @Ngone51

dongjoon-hyun · 2019-10-23T18:32:57Z

 // TODO: Garbage collect information about failure epochs when we know there are no more
 // stray messages to detect.
 private val failedEpoch = new HashMap[String, Long]
+ // There will be a regression when an executor lost and then causes 'fetch failed'.


There will be a regression is misleading because this mean this PR causes a regression.

dongjoon-hyun · 2019-10-23T18:34:46Z

 assert(mapStatus2(2).location.host === "hostB")
 }

+ test("All shuffle files on the executor should be cleaned up when executor lost " +


For the bug fix PR, we had better add SPARK-29551 prefix for the test case name.

dongjoon-hyun · 2019-10-23T18:35:44Z

 assert(initialMapStatus1.map{_.mapId}.toSet === Set(5, 6, 7))

 val initialMapStatus2 = mapOutputTracker.shuffleStatuses(secondShuffleId).mapStatuses
- // val initialMapStatus1 = mapOutputTracker.mapStatuses.get(0).get


Let's revert this change. This kind clean up is good in general of course, but this makes the backport difficult.

dongjoon-hyun · 2019-10-23T18:37:26Z

+ afterEach()
+ val conf = new SparkConf()
+ conf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
+ conf.set("spark.files.fetchFailure.unRegisterOutputOnHost", "false")


nit. "spark.files.fetchFailure.unRegisterOutputOnHost" -> UNREGISTER_OUTPUT_ON_HOST_ON_FETCH_FAILURE.key?

dongjoon-hyun · 2019-10-23T18:38:14Z

+ ))
+ // make sure our test setup is correct
+ val initialMapStatus1 = mapOutputTracker.shuffleStatuses(firstShuffleId).mapStatuses
+ // val initialMapStatus1 = mapOutputTracker.mapStatuses.get(0).get


Shall we remove this commented code?

dongjoon-hyun · 2019-10-23T18:38:56Z

+ assert(mapStatus2(2).location.executorId === "exec-hostB")
+ assert(mapStatus2(2).location.host === "hostB")
+ }
+ }


Indentation seems to be corrupted in some point before this line.

dongjoon-hyun · 2019-10-23T18:40:17Z

+ }
+ }
+
+ test("All shuffle files on the host should be cleaned up when host lost") {


The same comments for this test case (test case name, use config key instead of string literal, indentation)

dongjoon-hyun

Hi @weixiuli .
According to the previous comment, you need to keep the PR description up-to-date.
For example, #26206 (comment) should be in the PR description because the PR description will be a commit log eventually.

weixiuli · 2019-10-24T01:51:38Z

@dongjoon-hyun @cloud-fan @tgravescs @jiangxb1987 @Ngone51 @gatorsmile

cloud-fan · 2019-10-24T07:53:51Z

Does this PR introduce any user-facing change

Can you fill in this section? If it's a bug fix, how does the bug affect end users? Or it's only for performance?

jiangxb1987 · 2019-10-24T09:44:44Z

What problem are you trying to resolve here ？If some jobs failed, can you please attach the related logs in the JIRA ?

weixiuli · 2019-10-24T09:58:23Z

Some more stages of a job will be submitted and the job finish very slowly without this PR when the above happens . The user-affecting which I have described in the section.

AngersZhuuuu · 2019-10-25T10:29:35Z

Seems got your point, for fetch fail, you want it don't remove executor, if remove it, we need to re compute stage. In you pr, you allow it retry in task level, not just remove executor.

But what confused me is that, fetch data have retry mechanism ，we have add some pr to fix fetch failed problem in lower level such as :
#24533
#25469
#23590

Do we need to do that in this place.?

joshrosen-stripe · 2019-10-25T17:06:01Z

Disclaimer: I haven't read this thread in detail yet, so maybe this was already discussed.

Is this PR related to the problems I describe in SPARK-27736 "Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations"?

weixiuli · 2019-10-31T09:45:29Z

@joshrosen-stripe Yeah, the PR may solve the problem which has been discussed in SPARK-27736.

cloud-fan · 2019-10-31T09:59:45Z

@weixiuli can you describe the problem and your fix at high-level? i.e. do not mention the detailed code.

As we all know that the previous will callmapOutputTracker.removeOutputsOnHost(host)or
mapOutputTracker.removeOutputsOnExecutor(execId) when reduce stage fetches failed and the executor is active, while it does NOT for the above problems.

This is pretty hard to understand, please use phrases like "clear shuffle status for a host", "clear shuffle status for a mapper", etc.

turboFei · 2019-10-31T10:00:13Z

@joshrosen-stripe Yeah, the PR may solve the problem which has been discussed in SPARK-27736.

Just find this PR. We met the issue described by SPARK-27736 recently. And for some other reasons, we don't enable nodeManager recovery.
I also create a PR #26272, I think it is a supplement.

weixiuli · 2019-11-14T10:04:28Z

@cloud-fan @dongjoon-hyun @tgravescs @jiangxb1987 PTAL.

github-actions · 2020-02-23T00:12:20Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

weixiuli force-pushed the issues/SPARK-29551 branch from 1607bb4 to 78bf757Compare October 22, 2019 11:43

dongjoon-hyun added the SPARK CORE label Oct 22, 2019

[SPARK-29551][CORE] Fix a bug about fetch failed when an executor is …
e1574fe
…lost - There will be a regression when the executor lost and then causes 'fetch failed'. - Add fetchFailedEpoch to solve the above problem.

weixiuli force-pushed the issues/SPARK-29551 branch from 78bf757 to e1574feCompare October 23, 2019 03:31

gatorsmile changed the title ~~[SPARK-29551][CORE] Fix a bug about fetch failed when an executor is …~~[SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lostOct 23, 2019

dongjoon-hyun reviewed Oct 23, 2019
View reviewed changes

dongjoon-hyun requested changes Oct 23, 2019
View reviewed changes

Address comments
f23336d

github-actionsBot added the Stale label Feb 23, 2020

github-actionsBot closed this Feb 24, 2020

Uh oh!

Conversation

weixiuli commented Oct 22, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

weixiuli commented Oct 22, 2019

Uh oh!

AmplabJenkins commented Oct 22, 2019

Uh oh!

cloud-fan commented Oct 22, 2019

Uh oh!

weixiuli commented Oct 23, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weixiuli commented Oct 23, 2019

Uh oh!

tgravescs commented Oct 23, 2019

Uh oh!

cloud-fan commented Oct 23, 2019

Uh oh!

gatorsmile commented Oct 23, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

weixiuli commented Oct 24, 2019

Uh oh!

cloud-fan commented Oct 24, 2019

Uh oh!

jiangxb1987 commented Oct 24, 2019

Uh oh!

weixiuli commented Oct 24, 2019

Uh oh!

AngersZhuuuu commented Oct 25, 2019

Uh oh!

joshrosen-stripe commented Oct 25, 2019

Uh oh!

weixiuli commented Oct 31, 2019

Uh oh!

cloud-fan commented Oct 31, 2019

Uh oh!

turboFei commented Oct 31, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weixiuli commented Nov 14, 2019

Uh oh!

github-actionsBot commented Feb 23, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

weixiuli commented Oct 22, 2019•
edited
Loading

weixiuli commented Oct 23, 2019•
edited
Loading

turboFei commented Oct 31, 2019•
edited
Loading