[SPARK-4166][Core][WebUI] Display the executor ID in the Web UI when ExecutorLostFailure happens

[zsxwing]

Now when ExecutorLostFailure happens, it only displays ExecutorLostFailure (executor lost). Adding the executor id will help locate the faulted executor.

[SparkQA]

Test build #22603 has started for PR 3033 at commit c5c4cf2.

• This patch merges cleanly.

[SparkQA]

Test build #22603 has finished for PR 3033 at commit c5c4cf2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class ExecutorLostFailure(execId: String) extends TaskFailedReason

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22603/
Test PASSed.

[JoshRosen]

This looks good to me.

[JoshRosen]

Actually, there might be a backwards-compatibility concern here when event logs written by older Spark versions. Ping @andrewor14 for review.

[andrewor14]

Hey @zsxwing can you attach a screenshot of what it looks like before and after?

[andrewor14]

Yes this needs to be backward compatible.

[JoshRosen]

I guess this can just be extractOpt with a default executor ID of "unknown"

[andrewor14]

There's a Utils.jsonOption that we can use here

[JoshRosen]

@andrewor14 I think #3032 is the web UI display half of this PR, which contains screenshots; this looks like it's just adding additional information to ExecutorLostFailure.

[andrewor14]

Oh I see, this patch doesn't make any changes to the UI itself

[SparkQA]

Test build #22680 has started for PR 3033 at commit ff4664c.

• This patch merges cleanly.

[zsxwing]

I updated to use Utils.jsonOption to support backward compatible. Also update #3032

[SparkQA]

Test build #22680 has finished for PR 3033 at commit ff4664c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class ExecutorLostFailure(execId: String) extends TaskFailedReason

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22680/
Test PASSed.

[JoshRosen]

This looks good to me; I compared this against other recent changes to JsonProtocol and it seems to handle backwards-compatibility in the proper way, so I'm going to merge it. Thanks!

[andrewor14]

Hey @zsxwing actually can you add a backward compatibility test in JsonProtocolSuite? We actually require a test suite if people want to change the signature of the relevant classes.

[zsxwing]

Already added it in #3085