[SPARK-16304] LinkageError should not crash Spark executor

[petermaxlee]

What changes were proposed in this pull request?

This patch updates the failure handling logic so Spark executor does not crash when seeing LinkageError.

How was this patch tested?

Added an end-to-end test in FailureSuite.

[SparkQA]

Test build #3149 has finished for PR 13982 at commit 45f5003.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

cc @JoshRosen and @ericl

[srowen]

This can just be e.instanceOf[LinkageError] but an entire method for this is not appropriate

[SparkQA]

Test build #3160 has finished for PR 13982 at commit 050e524.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

@srowen any more feedback?

[srowen]

I don't feel strongly about it. It's something easy to revisit later to change if it turns out special-casing this is undesirable.

[rxin]

Alright merging in master. Thanks.

[zsxwing]

@rxin any objection to merge this one into 2.0? I noticed sometimes spark-shell will throw NoClassDefFoundError(a subclass of LinkageError) which just crashes the executor.

[rxin]

No objection.

On Wednesday, October 26, 2016, Shixiong Zhu notifications@github.com
wrote:

@rxinhttps://github.com/rxin any objection to merge this one into 2.0?
I noticed sometimes spark-shell will throw NoClassDefFoundError(a
subclass of LinkageError) which just crashes the executor.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#13982 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AATvPCFOsMrYaPmxf3VwHWN-H7MDsndJks5q3pwKgaJpZM4JBp3m
.

[zsxwing]

thanks! Merging to 2.0.