[SPARK-11872] Prevent the call to SparkContext#stop() in the listener bus's thread

[tedyu]

This is continuation of SPARK-11761

Andrew suggested adding this protection. See tail of #9741

[SparkQA]

Test build #46376 has finished for PR 9852 at commit 19c144e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tedyu]

@andrewor14@zsxwing
Kindly review

[zsxwing]

@tedyu could you add a unit test? Just to avoid someone breaking it in future.

[tedyu]

Should I add test in StreamingListenerSuite.scala (stopping SparkContext instead of stopping StreamingContext) ?
Which notification should be overridden ?

Thanks

[zsxwing]

You can add the test to SparkListenerSuite in core

[SparkQA]

Test build #46430 has finished for PR 9852 at commit 15fb7b2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

retest this please

[tedyu]

[error] Failed: Total 378, Failed 1, Errors 0, Passed 377, Ignored 2
[error] Failed tests:
[error] org.apache.spark.streaming.CheckpointSuite
[info] ExecutorClassLoaderSuite:
[info] - child first (4 seconds, 178 milliseconds)
[info] - parent first (49 milliseconds)
[info] - child first can fall back (47 milliseconds)
[info] - child first can fail (44 milliseconds)
[info] - failing to fetch classes from HTTP server should not leak resources (SPARK-6209) *** FAILED *** (1 second, 886 milliseconds)
[info] java.lang.ClassNotFoundException: ReplFakeClass2
[info] at org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:68)

I don't think the above failures were related to my change

[tedyu]

Manually ran org.apache.spark.streaming.CheckpointSuite (with this change) which passed.

[SparkQA]

Test build #46445 has finished for PR 9852 at commit 15fb7b2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

LGTM

[tedyu]

@zsxwing@andrewor14
Is there anything I need to do ?

[zsxwing]

Merged to master and 1.6. Thanks!