[SPARK-15359] [Mesos] Mesos dispatcher should handle DRIVER_ABORTED status from mesosDriver.run()

[devaraj-kavali]

What changes were proposed in this pull request?

When the mesosDriver.run() returns with the status as DRIVER_ABORTED then throwing the exception which can be handled from SparkUncaughtExceptionHandler to shutdown the dispatcher.

How was this patch tested?

I verified it manually, the driver thread throws exception when mesosDriver.run() returns with the DRIVER_ABORTED status.

[tnachen]

ok to test

[SparkQA]

Test build #59856 has finished for PR 13143 at commit c16fb5f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tnachen]

s/after the successful registration/after registration/g

[tnachen]

Also to simplify the code, can we just throw SparkExecption here? Then the catch will then handle all cases

[tnachen]

Is it because MesosDriver actually threw an exception?

[devaraj-kavali]

MesosDriver doesn't throw any exception, it just returns with the value as Status.DRIVER_ABORTED.

 registerLatch.await()
// propagate any error to the calling thread. This ensures that SparkContext creation fails
// without leaving a broken context that won't be able to schedule any tasks
error.foreach(throw _)

This code handles exceptions and throws if it gets Status.DRIVER_ABORTED during registration, once the registration completes there is no code to handle and will be skipped the status and also thread dies.

[SparkQA]

Test build #70648 has finished for PR 13143 at commit 946ad7d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[devaraj-kavali]

@tnachen Can you check this?

[devaraj-kavali]

@mgummelt /@tnachen, can you have a look into this?

[mgummelt]

What whole function is designed poorly. We need to totally change it instead of tacking this on. We shouldn't be calling driver.run() in a separate thread. We should be calling driver.start()

[devaraj-kavali]

Thanks @mgummelt for the feedback, will update the PR with the function rewrite.

[SparkQA]

Test build #75907 has finished for PR 13143 at commit 131edc3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #75908 has finished for PR 13143 at commit b9893a3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[skonto]

@ArtRand@susanxhuynh could please review this before we call for a merge?

[ArtRand]

@skonto@susanxhuynh@devaraj-kavali are people still interested in this? I was just playing around with this code to clean up ZK state.. Would be happy to try this when I have a few cycles.

[devaraj-kavali]

@ArtRand I think this is still an issue which needs to be merged, do you have any observations with this PR?

[ArtRand]

Hello @devaraj-kavali. Yes. I've been playing around with this because it's inconvenient to clean up ZK whenever you uninstall/reinstall the Dispatcher. The problem is that the only signal of a re-install vs. a failover is when Mesos gives you a DRIVER_ABORTED error (signals a re-install). I think a solution will involve having different methods to starting SchedulerDrivers for Spark Drivers (applications) and the Dispatcher because they inherently have different requirements with respect to state.

[AmplabJenkins]

Can one of the admins verify this patch?

[HyukjinKwon]

ping @devaraj-kavali for @ArtRand's comment above.