[SPARK-20814][mesos] Restore support for spark.executor.extraClassPath.

[vanzin]

Restore code that was removed as part of SPARK-17979, but instead of
using the deprecated env variable name to propagate the class path, use
a new one.

Verified by running "./bin/spark-class o.a.s.executor.CoarseGrainedExecutorBackend"
manually.

[vanzin]

@mgummelt

[SparkQA]

Test build #77106 has finished for PR 18037 at commit d129327.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #77129 has finished for PR 18037 at commit a861819.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

Ping? Let me invoke @srowen or @squito since we should have this in 2.2.0.

[mgummelt]

looking...

[mgummelt]

Was spark.executor.extraClassPath ever supported in Mesos? I see no code for it.

[vanzin]

Yes, it just set SPARK_CLASSPATH and relied on the launcher code to respect that env variable, but that code was recently removed.

[srowen]

I trust your judgment on this one

[mgummelt]

It looks like Standalone gets around this by constructing a raw java command instead of a spark-class command. Will YARN be using this new SPARK_EXECUTOR_CLASSPATH variable as well?

I don't really see why we would have deprecated SPARK_CLASSPATH only to introduce SPARK_EXECUTOR_CLASSPATH, but I'm fine with this change.

[vanzin]

Neither standalone nor YARN use env variables to create the executor's classpath, so I guess that's why Mesos was missed. I'd rather Mesos stop using env variables too, but that's a bigger change, and I want to get this in 2.2.0 to avoid a pretty user-visible regression.

[mgummelt]

It looks like YARN just sets the generic java CLASSPATH env var. Maybe Mesos should do that too, in the future.

[vanzin]

YARN does not run spark-class, Mesos does. And spark-class does not honor CLASSPATH, IIRC.

[vanzin]

I'm gonna merge this to unblock 2.2; if there's a desire to clean up this code later, we can do it separately.