[SPARK-12345] [CORE] Do not send SPARK_HOME through Spark submit REST interface

[skyluc]

It is usually an invalid location on the remote machine executing the job.
It is picked up by the Mesos support in cluster mode, and most of the time causes
the job to fail.

Fixes SPARK-12345

[andrewor14]

add to whitelist

[andrewor14]

This seems like a better place to filter it out. My only concern is that this one also affects standalone cluster mode, whereas #10332 is totally a Mesos thing.

[andrewor14]

I will go ahead and do a quick test on standalone cluster mode with this fix and report back. If there are no problems, we can just go ahead and merge this to unblock the release, since you guys already ran the tests.

[andrewor14]

we should add a comment here to explain why we need to filter out SPARK_HOME, and point to the JIRA. (though we can always do that in a separate patch so as to unblock the release)

[tnachen]

I have a fix that only affects Mesos cluster mode #10332
If standalone never had a problem then I suggest we don't affect it.

[andrewor14]

retest this please

[andrewor14]

Though we should probably never pass SPARK_HOME of the client machine to the cluster. For master I think this is the better fix, though it's a little late to merge this into 1.6.

[tnachen]

SGTM, I don't think we ever should either.

[andrewor14]

retest this please

[SparkQA]

Test build #2221 has finished for PR 10329 at commit f75815b.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

@skyluc looks like there is a style violation with this patch

[SparkQA]

Test build #47845 has finished for PR 10329 at commit f75815b.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[skyluc]

Fixed the 'style', added a comment, and switch to filterKeys.

[SparkQA]

Test build #47926 has finished for PR 10329 at commit 62f4d2f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

This is a better place to filter out SPARK_HOME than #10332. I'm going to merge this into master and revert the following patches in master:

• [SPARK-12345][MESOS] Filter SPARK_HOME when submitting Spark jobs with Mesos cluster mode. #10332 - which doesn't actually fixes the issue
• [SPARK-12345][Mesos] Properly filter out SPARK_HOME in the Mesos REST server #10359 - which fixes the previous patch
• [SPARK-12413] Fix Mesos ZK persistence #10366 - which fixes the HA issue introduced in the previous patch

Note: I'm only reverting these patches in master, but not in branch-1.6.

[tnachen]

@andrewor14 thanks a lot for the patience on this, this validates we need to really invest in automated testing a lot of these things and hopefully don't repeat again the same mistakes. Hopefully soon every Mesos tagged patch runs through these tests as well.

[andrewor14]

👍