[SPARK-17512][Core] Avoid formatting to python path for yarn and mesos cluster mode

[jerryshao]

What changes were proposed in this pull request?

Yarn and mesos cluster mode support remote python path (HDFS/S3 scheme) by their own mechanism, it is not necessary to check and format the python when running on these modes. This is a potential regression compared to 1.6, so here propose to fix it.

How was this patch tested?

Unit test to verify SparkSubmit arguments, also with local cluster verification. Because of lack of MiniDFSCluster support in Spark unit test, there's no integration test added.

[SparkQA]

Test build #65557 has finished for PR 15137 at commit d136155.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

Good catch. I'm merging this into master and 2.0. Thanks.

[andrewor14]

I've merged it. One more thing, would you mind correcting the comment in PythonRunner#formatPath? Right now it says "we currently only support local python files", which is apparently not true!

[jerryshao]

Try to think in another way from PythonRunner's point, this comment looks correct. For yarn and mesos cluster mode, it is because we leverage distributed cache or other to download python files to local workspace. But from PythonRunner's point, it still can only support local files.

[andrewor14]

Got it. I think back then when I wrote that comment we still haven't supported cluster mode with python yet. It's just a little confusing if we're reading that comment in isolation.

[steveloughran]

I see this in master; but the JIRA associated with the PR is still opened & unversioned. Which version did it make it into?

[srowen]

From the commits, it's in 2.0.1 and 2.1.0. Looks like the JIRA just wasn't resolved (I'll do that).