[SPARK-6327] [PySpark] fix launch spark-submit from python

[davies]

SparkSubmit should be launched without setting PYSPARK_SUBMIT_ARGS

cc @JoshRosen , this mode is actually used by python unit test, so I will not add more test for it.

[SparkQA]

Test build #28591 has started for PR 5019 at commit 2c20b0c.

• This patch merges cleanly.

[SparkQA]

Test build #28591 has finished for PR 5019 at commit 2c20b0c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28591/
Test PASSed.

[vanzin]

LGTM.

[jerryshao]

Hi @vanzin@davies , seems this change hides the pyspark-shell into the python code, so basically if we want to add some additional arguments with PYSPARK_SUBMIT_ARGS, do we need to override this environment variable as what I mentioned in #3916 , or how to handle this? Thanks a lot.

[vanzin]

@jerryshao you should be able to just set that variable. Doesn't that work for you? Davies's fix actually allows you to do that (before it would be overridden by the pyspark script).

[davies]

@jerryshao You can still PYSPARK_SUBMIT_ARGS by having pyspark-shell in the end. It's not perfect, but it's fine for a internal things.

[jerryshao]

OK, get it, thanks a lot.

[JoshRosen]

I was able to reproduce this bug locally and have verified that it's fixed by this patch. LGTM, so I'm going to merge this into master (1.4.0). Thanks!