[SPARK-2872] Fix conflict between code and doc in YarnClientSchedulerBackend.scala

[li-zhihui]

Doc say: system properties override environment variables.
https://github.com/apache/spark/blob/master/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala#L71

But code is conflict with it.

[AmplabJenkins]

Can one of the admins verify this patch?

[li-zhihui]

@tgravescs can you please to have a look at this patch?

[tgravescs]

At this point we have released it with env variable overriding configs. I think it would be better to just update the comment (since its just in the code and not user facing) to say what it does vs changing the user behavior.

[li-zhihui]

@tgravescs got it.

But, YarnClusterSchedulerBackend use the same strategy (code and comment).
https://github.com/apache/spark/blob/master/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala#L41
I think we should fix it to env variable overriding config too.

[li-zhihui]

@tgravescs I rollback previous commit and add a new commit just update comment.

[tgravescs]

@li-zhihui good point about YarnClusterSchedulerBackend, Taking a deeper look there is YarnClusterSchedulerBackend and then there are a couple other places in which the conf overrides the ENV variable. So lets keep it consistent and go with your original change. Sorry for the extra work.

[tgravescs]

can you also please file a jira for this and link the PR to it.

[li-zhihui]

@tgravescs done

[tgravescs]

Jenkins, test this please

[SparkQA]

QA tests have started for PR 1684. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18020/consoleFull

[li-zhihui]

@tgravescs
Test failure because " FileServerSuite:Build timed out (after 120 minutes)", but I guess the patch is irrelevant to the issue.

Can you ask jenkins test this again?

[li-zhihui]

@tgravescs

[tgravescs]

sorry I haven't had time to get back to this. Originally there were some issues with the order and make sure all the options worked properly from all the various ways to submit - directly using spark-class and then using spark-submit. I believe one of them was the SPARK_YARN_APP_NAME.

I think other things have been fixed where this shouldn't be an issue anymore but need to verify.

Can you test to make sure they all work, see #539.

[SparkQA]

Can one of the admins verify this patch?

[tgravescs]

Note this is going to conflict with #2350