SPARK-2116

[chu11]

If SPARK_CONF_DIR environment variable is set, search it for spark-defaults.conf.

[AmplabJenkins]

Can one of the admins verify this patch?

[vanzin]

Very similar to: #997

[mateiz]

When are you guys running into this, is it when you launch an app through an IDE or something like that?

[chu11]

No, just launching normally via command line.

The purpose of this patch is convenience. Its purpose is similar to the HADOOP_CONF_DIR, HBASE_CONF_DIR, etc. environment variables that exist with Hadoop, Hbase, etc.

[mateiz]

Alright, but I'm actually kind of confused then, won't spark-submit find its own conf dir from SPARK_HOME? Is that somehow missing in binary builds of Spark, or something like that? Because the way we envision this being used is that people install Spark in a location and call the spark-submit from that location, maybe adding it to their path. I guess you want one installation of Spark but multiple conf dirs?

[mateiz]

BTW for that second case, we could support it, but then we'd also need to make the start-cluster scripts support multiple conf dirs. If you'd like to do that, update the JIRA to say this, and we should patch all of them at once.

[chu11]

I wouldn't say "multiple" conf directories, but alternate ones from the default. In Hadoop I can stick all my config files in a /tmp/foo directory, set HADOOP_CONF_DIR, and hadoop will read all of its configuration files out of there instead of its default location.

spark-env.sh is already searched for in SPARK_CONF_DIR via load-spark-env.sh, so that isn't a problem. However, spark-defaults.conf is not searched for in SPARK_CONF_DIR. So I can't put all the config files in one directory.

I hope that clarifies things?

[mateiz]

I see, that makes sense. I didn't realize we supported a different conf dir for spark-env.sh through that variable.

Jenkins, test this please

[mateiz]

test this please

pretty please?

[mateiz]

Jenkins, test this please

[SparkQA]

QA tests have started for PR 1059. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17697/consoleFull

[SparkQA]

QA results for PR 1059:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17697/consoleFull

[mateiz]

Jenkins, test this please

[SparkQA]

QA tests have started for PR 1059. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17704/consoleFull

[SparkQA]

QA results for PR 1059:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17704/consoleFull

[mateiz]

Merged this into 1.1. Thanks!