[SPARK-2098] All Spark processes should support spark-defaults.conf, config file by witgo · Pull Request #1256 · apache/spark

witgo · 2014-06-28T14:15:23Z

No description provided.

AmplabJenkins · 2014-06-28T14:15:30Z

Merged build triggered.

AmplabJenkins · 2014-06-28T14:15:38Z

Merged build started.

AmplabJenkins · 2014-06-28T14:25:30Z

Merged build triggered.

AmplabJenkins · 2014-06-28T14:25:38Z

Merged build started.

AmplabJenkins · 2014-06-28T14:59:40Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-28T14:59:40Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16228/

AmplabJenkins · 2014-06-28T15:08:24Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-28T15:08:24Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16229/

AmplabJenkins · 2014-06-30T16:00:43Z

Merged build triggered.

AmplabJenkins · 2014-06-30T16:00:50Z

Merged build started.

AmplabJenkins · 2014-06-30T16:45:55Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-30T16:45:55Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16267/

vanzin · 2014-07-08T17:19:51Z

This is only called in one place, why the argument? Seems like we always want system properties to override other settings anyway.

vanzin · 2014-07-08T17:47:02Z

I'd like to see better documentation about the semantics of how the different sources for configuration override each other. Also, some tests would be nice.

AmplabJenkins · 2014-07-09T15:31:11Z

Merged build triggered.

AmplabJenkins · 2014-07-09T15:31:17Z

Merged build started.

AmplabJenkins · 2014-07-09T15:41:21Z

Merged build finished.

AmplabJenkins · 2014-07-09T15:41:22Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16465/

witgo · 2014-07-09T15:45:58Z

Jenkins, retest this please.

AmplabJenkins · 2014-07-09T15:46:11Z

Merged build triggered.

AmplabJenkins · 2014-07-09T15:46:17Z

Merged build started.

vanzin · 2014-07-09T17:37:54Z

If you're keeping this, you should document that this method will close the given input stream.

vanzin · 2014-07-30T17:13:16Z

You could use this method instead.

vanzin · 2014-07-30T17:15:07Z

Still a few nits left, but looks ok to me.

SparkQA · 2014-07-31T02:28:59Z

QA tests have started for PR 1256. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17539/consoleFull

SparkQA · 2014-07-31T03:23:49Z

QA results for PR 1256:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkConf(loadDefaults: Boolean, fileName: Option[String])

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17539/consoleFull

SparkQA · 2014-07-31T05:48:57Z

QA tests have started for PR 1256. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17555/consoleFull

SparkQA · 2014-07-31T06:28:49Z

QA results for PR 1256:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkConf(loadDefaults: Boolean, fileName: Option[String])

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17555/consoleFull

witgo · 2014-07-31T06:32:16Z

Jenkins, retest this please.

SparkQA · 2014-07-31T06:34:07Z

QA tests have started for PR 1256. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17564/consoleFull

SparkQA · 2014-07-31T07:30:03Z

QA results for PR 1256:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkConf(loadDefaults: Boolean, fileName: Option[String])

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17564/consoleFull

SparkQA · 2014-08-11T06:39:43Z

QA tests have started for PR 1256. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18302/consoleFull

SparkQA · 2014-08-11T07:30:12Z

QA results for PR 1256:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkConf(loadDefaults: Boolean, fileName: Option[String])

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18302/consoleFull

vanzin · 2014-08-22T00:54:30Z

@pwendell @andrewor14 could you guys take a look at this PR? Thanks!

andrewor14 · 2014-09-11T23:53:02Z

How about "Directory that contains application event logs to be loaded by the history server"

andrewor14 · 2014-09-12T00:14:44Z

@witgo I like the idea of having all processes (HistoryServer, Master, Worker etc.) read from the properties file in addition just SparkSubmit. However, I don't think this scope should include SparkConf, which is more application-specific. It would be good to keep the existing code in SparkSubmitArguments and only move the part where we actually read properties from the file into Utils, then all the other processes can handle --properties-file the same way. Do you understand what I mean?

witgo · 2014-09-12T02:13:40Z

OK, I generally understand what you mean,I will re-implement the feature at the weekend.

andrewor14 · 2014-09-13T08:00:35Z

@witgo would you mind closing this since you opened another one?

witgo · 2014-09-13T08:02:48Z

OK

…config file This is another implementation about #1256 cc andrewor14 vanzin Author: GuoQiang Li <witgo@qq.com> Closes#2379 from witgo/SPARK-2098-new and squashes the following commits: 4ef1cbd [GuoQiang Li] review commit 49ef70e [GuoQiang Li] Refactor getDefaultPropertiesFile c45d20c [GuoQiang Li] All Spark processes should support spark-defaults.conf, config file

* [CARMEL-6541] Support Query Level SQL Conf leveraging Hint * fix code style * fix ut * Add one more test Co-authored-by: Chao Sun <sunchao@apache.org>

witgo mentioned this pull request Jun 28, 2014
[WIP] Loading spark-defaults.conf when creating SparkConf instances #1233
Closed

vanzin reviewed Jul 8, 2014
View reviewed changes

vanzin reviewed Jul 9, 2014
View reviewed changes

vanzin reviewed Jul 30, 2014
View reviewed changes

Comment threadcore/src/test/scala/org/apache/spark/SparkConfSuite.scala Outdated
vanzinJul 30, 2014
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could use this method instead.

All Spark processes should support spark-defaults.conf, config file.
62f5677

andrewor14 reviewed Sep 11, 2014
View reviewed changes

witgo changed the title ~~SPARK-2098: All Spark processes should support spark-defaults.conf, config file~~[SPARK-2098] All Spark processes should support spark-defaults.conf, config fileSep 13, 2014

witgo mentioned this pull request Sep 13, 2014
[SPARK-2098] All Spark processes should support spark-defaults.conf, config file #2379
Closed

witgo closed this Sep 13, 2014

witgo deleted the SPARK-2098 branch October 15, 2014 10:22

Uh oh!

Conversation

witgo commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 28, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

vanzinJul 8, 2014

Choose a reason for hiding this comment

Uh oh!

vanzin commented Jul 8, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

witgo commented Jul 9, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

vanzinJul 9, 2014

Choose a reason for hiding this comment

Uh oh!

vanzinJul 30, 2014

Choose a reason for hiding this comment

Uh oh!

vanzin commented Jul 30, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

witgo commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

vanzin commented Aug 22, 2014

Uh oh!

andrewor14Sep 11, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Sep 12, 2014

Uh oh!