[SPARK-1953][YARN]yarn client mode Application Master memory size is same as driver memory... by WangTaoTheTonic · Pull Request #3607 · apache/spark

WangTaoTheTonic · 2014-12-04T13:58:16Z

... size

Ways to set Application Master's memory on yarn-client mode:

spark.yarn.am.memory in SparkConf or System Properties
default value 512m

Note: this arguments is only available in yarn-client mode.

…ory size

WangTaoTheTonic · 2014-12-04T14:01:03Z

@tgravescs

SparkQA · 2014-12-04T14:05:15Z

Test build #24141 has started for PR 3607 at commit 0566bb8.

This patch merges cleanly.

tgravescs · 2014-12-04T15:16:24Z

env variables are only for backwards compatibility we shouldn't add them for new configs so can you please remove it.

SparkQA · 2014-12-04T15:30:30Z

Test build #24141 has finished for PR 3607 at commit 0566bb8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-04T15:30:34Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24141/
Test PASSed.

tgravescs · 2014-12-04T15:33:48Z

I am a little bit on the fence here about having this config in spark-submit. I'm not sure if it will cause more confusion since it only applies to client mode. I'm wondering if perhaps we just add the config for now.
@vanzin @andrewor14 thoughts on that since you both commented on the am.extraJavaOptions pr

I'd prefer not to add this to SparkSubmit. I've never seen someone have to fiddle with that value, so my guess is that this is such an uncommon need that those who want to use it wouldn't be bothered by the more verbose "--conf" approach.
Also, should probably add a "memory overhead" config too.

AmplabJenkins · 2014-12-05T02:27:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24159/
Test FAILed.

AmplabJenkins · 2014-12-05T02:32:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24160/
Test FAILed.

WangTaoTheTonic · 2014-12-05T02:47:05Z

Jenkins, test this please.

AmplabJenkins · 2014-12-05T02:52:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24162/
Test FAILed.

WangTaoTheTonic · 2014-12-05T03:10:02Z

Jenkins, test this please.

AmplabJenkins · 2014-12-05T03:12:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24165/
Test FAILed.

WangTaoTheTonic · 2014-12-05T03:15:42Z

FATAL: Failed to fetch from https://github.com/apache/spark.git
hudson.plugins.git.GitException: Failed to fetch from https://github.com/apache/spark.git

Jenkins went wrong?

andrewor14 · 2014-12-05T04:27:10Z

retest this please. Hey @WangTaoTheTonic can you add [YARN] to the title? This will help us sort our PRs better

SparkQA · 2014-12-05T04:33:05Z

Test build #24171 has started for PR 3607 at commit 44e48c2.

This patch merges cleanly.

SparkQA · 2014-12-05T05:53:02Z

Test build #24171 has finished for PR 3607 at commit 44e48c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-05T05:53:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24171/
Test PASSed.

WangTaoTheTonic · 2014-12-05T07:27:40Z

Now 1. --am-memory MEM in SparkSubmit args and 3. SPARK_YARN_AM_MEMORY in System env was removed. spark.yarn.am.memoryOverhead was added.
And I think default values of amMemoryOverhead and executorMemoryOverhead are always 384MB before, so I changed the location of their assignment.

andrewor14 · 2014-12-08T22:54:04Z

This comment is no longer true

andrewor14 · 2014-12-09T00:22:33Z

Hey @WangTaoTheTonic I left a few comments. Also, could you document this? Thanks.

SparkQA · 2014-12-09T03:25:19Z

Test build #24235 has started for PR 3607 at commit ab16bb5.

This patch merges cleanly.

andrewor14 · 2015-01-07T15:57:13Z

This warning doesn't make sense. It's a perfectly reasonable thing for YARN users to set the driver memory in client mode.

andrewor14 · 2015-01-07T15:57:33Z

Hey @WangTaoTheTonic I believe the latest changes correctly addresses my last concern (that in client mode the amMemory shouldn't depend on --driver-memory at all). However, I believe the better approach is to define a var driverMemory along with the other variables, and set this in --driver-memory. Then later in cluster mode we can set amMemory = driverMemory, but not in client mode. Does that make sense?

SparkQA · 2015-01-08T07:52:38Z

Test build #25208 has started for PR 3607 at commit ddcd592.

This patch merges cleanly.

WangTaoTheTonic · 2015-01-08T08:20:10Z

@tgravescs @andrewor14

Sorry to produce so much issues in last rebase.

I've fixed them and tested again on my cluster. Here is the configuration(with yarn.scheduler.minimum-allocation-mb=256 in yarn):

spark.driver.memory=5G
spark.yarn.driver.memoryOverhead=1024
spark.yarn.am.memory=256m
spark.yarn.am.memoryOverhead=256
spark.yarn.executor.memoryOverhead=1024
spark.executor.memory=1g
spark.executor.instances=1

In cluster mode, it will launch two container: one used 6G, another 2G. In client mode they are 512M and 2G.

Then keep spark-defaults.conf unchanged with command:
./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster(yarn-client) --driver-memory 4G --executor-memory 1280m ../lib/spark-examples*.jar

In cluster mode, one used 5G, another 2.25G. In client mode, they are 512M and 2.25G.

SparkQA · 2015-01-08T08:59:49Z

Test build #25208 has finished for PR 3607 at commit ddcd592.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-08T08:59:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25208/
Test PASSed.

andrewor14 · 2015-01-08T18:45:47Z

The default value of 512 is duplicated here and up there. I would do this instead
sparkConf.getOption(amMemKey) .map(Utils.memoryStringToMb) .foreach { mem => amMemory = mem }

andrewor14 · 2015-01-08T18:55:24Z

Hi @WangTaoTheTonic the latest changes look pretty close. I believe the semantics of what should be set in what mode is as discussed. It would be good if other reviewers can confirm this. By the way you will need to rebase to master because ClientBase.scala no longer exists.

vanzin · 2015-01-08T20:43:36Z

I don't think you want to use println here. Also, I'm always a little conflicted about these logs, since they'll show up when you set these values in the default conf file (which would apply to a lot of different jobs unless they explicitly override the conf file). But not a big deal.

As ClientArguments.scala didn't extends Logging class, only println can be used here.
Yep, if user set the config values that never be used in that mode, we should give a prompt.
BTW, spark.driver.memory is used in both modes, so I deleted the meesage about it.

SparkQA · 2015-01-09T01:42:37Z

Test build #25284 has started for PR 3607 at commit 6c1b264.

This patch merges cleanly.

SparkQA · 2015-01-09T01:52:38Z

Test build #25285 has started for PR 3607 at commit d5ceb1b.

This patch merges cleanly.

SparkQA · 2015-01-09T02:47:47Z

Test build #25284 has finished for PR 3607 at commit 6c1b264.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-09T02:47:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25284/
Test PASSed.

SparkQA · 2015-01-09T03:01:22Z

Test build #25285 has finished for PR 3607 at commit d5ceb1b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-09T03:01:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25285/
Test PASSed.

andrewor14 · 2015-01-09T21:19:09Z

unindent, I will fix when I merge

andrewor14 · 2015-01-09T21:19:47Z

Ok I'm merging this into master thanks @WangTaoTheTonic for your repeated updates.

WangTaoTheTonic · 2015-01-10T00:45:11Z

Oh gosh it is merged finally.
Thanks guys for persistent comments. @andrewor14 @tgravescs @vanzin @sryza

yarn client mode Application Master memory size is same as driver mem…
0566bb8
…ory size

tgravescs reviewed Dec 4, 2014
View reviewed changes

add overhead mem and remove some configs
6fd13e1

minor fix
44e48c2

WangTaoTheTonic changed the title ~~[SPARK-1953]yarn client mode Application Master memory size is same as driver memory...~~[SPARK-1953][YARN]yarn client mode Application Master memory size is same as driver memory...Dec 5, 2014

andrewor14 reviewed Dec 8, 2014
View reviewed changes

fix bug and add comments
ab16bb5

andrewor14 reviewed Jan 7, 2015
View reviewed changes

fix the bug produced in rebase and some improvements
ddcd592

andrewor14 reviewed Jan 8, 2015
View reviewed changes

vanzin reviewed Jan 8, 2015
View reviewed changes

WangTaoTheTonic added 2 commits January 9, 2015 09:30

minor optiminzation
b8410c0

rebase
6c1b264

spark.driver.memeory is used in both modes
d5ceb1b

andrewor14 reviewed Jan 9, 2015
View reviewed changes

Comment threadyarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
andrewor14Jan 9, 2015
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unindent, I will fix when I merge

asfgit closed this in e966452Jan 9, 2015

andrewor14 mentioned this pull request Feb 26, 2015
[SPARK-5951][YARN] Remove unreachable driver memory properties in yarn client mode #4730
Closed

Uh oh!

Conversation

WangTaoTheTonic commented Dec 4, 2014

Uh oh!

WangTaoTheTonic commented Dec 4, 2014

Uh oh!

SparkQA commented Dec 4, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 4, 2014

Uh oh!

AmplabJenkins commented Dec 4, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Dec 5, 2014

Uh oh!

AmplabJenkins commented Dec 5, 2014

Uh oh!

WangTaoTheTonic commented Dec 5, 2014

Uh oh!

AmplabJenkins commented Dec 5, 2014

Uh oh!

WangTaoTheTonic commented Dec 5, 2014

Uh oh!

AmplabJenkins commented Dec 5, 2014

Uh oh!

WangTaoTheTonic commented Dec 5, 2014

Uh oh!

andrewor14 commented Dec 5, 2014

Uh oh!

SparkQA commented Dec 5, 2014

Uh oh!

SparkQA commented Dec 5, 2014

Uh oh!

AmplabJenkins commented Dec 5, 2014

Uh oh!

WangTaoTheTonic commented Dec 5, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Dec 9, 2014

Uh oh!

SparkQA commented Dec 9, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Jan 7, 2015

Uh oh!

SparkQA commented Jan 8, 2015

Uh oh!

WangTaoTheTonic commented Jan 8, 2015

Uh oh!

SparkQA commented Jan 8, 2015

Uh oh!

AmplabJenkins commented Jan 8, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Jan 8, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 9, 2015

Uh oh!

SparkQA commented Jan 9, 2015

Uh oh!

SparkQA commented Jan 9, 2015

Uh oh!

AmplabJenkins commented Jan 9, 2015

Uh oh!

SparkQA commented Jan 9, 2015

Uh oh!