Skip to content

[SPARK-22292][Mesos] Added spark.mem.max support for Mesos - #19510

Closed
windkit wants to merge 4 commits into
apache:masterfrom
windkit:mem_max
Closed

[SPARK-22292][Mesos] Added spark.mem.max support for Mesos#19510
windkit wants to merge 4 commits into
apache:masterfrom
windkit:mem_max

Conversation

@windkit

@windkitwindkit commented Oct 17, 2017

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

To limit the amount of resources a spark job accept from Mesos, currently we can only use spark.cores.max to limit in terms of cpu cores.
However, when we have big memory executors, it would consume all the resources.

This PR added spark.mem.max option for Mesos

How was this patch tested?

Added Unit Test

skonto
skonto reviewed Oct 18, 2017
View reviewed changes
Comment thread...scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala Outdated

@skontoskontoOct 18, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove space

@skonto

skonto commented Oct 18, 2017

Copy link
Copy Markdown
Contributor

@windkit there is an open issue here: https://issues.apache.org/jira/browse/SPARK-22133
Could you please add documentation of the new property along with the old ones.

skonto
skonto reviewed Oct 18, 2017
View reviewed changes
Comment thread...main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala Outdated

@skontoskontoOct 18, 2017•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we defend against minimum values and fail fast? For example default executor memory is 1.4MB. We could calculate the value returned by MesosSchedulerUtils.executorMemory. I don't think these values calculated in canLaunchTask ever change. Same applies for cpus.

@windkitwindkitOct 20, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skonto
For cpus, I think we can compare with minCoresPerExecutor
For mem, calling the MesosSchedulerUtils.executorMemory to get the minimum requirement.

Then at here, we parse the option, check the minimum and if it is too small, throw exception?

@skontoskontoOct 20, 2017•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exception I think would be ok, the idea if something is never going to work let the user know, especially for the novice user. In general, the minimum would be a warning if we dont want an exception thrown.

@windkitwindkitOct 23, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I add the check with this PR or a separate one?

@ArtRandArtRandOct 26, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to have a check similar to

spark/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala

Line 73 in 06df34d

// Maximum number of cores to acquire
but for memory, so that we know we'll "land" on the maximum?

@windkit

windkit commented Oct 20, 2017

Copy link
Copy Markdown
ContributorAuthor

@skonto sure I can add those in, can you point to me where the documentation source code is?

@skonto

skonto commented Oct 20, 2017

Copy link
Copy Markdown
Contributor

@windkithttps://github.com/apache/spark/blob/e2fea8cd6058a807ff4841b496ea345ff0553044/docs/running-on-mesos.md

@skonto

skonto commented Oct 20, 2017

Copy link
Copy Markdown
Contributor

@susanxhuynh pls review.

@windkitwindkit mentioned this pull request Oct 23, 2017
Closed
@susanxhuynh

susanxhuynh commented Oct 26, 2017

Copy link
Copy Markdown
Contributor

@windkit Trying to understand the need for this config ... could you accomplish the same thing by setting spark.cores.max, spark.executor.cores, and spark.executor.memory? Could you give an example scenario where this is needed?

susanxhuynh
susanxhuynh reviewed Oct 26, 2017
View reviewed changes
Comment threaddocs/running-on-mesos.md Outdated

@susanxhuynhsusanxhuynhOct 26, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: 'park'

Comment threaddocs/running-on-mesos.md Outdated

@susanxhuynhsusanxhuynhOct 26, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: 'park'

Comment threaddocs/running-on-mesos.md Outdated

@susanxhuynhsusanxhuynhOct 26, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: 'park'

Comment threaddocs/running-on-mesos.md Outdated

@susanxhuynhsusanxhuynhOct 26, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add "across the cluster (not from each machine)". And, something about there is no maximum if this property is not set.

Comment threaddocs/running-on-mesos.md Outdated

@susanxhuynhsusanxhuynhOct 26, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My suggestion: "Duration for which unused resources are considered declined, when maximum number of cores spark.cores.max has been reached."
@ArtRand Is this the documentation you had in mind in https://issues.apache.org/jira/browse/SPARK-22133 ? Is this enough information for a non-Mesos expert to set this?

@windkit

windkit commented Nov 1, 2017

Copy link
Copy Markdown
ContributorAuthor

@susanxhuynh Thanks for reviewing.
I want to use both spark.mem.max and spark.cores.max to limit resource one task can use within the cluster.
Now I am setting up a common cluster for several users, they are allowed to configure spark.executor.cores and spark.executor.memory according to their need. I then need a limit for both cpu cores and memory.

asfgit pushed a commit that referenced this pull request Nov 8, 2017
@windkit@srowen
[SPARK-22133][DOCS] Documentation for Mesos Reject Offer Configurations
6447d7b
## What changes were proposed in this pull request?
Documentation about Mesos Reject Offer Configurations
## Related PR
#19510 for `spark.mem.max`
Author: Li, YanKit | Wilson | RIT <yankit.li@rakuten.com>
Closes#19555 from windkit/spark_22133.
@skonto

skonto commented May 19, 2018

Copy link
Copy Markdown
Contributor

@windkit Could you update the PR? @susanxhuynh I think this is fine to merge, are you ok?

@AmplabJenkins

AmplabJenkins commented Jun 9, 2018

Copy link
Copy Markdown

Can one of the admins verify this patch?

@HyukjinKwon

HyukjinKwon commented Jul 16, 2018

Copy link
Copy Markdown
Member

gentle ping @windkit

@HyukjinKwonHyukjinKwon mentioned this pull request Jul 16, 2018
Closed
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
@HyukjinKwon@zifeif2
[INFRA] Close stale PR
b5ae1e1
Closesapache#17422Closesapache#17619Closesapache#18034Closesapache#18229Closesapache#18268Closesapache#17973Closesapache#18125Closesapache#18918Closesapache#19274Closesapache#19456Closesapache#19510Closesapache#19420Closesapache#20090Closesapache#20177Closesapache#20304Closesapache#20319Closesapache#20543Closesapache#20437Closesapache#21261Closesapache#21726Closesapache#14653Closesapache#13143Closesapache#17894Closesapache#19758Closesapache#12951Closesapache#17092Closesapache#21240Closesapache#16910Closesapache#12904Closesapache#21731Closesapache#21095
Added:
Closesapache#19233Closesapache#20100Closesapache#21453Closesapache#21455Closesapache#18477
Added:
Closesapache#21812Closesapache#21787
Author: hyukjinkwon <gurwls223@apache.org>
Closesapache#21781 from HyukjinKwon/closing-prs.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

3 more reviewers
@skontoskontoskonto left review comments
@ArtRandArtRandArtRand left review comments
@susanxhuynhsusanxhuynhsusanxhuynh left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@windkit@skonto@susanxhuynh@AmplabJenkins@HyukjinKwon@ArtRand