[SPARK-12832][MESOS] mesos scheduler respect agent attributes by atongen · Pull Request #10949 · apache/spark

atongen · 2016-01-27T14:42:54Z

We have a similar need to what is proposed in #10768 by @Astralidea and reviewed by @dragos. This pull request implements the suggestion in the comments of that PR.

It takes logic similar to what is found in #5563 for the executors and applies it to the mesos cluster scheduler.

The scheduler will now only accept offers from agents with attributes matching the constraints from the submission.

dragos · 2016-01-27T14:47:53Z

ok to test

dragos · 2016-01-27T14:48:12Z

I'll try it out. The code looks good otherwise, thanks for picking up that PR!

atongen · 2016-01-29T18:34:39Z

Hi @dragos, I'm not sure how to grant access rights to the CI server, but let me know if there's anything I can do to help out here. Thanks.

dragos · 2016-01-29T19:34:37Z

The failure is spurious, git failed to check out.

On 29 ian. 2016, at 19:35, Andrew Tongen notifications@github.com wrote:
Hi @dragos, I'm not sure how to grant access rights to the CI server, but let me know if there's anything I can do to help out here. Thanks.
—
Reply to this email directly or view it on GitHub.

andrewor14 · 2016-02-01T20:34:41Z

retest this please

andrewor14 · 2016-02-01T20:35:13Z

this will fail scalastyle tests; the line needs to be below 100 chars

andrewor14 · 2016-02-01T20:38:17Z

Also cc @tnachen who wrote this code originally. From the JIRA:

CoarseMesosSchedulerBackend have constraints feature but dispacher deploy use MesosClusterScheduler, it is different method.

Is this caused by duplicate code somewhere? Can we resolve that, either in this patch or separately?

andrewor14 · 2016-02-01T20:40:38Z

By the way, I'd just like to point out that there is another patch that fixes the same issue #10768. @tnachen @dragos what's the difference and which one should we proceed with?

SparkQA · 2016-02-01T21:04:12Z

Test build #50500 has finished for PR 10949 at commit 7c6650d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

dragos · 2016-02-02T10:29:42Z

@andrewor14 this is the one that should go forward. The first sentence of this PR says:

We have a similar need to what is proposed in #10768 by @Astralidea and reviewed by @dragos. This pull request implements the suggestion in the comments of that PR.

Unfortunately I can't close #10768.

dragos · 2016-02-02T10:33:22Z

Regarding sharing code: The logic to check constraints is already shared. The actual resource processing isn't. Maybe there is room to share more logic. I opened SPARK-10444.

SparkQA · 2016-02-11T15:21:15Z

Test build #51103 has started for PR 10949 at commit 6c934bd.

atongen · 2016-02-11T15:23:54Z

Pushed changes to address scalastyle test failures.

Also, in order to run multiple dispatchers on the same mesos cluster, you should set spark.deploy.zookeeper.dir to something other than the default, which is "/spark_mesos_dispatcher", for each dispatcher or mesos will use the same zookeeper path and there will be conflicts.

shaneknapp · 2016-02-11T16:07:49Z

jenkins, test this please

SparkQA · 2016-02-11T18:05:34Z

Test build #51113 has finished for PR 10949 at commit 6c934bd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tnachen · 2016-02-12T00:21:08Z

Btw can you add a quick unit test for this? We've added tests before already so should be straightforward to do so.

dragos · 2016-02-16T14:43:30Z

Good idea about the unit test. I don't think it's too hard to add one along the lines of what's already in MesosClusterSchedulerSuite.

dragos · 2016-02-23T16:21:55Z

Hey, @atongen will you have time to look into the additional test?

atongen · 2016-02-26T21:14:26Z

Yes, I'll be able to add tests. Sorry for the delay!

BrickXu · 2016-03-08T09:28:56Z

any updates here ?

Astralidea · 2016-03-11T05:14:00Z

@atongen this PR dose not merge into spark 1.6.1, I hope it could merge it to some version like 1.6.2.

atongen · 2016-03-16T21:57:08Z

@tnachen, @dragos : I reviewed the tests introduced by #5563, and from what I can tell they are mainly testing only MesosSchedulerUtils#matchesAttributeRequirements in regard to attribute constraints; which is the only significant thing introduced by this PR.

Without further refactoring, testing at the next level up (MesosClusterScheduler#scheduleTasks) would require a functioning MesosClusterPersistenceEngine (not the black hole), and quite a bit of additional scaffolding.

Let me know if this is still a requirement. I would like to confirm before putting in the effort.

tnachen · 2016-03-16T22:29:53Z

I think we should add tests and I don't think it requires that much refactoring, if you look at MesosClusterSchedulerSuite you can see the test "can handle multiple roles" already tries a submission and verifies it uses the Offer passed in, we can also test by doing a similiar setup where we have a Offer with attributes and without and verify it's performing the correct logic. Tests is very important and we're looking to really increase our coverage as it's getting harder and harder to catch things.

atongen · 2016-03-17T15:20:35Z

Ok, I'll look into it further, thanks!

SparkQA · 2016-03-23T00:27:48Z

Test build #53852 has finished for PR 10949 at commit 82f71bc.

This patch fails R style tests.
This patch does not merge cleanly.
This patch adds no public classes.

dragos · 2016-03-30T11:55:52Z

@atongen can you please rebase? The tests look good, but I'd like to see the test suite passing.

dragos · 2016-04-04T12:31:29Z

Hey @atongen, I think this is really close to being merged, can you please rebase?

dragos · 2016-04-27T14:58:25Z

ping @atongen

atongen · 2016-05-09T20:14:39Z

Hello @dragos, again, sorry for all the delays. This branch has been rebased onto master, but I've been having some trouble getting the test suite to run. Please advise.

SparkQA · 2016-05-09T21:44:56Z

Test build #58169 has finished for PR 10949 at commit e6001e9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

tnachen · 2016-06-02T17:16:54Z

@atongen please rebase and try again.

* mesos scheduler respect agent constraints * reduce line length for scalastyle test * update test suites

SparkQA · 2016-06-13T18:23:13Z

Test build #60406 has finished for PR 10949 at commit ce3047d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

atongen · 2016-06-13T18:55:13Z

@tnachen Tests updated and rebased. Let me know if there's anything else I can do to help out.

evilezh · 2017-02-06T21:36:36Z

any update on this ? It is real pain with driver. As i see patch is ready .. question is about when you can merge ?

AmplabJenkins · 2018-01-30T19:41:25Z

Can one of the admins verify this patch?

atongen · 2018-03-06T22:41:46Z

I am going to close this PR because there doesn't appear to be any interest in getting it merged. It's unfortunate, because it was a nice feature.

dragos · 2018-03-06T23:40:30Z

I think both @tnachen and I have moved on to the non-Spark world in the meantime. Anyway, neither of us had commit rights. I agree it's a pity to drop it, perhaps @andrewor14 could help.

andrewor14 reviewed Feb 1, 2016
View reviewed changes

andrewor14 mentioned this pull request Feb 1, 2016
[SPARK-12832][MESOS] Fix dispatcher does not have a constraints config #10768
Closed

atongen force-pushed the mesos-scheduler-constraints branch from 82f71bc to e6001e9Compare May 9, 2016 20:13

[SPARK-12832][MESOS] mesos scheduler respect agent attributes
ce3047d
* mesos scheduler respect agent constraints * reduce line length for scalastyle test * update test suites

atongen force-pushed the mesos-scheduler-constraints branch from e6001e9 to ce3047dCompare June 13, 2016 16:31

atongen closed this Mar 6, 2018

Uh oh!

Conversation

atongen commented Jan 27, 2016

Uh oh!

dragos commented Jan 27, 2016

Uh oh!

dragos commented Jan 27, 2016

Uh oh!

atongen commented Jan 29, 2016

Uh oh!

dragos commented Jan 29, 2016

Uh oh!

andrewor14 commented Feb 1, 2016

Uh oh!

andrewor14Feb 1, 2016

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Feb 1, 2016

Uh oh!

andrewor14 commented Feb 1, 2016

Uh oh!

SparkQA commented Feb 1, 2016

Uh oh!

dragos commented Feb 2, 2016

Uh oh!

dragos commented Feb 2, 2016

Uh oh!

SparkQA commented Feb 11, 2016

Uh oh!

atongen commented Feb 11, 2016

Uh oh!

shaneknapp commented Feb 11, 2016

Uh oh!

SparkQA commented Feb 11, 2016

Uh oh!

tnachen commented Feb 12, 2016

Uh oh!

dragos commented Feb 16, 2016

Uh oh!

dragos commented Feb 23, 2016

Uh oh!

atongen commented Feb 26, 2016

Uh oh!

BrickXu commented Mar 8, 2016

Uh oh!

Astralidea commented Mar 11, 2016

Uh oh!

atongen commented Mar 16, 2016

Uh oh!

tnachen commented Mar 16, 2016

Uh oh!

atongen commented Mar 17, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

dragos commented Mar 30, 2016

Uh oh!

dragos commented Apr 4, 2016

Uh oh!

dragos commented Apr 27, 2016

Uh oh!

atongen commented May 9, 2016

Uh oh!

SparkQA commented May 9, 2016

Uh oh!

tnachen commented Jun 2, 2016

Uh oh!

SparkQA commented Jun 13, 2016

Uh oh!

atongen commented Jun 13, 2016

Uh oh!

evilezh commented Feb 6, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AmplabJenkins commented Jan 30, 2018

Uh oh!

atongen commented Mar 6, 2018

Uh oh!

dragos commented Mar 6, 2018

Uh oh!

evilezh commented Feb 6, 2017•
edited
Loading