[SPARK-19956][Core]Optimize a location order of blocks with topology information by ConeyLiu · Pull Request #17300 · apache/spark

ConeyLiu · 2017-03-15T03:26:26Z

What changes were proposed in this pull request?

When call the method getLocations of BlockManager, we only compare the data block host. Random selection for non-local data blocks, this may cause the selected data block to be in a different rack. So in this patch to increase the sort of the rack.

How was this patch tested?

New test case.

Please review http://spark.apache.org/contributing.html before opening a pull request.

jerryshao · 2017-03-22T03:51:19Z

The fix LGTM, I think it is nice to have such topology priority. CC @mridulm .

mridulm · 2017-03-22T08:20:47Z

+ val getLocations = PrivateMethod[Seq[BlockManagerId]]('getLocations)
+ val locations = blockManager invokePrivate getLocations(BroadcastBlockId(0))
+ assert(locations.map(_.host).toSet
+ === Set(localHost, localHost, otherHost, otherHost, otherHost))


Remove toSet and make Set as Seq ?
Making it a Set sort of looses the point of this change (ordering is lost).
Same for topology info below below.

mridulm · 2017-03-22T08:22:33Z

 /**
 * Return a list of locations for the given block, prioritizing the local machine since
- * multiple block managers can share the same host.
+ * multiple block managers can share the same host, then try to get the same rack data.


How about "then try to get the same rack data" -> "followed by hosts on the same rack" ?

ConeyLiu · 2017-03-22T12:31:50Z

Hi, @jerryshao @mridulm Thanks for your review, I have updated the code.

ConeyLiu · 2017-03-22T12:33:10Z

 val getLocations = PrivateMethod[Seq[BlockManagerId]]('getLocations)
 val locations = blockManager invokePrivate getLocations(BroadcastBlockId(0))
- assert(locations.map(_.host).toSet === Set(localHost, localHost, otherHost))
+ assert(locations.map(_.host) === Seq(localHost, localHost, otherHost))


There should also be Seq.

mridulm

Looks good to me, will keep it open in case there are other comments.
+CC @jerryshao , @ConeyLiu

ConeyLiu · 2017-03-23T10:05:42Z

Ok, thanks a lot.

ConeyLiu · 2017-05-04T08:00:53Z

Hi, @cloud-fan @zsxwing Can you take a look? Thanks a lot.

cloud-fan · 2017-05-04T13:09:55Z

retest this please

cloud-fan · 2017-05-04T13:10:02Z

LGTM

SparkQA · 2017-05-04T15:22:03Z

Test build #76453 has finished for PR 17300 at commit 56f5231.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

ConeyLiu · 2017-05-05T00:45:54Z

retest this please

mridulm · 2017-05-05T01:04:47Z

Will merge when tests pass.

cloud-fan · 2017-05-05T02:38:12Z

retest this please

viirya · 2017-05-05T04:17:01Z

 val (preferredLocs, otherLocs) = locs.partition { loc => blockManagerId.host == loc.host }
- preferredLocs ++ otherLocs
+ val (sameRackLocs, differentRackLocs) = otherLocs.partition {
+ loc => blockManagerId.topologyInfo == loc.topologyInfo


If blockManagerId.topologyInfo is None, we will prefer the locations with empty topologyInfo. It is slightly different with what the shuffling wants to do here.

Modified, thanks a lot for the good advice.

ConeyLiu · 2017-05-05T05:54:47Z

Thanks both of you for review, I have addressed the comments and modified the test case. Please help calling jenkins for test, because I can't trigger that. Thanks again.

Also passed in local test.

SparkQA · 2017-05-05T06:52:36Z

Test build #76473 has finished for PR 17300 at commit 56f5231.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

mridulm · 2017-05-05T09:52:45Z

retest this please

gatorsmile · 2017-05-05T16:55:31Z

retest this please

mridulm · 2017-05-05T17:25:09Z

That is weird, jenkins should have restarted build when I commented ...

SparkQA · 2017-05-05T19:02:26Z

Test build #76500 has finished for PR 17300 at commit a3e979f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-05-05T19:11:32Z

retest this please

SparkQA · 2017-05-05T22:01:07Z

Test build #76502 has finished for PR 17300 at commit a3e979f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2017-05-06T08:57:25Z

LGTM

ConeyLiu · 2017-05-06T12:25:08Z

Thanks for your review.

cloud-fan · 2017-05-08T09:33:55Z

thanks, merging to master!

mridulm · 2017-05-08T09:52:31Z

Thanks for merging @cloud-fan, this PR kept dropping form my list ...

ConeyLiu · 2017-05-08T11:47:11Z

Thanks @cloud-fan @mridulm @gatorsmile

… information ## What changes were proposed in this pull request? When call the method getLocations of BlockManager, we only compare the data block host. Random selection for non-local data blocks, this may cause the selected data block to be in a different rack. So in this patch to increase the sort of the rack. ## How was this patch tested? New test case. Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Xianyang Liu <xianyang.liu@intel.com> Closesapache#17300 from ConeyLiu/blockmanager.

optimize a location order of blocks with topology information
926ea25

mridulm reviewed Mar 22, 2017
View reviewed changes

Address comments
56f5231

ConeyLiu commented Mar 22, 2017
View reviewed changes

mridulm approved these changes Mar 22, 2017
View reviewed changes

viirya reviewed May 5, 2017
View reviewed changes

ConeyLiu force-pushed the blockmanager branch from 039ba62 to 56f5231Compare May 5, 2017 04:53

ConeyLiu added 2 commits May 5, 2017 13:00

Merge branch 'master' into blockmanager
1d863b7

address comments, and fix the test error
a3e979f

asfgit closed this in 1552665May 8, 2017

ConeyLiu deleted the blockmanager branch May 8, 2017 11:47

Uh oh!

Conversation

ConeyLiu commented Mar 15, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

jerryshao commented Mar 22, 2017

Uh oh!

mridulmMar 22, 2017

Choose a reason for hiding this comment

Uh oh!

mridulmMar 22, 2017

Choose a reason for hiding this comment

Uh oh!

ConeyLiu commented Mar 22, 2017

Uh oh!

ConeyLiuMar 22, 2017

Choose a reason for hiding this comment

Uh oh!

mridulm left a comment

Choose a reason for hiding this comment

Uh oh!

ConeyLiu commented Mar 23, 2017

Uh oh!

ConeyLiu commented May 4, 2017

Uh oh!

cloud-fan commented May 4, 2017

Uh oh!

cloud-fan commented May 4, 2017

Uh oh!

SparkQA commented May 4, 2017

Uh oh!

ConeyLiu commented May 5, 2017

Uh oh!

mridulm commented May 5, 2017

Uh oh!

cloud-fan commented May 5, 2017

Uh oh!

viiryaMay 5, 2017

Choose a reason for hiding this comment

Uh oh!

ConeyLiuMay 5, 2017

Choose a reason for hiding this comment

Uh oh!

ConeyLiu commented May 5, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented May 5, 2017

Uh oh!

mridulm commented May 5, 2017

Uh oh!

gatorsmile commented May 5, 2017

Uh oh!

mridulm commented May 5, 2017

Uh oh!

SparkQA commented May 5, 2017

Uh oh!

gatorsmile commented May 5, 2017

Uh oh!

SparkQA commented May 5, 2017

Uh oh!

viirya commented May 6, 2017

Uh oh!

ConeyLiu commented May 6, 2017

Uh oh!

cloud-fan commented May 8, 2017

Uh oh!

mridulm commented May 8, 2017

Uh oh!

ConeyLiu commented May 8, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ConeyLiu commented May 5, 2017•
edited
Loading