[SPARK-6521][Core]executors in the same node read local shuffle file

[viper-kun]

In the past, executor read other executor's shuffle file in the same node by net. This pr make that executors in the same node read local shuffle file In sort-based Shuffle. It will reduce net transport.

[andrewor14]

ok to test

[andrewor14]

hey @viper-kun the style here and other places should be:

def getDataFile(
shuffleId: Int,
mapId: Int,
blockManagerId: BlockManagerId = ...): File = {
...
}

[SparkQA]

Test build #29167 has finished for PR 5178 at commit 5b766ca.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class GetLocalDirsPath(blockManagerId: BlockManagerId) extends ToBlockManagerMaster

[andrewor14]

style

blockManagerInfo
.filter { case (id, _) => (id != blockManagerId && id.host == blockManagerId.host) }
.mapValues { info => info.localDirsPath }
.toMap

[andrewor14]

Hi @viper-kun thanks for working on this. However, it seems that there are quite a few style violations. For more detail please see https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide or https://github.com/databricks/scala-style-guide. Once you fix those I will do a closer review.

[SparkQA]

Test build #29217 timed out for PR 5178 at commit 0eaf1af after a configured wait of 120m.

[SparkQA]

Test build #29218 timed out for PR 5178 at commit 6162aef after a configured wait of 120m.

[SparkQA]

Test build #29274 timed out for PR 5178 at commit bb94736 after a configured wait of 120m.

[viper-kun]

Hi @andrewor14. pls retest it, test build time out.

[WangTaoTheTonic]

Jenkins, test this please.

[SparkQA]

Test build #29296 timed out for PR 5178 at commit bb94736 after a configured wait of 120m.

[scwf]

Jenkins, test this please.

[SparkQA]

Test build #29346 has finished for PR 5178 at commit 9ddf84c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class GetLocalDirsPath(blockManagerId: BlockManagerId) extends ToBlockManagerMaster

[SparkQA]

Test build #29361 has finished for PR 5178 at commit 383a21f.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class GetLocalDirsPath(blockManagerId: BlockManagerId) extends ToBlockManagerMaster

[SparkQA]

Test build #29363 has finished for PR 5178 at commit 6ee8df2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class GetLocalDirsPath(blockManagerId: BlockManagerId) extends ToBlockManagerMaster

[SparkQA]

Test build #29385 has started for PR 5178 at commit 7ae6e46.

[SparkQA]

Test build #29381 has finished for PR 5178 at commit c429b66.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class GetLocalDirsPath(blockManagerId: BlockManagerId) extends ToBlockManagerMaster
• This patch does not change any dependencies.

[scwf]

Jenkins, test this please.

[SparkQA]

Test build #29391 has started for PR 5178 at commit 6c5c1d4.

[viper-kun]

@andrewor14 Pls review it. Thanks

[maropu]

Nit: Unnecessary parentheses.

[maropu]

One question; are there many cases for executors to share a single host in the Yarn mode?

[scwf]

@maropu , yeah i think it is a common case for yarn mode. We often specify more executors than nodemanager, that means there are more than one executor on one machine.

[maropu]

Understood.

[maropu]

We should check if SortShuffleManager is used because this patch only supports it. This logic fails when new shuffle managers will be implemented.

[maropu]

@viper-kun What's the status of this patch? If you don't make further updates, I'd like to brush up this patch.

[viper-kun]

@maropu I will update it.

[andrewor14]

@viper-kun@maropu any updates? Should we take over? I would recommend that we close this patch since it's mostly gone stale. We can always reopen an updated version later.

[AmplabJenkins]

Can one of the admins verify this patch?

[andrewor14]

Let's close this PR and reopen it later if necessary.