[SPARK-14437][Core]Use the address that NettyBlockTransferService listens to create BlockManagerId

[zsxwing]

What changes were proposed in this pull request?

Here is why SPARK-14437 happens:
BlockManagerId is created using NettyBlockTransferService.hostName which comes from customHostname. And Executor will set customHostname to the hostname which is detected by the driver. However, the driver may not be able to detect the correct address in some complicated network (Netty's Channel.remoteAddress doesn't always return a connectable address). In such case, BlockManagerId will be created using a wrong hostname.

To fix this issue, this PR uses hostname provided by SparkEnv.create to create NettyBlockTransferService and set NettyBlockTransferService.hostname to this one directly. A bonus of this approach is NettyBlockTransferService won't bound to 0.0.0.0 which is much safer.

How was this patch tested?

Manually checked the bound address using local-cluster.

[zsxwing]

cc @vanzin

[SparkQA]

Test build #55236 has finished for PR 12240 at commit 9037529.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class NettyBlockTransferService(

[zsxwing]

@rxin is it a mistake that NettyBlockTransferService was public?

[zsxwing]

Never mind. Just found it's in network package

[SparkQA]

Test build #55237 has finished for PR 12240 at commit 8c2d259.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #55239 has finished for PR 12240 at commit 4a836dc.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #55242 has finished for PR 12240 at commit 8a08eea.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #55250 has finished for PR 12240 at commit 6e08543.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #55317 has finished for PR 12240 at commit db9d497.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class DeserializeToObject(
  • case class SerializeFromObject(
  • case class DeserializeToObject(
  • case class SerializeFromObject(

[zsxwing]

retest this please

[zsxwing]

retest this please

[SparkQA]

Test build #55364 has finished for PR 12240 at commit db9d497.

• This patch fails some tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class DeserializeToObject(
  • case class SerializeFromObject(
  • case class DeserializeToObject(
  • case class SerializeFromObject(

[zsxwing]

retest this please

[SparkQA]

Test build #55376 has finished for PR 12240 at commit db9d497.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class DeserializeToObject(
  • case class SerializeFromObject(
  • case class DeserializeToObject(
  • case class SerializeFromObject(

[zsxwing]

retest this please

[SparkQA]

Test build #55379 has finished for PR 12240 at commit db9d497.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class DeserializeToObject(
  • case class SerializeFromObject(
  • case class DeserializeToObject(
  • case class SerializeFromObject(

[vanzin]

LGTM.

[zsxwing]

Thanks, Merging to master.