[SPARK-14849][CORE]Always set an address for the executor

[skyluc]

As specified in SPARK-14849, the address in the NettyRpcEnv for the executor is not set when running in standalone mode. To compensate, the driver try to guess the IP address of the executor during registration. But in a NAT situation, the IP address visible in the connection is not the one that should be used.
This address is sent back to the executor, and used later to describe the location of blocks.

The change is too always set the address as the host value, and to remove the code on the driver side which is trying to guess the right IP address. This kind of guessing is wrong in a NAT configuration, a possibly others.

Manually tested on the configuration describe on the ticket, and on a standard Mesos cluster.
Testing on a Yarn cluster would be useful, to check that there is no unexpected effects.

[SparkQA]

Test build #56705 has finished for PR 12613 at commit 1cf8322.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

cc @zsxwing

[skyluc]

Are the test failures real, or due to flaky tests?
I tried to reproduced the failures locally, but core/test passes most of the time.

[rxin]

this is no longer nullable

[rxin]

I think there are also some code in SparkEnv that deals with this?

 if (isDriver) {
conf.set("spark.driver.port", rpcEnv.address.port.toString)
} else if (rpcEnv.address != null) {
conf.set("spark.executor.port", rpcEnv.address.port.toString)
}

[skyluc]

Added comments, updated checks for cases when running in client mode, removed sending back the hostname to the executor.

[SparkQA]

Test build #57125 has finished for PR 12613 at commit 0624898.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #57133 has finished for PR 12613 at commit baaef11.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

-1 is actually confusing. All of executors in the same node will have the same address.

[zsxwing]

The host is easy to fix. However, the hard part is port. The executor cannot know the port until it sends a message. In addition, since the executor runs in the client mode, it may have multiple clients/ports.

[zsxwing]

@skyluc could you close this one, please? You can submit a new PR when you have a better idea. Thanks!