Uh oh!
There was an error while loading. Please reload this page.
[SPARK-10997] [core] Add "client mode" to netty rpc env. - #9210
Closed
vanzin wants to merge 5 commits into
Closed
Conversation
"Client mode" means the RPC env will not listen for incoming connections. This allows certain processes in the Spark stack (such as Executors or tha YARN client-mode AM) to act as pure clients when using the netty-based RPC backend, reducing the number of sockets needed by the app and also the number of open ports. Client connections are also preferred when endpoints that actually have a listening socket are involved; so, for example, if a Worker connects to a Master and the Master needs to send a message to a Worker endpoint, that client connection will be used, even though the Worker is also listening for incoming connections. With this change, the workaround for SPARK-10987 isn't necessary anymore, and is removed. The AM connects to the driver in "client mode", and that connection is used for all driver <-> AM communication, and so the AM is properly notified when the connection goes down. This change also removes the workaround for SPARK-10987.
ContributorAuthor
There was a problem hiding this comment.
note: this is unrelated but sbt started complaining about this dependency while testing this change.
vanzin
commented
Oct 22, 2015
ContributorAuthor
SparkQA
commented
Oct 22, 2015
Test build #44110 has finished for PR 9210 at commit
|
vanzin
commented
Oct 22, 2015
ContributorAuthor
retest this please |
SparkQA
commented
Oct 22, 2015
Test build #44127 has finished for PR 9210 at commit
|
Member
There was a problem hiding this comment.
nit: _address and _name can be val.
zsxwing
commented
Oct 22, 2015
Member
Just a nit. Otherwise LGTM |
SparkQA
commented
Oct 23, 2015
Test build #44185 has finished for PR 9210 at commit
|
vanzin
commented
Oct 23, 2015
ContributorAuthor
retest this please |
SparkQA
commented
Oct 23, 2015
Test build #44187 has finished for PR 9210 at commit
|
Contributor
There was a problem hiding this comment.
can you add some comment on what this is checking and when this can happen
added 2 commits
October 22, 2015 21:44
Conflicts: core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
SparkQA
commented
Oct 23, 2015
Test build #44200 has finished for PR 9210 at commit
|
SparkQA
commented
Oct 23, 2015
Test build #44199 has finished for PR 9210 at commit
|
SparkQA
commented
Oct 24, 2015
Test build #44298 has finished for PR 9210 at commit
|
vanzin
commented
Oct 28, 2015
ContributorAuthor
hi there, any remaining feedback here? |
vanzin
commented
Nov 1, 2015
ContributorAuthor
retest this please |
SparkQA
commented
Nov 2, 2015
Test build #44778 has finished for PR 9210 at commit
|
vanzin
commented
Nov 2, 2015
ContributorAuthor
hmm, akka test... it's failed for me in unrelated changes, also. retest this please. |
SparkQA
commented
Nov 2, 2015
Test build #44787 has finished for PR 9210 at commit
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
"Client mode" means the RPC env will not listen for incoming connections.
This allows certain processes in the Spark stack (such as Executors or
tha YARN client-mode AM) to act as pure clients when using the netty-based
RPC backend, reducing the number of sockets needed by the app and also the
number of open ports.
Client connections are also preferred when endpoints that actually have
a listening socket are involved; so, for example, if a Worker connects
to a Master and the Master needs to send a message to a Worker endpoint,
that client connection will be used, even though the Worker is also
listening for incoming connections.
With this change, the workaround for SPARK-10987 isn't necessary anymore, and
is removed. The AM connects to the driver in "client mode", and that connection
is used for all driver <-> AM communication, and so the AM is properly notified
when the connection goes down.