Uh oh!
There was an error while loading. Please reload this page.
[SPARK-29670][CORE][2.4] Make executor bind address configurable for Spark 2.4 - #32633
[SPARK-29670][CORE][2.4] Make executor bind address configurable for Spark 2.4#32633gedeh wants to merge 3 commits into
Conversation
AmplabJenkins
commented
May 22, 2021
Can one of the admins verify this patch? |
srowen
left a comment
There was a problem hiding this comment.
This could be OK, but I'm not sure there will be any more 2.4.x releases after 2.4.8 anyway
Uh oh!
There was an error while loading. Please reload this page.
gedeh
commented
May 22, 2021
Ow, I didn't aware of this. That's explains |
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Thank you for making a PR, @gedeh .
However, this PR is invalid in a few ways.
- SPARK-29670 is closed as
Duplicated. You cannot ask a backport of unmerged JIRA. - The actual merged JIRA for feature is SPARK-24203 which was an
Improvementat Apache Spark 3.0.0. Apache Spark community have a general policy which allows bug-fix backporting only. So, we cannot backport a new feature because we need to protect old branches. - Lastly, 2.4.8 was the EOL release already. Here is some explanation for that.
The last minor release within a major a release will typically be maintained for longer as an “LTS” release. For example, 2.4.0 was released in November 2nd 2018 and had been maintained for 31 months until 2.4.8 was released on May 2021. 2.4.8 is the last release and no more 2.4.x releases should be expected even for bug fixes.
Given that the above reasons, could you close this PR, @gedeh ?
gedeh
commented
May 23, 2021
Thank you @dongjoon-hyun for clarification. I wasn't aware of EOL of Spark 2.4 closing this PR. Thanks! |
dongjoon-hyun
commented
May 23, 2021
Thank you so much, @gedeh ! |
What changes were proposed in this pull request?
Backporting changes from #26331 to Spark 2.4, in a similar fashion, executor's
bindAddressis passed as an input parameter forRPCEnv.create.This PR also ensure
YarnAllocatoruse the config when allocating new executor and use default hostname when its not configured.Why are the changes needed?
Recently we've came across this issue with Spark running on Yarn in Istio enabled Kubernetes cluster. As I understand it, Spark 2.4 is a stepping stone for teams moving away from Scala 2.11 to Scala 2.12 and then moving to Spark 3.
Does this PR introduce any user-facing change?
Yes, new config is added and relevant doc is updated.
How was this patch tested?
I will run this on Kubenetes and Istio.