Uh oh!
There was an error while loading. Please reload this page.
[SPARK-34316][K8S] Support spark.kubernetes.executor.disableConfigMap - #31428
[SPARK-34316][K8S] Support spark.kubernetes.executor.disableConfigMap#31428dongjoon-hyun wants to merge 2 commits into
Conversation
Could you review this, @ScrapCodes ? There was a user request to disable the new feature of SPARK-30985 for some security reasons. |
SparkQA
commented
Feb 2, 2021
Test build #134750 has finished for PR 31428 at commit
|
SparkQA
commented
Feb 2, 2021
Kubernetes integration test starting |
SparkQA
commented
Feb 2, 2021
Kubernetes integration test status failure |
dongjoon-hyun
commented
Feb 2, 2021
Could you review this, @holdenk ? |
ScrapCodes
commented
Feb 2, 2021
Hi @dongjoon-hyun , Thank you for the PR. You mentioned due to a security issue, you would like add this config. Can you please describe the issue either here or if you think it makes sense then @Private list. I have taken a look a cursory look, no issues found. :) |
ScrapCodes
commented
Feb 2, 2021
Ok, +1 LGTM ! @dongjoon-hyun, Feel free to merge this. |
dongjoon-hyun
commented
Feb 2, 2021
Thank you, @ScrapCodes . Yes, for security, some namespaces do not allow the driver pod to create new ConfigMaps. As a result, Spark jobs are unable to be executed in those namespaces. This unblocks Spark jobs in those namespace. As I wrote in the PR description, this doesn't aim to hurt the functionality of the original contribution. Instead, this PR aims to provide a workaround for those security enforced namespace. |
dongjoon-hyun
commented
Feb 2, 2021
Merged to master. |
ScrapCodes
commented
Feb 2, 2021
Ahh, makes sense then. Wish, we could autodetect and fallback too. |
### What changes were proposed in this pull request? This PR aims to add a new configuration `spark.kubernetes.executor.disableConfigMap`. ### Why are the changes needed? This can be use to disable config map creating for executor pods due to apache#27735 . ### Does this PR introduce _any_ user-facing change? No. By default, this doesn't change AS-IS behavior. This is a new feature to add an ability to disable SPARK-30985. ### How was this patch tested? Pass the newly added UT. Closesapache#31428 from dongjoon-hyun/SPARK-34316. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
### What changes were proposed in this pull request? This PR aims to add a new configuration `spark.kubernetes.executor.disableConfigMap`. ### Why are the changes needed? This can be use to disable config map creating for executor pods due to apache#27735 . ### Does this PR introduce _any_ user-facing change? No. By default, this doesn't change AS-IS behavior. This is a new feature to add an ability to disable SPARK-30985. ### How was this patch tested? Pass the newly added UT. Closesapache#31428 from dongjoon-hyun/SPARK-34316. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit f66e38c) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
advancedxy
commented
May 18, 2023
Hi, @dongjoon-hyun and @ScrapCodes, When I was developing #41196, I initially thought it may be desirable to reuse the However after some digging and found some limitation exposed by K8S:
Therefore I think there should be some rules when developing feature steps for spark on K8S regarding config maps:
I'd like to make a proposal here:
WDYT? |
Yes, actually, that bit us as you see in this PR's goal.
Please send an email to dev@spark mailing list for further Q&A or discussion. We don't want to open a long discussion on the existing PR like this because that frequently causes unnecessary confusion for other audiences, @advancedxy . |
advancedxy
commented
May 18, 2023
Thanks. Let me open a discuss mail on the dev mail list. |
dongjoon-hyun
commented
May 18, 2023
Thank you! |
What changes were proposed in this pull request?
This PR aims to add a new configuration
spark.kubernetes.executor.disableConfigMap.Why are the changes needed?
This can be use to disable config map creating for executor pods due to #27735 .
Does this PR introduce any user-facing change?
No. By default, this doesn't change AS-IS behavior.
This is a new feature to add an ability to disable SPARK-30985.
How was this patch tested?
Pass the newly added UT.