Skip to content

[SPARK-37145][K8S] Add KubernetesCustom[Driver/Executor]FeatureConfigStep developer api - #35345

Closed
Yikun wants to merge 8 commits into
apache:masterfrom
Yikun:SPARK-37145-alt
Closed

[SPARK-37145][K8S] Add KubernetesCustom[Driver/Executor]FeatureConfigStep developer api#35345
Yikun wants to merge 8 commits into
apache:masterfrom
Yikun:SPARK-37145-alt

Conversation

@Yikun

@YikunYikun commented Jan 28, 2022•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This patch adds the support for extending user feature steps with configuration by adding 2 developer api:

  • KubernetesDriverCustomFeatureConfigStep: to help user extend custom feature step in executor side
  • KubernetesExecutorCustomFeatureConfigStep: to help user extend custom feature step in driver side

Before this patch user can only add feature step like:

  • class TestStep extends KubernetesFeatureConfigStep: without any kubernetes conf

After this patch user can add feature step with configuration like:

  • class TestStepWithDriverConf extends KubernetesDriverCustomFeatureConfigStep: only driver
  • class TestStepWithExecConf extends KubernetesExecutorCustomFeatureConfigStep: only executor
  • class TestStepWithK8SConf extends KubernetesDriverCustomFeatureConfigStep with KubernetesExecutorCustomFeatureConfigStep: both driver and executor

Why are the changes needed?

In #30206 , a developer API for custom feature steps has been added, but it didn't support initialize user feature step with kubernetes conf (like KubernetesConf/KubernetesDriverConf/KubernetesExecutorConf).

In most of scenarios, users want to make corresponding changes in their feature steps according to the configuration. Such as, the customized scheduler scenario, user wants to configure pod according to passed job configuration.

Does this PR introduce any user-facing change?

Improve the developer API for for custom feature steps.

How was this patch tested?

  • Added UT
  • Runing k8s integration test manaully: build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test

Closes: #34924

@YikunYikun mentioned this pull request Jan 28, 2022
Closed
@YikunYikun changed the title [SPARK-37145][K8S] (Alternative) Add support for extending user feature steps with configuration[SPARK-37145][K8S] Add KubernetesCustomFeatureConfigStep DeveloperApiJan 28, 2022
@YikunYikun changed the title [SPARK-37145][K8S] Add KubernetesCustomFeatureConfigStep DeveloperApi[SPARK-37145][K8S] Add KubernetesCustomFeatureConfigStep developer apiJan 28, 2022
@Yikun
Yikun marked this pull request as ready for review January 30, 2022 09:22
attilapiros
attilapiros reviewed Jan 31, 2022
View reviewed changes
Comment thread...s/core/src/main/scala/org/apache/spark/deploy/k8s/features/KubernetesFeatureConfigStep.scala Outdated
Comment thread...s/core/src/main/scala/org/apache/spark/deploy/k8s/features/KubernetesFeatureConfigStep.scala Outdated
Comment threadresource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Outdated
Comment threadresource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Outdated
Comment thread...s/core/src/main/scala/org/apache/spark/deploy/k8s/features/KubernetesFeatureConfigStep.scala Outdated
Comment thread...s/core/src/main/scala/org/apache/spark/deploy/k8s/features/KubernetesFeatureConfigStep.scala Outdated
Comment thread...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala Outdated
@Yikun

Yikun commented Feb 1, 2022•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

@attilapiros Thanks for your review, I will address soon.

(later)

updated.

@Yikun
Yikunforce-pushed the SPARK-37145-alt branch 4 times, most recently from 551a9a2 to d05b3d2CompareFebruary 1, 2022 09:46
@YikunYikun changed the title [SPARK-37145][K8S] Add KubernetesCustomFeatureConfigStep developer api[SPARK-37145][K8S] Add KubernetesCustom[Driver|Executor]FeatureConfigStep developer apiFeb 1, 2022
@YikunYikun changed the title [SPARK-37145][K8S] Add KubernetesCustom[Driver|Executor]FeatureConfigStep developer api[SPARK-37145][K8S] Add KubernetesCustom[Driver/Executor]FeatureConfigStep developer apiFeb 1, 2022
attilapiros
attilapiros reviewed Feb 1, 2022
View reviewed changes

@attilapirosattilapiros left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bunch of Nits and a test refactor.

Comment threaddocs/running-on-kubernetes.md Outdated
Comment threaddocs/running-on-kubernetes.md Outdated
Comment threadresource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Outdated
Comment threadresource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Outdated
Comment thread...s/core/src/main/scala/org/apache/spark/deploy/k8s/features/KubernetesFeatureConfigStep.scala Outdated
Comment thread...s/core/src/main/scala/org/apache/spark/deploy/k8s/features/KubernetesFeatureConfigStep.scala Outdated
Comment thread...ce-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/PodBuilderSuite.scala Outdated
Comment thread...ce-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/PodBuilderSuite.scala Outdated
@Yikun

Yikun commented Feb 2, 2022

Copy link
Copy Markdown
MemberAuthor

@attilapiros Thanks for your suggestion, it makes doc and test more clear and simple. Addressed.

dongjoon-hyun
dongjoon-hyun requested changes Feb 2, 2022
View reviewed changes

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for making this improvement, @Yikun .
I made a few comments.

@Yikun

Yikun commented Feb 2, 2022

Copy link
Copy Markdown
MemberAuthor

@dongjoon-hyun Thanks for your review, addressed. : )

dongjoon-hyun
dongjoon-hyun approved these changes Feb 2, 2022
View reviewed changes

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @Yikun and all.
Merged to master for Apache Spark 3.3.

@Yikun

Yikun commented Feb 3, 2022

Copy link
Copy Markdown
MemberAuthor

@dongjoon-hyun@attilapiros Thanks for your help!

@dongjoon-hyun

dongjoon-hyun commented Feb 3, 2022

Copy link
Copy Markdown
Member

Did I miss something from your comments, @attilapiros ?

@attilapiros

attilapiros commented Feb 3, 2022

Copy link
Copy Markdown
Contributor

@dongjoon-hyun It is fine, I just planned to do another review round (the last one) and I got surprised it is already in.

@dongjoon-hyun

dongjoon-hyun commented Feb 3, 2022

Copy link
Copy Markdown
Member

You can make comments still and we can handle it as follow-up PRs. Please let us know if you have any concern, @attilapiros .

@YikunYikun mentioned this pull request Feb 13, 2022
Closed
dongjoon-hyun pushed a commit that referenced this pull request Feb 16, 2022
@Yikun@dongjoon-hyun
[SPARK-37145][K8S][FOLLOWUP] Add note for `KubernetesCustom[Driver/Ex…
bb757b5
…ecutor]FeatureConfigStep`
### What changes were proposed in this pull request?
Add note for developers to show how to use `KubernetesDriverCustomFeatureConfigStep` and `KubernetesExecutorCustomFeatureConfigStep` (#35345).
### Why are the changes needed?
Give an example to show how to use it.
### Does this PR introduce _any_ user-facing change?
No, doc only
### How was this patch tested?
ci passed
Closes#35496 from Yikun/SPARK-37145-followup.
Authored-by: Yikun Jiang <yikunkero@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
@dongjoon-hyundongjoon-hyun mentioned this pull request Dec 5, 2022
Closed
@dongjoon-hyundongjoon-hyun mentioned this pull request Feb 24, 2026
Closed
dongjoon-hyun added a commit that referenced this pull request Feb 24, 2026
@dongjoon-hyun
[SPARK-55649][K8S] Promote `Kubernetes(Driver|Executor)?FeatureConfig…
2f37724
…Step` traits to `Stable`
### What changes were proposed in this pull request?
This PR aims to promote the following traits to `Stable` from Apache Spark 4.2.0:
- `KubernetesFeatureConfigStep`
- `KubernetesDriverCustomFeatureConfigStep`
- `KubernetesExecutorCustomFeatureConfigStep`
### Why are the changes needed?
Since Apache Spark 3.3.0, the `Kubernetes*FeatureConfigStep` traits have been serving stably without any modifications for 4 years. We had better promote them to `Stable`.
- #35345
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: `Gemini 3.1 Pro (High)` on `Antigravity`
Closes#54439 from dongjoon-hyun/SPARK-55649.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@attilapirosattilapirosattilapiros left review comments
@dongjoon-hyundongjoon-hyundongjoon-hyun approved these changes

Assignees

No one assigned

Labels

DOCSKUBERNETES

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Yikun@dongjoon-hyun@attilapiros