[SPARK-53324][K8S] Introduce pending pod limit per ResourceProfile by ForVic · Pull Request #51913 · apache/spark

ForVic · 2025-08-07T22:13:39Z

What changes were proposed in this pull request?

Introducing a limit for pending PODs (newly created/requested executors included) per resource profile.
There exists a config for a global limit for all resource profiles, but here we add a limit per resource profile. #33492 does a lot of the plumbing for us already, counting newly created and pending pods, and we can just pass through the pending pods per resource profile, and limit the number of requests we were going to make for pods for that resource profile to min(previousRequest, maxPodsPerRP).

Why are the changes needed?

For multiple resource profile use cases you can set limits that apply at the resource profile level, instead of globally.

Does this PR introduce any user-facing change?

No

How was this patch tested?

unit tests added

Was this patch authored or co-authored using generative AI tooling?

No

ForVic · 2025-08-18T22:09:45Z

@attilapiros @dongjoon-hyun who created / reviewed #33492 respectively

mridulm · 2025-09-12T15:41:55Z

~~Why does this need to be specific to k8s ? Is it not applicable in case of YARN ?~~
Scratch that, yarn model makes it less susceptible to this issue.

mridulm

Looks reasonable to me.
Will let @attilapiros or @dongjoon-hyun review this though, as they are closer to it.

For context, we are dynamically adding resource profiles, and so a per RP limit will be helpful (so that a single RP does not end up taking too many pod allocation slots) - this PR will help towards usecases of that nature.

dongjoon-hyun

Thank you, @ForVic and @mridulm .

+1, LGTM, too.

Merged to master for Apache Spark 4.1.0-preview3.

…ndingPods` ### What changes were proposed in this pull request? This is a follow-up of the following to apply the latest `master` branch change. - #51913 ### Why are the changes needed? The master branch compilation is broken currently. - https://github.com/apache/spark/actions/runs/18760939731 Recently, we changed `numOutstandingPods`'s visibility, but I missed that the old open PRs didn't have this change. - #52614 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes#52714 from dongjoon-hyun/SPARK-53324. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? Introducing a limit for pending PODs (newly created/requested executors included) per resource profile. There exists a config for a global limit for all resource profiles, but here we add a limit per resource profile. apache#33492 does a lot of the plumbing for us already, counting newly created and pending pods, and we can just pass through the pending pods per resource profile, and limit the number of requests we were going to make for pods for that resource profile to min(previousRequest, maxPodsPerRP). ### Why are the changes needed? For multiple resource profile use cases you can set limits that apply at the resource profile level, instead of globally. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? unit tests added ### Was this patch authored or co-authored using generative AI tooling? No Closesapache#51913 from ForVic/vsunderl/max_pending_pods_per_rpid. Authored-by: ForVic <vsunderland@linkedin.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…ndingPods` ### What changes were proposed in this pull request? This is a follow-up of the following to apply the latest `master` branch change. - apache#51913 ### Why are the changes needed? The master branch compilation is broken currently. - https://github.com/apache/spark/actions/runs/18760939731 Recently, we changed `numOutstandingPods`'s visibility, but I missed that the old open PRs didn't have this change. - apache#52614 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closesapache#52714 from dongjoon-hyun/SPARK-53324. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…erPodIP|allocation.maxPendingPodsPerRp|driver.annotateExitException)` ### What changes were proposed in this pull request? This PR documents three missing Spark configurations introduced in Spark 4.1.0. | Property | Default | Since | Introduced in | | --- | --- | --- | --- | | `spark.kubernetes.executor.useDriverPodIP` | `false` | 4.1.0 | SPARK-53944 | | `spark.kubernetes.driver.annotateExitException` | `false` | 4.1.0 | SPARK-53335 | | `spark.kubernetes.allocation.maxPendingPodsPerRp` | `Int.MaxValue` | 4.1.0 | SPARK-53324 | ### Why are the changes needed? Three user-facing Kubernetes configurations added in Spark 4.1.0 but weren't documented yet. - #52650 - #52068 - #51913 ### Does this PR introduce _any_ user-facing change? No. Documentation-only update. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) Closes#55591 from dongjoon-hyun/SPARK-56653. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…erPodIP|allocation.maxPendingPodsPerRp|driver.annotateExitException)` ### What changes were proposed in this pull request? This PR documents three missing Spark configurations introduced in Spark 4.1.0. | Property | Default | Since | Introduced in | | --- | --- | --- | --- | | `spark.kubernetes.executor.useDriverPodIP` | `false` | 4.1.0 | SPARK-53944 | | `spark.kubernetes.driver.annotateExitException` | `false` | 4.1.0 | SPARK-53335 | | `spark.kubernetes.allocation.maxPendingPodsPerRp` | `Int.MaxValue` | 4.1.0 | SPARK-53324 | ### Why are the changes needed? Three user-facing Kubernetes configurations added in Spark 4.1.0 but weren't documented yet. - #52650 - #52068 - #51913 ### Does this PR introduce _any_ user-facing change? No. Documentation-only update. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) Closes#55591 from dongjoon-hyun/SPARK-56653. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 68456a6) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

ForVic added 5 commits August 7, 2025 14:14

initial impl need to verify
52549d5

fix logic
b28ddea

reformat
0fd9417

compile error
6dad084

add unit test
c4daa0d

github-actionsBot added the KUBERNETES label Aug 7, 2025

ForVic changed the title ~~initial impl need to verify~~[IN PROGRESS][K8S] Introduce pending pod limit per ResourceProfileAug 7, 2025

ForVic added 6 commits August 7, 2025 16:30

typo
800159f

Linting fixes
33054be

formatting
42ae4a9

breaking log
9fcbf25

rework the test
6fd9b90

remove whitespace
a5e513f

ForVic changed the title ~~[IN PROGRESS][K8S] Introduce pending pod limit per ResourceProfile~~[SPARK-53324][K8S] Introduce pending pod limit per ResourceProfileAug 18, 2025

ForVic marked this pull request as ready for review August 18, 2025 22:07

mridulm reviewed Sep 12, 2025
View reviewed changes

Comment thread...netes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
Comment thread...netes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala Outdated

ForVic added 2 commits September 15, 2025 10:33

Address comments, style + enforcement
60321ac

flag + comment
b4bc807

dongjoon-hyun approved these changes Oct 23, 2025
View reviewed changes

dongjoon-hyun closed this in 9b465a2Oct 23, 2025

dongjoon-hyun mentioned this pull request Oct 23, 2025
[SPARK-53324][K8S][FOLLOWUP] Use invokePrivate to access numOutstandingPods#52714
Closed

dongjoon-hyun mentioned this pull request Apr 29, 2026
[SPARK-56653][K8S][DOCS] Document spark.kubernetes.(executor.useDriverPodIP|allocation.maxPendingPodsPerRp|driver.annotateExitException)#55591
Closed

Navigation Menu

Uh oh!

[SPARK-53324][K8S] Introduce pending pod limit per ResourceProfile - #51913

[SPARK-53324][K8S] Introduce pending pod limit per ResourceProfile#51913
ForVic wants to merge 13 commits into
apache:masterfrom
ForVic:vsunderl/max_pending_pods_per_rpid

ForVic commented Aug 7, 2025•
edited
Loading

Uh oh!

ForVic commented Aug 18, 2025

Uh oh!

mridulm commented Sep 12, 2025•
edited
Loading

Uh oh!

mridulm left a comment •
edited
Loading

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ForVic commented Aug 7, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

ForVic commented Aug 18, 2025

Uh oh!

mridulm commented Sep 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mridulm left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ForVic commented Aug 7, 2025•
edited
Loading

mridulm commented Sep 12, 2025•
edited
Loading

mridulm left a comment •
edited
Loading