[SPARK-52933][K8S] Verify if the executor cpu request exceeds limit by dongjoon-hyun · Pull Request #51678 · apache/spark

dongjoon-hyun · 2025-07-27T21:46:56Z

What changes were proposed in this pull request?

This PR aims to verify if the executor pod's cpu request exceeds cpu limit or not in order to make it fail-fast.

Why are the changes needed?

Since Spark creates many executor pods, we had better do fail-fast on the invalid settings before submitting invalid pod spec to K8s cluster. It wastes lots of K8s resources.

Note that newly added validation check only happens when spark.kubernetes.executor.limit.cores is given explicitly.

Does this PR introduce any user-facing change?

No behavior change eventually because the existing misconfigured spark.kubernetes.executor.limit.cores means Spark driver cannot get any executor pods and the job will hang or fail eventually.

How was this patch tested?

Pass the CIs with the newly added test case.

Was this patch authored or co-authored using generative AI tooling?

No.

dongjoon-hyun · 2025-07-27T23:05:35Z

Could you review this when you have some time, @peter-toth ?

dongjoon-hyun · 2025-07-28T00:53:59Z

Could you review this PR when you have some time, @HyukjinKwon ?

LuciferYang · 2025-07-28T01:02:52Z

 val executorCpuLimitQuantity = new Quantity(limitCores)
+ if (executorCpuLimitQuantity.compareTo(executorCpuQuantity) < 0) {
+ throw new SparkException(
+ "The executor cpu request should be less than or equal to cpu limit")


nit: Should the request value and limit value be included in the error message?

dongjoon-hyun · 2025-07-28T03:08:25Z

Thank you, @LuciferYang and @HyukjinKwon .

LuciferYang

+1, LGTM
Thank you @dongjoon-hyun

dongjoon-hyun · 2025-07-28T03:16:19Z

All comments are addressed and I verified the test result manually because the last commit changes only the exception message string.

[info] BasicExecutorFeatureStepSuite:
[info] - test spark resource missing vendor (6 milliseconds)
[info] - test spark resource missing amount (1 millisecond)
[info] - SPARK-52933: Verify if the executor cpu request exceeds limit (5 milliseconds)

Merged to master for Apache Spark 4.1.0.

peter-toth

Late LGTM.

zemin-piao · 2025-08-25T23:33:29Z

Thanks a lot for pushing this PR! @dongjoon-hyun

### What changes were proposed in this pull request? This PR aims to upgrade Spark to `4.1.0-preview2` for `4.0.1`. ### Why are the changes needed? Since Apache Spark 4.1.0 is planned next month, we had better prepare to use new features via using `4.1.0-preview2` (September) and `4.1.0-preview2 (October)` gradually. - apache/spark#51678 - apache/spark#51522 - apache/spark#50925 ### Does this PR introduce _any_ user-facing change? No behavior change. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes#364 from dongjoon-hyun/SPARK-53787. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR aims to verify if the executor pod's cpu request exceeds cpu limit or not in order to make it fail-fast. ### Why are the changes needed? Since Spark creates many executor pods, we had better do fail-fast on the invalid settings before submitting invalid pod spec to K8s cluster. It wastes lots of K8s resources. Note that newly added validation check only happens when `spark.kubernetes.executor.limit.cores` is given explicitly. ### Does this PR introduce _any_ user-facing change? No behavior change eventually because the existing misconfigured `spark.kubernetes.executor.limit.cores` means Spark driver cannot get any executor pods and the job will hang or fail eventually. ### How was this patch tested? Pass the CIs with the newly added test case. ### Was this patch authored or co-authored using generative AI tooling? No. Closesapache#51678 from dongjoon-hyun/SPARK-52933. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 4dc3f0f)

…tException` for executor cpu misconfigs ### What changes were proposed in this pull request? This PR aims to fix `BasicExecutorFeatureStep` to throw `IllegalArgumentException` for executor cpu misconfigs in order the Spark jobs ASAP. ### Why are the changes needed? From Apache Spark 4.1.0, Spark driver pod throws `SparkException` for the executor cpu misconfiguration before requesting to the K8s control plain. This improvement reduces the burden of K8s control plane. - #51678 ``` 26/01/24 06:55:31 INFO ExecutorPodsAllocator: Going to request 5 executors from Kubernetes for ResourceProfile Id: 0, target: 5, known: 0, sharedSlotFromPendingPods: 2147483647. 26/01/24 06:55:31 INFO ExecutorPodsAllocator: Found 0 reusable PVCs from 0 PVCs 26/01/24 06:55:31 WARN ExecutorPodsSnapshotsStoreImpl: Exception when notifying snapshot subscriber. org.apache.spark.SparkException: The executor cpu request (4) should be less than or equal to cpu limit (1) at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.$anonfun$configurePod$11(BasicExecutorFeatureStep.scala:236) ``` However, the Spark driver keeps re-trying to create executor pods in any way if the users didn't have an additional `spark.driver.timeout` configuration. - #45313 So, we had better exit the Spark job in this case ASAP. We can do that simply switches `SparkException` to `IllegalArgumentException` like the other steps. - #30084 ### Does this PR introduce _any_ user-facing change? Technically no because previously those misconfigured Spark job didn't get any resources. ### How was this patch tested? Pass the CIs with the updated test case. Also, I checked manually via `spark-submit`: ``` $ bin/spark-submit --master k8s://$K8S_MASTER \ --deploy-mode cluster \ -c spark.executor.instances=5 \ -c spark.kubernetes.executor.request.cores=4 \ -c spark.kubernetes.executor.limit.cores=1 \ -c spark.kubernetes.container.image=apache/spark:SPARK-55134 \ -c spark.kubernetes.authenticate.driver.serviceAccountName=spark \ -c spark.kubernetes.executor.useDriverPodIP=true \ --class org.apache.spark.examples.SparkPi \ local:///opt/spark/examples/jars/spark-examples.jar 200000 ... 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: State changed, new state: pod name: org-apache-spark-examples-sparkpi-0482f19beeec7491-driver namespace: default labels: spark-app-name -> org-apache-spark-examples-sparkpi, spark-app-selector -> spark-ee23f03db88b43fb906b0dbc1b04ad63, spark-role -> driver, spark-version -> 4.2.0-SNAPSHOT pod uid: c6d41845-5893-4135-a065-278d94500315 creation time: 2026-01-24T07:33:52Z service account name: spark volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-8rbc8 node name: lima-rancher-desktop start time: 2026-01-24T07:33:52Z phase: Failed container status: container name: spark-kubernetes-driver container image: apache/spark:SPARK-55134 container state: terminated container started at: 2026-01-24T07:33:53Z container finished at: 2026-01-24T07:33:55Z exit code: 1 termination reason: Error 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Application status for spark-ee23f03db88b43fb906b0dbc1b04ad63 (phase: Failed) 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Container final statuses: container name: spark-kubernetes-driver container image: apache/spark:SPARK-55134 container state: terminated container started at: 2026-01-24T07:33:53Z container finished at: 2026-01-24T07:33:55Z exit code: 1 termination reason: Error 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Application org.apache.spark.examples.SparkPi with application ID spark-ee23f03db88b43fb906b0dbc1b04ad63 and submission ID default:org-apache-spark-examples-sparkpi-0482f19beeec7491-driver finished ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes#53948 from dongjoon-hyun/SPARK-55134. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…tException` for executor cpu misconfigs ### What changes were proposed in this pull request? This PR aims to fix `BasicExecutorFeatureStep` to throw `IllegalArgumentException` for executor cpu misconfigs in order the Spark jobs ASAP. ### Why are the changes needed? From Apache Spark 4.1.0, Spark driver pod throws `SparkException` for the executor cpu misconfiguration before requesting to the K8s control plain. This improvement reduces the burden of K8s control plane. - #51678 ``` 26/01/24 06:55:31 INFO ExecutorPodsAllocator: Going to request 5 executors from Kubernetes for ResourceProfile Id: 0, target: 5, known: 0, sharedSlotFromPendingPods: 2147483647. 26/01/24 06:55:31 INFO ExecutorPodsAllocator: Found 0 reusable PVCs from 0 PVCs 26/01/24 06:55:31 WARN ExecutorPodsSnapshotsStoreImpl: Exception when notifying snapshot subscriber. org.apache.spark.SparkException: The executor cpu request (4) should be less than or equal to cpu limit (1) at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.$anonfun$configurePod$11(BasicExecutorFeatureStep.scala:236) ``` However, the Spark driver keeps re-trying to create executor pods in any way if the users didn't have an additional `spark.driver.timeout` configuration. - #45313 So, we had better exit the Spark job in this case ASAP. We can do that simply switches `SparkException` to `IllegalArgumentException` like the other steps. - #30084 ### Does this PR introduce _any_ user-facing change? Technically no because previously those misconfigured Spark job didn't get any resources. ### How was this patch tested? Pass the CIs with the updated test case. Also, I checked manually via `spark-submit`: ``` $ bin/spark-submit --master k8s://$K8S_MASTER \ --deploy-mode cluster \ -c spark.executor.instances=5 \ -c spark.kubernetes.executor.request.cores=4 \ -c spark.kubernetes.executor.limit.cores=1 \ -c spark.kubernetes.container.image=apache/spark:SPARK-55134 \ -c spark.kubernetes.authenticate.driver.serviceAccountName=spark \ -c spark.kubernetes.executor.useDriverPodIP=true \ --class org.apache.spark.examples.SparkPi \ local:///opt/spark/examples/jars/spark-examples.jar 200000 ... 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: State changed, new state: pod name: org-apache-spark-examples-sparkpi-0482f19beeec7491-driver namespace: default labels: spark-app-name -> org-apache-spark-examples-sparkpi, spark-app-selector -> spark-ee23f03db88b43fb906b0dbc1b04ad63, spark-role -> driver, spark-version -> 4.2.0-SNAPSHOT pod uid: c6d41845-5893-4135-a065-278d94500315 creation time: 2026-01-24T07:33:52Z service account name: spark volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-8rbc8 node name: lima-rancher-desktop start time: 2026-01-24T07:33:52Z phase: Failed container status: container name: spark-kubernetes-driver container image: apache/spark:SPARK-55134 container state: terminated container started at: 2026-01-24T07:33:53Z container finished at: 2026-01-24T07:33:55Z exit code: 1 termination reason: Error 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Application status for spark-ee23f03db88b43fb906b0dbc1b04ad63 (phase: Failed) 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Container final statuses: container name: spark-kubernetes-driver container image: apache/spark:SPARK-55134 container state: terminated container started at: 2026-01-24T07:33:53Z container finished at: 2026-01-24T07:33:55Z exit code: 1 termination reason: Error 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Application org.apache.spark.examples.SparkPi with application ID spark-ee23f03db88b43fb906b0dbc1b04ad63 and submission ID default:org-apache-spark-examples-sparkpi-0482f19beeec7491-driver finished ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes#53948 from dongjoon-hyun/SPARK-55134. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit ab3ec9e) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

[SPARK-52933][K8S] Verify if the executor cpu request exceeds limit
df03689

github-actionsBot added the KUBERNETES label Jul 27, 2025

indentation
202dce7

LuciferYang approved these changes Jul 28, 2025
View reviewed changes

LuciferYang reviewed Jul 28, 2025
View reviewed changes

HyukjinKwon approved these changes Jul 28, 2025
View reviewed changes

Address comments
34c1d39

LuciferYang approved these changes Jul 28, 2025
View reviewed changes

dongjoon-hyun closed this in 4dc3f0fJul 28, 2025

dongjoon-hyun deleted the SPARK-52933 branch July 28, 2025 03:16

peter-toth reviewed Jul 28, 2025
View reviewed changes

dongjoon-hyun mentioned this pull request Oct 2, 2025
[SPARK-53787] Upgrade Spark to 4.1.0-preview2apache/spark-kubernetes-operator#364
Closed

dongjoon-hyun mentioned this pull request Jan 24, 2026
[SPARK-55134] Fix BasicExecutorFeatureStep to throw IllegalArgumentException for executor cpu misconfigs #53948
Closed

Navigation Menu

Uh oh!

[SPARK-52933][K8S] Verify if the executor cpu request exceeds limit - #51678

[SPARK-52933][K8S] Verify if the executor cpu request exceeds limit#51678
dongjoon-hyun wants to merge 3 commits into
apache:masterfrom
dongjoon-hyun:SPARK-52933

dongjoon-hyun commented Jul 27, 2025•
edited
Loading

Uh oh!

dongjoon-hyun commented Jul 27, 2025

Uh oh!

dongjoon-hyun commented Jul 28, 2025

Uh oh!

LuciferYangJul 28, 2025

Uh oh!

dongjoon-hyun commented Jul 28, 2025

Uh oh!

LuciferYang left a comment •
edited
Loading

Uh oh!

dongjoon-hyun commented Jul 28, 2025

Uh oh!

peter-toth left a comment

Uh oh!

zemin-piao commented Aug 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

dongjoon-hyun commented Jul 27, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun commented Jul 27, 2025

Uh oh!

dongjoon-hyun commented Jul 28, 2025

Uh oh!

LuciferYangJul 28, 2025

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jul 28, 2025

Uh oh!

LuciferYang left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jul 28, 2025

Uh oh!

peter-toth left a comment

Choose a reason for hiding this comment

Uh oh!

zemin-piao commented Aug 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dongjoon-hyun commented Jul 27, 2025•
edited
Loading

LuciferYang left a comment •
edited
Loading