Uh oh!
There was an error while loading. Please reload this page.
[SPARK-47207][CORE] Support spark.driver.timeout and DriverTimeoutPlugin - #45313
Closed
dongjoon-hyun wants to merge 2 commits into
Closed
[SPARK-47207][CORE] Support spark.driver.timeout and DriverTimeoutPlugin#45313dongjoon-hyun wants to merge 2 commits into
spark.driver.timeout and DriverTimeoutPlugin#45313dongjoon-hyun wants to merge 2 commits into
Conversation
dongjoon-hyunforce-pushed
the
SPARK-47207
branch
from
February 28, 2024 09:13
2bec717 to
b61a68aComparedongjoon-hyunforce-pushed
the
SPARK-47207
branch
from
February 28, 2024 14:48
e5ea037 to
95ac5adComparedongjoon-hyun
commented
Feb 28, 2024
MemberAuthor
Could you review this PR too, @LuciferYang ? |
| .booleanConf | ||
| .createWithDefault(false) | ||
| private[spark] val DRIVER_TIMEOUT = ConfigBuilder("spark.driver.timeout") |
Contributor
There was a problem hiding this comment.
Perhaps we should also add corresponding entries in configuration.md?
| logWarning(s"Terminate Driver JVM because it runs after $timeout minute" + | ||
| (if (timeout == 1) "" else "s")) | ||
| // We cannot use 'SparkContext.stop' because SparkContext might be in abnormal situation. | ||
| System.exit(124) |
Contributor
There was a problem hiding this comment.
I suggest adding a definition for 124 in SparkExitCode
dongjoon-hyun
commented
Feb 28, 2024
MemberAuthor
The PR is updated according to your advice, @LuciferYang . |
dongjoon-hyun
commented
Feb 28, 2024
MemberAuthor
Thank you so much! Merged to master for Apache Spark 4.0.0. This plugin PR is irrelevant to CIs. |
TakawaAkirayo pushed a commit
to TakawaAkirayo/spark
that referenced
this pull request
Mar 4, 2024
…Plugin` ### What changes were proposed in this pull request? This PR aims to support `spark.driver.timeout` and `DriverTimeoutPlugin`. ### Why are the changes needed? Sometime, Spark applications fall into abnormal situation and hang. We had better provide a way to guarantee the termination after pre-defined timeout via a standard way. - spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin - spark.driver.timeout=1min ``` $ bin/spark-shell -c spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin -c spark.driver.timeout=1min Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.10) Type in expressions to have them evaluated. Type :help for more information. 24/02/28 06:53:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1709132014477). Spark session available as 'spark'. scala> 24/02/28 06:54:34 WARN DriverTimeoutDriverPlugin: Terminate Driver JVM because it runs after 1 minute $ echo $? 124 ``` ### Does this PR introduce _any_ user-facing change? No, this is a new feature and a built-in plugin. ### How was this patch tested? Manually because this invokes `System.exit`. 1. Timeout with 1 minute ``` $ bin/spark-shell -c spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin -c spark.driver.timeout=1min ... scala> 24/02/28 06:54:34 WARN DriverTimeoutDriverPlugin: Terminate Driver JVM because it runs after 1 minute $ echo $? 124 ``` 2. `DriverTimeoutPlugin` will be ignored if the default value of `spark.driver.timeout` is used. ``` $ bin/spark-shell -c spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin ... 24/02/28 01:02:57 WARN DriverTimeoutDriverPlugin: Disabled with the timeout value 0. ... scala> ``` 3. `spark.driver.timeout` will be ignored if `DriverTimeoutPlugin` is not provided. ### Was this patch authored or co-authored using generative AI tooling? No. Closesapache#45313 from dongjoon-hyun/SPARK-47207. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
ericm-db pushed a commit
to ericm-db/spark
that referenced
this pull request
Mar 5, 2024
…Plugin` ### What changes were proposed in this pull request? This PR aims to support `spark.driver.timeout` and `DriverTimeoutPlugin`. ### Why are the changes needed? Sometime, Spark applications fall into abnormal situation and hang. We had better provide a way to guarantee the termination after pre-defined timeout via a standard way. - spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin - spark.driver.timeout=1min ``` $ bin/spark-shell -c spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin -c spark.driver.timeout=1min Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.10) Type in expressions to have them evaluated. Type :help for more information. 24/02/28 06:53:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1709132014477). Spark session available as 'spark'. scala> 24/02/28 06:54:34 WARN DriverTimeoutDriverPlugin: Terminate Driver JVM because it runs after 1 minute $ echo $? 124 ``` ### Does this PR introduce _any_ user-facing change? No, this is a new feature and a built-in plugin. ### How was this patch tested? Manually because this invokes `System.exit`. 1. Timeout with 1 minute ``` $ bin/spark-shell -c spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin -c spark.driver.timeout=1min ... scala> 24/02/28 06:54:34 WARN DriverTimeoutDriverPlugin: Terminate Driver JVM because it runs after 1 minute $ echo $? 124 ``` 2. `DriverTimeoutPlugin` will be ignored if the default value of `spark.driver.timeout` is used. ``` $ bin/spark-shell -c spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin ... 24/02/28 01:02:57 WARN DriverTimeoutDriverPlugin: Disabled with the timeout value 0. ... scala> ``` 3. `spark.driver.timeout` will be ignored if `DriverTimeoutPlugin` is not provided. ### Was this patch authored or co-authored using generative AI tooling? No. Closesapache#45313 from dongjoon-hyun/SPARK-47207. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
snmvaughan pushed a commit
to snmvaughan/spark
that referenced
this pull request
Mar 26, 2024
…Plugin` This PR aims to support `spark.driver.timeout` and `DriverTimeoutPlugin`. Sometime, Spark applications fall into abnormal situation and hang. We had better provide a way to guarantee the termination after pre-defined timeout via a standard way. - spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin - spark.driver.timeout=1min ``` $ bin/spark-shell -c spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin -c spark.driver.timeout=1min Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.10) Type in expressions to have them evaluated. Type :help for more information. 24/02/28 06:53:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1709132014477). Spark session available as 'spark'. scala> 24/02/28 06:54:34 WARN DriverTimeoutDriverPlugin: Terminate Driver JVM because it runs after 1 minute $ echo $? 124 ``` No, this is a new feature and a built-in plugin. Manually because this invokes `System.exit`. 1. Timeout with 1 minute ``` $ bin/spark-shell -c spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin -c spark.driver.timeout=1min ... scala> 24/02/28 06:54:34 WARN DriverTimeoutDriverPlugin: Terminate Driver JVM because it runs after 1 minute $ echo $? 124 ``` 2. `DriverTimeoutPlugin` will be ignored if the default value of `spark.driver.timeout` is used. ``` $ bin/spark-shell -c spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin ... 24/02/28 01:02:57 WARN DriverTimeoutDriverPlugin: Disabled with the timeout value 0. ... scala> ``` 3. `spark.driver.timeout` will be ignored if `DriverTimeoutPlugin` is not provided. No. Closesapache#45313 from dongjoon-hyun/SPARK-47207. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
dongjoon-hyun added a commit
that referenced
this pull request
Jan 24, 2026
…tException` for executor cpu misconfigs ### What changes were proposed in this pull request? This PR aims to fix `BasicExecutorFeatureStep` to throw `IllegalArgumentException` for executor cpu misconfigs in order the Spark jobs ASAP. ### Why are the changes needed? From Apache Spark 4.1.0, Spark driver pod throws `SparkException` for the executor cpu misconfiguration before requesting to the K8s control plain. This improvement reduces the burden of K8s control plane. - #51678 ``` 26/01/24 06:55:31 INFO ExecutorPodsAllocator: Going to request 5 executors from Kubernetes for ResourceProfile Id: 0, target: 5, known: 0, sharedSlotFromPendingPods: 2147483647. 26/01/24 06:55:31 INFO ExecutorPodsAllocator: Found 0 reusable PVCs from 0 PVCs 26/01/24 06:55:31 WARN ExecutorPodsSnapshotsStoreImpl: Exception when notifying snapshot subscriber. org.apache.spark.SparkException: The executor cpu request (4) should be less than or equal to cpu limit (1) at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.$anonfun$configurePod$11(BasicExecutorFeatureStep.scala:236) ``` However, the Spark driver keeps re-trying to create executor pods in any way if the users didn't have an additional `spark.driver.timeout` configuration. - #45313 So, we had better exit the Spark job in this case ASAP. We can do that simply switches `SparkException` to `IllegalArgumentException` like the other steps. - #30084 ### Does this PR introduce _any_ user-facing change? Technically no because previously those misconfigured Spark job didn't get any resources. ### How was this patch tested? Pass the CIs with the updated test case. Also, I checked manually via `spark-submit`: ``` $ bin/spark-submit --master k8s://$K8S_MASTER \ --deploy-mode cluster \ -c spark.executor.instances=5 \ -c spark.kubernetes.executor.request.cores=4 \ -c spark.kubernetes.executor.limit.cores=1 \ -c spark.kubernetes.container.image=apache/spark:SPARK-55134 \ -c spark.kubernetes.authenticate.driver.serviceAccountName=spark \ -c spark.kubernetes.executor.useDriverPodIP=true \ --class org.apache.spark.examples.SparkPi \ local:///opt/spark/examples/jars/spark-examples.jar 200000 ... 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: State changed, new state: pod name: org-apache-spark-examples-sparkpi-0482f19beeec7491-driver namespace: default labels: spark-app-name -> org-apache-spark-examples-sparkpi, spark-app-selector -> spark-ee23f03db88b43fb906b0dbc1b04ad63, spark-role -> driver, spark-version -> 4.2.0-SNAPSHOT pod uid: c6d41845-5893-4135-a065-278d94500315 creation time: 2026-01-24T07:33:52Z service account name: spark volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-8rbc8 node name: lima-rancher-desktop start time: 2026-01-24T07:33:52Z phase: Failed container status: container name: spark-kubernetes-driver container image: apache/spark:SPARK-55134 container state: terminated container started at: 2026-01-24T07:33:53Z container finished at: 2026-01-24T07:33:55Z exit code: 1 termination reason: Error 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Application status for spark-ee23f03db88b43fb906b0dbc1b04ad63 (phase: Failed) 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Container final statuses: container name: spark-kubernetes-driver container image: apache/spark:SPARK-55134 container state: terminated container started at: 2026-01-24T07:33:53Z container finished at: 2026-01-24T07:33:55Z exit code: 1 termination reason: Error 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Application org.apache.spark.examples.SparkPi with application ID spark-ee23f03db88b43fb906b0dbc1b04ad63 and submission ID default:org-apache-spark-examples-sparkpi-0482f19beeec7491-driver finished ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes#53948 from dongjoon-hyun/SPARK-55134. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
dongjoon-hyun added a commit
that referenced
this pull request
Jan 24, 2026
…tException` for executor cpu misconfigs ### What changes were proposed in this pull request? This PR aims to fix `BasicExecutorFeatureStep` to throw `IllegalArgumentException` for executor cpu misconfigs in order the Spark jobs ASAP. ### Why are the changes needed? From Apache Spark 4.1.0, Spark driver pod throws `SparkException` for the executor cpu misconfiguration before requesting to the K8s control plain. This improvement reduces the burden of K8s control plane. - #51678 ``` 26/01/24 06:55:31 INFO ExecutorPodsAllocator: Going to request 5 executors from Kubernetes for ResourceProfile Id: 0, target: 5, known: 0, sharedSlotFromPendingPods: 2147483647. 26/01/24 06:55:31 INFO ExecutorPodsAllocator: Found 0 reusable PVCs from 0 PVCs 26/01/24 06:55:31 WARN ExecutorPodsSnapshotsStoreImpl: Exception when notifying snapshot subscriber. org.apache.spark.SparkException: The executor cpu request (4) should be less than or equal to cpu limit (1) at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.$anonfun$configurePod$11(BasicExecutorFeatureStep.scala:236) ``` However, the Spark driver keeps re-trying to create executor pods in any way if the users didn't have an additional `spark.driver.timeout` configuration. - #45313 So, we had better exit the Spark job in this case ASAP. We can do that simply switches `SparkException` to `IllegalArgumentException` like the other steps. - #30084 ### Does this PR introduce _any_ user-facing change? Technically no because previously those misconfigured Spark job didn't get any resources. ### How was this patch tested? Pass the CIs with the updated test case. Also, I checked manually via `spark-submit`: ``` $ bin/spark-submit --master k8s://$K8S_MASTER \ --deploy-mode cluster \ -c spark.executor.instances=5 \ -c spark.kubernetes.executor.request.cores=4 \ -c spark.kubernetes.executor.limit.cores=1 \ -c spark.kubernetes.container.image=apache/spark:SPARK-55134 \ -c spark.kubernetes.authenticate.driver.serviceAccountName=spark \ -c spark.kubernetes.executor.useDriverPodIP=true \ --class org.apache.spark.examples.SparkPi \ local:///opt/spark/examples/jars/spark-examples.jar 200000 ... 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: State changed, new state: pod name: org-apache-spark-examples-sparkpi-0482f19beeec7491-driver namespace: default labels: spark-app-name -> org-apache-spark-examples-sparkpi, spark-app-selector -> spark-ee23f03db88b43fb906b0dbc1b04ad63, spark-role -> driver, spark-version -> 4.2.0-SNAPSHOT pod uid: c6d41845-5893-4135-a065-278d94500315 creation time: 2026-01-24T07:33:52Z service account name: spark volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-8rbc8 node name: lima-rancher-desktop start time: 2026-01-24T07:33:52Z phase: Failed container status: container name: spark-kubernetes-driver container image: apache/spark:SPARK-55134 container state: terminated container started at: 2026-01-24T07:33:53Z container finished at: 2026-01-24T07:33:55Z exit code: 1 termination reason: Error 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Application status for spark-ee23f03db88b43fb906b0dbc1b04ad63 (phase: Failed) 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Container final statuses: container name: spark-kubernetes-driver container image: apache/spark:SPARK-55134 container state: terminated container started at: 2026-01-24T07:33:53Z container finished at: 2026-01-24T07:33:55Z exit code: 1 termination reason: Error 26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Application org.apache.spark.examples.SparkPi with application ID spark-ee23f03db88b43fb906b0dbc1b04ad63 and submission ID default:org-apache-spark-examples-sparkpi-0482f19beeec7491-driver finished ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes#53948 from dongjoon-hyun/SPARK-55134. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit ab3ec9e) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR aims to support
spark.driver.timeoutandDriverTimeoutPlugin.Why are the changes needed?
Sometime, Spark applications fall into abnormal situation and hang.
We had better provide a way to guarantee the termination after pre-defined timeout via a standard way.
Does this PR introduce any user-facing change?
No, this is a new feature and a built-in plugin.
How was this patch tested?
Manually because this invokes
System.exit.DriverTimeoutPluginwill be ignored if the default value ofspark.driver.timeoutis used.spark.driver.timeoutwill be ignored ifDriverTimeoutPluginis not provided.Was this patch authored or co-authored using generative AI tooling?
No.