[SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods. by ScrapCodes · Pull Request #27735 · apache/spark

ScrapCodes · 2020-02-28T11:15:59Z

What changes were proposed in this pull request?

This is an improvement, we mount all the user specific configuration files(except the templates and spark properties files) from SPARK_CONF_DIR at the point of spark-submit, to both executor and driver pods. Currently, only spark.properties is mounted, only on driver.

Why are the changes needed?

SPARK_CONF_DIR hosts several configuration files, for example,

spark-defaults.conf - containing all the spark properties.
log4j.properties - Logger configuration.
core-site.xml - Hadoop related configuration.
fairscheduler.xml - Spark's fair scheduling policy at the job level.
metrics.properties - Spark metrics.
Any user specific - library or framework specific configuration file.

At the moment, we can cannot propagate these files to the driver and executor configuration directory.

There is a design doc, with more details, and this patch is currently providing a reference implementation. Please take a look at the doc and comment, how we can improve. google docs link to the doc

Further scope

Support user defined configMaps.

Does this PR introduce any user-facing change?

Yes, previously the user configuration files(e.g. hdfs-site.xml, log4j.properties etc...) were not propagated by default, now after this patch it is propagated to driver and executor pods' SPARK_CONF_DIR.

How was this patch tested?

Added tests.

Also manually tested, by deploying it to a minikube cluster and observing the additional configuration files were present, and taking effect. For example, changes to log4j.properties was properly applied to executors.

SparkQA · 2020-02-28T11:34:33Z

Test build #119087 has finished for PR 27735 at commit 2b71a77.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-28T12:00:54Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23831/

SparkQA · 2020-02-28T12:31:45Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23831/

ScrapCodes · 2020-02-28T15:50:24Z

Integration test decommission suite passed locally, however it seems to be a flaky one.

retest please

tgravescs · 2020-02-28T16:40:49Z

ill let the user specific configuration files be mounted on the driver and executor pods'

Can you expand on this, what does "mount" mean here in a distributed environment. The place $SPARK_CONF_DIR is pointing has to be on distributed filesystem. if its pointing to local disk how do you mount that on remote pod.

ScrapCodes · 2020-03-02T07:03:29Z

ill let the user specific configuration files be mounted on the driver and executor pods'
Can you expand on this, what does "mount" mean here in a distributed environment. The place $SPARK_CONF_DIR is pointing has to be on distributed filesystem. if its pointing to local disk how do you mount that on remote pod.

@tgravescs, Hi Thomas Graves, the mount here refers to the kubernetes feature that it lets us mount ConfigMaps as Volumes. More details here. https://kubernetes.io/docs/concepts/storage/volumes/#configmap. SPARK_CONF_DIR is not required to be on a distributed filesystem.

I have also updated the description with more details how the Kubernetes config map will work here.

Thanks a lot for taking a look though, do you have more comments? can you please help review this patch !

SparkQA · 2020-03-02T07:28:36Z

Test build #119155 has finished for PR 27735 at commit f137617.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-02T08:09:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23897/

SparkQA · 2020-03-02T08:38:31Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23897/

SparkQA · 2020-03-02T08:50:23Z

Test build #119159 has finished for PR 27735 at commit fe10c37.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-02T09:14:29Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23901/

SparkQA · 2020-03-02T09:44:39Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23901/

ScrapCodes · 2020-03-02T11:55:05Z

@holdenk Hi Holden, Would you mind taking a look? I am not sure why DecommissionSuite fails on jenkins ? It passed locally though. Do you think this patch has broken the suite, in some way? Thanks !

ScrapCodes · 2020-03-03T09:59:53Z

retest this please

SparkQA · 2020-03-03T10:21:02Z

Test build #119222 has finished for PR 27735 at commit fe10c37.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-03T10:51:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23961/

SparkQA · 2020-03-03T11:11:29Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23961/

ScrapCodes · 2020-03-04T10:58:07Z

@skonto @ifilonenko, can you please review this PR?

SparkQA · 2020-06-11T08:39:49Z

Test build #123835 has finished for PR 27735 at commit fb31e84.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-11T09:04:36Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28460/

SparkQA · 2020-06-11T09:34:02Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28460/

ScrapCodes · 2020-11-07T15:24:25Z

Do we need to rename SecretVolumeUtils.scala to K8sVolumeTestUtils? Actually, the code is still using SecretVolumeUtils.podHasVolume because we didn't change the class name. If you don't mind, shall we keep the original file name?

Hi @dongjoon-hyun , I have already reverted this change. Actually, I did this change initially, because SecretVolumeUtils.podHasVolume is used at more places than it was initially intended and the name Secret has no relevance anymore. But, we can tackle this in a separate patch.

I did another pass to see if all the comments are addressed. Do you think, I am missing something?

Thanks !

dongjoon-hyun

Hi, @ScrapCodes .
Sorry for the long delay. I believe this PR is almost ready for merge. Could you resolve the conflicts once more?

dongjoon-hyun · 2020-11-13T22:54:34Z

 object Fabric8Aliases {
 type PODS = MixedOperation[Pod, PodList, DoneablePod, PodResource[Pod, DoneablePod]]
+ type CONFIG_MAPS = MixedOperation[ConfigMap,
+ ConfigMapList, DoneableConfigMap, Resource[ConfigMap, DoneableConfigMap]]


If you don't mind, shall we use like the following
-typeCONFIG_MAPS=MixedOperation[ConfigMap, -ConfigMapList, DoneableConfigMap, Resource[ConfigMap, DoneableConfigMap]] +typeCONFIG_MAPS=MixedOperation[ +ConfigMap, ConfigMapList, DoneableConfigMap, Resource[ConfigMap, DoneableConfigMap]]

dongjoon-hyun · 2020-11-13T22:54:47Z

 type LABELED_PODS = FilterWatchListDeletable[
 Pod, PodList, java.lang.Boolean, Watch, Watcher[Pod]]
+ type LABELED_CONFIG_MAPS = FilterWatchListDeletable[ConfigMap, ConfigMapList,
+ java.lang.Boolean, Watch, Watcher[ConfigMap]]


-typeLABELED_CONFIG_MAPS=FilterWatchListDeletable[ConfigMap, ConfigMapList, - java.lang.Boolean, Watch, Watcher[ConfigMap]] +typeLABELED_CONFIG_MAPS=FilterWatchListDeletable[ +ConfigMap, ConfigMapList, java.lang.Boolean, Watch, Watcher[ConfigMap]]

dongjoon-hyun · 2020-11-13T23:03:26Z

+ mapping
+ }.toMap
+ } else {
+ logInfo(s"Spark configuration directory is not detected, please set env:$ENV_SPARK_CONF_DIR")


For the existing users, I guess we don't need to give this additional direction because they don't need this.

dongjoon-hyun · 2020-11-13T23:06:35Z

Hi, @holdenk and @tgravescs . Do you have any concern on this proposal?

…pods.

SparkQA · 2020-11-16T07:01:47Z

Test build #131135 has finished for PR 27735 at commit ba93111.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-16T07:38:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35738/

dongjoon-hyun

+1, LGTM, @ScrapCodes .

I verified this manually too.

$ k get cm
NAME DATA AGE
spark-drv-b6d61375d00e2207-conf-map 3 9s
spark-exec-6fef2f75cf696182-conf-map 2 2s
$ k get pod
NAME READY STATUS RESTARTS AGE
org-apache-spark-examples-sparkpi-3aca7e75d00e1f8f-driver 1/1 Running 0 15s
spark-pi-b2de0775cf695c4a-exec-1 1/1 Running 0 8s
$ k exec -it org-apache-spark-examples-sparkpi-3aca7e75d00e1f8f-driver -- ls -al /opt/spark/conf
total 12
drwxrwxrwx 3 root root 4096 Nov 16 04:56 .
drwxr-xr-x 1 root root 4096 Nov 16 04:56 ..
drwxr-xr-x 2 root root 4096 Nov 16 04:56 ..2020_11_16_04_56_26.581426662
lrwxrwxrwx 1 root root 31 Nov 16 04:56 ..data -> ..2020_11_16_04_56_26.581426662
lrwxrwxrwx 1 root root 23 Nov 16 04:56 log4j.properties -> ..data/log4j.properties
lrwxrwxrwx 1 root root 25 Nov 16 04:56 metrics.properties -> ..data/metrics.properties
lrwxrwxrwx 1 root root 23 Nov 16 04:56 spark.properties -> ..data/spark.properties
$ k exec -it spark-pi-b2de0775cf695c4a-exec-1 -- ls -al /opt/spark/conf
total 16
drwxrwxrwx 3 root root 4096 Nov 16 04:56 .
drwxr-xr-x 1 root root 4096 Nov 16 04:56 ..
drwxr-xr-x 2 root root 4096 Nov 16 04:56 ..2020_11_16_04_56_31.214047175
lrwxrwxrwx 1 root root 31 Nov 16 04:56 ..data -> ..2020_11_16_04_56_31.214047175
lrwxrwxrwx 1 root root 23 Nov 16 04:56 log4j.properties -> ..data/log4j.properties
lrwxrwxrwx 1 root root 25 Nov 16 04:56 metrics.properties -> ..data/metrics.properties
$ k exec -it spark-pi-b2de0775cf695c4a-exec-1 -- cat /opt/spark/conf/log4j.properties
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with

dongjoon-hyun · 2020-11-16T08:00:48Z

The current running K8s IT has one irrelevant failure. We can ignore it.

- Test basic decommissioning *** FAILED ***

dongjoon-hyun · 2020-11-16T08:03:04Z

Merged to master for Apache Spark 3.1!

SparkQA · 2020-11-16T08:03:20Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35738/

ScrapCodes · 2020-11-16T08:48:27Z

@dongjoon-hyun Thanks a lot, for reviewing and helping this move forward. :)

dongjoon-hyun · 2020-11-16T20:39:33Z

Sorry for the delay, @ScrapCodes . This feature and the related JIRAs are targeting at Apache Spark 3.1, right?

ScrapCodes · 2020-11-17T02:50:53Z

@dongjoon-hyun Yes !

### What changes were proposed in this pull request? This PR aims to add a new configuration `spark.kubernetes.executor.disableConfigMap`. ### Why are the changes needed? This can be use to disable config map creating for executor pods due to #27735 . ### Does this PR introduce _any_ user-facing change? No. By default, this doesn't change AS-IS behavior. This is a new feature to add an ability to disable SPARK-30985. ### How was this patch tested? Pass the newly added UT. Closes#31428 from dongjoon-hyun/SPARK-34316. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This PR aims to add a new configuration `spark.kubernetes.executor.disableConfigMap`. ### Why are the changes needed? This can be use to disable config map creating for executor pods due to apache#27735 . ### Does this PR introduce _any_ user-facing change? No. By default, this doesn't change AS-IS behavior. This is a new feature to add an ability to disable SPARK-30985. ### How was this patch tested? Pass the newly added UT. Closesapache#31428 from dongjoon-hyun/SPARK-34316. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This PR aims to add a new configuration `spark.kubernetes.executor.disableConfigMap`. ### Why are the changes needed? This can be use to disable config map creating for executor pods due to apache#27735 . ### Does this PR introduce _any_ user-facing change? No. By default, this doesn't change AS-IS behavior. This is a new feature to add an ability to disable SPARK-30985. ### How was this patch tested? Pass the newly added UT. Closesapache#31428 from dongjoon-hyun/SPARK-34316. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit f66e38c) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This PR aims to promote `KubernetesClientUtils` from `Unstable` to `Stable` API. ### Why are the changes needed? `KubernetesClientUtils` is a `DeveloperApi` utility used by both Spark's internal K8s submission path and the Spark K8s Operator. It was introduced in Apache Spark 3.1.0 (SPARK-30985) and becomes `DeveloperAPI` at Spark 4.1.0. - #27735 - #46327 Since 3.1.0, its public surface has remained stable for multiple releases. Promoting it to `Stable` signals this stability to downstream users like the Spark K8s Operator. ### Does this PR introduce _any_ user-facing change? No, this is an annotation-only change (API stability signaling). No behavior change. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.7 Closes#55513 from dongjoon-hyun/SPARK-56601. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

ScrapCodes mentioned this pull request Feb 28, 2020
[SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor. #26193
Closed

ScrapCodes force-pushed the SPARK-30985/spark-conf-k8s-propagate branch from f137617 to fe10c37Compare March 2, 2020 08:30

dongjoon-hyun added the KUBERNETES label Apr 12, 2020

ScrapCodes changed the title ~~[SPARK-30985][SPARK-25065][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.~~[SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.Jun 11, 2020

ScrapCodes force-pushed the SPARK-30985/spark-conf-k8s-propagate branch from fe10c37 to edea4b3Compare June 11, 2020 08:14

probot-autolabelerBot added the DOCS label Jun 11, 2020

ScrapCodes force-pushed the SPARK-30985/spark-conf-k8s-propagate branch from edea4b3 to fb31e84Compare June 11, 2020 08:16

ScrapCodes changed the title ~~[SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.~~[WIP][SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.Jun 12, 2020

ScrapCodes force-pushed the SPARK-30985/spark-conf-k8s-propagate branch from fb31e84 to a596378Compare July 1, 2020 11:47

dongjoon-hyun reviewed Nov 13, 2020
View reviewed changes

Comment thread...managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/ClientSuite.scala

dongjoon-hyun reviewed Nov 13, 2020
View reviewed changes

Comment thread...managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/ClientSuite.scala

dongjoon-hyun reviewed Nov 13, 2020
View reviewed changes

[SPARK-30985][k8s] Propagate SPARK_CONF_DIR files to driver and exec …
ba93111
…pods.

ScrapCodes force-pushed the SPARK-30985/spark-conf-k8s-propagate branch from 4d66b37 to ba93111Compare November 16, 2020 06:49

dongjoon-hyun approved these changes Nov 16, 2020
View reviewed changes

dongjoon-hyun closed this in 8615f35Nov 16, 2020

ScrapCodes deleted the SPARK-30985/spark-conf-k8s-propagate branch November 16, 2020 08:47

dongjoon-hyun mentioned this pull request Feb 2, 2021
[SPARK-34316][K8S] Support spark.kubernetes.executor.disableConfigMap #31428
Closed

ScrapCodes mentioned this pull request Jul 19, 2021
[SPARK-36039][K8S] Fix executor pod hadoop conf mount #33257
Closed

yaooqinn mentioned this pull request May 19, 2023
[SPARK-43504][K8S] Mounts the hadoop config map on the executor pod #41181
Closed

dongjoon-hyun mentioned this pull request Apr 23, 2026
[SPARK-56601][K8S] Promote KubernetesClientUtils to Stable#55513
Closed

Uh oh!

Conversation

ScrapCodes commented Feb 28, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Further scope

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Feb 28, 2020

Uh oh!

SparkQA commented Feb 28, 2020

Uh oh!

SparkQA commented Feb 28, 2020

Uh oh!

ScrapCodes commented Feb 28, 2020

Uh oh!

tgravescs commented Feb 28, 2020

Uh oh!

ScrapCodes commented Mar 2, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Mar 2, 2020

Uh oh!

SparkQA commented Mar 2, 2020

Uh oh!

SparkQA commented Mar 2, 2020

Uh oh!

SparkQA commented Mar 2, 2020

Uh oh!

SparkQA commented Mar 2, 2020

Uh oh!

SparkQA commented Mar 2, 2020

Uh oh!

ScrapCodes commented Mar 2, 2020

Uh oh!

ScrapCodes commented Mar 3, 2020

Uh oh!

SparkQA commented Mar 3, 2020

Uh oh!

SparkQA commented Mar 3, 2020

Uh oh!

SparkQA commented Mar 3, 2020

Uh oh!

ScrapCodes commented Mar 4, 2020

Uh oh!

SparkQA commented Jun 11, 2020

Uh oh!

SparkQA commented Jun 11, 2020

Uh oh!

SparkQA commented Jun 11, 2020

Uh oh!

ScrapCodes commented Nov 7, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyunNov 13, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyunNov 13, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyunNov 13, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Nov 13, 2020

Uh oh!

SparkQA commented Nov 16, 2020

Uh oh!

SparkQA commented Nov 16, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Nov 16, 2020

ScrapCodes commented Feb 28, 2020•
edited
Loading

ScrapCodes commented Mar 2, 2020•
edited
Loading

dongjoon-hyunNov 13, 2020•
edited
Loading

ScrapCodes commented Nov 17, 2020•
edited
Loading