[SPARK-50758][K8S] Mounts the krb5 config map on the executor pod

[maomaodev]

What changes were proposed in this pull request?

In this pr, for spark on k8s, the krb5.conf config map will be mounted in executor side as well. Before, the krb5.conf config map is only mounted in driver side. But according to the parameter description (spark.kubernetes.kerberos.krb5.path and spark.kubernetes.kerberos.krb5.configMapName), the krb5.conf file should be mounted on both the driver and the executor.

This is the second attempt of #49467, which added the ability to mount the krb5.conf ConfigMap on executor pods. The first attempt only worked in cluster mode; this follow-up adds client mode compatibility. The two modes work as follows:

Mode	Who creates the krb5 ConfigMap	Naming
cluster	KerberosConfDriverFeatureStep (driver pod)	<resourceNamePrefix>-krb5-file
client	KubernetesClusterSchedulerBackend.setUpExecutorKrb5ConfigMap	spark-krb5-<uniqueID>-conf-map

Both paths write KRB_CONFIG_MAP_NAME into the executor SparkConf, and KerberosConfExecutorFeatureStep reads it to mount /etc/krb5.conf.

Why are the changes needed?

After SPARK-43504, the hadoop config map will be mounted on the executor pod.
Now the executor pod fails to start because the hadoop conf file contains Kerberos authentication configuration, but the executor does not mount krb5.conf correctly. See the #41181 discuss.

Does this PR introduce any user-facing change?

Yes, users do not need to take workarounds to make executors load the krb5.conf. Such as:

• including krb5.conf in executor image
• placing krb5.conf in executor working directory using --files

How was this patch tested?

Added UT:

• KerberosConfExecutorFeatureStepSuite — mount / no-op / driver-step publishing.
• KubernetesClusterSchedulerBackendSuite — setUpExecutorKrb5ConfigMap covers 4 branches (no krb5 config / cluster mode already set / user-provided ConfigMap / local file).
• KerberosConfDriverFeatureStepSuite — updated assertions for the published system property.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: DeepSeek-V4-Pro

[maomaodev]

@pan3793 Could you please take a look when you have time? Thanks!

[uros-b]

I updated the PR title ever so slightly (space nit)

[uros-b]

Change seems correct to me, but adding @pan3793 to review

[maomaodev]

Change seems correct to me, but adding @pan3793 to review

Thanks for the review!

[uros-b]

also adding @yaooqinn to help load balancing with @pan3793

[tdcmeehan]

1 blocking, 0 non-blocking, 0 nits.
The cross-mode design is coherent, but the local-file path can create an executor volume that lacks the fixed subpath it tries to mount.

Correctness (1)

• resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala:123: The client-mode local-file path can create a ConfigMap without the krb5.conf subpath mounted by executors. -- see inline

Verification

Traced both ConfigMap-name producers into KerberosConfExecutorFeatureStep. For the client-mode local-file branch, the ConfigMap data key is file.getName, while the executor volume mount always requests the krb5.conf subpath; the added test explicitly permits a generated non-krb5.conf basename but does not validate the projected path.

[tdcmeehan]

Store this entry under krb5.conf, or project the source key to that path. The executor mount always selects subPath("krb5.conf"), so a supported local file with any other basename creates a ConfigMap volume without the requested path and the executor pod cannot start. Please also assert the projected key/path in the local-file test.