[SPARK-58967][K8S] Use FQDN with configurable cluster domain for driver hostname

[shazebkhan1]

What changes were proposed in this pull request?

Introduces a new config spark.kubernetes.clusterDomain (default: cluster.local) and changes
the driver hostname constructed in DriverServiceFeatureStep from a partial DNS name to a
fully-qualified domain name (FQDN) with a trailing dot:

Before:

<service>.<namespace>.svc

After:

<service>.<namespace>.svc.<clusterDomain>.

The trailing dot tells the DNS resolver to treat the name as absolute, bypassing the pod's
configured search domain list entirely.

Why are the changes needed?

When executors resolve spark.driver.host, the standard Kubernetes pod DNS configuration uses
ndots:5 with search domains:

search <namespace>.svc.cluster.local svc.cluster.local cluster.local

The existing partial hostname <service>.<namespace>.svc contains only 2 dots, which is below
the ndots:5 threshold. The resolver therefore attempts to resolve it by appending each search
domain in turn, producing two NXDOMAIN responses before arriving at the correct address:

Attempt 1: <service>.<namespace>.svc.<namespace>.svc.cluster.local. → NXDOMAIN
Attempt 2: <service>.<namespace>.svc.svc.cluster.local.             → NXDOMAIN
Attempt 3: <service>.<namespace>.svc.cluster.local.                 → NOERROR

In production environments running many concurrent Spark pipelines, each executor startup
generates these spurious failed lookups. This adds unnecessary NXDOMAIN traffic to CoreDNS
(a shared cluster-wide component) and can contribute to delays during executor-to-driver
connection establishment.

Using a trailing-dot FQDN forces the resolver to issue a single absolute lookup, eliminating
all NXDOMAIN responses. The new spark.kubernetes.clusterDomain config accommodates clusters
that use a non-default domain (i.e., something other than cluster.local).

Does this PR introduce any user-facing change?

Yes. spark.driver.host is now set to <service>.<namespace>.svc.cluster.local. by default
instead of <service>.<namespace>.svc. This is backward compatible for standard clusters.
Users on clusters with a non-default domain can configure spark.kubernetes.clusterDomain.

How was this patch tested?

The fix was validated by manually patching the jar and testing it in a local Kubernetes
development environment. The following scenarios were verified:

• Tested with the default cluster.local domain.
• Tested with a custom cluster DNS domain.
• Verified the resulting driver hostname is a trailing-dot FQDN.
• Confirmed that executor-to-driver connections succeed without any NXDOMAIN lookups in CoreDNS.

[dongjoon-hyun]

Thank you for the PR, and for the detailed description of the ndots:5 behavior.

I have a concern about the default value being a silent behavior change.

spark.kubernetes.clusterDomain defaults to cluster.local, and the driver hostname is now always emitted as an absolute name with a trailing dot. On a cluster configured with a non-default DNS domain (kubelet --cluster-domain=...), the existing <service>.<namespace>.svc resolves correctly through the pod's search list, but <service>.<namespace>.svc.cluster.local. bypasses the search list entirely and gets NXDOMAIN. Such clusters would break after upgrading, and there is no way to restore the previous behavior because every value of the new config produces an FQDN.

Note also that DriverServiceFeatureStep runs on the submission client, which is often outside the cluster, so the correct domain cannot be auto-detected there.

Could we make this opt-in instead? For example, createOptional, and keep the current hostname when it is unset:

val driverHostname = kubernetesConf.get(KUBERNETES_CLUSTER_DOMAIN)
  .map(d => s"$resolvedServiceName.${kubernetesConf.namespace}.svc.$d.")
  .getOrElse(s"$resolvedServiceName.${kubernetesConf.namespace}.svc")

If we want to keep cluster.local as the default, we would at least need a documented way to opt out and restore the legacy hostname.

[dongjoon-hyun]

One more note on the motivation section, which no longer matches master.

As I mentioned in dev mailing list, spark.kubernetes.executor.useDriverPodIP is enabled by default, so executor pods connect to the driver RPC endpoint via the driver pod IP rather than the driver's K8s Service DNS name:

https://github.com/apache/spark/blob/841f3a0332d/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L56-L61

So "each executor startup generates these spurious failed lookups" is not accurate for the default configuration on master anymore. What still resolves spark.driver.host is mainly the driver's advertised BlockManager address that executors use for block transfers:

https://github.com/apache/spark/blob/841f3a0332d/core/src/main/scala/org/apache/spark/SparkEnv.scala#L718-L720

and those lookups are additionally subject to the JVM DNS cache.

Could you update the description to describe the paths that still resolve the name on master, and share the measured effect on a current build (e.g. CoreDNS NXDOMAIN counts before/after)? Also, the test section mentions patching a jar manually. Which Spark version was that built from? If it predates 4.3, the observed NXDOMAIN volume would be from the driver service path that is no longer the default.

[shazebkhan1]

Hi @dongjoon-hyun,

We wanted to share our experience with SPARK-53944 and also report a related issue we discovered during testing.

SPARK-53944 — Backported and validated on Spark 3.5.2

We cherry-picked the changes from PRs #52650, #52923, and #52954 (Config.scala, BasicExecutorFeatureStep.scala, SparkContext.scala) onto a clean v3.5.2 base, along with the SPARK-57351 default-to-true change from PR #56412. After rebuilding spark-core and spark-kubernetes JARs and deploying the patched artifacts:

• Executor pods now connect directly to the driver pod IP.
• NXDOMAIN queries for the driver Service name are gone from CoreDNS.

The fix works as expected. Thank you for the work on this.

Separate issue — API server NXDOMAIN queries from the driver pod

After resolving the executor-to-driver DNS issue, we observed a different set of NXDOMAIN queries in CoreDNS originating from the driver pod itself on every Kubernetes API server call:

NXDOMAIN  kubernetes.default.svc.default.svc.cluster.local.
NXDOMAIN  kubernetes.default.svc.svc.cluster.local.
NOERROR   kubernetes.default.svc.cluster.local.

This is the ndots:5 search-domain expansion: kubernetes.default.svc has only 2 dots so the resolver appends all search domains before trying it as absolute, producing 2 NXDOMAIN responses on every API server call.

Root cause

SparkKubernetesClientFactory.createKubernetesClient() calls Fabric8's autoConfigure(), which correctly detects in-cluster execution via KUBERNETES_SERVICE_HOST / KUBERNETES_SERVICE_PORT and sets the master URL to the raw API server IP (e.g. https://10.96.0.1:443). Spark then unconditionally overwrites this with .withMasterUrl(master) where master defaults to KUBERNETES_MASTER_INTERNAL_URL = "https://kubernetes.default.svc", negating Fabric8's work and forcing a hostname-based DNS lookup on every request.

We confirmed this is present in Spark 3.5.x, branch-4.1, and current master (5.0-SNAPSHOT). This is orthogonal to SPARK-53944 — that fix addresses executor-to-driver connectivity; this affects driver-to-API-server connectivity.

Proposed fix

A minimal, single-file change to SparkKubernetesClientFactory.scala. The fix uses clientType, which is already threaded through createKubernetesClient, to reliably identify the in-cluster driver. It also checks that spark.kubernetes.driver.master has not been explicitly configured, so existing user overrides are fully preserved.

val baseConfig = new ConfigBuilder(autoConfigure(kubeContext.orNull))
  .withApiVersion("v1")

// For the submission client, always use the configured spark.master as-is.
// This is important for both same-cluster and cross-cluster submissions.
//
// For the driver running inside Kubernetes, use the API server IP and port
// injected by Kubernetes via KUBERNETES_SERVICE_HOST / KUBERNETES_SERVICE_PORT.
// This avoids DNS resolution of kubernetes.default.svc and the associated
// ndots:5 search-domain lookups that cause NXDOMAIN responses on every API call.
//
// This only applies when spark.kubernetes.driver.master is not explicitly set.
// If the user has configured it, that value is respected as-is (existing behaviour).
val driverMasterExplicitlySet = sparkConf.contains(KUBERNETES_DRIVER_MASTER_URL.key)
val configWithMaster =
  if (clientType == ClientType.Driver &&
      sys.env.contains("KUBERNETES_SERVICE_HOST") &&
      sys.env.contains("KUBERNETES_SERVICE_PORT") &&
      !driverMasterExplicitlySet) {

    val host = sys.env("KUBERNETES_SERVICE_HOST")
    val port = sys.env("KUBERNETES_SERVICE_PORT")
    val inClusterApiServer = s"https://$host:$port"

    logInfo(
      s"Running in-cluster driver; using Kubernetes API server $inClusterApiServer " +
      "for Driver-to-API-server communication."
    )

    baseConfig.withMasterUrl(inClusterApiServer)
  } else {
    baseConfig.withMasterUrl(master)
  }

val config = configWithMaster
  .withRequestTimeout(...)  // rest unchanged

Compatibility

All four conditions must be true for the fix to activate. If any one is false the code falls through to baseConfig.withMasterUrl(master) — identical to the original behaviour.

Scenario	Fix activates?	What is used
Submission client (any environment)	No	spark.master — unchanged
spark.kubernetes.driver.master explicitly set	No	Configured value — unchanged
Driver pod, running outside cluster	No	spark.master — unchanged
Driver pod, in-cluster, no explicit override	Yes	KUBERNETES_SERVICE_HOST:PORT — raw ClusterIP, zero DNS

Would the team be open to a PR for this against master?

Best regards,
Shazeb Khan