[SPARK-57900][K8S][TESTS] Add OIDC credential propagation E2E tests on Minikube with moto

[sarutak]

What changes were proposed in this pull request?

Add a new optional integration-test module, connector/credential-aws-integration-tests, that validates the end-to-end OIDC credential propagation pipeline on a real Kubernetes cluster (Minikube). This is Sub-task 11 of the OIDC Credential Propagation SPIP (SPARK-57703), and it exercises the
whole feature together: projected ServiceAccount token -> FileTokenIngestor -> AwsStsCredentialProvider -> STS -> S3A read/write, plus mid-job token rotation and late-registering executors.

The tests use moto (Apache 2.0-licensed) as a lightweight S3 + STS backend. The original SPIP mentioned LocalStack, but both LocalStack and MinIO have moved away from freely usable OSS distributions and are incompatible with the ASF license policy. moto runs as a plain HTTP server (no extra container) and does not
verify the OIDC JWT, keeping the test focused on Spark's credential propagation logic.

Three scenarios are implemented:

1. Basic flow (OidcS3ReadWriteJob): a Spark job on Minikube exchanges the identity token for STS credentials and reads/writes S3 via S3A.
2. Mid-job token rotation (OidcTokenRotationJob): a long-running job writes to S3 repeatedly while the test rewrites the identity token file in the driver pod. The initial token is supplied by an init container into an emptyDir (an externally-provided, rotatable token file, as the SPIP assumes). The rotated token carries a different principal; with a short renewal interval, UserCredentialManager re-reads it, re-exchanges it via STS, and propagates fresh credentials. The test asserts the driver logged the rotated principal (proving the new token was actually read, not a no-op) and that S3 output for all iterations spanning the rotation is present.
3. Late-registering executor (OidcLateExecutorJob): with dynamic allocation and a short idle timeout, a job warms up, idles until executors scale down, then runs a wider stage that forces new executors to register after credentials were acquired. The test asserts more than one distinct executor registered over the run (evidence of a genuinely late-registering executor) and that the wide stage produced all outputs — an executor that did not receive credentials via the SparkAppConfig registration response would have failed its task.

Structure and design:

• The module is gated behind the -Poidc-e2e Maven profile (and requires -Pkubernetes), so it is skipped by default.
• Jobs that run on the cluster live in src/main so they are packaged into the module jar and baked into the Spark image; test classes are not packaged.
• S3A support (hadoop-aws + AWS SDK) is provided by building the image with -Phadoop-cloud.
• Image building is handled by an explicit step (docker-image-tool.sh) in CI and by dev-run-integration-tests.sh locally, rather than being bound to the sbt test task.
• The spark-submit helpers (SparkAppLauncher, SparkAppConf, SparkAppArguments, ProcessUtils) are implemented locally instead of depending on the spark-kubernetes-integration-tests test-jar, which sbt could not resolve as an inter-project reference. They mirror the equivalents there.
• moto is reached from two vantage points: pods use the host gateway IP (spark.oidc.test.s3Endpoint / stsEndpoint), while the test process uses loopback (spark.oidc.test.s3ClientEndpoint). In CI, moto is installed into an isolated virtualenv
  (to avoid the OS-provided urllib3/pyOpenSSL that crashes moto on startup) and started inside the same workflow step that runs the tests.

New files:

• connector/credential-aws-integration-tests/ — module with the test suite (OidcCredentialE2ESuite), the three Spark jobs, spark-submit helpers, pom.xml, log4j2.properties, a local runner script (dev-run-integration-tests.sh), and README.md.

Modified files:

• pom.xml (root) — add the oidc-e2e profile / module.
• project/SparkBuild.scala — register credentialAwsIntegrationTests.
• .github/workflows/build_and_test.yml — add the oidc-e2e job (moto + Minikube).

Why are the changes needed?

The SPIP calls for an end-to-end test that validates the full credential propagation pipeline in a realistic Kubernetes environment. The prior sub-tasks each cover a slice with unit/integration tests, but nothing exercised the entire flow — token ingestion, STS exchange, RPC + SparkAppConfig propagation, S3A read/write, and mid-job refresh — against a real cluster. This module provides that coverage and guards against regressions in how the pieces fit together.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

This is the test. The suite was run on a local Minikube (with moto) under both build tools and all three scenarios passed:

• sbt: build/sbt -Phadoop-3 -Pkubernetes -Pcredential-aws -Poidc-e2e ... credential-aws-integration-tests/test
• Maven: build/mvn integration-test -pl connector/credential-aws-integration-tests -Phadoop-3 -Pkubernetes -Pcredential-aws -Poidc-e2e ...

The new oidc-e2e GitHub Actions job (Minikube + moto) is green. dev-run-integration-tests.sh was also verified to build the image, start/stop moto, and run the suite end-to-end.

Was this patch authored or co-authored using generative AI tooling?

Kiro CLI / Claude

[dongjoon-hyun]

Thanks for working on this! The test design is genuinely good — I especially like that the assertions can't pass vacuously: the rotation test checks for the rotated principal in the driver log rather than just "all iterations wrote", and the late-executor test requires more than one distinct executor ID from the Registered executor ... with ID lines. Choosing moto over LocalStack for license reasons is the right call too, and the job finishes in ~12 minutes (vs ~70 for the k8s IT job), so the added CI cost is modest.

My main concern is the CI wiring rather than the test code: as written, the new job will rarely (and on apache/spark, never) run.

1. oidc-e2e is not wired into the scheduled daily builds

build_java17.yml, build_java21.yml, and build_java25.yml pass an explicit jobs JSON that includes "k8s-integration-tests": "true" but not "oidc-e2e": "true". Those workflows are gated on github.repository == 'apache/spark' and are the only place the K8s integration tests run against master, so the new job is skipped there permanently.

2. The trigger condition doesn't cover the code under test

\"oidc-e2e\" : \"$kubernetes\",

kubernetes comes from ./dev/is-changed.py -m kubernetes, whose source_file_regexes is only resource-managers/kubernetes. So a change to connector/credential-aws — the module this suite exists to guard — does not trigger the job:

$ python3 -c "import sparktestsupport.utils as u; print(sorted(m.name for m in u.determine_modules_for_files(['connector/credential-aws/pom.xml'])))"
['build', 'credential-aws']

Something like is-changed.py -m kubernetes,credential-aws would match the intent.

3. The new module path isn't registered in dev/sparktestsupport/modules.py

Files under connector/credential-aws-integration-tests/ match no module, so they fall through to root:

$ python3 -c "import sparktestsupport.utils as u; print(sorted(m.name for m in u.determine_modules_for_files(['connector/credential-aws-integration-tests/src/test/scala/A.scala'])))"
['root']

root triggers the entire CI matrix, so a one-line edit to an optional integration-test module runs everything — the opposite problem from #2. Adding the path to credential_aws.source_file_regexes (or a dedicated Module) fixes both directions.

---

Correctness / robustness

No pod cleanup between tests.KubernetesSuite deletes the driver and executor pods in after; this suite only deletes the namespace in afterAll, and only when it created it. Tests 2 and 3 run with waitAppCompletion=false, so if test 2 fails mid-run its driver and executors keep running while test 3 asks for up to 3 executors on a 2 CPU / 6 GB Minikube. A finally (or after) that deletes the driver pod would make this much less fragile.

afterAll can skip closing the clients.

if (createdNamespace) { kubernetesClient.namespaces().withName(namespace).delete() }
Option(s3Client).foreach(_.close())
Option(kubernetesClient).foreach(_.close())

If delete() throws, neither client is closed. Wrapping each step in Utils.tryLogNonFatalError would be safer.

busybox:1.36 init container pulls from Docker Hub at test time. This adds an external image pull inside Minikube during the run, which is exposed to Docker Hub rate limits and network flakiness. The Spark image already has sh and is guaranteed to be present in the local daemon — using ${sparkImage} for the init container removes the external dependency entirely.

Dead code / docs that don't match behavior

• baseSparkConf sets spark.oidc.test.outputPath (commented "OidcS3ReadWriteJob parameters"), but no job reads it — the path is passed as argv(0). Can be dropped.
• resolveSparkHomeDir() lists spark.kubernetes.test.unpackSparkDir as its first candidate, but that property is never set: it isn't declared in this module's pom.xml (neither as a property nor in <systemProperties>), and the sbt KubernetesIntegrationTests.settings that would set it is not enabled for this project. Only spark.test.home actually resolves, so both the candidate and the error message's advice are misleading.
• dev-run-integration-tests.sh --spark-tgz passes -Dspark.kubernetes.test.sparkTgz, which nothing reads — it's a no-op, but it's documented as a real option in the README table.
• --java-version is parsed and documented but never used.

Minor

• kubernetes-version: "1.36.0" here vs "1.37.0" in the existing k8s job in the same file — two pinned versions to keep in sync.
• class OidcCredentialE2ESuite extends SparkFunSuite with BeforeAndAfterAll ... with Logging — SparkFunSuite already mixes in both via SparkTestSuite.
• spark.executor.instances=1 from baseSparkConf carries into the dynamic-allocation test. Harmless (initialExecutors = max(min, initial, instances) = 1) but it muddies the intent.
• moto[server,s3,sts]>=5.0.0 has no upper bound, so a moto release can break CI without any change here.
• dev-run-integration-tests.sh builds with build/sbt ... package and then runs build/mvn integration-test -am, which recompiles the dependency chain from scratch under Maven — the sbt build ends up wasted. Using one build tool for both steps would be simpler.
• -Poidc-e2e without -Pcredential-aws fails with an unresolved-dependency error. The docs consistently pass both, so this is low priority.

Items 1–3 are the ones I'd like to see addressed before merge, since without them the suite wouldn't actually catch regressions in the feature it covers.

[sarutak]

Thank you for the comments, @dongjoon-hyun . I've addressed all of them except the following one.

dev-run-integration-tests.sh builds with build/sbt ... package and then runs build/mvn integration-test -am, which recompiles the dependency chain from scratch under Maven — the sbt build ends up wasted. Using one build tool for both steps would be simpler.

I'd like to keep the current, verified-working layout for this PR and unify the build tool as a follow-up once I've confirmed the unified flow end-to-end on Minikube. The double build is a local-dev-only inefficiency and doesn't affect CI (which uses sbt throughout).

[dongjoon-hyun]

Thanks for the update, @sarutak. I confirmed all the items from the previous round are in the head commit (daily-build wiring, is-changed.py -m kubernetes,credential-aws, modules.py, per-test driver pod cleanup, afterAll isolation, Spark image for the init container, dead-code removal, moto upper bound, -Poidc-e2e listing connector/credential-aws). Maven de-duplicates the module listed in two active profiles, so -Pcredential-aws -Poidc-e2e together is fine.

A second pass surfaced a few more things. The first two affect what the suite actually proves, so I'd like to see them addressed before merge; the rest are robustness/docs.

Assertions

1. The basic test passes even when the driver ends in Failed.runOidcJobAndVerify relies on spark-submit's exit code plus a non-empty S3 listing, but LoggingPodStatusWatcherImpl.hasCompleted is true for both Succeeded and Failed and KubernetesClientApplication just break()s, so spark-submit exits 0 for a failed driver. A driver that writes the part files and then fails in the read-back phase (e.g. propagated credentials rejected on the read path) is reported as PASS. The comments at lines 442-445 / 483-485 ("a non-zero exit means the driver failed") describe behaviour Spark doesn't have, and OidcS3ReadWriteJob.SUCCESS_MARKER (whose comment says the suite looks for it) is never asserted. Calling awaitDriverSucceeded and/or asserting the marker, as tests 2 and 3 do, closes this.

2. The != "Failed" guards inside eventually never fail fast. ScalaTest's eventually retries TestFailedException, so the assert(driverPodPhase(...) != "Failed") at lines 272, 291, 371 and 385 just keeps retrying on a dead pod until the 3/2/3/10-minute timeouts expire. Only awaitDriverSucceeded really fails fast. Checking the phase outside the eventually (or via a small poll helper that fail()s on Failed before the log assertion) restores the intent.

Local run path

3. RBAC is only granted in CI. The suite comment says RBAC is granted "by the CI workflow / dev-run script", but the dev script and README contain no rolebinding; on a fresh RBAC-enabled Minikube the default SA can't create executor pods and every test times out after 10 minutes. Either create a namespaced RoleBinding in ensureNamespace() (which would also let CI drop the cluster-wide cluster-admin grant), or add the grant to the script and document it like resource-managers/kubernetes/integration-tests/dev/spark-rbac.yaml.

4. --image-tag <tag> --skip-build selects the image without the job jar. The build path tags the runnable image <tag>-job, but with --skip-buildSPARK_IMAGE stays empty and only -Dspark.kubernetes.test.imageTag=<tag> is forwarded, so the suite falls back to spark:<tag> and the driver dies with ClassNotFoundException (and, per #1, only after the 2-minute S3 poll). Deriving SPARK_IMAGE="${IMAGE_REPO}/spark:${IMAGE_TAG}-job" in the skip-build branch (or tagging the final image with the plain tag) and forwarding only spark.oidc.test.sparkImage fixes the README's documented re-run flow.

5. --deploy-mode / spark.kubernetes.test.deployMode is never read. It's declared in pom.xml, forwarded via systemProperties, passed by CI and documented, but nothing under src/ consumes it; the suite always uses the current kubeconfig. For the advertised non-minikube modes the script also sets MOTO_HOST=localhost, which inside a pod is the pod itself. I'd drop the option/property and state Minikube-only.

6. spark.kubernetes.test.master is neither normalized nor applied to the fabric8 client. It is passed verbatim as --master, so the README's "Kubernetes API server URL" form (https://...) is rejected by SparkSubmit, and a k8s://https://clusterA value while kubeconfig points at clusterB splits namespace/pod/exec handling and the submit across two clusters. KubeConfigBackend normalizes via Utils.checkAndGetK8sMasterUrl and calls config.setMasterUrl; alternatively drop the property and derive spark.master from kubernetesClient.getMasterUrl only, as KubernetesTestComponents does.

Cleanup / coverage

7. Namespace and pod deletes aren't awaited.afterAll and deleteDriverPod are fire-and-forget, unlike KubernetesTestComponents.deleteNamespace / KubernetesSuite.deleteDriverPod. With a fixed -Dspark.kubernetes.test.namespace, an immediate re-run finds the namespace still Terminating, treats it as pre-existing (createdNamespace=false), fails on submit, and never cleans it up; a failed test also lets the next one submit while the previous driver/executors are still terminating. Polling until get() == null matches the reference.

8. Rotation test launches outside try/finally.SparkAppLauncher.launch at line 264 runs before the try at 267 (test 3 has it inside), so a launch failure skips deleteDriverPod and Files.deleteIfExists(podTemplatePath).

9. The new module is never scalastyle-checked.dev/scalastyle's default SPARK_PROFILES includes -Pkubernetes-integration-tests and -Pdocker-integration-tests but not -Poidc-e2e, and the lint job runs ./dev/lint-scala with no arguments, so sbt never loads the project during linting. Appending -Poidc-e2e there closes the gap.

10. The SPARK-43540 justification is inverted.SPARK-43540added the working directory to the driver classpath, and SparkSubmit already puts a local:// primary jar on the driver classpath (childClasspath += localPrimaryResource) and into spark.jars for executors -- that's why kubernetes-integration-tests runs local:///opt/spark/examples/jars/..., outside /opt/spark/jars. The jar still has to be copied into the image (docker-image-tool.sh only copies examples/jars), so a copy step is needed, but the comments in the workflow, the script and jobJarResource should be corrected, and the second docker build could be folded into the standard image build.

Minor, no action needed: the SparkAppLauncher.scala comment says the k8s IT test-jar "could not be resolved as an inter-project reference under sbt"; sbt-pom-reader does map <type>test-jar</type> on in-reactor modules to test->test (that's how KubernetesSuite gets SparkFunSuite). The real cost is that every invocation would then need -Pkubernetes-integration-tests, which is a fair reason to keep the copy -- the comment could just say that.

[dongjoon-hyun]

+1, LGTM (Pending CIs). Thank you, @sarutak .

[uros-b]

Looks good!

[dongjoon-hyun]

Merge Summary:

• merged into master bcea2b2
• merged into branch-4.x 81d47f7

Posted by merge_spark_pr.py

[dongjoon-hyun]

There was a conflict on branch-4.3. If you need this in order to complete the SPIP, please make a backporting PR to branch-4.3, @sarutak .

[sarutak]

Thank you @dongjoon-hyun and @uros-b !

There was a conflict on branch-4.3. If you need this in order to complete the SPIP, please make a backporting PR to branch-4.3, @sarutak .

Since branch-4.3 was cut in early August (code freeze), I have decided to change the target version to 4.4. So it's not necessary to port this to branch-4.3.

[dongjoon-hyun]

Got it. Sounds good to me too because we can have more time to validate before the official announcement, @sarutak .