Skip to content

[v3-3-test] Speed up lang-SDK k8s test by building Go/Java natively in CI (#69411) - #69580

Merged
jason810496 merged 1 commit into
apache:v3-3-testfrom
jason810496:backport-673183b-v3-3-test
Jul 8, 2026
Merged

[v3-3-test] Speed up lang-SDK k8s test by building Go/Java natively in CI (#69411)#69580
jason810496 merged 1 commit into
apache:v3-3-testfrom
jason810496:backport-673183b-v3-3-test

Conversation

@jason810496

Copy link
Copy Markdown
Member
  • Speed up lang-SDK k8s test by building Go/Java natively in CI

The Multi-Lang KubernetesExecutor system test builds a Go bundle and a Java jar inside throwaway toolchain containers. On a dev host the module, Gradle-distribution and dependency caches persist between runs, but CI runners are ephemeral and nothing cached them, so every scheduled run re-pulled the golang/temurin images and re-downloaded the Gradle distribution and all dependencies — adding roughly ten minutes to each of the six KubernetesExecutor variants that run the test.

Build the artifacts with the host toolchain in CI instead, provisioned and cached through actions/setup-go and actions/setup-java, so the toolchain image pulls and cold dependency downloads no longer happen on every run. Local runs keep the containerised build so a dev host still needs neither Go nor a JDK installed.

  • Centralize lang-SDK JDK version and add a bustable cache key in CI

Follow-ups to the native lang-SDK build:

  • Move the JDK version (17) into a JAVA_SDK_VERSION breeze constant and surface it as the java-sdk-version selective-checks / build-info output, so the k8s workflow reads it instead of hardcoding the version in YAML.
  • Restore the Go module/build and Gradle caches with an explicit actions/cache keyed on a "-v1-" salt rather than the setup-* built-in caching, giving an in-repo knob to force-invalidate a poisoned cache (bump the salt) without waiting for a dependency change to rotate it. (cherry picked from commit 673183b)

…n CI (apache#69411)
* Speed up lang-SDK k8s test by building Go/Java natively in CI
The Multi-Lang KubernetesExecutor system test builds a Go bundle and a
Java jar inside throwaway toolchain containers. On a dev host the module,
Gradle-distribution and dependency caches persist between runs, but CI
runners are ephemeral and nothing cached them, so every scheduled run
re-pulled the golang/temurin images and re-downloaded the Gradle
distribution and all dependencies — adding roughly ten minutes to each of
the six KubernetesExecutor variants that run the test.
Build the artifacts with the host toolchain in CI instead, provisioned and
cached through actions/setup-go and actions/setup-java, so the toolchain
image pulls and cold dependency downloads no longer happen on every run.
Local runs keep the containerised build so a dev host still needs neither
Go nor a JDK installed.
* Centralize lang-SDK JDK version and add a bustable cache key in CI
Follow-ups to the native lang-SDK build:
- Move the JDK version (17) into a JAVA_SDK_VERSION breeze constant and
surface it as the java-sdk-version selective-checks / build-info output,
so the k8s workflow reads it instead of hardcoding the version in YAML.
- Restore the Go module/build and Gradle caches with an explicit
actions/cache keyed on a "-v1-" salt rather than the setup-* built-in
caching, giving an in-repo knob to force-invalidate a poisoned cache
(bump the salt) without waiting for a dependency change to rotate it.
(cherry picked from commit 673183b)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
@jason810496
jason810496 merged commit b408922 into apache:v3-3-testJul 8, 2026
117 checks passed
vatsrahul1001 pushed a commit that referenced this pull request Jul 9, 2026
…n CI (#69411) (#69580)
* Speed up lang-SDK k8s test by building Go/Java natively in CI
The Multi-Lang KubernetesExecutor system test builds a Go bundle and a
Java jar inside throwaway toolchain containers. On a dev host the module,
Gradle-distribution and dependency caches persist between runs, but CI
runners are ephemeral and nothing cached them, so every scheduled run
re-pulled the golang/temurin images and re-downloaded the Gradle
distribution and all dependencies — adding roughly ten minutes to each of
the six KubernetesExecutor variants that run the test.
Build the artifacts with the host toolchain in CI instead, provisioned and
cached through actions/setup-go and actions/setup-java, so the toolchain
image pulls and cold dependency downloads no longer happen on every run.
Local runs keep the containerised build so a dev host still needs neither
Go nor a JDK installed.
* Centralize lang-SDK JDK version and add a bustable cache key in CI
Follow-ups to the native lang-SDK build:
- Move the JDK version (17) into a JAVA_SDK_VERSION breeze constant and
surface it as the java-sdk-version selective-checks / build-info output,
so the k8s workflow reads it instead of hardcoding the version in YAML.
- Restore the Go module/build and Gradle caches with an explicit
actions/cache keyed on a "-v1-" salt rather than the setup-* built-in
caching, giving an in-repo knob to force-invalidate a poisoned cache
(bump the salt) without waiting for a dependency change to rotate it.
(cherry picked from commit 673183b)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
@vatsrahul1001vatsrahul1001 added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jul 27, 2026
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.3.1 milestone Jul 27, 2026
vatsrahul1001 pushed a commit that referenced this pull request Aug 5, 2026
…n CI (#69411) (#69580)
* Speed up lang-SDK k8s test by building Go/Java natively in CI
The Multi-Lang KubernetesExecutor system test builds a Go bundle and a
Java jar inside throwaway toolchain containers. On a dev host the module,
Gradle-distribution and dependency caches persist between runs, but CI
runners are ephemeral and nothing cached them, so every scheduled run
re-pulled the golang/temurin images and re-downloaded the Gradle
distribution and all dependencies — adding roughly ten minutes to each of
the six KubernetesExecutor variants that run the test.
Build the artifacts with the host toolchain in CI instead, provisioned and
cached through actions/setup-go and actions/setup-java, so the toolchain
image pulls and cold dependency downloads no longer happen on every run.
Local runs keep the containerised build so a dev host still needs neither
Go nor a JDK installed.
* Centralize lang-SDK JDK version and add a bustable cache key in CI
Follow-ups to the native lang-SDK build:
- Move the JDK version (17) into a JAVA_SDK_VERSION breeze constant and
surface it as the java-sdk-version selective-checks / build-info output,
so the k8s workflow reads it instead of hardcoding the version in YAML.
- Restore the Go module/build and Gradle caches with an explicit
actions/cache keyed on a "-v1-" salt rather than the setup-* built-in
caching, giving an in-repo knob to force-invalidate a poisoned cache
(bump the salt) without waiting for a dependency change to rotate it.
(cherry picked from commit 673183b)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-toolsarea:kubernetes-testschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@jason810496@Lee-W@guan404ming@vatsrahul1001@potiuk