Uh oh!
There was an error while loading. Please reload this page.
[SPARK-55115][INFRA] Use composable Dockerfile for release builds - #53905
Closed
cloud-fan wants to merge 3 commits into
Closed
[SPARK-55115][INFRA] Use composable Dockerfile for release builds#53905cloud-fan wants to merge 3 commits into
cloud-fan wants to merge 3 commits into
Conversation
JIRA Issue Information=== Improvement SPARK-55115 === This comment was automatically generated by GitHub Actions |
This was referenced Jan 22, 2026
cloud-fan
commented
Jan 22, 2026
Uh oh!
There was an error while loading. Please reload this page.
This changes the release Dockerfile to use a composable approach: - Dockerfile.base: Contains common tools shared across all Spark versions (Ubuntu 22.04, R packages, Ruby/bundler, TeX, Node.js) Does NOT include Java or Python - those are branch-specific. - Dockerfile: Branch-specific configuration that extends the base image (installs Java, Python, and version-specific pip packages) For master (4.2-SNAPSHOT): - Uses Java 17 - Uses Python 3.10 - Installs Sphinx 4.5.0, torch, and related packages for documentation
cloud-fanforce-pushed
the
release-infra-master
branch
from
January 22, 2026 03:51
bfc61b4 to
0336d34Comparecloud-fan
commented
Jan 22, 2026
Uh oh!
There was an error while loading. Please reload this page.
The original Dockerfile didn't set JAVA_HOME. The do-release.sh script has fallback logic that sets JAVA_HOME=/usr which works correctly. Setting JAVA_HOME in ENV at build time causes architecture mismatch issues.
ContributorAuthor
I want to use the master-branch base docker image for all active branches, as we can't really "freeze" the docker file in old branches. Things will go stale over time, and it's more stable to use the actively maintained docker file in the master branch. Every time I try to build release from an old branch, the docker environment is always broken. For example, building docker image with branch-3.5 fails now for various issues. I have PRs for all the active branches, and the base docker file is the same: I've done tests locally for all of them, for all the release steps (except for uploading). The produced artifacts are also verified. After these are merged, I'll update the github action release job to always use the master-branch base docker file: #53890 |
HyukjinKwon
approved these changes
Jan 23, 2026
cloud-fan
commented
Jan 23, 2026
ContributorAuthor
This is release infra change and CI failures is unrelated, thanks for review, merging to master! |
dongjoon-hyun added a commit
that referenced
this pull request
Jun 30, 2026
### What changes were proposed in this pull request? This PR uses `Python 3.11` instead of 3.10 in `dev/create-release/spark-rm/Dockerfile` for Apache Spark 4.3.0+. ### Why are the changes needed? Since `Python 3.10` reaches end-of-life in October 2026 before Apache Spark 4.3.0 release, Apache Spark 3.4.0 dropped it. - #55914 So, ideally, we need to use Python 3.11+ in the release script. However, recently, on 2026-01-23, we share `Dockerfile.base` for all maintenance releases and `ubuntu:jammy` has only `Python 3.11.0rc1`. Since `Python 3.11.0rc1` is better than the EOL `Python 3.10`, this PR upgrades it only for Spark release. - #53905https://github.com/apache/spark/blob/e28edaf0480e19abca83cd2b215450b85fdaf438/dev/create-release/spark-rm/Dockerfile.base#L30 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 Closes#56885 from dongjoon-hyun/SPARK-57771. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
dongjoon-hyun added a commit
that referenced
this pull request
Jun 30, 2026
### What changes were proposed in this pull request? This PR uses `Python 3.11` instead of 3.10 in `dev/create-release/spark-rm/Dockerfile` for Apache Spark 4.3.0+. ### Why are the changes needed? Since `Python 3.10` reaches end-of-life in October 2026 before Apache Spark 4.3.0 release, Apache Spark 3.4.0 dropped it. - #55914 So, ideally, we need to use Python 3.11+ in the release script. However, recently, on 2026-01-23, we share `Dockerfile.base` for all maintenance releases and `ubuntu:jammy` has only `Python 3.11.0rc1`. Since `Python 3.11.0rc1` is better than the EOL `Python 3.10`, this PR upgrades it only for Spark release. - #53905https://github.com/apache/spark/blob/e28edaf0480e19abca83cd2b215450b85fdaf438/dev/create-release/spark-rm/Dockerfile.base#L30 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 Closes#56885 from dongjoon-hyun/SPARK-57771. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit a57c537) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR refactors the release Docker image build process to use a composable Dockerfile approach:
Dockerfile.base: A shared base image containing common tools (Ubuntu 22.04, R packages, Ruby/bundler, TeX, Node.js)Dockerfile: Branch-specific image that extends the base with Java/Python versions and packages for this branchdo-release-docker.sh: Updated to build the base image first, then the branch-specific imageWhy are the changes needed?
Currently, each branch maintains its own full Dockerfile which leads to:
With the composable approach:
Dockerfile.baseVersion changes
This is a pure refactor. No package versions were changed.
Does this PR introduce any user-facing change?
No. This only affects the release infrastructure.
How was this patch tested?
Docker image built and verified successfully on remote machine.
Was this patch authored or co-authored using generative AI tooling?
Yes