Uh oh!
There was an error while loading. Please reload this page.
[SPARK-55115][INFRA][4.1] Use composable Dockerfile for release builds - #53906
Closed
cloud-fan wants to merge 3 commits into
Closed
[SPARK-55115][INFRA][4.1] Use composable Dockerfile for release builds#53906cloud-fan wants to merge 3 commits into
cloud-fan wants to merge 3 commits into
Conversation
JIRA Issue Information=== Improvement SPARK-55115 === This comment was automatically generated by GitHub Actions |
cloud-fan
commented
Jan 22, 2026
Uh oh!
There was an error while loading. Please reload this page.
This changes the release Dockerfile to use a composable approach: - Dockerfile.base: Contains common tools shared across all Spark versions (Ubuntu 22.04, R packages, Ruby/bundler, TeX, Node.js) Does NOT include Java or Python - those are branch-specific. - Dockerfile: Branch-specific configuration that extends the base image (installs Java, Python, and version-specific pip packages) For branch-4.1: - Uses Java 17 - Uses Python 3.10 - Installs Sphinx 4.5.0, torch, and related packages for documentation
cloud-fanforce-pushed
the
release-infra-4.1
branch
from
January 22, 2026 03:51
ca460d6 to
41d6f96Comparecloud-fan
commented
Jan 22, 2026
Uh oh!
There was an error while loading. Please reload this page.
The original Dockerfile didn't set JAVA_HOME. The do-release.sh script has fallback logic that sets JAVA_HOME=/usr which works correctly. Setting JAVA_HOME in ENV at build time causes architecture mismatch issues.
HyukjinKwon
approved these changes
Jan 22, 2026
ContributorAuthor
This is release infra change and CI failures is unrelated, thanks for review, merging to 4.1! |
cloud-fan added a commit
that referenced
this pull request
Jan 23, 2026
### What changes were proposed in this pull request? This PR refactors the release Docker image build process to use a composable Dockerfile approach: 1. **`Dockerfile.base`**: A shared base image containing common tools (Ubuntu 22.04, R packages, Ruby/bundler, TeX, Node.js) 2. **`Dockerfile`**: Branch-specific image that extends the base with Java/Python versions and packages for this branch 3. **`do-release-docker.sh`**: Updated to build the base image first, then the branch-specific image ### Why are the changes needed? Currently, each branch maintains its own full Dockerfile which leads to: - Duplicated common configuration across branches - Difficulty keeping base tools (R packages, Ruby, etc.) in sync - Expired GPG keys or outdated base images affecting all branches With the composable approach: - Common tools are defined once in `Dockerfile.base` - Each branch only specifies its unique Java/Python requirements - Updates to base tools can be applied consistently ### Version changes This is a pure refactor. No package versions were changed. ### Does this PR introduce _any_ user-facing change? No. This only affects the release infrastructure. ### How was this patch tested? Docker image built and verified successfully on remote machine. ### Was this patch authored or co-authored using generative AI tooling? Yes Closes#53906 from cloud-fan/release-infra-4.1. Lead-authored-by: Wenchen Fan <wenchen@databricks.com> Co-authored-by: Wenchen Fan <cloud0fan@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR refactors the release Docker image build process to use a composable Dockerfile approach:
Dockerfile.base: A shared base image containing common tools (Ubuntu 22.04, R packages, Ruby/bundler, TeX, Node.js)Dockerfile: Branch-specific image that extends the base with Java/Python versions and packages for this branchdo-release-docker.sh: Updated to build the base image first, then the branch-specific imageWhy are the changes needed?
Currently, each branch maintains its own full Dockerfile which leads to:
With the composable approach:
Dockerfile.baseVersion changes
This is a pure refactor. No package versions were changed.
Does this PR introduce any user-facing change?
No. This only affects the release infrastructure.
How was this patch tested?
Docker image built and verified successfully on remote machine.
Was this patch authored or co-authored using generative AI tooling?
Yes