Uh oh!
There was an error while loading. Please reload this page.
[SPARK-55115][INFRA][3.5] Use composable Dockerfile for release builds - #53908
Closed
cloud-fan wants to merge 2 commits into
Closed
[SPARK-55115][INFRA][3.5] Use composable Dockerfile for release builds#53908cloud-fan wants to merge 2 commits into
cloud-fan wants to merge 2 commits into
Conversation
JIRA Issue Information=== Improvement SPARK-55115 === This comment was automatically generated by GitHub Actions |
This changes the release Dockerfile to use a composable approach: - Dockerfile.base: Contains common tools shared across all Spark versions (Ubuntu 22.04, R packages, Ruby/bundler, TeX, Node.js) Does NOT include Java or Python - those are branch-specific. - Dockerfile: Branch-specific configuration that extends the base image (installs Java, Python, and version-specific pip packages) For branch-3.5: - Uses Java 8 - Uses Python 3.8 - Installs Sphinx 3.0.4 and related packages for documentation
cloud-fanforce-pushed
the
release-infra-3.5
branch
from
January 22, 2026 03:51
8dc9ffc to
dab43eeCompareThe original Dockerfile didn't set JAVA_HOME. The do-release.sh script has fallback logic that sets JAVA_HOME=/usr which works correctly. Setting JAVA_HOME in ENV at build time causes architecture mismatch issues.
HyukjinKwon
approved these changes
Jan 22, 2026
cloud-fan
commented
Jan 23, 2026
ContributorAuthor
thanks for review, merging to 3.5! |
cloud-fan added a commit
that referenced
this pull request
Jan 23, 2026
### What changes were proposed in this pull request? This PR refactors the release Docker image build process to use a composable Dockerfile approach: 1. **`Dockerfile.base`**: A shared base image containing common tools (Ubuntu 22.04, R packages, Ruby/bundler, TeX, Node.js) 2. **`Dockerfile`**: Branch-specific image that extends the base with Java/Python versions and packages for this branch 3. **`do-release-docker.sh`**: Updated to build the base image first, then the branch-specific image ### Why are the changes needed? Currently, each branch maintains its own full Dockerfile which leads to: - Duplicated common configuration across branches - Difficulty keeping base tools (R packages, Ruby, etc.) in sync - Expired GPG keys or outdated base images affecting all branches With the composable approach: - Common tools are defined once in `Dockerfile.base` - Each branch only specifies its unique Java/Python requirements - Updates to base tools can be applied consistently ### Version changes | Component | Before | After | |-----------|--------|-------| | Ubuntu | 20.04 | 22.04 (jammy-20250819) | | bundler | 2.3.8 | 2.4.22 | | R CRAN repo | focal-cran40 | jammy-cran40 | | docutils | <0.17 | ==0.16 | | FULL_REFRESH_DATE | (none) | 20250819 | **Note**: The Ubuntu upgrade from 20.04 to 22.04 is necessary to use a shared base image across all branches. Ubuntu 20.04 reached end of standard support in April 2025. ### Does this PR introduce _any_ user-facing change? No. This only affects the release infrastructure. ### How was this patch tested? Docker image built and verified successfully on remote machine. ### Was this patch authored or co-authored using generative AI tooling? Yes Closes#53908 from cloud-fan/release-infra-3.5. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR refactors the release Docker image build process to use a composable Dockerfile approach:
Dockerfile.base: A shared base image containing common tools (Ubuntu 22.04, R packages, Ruby/bundler, TeX, Node.js)Dockerfile: Branch-specific image that extends the base with Java/Python versions and packages for this branchdo-release-docker.sh: Updated to build the base image first, then the branch-specific imageWhy are the changes needed?
Currently, each branch maintains its own full Dockerfile which leads to:
With the composable approach:
Dockerfile.baseVersion changes
Note: The Ubuntu upgrade from 20.04 to 22.04 is necessary to use a shared base image across all branches. Ubuntu 20.04 reached end of standard support in April 2025.
Does this PR introduce any user-facing change?
No. This only affects the release infrastructure.
How was this patch tested?
Docker image built and verified successfully on remote machine.
Was this patch authored or co-authored using generative AI tooling?
Yes