[SPARK-58771][INFRA] Clean up the dependency list for classic-only docker image - #58006
Closed
gaogaotiantian wants to merge 1 commit into
Closed
[SPARK-58771][INFRA] Clean up the dependency list for classic-only docker image#58006gaogaotiantian wants to merge 1 commit into
gaogaotiantian wants to merge 1 commit into
Conversation
uros-b
approved these changes
Aug 14, 2026
Member
|
Thank you @gaogaotiantian! |
nchammas
approved these changes
Aug 16, 2026
nchammas
left a comment
Contributor
There was a problem hiding this comment.
Would it make sense to do the same cleanup to these other Dockerfiles?
Perhaps you're already planning to do that. Just wanted to check.
nchammas
reviewed
Aug 16, 2026
| RUN python3.12 -m venv $VIRTUAL_ENV | ||
| ENV PATH="$VIRTUAL_ENV/bin:$PATH" | ||
|
|
||
| ARG BASIC_PIP_PKGS="numpy pyarrow>=23.0.0 pandas==2.3.3 plotly<6.0.0 matplotlib openpyxl memory-profiler>=0.61.0 mlflow>=2.8.1 scipy scikit-learn>=1.3.2 pystack>=1.6.0 psutil" |
Contributor
There was a problem hiding this comment.
Just FYI, some of these pins are different with the new dependency group:
mlflowwent from>=2.8.1to>=2.3.1scikit-learnis no longer pinned
Contributor
Author
|
This cleanup will eventually be done to all the docker images. I'm doing it slowly because I don't want to break too many CIs at the same time. It's easier to revert/fix a simple change to a single docker file. Yes there are some minor diff for versions. For this specific case, it should not have any real impact because docker will just install the latest version. We kind of want to eliminate these inconsistency during the refactoring because some of them are definitely not intentional. One of the goals to move all the dependencies to a single file is to make it easier to manage and reduce the minor inconsistency between all the docker images. |
gaogaotiantian
added a commit
that referenced
this pull request
Aug 18, 2026
…cker image ### What changes were proposed in this pull request? * Use `pyproject.toml` as the dependency reference for classic-only * Remove the unnecessary pip bootstrap for the docker image ### Why are the changes needed? To clean up docker dependency list. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? This docker image is only used by scheduled CI so we need to wait for the scheduled CI. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #58006 from gaogaotiantian/docker-classic-only. Authored-by: Tian Gao <gaogaotiantian@hotmail.com> Signed-off-by: Tian Gao <gaogaotiantian@hotmail.com> (cherry picked from commit ff26d01) Signed-off-by: Tian Gao <gaogaotiantian@hotmail.com>
Contributor
Author
gaogaotiantian
added a commit
that referenced
this pull request
Aug 18, 2026
### What changes were proposed in this pull request? Add `build-contexts` for classic-only docker image build. ### Why are the changes needed? It was missing from #58006 and is breaking our CI. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI should be restored. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #58072 from gaogaotiantian/fix-classic-docker-build. Authored-by: Tian Gao <gaogaotiantian@hotmail.com> Signed-off-by: Tian Gao <gaogaotiantian@hotmail.com>
gaogaotiantian
added a commit
that referenced
this pull request
Aug 18, 2026
### What changes were proposed in this pull request? Add `build-contexts` for classic-only docker image build. ### Why are the changes needed? It was missing from #58006 and is breaking our CI. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI should be restored. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #58072 from gaogaotiantian/fix-classic-docker-build. Authored-by: Tian Gao <gaogaotiantian@hotmail.com> Signed-off-by: Tian Gao <gaogaotiantian@hotmail.com> (cherry picked from commit 0790be9) Signed-off-by: Tian Gao <gaogaotiantian@hotmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
pyproject.tomlas the dependency reference for classic-onlyWhy are the changes needed?
To clean up docker dependency list.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
This docker image is only used by scheduled CI so we need to wait for the scheduled CI.
Was this patch authored or co-authored using generative AI tooling?
No.