[SPARK-59280][BUILD] Expand and group sparktestsupport ignore patterns - #58555
Open
nchammas wants to merge 4 commits into
Open
[SPARK-59280][BUILD] Expand and group sparktestsupport ignore patterns#58555nchammas wants to merge 4 commits into
sparktestsupport ignore patterns#58555nchammas wants to merge 4 commits into
Conversation
uros-b
reviewed
Sep 10, 2026
uros-b
left a comment
Member
There was a problem hiding this comment.
This looks good to me, thank you @nchammas! Let's please ping someone with a bit more context to review this too, e.g. @zhengruifeng and/or @HyukjinKwon
| "/.github/workflows/build_codegen_jdk.yml", | ||
| "/.github/workflows/build_coverage.yml", | ||
| "/.github/workflows/build_java*.yml", | ||
| "/.github/workflows/build_main.yml", |
Contributor
There was a problem hiding this comment.
Not sure if we should skip these (and the docker file). If our testing environment is changed, we should run a full test against the new environment right?
Contributor
There was a problem hiding this comment.
yes, basically a infra change triggers a full test
| "/sbin/decommission-worker.sh", | ||
| "/sbin/spark-daemons.sh", | ||
| "/sbin/start-all.sh", | ||
| "/sbin/start-connect-server.sh", |
Contributor
There was a problem hiding this comment.
If I remember correctly, at least some of these shell scripts are directly used by some tests (local_server_pool or something) and can't be ignored.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
sparktestsupportto cover almost all cases where we do not want any test modules to fire.Why are the changes needed?
master. I found that about 4% of them triggered a full CI run that they could have completely skipped.Here are 10 examples, including the CI time consumed: #55685 (30h), #58072 (28h), #55800 (28h), #55878 (27h), #57972 (27h), #56996 (27h), #57741 (27h), #58206 (27h), #58098 (27h), #55972 (27h)
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing doctests and CI.
Was this patch authored or co-authored using generative AI tooling?
I wrote this patch and the associated analysis scripts (not posted; happy to share if interested) with assistance from GitHub Copilot.