Uh oh!
There was an error while loading. Please reload this page.
[SPARK-50603][SQL] Respect user-provided basePath for streaming file source reads without glob - #51267
Open
Kimahriman wants to merge 1 commit into
Open
[SPARK-50603][SQL] Respect user-provided basePath for streaming file source reads without glob#51267Kimahriman wants to merge 1 commit into
Kimahriman wants to merge 1 commit into
Conversation
Kimahriman
commented
Jun 24, 2025
ContributorAuthor
Re-opening from #49220 which was approved but never merged |
Kimahrimanforce-pushed
the
respect-base-path
branch
from
August 15, 2025 11:21
8aa9799 to
f3a46d1CompareKimahrimanforce-pushed
the
respect-base-path
branch
2 times, most recently
from
November 5, 2025 14:42
8f7a1df to
1496ab1CompareKimahrimanforce-pushed
the
respect-base-path
branch
2 times, most recently
from
November 28, 2025 18:58
2832612 to
1ce12e3Comparezhixingheyi-tian
approved these changes
Jan 19, 2026
Kimahrimanforce-pushed
the
respect-base-path
branch
from
February 3, 2026 15:56
1ce12e3 to
34d6edbCompareKimahrimanforce-pushed
the
respect-base-path
branch
from
March 11, 2026 15:15
34d6edb to
7223924CompareKimahrimanforce-pushed
the
respect-base-path
branch
from
May 11, 2026 21:05
7223924 to
f01364aCompareKimahrimanforce-pushed
the
respect-base-path
branch
from
July 7, 2026 17:22
f01364a to
a8cf36eCompareKimahrimanforce-pushed
the
respect-base-path
branch
from
August 21, 2026 18:12
a8cf36e to
a65145fCompare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Don't override the user-provided
basePathfor streaming file source reads without globbing.Why are the changes needed?
#14803 fixed an issue reading partition values for streaming file source reads without globbing, but it always overwrites the
basePathoption. If the user provides a custombasePathin this case, there can be a mismatch in partition schema from when the stream is loaded to when a batch is executed, causing errors.Does this PR introduce any user-facing change?
Bug fix.
How was this patch tested?
New UT
Was this patch authored or co-authored using generative AI tooling?
No