Uh oh!
There was an error while loading. Please reload this page.
[SPARK-43281][SQL] Fix concurrent writer does not update file metrics - #40952
Closed
ulysses-you wants to merge 2 commits into
Closed
[SPARK-43281][SQL] Fix concurrent writer does not update file metrics#40952ulysses-you wants to merge 2 commits into
ulysses-you wants to merge 2 commits into
Conversation
ulysses-you
commented
Apr 26, 2023
ContributorAuthor
cc @cloud-fan |
cloud-fan
approved these changes
Apr 26, 2023
cloud-fan
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Is this a long-standing bug?
ulysses-you
commented
Apr 26, 2023
ContributorAuthor
@cloud-fan , it happened since #32198 and with concurrent writer on. |
ulysses-you
commented
Apr 26, 2023
| val missing = new Path(tempDirPath, "missing") | ||
| val tracker = new BasicWriteTaskStatsTracker(conf) | ||
| tracker.newFile(missing.toString) | ||
| tracker.closeFile(missing.toString) |
ContributorAuthor
There was a problem hiding this comment.
after refactor of #32198, one newFile should have one closeFile
ulysses-you
commented
Apr 28, 2023
ContributorAuthor
@cloud-fan any comments ? |
dongjoon-hyun
commented
May 12, 2023
Member
cc @mridulm , too. |
cloud-fan
commented
May 16, 2023
Contributor
thanks, merging to master/3.4! |
cloud-fan pushed a commit
that referenced
this pull request
May 16, 2023
### What changes were proposed in this pull request? `DynamicPartitionDataConcurrentWriter` it uses temp file path to get file status after commit task. However, the temp file has already moved to new path during commit task. This pr calls `closeFile` before commit task. ### Why are the changes needed? fix bug ### Does this PR introduce _any_ user-facing change? yes, after this pr the metrics is correct ### How was this patch tested? add test Closes#40952 from ulysses-you/SPARK-43281. Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 592e922) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
snmvaughan pushed a commit
to snmvaughan/spark
that referenced
this pull request
Jun 20, 2023
### What changes were proposed in this pull request? `DynamicPartitionDataConcurrentWriter` it uses temp file path to get file status after commit task. However, the temp file has already moved to new path during commit task. This pr calls `closeFile` before commit task. ### Why are the changes needed? fix bug ### Does this PR introduce _any_ user-facing change? yes, after this pr the metrics is correct ### How was this patch tested? add test Closesapache#40952 from ulysses-you/SPARK-43281. Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 592e922) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
GladwinLee pushed a commit
to lyft/spark
that referenced
this pull request
Oct 10, 2023
### What changes were proposed in this pull request? `DynamicPartitionDataConcurrentWriter` it uses temp file path to get file status after commit task. However, the temp file has already moved to new path during commit task. This pr calls `closeFile` before commit task. ### Why are the changes needed? fix bug ### Does this PR introduce _any_ user-facing change? yes, after this pr the metrics is correct ### How was this patch tested? add test Closesapache#40952 from ulysses-you/SPARK-43281. Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 592e922) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
catalinii pushed a commit
to lyft/spark
that referenced
this pull request
Oct 10, 2023
### What changes were proposed in this pull request? `DynamicPartitionDataConcurrentWriter` it uses temp file path to get file status after commit task. However, the temp file has already moved to new path during commit task. This pr calls `closeFile` before commit task. ### Why are the changes needed? fix bug ### Does this PR introduce _any_ user-facing change? yes, after this pr the metrics is correct ### How was this patch tested? add test Closesapache#40952 from ulysses-you/SPARK-43281. Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 592e922) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
DynamicPartitionDataConcurrentWriterit uses temp file path to get file status after commit task. However, the temp file has already moved to new path during commit task.This pr calls
closeFilebefore commit task.Why are the changes needed?
fix bug
Does this PR introduce any user-facing change?
yes, after this pr the metrics is correct
How was this patch tested?
add test