Core: Parallelize the determining of reachable manifests during file cleanup - #5981

Merged

rdblue merged 1 commit into

apache:masterfrom

amogh-jahagirdar:reachability-cleanup

Oct 19, 2022

amogh-jahagirdar commented Oct 14, 2022

Contributor

Follow up PR to #5669.

1.) Parallelizing determining the reachable manifests given a set of snapshots

2.) Cleaning up issues in log messages #5669 (comment) and #5669 (comment)

github-actionsBot added the core label

amogh-jahagirdar commented

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated

amogh-jahagirdar commented

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated

amogh-jahagirdar force-pushed the reachability-cleanup branch from f523fb8 to 2cdcd4fCompare

October 14, 2022 02:07

amogh-jahagirdar changed the title ~~Core: Parallelize the determining of reachable manifests during snapshot expiration file cleanp~~Core: Parallelize the determining of reachable manifests during file cleanup

rdblue reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated

rdblue reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated

amogh-jahagirdar force-pushed the reachability-cleanup branch 5 times, most recently from ed9e051 to b061989Compare

October 15, 2022 04:04

kbendick reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated

kbendick reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated

kbendick reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated

amogh-jahagirdar force-pushed the reachability-cleanup branch from b061989 to cc8f539Compare

October 16, 2022 16:49

amogh-jahagirdar requested review from kbendick and rdblue and removed request for kbendick and rdblue

October 16, 2022 16:57

amogh-jahagirdar force-pushed the reachability-cleanup branch 2 times, most recently from 10eca9a to 24a456aCompare

October 16, 2022 17:11

rdblue reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated

rdblue reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated

rdblue reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated

amogh-jahagirdar force-pushed the reachability-cleanup branch 2 times, most recently from 06cc886 to a205384Compare

October 17, 2022 01:55

Core: Parallelize the determining of reachable manifests and cleanup …

3846bf7

…log messages

amogh-jahagirdar force-pushed the reachability-cleanup branch from a205384 to 3846bf7Compare

October 17, 2022 04:16

amogh-jahagirdar requested a review from rdblue

October 17, 2022 05:41

jackye1995 approved these changes

View reviewed changes

jackye1995 left a comment

Contributor

looks good to me!

rdblue reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java

+ return;
+ }
+ currentManifestCallback.accept(manifestFile.copy());

rdblueOct 19, 2022

Contributor

Very minor, but it seems weird to copy here rather than in the callback. If the callback were a noop, we'd be doing work for nothing. Since we know it needs to be copied, it seems fine though.

rdblue approved these changes

View reviewed changes

rdblue merged commit 3ae58a1 into apache:master

rdblue commented Oct 19, 2022

Contributor

Thanks, @amogh-jahagirdar! Looks great.

amogh-jahagirdar commented Oct 19, 2022

ContributorAuthor

Thanks for the reviews @rdblue @kbendick @jackye1995 !

zhongyujiang pushed a commit to zhongyujiang/iceberg that referenced this pull request

[Cherry-pick] Core: Parallelize reading reachable manifests, improve …

d90e704

…logs (apache#5981)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment