Skip to content

Backport "HBASE-29003 Proper bulk load tracking (#6506) (#6921)" to branch-2.6 - #6927

Merged
rmdmattingly merged 1 commit into
branch-2.6from
HBASE-29003-branch-2.6
Apr 27, 2025
Merged

Backport "HBASE-29003 Proper bulk load tracking (#6506) (#6921)" to branch-2.6#6927
rmdmattingly merged 1 commit into
branch-2.6from
HBASE-29003-branch-2.6

Conversation

@rmdmattingly

@rmdmattinglyrmdmattingly commented Apr 23, 2025

Copy link
Copy Markdown
Contributor

No description provided.

@Apache-HBase

This comment has been minimized.

Sign in to view
@rmdmattingly@DieterDP-ngRay Mattingly
HBASE-29003Proper bulk load tracking (#6506) (#6921)
43afb37
Signed-off-by: Ray Mattingly <rmattingly@apache.org>
The HBase backup mechanism keeps track of which HFiles
were bulk loaded, so they can be included in incremental
backups.
Before this ticket, these bulk load records were only
deleted when an incremental backup is created. This
commit adds 2 more locations:
1) after a full backup. Since a full backup already
captures all data, this meant that unnecessary HFiles
were being included in the next incremental backup.
2) after a table delete/truncate/CF-deletion. Previously,
if an HFile was loaded before a table was cleared, the next
incremental backup would effectively still include the
HFile. This lead to incorrect data being restored.
This commit also completely refactors & simplifies the
test for this functionality.
Co-authored-by: DieterDP <90392398+DieterDP-ng@users.noreply.github.com>
@rmdmattingly
rmdmattinglyforce-pushed the HBASE-29003-branch-2.6 branch from f24a09d to 43afb37CompareApril 25, 2025 15:04
@Apache-HBase

Apache-HBase commented Apr 26, 2025

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 58sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗codespell0m 0scodespell was not available.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
_ branch-2.6 Compile Tests _
+0 🆗mvndep0m 58sMaven dependency ordering for branch
+1 💚mvninstall5m 18sbranch-2.6 passed
+1 💚compile4m 32sbranch-2.6 passed
+1 💚checkstyle0m 45sbranch-2.6 passed
+1 💚spotbugs2m 5sbranch-2.6 passed
+1 💚spotless0m 53sbranch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 🆗mvndep0m 12sMaven dependency ordering for patch
+1 💚mvninstall2m 59sthe patch passed
+1 💚compile3m 17sthe patch passed
+1 💚javac3m 17sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 46sthe patch passed
+1 💚spotbugs2m 12sthe patch passed
+1 💚hadoopcheck16m 21sPatch does not cause any errors with Hadoop 2.10.2 or 3.3.6 3.4.0.
+1 💚spotless0m 44spatch has no errors when running spotless:check.
_ Other Tests _
+1 💚asflicense0m 18sThe patch does not generate ASF License warnings.
44m 32s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#6927
Optional Testsdupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
unameLinux 63b4ce09bb00 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2.6 / 43afb37
Default JavaEclipse Adoptium-11.0.23+9
Max. process+thread count78 (vs. ulimit of 30000)
modulesC: hbase-server hbase-backup U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/console
versionsgit=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Apr 26, 2025

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 47sDocker mode activated.
-0 ⚠️yetus0m 6sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2.6 Compile Tests _
+0 🆗mvndep0m 10sMaven dependency ordering for branch
+1 💚mvninstall3m 13sbranch-2.6 passed
+1 💚compile1m 15sbranch-2.6 passed
+1 💚javadoc0m 41sbranch-2.6 passed
+1 💚shadedjars6m 29sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 🆗mvndep0m 20sMaven dependency ordering for patch
+1 💚mvninstall3m 54sthe patch passed
+1 💚compile1m 33sthe patch passed
+1 💚javac1m 33sthe patch passed
+1 💚javadoc0m 52sthe patch passed
+1 💚shadedjars7m 12spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit210m 48shbase-server in the patch passed.
+1 💚unit16m 6shbase-backup in the patch passed.
258m 43s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR#6927
Optional Testsjavac javadoc unit compile shadedjars
unameLinux 79934f8aa11f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2.6 / 43afb37
Default JavaEclipse Adoptium-17.0.11+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/testReport/
Max. process+thread count4395 (vs. ulimit of 30000)
modulesC: hbase-server hbase-backup U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Apr 26, 2025

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 52sDocker mode activated.
-0 ⚠️yetus0m 6sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2.6 Compile Tests _
+0 🆗mvndep0m 12sMaven dependency ordering for branch
+1 💚mvninstall3m 36sbranch-2.6 passed
+1 💚compile1m 25sbranch-2.6 passed
+1 💚javadoc0m 47sbranch-2.6 passed
+1 💚shadedjars6m 42sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 🆗mvndep0m 18sMaven dependency ordering for patch
+1 💚mvninstall3m 10sthe patch passed
+1 💚compile1m 13sthe patch passed
+1 💚javac1m 13sthe patch passed
+1 💚javadoc0m 40sthe patch passed
+1 💚shadedjars7m 16spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit218m 4shbase-server in the patch passed.
+1 💚unit15m 37shbase-backup in the patch passed.
264m 32s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#6927
Optional Testsjavac javadoc unit compile shadedjars
unameLinux e38c19ed8eec 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2.6 / 43afb37
Default JavaEclipse Adoptium-11.0.23+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/testReport/
Max. process+thread count4131 (vs. ulimit of 30000)
modulesC: hbase-server hbase-backup U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Apr 26, 2025

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 57sDocker mode activated.
-0 ⚠️yetus0m 8sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2.6 Compile Tests _
+0 🆗mvndep0m 21sMaven dependency ordering for branch
+1 💚mvninstall5m 56sbranch-2.6 passed
+1 💚compile2m 14sbranch-2.6 passed
+1 💚javadoc1m 31sbranch-2.6 passed
+1 💚shadedjars10m 6sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 🆗mvndep0m 23sMaven dependency ordering for patch
+1 💚mvninstall5m 38sthe patch passed
+1 💚compile2m 8sthe patch passed
+1 💚javac2m 8sthe patch passed
+1 💚javadoc1m 26sthe patch passed
+1 💚shadedjars8m 51spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit273m 52shbase-server in the patch passed.
+1 💚unit13m 36shbase-backup in the patch passed.
332m 31s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#6927
Optional Testsjavac javadoc unit compile shadedjars
unameLinux 10d1f29a68dc 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2.6 / 43afb37
Default JavaTemurin-1.8.0_412-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/testReport/
Max. process+thread count4111 (vs. ulimit of 30000)
modulesC: hbase-server hbase-backup U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6927/2/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@rmdmattingly
rmdmattingly merged commit 18a28f8 into branch-2.6Apr 27, 2025
@rmdmattingly
rmdmattingly deleted the HBASE-29003-branch-2.6 branch April 27, 2025 18:41
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rmdmattingly@Apache-HBase