Skip to content

HBASE-28460 Full backup restore failed on empty HFiles - #5782

Merged
bbeaudreault merged 2 commits into
apache:masterfrom
DieterDP-ng:HBase-28460_Restore-fails-on-empty-hfiles
Apr 2, 2024
Merged

HBASE-28460 Full backup restore failed on empty HFiles#5782
bbeaudreault merged 2 commits into
apache:masterfrom
DieterDP-ng:HBase-28460_Restore-fails-on-empty-hfiles

Conversation

@DieterDP-ng

@DieterDP-ngDieterDP-ng commented Mar 27, 2024

Copy link
Copy Markdown
Contributor

No description provided.

@Apache-HBase

Apache-HBase commented Mar 27, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 37sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 58smaster passed
+1 💚compile0m 27smaster passed
+1 💚checkstyle0m 12smaster passed
+1 💚spotless0m 45sbranch has no errors when running spotless:check.
+1 💚spotbugs0m 32smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 10sthe patch passed
+1 💚compile0m 29sthe patch passed
+1 💚javac0m 29sthe patch passed
+1 💚checkstyle0m 13sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck5m 56sPatch does not cause any errors with Hadoop 3.3.6.
+1 💚spotless0m 50spatch has no errors when running spotless:check.
+1 💚spotbugs0m 45sthe patch passed
_ Other Tests _
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
25m 42s
SubsystemReport/Notes
DockerClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5782
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 8cf984148b3f 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 298c550
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count79 (vs. ulimit of 30000)
modulesC: hbase-backup U: hbase-backup
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Mar 27, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 25sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 10smaster passed
+1 💚compile0m 16smaster passed
+1 💚shadedjars5m 35sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 13smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 54sthe patch passed
+1 💚compile0m 16sthe patch passed
+1 💚javac0m 16sthe patch passed
+1 💚shadedjars5m 34spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 12sthe patch passed
_ Other Tests _
+1 💚unit9m 48shbase-backup in the patch passed.
29m 20s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR#5782
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 12d25b1bf672 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 298c550
Default JavaEclipse Adoptium-17.0.10+7
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/testReport/
Max. process+thread count3877 (vs. ulimit of 30000)
modulesC: hbase-backup U: hbase-backup
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Mar 27, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 53sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 43smaster passed
+1 💚compile0m 21smaster passed
+1 💚shadedjars6m 13sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 17smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 16sthe patch passed
+1 💚compile0m 21sthe patch passed
+1 💚javac0m 21sthe patch passed
+1 💚shadedjars5m 29spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 13sthe patch passed
_ Other Tests _
+1 💚unit11m 37shbase-backup in the patch passed.
33m 34s
SubsystemReport/Notes
DockerClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5782
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 35f0111cf1c0 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 298c550
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/testReport/
Max. process+thread count3476 (vs. ulimit of 30000)
modulesC: hbase-backup U: hbase-backup
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Mar 27, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 13smaster passed
+1 💚compile0m 19smaster passed
+1 💚shadedjars7m 3sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 16smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 13sthe patch passed
+1 💚compile0m 20sthe patch passed
+1 💚javac0m 20sthe patch passed
+1 💚shadedjars6m 7spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 13sthe patch passed
_ Other Tests _
+1 💚unit12m 1shbase-backup in the patch passed.
34m 25s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5782
Optional Testsjavac javadoc unit shadedjars compile
unameLinux d3b46a6b2794 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 298c550
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/testReport/
Max. process+thread count3756 (vs. ulimit of 30000)
modulesC: hbase-backup U: hbase-backup
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

bbeaudreault
bbeaudreault reviewed Apr 1, 2024
View reviewed changes

@bbeaudreaultbbeaudreault left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While you are here, can you fix the LOG name in RestoreTool to match the classname? Right now it's set as BackupUtils which is misleading.

As a big aside, I wonder if we should simplify FULL restore to just use the existing region boundaries in the snapshot manifest

Comment threadhbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestFullRestore.java Outdated
Comment threadhbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceRestoreJob.java
throw new IOException("Can not restore from backup directory " + dirs
+ " (check Hadoop and HBase logs). Bulk loader returns null");
}
loader.bulkLoad(newTableNames[i], bulkOutputPath);

@bbeaudreaultbbeaudreaultApr 1, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a safe change? It solves for 1 case where empty hfiles could fail the restore, but I wonder if there are other cases where the bulkload might unexpectedly not restore any files.

@DieterDP-ngDieterDP-ngApr 2, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have enough knowledge about the internals to give an informed opinion I'm afraid. The test suite finds no problem with the change, but that's no guarantee of course.

But I think that if there would be cases where not loading anything in the bulkload indicated an error, it would have to be detected at other locations rather than here. Restoring no files is a valid usecase, and so the only thing I can think of is that there would be an issue when those HFiles are written, or that something happens to those files before they are restored.

  • The former should already be detected as an issue I'd hope.
  • The latter could be something that damages or deletes the HFiles. The only (valid) to detect that would be to track how many (or even which specific HFiles) should be loaded. So metadata that would be recorded when creating the backup, and verified when restoring. But - the assumption that this could happen, would have a lot more repercussions outside of backup&restore in my opinion.

@Apache-HBase

Apache-HBase commented Apr 2, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec3m 59sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 2smaster passed
+1 💚compile0m 26smaster passed
+1 💚checkstyle0m 11smaster passed
+1 💚spotless0m 44sbranch has no errors when running spotless:check.
+1 💚spotbugs0m 31smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 44sthe patch passed
+1 💚compile0m 24sthe patch passed
+1 💚javac0m 24sthe patch passed
+1 💚checkstyle0m 11sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck4m 54sPatch does not cause any errors with Hadoop 3.3.6.
+1 💚spotless0m 41spatch has no errors when running spotless:check.
+1 💚spotbugs0m 38sthe patch passed
_ Other Tests _
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
24m 42s
SubsystemReport/Notes
DockerClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5782
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 6634d1785539 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 2941d6e
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count79 (vs. ulimit of 30000)
modulesC: hbase-backup U: hbase-backup
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Apr 2, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 27sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 49smaster passed
+1 💚compile0m 16smaster passed
+1 💚shadedjars5m 41sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 12smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 30sthe patch passed
+1 💚compile0m 15sthe patch passed
+1 💚javac0m 15sthe patch passed
+1 💚shadedjars5m 38spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 11sthe patch passed
_ Other Tests _
+1 💚unit10m 36shbase-backup in the patch passed.
29m 32s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5782
Optional Testsjavac javadoc unit shadedjars compile
unameLinux d885ffc4bd10 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 2941d6e
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/testReport/
Max. process+thread count3535 (vs. ulimit of 30000)
modulesC: hbase-backup U: hbase-backup
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Apr 2, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec2m 25sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 54smaster passed
+1 💚compile0m 17smaster passed
+1 💚shadedjars5m 33sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 11smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 53sthe patch passed
+1 💚compile0m 15sthe patch passed
+1 💚javac0m 15sthe patch passed
+1 💚shadedjars5m 33spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 11sthe patch passed
_ Other Tests _
+1 💚unit9m 17shbase-backup in the patch passed.
30m 25s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR#5782
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 311353d117e7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 2941d6e
Default JavaEclipse Adoptium-17.0.10+7
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/testReport/
Max. process+thread count3819 (vs. ulimit of 30000)
modulesC: hbase-backup U: hbase-backup
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Apr 2, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec3m 33sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 3smaster passed
+1 💚compile0m 19smaster passed
+1 💚shadedjars5m 8sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 16smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 43sthe patch passed
+1 💚compile0m 19sthe patch passed
+1 💚javac0m 19sthe patch passed
+1 💚shadedjars5m 6spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 15sthe patch passed
_ Other Tests _
+1 💚unit10m 31shbase-backup in the patch passed.
31m 59s
SubsystemReport/Notes
DockerClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5782
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 1a1470ba9288 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 2941d6e
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/testReport/
Max. process+thread count3815 (vs. ulimit of 30000)
modulesC: hbase-backup U: hbase-backup
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@bbeaudreault
bbeaudreault merged commit 90639d7 into apache:masterApr 2, 2024
bbeaudreault pushed a commit that referenced this pull request Apr 2, 2024
@DieterDP-ng@bbeaudreault
HBASE-28460Full backup restore failed on empty HFiles (#5782)
2f05ee3
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
bbeaudreault pushed a commit that referenced this pull request Apr 2, 2024
@DieterDP-ng@bbeaudreault
HBASE-28460Full backup restore failed on empty HFiles (#5782)
691fb65
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
bbeaudreault pushed a commit that referenced this pull request Apr 2, 2024
@DieterDP-ng@bbeaudreault
HBASE-28460Full backup restore failed on empty HFiles (#5782)
e8d24fa
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
bbeaudreault pushed a commit to HubSpot/hbase that referenced this pull request Apr 2, 2024
@DieterDP-ng@bbeaudreault
HubSpot Backport: HBASE-28460 Full backup restore failed on empty HFi…
a15bdbd
…les (apache#5782)
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bbeaudreaultbbeaudreaultbbeaudreault approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@DieterDP-ng@Apache-HBase@bbeaudreault