Skip to content

Forward-port "HBASE-28065 Corrupt HFile data is mishandled in several cases" to branch-3 - #5418

Merged
ndimiduk merged 2 commits into
apache:branch-3from
ndimiduk:28065-hfile-corrupt-reads-branch-3
Sep 21, 2023
Merged

Forward-port "HBASE-28065 Corrupt HFile data is mishandled in several cases" to branch-3#5418
ndimiduk merged 2 commits into
apache:branch-3from
ndimiduk:28065-hfile-corrupt-reads-branch-3

Conversation

@ndimiduk

@ndimidukndimiduk commented Sep 20, 2023

Copy link
Copy Markdown
Member

No description provided.

@ndimiduk
HBASE-28065 Corrupt HFile data is mishandled in several cases
5c58af1
* when no block size is provided and there's not a preread headerBuf, treat the value with
caution.
* verify HBase checksums before making use of the block header.
* inline verifyOnDiskSizeMatchesHeader to keep throw/return logic in the method body.
* separate validation of onDiskSizeWithHeader as input parameter from as read from block header
* simplify branching around fetching and populating onDiskSizeWithHeader.
* inline retrieving nextOnDiskBlockSize ; add basic validation.
* whenever a read is determined to be corrupt and fallback to HDFS checksum is necessary, also
invalidate the cached value of headerBuf.
* build out a test suite covering various forms of block header corruption, for blocks in first
and second positions.
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
@ndimiduk
adjustment for master branch
4cc944f
@ndimidukndimiduk added the backport This PR is a back port of some issue or issues already committed to master label Sep 20, 2023
@Apache-HBase

Apache-HBase commented Sep 20, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 38sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ branch-3 Compile Tests _
+1 💚mvninstall3m 48sbranch-3 passed
+1 💚compile3m 3sbranch-3 passed
+1 💚checkstyle0m 43sbranch-3 passed
+1 💚spotless0m 58sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 46sbranch-3 passed
_ Patch Compile Tests _
+1 💚mvninstall3m 13sthe patch passed
+1 💚compile2m 46sthe patch passed
+1 💚javac2m 46sthe patch passed
-0 ⚠️checkstyle0m 39shbase-server: The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck10m 19sPatch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚spotless0m 40spatch has no errors when running spotless:check.
+1 💚spotbugs1m 33sthe patch passed
_ Other Tests _
+1 💚asflicense0m 10sThe patch does not generate ASF License warnings.
36m 31s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5418
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 10e8bb68ec8c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-3 / 6cac6c2
Default JavaEclipse Adoptium-11.0.17+8
checkstylehttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count79 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 20, 2023

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 20sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-3 Compile Tests _
+1 💚mvninstall3m 47sbranch-3 passed
+1 💚compile1m 3sbranch-3 passed
+1 💚shadedjars7m 14sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 36sbranch-3 passed
_ Patch Compile Tests _
+1 💚mvninstall3m 50sthe patch passed
+1 💚compile1m 5sthe patch passed
+1 💚javac1m 5sthe patch passed
+1 💚shadedjars6m 10spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 38sthe patch passed
_ Other Tests _
-1 ❌unit271m 47shbase-server in the patch failed.
301m 33s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5418
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 8ca36b7d1cb8 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-3 / 6cac6c2
Default JavaEclipse Adoptium-11.0.17+8
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/testReport/
Max. process+thread count4856 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 20, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 43sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-3 Compile Tests _
+1 💚mvninstall3m 41sbranch-3 passed
+1 💚compile1m 1sbranch-3 passed
+1 💚shadedjars7m 3sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 37sbranch-3 passed
_ Patch Compile Tests _
+1 💚mvninstall3m 11sthe patch passed
+1 💚compile1m 0sthe patch passed
+1 💚javac1m 0sthe patch passed
+1 💚shadedjars7m 13spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 38sthe patch passed
_ Other Tests _
+1 💚unit304m 33shbase-server in the patch passed.
334m 37s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5418
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 132c45082282 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-3 / 6cac6c2
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/testReport/
Max. process+thread count4472 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@ndimiduk
ndimiduk merged commit a4b8c6d into apache:branch-3Sep 21, 2023
@ndimiduk
ndimiduk deleted the 28065-hfile-corrupt-reads-branch-3 branch September 21, 2023 10:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backportThis PR is a back port of some issue or issues already committed to master

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ndimiduk@Apache-HBase