Skip to content

HBASE-27940 Midkey metadata in root index block would always be ignor… - #5296

Merged
comnetwork merged 3 commits into
apache:masterfrom
comnetwork:midkey
Jun 17, 2023
Merged

HBASE-27940 Midkey metadata in root index block would always be ignor…#5296
comnetwork merged 3 commits into
apache:masterfrom
comnetwork:midkey

Conversation

@comnetwork

@comnetworkcomnetwork commented Jun 17, 2023

Copy link
Copy Markdown
Contributor

…ed by BlockIndexReader.readMultiLevelIndexRoot

@Apache-HBase

Apache-HBase commented Jun 17, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 25sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 50smaster passed
+1 💚compile2m 23smaster passed
+1 💚checkstyle0m 31smaster passed
+1 💚spotless0m 40sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 24smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 27sthe patch passed
+1 💚compile2m 23sthe patch passed
+1 💚javac2m 23sthe patch passed
+1 💚checkstyle0m 33sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck9m 30sPatch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚spotless0m 39spatch has no errors when running spotless:check.
+1 💚spotbugs1m 30sthe patch passed
_ Other Tests _
+1 💚asflicense0m 9sThe patch does not generate ASF License warnings.
31m 24s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5296
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux b35b1447e437 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 622f4ae
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count79 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9
Apache9 approved these changes Jun 17, 2023
View reviewed changes

@Apache9Apache9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch.

So we do not need to store the checksum bytes in HFileBlock right? We only need to verify it while reading the data from the file system?

@Apache-HBase

Apache-HBase commented Jun 17, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 46sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 30smaster passed
+1 💚compile0m 39smaster passed
+1 💚shadedjars4m 29sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 17sthe patch passed
+1 💚compile0m 40sthe patch passed
+1 💚javac0m 40sthe patch passed
+1 💚shadedjars4m 30spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23sthe patch passed
_ Other Tests _
+1 💚unit223m 25shbase-server in the patch passed.
244m 48s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5296
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 2381e3046c12 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 622f4ae
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/1/testReport/
Max. process+thread count4134 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jun 17, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 51sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 51smaster passed
+1 💚compile0m 47smaster passed
+1 💚shadedjars4m 44sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 27smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 38sthe patch passed
+1 💚compile0m 47sthe patch passed
+1 💚javac0m 47sthe patch passed
+1 💚shadedjars4m 41spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26sthe patch passed
_ Other Tests _
+1 💚unit224m 26shbase-server in the patch passed.
247m 17s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5296
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 65bf7f7af2e6 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 622f4ae
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/1/testReport/
Max. process+thread count4791 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jun 17, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 24sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 48smaster passed
+1 💚compile2m 23smaster passed
+1 💚checkstyle0m 32smaster passed
+1 💚spotless0m 41sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 23smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 28sthe patch passed
+1 💚compile2m 23sthe patch passed
+1 💚javac2m 23sthe patch passed
+1 💚checkstyle0m 31sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck9m 21sPatch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚spotless0m 38spatch has no errors when running spotless:check.
+1 💚spotbugs1m 28sthe patch passed
_ Other Tests _
+1 💚asflicense0m 10sThe patch does not generate ASF License warnings.
31m 5s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5296
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 9c7189c77795 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / b9a244c
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count77 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@comnetwork

comnetwork commented Jun 17, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@Apache9 , thanks for review, yes , for HBASE-27053 , in FSReaderImpl.readBlockDataInternal, checksum is removed from the the curBlock ByteBuff once the checksum is verified, and it create HFileBlock by the curBlock ByteBuff
which checksum is removed.

 // remove checksum from buffer now that it's verified
int sizeWithoutChecksum = curBlock.getInt(Header.ON_DISK_DATA_SIZE_WITH_HEADER_INDEX);
curBlock.limit(sizeWithoutChecksum);
long duration = EnvironmentEdgeManager.currentTime() - startTime;
if (updateMetrics) {
HFile.updateReadLatency(duration, pread);
}
HFileBlock hFileBlock = createFromBuff(curBlock, checksumSupport, offset,
nextBlockOnDiskSize, fileContext, intoHeap ? HEAP : allocator);

@comnetwork
comnetwork merged commit cd3f94d into apache:masterJun 17, 2023
asfgit pushed a commit that referenced this pull request Jun 17, 2023
@comnetwork
HBASE-27940Midkey metadata in root index block would always be ignor… (
8f3cb6e
#5296)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
@Apache-HBase

Apache-HBase commented Jun 17, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 53sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 35smaster passed
+1 💚compile0m 48smaster passed
+1 💚shadedjars4m 40sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 39sthe patch passed
+1 💚compile0m 48sthe patch passed
+1 💚javac0m 48sthe patch passed
+1 💚shadedjars4m 43spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26sthe patch passed
_ Other Tests _
+1 💚unit246m 49shbase-server in the patch passed.
269m 21s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5296
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 7ee03f52c2aa 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / b9a244c
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/2/testReport/
Max. process+thread count4708 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jun 17, 2023

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 47sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 11smaster passed
+1 💚compile0m 40smaster passed
+1 💚shadedjars4m 29sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 12sthe patch passed
+1 💚compile0m 40sthe patch passed
+1 💚javac0m 40sthe patch passed
+1 💚shadedjars4m 29spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 22sthe patch passed
_ Other Tests _
-1 ❌unit256m 58shbase-server in the patch failed.
277m 58s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5296
Optional Testsjavac javadoc unit shadedjars compile
unameLinux abe8eac6d204 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / b9a244c
Default JavaTemurin-1.8.0_352-b08
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/2/testReport/
Max. process+thread count4176 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5296/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Apache9Apache9Apache9 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@comnetwork@Apache-HBase@Apache9