Skip to content

HBASE-28189 Fix the miss count in one of CombinedBlockCache getBlock implementations - #5506

Merged
wchevreuil merged 2 commits into
apache:masterfrom
wchevreuil:HBASE-28189
Nov 9, 2023
Merged

HBASE-28189 Fix the miss count in one of CombinedBlockCache getBlock implementations#5506
wchevreuil merged 2 commits into
apache:masterfrom
wchevreuil:HBASE-28189

Conversation

@wchevreuil

@wchevreuilwchevreuil commented Nov 6, 2023

Copy link
Copy Markdown
Contributor

No description provided.

@wchevreuil
HBASE-28189 Fix the miss count in one of CombinedBlockCache getBlock …
0539610
…implementations
Change-Id: I9edff6fc2ad88128a4f695bd7477d2fca5bc28a8
@Apache-HBase

Apache-HBase commented Nov 6, 2023

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 27sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 43smaster passed
+1 💚compile0m 37smaster passed
+1 💚shadedjars5m 23sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 20sthe patch passed
+1 💚compile0m 37sthe patch passed
+1 💚javac0m 37sthe patch passed
+1 💚shadedjars5m 19spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 22sthe patch passed
_ Other Tests _
-1 ❌unit10m 50shbase-server in the patch failed.
30m 40s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5506
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 3d64daa66951 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 027a119
Default JavaTemurin-1.8.0_352-b08
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/1/testReport/
Max. process+thread count1792 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Nov 6, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 53sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 52smaster passed
+1 💚compile3m 3smaster passed
+1 💚checkstyle0m 46smaster passed
+1 💚spotless0m 50sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 45smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 24sthe patch passed
+1 💚compile2m 56sthe patch passed
+1 💚javac2m 56sthe patch passed
+1 💚checkstyle0m 45sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck10m 56sPatch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚spotless0m 48spatch has no errors when running spotless:check.
+1 💚spotbugs1m 50sthe patch passed
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
39m 19s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5506
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux ed93e7ab0081 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 027a119
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count78 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Nov 6, 2023

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 36sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 24smaster passed
+1 💚compile0m 58smaster passed
+1 💚shadedjars5m 39sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 28smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 14sthe patch passed
+1 💚compile0m 55sthe patch passed
+1 💚javac0m 55sthe patch passed
+1 💚shadedjars5m 55spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 30sthe patch passed
_ Other Tests _
-1 ❌unit16m 55shbase-server in the patch failed.
40m 13s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5506
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 7e7da0cd2ecf 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 027a119
Default JavaEclipse Adoptium-11.0.17+8
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/1/testReport/
Max. process+thread count1988 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@wchevreuil
addressing cases where block type in cache key differs from the actua…
cf7f64f
…l type of block
Change-Id: Ia3354a4801ae505963f7515b1324ecddf4dd97dc
@Apache-HBase

Apache-HBase commented Nov 7, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 12sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 54smaster passed
+1 💚compile2m 26smaster passed
+1 💚checkstyle0m 36smaster passed
+1 💚spotless0m 43sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 28smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 38sthe patch passed
+1 💚compile2m 24sthe patch passed
+1 💚javac2m 24sthe patch passed
+1 💚checkstyle0m 34sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck9m 16sPatch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚spotless0m 40spatch has no errors when running spotless:check.
+1 💚spotbugs1m 40sthe patch passed
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
31m 22s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5506
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux f7424ca248b0 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 027a119
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count78 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

petersomogyi
petersomogyi reviewed Nov 7, 2023
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java
boolean metaBlock = isMetaBlock(cacheKey.getBlockType());
if (metaBlock) {
if (!existInL1 && block != null) {
LOG.warn("Cache key {} had block type {}, but was found in L2 cache.", cacheKey,

@petersomogyipetersomogyiNov 7, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can this happen? Is this a bug?

@Apache9Apache9Nov 8, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is just a defensive logging? It should not happen normally. If we see this log, then there could be a bug?

@wchevreuilwchevreuilNov 8, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I decided to put this as I saw a UT doing the opposite (creating a meta block then wrongly passing block type data in the cachekey), so I wonder if this mistake might be happening anywhere in the "real" code. I tried to some review of all callees to this method and haven't found it, but I could had missed something.

@Apache-HBase

Apache-HBase commented Nov 7, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 31smaster passed
+1 💚compile0m 39smaster passed
+1 💚shadedjars4m 53sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 22sthe patch passed
+1 💚compile0m 41sthe patch passed
+1 💚javac0m 41sthe patch passed
+1 💚shadedjars4m 56spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24sthe patch passed
_ Other Tests _
+1 💚unit229m 11shbase-server in the patch passed.
250m 39s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5506
Optional Testsjavac javadoc unit shadedjars compile
unameLinux a8ef186fc88e 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 027a119
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/2/testReport/
Max. process+thread count4591 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Nov 7, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 34smaster passed
+1 💚compile0m 48smaster passed
+1 💚shadedjars4m 52sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 43sthe patch passed
+1 💚compile0m 48sthe patch passed
+1 💚javac0m 48sthe patch passed
+1 💚shadedjars4m 55spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26sthe patch passed
_ Other Tests _
+1 💚unit230m 13shbase-server in the patch passed.
252m 43s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5506
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 5737ab7e3d71 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 027a119
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/2/testReport/
Max. process+thread count4564 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5506/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@wchevreuil
wchevreuil merged commit 8821c6a into apache:masterNov 9, 2023
wchevreuil added a commit that referenced this pull request Nov 9, 2023
@wchevreuil
HBASE-28189Fix the miss count in one of CombinedBlockCache getBlock …
f00a176
…implementations (#5506)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
wchevreuil added a commit that referenced this pull request Nov 9, 2023
@wchevreuil
HBASE-28189Fix the miss count in one of CombinedBlockCache getBlock …
a5af15e
…implementations (#5506)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Change-Id: I472b6019c3fe744f7afc4219e47967a0d5823c6f
wchevreuil added a commit that referenced this pull request Nov 9, 2023
@wchevreuil
HBASE-28189Fix the miss count in one of CombinedBlockCache getBlock …
ce309a7
…implementations (#5506)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
wchevreuil added a commit that referenced this pull request Nov 9, 2023
@wchevreuil
HBASE-28189Fix the miss count in one of CombinedBlockCache getBlock …
43a1090
…implementations (#5506)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
@wchevreuil
CDPD-45890: HBASE-28189 Fix the miss count in one of CombinedBlockCac…
1bfabc0
…he getBlock implementations (apache#5506)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Change-Id: Icee78a9c839b3c4e4aac2176aa4091cdcd47a08b
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@petersomogyipetersomogyipetersomogyi approved these changes
@taklwutaklwuAwaiting requested review from taklwu
@Apache9Apache9Awaiting requested review from Apache9
@YutSeanYutSeanAwaiting requested review from YutSean

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@wchevreuil@Apache-HBase@petersomogyi@Apache9