Skip to content

HBASE-28011 : The logStats about LruBlockCache is not accurate - #5344

Merged
virajjasani merged 2 commits into
apache:masterfrom
guluo2016:hbase_lru
Aug 9, 2023
Merged

HBASE-28011 : The logStats about LruBlockCache is not accurate#5344
virajjasani merged 2 commits into
apache:masterfrom
guluo2016:hbase_lru

Conversation

@guluo2016

@guluo2016guluo2016 commented Aug 7, 2023•
edited
Loading

Copy link
Copy Markdown
Member

Details see: HBASE-28011

Apply this commit, LruBlockCache.logStats would print info,as follow:

INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.20 GB, usedSize=2.41 MB, freeSize=3.20 GB, max=3.20 GB, blockCount=4, accesses=13, hits=9, hitRatio=69.23%, , cachingAccesses=13, cachingHits=9, cachingHitsRatio=69.23%, evictions=29, evicted=0, evictedPerRun=0.0

@Apache-HBase

Apache-HBase commented Aug 7, 2023

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 26sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 34smaster passed
+1 💚compile0m 35smaster passed
+1 💚shadedjars4m 57sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 11sthe patch passed
+1 💚compile0m 36sthe patch passed
+1 💚javac0m 36sthe patch passed
+1 💚shadedjars4m 53spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 21sthe patch passed
_ Other Tests _
-1 ❌unit10m 3shbase-server in the patch failed.
28m 52s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5344
JIRA IssueHBASE-28011
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 02ad79bdc704 5.4.0-1101-aws #109~18.04.1-Ubuntu SMP Mon Apr 24 20:40:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7cfa47d
Default JavaTemurin-1.8.0_352-b08
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/1/testReport/
Max. process+thread count1776 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 7, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 24sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 54smaster passed
+1 💚compile2m 24smaster passed
+1 💚checkstyle0m 34smaster passed
+1 💚spotless0m 41sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 25smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 28sthe patch passed
+1 💚compile2m 23sthe patch passed
+1 💚javac2m 23sthe patch passed
+1 💚checkstyle0m 33sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck9m 29sPatch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚spotless0m 37spatch has no errors when running spotless:check.
+1 💚spotbugs1m 29sthe patch passed
_ Other Tests _
+1 💚asflicense0m 10sThe patch does not generate ASF License warnings.
31m 40s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5344
JIRA IssueHBASE-28011
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 3218f0ac80e1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7cfa47d
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count79 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 7, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 29sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 56smaster passed
+1 💚compile0m 42smaster passed
+1 💚shadedjars5m 4sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 34sthe patch passed
+1 💚compile0m 43sthe patch passed
+1 💚javac0m 43sthe patch passed
+1 💚shadedjars4m 54spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24sthe patch passed
_ Other Tests _
+1 💚unit207m 41shbase-server in the patch passed.
229m 59s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5344
JIRA IssueHBASE-28011
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 159e9f18d742 5.4.0-1101-aws #109~18.04.1-Ubuntu SMP Mon Apr 24 20:40:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7cfa47d
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/1/testReport/
Max. process+thread count4651 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

virajjasani
virajjasani reviewed Aug 7, 2023
View reviewed changes

@virajjasanivirajjasani left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one minor nit, else looks good

Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java Outdated
LruBlockCache.LOG.info("totalSize=" + StringUtils.byteDesc(totalSize) + ", " + "freeSize="
long usedSize = heapSize();
long freeSize = maxSize - usedSize;
LruBlockCache.LOG.info("usedSize=" + StringUtils.byteDesc(usedSize) + ", " + "freeSize="

@virajjasanivirajjasaniAug 7, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about we also add total size?

@guluo2016guluo2016Aug 8, 2023

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank for you comment, I will update later

@virajjasanivirajjasani changed the title HBASE-28011:The logStats about LruBlockCache is not accurateHBASE-28011 : The logStats about LruBlockCache is not accurateAug 8, 2023
@Apache-HBase

Apache-HBase commented Aug 8, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 24sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 31smaster passed
+1 💚compile2m 24smaster passed
+1 💚checkstyle0m 32smaster passed
+1 💚spotless0m 39sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 22smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 30sthe patch passed
+1 💚compile2m 22sthe patch passed
+1 💚javac2m 22sthe patch passed
+1 💚checkstyle0m 33sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck9m 25sPatch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚spotless0m 41spatch has no errors when running spotless:check.
+1 💚spotbugs1m 35sthe patch passed
_ Other Tests _
+1 💚asflicense0m 8sThe patch does not generate ASF License warnings.
30m 55s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5344
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux b3960f7b3571 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7cfa47d
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count79 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

virajjasani
virajjasani approved these changes Aug 8, 2023
View reviewed changes

@virajjasanivirajjasani left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@Apache-HBase

Apache-HBase commented Aug 8, 2023

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 27sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 44smaster passed
+1 💚compile0m 47smaster passed
+1 💚shadedjars4m 55sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 22smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 36sthe patch passed
+1 💚compile0m 43sthe patch passed
+1 💚javac0m 43sthe patch passed
+1 💚shadedjars5m 0spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 21sthe patch passed
_ Other Tests _
-1 ❌unit209m 17shbase-server in the patch failed.
231m 14s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5344
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 80f3170f21a2 5.4.0-1101-aws #109~18.04.1-Ubuntu SMP Mon Apr 24 20:40:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7cfa47d
Default JavaEclipse Adoptium-11.0.17+8
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/2/testReport/
Max. process+thread count4870 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 8, 2023

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 56smaster passed
+1 💚compile0m 42smaster passed
+1 💚shadedjars5m 45sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 31smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 57sthe patch passed
+1 💚compile0m 46sthe patch passed
+1 💚javac0m 46sthe patch passed
+1 💚shadedjars5m 35spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 27sthe patch passed
_ Other Tests _
-1 ❌unit251m 48shbase-server in the patch failed.
276m 12s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5344
Optional Testsjavac javadoc unit shadedjars compile
unameLinux b80732a52174 5.4.0-1101-aws #109~18.04.1-Ubuntu SMP Mon Apr 24 20:40:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7cfa47d
Default JavaTemurin-1.8.0_352-b08
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/2/testReport/
Max. process+thread count4611 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5344/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@virajjasani
virajjasani merged commit ab4b1d8 into apache:masterAug 9, 2023
virajjasani pushed a commit that referenced this pull request Aug 9, 2023
@guluo2016@virajjasani
HBASE-28011: The logStats about LruBlockCache is not accurate (#5344)
d7be336
virajjasani pushed a commit that referenced this pull request Aug 9, 2023
@guluo2016@virajjasani
HBASE-28011: The logStats about LruBlockCache is not accurate (#5344)
438b591
virajjasani pushed a commit that referenced this pull request Aug 9, 2023
@guluo2016@virajjasani
HBASE-28011: The logStats about LruBlockCache is not accurate (#5344)
f27659b
virajjasani pushed a commit that referenced this pull request Aug 9, 2023
@guluo2016@virajjasani
HBASE-28011: The logStats about LruBlockCache is not accurate (#5344)
c26f2c4
@guluo2016
guluo2016 deleted the hbase_lru branch August 27, 2023 14:32
vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
@guluo2016Jenkins
HBASE-28011 : The logStats about LruBlockCache is not accurate (apach…
5eb4b83
…e#5344)
(cherry picked from commit c26f2c4)
Change-Id: I4e0d66c0b5015ca7e50efa3bf5c9fe9fee4ad78d
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@virajjasanivirajjasanivirajjasani approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@guluo2016@Apache-HBase@virajjasani