Skip to content

HBASE-27229 BucketCache statistics should not count evictions by hfile - #4639

Merged
bbeaudreault merged 1 commit into
apache:masterfrom
HubSpot:HBASE-27229
Jul 26, 2022
Merged

HBASE-27229 BucketCache statistics should not count evictions by hfile#4639
bbeaudreault merged 1 commit into
apache:masterfrom
HubSpot:HBASE-27229

Conversation

@bbeaudreault

@bbeaudreaultbbeaudreault commented Jul 21, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

This change mirrors how it works in LruBlockCache and the other implementations. A evictedByEvictionProcess boolean is added to doEvictBlock and blockEvicted, which wraps the cacheStats.evicted calls. Then, all calls to those methods are set to pass in false except the calls originating from the freeSpace(String why) method.

There are various edge cases which can result in a cache eviction. It's hard to test all of those, but I added a test which covers the 3 main cases.

@Apache-HBase

Apache-HBase commented Jul 21, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 40sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 21smaster passed
+1 💚compile7m 19smaster passed
+1 💚checkstyle0m 40smaster passed
+1 💚spotless0m 56sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 32smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 41sthe patch passed
+1 💚compile2m 37sthe patch passed
+1 💚javac2m 37sthe patch passed
+1 💚checkstyle0m 30sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck15m 31sPatch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚spotless1m 9spatch has no errors when running spotless:check.
+1 💚spotbugs2m 20sthe patch passed
_ Other Tests _
+1 💚asflicense0m 15sThe patch does not generate ASF License warnings.
47m 17s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4639
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 95969b20207a 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / da27a67
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count64 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jul 21, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 16sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 3smaster passed
+1 💚compile2m 50smaster passed
+1 💚checkstyle0m 34smaster passed
+1 💚spotless0m 49sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 36smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 51sthe patch passed
+1 💚compile2m 38sthe patch passed
+1 💚javac2m 38sthe patch passed
+1 💚checkstyle0m 36sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck13m 24sPatch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚spotless0m 47spatch has no errors when running spotless:check.
+1 💚spotbugs1m 36sthe patch passed
_ Other Tests _
+1 💚asflicense0m 10sThe patch does not generate ASF License warnings.
37m 58s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4639
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux abb3f991cd1d 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 31fc97e
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count64 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jul 21, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 10sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 35smaster passed
+1 💚compile0m 40smaster passed
+1 💚shadedjars3m 47sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 17sthe patch passed
+1 💚compile0m 40sthe patch passed
+1 💚javac0m 40sthe patch passed
+1 💚shadedjars3m 46spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25sthe patch passed
_ Other Tests _
+1 💚unit211m 56shbase-server in the patch passed.
229m 20s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#4639
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 9b4ce8c9d477 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 31fc97e
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/testReport/
Max. process+thread count2453 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jul 21, 2022

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 7sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 54smaster passed
+1 💚compile0m 46smaster passed
+1 💚shadedjars3m 45sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 29smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 39sthe patch passed
+1 💚compile0m 47sthe patch passed
+1 💚javac0m 47sthe patch passed
+1 💚shadedjars3m 47spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 27sthe patch passed
_ Other Tests _
-1 ❌unit244m 28shbase-server in the patch failed.
262m 49s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4639
Optional Testsjavac javadoc unit shadedjars compile
unameLinux b58d740b4ea4 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 31fc97e
Default JavaAdoptOpenJDK-11.0.10+9
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/testReport/
Max. process+thread count2789 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jul 25, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 9sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 1sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 15smaster passed
+1 💚compile2m 14smaster passed
+1 💚checkstyle0m 31smaster passed
+1 💚spotless0m 44sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 18smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 15sthe patch passed
+1 💚compile2m 13sthe patch passed
+1 💚javac2m 13sthe patch passed
+1 💚checkstyle0m 30sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck11m 28sPatch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚spotless0m 41spatch has no errors when running spotless:check.
+1 💚spotbugs1m 24sthe patch passed
_ Other Tests _
+1 💚asflicense0m 9sThe patch does not generate ASF License warnings.
31m 41s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4639
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 28daa80393db 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 0ae42dd
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count64 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jul 25, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 10sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 17smaster passed
+1 💚compile0m 41smaster passed
+1 💚shadedjars3m 45sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 21sthe patch passed
+1 💚compile0m 39sthe patch passed
+1 💚javac0m 39sthe patch passed
+1 💚shadedjars3m 47spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25sthe patch passed
_ Other Tests _
+1 💚unit212m 57shbase-server in the patch passed.
229m 48s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#4639
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 2104842518a3 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 0ae42dd
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/testReport/
Max. process+thread count2592 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jul 25, 2022

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 49sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 17smaster passed
+1 💚compile1m 9smaster passed
+1 💚shadedjars4m 55sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 33smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 43sthe patch passed
+1 💚compile1m 6sthe patch passed
+1 💚javac1m 6sthe patch passed
+1 💚shadedjars4m 40spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 29sthe patch passed
_ Other Tests _
-1 ❌unit305m 36shbase-server in the patch failed.
330m 11s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4639
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 1754aff4332a 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 0ae42dd
Default JavaAdoptOpenJDK-11.0.10+9
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/testReport/
Max. process+thread count2673 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4639/3/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@bbeaudreault
bbeaudreault merged commit 277058e into apache:masterJul 26, 2022
@bbeaudreault
bbeaudreault deleted the HBASE-27229 branch July 26, 2022 21:00
bbeaudreault added a commit that referenced this pull request Jul 26, 2022
@bbeaudreault
HBASE-27229BucketCache statistics should not count evictions by hfile (
827c1ab
#4639)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
bbeaudreault added a commit that referenced this pull request Jul 26, 2022
@bbeaudreault
HBASE-27229BucketCache statistics should not count evictions by hfile (
25136c3
#4639)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
bbeaudreault added a commit to HubSpot/hbase that referenced this pull request Jul 28, 2022
@bbeaudreault
HubSpot Backport: HBASE-27229 BucketCache statistics should not count…
c6e14f9
… evictions by hfile (apache#4639)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
@bbeaudreaultRichard Antal
HBASE-27229 BucketCache statistics should not count evictions by hfile (
df9d727
apache#4639)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
(cherry picked from commit 25136c3)
Change-Id: I44ade79a5d6011e86c8c3f7dff9ecef72dbad3bb
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@apurtellapurtellapurtell approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@bbeaudreault@Apache-HBase@apurtell