Skip to content

HBASE-28841: Modify default value of hbase.bucketcache.persistence.chunksize to 10K - #6252

Merged
wchevreuil merged 1 commit into
apache:masterfrom
janardhanrh:HBASE-28841
Sep 16, 2024
Merged

HBASE-28841: Modify default value of hbase.bucketcache.persistence.chunksize to 10K#6252
wchevreuil merged 1 commit into
apache:masterfrom
janardhanrh:HBASE-28841

Conversation

@jhungund

Copy link
Copy Markdown
Contributor

Currently, the default value of the configuration parameter "hbase.bucketcache.persistence.chunksize" is 10 million (10000000). This is the number of block entries that are processed during the persistence of bucket-cache backing map to the persistence file. During the testing, it was found that, this high number of chunksize resulted in high heap utilisation in region servers leading to longer GC pauses which also led to server crashes intermittently.

When the value of this configuration is set to 10K(10000), the cache remains stable. No GC delays are observed. Also no server crashes are observed. The jmap outputs collected against the regionservers showed reduced memory utilisation from 4.5-5GB to 1-1.5GB for the objects related to persistence code.

Change-Id: I83ea05e694fa9b76683a837f83162b6a72ef1a3c

…unksize to 10K
Currently, the default value of the configuration parameter
"hbase.bucketcache.persistence.chunksize" is 10 million (10000000).
This is the number of block entries that are processed during the persistence of bucket-cache
backing map to the persistence file. During the testing, it was found that, this high number
of chunksize resulted in high heap utilisation in region servers leading to longer GC pauses
which also led to server crashes intermittently.
When the value of this configuration is set to 10K(10000), the cache remains stable.
No GC delays are observed. Also no server crashes are observed.
The jmap outputs collected against the regionservers showed reduced memory utilisation from
4.5-5GB to 1-1.5GB for the objects related to persistence code.
Change-Id: I83ea05e694fa9b76683a837f83162b6a72ef1a3c
@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 26sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗codespell0m 0scodespell was not available.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
_ master Compile Tests _
+1 💚mvninstall3m 6smaster passed
+1 💚compile3m 2smaster passed
+1 💚checkstyle0m 35smaster passed
+1 💚spotbugs1m 29smaster passed
+1 💚spotless0m 42sbranch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚mvninstall3m 6sthe patch passed
+1 💚compile3m 1sthe patch passed
+1 💚javac3m 1sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 34sthe patch passed
+1 💚spotbugs1m 35sthe patch passed
+1 💚hadoopcheck10m 50sPatch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚spotless0m 41spatch has no errors when running spotless:check.
_ Other Tests _
+1 💚asflicense0m 10sThe patch does not generate ASF License warnings.
36m 2s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6252/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#6252
JIRA IssueHBASE-28841
Optional Testsdupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
unameLinux c5cf15866fa4 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 2b6db4e
Default JavaEclipse Adoptium-17.0.11+9
Max. process+thread count85 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6252/1/console
versionsgit=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 47sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 48smaster passed
+1 💚compile1m 10smaster passed
+1 💚javadoc0m 37smaster passed
+1 💚shadedjars6m 19sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚mvninstall3m 28sthe patch passed
+1 💚compile1m 22sthe patch passed
+1 💚javac1m 22sthe patch passed
+1 💚javadoc0m 36sthe patch passed
+1 💚shadedjars6m 42spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit268m 37shbase-server in the patch passed.
298m 16s
SubsystemReport/Notes
DockerClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6252/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR#6252
JIRA IssueHBASE-28841
Optional Testsjavac javadoc unit compile shadedjars
unameLinux 7370f01395b9 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 2b6db4e
Default JavaEclipse Adoptium-17.0.11+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6252/1/testReport/
Max. process+thread count5395 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6252/1/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@wchevreuil
wchevreuil merged commit 3f9dbf2 into apache:masterSep 16, 2024
wchevreuil pushed a commit that referenced this pull request Sep 16, 2024
…unksize to 10K (#6252)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wchevreuil pushed a commit that referenced this pull request Sep 16, 2024
…unksize to 10K (#6252)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@jhungund@Apache-HBase@wchevreuil@ragarkar