Skip to content

HBASE-26945 Quotas causes too much load on meta for large clusters - #4567

Merged
bbeaudreault merged 1 commit into
apache:masterfrom
HubSpot:HBASE-26945
Jun 23, 2022
Merged

HBASE-26945 Quotas causes too much load on meta for large clusters#4567
bbeaudreault merged 1 commit into
apache:masterfrom
HubSpot:HBASE-26945

Conversation

@bbeaudreault

Copy link
Copy Markdown
Contributor

This is a relatively simple change -- We were already fetching ClusterMetrics. This change adds Option.TABLE_TO_REGIONS_COUNT to that and then uses clusterMetrics.getTableRegionStatesCount() instead of MetaTableAccessor.

I couldn't think of any new tests to add here. There are substantial tests in TestClusterScopeQuotaThrottle, TestQuotaThrottle, and TestQuotaAdmin. I did some messing with the logic to prove that existing tests would have failed if there were a bug here. This is just a different implementation for an existing functionality, rather than a net new functionality to test.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 38sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 24smaster passed
+1 💚compile2m 18smaster passed
+1 💚checkstyle0m 29smaster passed
+1 💚spotless0m 42sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 15smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 9sthe patch passed
+1 💚compile2m 14sthe patch passed
+1 💚javac2m 14sthe patch passed
+1 💚checkstyle0m 27sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck11m 55sPatch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚spotless0m 39spatch has no errors when running spotless:check.
+1 💚spotbugs1m 18sthe patch passed
_ Other Tests _
+1 💚asflicense0m 10sThe patch does not generate ASF License warnings.
31m 53s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4567/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4567
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 46fe0a28a197 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 9bae50f
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count64 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4567/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 37sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 48smaster passed
+1 💚compile0m 42smaster passed
+1 💚shadedjars3m 49sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 27smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 33sthe patch passed
+1 💚compile0m 41sthe patch passed
+1 💚javac0m 41sthe patch passed
+1 💚shadedjars3m 46spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23sthe patch passed
_ Other Tests _
+1 💚unit203m 21shbase-server in the patch passed.
220m 32s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4567/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4567
Optional Testsjavac javadoc unit shadedjars compile
unameLinux fce63e003fa3 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 9bae50f
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4567/1/testReport/
Max. process+thread count2670 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4567/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 27sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 33smaster passed
+1 💚compile0m 39smaster passed
+1 💚shadedjars3m 39sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 13sthe patch passed
+1 💚compile0m 39sthe patch passed
+1 💚javac0m 39sthe patch passed
+1 💚shadedjars3m 39spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24sthe patch passed
_ Other Tests _
+1 💚unit204m 45shbase-server in the patch passed.
221m 10s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4567/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#4567
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 255640c58cea 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 9bae50f
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4567/1/testReport/
Max. process+thread count2582 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4567/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@bbeaudreault
bbeaudreault merged commit e84cd3d into apache:masterJun 23, 2022
@bbeaudreault

Copy link
Copy Markdown
ContributorAuthor

Thanks @sunhelly!

@bbeaudreault
bbeaudreault deleted the HBASE-26945 branch June 23, 2022 14:34
bbeaudreault added a commit to HubSpot/hbase that referenced this pull request Jun 23, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@bbeaudreault@Apache-HBase@sunhelly