Skip to content

HBASE-27800: Add support for default user quotas - #5666

Merged
bbeaudreault merged 2 commits into
apache:masterfrom
HubSpot:HBASE-27800
Feb 7, 2024
Merged

HBASE-27800: Add support for default user quotas#5666
bbeaudreault merged 2 commits into
apache:masterfrom
HubSpot:HBASE-27800

Conversation

@rmdmattingly

@rmdmattinglyrmdmattingly commented Feb 6, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

If you're managing HBase for many users, then managing effective user quotas can be a difficult operational burden. At my day job we have thousands of web services, workers, and recurring jobs — each of which we consider a distinct user via request attributes and quota user overrides. A common reliability issue is that some single system runs awry, for whatever reason, and generates a pathologic amount of traffic at the HBase level; this can cause pain for any other systems dependent on the given HBase cluster.

The above situation could potentially be solved by maintaining our own automated user quota management, but even better would be the ability to specify default user quotas at the HBase level. So that's exactly what this PR introduces.

In this PR we add six new configuration options to the QuotaUtil class:

/* * The below defaults, if configured, will be applied to otherwise unthrottled users. For example, * set `hbase.quota.default.user.machine.read.size` to `1048576` in your hbase-site.xml to ensure * that any given user may not query more than 1mb per second from any given machine, unless * explicitly permitted by a persisted quota. All of these defaults use TimeUnit.SECONDS and * QuotaScope.MACHINE. */publicstaticfinalStringQUOTA_DEFAULT_USER_MACHINE_READ_NUM =
"hbase.quota.default.user.machine.read.num";
publicstaticfinalStringQUOTA_DEFAULT_USER_MACHINE_READ_SIZE =
"hbase.quota.default.user.machine.read.size";
publicstaticfinalStringQUOTA_DEFAULT_USER_MACHINE_REQUEST_NUM =
"hbase.quota.default.user.machine.request.num";
publicstaticfinalStringQUOTA_DEFAULT_USER_MACHINE_REQUEST_SIZE =
"hbase.quota.default.user.machine.request.size";
publicstaticfinalStringQUOTA_DEFAULT_USER_MACHINE_WRITE_NUM =
"hbase.quota.default.user.machine.write.num";
publicstaticfinalStringQUOTA_DEFAULT_USER_MACHINE_WRITE_SIZE =
"hbase.quota.default.user.machine.write.size";

These options are exactly what they sound like — one could configure hbase.quota.default.user.machine.read.size and hbase.quota.default.user.machine.req.num to ensure that no users can exceed a given read size and request per-machine throughput (unless explicitly given larger allowance via a defined quota).

cc @bbeaudreault@hgromer@eab148@bozzkar

bbeaudreault
bbeaudreault reviewed Feb 6, 2024
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
@Apache-HBase

Apache-HBase commented Feb 6, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 42sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 2smaster passed
+1 💚compile2m 30smaster passed
+1 💚checkstyle0m 38smaster passed
+1 💚spotless0m 47sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 33smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 44sthe patch passed
+1 💚compile2m 26sthe patch passed
+1 💚javac2m 26sthe patch passed
+1 💚checkstyle0m 36shbase-server: The patch generated 0 new + 2 unchanged - 1 fixed = 2 total (was 3)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck4m 44sPatch does not cause any errors with Hadoop 3.3.6.
+1 💚spotless0m 42spatch has no errors when running spotless:check.
+1 💚spotbugs1m 37sthe patch passed
_ Other Tests _
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
28m 18s
SubsystemReport/Notes
DockerClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5666
JIRA IssueHBASE-27800
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 6fdfddf6256b 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / afbf2ad
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count80 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Feb 7, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec2m 9sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 11smaster passed
+1 💚compile0m 46smaster passed
+1 💚shadedjars5m 47sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 58sthe patch passed
+1 💚compile0m 45sthe patch passed
+1 💚javac0m 45sthe patch passed
+1 💚shadedjars5m 41spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23sthe patch passed
_ Other Tests _
+1 💚unit220m 24shbase-server in the patch passed.
246m 29s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5666
JIRA IssueHBASE-27800
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 074c5b6f6f7f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / afbf2ad
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/testReport/
Max. process+thread count4667 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Feb 7, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec2m 11sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 41smaster passed
+1 💚compile0m 38smaster passed
+1 💚shadedjars5m 32sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 26sthe patch passed
+1 💚compile0m 38sthe patch passed
+1 💚javac0m 38sthe patch passed
+1 💚shadedjars5m 31spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 22sthe patch passed
_ Other Tests _
+1 💚unit226m 44shbase-server in the patch passed.
251m 39s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5666
JIRA IssueHBASE-27800
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 99e71ce97c8d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / afbf2ad
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/testReport/
Max. process+thread count4803 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Feb 7, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 40sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 46smaster passed
+1 💚compile2m 28smaster passed
+1 💚checkstyle0m 36smaster passed
+1 💚spotless0m 41sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 28smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 47sthe patch passed
+1 💚compile2m 26sthe patch passed
+1 💚javac2m 26sthe patch passed
+1 💚checkstyle0m 36shbase-server: The patch generated 0 new + 2 unchanged - 1 fixed = 2 total (was 3)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck4m 48sPatch does not cause any errors with Hadoop 3.3.6.
+1 💚spotless0m 42spatch has no errors when running spotless:check.
+1 💚spotbugs1m 39sthe patch passed
_ Other Tests _
+1 💚asflicense0m 10sThe patch does not generate ASF License warnings.
27m 50s
SubsystemReport/Notes
DockerClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5666
JIRA IssueHBASE-27800
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux d48a820d6911 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / afbf2ad
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count78 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Feb 7, 2024

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 27sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 56smaster passed
+1 💚compile0m 46smaster passed
+1 💚shadedjars5m 44sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 50sthe patch passed
+1 💚compile0m 45sthe patch passed
+1 💚javac0m 45sthe patch passed
+1 💚shadedjars5m 45spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23sthe patch passed
_ Other Tests _
-1 ❌unit222m 59shbase-server in the patch failed.
246m 52s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5666
JIRA IssueHBASE-27800
Optional Testsjavac javadoc unit shadedjars compile
unameLinux f8a2975dddb0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / afbf2ad
Default JavaEclipse Adoptium-11.0.17+8
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/testReport/
Max. process+thread count5528 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Feb 7, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 26sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 45smaster passed
+1 💚compile0m 38smaster passed
+1 💚shadedjars5m 28sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 22sthe patch passed
+1 💚compile0m 39sthe patch passed
+1 💚javac0m 39sthe patch passed
+1 💚shadedjars5m 27spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 21sthe patch passed
_ Other Tests _
+1 💚unit225m 45shbase-server in the patch passed.
248m 9s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5666
JIRA IssueHBASE-27800
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 20d373a14779 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / afbf2ad
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/testReport/
Max. process+thread count5710 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@rmdmattingly

rmdmattingly commented Feb 7, 2024

Copy link
Copy Markdown
ContributorAuthor

The TestAcidGuaranteesWithBasicPolicy failure appears unrelated

@bbeaudreault
bbeaudreault merged commit 275d928 into apache:masterFeb 7, 2024
@bbeaudreault
bbeaudreault deleted the HBASE-27800 branch February 7, 2024 14:23
bbeaudreault pushed a commit that referenced this pull request Feb 7, 2024
@rmdmattingly@bbeaudreault
HBASE-27800: Add support for default user quotas (#5666)
7f531f3
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
bbeaudreault pushed a commit that referenced this pull request Feb 7, 2024
@rmdmattingly@bbeaudreault
HBASE-27800: Add support for default user quotas (#5666)
6062732
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
bbeaudreault pushed a commit that referenced this pull request Feb 7, 2024
@rmdmattingly@bbeaudreault
HBASE-27800: Add support for default user quotas (#5666)
240dfa0
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
bbeaudreault pushed a commit to HubSpot/hbase that referenced this pull request Feb 7, 2024
@rmdmattingly@bbeaudreault
HubSpot Backport: HBASE-27800: Add support for default user quotas (a…
53f2137
…pache#5666)
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
@rmdmattinglyrmdmattingly mentioned this pull request Feb 16, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bbeaudreaultbbeaudreaultbbeaudreault approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@rmdmattingly@Apache-HBase@bbeaudreault