HBASE-28359 Improve quota RateLimiter synchronization by rmdmattingly · Pull Request #5683 · apache/hbase

rmdmattingly · 2024-02-15T14:29:38Z

We've been experiencing RpcThrottlingException with 0ms waitInterval. This seems odd and wasteful, since the client side will immediately retry without backoff. I think the problem is related to the synchronization of RateLimiter.

The TimeBasedLimiter checkQuota method does the following:

if (!reqSizeLimiter.canExecute(estimateWriteSize + estimateReadSize)) {
RpcThrottlingException.throwRequestSizeExceeded(
reqSizeLimiter.waitInterval(estimateWriteSize + estimateReadSize));
}

Both canExecute and waitInterval are synchronized, but we're calling them independently. So it's possible under high concurrency for canExecute to return false, but then waitInterval returns 0 (would have been true).

This simplifies the API by having canExecute return the waitInterval, it being greater than 0 if we should throttle the client.

This also implicitly fixes a bug with request number quotas — we were returning a waitInterval assuming a single resource consumption, regardless of the resources consumed by the operation:

hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/TimeBasedLimiter.java

Lines 144 to 146 in 9c8c9e7

if (!reqsLimiter.canExecute(writeReqs + readReqs)) { 
RpcThrottlingException.throwNumRequestsExceeded(reqsLimiter.waitInterval()); 
 } 

I'm marking this as a draft while I deploy this onto a test cluster to confirm that it resolves our 0ms throttle backoffs

cc @hgromer @eab148 @bozzkar

Apache-HBase · 2024-02-15T15:06:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 42s	master passed
+1 💚	compile	0m 42s	master passed
+1 💚	shadedjars	5m 5s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 28s	the patch passed
+1 💚	compile	0m 42s	the patch passed
+1 💚	javac	0m 42s	the patch passed
+1 💚	shadedjars	5m 0s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	11m 20s	hbase-server in the patch failed.
		31m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5683
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b329ea2b8cff 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d925754`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/1/testReport/
Max. process+thread count	1717 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-15T15:09:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 59s	master passed
+1 💚	compile	0m 46s	master passed
+1 💚	shadedjars	5m 30s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 42s	the patch passed
+1 💚	compile	0m 44s	the patch passed
+1 💚	javac	0m 44s	the patch passed
+1 💚	shadedjars	5m 29s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	the patch passed
		_ Other Tests _
-1 ❌	unit	13m 23s	hbase-server in the patch failed.
		34m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5683
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 0d99844e4a74 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d925754`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/1/testReport/
Max. process+thread count	1843 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-15T15:11:18Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 39s	master passed
+1 💚	compile	2m 59s	master passed
+1 💚	checkstyle	0m 42s	master passed
+1 💚	spotless	0m 50s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 45s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 7s	the patch passed
+1 💚	compile	2m 50s	the patch passed
+1 💚	javac	2m 50s	the patch passed
+1 💚	checkstyle	0m 48s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	6m 29s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 58s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 37s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		35m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5683
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux c27ac091b395 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d925754`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	77 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-15T21:03:52Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 5s	master passed
+1 💚	compile	2m 27s	master passed
+1 💚	checkstyle	0m 35s	master passed
+1 💚	spotless	0m 42s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 29s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 43s	the patch passed
+1 💚	compile	2m 24s	the patch passed
+1 💚	javac	2m 24s	the patch passed
+1 💚	checkstyle	0m 32s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	5m 10s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 38s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		28m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5683
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux a900e9d54d06 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d925754`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	82 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-16T00:49:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 53s	master passed
+1 💚	compile	0m 49s	master passed
+1 💚	shadedjars	5m 9s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 47s	the patch passed
+1 💚	compile	0m 50s	the patch passed
+1 💚	javac	0m 50s	the patch passed
+1 💚	shadedjars	5m 5s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	230m 18s	hbase-server in the patch passed.
		254m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5683
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a5f81154eefa 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d925754`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/2/testReport/
Max. process+thread count	5656 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-16T00:54:40Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 48s	master passed
+1 💚	compile	0m 37s	master passed
+1 💚	shadedjars	5m 31s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 27s	the patch passed
+1 💚	compile	0m 38s	the patch passed
+1 💚	javac	0m 38s	the patch passed
+1 💚	shadedjars	5m 29s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	the patch passed
		_ Other Tests _
+1 💚	unit	236m 23s	hbase-server in the patch passed.
		259m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5683
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 8468a0cc5573 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d925754`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/2/testReport/
Max. process+thread count	5009 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

rmdmattingly · 2024-02-16T19:08:21Z

- return readAvailable;
+ // at this point we've grabbed some quota, so we should use at least that
+ return Math.max(readAvailable, readConsumed);


Without this change we've experienced erroneous MultiActionResultTooLarge when riding close to quotas. As readAvailable can be negative, this can even affect otherwise irrelevant operation types. For example, riding close to a read quota and entering negative read availability would cause multiPuts, which do no reading, to throw MultiActionResultTooLarge. The logic for throwing a MultiActionResultTooLarge can be found here. Worse, this exception is retried immediately, so this can snowball into a DDOS of your RPC layer.

Should we move this Math.max to caller? Theoretically readAvailable could be negative, the caller should decide how to deal with this.

Yeah, I wonder whether it's worth having a separate method, OperationQuota#getMaxResultSize, which encapsulates this logic and is not hiding business logic in what appears to be a simple getter

I think we should keep this here. I have three reasons:
There are only 2 callers to this method, and both would need the Math.max() added
We don't currently expose readConsumed
Logically, I think it makes sense to consider readAvailable in this way for all cases. I could even see it being Math.max(readAvailable + readConsumed, readConsumed). This is because of the nuanced quota logic in checkQuota. Details on this one below:
In checkQuota, it iterates all limiters twice. Once to check for available (or throw throttle exception). Once it gets through all limiters, it loops them again to grab the quota. This loop can over consume the quota. At this point we've grabbed the quota and there's no easy way to return it. So we might as well consider this as "available" for us.
The quotas are complicated because there can be many limiters and checkQuota is not synchronized. It'd be more accurate to synchronize the entire checkQuota, but I think it's better for performance to let it be.

Then better rename the method, or at least add more comments to describe the logic.

Ah sorry Bryan, I didn't see your comment until I pushed a change. I think the only difference though would be to remove the readAvailable and readConsumed getters, and move the getMaxResultSize logic out of the interface. Do you think that's worth doing, or do you like the changeset in its current form?

Personally I prefer the original name, which has the javadoc:
/** Returns the number of bytes available to read to avoid exceeding the quota *
To me "read available" makes sense to be "how much did we grab, plus how much is left". But I can see that might be subjective. So I don't have a strong opinion.

I see both sides. I think it's ok for us to be more specific with the naming here because, like you said, the current usage scope is small enough for us to be specific

Apache-HBase · 2024-02-16T19:36:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 45s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 52s	master passed
+1 💚	compile	2m 26s	master passed
+1 💚	checkstyle	0m 35s	master passed
+1 💚	spotless	0m 42s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 43s	the patch passed
+1 💚	compile	2m 25s	the patch passed
+1 💚	javac	2m 25s	the patch passed
+1 💚	checkstyle	0m 37s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	4m 48s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 39s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		30m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5683
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux f8c71fdec776 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d925754`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	78 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/3/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-16T23:18:04Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 46s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 50s	master passed
+1 💚	compile	0m 48s	master passed
+1 💚	shadedjars	5m 7s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 45s	the patch passed
+1 💚	compile	0m 48s	the patch passed
+1 💚	javac	0m 48s	the patch passed
+1 💚	shadedjars	5m 5s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	the patch passed
		_ Other Tests _
+1 💚	unit	229m 7s	hbase-server in the patch passed.
		252m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5683
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 0901cae12482 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d925754`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/3/testReport/
Max. process+thread count	5554 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/3/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-16T23:18:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 6s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 26s	master passed
+1 💚	compile	0m 38s	master passed
+1 💚	shadedjars	5m 29s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 23s	the patch passed
+1 💚	compile	0m 39s	the patch passed
+1 💚	javac	0m 39s	the patch passed
+1 💚	shadedjars	5m 30s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	the patch passed
		_ Other Tests _
+1 💚	unit	229m 28s	hbase-server in the patch passed.
		253m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5683
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 2d61f1aae35d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d925754`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/3/testReport/
Max. process+thread count	5005 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/3/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2024-02-18T10:26:24Z

- return readAvailable;
+ // at this point we've grabbed some quota, so we should use at least that
+ return Math.max(readAvailable, readConsumed);


Should we move this Math.max to caller? Theoretically readAvailable could be negative, the caller should decide how to deal with this.

Apache-HBase · 2024-02-18T16:03:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 3s	master passed
+1 💚	compile	2m 30s	master passed
+1 💚	checkstyle	0m 39s	master passed
+1 💚	spotless	0m 44s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 31s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 46s	the patch passed
+1 💚	compile	2m 25s	the patch passed
+1 💚	javac	2m 25s	the patch passed
+1 💚	checkstyle	0m 36s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	4m 52s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 40s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 38s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		28m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5683
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 9361c2d30f40 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `96a447f`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	80 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/4/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-18T19:46:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 25s	master passed
+1 💚	compile	0m 38s	master passed
+1 💚	shadedjars	5m 29s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 20s	the patch passed
+1 💚	compile	0m 38s	the patch passed
+1 💚	javac	0m 38s	the patch passed
+1 💚	shadedjars	5m 24s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	the patch passed
		_ Other Tests _
+1 💚	unit	228m 44s	hbase-server in the patch passed.
		250m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5683
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a8ded6bc9e60 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `96a447f`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/4/testReport/
Max. process+thread count	5252 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-18T19:47:42Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 41s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 58s	master passed
+1 💚	compile	0m 49s	master passed
+1 💚	shadedjars	5m 8s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 47s	the patch passed
+1 💚	compile	0m 50s	the patch passed
+1 💚	javac	0m 50s	the patch passed
+1 💚	shadedjars	5m 6s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 25s	hbase-server generated 1 new + 95 unchanged - 0 fixed = 96 total (was 95)
		_ Other Tests _
+1 💚	unit	228m 31s	hbase-server in the patch passed.
		252m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5683
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a31e421702b9 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `96a447f`
Default Java	Eclipse Adoptium-11.0.17+8
javadoc	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/4/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/4/testReport/
Max. process+thread count	5420 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5683/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

fix RateLimiter canExecute and waitInterval synchronization
334567e

bbeaudreault reviewed Feb 15, 2024
View reviewed changes

Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java Outdated

cleanup
11cd56c

rmdmattingly marked this pull request as ready for review February 16, 2024 14:11

fix underrepresented read availability
abe1a32

rmdmattingly commented Feb 16, 2024
View reviewed changes

Apache9 reviewed Feb 18, 2024
View reviewed changes

PR feedback: new method, no private sync, naming
6d4abbf

Apache9 approved these changes Feb 22, 2024
View reviewed changes

bbeaudreault merged commit e3825ca into apache:masterMar 7, 2024

bbeaudreault deleted the HBASE-28359 branch March 7, 2024 01:06

bbeaudreault pushed a commit that referenced this pull request Mar 7, 2024
HBASE-28359Improve quota RateLimiter synchronization (#5683)
b374e52
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

bbeaudreault pushed a commit that referenced this pull request Mar 7, 2024
HBASE-28359Improve quota RateLimiter synchronization (#5683)
c7eb3d8
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

bbeaudreault pushed a commit that referenced this pull request Mar 7, 2024
HBASE-28359Improve quota RateLimiter synchronization (#5683)
12a07b6
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

bbeaudreault pushed a commit to HubSpot/hbase that referenced this pull request Mar 7, 2024
HBASE-28359 Improve quota RateLimiter synchronization (apache#5683)
7aedd84
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

	if (!reqsLimiter.canExecute(writeReqs + readReqs)) {
	RpcThrottlingException.throwNumRequestsExceeded(reqsLimiter.waitInterval());
	}

Uh oh!

Conversation

rmdmattingly commented Feb 15, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Apache-HBase commented Feb 15, 2024

Uh oh!

Apache-HBase commented Feb 15, 2024

Uh oh!

Apache-HBase commented Feb 15, 2024

Uh oh!

Apache-HBase commented Feb 15, 2024

Uh oh!

Apache-HBase commented Feb 16, 2024

Uh oh!

Apache-HBase commented Feb 16, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Feb 16, 2024

Uh oh!

Apache-HBase commented Feb 16, 2024

Uh oh!

Apache-HBase commented Feb 16, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Apache-HBase commented Feb 18, 2024

Uh oh!

Apache-HBase commented Feb 18, 2024

Uh oh!

Apache-HBase commented Feb 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rmdmattingly commented Feb 15, 2024•
edited
Loading