Skip to content

HBASE-29141 Calculate default maxQueueLength call queues correctly - #7490

Merged
apurtell merged 5 commits into
apache:masterfrom
Umeshkumar9414:HBASE-29141
Jan 26, 2026
Merged

HBASE-29141 Calculate default maxQueueLength call queues correctly#7490
apurtell merged 5 commits into
apache:masterfrom
Umeshkumar9414:HBASE-29141

Conversation

@Umeshkumar9414

@Umeshkumar9414Umeshkumar9414 commented Nov 28, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

Issues solved -

  • While calculating default maxQueueLength, all handlers are considered instead of handler of one queue.
  • Soft limit was getting overridden becuase calling initializeQueues multiple times.

@Umeshkumar9414Umeshkumar9414 changed the title @HBASE-29141 Resolution for high default maxQueueLength for call queuesHBASE-29141 Resolution for high default maxQueueLength for call queuesNov 28, 2025
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Umeshkumar9414

Umeshkumar9414 commented Nov 29, 2025

Copy link
Copy Markdown
ContributorAuthor

@Apache9 , @apurtell , @virajjasani need help in review.

mnpoonia
mnpoonia approved these changes Nov 30, 2025
View reviewed changes

@mnpooniamnpoonia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Umeshkumar9414 This is surely a change in good direction.

d-c-manning
d-c-manning reviewed Dec 1, 2025
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BalancedQueueRpcExecutor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java Outdated
currentQueueLimit = (int) queueInitArgs[0];
queueInitArgs[0] = Math.max((int) queueInitArgs[0], DEFAULT_CALL_QUEUE_SIZE_HARD_LIMIT);
queueInitArgs[0] = Math.min((int) queueInitArgs[0], DEFAULT_CALL_QUEUE_SIZE_HARD_LIMIT);
// queue should neven be initialised with 0 or less length

@d-c-manningd-c-manningDec 1, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit typo

Suggested change
// queue should neven be initialised with 0 or less length
// queue should never be initialised with 0 or less length

Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestRWQueueRpcExecutor.java
PriorityFunction qosFunction = mock(PriorityFunction.class);
RWQueueRpcExecutor executor =
new RWQueueRpcExecutor(testName.getMethodName(), 100, 100, qosFunction, conf, null);
RWQueueRpcExecutor executor = new RWQueueRpcExecutor(testName.getMethodName(), 100,

@d-c-manningd-c-manningDec 1, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we have a test that validates the queue length limits? Could we validate RpcExecutor#currentQueueLimit directly? Or some other validation that tries to fill the queue, but that may require more work and complexity.

@Umeshkumar9414Umeshkumar9414Jan 8, 2026

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a test

@Umeshkumar9414Umeshkumar9414 mentioned this pull request Dec 9, 2025
Merged
@Umeshkumar9414Umeshkumar9414 changed the title HBASE-29141 Resolution for high default maxQueueLength for call queuesHBASE-29141 Calculate default maxQueueLength call queues correctlyJan 8, 2026
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Umeshkumar9414

Umeshkumar9414 commented Jan 9, 2026

Copy link
Copy Markdown
ContributorAuthor

I tried both the tests (TestAcidGuaranteesWithBasicPolicy and TestAcidGuaranteesWithEagerPolicy) on my local and both are working fine.
@Apache9 , @d-c-manning , @apurtell can you please review this PR.

@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
d-c-manning
d-c-manning approved these changes Jan 13, 2026
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java Outdated
.collect(Collectors.groupingBy(Pair::getFirst, Collectors.summingLong(Pair::getSecond)));
}

// IMPORTANT: Call this method only ONCE per executor instance.

@d-c-manningd-c-manningJan 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we could just enforce this in the code? Only allow it to be called once, and if it is called a second time then throw an "already initialized" error?

@apurtellapurtellJan 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do this.

@Umeshkumar9414Umeshkumar9414Jan 13, 2026

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java Outdated
handlerCount * RpcServer.DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER);
int maxPriorityQueueLength = conf.getInt(RpcScheduler.IPC_SERVER_PRIORITY_MAX_CALLQUEUE_LENGTH,
priorityHandlerCount * RpcServer.DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER);
int maxQueueLength = conf.getInt(RpcScheduler.IPC_SERVER_MAX_CALLQUEUE_LENGTH, -1);

@d-c-manningd-c-manningJan 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we make the -1 a constant too? something like MAX_CALLQUEUE_LENGTH_UNDEFINED?

Suggested change
intmaxQueueLength = conf.getInt(RpcScheduler.IPC_SERVER_MAX_CALLQUEUE_LENGTH, -1);
intmaxQueueLength = conf.getInt(RpcScheduler.IPC_SERVER_MAX_CALLQUEUE_LENGTH, MAX_CALLQUEUE_LENGTH_UNDEFINED);

@Umeshkumar9414Umeshkumar9414Jan 13, 2026

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java
private static final QueueBalancer ONE_QUEUE = val -> 0;

public static QueueBalancer getBalancer(final String executorName, final Configuration conf,
protected static QueueBalancer getBalancer(final String executorName, final Configuration conf,

@d-c-manningd-c-manningJan 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this does seem like a useful visibility change, but must it be included here? Changing a public method of a "public" class (LimitedPrivate for COPROC, PHOENIX) is a breaking change. That being said, it's hard to imagine how anyone is using it outside of hbase... and the return class is not LimitedPrivate for COPROC... and I don't see it used in Phoenix... so 🤷‍♂️

@Umeshkumar9414Umeshkumar9414Jan 13, 2026

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method creates the balancer for RPC queues, so I don't think it should be visible outside of the protected visibility.

This was referenced Jan 13, 2026
Closed
Merged
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Umeshkumar9414

Umeshkumar9414 commented Jan 19, 2026

Copy link
Copy Markdown
ContributorAuthor

Both test - TestAcidGuaranteesWithBasicPolicy,TestAcidGuaranteesWithEagerPolicy, are working in local. Seems like flacky or ongoing issue.

@Apache-HBase

Apache-HBase commented Jan 21, 2026

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 31sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗codespell0m 0scodespell was not available.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
_ master Compile Tests _
+1 💚mvninstall3m 22smaster passed
+1 💚compile3m 26smaster passed
+1 💚checkstyle0m 59smaster passed
+1 💚spotbugs1m 41smaster passed
+1 💚spotless1m 4sbranch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚mvninstall2m 59sthe patch passed
+1 💚compile3m 28sthe patch passed
+1 💚javac3m 28sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 59shbase-server: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
+1 💚spotbugs1m 45sthe patch passed
+1 💚hadoopcheck11m 24sPatch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 💚spotless0m 45spatch has no errors when running spotless:check.
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
40m 8s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7490/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#7490
Optional Testsdupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
unameLinux e98b6ae26245 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 5d18bb6
Default JavaEclipse Adoptium-17.0.11+9
Max. process+thread count84 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7490/6/console
versionsgit=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 21, 2026

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec1m 40sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 29smaster passed
+1 💚compile0m 45smaster passed
+1 💚javadoc0m 22smaster passed
+1 💚shadedjars4m 16sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚mvninstall2m 15sthe patch passed
+1 💚compile0m 44sthe patch passed
+1 💚javac0m 44sthe patch passed
+1 💚javadoc0m 21sthe patch passed
+1 💚shadedjars4m 15spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
-1 ❌unit222m 27s/patch-unit-hbase-server.txthbase-server in the patch failed.
243m 25s
SubsystemReport/Notes
DockerClientAPI=1.48 ServerAPI=1.48 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7490/6/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR#7490
Optional Testsjavac javadoc unit compile shadedjars
unameLinux 6f26402413b9 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 5d18bb6
Default JavaEclipse Adoptium-17.0.11+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7490/6/testReport/
Max. process+thread count5772 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7490/6/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

apurtell
apurtell approved these changes Jan 21, 2026
View reviewed changes

@apurtellapurtell left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, will merge soon unless objection

@apurtell

apurtell commented Jan 21, 2026

Copy link
Copy Markdown
Contributor

In order to get clean cherry picks back to branch-2 based code, #7620 should be merged first.

@Umeshkumar9414

Umeshkumar9414 commented Jan 22, 2026

Copy link
Copy Markdown
ContributorAuthor

In order to get clean cherry picks back to branch-2 based code, #7620 should be merged first.
Sorry I made it confusing. typo in this msg #7652 shoud be merged first.

@apurtell
apurtell merged commit c770b92 into apache:masterJan 26, 2026
1 check failed
asf-gitbox-commits pushed a commit that referenced this pull request Jan 26, 2026
@Umeshkumar9414ukumawat@apurtell
HBASE-29141Calculate default maxQueueLength call queues correctly (#…
724c409
…7490)
Co-authored-by: ukumawat <ukumawat@salesforce.com>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
asf-gitbox-commits pushed a commit that referenced this pull request Jan 26, 2026
@Umeshkumar9414ukumawat@apurtell
HBASE-29141Calculate default maxQueueLength call queues correctly (#…
6b04a7a
…7490)
Co-authored-by: ukumawat <ukumawat@salesforce.com>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
asf-gitbox-commits pushed a commit that referenced this pull request Jan 26, 2026
@Umeshkumar9414ukumawat@apurtell
HBASE-29141Calculate default maxQueueLength call queues correctly (#…
43dbd89
…7490)
Co-authored-by: ukumawat <ukumawat@salesforce.com>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@apurtellapurtellapurtell approved these changes
+2 more reviewers
@mnpooniamnpooniamnpoonia approved these changes
@d-c-manningd-c-manningd-c-manning approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Umeshkumar9414@Apache-HBase@apurtell@mnpoonia@d-c-manning