Skip to content

HBASE-29252: Reduce allocations in RowIndexSeekerV1 - #6902

Merged
ndimiduk merged 2 commits into
apache:masterfrom
HubSpot:HBASE-29252/row-index-v1-allocations
Apr 25, 2025
Merged

HBASE-29252: Reduce allocations in RowIndexSeekerV1#6902
ndimiduk merged 2 commits into
apache:masterfrom
HubSpot:HBASE-29252/row-index-v1-allocations

Conversation

@charlesconnell

@charlesconnellcharlesconnell commented Apr 11, 2025

Copy link
Copy Markdown
Contributor

I've looked at a lot of allocation profiles of RegionServers doing a read-heavy workload. Some allocations that dominate the chart can be easily avoided.

The following code in the main decode method

currentBuffer.asSubByteBuffer(currentBuffer.position(), current.keyLength, tmpPair);
ByteBuffer key = tmpPair.getFirst().duplicate();
key.position(tmpPair.getSecond()).limit(tmpPair.getSecond() + current.keyLength);
current.keyBuffer = key; 

results in a new ByteBuffer for every cell. The reason to have this duplicate ByteBuffer is to hold the result of tmpPair.getSecond() as its position state. But this is just an integer that can be more cheaply stored in a different way. We can introduce a current.keyOffset variable and do this instead:

currentBuffer.asSubByteBuffer(currentBuffer.position(), current.keyLength, tmpPair);
current.keyBuffer = tmpPair.getFirst();
current.keyOffset = tmpPair.getSecond();

and then reference current.keyOffset where we previously referenced current.keyBuffer.position().

Additionally, RowIndexSeekerV1.SeekerState contains a ByteBufferKeyOnlyKeyValue field that is replaced on every cell read. This object can be reset and re-used instead.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache9
Apache9 requested a review from CopilotApril 12, 2025 06:21

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

@charlesconnell
charlesconnellforce-pushed the HBASE-29252/row-index-v1-allocations branch from 05789be to 22e8d01CompareApril 12, 2025 17:54
@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@charlesconnell

Copy link
Copy Markdown
ContributorAuthor

test failures are unrelated

protected int nextKvOffset;
// buffer backed keyonlyKV
// buffer backed keyonlyKV, reset and re-used as necessary to avoid allocations
private ByteBufferKeyOnlyKeyValue currentKey = new ByteBufferKeyOnlyKeyValue();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make it final if we do not need to recreate it?


protected ByteBuff currentBuffer;
protected int startOffset = -1;
protected int keyOffset = -1;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better add some comments to explain what do these offsets and lengths mean?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I won't request further changes, but for future reference, these comments would ideally be javadoc and then this additional information would be more readily onhand for folks pursuing the code in an IDE environment.

protected int valueLength;
// Tags start after values and end after tagsLength
protected int tagsLength = 0;
protected int tagsOffset = -1;

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that tagsOffset was effectively unused so I've removed it

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 29sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗codespell0m 0scodespell was not available.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
_ master Compile Tests _
+0 🆗mvndep0m 15sMaven dependency ordering for branch
+1 💚mvninstall5m 10smaster passed
+1 💚compile5m 30smaster passed
+1 💚checkstyle1m 17smaster passed
+1 💚spotbugs3m 18smaster passed
+1 💚spotless1m 12sbranch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 🆗mvndep0m 15sMaven dependency ordering for patch
+1 💚mvninstall5m 9sthe patch passed
+1 💚compile8m 10sthe patch passed
+1 💚javac8m 10sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle2m 3sthe patch passed
+1 💚spotbugs5m 19sthe patch passed
+1 💚hadoopcheck21m 19sPatch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚spotless2m 0spatch has no errors when running spotless:check.
_ Other Tests _
+1 💚asflicense1m 10sThe patch does not generate ASF License warnings.
76m 2s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6902/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#6902
JIRA IssueHBASE-29252
Optional Testsdupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
unameLinux 0a7f92a72c35 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / db17a2b
Default JavaEclipse Adoptium-17.0.11+9
Max. process+thread count85 (vs. ulimit of 30000)
modulesC: hbase-common hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6902/3/console
versionsgit=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 28sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for branch
+1 💚mvninstall3m 28smaster passed
+1 💚compile1m 21smaster passed
+1 💚javadoc0m 47smaster passed
+1 💚shadedjars6m 6sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for patch
+1 💚mvninstall3m 23sthe patch passed
+1 💚compile1m 26sthe patch passed
+1 💚javac1m 26sthe patch passed
+1 💚javadoc0m 48sthe patch passed
+1 💚shadedjars6m 16spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit2m 26shbase-common in the patch passed.
+1 💚unit233m 19shbase-server in the patch passed.
265m 57s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6902/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR#6902
JIRA IssueHBASE-29252
Optional Testsjavac javadoc unit compile shadedjars
unameLinux 486b25b924e4 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / db17a2b
Default JavaEclipse Adoptium-17.0.11+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6902/3/testReport/
Max. process+thread count5185 (vs. ulimit of 30000)
modulesC: hbase-common hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6902/3/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.


protected ByteBuff currentBuffer;
protected int startOffset = -1;
protected int keyOffset = -1;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I won't request further changes, but for future reference, these comments would ideally be javadoc and then this additional information would be more readily onhand for folks pursuing the code in an IDE environment.

@ndimiduk
ndimiduk merged commit 30ce21b into apache:masterApr 25, 2025
@ndimiduk
ndimiduk deleted the HBASE-29252/row-index-v1-allocations branch April 25, 2025 09:45
ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Apr 25, 2025
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Apr 25, 2025
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Apr 25, 2025
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Apr 25, 2025
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit that referenced this pull request Apr 26, 2025
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit that referenced this pull request Apr 28, 2025
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit that referenced this pull request Apr 28, 2025
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit that referenced this pull request Apr 28, 2025
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
mokai87 pushed a commit to mokai87/hbase that referenced this pull request Aug 7, 2025
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
sanjeet006py pushed a commit to sanjeet006py/hbase that referenced this pull request Sep 26, 2025
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@charlesconnell@Apache-HBase@ndimiduk@Apache9