Skip to content

HBASE-28012 Avoid CellUtil.cloneRow in BufferedEncodedSeeker - #5347

Merged
Apache9 merged 2 commits into
apache:masterfrom
jbewing:HBASE-28012-avoid-clone-row-in-dbe-seek
Aug 15, 2023
Merged

HBASE-28012 Avoid CellUtil.cloneRow in BufferedEncodedSeeker#5347
Apache9 merged 2 commits into
apache:masterfrom
jbewing:HBASE-28012-avoid-clone-row-in-dbe-seek

Conversation

@jbewing

@jbewingjbewing commented Aug 9, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

What

This PR updates BufferedEncodedSeeker#seekToKeyInBlock (used extensively in the reverse scan path) to avoid calling CellUtil.cloneRow for off-heap Cell seek targets.

Implementation Notes

For private "findCommonPrefix*" methods, I used an invariant that the "right" cell is always a keyOnlyKv and therefore on-heap.

For the public "compareCommonPrefix*" methods, I added extra paths to them to account for either the left or right cell to be on or off-heap to keep the behavior optimized for any external callers. In practice, I believe these methods are internal to BufferedEncodedSeeker except for a few unit tests so I'd be open to reducing the branching to just optimize for the left cell being able to be on/off-heap. I'd love some feedback on this approach as it adds a fair amount of extra code that isn't strictly necessary for this PR.

I also added a test that is a clone of a current seekToKeyInBlock over a sample of data with the only variation being that the seek target cell is an off-heap cell instead of an on-heap one to give coverage over the new path here. Let me know if you'd like to see any more tests.

Testing

Some (naive and non-JMH) testing has been done of the new code path here. It looks to be in the range of about 20-30% faster than the old path for off-heap cell seek targets. Obviously, take these numbers with a grain of salt as they're not from JMH.

HBASE-28012

@Apache-HBase

Apache-HBase commented Aug 9, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 25sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗mvndep0m 14sMaven dependency ordering for branch
+1 💚mvninstall2m 50smaster passed
+1 💚compile2m 56smaster passed
+1 💚checkstyle0m 46smaster passed
+1 💚spotless0m 42sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 52smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 10sMaven dependency ordering for patch
+1 💚mvninstall2m 33sthe patch passed
+1 💚compile2m 55sthe patch passed
+1 💚javac2m 55sthe patch passed
+1 💚checkstyle0m 45sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck9m 32sPatch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚spotless0m 39spatch has no errors when running spotless:check.
+1 💚spotbugs2m 12sthe patch passed
_ Other Tests _
+1 💚asflicense0m 16sThe patch does not generate ASF License warnings.
34m 48s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5347
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux cf8bb8302e44 5.4.0-1101-aws #109~18.04.1-Ubuntu SMP Mon Apr 24 20:40:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7cfa47d
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count80 (vs. ulimit of 30000)
modulesC: hbase-common hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 9, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 25sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for branch
+1 💚mvninstall2m 33smaster passed
+1 💚compile0m 53smaster passed
+1 💚shadedjars5m 4sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 36smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for patch
+1 💚mvninstall2m 20sthe patch passed
+1 💚compile0m 52sthe patch passed
+1 💚javac0m 52sthe patch passed
+1 💚shadedjars5m 0spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 34sthe patch passed
_ Other Tests _
+1 💚unit1m 45shbase-common in the patch passed.
+1 💚unit219m 13shbase-server in the patch passed.
244m 12s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5347
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 3848d38a8d87 5.4.0-1101-aws #109~18.04.1-Ubuntu SMP Mon Apr 24 20:40:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7cfa47d
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/1/testReport/
Max. process+thread count4613 (vs. ulimit of 30000)
modulesC: hbase-common hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 9, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 28sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for branch
+1 💚mvninstall2m 43smaster passed
+1 💚compile1m 5smaster passed
+1 💚shadedjars4m 38sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 43smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 14sMaven dependency ordering for patch
+1 💚mvninstall2m 34sthe patch passed
+1 💚compile1m 5sthe patch passed
+1 💚javac1m 5sthe patch passed
+1 💚shadedjars4m 35spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 42sthe patch passed
_ Other Tests _
+1 💚unit2m 9shbase-common in the patch passed.
+1 💚unit222m 39shbase-server in the patch passed.
248m 29s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5347
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 6a821be5ea50 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7cfa47d
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/1/testReport/
Max. process+thread count4386 (vs. ulimit of 30000)
modulesC: hbase-common hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9
Apache9 reviewed Aug 13, 2023
View reviewed changes
Comment threadhbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
Comment threadhbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java
@Apache9

Apache9 commented Aug 13, 2023

Copy link
Copy Markdown
Contributor

Thanks for opening this PR.

Overall LGTM. Just a simple question and a follow on task.

@jbewing
jbewing requested a review from Apache9August 14, 2023 21:34
@Apache-HBase

Apache-HBase commented Aug 14, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 10sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for branch
+1 💚mvninstall2m 45smaster passed
+1 💚compile2m 56smaster passed
+1 💚checkstyle0m 52smaster passed
+1 💚spotless0m 43sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 58smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for patch
+1 💚mvninstall2m 29sthe patch passed
+1 💚compile2m 57sthe patch passed
+1 💚javac2m 57sthe patch passed
+1 💚checkstyle0m 49sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck8m 58sPatch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚spotless0m 41spatch has no errors when running spotless:check.
+1 💚spotbugs2m 14sthe patch passed
_ Other Tests _
+1 💚asflicense0m 19sThe patch does not generate ASF License warnings.
34m 0s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5347
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 3b10fdde423f 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 20c9e4b
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count80 (vs. ulimit of 30000)
modulesC: hbase-common hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 15, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 24sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 30sMaven dependency ordering for branch
+1 💚mvninstall2m 25smaster passed
+1 💚compile0m 49smaster passed
+1 💚shadedjars4m 53sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 33smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 10sMaven dependency ordering for patch
+1 💚mvninstall2m 14sthe patch passed
+1 💚compile0m 49sthe patch passed
+1 💚javac0m 49sthe patch passed
+1 💚shadedjars4m 51spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 32sthe patch passed
_ Other Tests _
+1 💚unit1m 43shbase-common in the patch passed.
+1 💚unit215m 48shbase-server in the patch passed.
239m 57s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5347
Optional Testsjavac javadoc unit shadedjars compile
unameLinux a0145901c91f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 20c9e4b
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/2/testReport/
Max. process+thread count4976 (vs. ulimit of 30000)
modulesC: hbase-common hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 15, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 43sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 12sMaven dependency ordering for branch
+1 💚mvninstall3m 30smaster passed
+1 💚compile1m 19smaster passed
+1 💚shadedjars5m 28sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 48smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for patch
+1 💚mvninstall3m 15sthe patch passed
+1 💚compile1m 16sthe patch passed
+1 💚javac1m 16sthe patch passed
+1 💚shadedjars5m 43spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 48sthe patch passed
_ Other Tests _
+1 💚unit2m 35shbase-common in the patch passed.
+1 💚unit256m 33shbase-server in the patch passed.
287m 29s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5347
Optional Testsjavac javadoc unit shadedjars compile
unameLinux e3be821c9669 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 20c9e4b
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/2/testReport/
Max. process+thread count4410 (vs. ulimit of 30000)
modulesC: hbase-common hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5347/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9
Apache9 merged commit 2fb2ae1 into apache:masterAug 15, 2023
Apache9 pushed a commit that referenced this pull request Aug 15, 2023
@jbewing@Apache9
HBASE-28012Avoid CellUtil.cloneRow in BufferedEncodedSeeker (#5347)
9aa8b9b
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 2fb2ae1)
Apache9 pushed a commit that referenced this pull request Aug 15, 2023
@jbewing@Apache9
HBASE-28012Avoid CellUtil.cloneRow in BufferedEncodedSeeker (#5347)
e61207a
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 2fb2ae1)
Apache9 pushed a commit that referenced this pull request Aug 15, 2023
@jbewing@Apache9
HBASE-28012Avoid CellUtil.cloneRow in BufferedEncodedSeeker (#5347)
bf945f2
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 2fb2ae1)
Apache9 pushed a commit that referenced this pull request Aug 15, 2023
@jbewing@Apache9
HBASE-28012Avoid CellUtil.cloneRow in BufferedEncodedSeeker (#5347)
a5aab42
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 2fb2ae1)
bbeaudreault pushed a commit to HubSpot/hbase that referenced this pull request Aug 21, 2023
@jbewing@bbeaudreault
HubSpot Backport: HBASE-28012 Avoid CellUtil.cloneRow in BufferedEnco…
328e339
…dedSeeker (apache#5347)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 2fb2ae1)
vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
@jbewingJenkins
HBASE-28012 Avoid CellUtil.cloneRow in BufferedEncodedSeeker (apache#…
22c8dd2
…5347)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 2fb2ae1)
(cherry picked from commit 9aa8b9b)
Change-Id: I246d035e9138fa4c0927b6a0ecbaf1bc9f9a31ad
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Apache9Apache9Apache9 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jbewing@Apache-HBase@Apache9