Skip to content

HBASE-25364 Redo the getMidPoint() in HFileWriterImpl to get rid of t… - #2741

Merged
saintstack merged 1 commit into
apache:masterfrom
GeorryHuang:getMidPoint
Feb 10, 2021
Merged

HBASE-25364 Redo the getMidPoint() in HFileWriterImpl to get rid of t…#2741
saintstack merged 1 commit into
apache:masterfrom
GeorryHuang:getMidPoint

Conversation

@GeorryHuang

Copy link
Copy Markdown
Contributor

…he double comparison process

There is a TODO like this "TODO: Redo so only a single pass over the arrays rather than one to compare and then a second composing midpoint." in getMidpoint() of class ​HFileWriteImpl​

The old logic compares the left byte array and the right byte array twice:

  1. A comparison is performed before composing MinimumMidpointArray
  2. During composing of MinimumMidpointArray, bytes were comparing again

My optimization combines them into one

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 8sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 9smaster passed
+1 💚checkstyle1m 12smaster passed
+1 💚spotbugs2m 11smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 47sthe patch passed
-0 ⚠️checkstyle1m 9shbase-server: The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck19m 7sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 18sthe patch passed
_ Other Tests _
+1 💚asflicense0m 13sThe patch does not generate ASF License warnings.
43m 5s
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2741/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2741
Optional Testsdupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
unameLinux c64fb3ef40e9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7d0a687
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2741/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count84 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2741/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 6sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 17smaster passed
+1 💚compile1m 5smaster passed
+1 💚shadedjars6m 42sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 42smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 1sthe patch passed
+1 💚compile1m 4sthe patch passed
+1 💚javac1m 4sthe patch passed
+1 💚shadedjars6m 35spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 38sthe patch passed
_ Other Tests _
+1 💚unit137m 39shbase-server in the patch passed.
165m 53s
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2741/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2741
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 4e341520fd43 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7d0a687
Default JavaAdoptOpenJDK-11.0.6+10
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2741/1/testReport/
Max. process+thread count4231 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2741/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 6sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 12smaster passed
+1 💚compile0m 59smaster passed
+1 💚shadedjars7m 10sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 39smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 50sthe patch passed
+1 💚compile1m 0sthe patch passed
+1 💚javac1m 0sthe patch passed
+1 💚shadedjars7m 9spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 36sthe patch passed
_ Other Tests _
+1 💚unit204m 1shbase-server in the patch passed.
232m 29s
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2741/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2741
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 0e740dfd9e37 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7d0a687
Default JavaAdoptOpenJDK-1.8.0_232-b09
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2741/1/testReport/
Max. process+thread count3241 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2741/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

}
}
return minMidpoint;
//Note that left[diffIdx] can never be equal to 0xff since left < right

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic here is different from original?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic of composing midPoint is basically the same, with some small differences

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(diffByte + 1) < (ByteBufferUtils.toByte(right, rightOffset + diffIdx) & 0xff)
I think this logical branch is lost

byte[] minimumMidpointArray = new byte[minLength + 1];
ByteBufferUtils
.copyFromBufferToArray(minimumMidpointArray, right, rightOffset, 0, minLength + 1);
minimumMidpointArray[minLength] = 0x00;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is different from original?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I made some adjustments.

@saintstacksaintstack left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skimmed. I like your addition of extra boundary checking test. LGTM.

*/
public static Cell getMidpoint(final CellComparator comparator, final Cell left,
final Cell right) {
// TODO: Redo so only a single pass over the arrays rather than one to

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From HBASE-10800 ... By our Ramkrishna

@GeorryHuang

Copy link
Copy Markdown
ContributorAuthor

@saintstack Thanks for reviewing

@saintstack

Copy link
Copy Markdown
Contributor

@nyl3532016 You good w/ this change? If so, would like to add you as a sign-off. Thanks.

@nyl3532016

Copy link
Copy Markdown
Contributor

@nyl3532016 You good w/ this change? If so, would like to add you as a sign-off. Thanks.

@saintstack, I am good with the patch after @GeorryHuang amend a bit

@saintstack
saintstack merged commit 3e743df into apache:masterFeb 10, 2021
saintstack pushed a commit that referenced this pull request Feb 10, 2021
…he double comparison process (#2741)
Signed-off-by: niuyulin <nyl353@163.com>
Signed-off-by: stack <stack@apache.org>
ddupg pushed a commit to ddupg/hbase that referenced this pull request Feb 19, 2021
…he double comparison process (apache#2741)
Signed-off-by: niuyulin <nyl353@163.com>
Signed-off-by: stack <stack@apache.org>
wchevreuil pushed a commit to wchevreuil/hbase that referenced this pull request Feb 24, 2021
…he double comparison process (apache#2741)
Signed-off-by: niuyulin <nyl353@163.com>
Signed-off-by: stack <stack@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@GeorryHuang@Apache-HBase@saintstack@nyl3532016