Skip to content

HBASE-25510 Optimize TableName.valueOf from O(n) to O(1) - #2885

Open
zhengzhuobinzzb wants to merge 1 commit into
apache:masterfrom
zhengzhuobinzzb:tablename-optimize
Open

HBASE-25510 Optimize TableName.valueOf from O(n) to O(1)#2885
zhengzhuobinzzb wants to merge 1 commit into
apache:masterfrom
zhengzhuobinzzb:tablename-optimize

Conversation

@zhengzhuobinzzb

Copy link
Copy Markdown
Contributor

Optimize TableName.valueOf from O(n) to O(1). We can get benefits when the number of tables in the cluster is greater than dozens

@saintstack

Copy link
Copy Markdown
Contributor

I edited the summary to remove square brackets around the JIRA number.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec3m 55sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 43smaster passed
+1 💚compile0m 30smaster passed
+1 💚shadedjars7m 34sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 22smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 26sthe patch passed
+1 💚compile0m 24sthe patch passed
+1 💚javac0m 24sthe patch passed
+1 💚shadedjars7m 55spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25sthe patch passed
_ Other Tests _
+1 💚unit1m 48shbase-common in the patch passed.
33m 0s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2885
JIRA IssueHBASE-25510
Optional Testsjavac javadoc unit shadedjars compile
unameLinux b26393b36a26 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 3cc2468
Default JavaAdoptOpenJDK-1.8.0_232-b09
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/testReport/
Max. process+thread count227 (vs. ulimit of 30000)
modulesC: hbase-common U: hbase-common
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 44sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall5m 29smaster passed
+1 💚compile0m 30smaster passed
+1 💚shadedjars8m 5sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 35smaster passed
_ Patch Compile Tests _
+1 💚mvninstall5m 3sthe patch passed
+1 💚compile0m 27sthe patch passed
+1 💚javac0m 27sthe patch passed
+1 💚shadedjars7m 58spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 31sthe patch passed
_ Other Tests _
+1 💚unit2m 30shbase-common in the patch passed.
33m 53s
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2885
JIRA IssueHBASE-25510
Optional Testsjavac javadoc unit shadedjars compile
unameLinux e3d8ce757d3f 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 3cc2468
Default JavaAdoptOpenJDK-11.0.6+10
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/testReport/
Max. process+thread count205 (vs. ulimit of 30000)
modulesC: hbase-common U: hbase-common
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@zhengzhuobinzzbzhengzhuobinzzb changed the title [HBASE-25510] Optimize TableName.valueOf from O(n) to O(1)HBASE-25510 Optimize TableName.valueOf from O(n) to O(1)Jan 15, 2021
@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 8sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 23smaster passed
+1 💚compile0m 48smaster passed
+1 💚checkstyle0m 25smaster passed
+1 💚spotbugs0m 45smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 47sthe patch passed
+1 💚compile0m 45sthe patch passed
+1 💚javac0m 45sthe patch passed
-0 ⚠️checkstyle0m 24shbase-common: The patch generated 2 new + 17 unchanged - 0 fixed = 19 total (was 17)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck21m 25sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs0m 54sthe patch passed
_ Other Tests _
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
42m 43s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2885
JIRA IssueHBASE-25510
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux a852dd790e34 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 3cc2468
Default JavaAdoptOpenJDK-1.8.0_232-b09
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt
Max. process+thread count85 (vs. ulimit of 30000)
modulesC: hbase-common U: hbase-common
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9

Copy link
Copy Markdown
Contributor

It is a bit strange that we convert bytes or ByteBuffer to String, and then convert back to store them...

@zhengzhuobinzzb

Copy link
Copy Markdown
ContributorAuthor

It is a bit strange that we convert bytes or ByteBuffer to String, and then convert back to store them...
Thank you very much for reviewing my code.

In face, i tryed to use ByteBuffer as hash key. But there ars some reasons i choose String finally.

  1. Because we use hashmap, so we must choose a class as hash key. Therefore, we must convert one of String and ByteBuffer to the other.
  2. Use ByteBuffer alse need to copy underlying array. If not, might cause memory leak.
  3. The code for construct hash key(${namespace} + ':' + ${tablename}), String is more concise than ByteBuffer.
  4. In my benchmark(code can find in jira attachment). The efficiency of String is a little better than ByteBuffer overall.

Benchmark Result For ByteBuffer:
Benchmark Mode Cnt Score Error Units
TestTableNameJMH.testBB1 thrpt 10 73319.634 ± 5162.112 ops/ms
TestTableNameJMH.testBB10 thrpt 10 69742.749 ± 1238.255 ops/ms
TestTableNameJMH.testBB100 thrpt 10 65220.820 ± 3357.727 ops/ms
TestTableNameJMH.testBB1000 thrpt 10 47865.385 ± 2510.229 ops/ms
TestTableNameJMH.testBB10000 thrpt 10 29383.002 ± 5535.025 ops/ms
TestTableNameJMH.testBB100000 thrpt 10 15333.042 ± 3451.444 ops/ms
TestTableNameJMH.testStr1 thrpt 10 17342.604 ± 2997.854 ops/ms
TestTableNameJMH.testStr10 thrpt 10 18562.438 ± 1984.190 ops/ms
TestTableNameJMH.testStr100 thrpt 10 16152.466 ± 3349.675 ops/ms
TestTableNameJMH.testStr1000 thrpt 10 14944.087 ± 1132.673 ops/ms
TestTableNameJMH.testStr10000 thrpt 10 9231.439 ± 2480.242 ops/ms
TestTableNameJMH.testStr100000 thrpt 10 7458.192 ± 1046.736 ops/ms

Benchmark Result For String:
Benchmark Mode Cnt Score Error Units
TestTableNameJMH.testBB1 thrpt 10 20719.730 ± 987.327 ops/ms
TestTableNameJMH.testBB10 thrpt 10 20698.180 ± 240.704 ops/ms
TestTableNameJMH.testBB100 thrpt 10 19331.825 ± 232.413 ops/ms
TestTableNameJMH.testBB1000 thrpt 10 18865.971 ± 260.727 ops/ms
TestTableNameJMH.testBB10000 thrpt 10 13420.776 ± 1473.567 ops/ms
TestTableNameJMH.testBB100000 thrpt 10 7209.563 ± 2928.791 ops/ms
TestTableNameJMH.testStr1 thrpt 10 133099.686 ± 5902.544 ops/ms
TestTableNameJMH.testStr10 thrpt 10 141280.563 ± 7157.877 ops/ms
TestTableNameJMH.testStr100 thrpt 10 127057.065 ± 11665.471 ops/ms
TestTableNameJMH.testStr1000 thrpt 10 85985.687 ± 18432.481 ops/ms
TestTableNameJMH.testStr10000 thrpt 10 51825.915 ± 1824.001 ops/ms
TestTableNameJMH.testStr100000 thrpt 10 19002.112 ± 3337.013 ops/ms

@Apache9

Copy link
Copy Markdown
Contributor

It is a bit strange that we convert bytes or ByteBuffer to String, and then convert back to store them...
Thank you very much for reviewing my code.

In face, i tryed to use ByteBuffer as hash key. But there ars some reasons i choose String finally.

  1. Because we use hashmap, so we must choose a class as hash key. Therefore, we must convert one of String and ByteBuffer to the other.
  2. Use ByteBuffer alse need to copy underlying array. If not, might cause memory leak.
  3. The code for construct hash key(${namespace} + ':' + ${tablename}), String is more concise than ByteBuffer.
  4. In my benchmark(code can find in jira attachment). The efficiency of String is a little better than ByteBuffer overall.

Benchmark Result For ByteBuffer:
Benchmark Mode Cnt Score Error Units
TestTableNameJMH.testBB1 thrpt 10 73319.634 ± 5162.112 ops/ms
TestTableNameJMH.testBB10 thrpt 10 69742.749 ± 1238.255 ops/ms
TestTableNameJMH.testBB100 thrpt 10 65220.820 ± 3357.727 ops/ms
TestTableNameJMH.testBB1000 thrpt 10 47865.385 ± 2510.229 ops/ms
TestTableNameJMH.testBB10000 thrpt 10 29383.002 ± 5535.025 ops/ms
TestTableNameJMH.testBB100000 thrpt 10 15333.042 ± 3451.444 ops/ms
TestTableNameJMH.testStr1 thrpt 10 17342.604 ± 2997.854 ops/ms
TestTableNameJMH.testStr10 thrpt 10 18562.438 ± 1984.190 ops/ms
TestTableNameJMH.testStr100 thrpt 10 16152.466 ± 3349.675 ops/ms
TestTableNameJMH.testStr1000 thrpt 10 14944.087 ± 1132.673 ops/ms
TestTableNameJMH.testStr10000 thrpt 10 9231.439 ± 2480.242 ops/ms
TestTableNameJMH.testStr100000 thrpt 10 7458.192 ± 1046.736 ops/ms

Benchmark Result For String:
Benchmark Mode Cnt Score Error Units
TestTableNameJMH.testBB1 thrpt 10 20719.730 ± 987.327 ops/ms
TestTableNameJMH.testBB10 thrpt 10 20698.180 ± 240.704 ops/ms
TestTableNameJMH.testBB100 thrpt 10 19331.825 ± 232.413 ops/ms
TestTableNameJMH.testBB1000 thrpt 10 18865.971 ± 260.727 ops/ms
TestTableNameJMH.testBB10000 thrpt 10 13420.776 ± 1473.567 ops/ms
TestTableNameJMH.testBB100000 thrpt 10 7209.563 ± 2928.791 ops/ms
TestTableNameJMH.testStr1 thrpt 10 133099.686 ± 5902.544 ops/ms
TestTableNameJMH.testStr10 thrpt 10 141280.563 ± 7157.877 ops/ms
TestTableNameJMH.testStr100 thrpt 10 127057.065 ± 11665.471 ops/ms
TestTableNameJMH.testStr1000 thrpt 10 85985.687 ± 18432.481 ops/ms
TestTableNameJMH.testStr10000 thrpt 10 51825.915 ± 1824.001 ops/ms
TestTableNameJMH.testStr100000 thrpt 10 19002.112 ± 3337.013 ops/ms

I think the problem is we convert it back to bytes.

Could we introduce an internal methods which takes both string and bytes? So after checking the cache, we need to use bytes to create a new TableName, we just use it directly instead of converting the String back.

@zhengzhuobinzzb

Copy link
Copy Markdown
ContributorAuthor

I think the problem is we convert it back to bytes.

Could we introduce an internal methods which takes both string and bytes? So after checking the cache, we need to use bytes to create a new TableName, we just use it directly instead of converting the String back.

The matter is we can't believe outside bytes(because it can be modifyed outside). So we have to go through two conversions. The diff is below:

  1. bytes -> string -> bytes
  2. bytes -> string + bytes -> inner bytes

So, I think we choose 1 here is not a problem

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 59sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 19smaster passed
+1 💚compile0m 23smaster passed
+1 💚shadedjars8m 58sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 22smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 5sthe patch passed
+1 💚compile0m 23sthe patch passed
+1 💚javac0m 23sthe patch passed
+1 💚shadedjars8m 58spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 20sthe patch passed
_ Other Tests _
+1 💚unit1m 50shbase-common in the patch passed.
31m 36s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2885
Optional Testsjavac javadoc unit shadedjars compile
unameLinux d8aec776018a 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 0955a7a
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/testReport/
Max. process+thread count222 (vs. ulimit of 30000)
modulesC: hbase-common U: hbase-common
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 38sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall5m 29smaster passed
+1 💚compile0m 33smaster passed
+1 💚shadedjars9m 26sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 30smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 54sthe patch passed
+1 💚compile0m 28sthe patch passed
+1 💚javac0m 28sthe patch passed
+1 💚shadedjars9m 2spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24sthe patch passed
_ Other Tests _
+1 💚unit2m 46shbase-common in the patch passed.
36m 12s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2885
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 34a65c8e5acd 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 0955a7a
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/testReport/
Max. process+thread count213 (vs. ulimit of 30000)
modulesC: hbase-common U: hbase-common
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 28sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 56smaster passed
+1 💚compile0m 51smaster passed
+1 💚checkstyle0m 26smaster passed
+1 💚spotbugs0m 46smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 34sthe patch passed
+1 💚compile0m 48sthe patch passed
+1 💚javac0m 48sthe patch passed
-0 ⚠️checkstyle0m 25shbase-common: The patch generated 2 new + 17 unchanged - 0 fixed = 19 total (was 17)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck18m 2sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs0m 55sthe patch passed
_ Other Tests _
+1 💚asflicense0m 15sThe patch does not generate ASF License warnings.
38m 14s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2885
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux e6b40efee63a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 0955a7a
Default JavaAdoptOpenJDK-1.8.0_282-b08
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt
Max. process+thread count96 (vs. ulimit of 30000)
modulesC: hbase-common U: hbase-common
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2885/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9

Copy link
Copy Markdown
Contributor

I think the problem is we convert it back to bytes.
Could we introduce an internal methods which takes both string and bytes? So after checking the cache, we need to use bytes to create a new TableName, we just use it directly instead of converting the String back.

The matter is we can't believe outside bytes(because it can be modifyed outside). So we have to go through two conversions. The diff is below:

  1. bytes -> string -> bytes
  2. bytes -> string + bytes -> inner bytes

So, I think we choose 1 here is not a problem

In the constructor of TableName, we will copy the bytes out. So let's avoid the extra converting?

@Apache9

Copy link
Copy Markdown
Contributor

Any updates here?

Thanks.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@zhengzhuobinzzb@saintstack@Apache-HBase@Apache9