Skip to content

HBASE-25894 Improve the performance for region load and region count related cost functions - #3276

Merged
Apache9 merged 1 commit into
apache:masterfrom
Apache9:HBASE-25894
May 25, 2021
Merged

HBASE-25894 Improve the performance for region load and region count related cost functions#3276
Apache9 merged 1 commit into
apache:masterfrom
Apache9:HBASE-25894

Conversation

@Apache9

@Apache9Apache9 commented May 18, 2021

Copy link
Copy Markdown
Contributor

No description provided.

@Apache9

Apache9 commented May 18, 2021

Copy link
Copy Markdown
ContributorAuthor

Depends on #3274

@Apache9Apache9 self-assigned this May 18, 2021
@Apache-HBase

Apache-HBase commented May 18, 2021

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 28sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 24sMaven dependency ordering for branch
+1 💚mvninstall3m 59smaster passed
+1 💚compile1m 22smaster passed
+1 💚shadedjars8m 40sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 58smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 15sMaven dependency ordering for patch
+1 💚mvninstall3m 43sthe patch passed
+1 💚compile1m 19sthe patch passed
+1 💚javac1m 19sthe patch passed
+1 💚shadedjars8m 31spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 55sthe patch passed
_ Other Tests _
+1 💚unit0m 31shbase-balancer in the patch passed.
-1 ❌unit8m 2shbase-server in the patch failed.
40m 52s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3276
Optional Testsjavac javadoc unit shadedjars compile
unameLinux fc76517af058 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 741b4b4
Default JavaAdoptOpenJDK-1.8.0_282-b08
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/testReport/
Max. process+thread count1146 (vs. ulimit of 30000)
modulesC: hbase-balancer hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented May 18, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 1sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗mvndep0m 23sMaven dependency ordering for branch
+1 💚mvninstall3m 50smaster passed
+1 💚compile3m 44smaster passed
+1 💚checkstyle1m 16smaster passed
+1 💚spotbugs2m 36smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for patch
+1 💚mvninstall3m 36sthe patch passed
+1 💚compile3m 38sthe patch passed
+1 💚javac3m 38sthe patch passed
+1 💚checkstyle0m 14sThe patch passed checkstyle in hbase-balancer
+1 💚checkstyle1m 2shbase-server: The patch generated 0 new + 6 unchanged - 1 fixed = 6 total (was 7)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck18m 23sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 55sthe patch passed
_ Other Tests _
+1 💚asflicense0m 26sThe patch does not generate ASF License warnings.
51m 26s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3276
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 50484385897a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 741b4b4
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count96 (vs. ulimit of 30000)
modulesC: hbase-balancer hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented May 18, 2021

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec3m 8sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 22sMaven dependency ordering for branch
+1 💚mvninstall5m 17smaster passed
+1 💚compile1m 54smaster passed
+1 💚shadedjars9m 11sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 1smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 14sMaven dependency ordering for patch
+1 💚mvninstall5m 1sthe patch passed
+1 💚compile1m 55sthe patch passed
+1 💚javac1m 55sthe patch passed
+1 💚shadedjars9m 42spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 1sthe patch passed
_ Other Tests _
+1 💚unit0m 33shbase-balancer in the patch passed.
-1 ❌unit13m 3shbase-server in the patch failed.
54m 0s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3276
Optional Testsjavac javadoc unit shadedjars compile
unameLinux b2c8d9601be4 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 741b4b4
Default JavaAdoptOpenJDK-11.0.10+9
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/testReport/
Max. process+thread count649 (vs. ulimit of 30000)
modulesC: hbase-balancer hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented May 18, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 59sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 1sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗mvndep0m 14sMaven dependency ordering for branch
+1 💚mvninstall3m 34smaster passed
+1 💚compile3m 37smaster passed
+1 💚checkstyle1m 16smaster passed
+1 💚spotbugs2m 33smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for patch
+1 💚mvninstall3m 35sthe patch passed
+1 💚compile3m 36sthe patch passed
+1 💚javac3m 36sthe patch passed
+1 💚checkstyle0m 14sThe patch passed checkstyle in hbase-balancer
+1 💚checkstyle1m 1shbase-server: The patch generated 0 new + 6 unchanged - 1 fixed = 6 total (was 7)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck18m 4sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 55sthe patch passed
_ Other Tests _
+1 💚asflicense0m 25sThe patch does not generate ASF License warnings.
50m 30s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3276
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 9d7344048438 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 741b4b4
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count96 (vs. ulimit of 30000)
modulesC: hbase-balancer hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented May 18, 2021

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 28sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for branch
+1 💚mvninstall3m 40smaster passed
+1 💚compile1m 20smaster passed
+1 💚shadedjars8m 11sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 52smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 17sMaven dependency ordering for patch
+1 💚mvninstall3m 45sthe patch passed
+1 💚compile1m 20sthe patch passed
+1 💚javac1m 20sthe patch passed
+1 💚shadedjars8m 32spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 58sthe patch passed
_ Other Tests _
+1 💚unit0m 30shbase-balancer in the patch passed.
-1 ❌unit150m 53shbase-server in the patch failed.
183m 27s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3276
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 127d326e0ee5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 741b4b4
Default JavaAdoptOpenJDK-1.8.0_282-b08
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/testReport/
Max. process+thread count3342 (vs. ulimit of 30000)
modulesC: hbase-balancer hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented May 18, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 21sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 12sMaven dependency ordering for branch
+1 💚mvninstall5m 24smaster passed
+1 💚compile1m 56smaster passed
+1 💚shadedjars9m 55sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 0smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 16sMaven dependency ordering for patch
+1 💚mvninstall5m 43sthe patch passed
+1 💚compile1m 56sthe patch passed
+1 💚javac1m 56sthe patch passed
+1 💚shadedjars9m 55spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 6sthe patch passed
_ Other Tests _
+1 💚unit0m 32shbase-balancer in the patch passed.
+1 💚unit224m 7shbase-server in the patch passed.
266m 10s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3276
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 745eba1f14a9 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 741b4b4
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/testReport/
Max. process+thread count2778 (vs. ulimit of 30000)
modulesC: hbase-balancer hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented May 19, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 29sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 1sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗mvndep0m 22sMaven dependency ordering for branch
+1 💚mvninstall4m 10smaster passed
+1 💚compile3m 39smaster passed
+1 💚checkstyle1m 19smaster passed
+1 💚spotbugs2m 39smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 14sMaven dependency ordering for patch
+1 💚mvninstall3m 39sthe patch passed
+1 💚compile3m 38sthe patch passed
+1 💚javac3m 38sthe patch passed
+1 💚checkstyle0m 13sThe patch passed checkstyle in hbase-balancer
+1 💚checkstyle1m 3shbase-server: The patch generated 0 new + 6 unchanged - 1 fixed = 6 total (was 7)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck17m 58sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 57sthe patch passed
_ Other Tests _
+1 💚asflicense0m 27sThe patch does not generate ASF License warnings.
50m 45s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3276
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 97dda1208770 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 5b99409
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count96 (vs. ulimit of 30000)
modulesC: hbase-balancer hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented May 19, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 28sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 23sMaven dependency ordering for branch
+1 💚mvninstall3m 52smaster passed
+1 💚compile1m 18smaster passed
+1 💚shadedjars8m 3sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 54smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 16sMaven dependency ordering for patch
+1 💚mvninstall3m 38sthe patch passed
+1 💚compile1m 19sthe patch passed
+1 💚javac1m 19sthe patch passed
+1 💚shadedjars8m 13spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 54sthe patch passed
_ Other Tests _
+1 💚unit0m 31shbase-balancer in the patch passed.
+1 💚unit147m 52shbase-server in the patch passed.
179m 59s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3276
Optional Testsjavac javadoc unit shadedjars compile
unameLinux ce9914c88f1f 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 5b99409
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/testReport/
Max. process+thread count3466 (vs. ulimit of 30000)
modulesC: hbase-balancer hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented May 19, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec3m 7sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 20sMaven dependency ordering for branch
+1 💚mvninstall4m 59smaster passed
+1 💚compile1m 40smaster passed
+1 💚shadedjars9m 10sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 0smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for patch
+1 💚mvninstall4m 45sthe patch passed
+1 💚compile1m 40sthe patch passed
+1 💚javac1m 40sthe patch passed
+1 💚shadedjars9m 6spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 58sthe patch passed
_ Other Tests _
+1 💚unit0m 31shbase-balancer in the patch passed.
+1 💚unit208m 58shbase-server in the patch passed.
248m 23s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3276
Optional Testsjavac javadoc unit shadedjars compile
unameLinux a891de2b9390 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 5b99409
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/testReport/
Max. process+thread count2430 (vs. ulimit of 30000)
modulesC: hbase-balancer hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/3/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9

Apache9 commented May 20, 2021

Copy link
Copy Markdown
ContributorAuthor

Wrote a simple test to verify the performance improvement

/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.master.balancer;
import java.util.List;
import java.util.Map;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.RegionInfo;
public class StohasticBalancerPerformanceEvaluation extends StochasticBalancerTestBase {
private void runBalancer(int numNodes, int numRegions, int numRegionsPerServer, int numTables,
int replication) {
Map<ServerName, List<RegionInfo>> serverMap =
createServerMap(numNodes, numRegions, numRegionsPerServer, replication, numTables);
Map<TableName, Map<ServerName, List<RegionInfo>>> LoadOfAllTable =
(Map) mockClusterServersWithTables(serverMap);
loadBalancer.balanceCluster(LoadOfAllTable);
}
public void run() throws Exception {
beforeAllTests();
loadBalancer.setRackManager(null);
int numNodes = 1000;
int numRegions = 100000; // 100 regions per RS
int numRegionsPerServer = 80; // all servers except one
int numTables = 100;
int replication = 1;
for (int i = 0; i < 3; i++) {
runBalancer(numNodes, numRegions, numRegionsPerServer, numTables, replication);
}
long startTimeNs = System.nanoTime();
runBalancer(numNodes, numRegions, numRegionsPerServer, numTables, replication);
long costNs = System.nanoTime() - startTimeNs;
System.out.println(String.format("Total time: %.3f secs", (double) costNs / 1000000000L));
}
public static void main(String[] args) throws Exception {
new StohasticBalancerPerformanceEvaluation().run();
}
}

After applied HBASE-25873, the result is
Total time: 51.990 secs
And after applied the patch here, the result is
Total time: 32.307 secs

About 37.8% down of the execution time.

@Apache-HBase

Apache-HBase commented May 24, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 30sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 3smaster passed
+1 💚compile3m 14smaster passed
+1 💚checkstyle1m 5smaster passed
+1 💚spotbugs2m 6smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 59sthe patch passed
+1 💚compile3m 29sthe patch passed
+1 💚javac3m 29sthe patch passed
+1 💚checkstyle1m 6sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck20m 5sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 31sthe patch passed
_ Other Tests _
+1 💚asflicense0m 16sThe patch does not generate ASF License warnings.
51m 46s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3276
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux e00516147d41 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / b02c810
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count96 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented May 24, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 54smaster passed
+1 💚compile1m 2smaster passed
+1 💚shadedjars8m 39sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 39smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 48sthe patch passed
+1 💚compile1m 1sthe patch passed
+1 💚javac1m 1sthe patch passed
+1 💚shadedjars8m 27spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 40sthe patch passed
_ Other Tests _
+1 💚unit150m 57shbase-server in the patch passed.
181m 57s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3276
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 6022185d17d4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / b02c810
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/testReport/
Max. process+thread count3359 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented May 24, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 21sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall5m 19smaster passed
+1 💚compile1m 23smaster passed
+1 💚shadedjars9m 37sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 42smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 45sthe patch passed
+1 💚compile1m 19sthe patch passed
+1 💚javac1m 19sthe patch passed
+1 💚shadedjars9m 0spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 41sthe patch passed
_ Other Tests _
+1 💚unit218m 41shbase-server in the patch passed.
254m 50s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3276
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 099027cdce98 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / b02c810
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/testReport/
Max. process+thread count2569 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9
Apache9 merged commit 6a77872 into apache:masterMay 25, 2021
Apache9 added a commit that referenced this pull request May 25, 2021
@Apache9
HBASE-25894Improve the performance for region load and region count …
2cb6cc8
…related cost functions (#3276)
Signed-off-by: Yi Mei <myimeiyi@gmail.com>
ndimiduk
ndimiduk reviewed Jun 1, 2021
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
}
}

void setCosts(Consumer<double[]> consumer) {

@ndimidukndimidukJun 1, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: having a setFoo method that is not a simple POJO field assignment is surprising!

@Apache9Apache9Jun 2, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any suggestion on the naming? fillCosts?

@ndimidukndimidukJun 10, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not off the top of my head, no. This interface for mutability by an external actor is a little strange. Maybe applyCostsConsumer ?

@Apache9Apache9Jun 11, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Let file another issue to land this naming change.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ndimidukndimidukndimiduk left review comments
@mymeiyimymeiyimymeiyi approved these changes

Assignees

@Apache9Apache9

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Apache9@Apache-HBase@ndimiduk@mymeiyi