Skip to content

HBASE-26872 Load rate calculator for cost functions should be more precise - #4253

Merged
sunhelly merged 2 commits into
apache:masterfrom
sunhelly:HBASE-26872
Mar 27, 2022
Merged

HBASE-26872 Load rate calculator for cost functions should be more precise#4253
sunhelly merged 2 commits into
apache:masterfrom
sunhelly:HBASE-26872

Conversation

@sunhelly

@sunhellysunhelly commented Mar 21, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

No description provided.

@Apache-HBase

Apache-HBase commented Mar 21, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 47sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 19smaster passed
+1 💚compile0m 33smaster passed
+1 💚checkstyle0m 13smaster passed
+1 💚spotbugs0m 27smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 49sthe patch passed
+1 💚compile0m 26sthe patch passed
+1 💚javac0m 26sthe patch passed
+1 💚checkstyle0m 10sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck14m 14sPatch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚spotbugs0m 35sthe patch passed
_ Other Tests _
+1 💚asflicense0m 10sThe patch does not generate ASF License warnings.
30m 14s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4253
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 23996c647168 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 0d7638d
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count60 (vs. ulimit of 30000)
modulesC: hbase-balancer U: hbase-balancer
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Mar 21, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 40sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 42smaster passed
+1 💚compile0m 17smaster passed
+1 💚shadedjars4m 42sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 17smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 18sthe patch passed
+1 💚compile0m 17sthe patch passed
+1 💚javac0m 17sthe patch passed
+1 💚shadedjars4m 41spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 14sthe patch passed
_ Other Tests _
+1 💚unit10m 10shbase-balancer in the patch passed.
30m 7s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4253
Optional Testsjavac javadoc unit shadedjars compile
unameLinux ffe4e57827db 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 0d7638d
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/testReport/
Max. process+thread count190 (vs. ulimit of 30000)
modulesC: hbase-balancer U: hbase-balancer
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Mar 21, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 55sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 36smaster passed
+1 💚compile0m 24smaster passed
+1 💚shadedjars5m 40sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 18smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 40sthe patch passed
+1 💚compile0m 23sthe patch passed
+1 💚javac0m 23sthe patch passed
+1 💚shadedjars6m 8spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 17sthe patch passed
_ Other Tests _
+1 💚unit12m 26shbase-balancer in the patch passed.
35m 48s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#4253
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 1bc43c365842 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 0d7638d
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/testReport/
Max. process+thread count161 (vs. ulimit of 30000)
modulesC: hbase-balancer U: hbase-balancer
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sunhellysunhelly changed the title HBASE-26872 Load rate calculator for cost functions should be more pr…HBASE-26872 Load rate calculator for cost functions should be more preciseMar 21, 2022
@sunhelly
sunhelly requested a review from Apache9March 21, 2022 12:34
@sunhelly

sunhelly commented Mar 24, 2022

Copy link
Copy Markdown
ContributorAuthor

Hi, @bbeaudreault , would you want to review this PR in your convenience? Thanks.

bbeaudreault
bbeaudreault approved these changes Mar 24, 2022
View reviewed changes

@bbeaudreaultbbeaudreault left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change makes sense! The code change itself looks good. I had 1 minor request for the test, if it makes sense

Comment thread...lancer/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java
}

@Test
public void testRegionLoadCostWhenDecrease() {

@bbeaudreaultbbeaudreaultMar 24, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct me if im wrong, but i believe this is testing the case where region loads go 1, 2, 0, 4. So it uses a 0 value in the 3rd slot. This works to verify your change, but I wonder if it would be more comprehensive to test 1, 2, 1, 4. This was just be more indicative of a real reset where the value starts at 0 but then gets incremented again.

@sunhellysunhellyMar 25, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, good idea, @bbeaudreault. It's reasonable to test the [1, 2, 1, 4], since it's rare to see 0 in the real scenarios, and I used 0 to describe the issue more clearly. Thanks.

@Apache-HBase

Apache-HBase commented Mar 25, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 46sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 24smaster passed
+1 💚compile0m 20smaster passed
+1 💚checkstyle0m 8smaster passed
+1 💚spotbugs0m 22smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 3sthe patch passed
+1 💚compile0m 17sthe patch passed
+1 💚javac0m 17sthe patch passed
+1 💚checkstyle0m 7sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck11m 2sPatch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚spotbugs0m 23sthe patch passed
_ Other Tests _
+1 💚asflicense0m 7sThe patch does not generate ASF License warnings.
22m 36s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4253
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 84bd4ab8d986 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 4ac600b
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count60 (vs. ulimit of 30000)
modulesC: hbase-balancer U: hbase-balancer
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Mar 25, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 8sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 50smaster passed
+1 💚compile0m 16smaster passed
+1 💚shadedjars3m 44sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 15smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 42sthe patch passed
+1 💚compile0m 14sthe patch passed
+1 💚javac0m 14sthe patch passed
+1 💚shadedjars3m 44spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 13sthe patch passed
_ Other Tests _
+1 💚unit13m 19shbase-balancer in the patch passed.
29m 10s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4253
Optional Testsjavac javadoc unit shadedjars compile
unameLinux c32b3b25f9fc 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 4ac600b
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/testReport/
Max. process+thread count195 (vs. ulimit of 30000)
modulesC: hbase-balancer U: hbase-balancer
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Mar 25, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 53sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 38smaster passed
+1 💚compile0m 23smaster passed
+1 💚shadedjars5m 39sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 27smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 16sthe patch passed
+1 💚compile0m 17sthe patch passed
+1 💚javac0m 17sthe patch passed
+1 💚shadedjars5m 38spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 20sthe patch passed
_ Other Tests _
+1 💚unit10m 50shbase-balancer in the patch passed.
32m 29s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#4253
Optional Testsjavac javadoc unit shadedjars compile
unameLinux dbba970960ef 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 4ac600b
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/testReport/
Max. process+thread count162 (vs. ulimit of 30000)
modulesC: hbase-balancer U: hbase-balancer
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sunhelly
sunhelly merged commit ad70ac2 into apache:masterMar 27, 2022
asfgit pushed a commit that referenced this pull request Mar 27, 2022
@sunhelly
HBASE-26872Load rate calculator for cost functions should be more pr…
7ae6be7
…ecise (#4253)
Signed-off-by: Bryan Beaudreault <bbeaudreault@hubspot.com>
Signed-off-by: Viraj Jasani<virajjasani@apache.org>
asfgit pushed a commit that referenced this pull request Mar 27, 2022
@sunhelly
HBASE-26872Load rate calculator for cost functions should be more pr…
568b335
…ecise (#4253)
Signed-off-by: Bryan Beaudreault <bbeaudreault@hubspot.com>
Signed-off-by: Viraj Jasani<virajjasani@apache.org>
asfgit pushed a commit that referenced this pull request Mar 28, 2022
@sunhelly
Revert "HBASE-26872Load rate calculator for cost functions should be…
7623900
… more precise (#4253)"
This reverts commit 7ae6be7.
asfgit pushed a commit that referenced this pull request Mar 28, 2022
@sunhelly
Revert "HBASE-26872Load rate calculator for cost functions should be…
2452290
… more precise (#4253)"
This reverts commit 568b335.
asfgit pushed a commit that referenced this pull request Mar 28, 2022
@sunhelly
HBASE-26872Load rate calculator for cost functions should be more pr…
f0088df
…ecise (#4253)
Signed-off-by: Bryan Beaudreault <bbeaudreault@hubspot.com>
Signed-off-by: Viraj Jasani<virajjasani@apache.org>
asfgit pushed a commit that referenced this pull request Mar 28, 2022
@sunhelly
HBASE-26872Load rate calculator for cost functions should be more pr…
170570b
…ecise (#4253)
Signed-off-by: Bryan Beaudreault <bbeaudreault@hubspot.com>
Signed-off-by: Viraj Jasani<virajjasani@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bbeaudreaultbbeaudreaultbbeaudreault approved these changes
@virajjasanivirajjasanivirajjasani approved these changes
@Apache9Apache9Awaiting requested review from Apache9

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sunhelly@Apache-HBase@bbeaudreault@virajjasani