Skip to content

HBASE-25482 Improve SimpleRegionNormalizer#getAverageRegionSizeMb - #2858

Merged
ndimiduk merged 2 commits into
apache:masterfrom
ZhaoBQ:HBASE-25482
Jan 19, 2021
Merged

HBASE-25482 Improve SimpleRegionNormalizer#getAverageRegionSizeMb#2858
ndimiduk merged 2 commits into
apache:masterfrom
ZhaoBQ:HBASE-25482

Conversation

@ZhaoBQ

Copy link
Copy Markdown
Contributor

HBASE-25482
If the table is set NormalizerTargetRegionSize, we take NormalizerTargetRegionSize as avgRegionSize and return it. So the totalSizeMb of table is not always calculated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec2m 2sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 29smaster passed
+1 💚checkstyle1m 16smaster passed
+1 💚spotbugs2m 21smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 53sthe patch passed
+1 💚checkstyle1m 11sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck20m 14sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 53sthe patch passed
_ Other Tests _
+1 💚asflicense0m 15sThe patch does not generate ASF License warnings.
48m 20s
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2858
Optional Testsdupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
unameLinux fa8048ee99e9 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 2444d26
Max. process+thread count84 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 20sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 15smaster passed
+1 💚compile1m 10smaster passed
+1 💚shadedjars7m 15sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 41smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 2sthe patch passed
+1 💚compile1m 3sthe patch passed
+1 💚javac1m 3sthe patch passed
+1 💚shadedjars7m 10spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 39sthe patch passed
_ Other Tests _
+1 💚unit204m 34shbase-server in the patch passed.
234m 2s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2858
Optional Testsjavac javadoc unit shadedjars compile
unameLinux cf42b185ae39 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 2444d26
Default JavaAdoptOpenJDK-1.8.0_232-b09
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/1/testReport/
Max. process+thread count3102 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec2m 1sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall5m 7smaster passed
+1 💚compile1m 15smaster passed
+1 💚shadedjars7m 39sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 48smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 41sthe patch passed
+1 💚compile1m 16sthe patch passed
+1 💚javac1m 16sthe patch passed
+1 💚shadedjars7m 31spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 45sthe patch passed
_ Other Tests _
-1 ❌unit214m 21shbase-server in the patch failed.
247m 17s
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2858
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 545eb33478b7 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 2444d26
Default JavaAdoptOpenJDK-11.0.6+10
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/1/testReport/
Max. process+thread count2944 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 53smaster passed
+1 💚compile0m 56smaster passed
+1 💚shadedjars6m 39sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 39smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 30sthe patch passed
+1 💚compile0m 56sthe patch passed
+1 💚javac0m 56sthe patch passed
+1 💚shadedjars6m 31spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 37sthe patch passed
_ Other Tests _
-1 ❌unit5m 25shbase-server in the patch failed.
30m 55s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2858
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 89572b4252ee 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 686b72c
Default JavaAdoptOpenJDK-1.8.0_232-b09
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/2/testReport/
Max. process+thread count583 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 1sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 57smaster passed
+1 💚checkstyle1m 6smaster passed
+1 💚spotbugs2m 7smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 28sthe patch passed
+1 💚checkstyle1m 5sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck18m 48sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 10sthe patch passed
_ Other Tests _
+1 💚asflicense0m 15sThe patch does not generate ASF License warnings.
40m 41s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2858
Optional Testsdupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
unameLinux c3d004cf54e5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 686b72c
Max. process+thread count94 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/2/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Comment on lines +280 to +282
final long totalSizeMb = tableRegions.stream()
.mapToLong(this::getRegionSizeMB)
.sum();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, just to be sure on the goal here, this is just to avoid this loop always happen, even when totalSizeMb is not been used?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is origin goal. If we set NormalizerTargetRegionSize, we use NormalizerTargetRegionSize as avgRegionSize and return. But now I found another serious bug, NormalizerTargetRegionCount and NormalizerTargetRegionSize will never used when log level is not debug.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 11sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 52smaster passed
+1 💚compile1m 11smaster passed
+1 💚shadedjars7m 30sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 42smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 38sthe patch passed
+1 💚compile1m 11sthe patch passed
+1 💚javac1m 11sthe patch passed
+1 💚shadedjars7m 26spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 40sthe patch passed
_ Other Tests _
-1 ❌unit46m 47shbase-server in the patch failed.
77m 25s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2858
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 8248165d9bd3 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 686b72c
Default JavaAdoptOpenJDK-11.0.6+10
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/2/testReport/
Max. process+thread count1138 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@ndimidukndimiduk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, ouch. Good find 👍

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 1sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 30smaster passed
+1 💚compile3m 18smaster passed
+1 💚checkstyle1m 2smaster passed
+1 💚spotbugs1m 59smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 36sthe patch passed
+1 💚compile3m 14sthe patch passed
+1 💚javac3m 14sthe patch passed
+1 💚checkstyle1m 3sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck20m 40sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 52sthe patch passed
_ Other Tests _
+1 💚asflicense0m 13sThe patch does not generate ASF License warnings.
50m 25s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2858
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux a6e3c955836f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 4caab90
Default JavaAdoptOpenJDK-1.8.0_232-b09
Max. process+thread count97 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/3/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 31sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 28smaster passed
+1 💚compile1m 1smaster passed
+1 💚shadedjars6m 44sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 38smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 34sthe patch passed
+1 💚compile0m 56sthe patch passed
+1 💚javac0m 56sthe patch passed
+1 💚shadedjars6m 36spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 36sthe patch passed
_ Other Tests _
+1 💚unit144m 59shbase-server in the patch passed.
171m 7s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2858
Optional Testsjavac javadoc unit shadedjars compile
unameLinux d57292bf8a88 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 4caab90
Default JavaAdoptOpenJDK-1.8.0_232-b09
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/3/testReport/
Max. process+thread count4417 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/3/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 6sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 47smaster passed
+1 💚compile1m 13smaster passed
+1 💚shadedjars7m 25sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 42smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 33sthe patch passed
+1 💚compile1m 12sthe patch passed
+1 💚javac1m 12sthe patch passed
+1 💚shadedjars7m 29spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 40sthe patch passed
_ Other Tests _
+1 💚unit212m 51shbase-server in the patch passed.
244m 0s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2858
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 09d9fce59be2 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 4caab90
Default JavaAdoptOpenJDK-11.0.6+10
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/3/testReport/
Max. process+thread count3601 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2858/3/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@ndimiduk

Copy link
Copy Markdown
Member

Any further comments/concerns @wchevreuil ?

@ndimiduk
ndimiduk merged commit 871eb09 into apache:masterJan 19, 2021
ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Jan 19, 2021
ndimiduk pushed a commit that referenced this pull request Jan 19, 2021
)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Apr 12, 2021
ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Apr 12, 2021
ndimiduk pushed a commit that referenced this pull request Apr 12, 2021
)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit that referenced this pull request Apr 12, 2021
)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@ZhaoBQ@Apache-HBase@ndimiduk@wchevreuil