Skip to content

HBASE-26229 Limit count and size of L0 files compaction in StripeCompactionPolicy - #3646

Merged
sunhelly merged 1 commit into
apache:masterfrom
sunhelly:HBASE-26229
Sep 9, 2021
Merged

HBASE-26229 Limit count and size of L0 files compaction in StripeCompactionPolicy#3646
sunhelly merged 1 commit into
apache:masterfrom
sunhelly:HBASE-26229

Conversation

@sunhelly

@sunhellysunhelly commented Aug 30, 2021

Copy link
Copy Markdown
Contributor

… and size under control

@Apache-HBase

Apache-HBase commented Aug 30, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 2sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 22smaster passed
+1 💚compile3m 22smaster passed
+1 💚checkstyle1m 11smaster passed
+1 💚spotbugs2m 13smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 7sthe patch passed
+1 💚compile3m 23sthe patch passed
+1 💚javac3m 23sthe patch passed
+1 💚checkstyle1m 9shbase-server: The patch generated 0 new + 7 unchanged - 3 fixed = 7 total (was 10)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck20m 13sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 23sthe patch passed
_ Other Tests _
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
52m 13s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3646
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux de187d6c3c23 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 889049e
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count85 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 30, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 27sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 59smaster passed
+1 💚compile1m 0smaster passed
+1 💚shadedjars8m 18sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 39smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 44sthe patch passed
+1 💚compile1m 1sthe patch passed
+1 💚javac1m 1sthe patch passed
+1 💚shadedjars8m 10spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 36sthe patch passed
_ Other Tests _
+1 💚unit147m 40shbase-server in the patch passed.
177m 54s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3646
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 190bb0bb5033 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 889049e
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/1/testReport/
Max. process+thread count4846 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 30, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 29sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall6m 3smaster passed
+1 💚compile1m 27smaster passed
+1 💚shadedjars10m 14sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 54smaster passed
_ Patch Compile Tests _
+1 💚mvninstall5m 41sthe patch passed
+1 💚compile1m 31sthe patch passed
+1 💚javac1m 31sthe patch passed
+1 💚shadedjars9m 49spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 48sthe patch passed
_ Other Tests _
+1 💚unit215m 54shbase-server in the patch passed.
256m 2s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3646
Optional Testsjavac javadoc unit shadedjars compile
unameLinux b1c27736605c 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 889049e
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/1/testReport/
Max. process+thread count3506 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9
Apache9 reviewed Sep 1, 2021
View reviewed changes

@Apache9Apache9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The title of the jira is a bit general, better describe more on how you plan to make it faster?

Thanks.

Comment thread...rc/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java Outdated
@sunhelly

sunhelly commented Sep 1, 2021

Copy link
Copy Markdown
ContributorAuthor

The title of the jira is a bit general, better describe more on how you plan to make it faster?

Thanks.

Thanks, @Apache9 , I will change the title of the jira.

@sunhellysunhelly changed the title HBASE-26229 Make L0 files compaction in StripeCompactionPolicy faster…HBASE-26229 Limit count and size of L0 files compaction in StripeCompactionPolicySep 1, 2021
@Apache-HBase

Apache-HBase commented Sep 1, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 26sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 1sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 37smaster passed
+1 💚compile3m 12smaster passed
+1 💚checkstyle1m 1smaster passed
+1 💚spotbugs2m 9smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 48sthe patch passed
+1 💚compile3m 21sthe patch passed
+1 💚javac3m 21sthe patch passed
+1 💚checkstyle1m 4shbase-server: The patch generated 0 new + 7 unchanged - 3 fixed = 7 total (was 10)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck18m 32sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 34sthe patch passed
_ Other Tests _
+1 💚asflicense0m 14sThe patch does not generate ASF License warnings.
48m 52s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3646
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux b5c792374693 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 36884c3
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count96 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/2/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 1, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 28sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 1smaster passed
+1 💚compile1m 2smaster passed
+1 💚shadedjars8m 10sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 39smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 41sthe patch passed
+1 💚compile1m 0sthe patch passed
+1 💚javac1m 0sthe patch passed
+1 💚shadedjars8m 9spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 37sthe patch passed
_ Other Tests _
+1 💚unit148m 13shbase-server in the patch passed.
178m 19s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3646
Optional Testsjavac javadoc unit shadedjars compile
unameLinux edc6cde0403f 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 36884c3
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/2/testReport/
Max. process+thread count4372 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 1, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 3sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall5m 2smaster passed
+1 💚compile1m 20smaster passed
+1 💚shadedjars9m 5sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 43smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 50sthe patch passed
+1 💚compile1m 20sthe patch passed
+1 💚javac1m 20sthe patch passed
+1 💚shadedjars9m 2spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 42sthe patch passed
_ Other Tests _
+1 💚unit207m 18shbase-server in the patch passed.
242m 55s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3646
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 0ae0b35ee37f 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 36884c3
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/2/testReport/
Max. process+thread count3639 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 1, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 5sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 13smaster passed
+1 💚compile3m 20smaster passed
+1 💚checkstyle1m 9smaster passed
+1 💚spotbugs2m 12smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 4sthe patch passed
+1 💚compile3m 22sthe patch passed
+1 💚javac3m 22sthe patch passed
+1 💚checkstyle1m 13shbase-server: The patch generated 0 new + 7 unchanged - 3 fixed = 7 total (was 10)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck24m 18sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 23sthe patch passed
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
56m 4s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3646
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 8cecb7f0128d 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / f62caa5
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count86 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/3/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 1, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 27sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 13smaster passed
+1 💚compile1m 5smaster passed
+1 💚shadedjars8m 40sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 42smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 54sthe patch passed
+1 💚compile1m 4sthe patch passed
+1 💚javac1m 4sthe patch passed
+1 💚shadedjars8m 37spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 41sthe patch passed
_ Other Tests _
+1 💚unit158m 2shbase-server in the patch passed.
189m 36s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3646
Optional Testsjavac javadoc unit shadedjars compile
unameLinux e1e6df866dd2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / f62caa5
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/3/testReport/
Max. process+thread count4452 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/3/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 1, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 5sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 45smaster passed
+1 💚compile1m 18smaster passed
+1 💚shadedjars9m 1sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 41smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 50sthe patch passed
+1 💚compile1m 18sthe patch passed
+1 💚javac1m 18sthe patch passed
+1 💚shadedjars9m 8spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 41sthe patch passed
_ Other Tests _
+1 💚unit225m 31shbase-server in the patch passed.
260m 53s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3646
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 32ffe32826c7 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / f62caa5
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/3/testReport/
Max. process+thread count3253 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3646/3/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sunhelly
sunhelly merged commit 8d2b995 into apache:masterSep 9, 2021
asfgit pushed a commit that referenced this pull request Sep 9, 2021
@sunhelly
HBASE-26229Limit count and size of L0 files compaction in StripeComp…
f670fc5
…actionPolicy (#3646)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
asfgit pushed a commit that referenced this pull request Sep 9, 2021
@sunhelly
HBASE-26229Limit count and size of L0 files compaction in StripeComp…
466d2d5
…actionPolicy (#3646)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
@sunhelly@stoty
HBASE-26229 Limit count and size of L0 files compaction in StripeComp…
85e6f62
…actionPolicy (apache#3646)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Apache9Apache9Apache9 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sunhelly@Apache-HBase@Apache9