Skip to content

HBASE-26434 Compact L0 files for cold regions using StripeCompactionPolicy - #3830

Merged
sunhelly merged 1 commit into
apache:masterfrom
sunhelly:HBASE-26434
Feb 15, 2022
Merged

HBASE-26434 Compact L0 files for cold regions using StripeCompactionPolicy#3830
sunhelly merged 1 commit into
apache:masterfrom
sunhelly:HBASE-26434

Conversation

@sunhelly

@sunhellysunhelly commented Nov 9, 2021

Copy link
Copy Markdown
Contributor

This issue solves the similar problem to HBASE-25754. StripeCompactionPolicy doesn't support major compaction, but the expired data in the cold region need to be cleared.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec4m 5sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 19smaster passed
+1 💚compile3m 18smaster passed
+1 💚checkstyle1m 3smaster passed
+1 💚spotbugs2m 6smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 46sthe patch passed
+1 💚compile3m 10sthe patch passed
+1 💚javac3m 10sthe patch passed
+1 💚checkstyle1m 1sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck19m 19sPatch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚spotbugs2m 13sthe patch passed
_ Other Tests _
+1 💚asflicense0m 16sThe patch does not generate ASF License warnings.
52m 49s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3830
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 7f987274da3d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 407a5bd
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count96 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 1sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 46smaster passed
+1 💚compile1m 11smaster passed
+1 💚shadedjars8m 20sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 45smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 26sthe patch passed
+1 💚compile1m 15sthe patch passed
+1 💚javac1m 15sthe patch passed
+1 💚shadedjars8m 6spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 42sthe patch passed
_ Other Tests _
+1 💚unit144m 11shbase-server in the patch passed.
177m 5s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3830
Optional Testsjavac javadoc unit shadedjars compile
unameLinux f9aaa8d44452 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 407a5bd
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/testReport/
Max. process+thread count3474 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 26sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 10smaster passed
+1 💚compile1m 1smaster passed
+1 💚shadedjars8m 19sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 41smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 50sthe patch passed
+1 💚compile1m 1sthe patch passed
+1 💚javac1m 1sthe patch passed
+1 💚shadedjars8m 24spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 38sthe patch passed
_ Other Tests _
+1 💚unit148m 47shbase-server in the patch passed.
179m 34s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3830
Optional Testsjavac javadoc unit shadedjars compile
unameLinux cf9ffa4e7fc7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 407a5bd
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/testReport/
Max. process+thread count3532 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sunhellysunhelly changed the title HBASE-26434 Do compact when all L0 files are expiredHBASE-26434 Do compact when all L0 files are expired in StripeCompactionPolicyNov 16, 2021
@sunhellysunhelly changed the title HBASE-26434 Do compact when all L0 files are expired in StripeCompactionPolicyHBASE-26434 Compact L0 files for cold regions using StripeCompactionPolicyNov 16, 2021
@sunhelly

Copy link
Copy Markdown
ContributorAuthor

@Apache9 can you help review this PR at your convenience? Thanks.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 1sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 18smaster passed
+1 💚compile3m 15smaster passed
+1 💚checkstyle1m 4smaster passed
+1 💚spotbugs2m 10smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 55sthe patch passed
+1 💚compile3m 13sthe patch passed
+1 💚javac3m 13sthe patch passed
+1 💚checkstyle1m 4sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck19m 33sPatch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚spotbugs2m 17sthe patch passed
_ Other Tests _
+1 💚asflicense0m 16sThe patch does not generate ASF License warnings.
50m 11s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3830
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 99c6dd62fe5d 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / f6348d4
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count96 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 30sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 17smaster passed
+1 💚compile1m 6smaster passed
+1 💚shadedjars8m 26sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 40smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 1sthe patch passed
+1 💚compile1m 5sthe patch passed
+1 💚javac1m 5sthe patch passed
+1 💚shadedjars8m 39spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 44sthe patch passed
_ Other Tests _
-1 ❌unit153m 25shbase-server in the patch failed.
185m 1s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3830
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 1c0c5502fbd1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / f6348d4
Default JavaAdoptOpenJDK-1.8.0_282-b08
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/testReport/
Max. process+thread count3905 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 31sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall5m 42smaster passed
+1 💚compile1m 32smaster passed
+1 💚shadedjars9m 25sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 47smaster passed
_ Patch Compile Tests _
+1 💚mvninstall5m 29sthe patch passed
+1 💚compile1m 25sthe patch passed
+1 💚javac1m 25sthe patch passed
+1 💚shadedjars9m 29spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 45sthe patch passed
_ Other Tests _
-1 ❌unit251m 1shbase-server in the patch failed.
288m 56s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3830
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 611af2ca6313 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / f6348d4
Default JavaAdoptOpenJDK-11.0.10+9
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/testReport/
Max. process+thread count3289 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 12sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 41smaster passed
+1 💚compile1m 57smaster passed
+1 💚checkstyle0m 36smaster passed
+1 💚spotbugs1m 17smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 23sthe patch passed
+1 💚compile1m 55sthe patch passed
+1 💚javac1m 55sthe patch passed
+1 💚checkstyle0m 35sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck11m 22sPatch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚spotbugs1m 26sthe patch passed
_ Other Tests _
+1 💚asflicense0m 9sThe patch does not generate ASF License warnings.
30m 36s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3830
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux a030e63c3d4c 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 1af2262
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count60 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sunhelly
sunhelly merged commit 7c52895 into apache:masterFeb 15, 2022
asfgit pushed a commit that referenced this pull request Feb 15, 2022
Signed-off-by: Duo Zhang <zhangduo@apache.org>
asfgit pushed a commit that referenced this pull request Feb 15, 2022
Signed-off-by: Duo Zhang <zhangduo@apache.org>
asfgit pushed a commit that referenced this pull request Feb 15, 2022
Signed-off-by: Duo Zhang <zhangduo@apache.org>
@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 41sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 4smaster passed
+1 💚compile0m 40smaster passed
+1 💚shadedjars4m 59sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 42sthe patch passed
+1 💚compile0m 41sthe patch passed
+1 💚javac0m 41sthe patch passed
+1 💚shadedjars4m 56spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23sthe patch passed
_ Other Tests _
+1 💚unit309m 19shbase-server in the patch passed.
329m 9s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3830
Optional Testsjavac javadoc unit shadedjars compile
unameLinux c75ef1ead39a 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 1af2262
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/testReport/
Max. process+thread count1561 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 39sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 6smaster passed
+1 💚compile0m 33smaster passed
+1 💚shadedjars4m 56sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 18sthe patch passed
+1 💚compile0m 34sthe patch passed
+1 💚javac0m 34sthe patch passed
+1 💚shadedjars4m 56spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 20sthe patch passed
_ Other Tests _
+1 💚unit319m 36shbase-server in the patch passed.
339m 42s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3830
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 744640ec1a44 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 1af2262
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/testReport/
Max. process+thread count1628 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3830/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit c8ccbde)
Change-Id: Ia8bfff1bfb0ecd69443173ef10f727b743d5ac31
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@sunhelly@Apache-HBase@Apache9