Skip to content

HBASE-22749: Distributed MOB compactions - #921

Closed
VladRodionov wants to merge 17 commits into
apache:masterfrom
VladRodionov:master
Closed

HBASE-22749: Distributed MOB compactions#921
VladRodionov wants to merge 17 commits into
apache:masterfrom
VladRodionov:master

Conversation

@VladRodionov

@VladRodionovVladRodionov commented Dec 9, 2019

Copy link
Copy Markdown
Contributor

Finalizing HBASE-22749. Rebased patch against current HBase master, created new pull PR

@busbey

busbey commented Dec 9, 2019

Copy link
Copy Markdown
Contributor

just to be clear, this includes all the follow-on commits from #623 as well right? i.e. HBASE-23209, HBASE-23188, HBASE-23266, HBASE-23267, and HBASE-23189?

@busbey
busbey self-requested a review December 9, 2019 20:39
@VladRodionov

VladRodionov commented Dec 9, 2019

Copy link
Copy Markdown
ContributorAuthor

Yes, it includes all from PR #623 + some fixes to make this patch mergeable with current HBase master.

@Apache-HBase

This comment has been minimized.

Sign in to view
@busbey

busbey commented Dec 10, 2019

Copy link
Copy Markdown
Contributor

Please clean up qabot flagged issues. The unit test failures look related at first glance.

busbey
busbey requested changes Dec 12, 2019
View reviewed changes

@busbeybusbey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review in progress.

Comment threadhbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestMobCompaction.java
Comment threadhbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestMobCompaction.java Outdated
Comment threadhbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestMobCompaction.java
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCleanerChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCleanerChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCleanerChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCleanerChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCleanerChore.java Outdated
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
busbey
busbey requested changes Dec 16, 2019
View reviewed changes

@busbeybusbey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still WIP

Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java Outdated
busbey
busbey requested changes Dec 17, 2019
View reviewed changes

@busbeybusbey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more WIP

Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
busbey
busbey requested changes Dec 19, 2019
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreFlusher.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreFlusher.java
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobConstants.java
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobConstants.java
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobConstants.java
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobConstants.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobConstants.java Outdated
busbey
busbey requested changes Dec 19, 2019
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
busbey
busbey requested changes Jan 9, 2020
View reviewed changes

@busbeybusbey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

finished getting through all the changes. yay!

Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TesMobFileCleanerChore.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompaction.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompaction.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompaction.java
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompaction.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompactionBase.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompactionBase.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompactionBase.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobFileName.java
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMobStoreCompaction.java Outdated
busbey
busbey requested changes Jan 14, 2020
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCompactionChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCompactionChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCompactionChore.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCompactionChore.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TesMobFileCleanerChore.java Outdated
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@VladRodionov
VladRodionovforce-pushed the master branch 2 times, most recently from 926fcff to f7dc66dCompareJanuary 24, 2020 18:46
@busbeybusbey closed this Feb 2, 2020
@busbeybusbey reopened this Feb 3, 2020
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
busbey
busbey requested changes Feb 6, 2020
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompactionOptRegionBatchMode.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompactionRegularMode.java Outdated
Comment thread...erver/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompactionRegularRegionBatchMode.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobFileCleanerChore.java Outdated
Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobFileCleanerChore.java Outdated
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@busbey

busbey commented Feb 15, 2020

Copy link
Copy Markdown
Contributor

a precommit check of e11ec68 is in progress; about 20 minutes into the usual 2.5 hours.

busbey
busbey approved these changes Feb 15, 2020
View reviewed changes

@busbeybusbey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QA bot should be back in a couple of hours, I'd expect with a clean run.

I'm going to push a branch with these changes so we can get a full nightly run with the changes in place. Once I get that clean I'll move forward with merging to master.

Thanks for sticking through on this large improvement @VladRodionov!

@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

Apache-HBase commented Feb 15, 2020

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 43sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 1sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 23 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep0m 41sMaven dependency ordering for branch
+1 💚mvninstall7m 5smaster passed
+1 💚compile2m 25smaster passed
+1 💚checkstyle3m 0smaster passed
+0 🆗refguide6m 23sbranch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚shadedjars6m 1sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 37smaster passed
+0 🆗spotbugs5m 7sUsed deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗findbugs0m 17sbranch/hbase-checkstyle no findbugs output file (findbugsXml.xml)
-0 ⚠️patch5m 38sUsed diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗mvndep0m 16sMaven dependency ordering for patch
+1 💚mvninstall6m 33sthe patch passed
+1 💚compile2m 24sthe patch passed
+1 💚javac2m 24sthe patch passed
+1 💚checkstyle2m 59sroot: The patch generated 0 new + 427 unchanged - 58 fixed = 427 total (was 485)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚xml0m 4sThe patch has no ill-formed XML file.
+0 🆗refguide6m 25spatch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚shadedjars6m 1spatch has no errors when building our shaded downstream artifacts.
+1 💚hadoopcheck20m 10sPatch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 💚javadoc1m 26sthe patch passed
+0 🆗findbugs0m 14shbase-checkstyle has no data from findbugs
_ Other Tests _
+1 💚unit0m 14shbase-checkstyle in the patch passed.
+1 💚unit1m 3shbase-common in the patch passed.
+1 💚unit66m 16shbase-server in the patch passed.
+1 💚unit1m 16shbase-it in the patch passed.
+1 💚asflicense2m 15sThe patch does not generate ASF License warnings.
165m 14s
SubsystemReport/Notes
DockerClient=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/artifact/out/Dockerfile
GITHUB PR#921
JIRA IssueHBASE-22749
Optional Testsdupname asflicense checkstyle javac javadoc unit xml refguide spotbugs findbugs shadedjars hadoopcheck hbaseanti compile
unameLinux 514b8f7656b0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-921/out/precommit/personality/provided.sh
git revisionmaster / ae781e5
Default Java1.8.0_181
refguidehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/artifact/out/branch-site/book.html
refguidehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/artifact/out/patch-site/book.html
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/testReport/
Max. process+thread count8774 (vs. ulimit of 10000)
modulesC: hbase-checkstyle hbase-common hbase-server hbase-it U: .
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@busbey

busbey commented Feb 18, 2020

Copy link
Copy Markdown
Contributor

qabot looks great. I got some noise on my nightly branch. checking on it today.

@busbey

busbey commented Feb 19, 2020

Copy link
Copy Markdown
Contributor

nightly looks good as well. I'll post details about that to jira and then move forward on merging.

thanks for chasing this down @VladRodionov

@busbey

busbey commented Feb 19, 2020

Copy link
Copy Markdown
Contributor

merged via squashed commit b8194b4

@busbeybusbey closed this Feb 19, 2020
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@busbeybusbeybusbey approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@VladRodionov@busbey@Apache-HBase