Skip to content

HBASE-22810 Initialize an separate ThreadPoolExecutor for taking/restoring snapshot - #486

Merged
openinx merged 1 commit into
apache:masterfrom
openinx:master
Aug 15, 2019
Merged

HBASE-22810 Initialize an separate ThreadPoolExecutor for taking/restoring snapshot#486
openinx merged 1 commit into
apache:masterfrom
openinx:master

Conversation

@openinx

@openinxopeninx commented Aug 13, 2019

Copy link
Copy Markdown
Member

…oring snapshot

@Apache-HBase

Apache-HBase commented Aug 13, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec38Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1mvninstall337master passed
+1compile54master passed
+1checkstyle80master passed
+1shadedjars274branch has no errors when building our shaded downstream artifacts.
+1javadoc37master passed
0spotbugs254Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs250master passed
_ Patch Compile Tests _
+1mvninstall304the patch passed
+1compile56the patch passed
+1javac56the patch passed
-1checkstyle77hbase-server: The patch generated 4 new + 167 unchanged - 2 fixed = 171 total (was 169)
+1whitespace0The patch has no whitespace issues.
+1shadedjars274patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck946Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1javadoc33the patch passed
+1findbugs255the patch passed
_ Other Tests _
+1unit6636hbase-server in the patch passed.
+1asflicense27The patch does not generate ASF License warnings.
10077
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/1/artifact/out/Dockerfile
GITHUB PR#486
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 4d977315dc04 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-486/out/precommit/personality/provided.sh
git revisionmaster / 8c1edb3
Default Java1.8.0_181
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/1/artifact/out/diff-checkstyle-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/1/testReport/
Max. process+thread count5094 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/1/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 13, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec46Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1mvninstall331master passed
+1compile52master passed
+1checkstyle75master passed
+1shadedjars275branch has no errors when building our shaded downstream artifacts.
+1javadoc36master passed
0spotbugs256Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs253master passed
_ Patch Compile Tests _
+1mvninstall301the patch passed
+1compile53the patch passed
+1javac53the patch passed
-1checkstyle75hbase-server: The patch generated 4 new + 167 unchanged - 2 fixed = 171 total (was 169)
+1whitespace1The patch has no whitespace issues.
+1shadedjars265patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck970Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1javadoc33the patch passed
+1findbugs249the patch passed
_ Other Tests _
-1unit6854hbase-server in the patch failed.
+1asflicense32The patch does not generate ASF License warnings.
10271
ReasonTests
Failed junit testshadoop.hbase.master.assignment.TestOpenRegionProcedureHang
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/2/artifact/out/Dockerfile
GITHUB PR#486
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 499a55e83057 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-486/out/precommit/personality/provided.sh
git revisionmaster / e69af5a
Default Java1.8.0_181
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/2/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/2/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/2/testReport/
Max. process+thread count4524 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/2/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 14, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec37Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1mvninstall323master passed
+1compile53master passed
+1checkstyle74master passed
+1shadedjars261branch has no errors when building our shaded downstream artifacts.
+1javadoc38master passed
0spotbugs207Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs206master passed
_ Patch Compile Tests _
+1mvninstall297the patch passed
+1compile52the patch passed
+1javac52the patch passed
-1checkstyle72hbase-server: The patch generated 4 new + 167 unchanged - 2 fixed = 171 total (was 169)
+1whitespace0The patch has no whitespace issues.
+1shadedjars263patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck905Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1javadoc32the patch passed
+1findbugs243the patch passed
_ Other Tests _
+1unit6706hbase-server in the patch passed.
+1asflicense24The patch does not generate ASF License warnings.
9937
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/3/artifact/out/Dockerfile
GITHUB PR#486
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux aec1cc3557ba 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-486/out/precommit/personality/provided.sh
git revisionmaster / 27ed2ac
Default Java1.8.0_181
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/3/artifact/out/diff-checkstyle-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/3/testReport/
Max. process+thread count4995 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/3/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Apache9
Apache9 requested changes Aug 14, 2019
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Outdated
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Outdated
this.executorService.startExecutorService(ExecutorType.M_LOG_REPLAY_OPS,
conf.getInt("hbase.master.executor.logreplayops.threads", 10));
this.executorService.startExecutorService(ExecutorType.MASTER_SNAPSHOT_OPERATIONS,
conf.getInt("hbase.master.executor.snapshot.threads", 3));

@Apache9Apache9Aug 14, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better make a static field for this.

@openinxopeninxAug 14, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine

@Apache-HBase

Apache-HBase commented Aug 14, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec84Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ master Compile Tests _
0mvndep40Maven dependency ordering for branch
+1mvninstall455master passed
+1compile96master passed
+1checkstyle136master passed
+1shadedjars376branch has no errors when building our shaded downstream artifacts.
+1javadoc64master passed
0spotbugs308Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs366master passed
_ Patch Compile Tests _
0mvndep17Maven dependency ordering for patch
+1mvninstall413the patch passed
+1compile100the patch passed
+1javac100the patch passed
-1checkstyle104hbase-server: The patch generated 3 new + 162 unchanged - 7 fixed = 165 total (was 169)
+1whitespace0The patch has no whitespace issues.
+1shadedjars371patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck1282Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1javadoc65the patch passed
+1findbugs393the patch passed
_ Other Tests _
+1unit204hbase-common in the patch passed.
-1unit13817hbase-server in the patch failed.
+1asflicense66The patch does not generate ASF License warnings.
18990
ReasonTests
Failed junit testshadoop.hbase.util.TestFromClientSide3WoUnsafe
hadoop.hbase.client.TestFromClientSide3
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/4/artifact/out/Dockerfile
GITHUB PR#486
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux cd6837ae2514 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-486/out/precommit/personality/provided.sh
git revisionmaster / 27ed2ac
Default Java1.8.0_181
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/4/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/4/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/4/testReport/
Max. process+thread count4725 (vs. ulimit of 10000)
modulesC: hbase-common hbase-server U: .
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/4/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 14, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec38Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ master Compile Tests _
0mvndep39Maven dependency ordering for branch
+1mvninstall340master passed
+1compile74master passed
+1checkstyle93master passed
+1shadedjars263branch has no errors when building our shaded downstream artifacts.
+1javadoc54master passed
0spotbugs257Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs305master passed
_ Patch Compile Tests _
0mvndep14Maven dependency ordering for patch
+1mvninstall286the patch passed
+1compile71the patch passed
+1javac71the patch passed
-1checkstyle69hbase-server: The patch generated 3 new + 162 unchanged - 7 fixed = 165 total (was 169)
+1whitespace0The patch has no whitespace issues.
+1shadedjars264patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck893Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1javadoc55the patch passed
+1findbugs310the patch passed
_ Other Tests _
+1unit174hbase-common in the patch passed.
-1unit6870hbase-server in the patch failed.
+1asflicense38The patch does not generate ASF License warnings.
10643
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/5/artifact/out/Dockerfile
GITHUB PR#486
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 2b37abec7345 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-486/out/precommit/personality/provided.sh
git revisionmaster / 53db390
Default Java1.8.0_181
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/5/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/5/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/5/testReport/
Max. process+thread count4656 (vs. ulimit of 10000)
modulesC: hbase-common hbase-server U: .
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/5/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Apache9
Apache9 approved these changes Aug 15, 2019
View reviewed changes

@Apache9Apache9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1.

@Apache9

Apache9 commented Aug 15, 2019

Copy link
Copy Markdown
Contributor

But please fix the checkstyle issues before merging...

@openinx
openinx merged commit 8ffc45a into apache:masterAug 15, 2019
@openinx

openinx commented Aug 15, 2019

Copy link
Copy Markdown
MemberAuthor

The checkstyle is not an issue I think, the '(' is preceded by a whitespace because we want to make the code align as before :

/** * Messages originating from Client to Master.<br> * C_M_CREATE_TABLE<br> * Client asking Master to create a table. */C_M_CREATE_TABLE (47, ExecutorType.MASTER_TABLE_OPERATIONS),
/** * Messages originating from Client to Master.<br> * C_M_SNAPSHOT_TABLE<br> * Client asking Master to snapshot an offline table. */C_M_SNAPSHOT_TABLE (48, ExecutorType.MASTER_SNAPSHOT_OPERATIONS),
/** * Messages originating from Client to Master.<br> * C_M_RESTORE_SNAPSHOT<br> * Client asking Master to restore a snapshot. */C_M_RESTORE_SNAPSHOT (49, ExecutorType.MASTER_SNAPSHOT_OPERATIONS),

asfgit pushed a commit that referenced this pull request Aug 15, 2019
@openinx
HBASE-22810Initialize an separate ThreadPoolExecutor for taking/rest…
b4734e7
…oring snapshot (#486)
asfgit pushed a commit that referenced this pull request Aug 15, 2019
@openinx
HBASE-22810Initialize an separate ThreadPoolExecutor for taking/rest…
f49a86a
…oring snapshot (#486)
asfgit pushed a commit that referenced this pull request Aug 15, 2019
@openinx
HBASE-22810Initialize an separate ThreadPoolExecutor for taking/rest…
1a8fb25
…oring snapshot (#486)
asfgit pushed a commit that referenced this pull request Aug 15, 2019
@openinx
HBASE-22810Initialize an separate ThreadPoolExecutor for taking/rest…
6f9db46
…oring snapshot (#486)
asfgit pushed a commit that referenced this pull request Aug 15, 2019
@openinx
HBASE-22810Initialize an separate ThreadPoolExecutor for taking/rest…
eb6b617
…oring snapshot (#486)
asfgit pushed a commit that referenced this pull request Aug 15, 2019
@openinx
HBASE-22810Initialize an separate ThreadPoolExecutor for taking/rest…
b97621b
…oring snapshot (#486)
asfgit pushed a commit that referenced this pull request Aug 15, 2019
@openinx
HBASE-22810Initialize an separate ThreadPoolExecutor for taking/rest…
6fc459d
…oring snapshot (#486)
@Apache-HBase

Apache-HBase commented Aug 15, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec58Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ master Compile Tests _
0mvndep39Maven dependency ordering for branch
+1mvninstall426master passed
+1compile100master passed
+1checkstyle139master passed
+1shadedjars343branch has no errors when building our shaded downstream artifacts.
+1javadoc70master passed
0spotbugs327Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs390master passed
_ Patch Compile Tests _
0mvndep18Maven dependency ordering for patch
+1mvninstall405the patch passed
+1compile107the patch passed
+1javac107the patch passed
-1checkstyle104hbase-server: The patch generated 3 new + 162 unchanged - 7 fixed = 165 total (was 169)
+1whitespace0The patch has no whitespace issues.
+1shadedjars359patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck1294Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1javadoc70the patch passed
+1findbugs432the patch passed
_ Other Tests _
+1unit217hbase-common in the patch passed.
-1unit19484hbase-server in the patch failed.
+1asflicense58The patch does not generate ASF License warnings.
24660
ReasonTests
Failed junit testshadoop.hbase.client.TestFromClientSide
hadoop.hbase.replication.TestReplicationDisableInactivePeer
hadoop.hbase.snapshot.TestFlushSnapshotFromClient
hadoop.hbase.client.TestFromClientSide3
hadoop.hbase.replication.TestReplicationKillSlaveRSWithSeparateOldWALs
hadoop.hbase.replication.TestReplicationSmallTests
hadoop.hbase.master.TestAssignmentManagerMetrics
hadoop.hbase.replication.TestReplicationSmallTestsSync
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/6/artifact/out/Dockerfile
GITHUB PR#486
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 8e8f7bc61373 4.4.0-154-generic #181-Ubuntu SMP Tue Jun 25 05:29:03 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-486/out/precommit/personality/provided.sh
git revisionmaster / d9d5f69
Default Java1.8.0_181
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/6/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/6/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/6/testReport/
Max. process+thread count4829 (vs. ulimit of 10000)
modulesC: hbase-common hbase-server U: .
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/6/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

infraio pushed a commit to infraio/hbase that referenced this pull request Aug 17, 2020
@openinx@Apache9
HBASE-22810 Initialize an separate ThreadPoolExecutor for taking/rest…
e567a85
…oring snapshot (apache#486)
symat pushed a commit to symat/hbase that referenced this pull request Feb 17, 2021
@openinxJenkins
HBASE-22810 Initialize an separate ThreadPoolExecutor for taking/rest…
085feaf
…oring snapshot (apache#486)
(cherry picked from commit b4734e7)
Change-Id: Id5c952da1949f7039d6caa5329c6599ca54fbba4
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@infraioinfraioinfraio approved these changes
@Apache9Apache9Apache9 approved these changes
@saintstacksaintstackAwaiting requested review from saintstack

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@openinx@Apache-HBase@Apache9@infraio