HBASE-22923 Consider minVersionToMoveSysTables while moving region and creating regionPlan (ADDENDUM) by virajjasani · Pull Request #3455 · apache/hbase

virajjasani · 2021-07-02T19:22:48Z

No description provided.

…d creating regionPlan (ADDENDUM)

virajjasani · 2021-07-02T19:24:50Z

FYI @d-c-manning
Tested region_mover and balancer with mixed mode of servers, works as expected, meaning no workflow considers moving system tables to higher version servers. Please review.
Thanks

Apache-HBase · 2021-07-02T20:10:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 55s	master passed
+1 💚	compile	3m 10s	master passed
+1 💚	checkstyle	1m 4s	master passed
+1 💚	spotbugs	2m 5s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 37s	the patch passed
+1 💚	compile	3m 10s	the patch passed
+1 💚	javac	3m 10s	the patch passed
+1 💚	checkstyle	1m 1s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	18m 15s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 13s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		46m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3455
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 5f25bc424dba 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5ef5e64`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

d-c-manning

thanks @virajjasani !

Apache-HBase · 2021-07-02T22:19:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 41s	master passed
+1 💚	compile	1m 16s	master passed
+1 💚	shadedjars	8m 20s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 20s	the patch passed
+1 💚	compile	1m 14s	the patch passed
+1 💚	javac	1m 14s	the patch passed
+1 💚	shadedjars	8m 28s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 50s	the patch passed
		_ Other Tests _
+1 💚	unit	143m 33s	hbase-server in the patch passed.
		175m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3455
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 927414ddb83f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5ef5e64`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/testReport/
Max. process+thread count	3105 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-07-02T22:27:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 4s	master passed
+1 💚	compile	1m 2s	master passed
+1 💚	shadedjars	8m 5s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 39s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 46s	the patch passed
+1 💚	compile	0m 59s	the patch passed
+1 💚	javac	0m 59s	the patch passed
+1 💚	shadedjars	8m 4s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	the patch passed
		_ Other Tests _
-1 ❌	unit	153m 49s	hbase-server in the patch failed.
		183m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3455
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 52d015dfa3c5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5ef5e64`
Default Java	AdoptOpenJDK-1.8.0_282-b08
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/testReport/
Max. process+thread count	3358 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3455/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2021-07-03T12:52:35Z

So what's the problem here?

virajjasani · 2021-07-03T15:39:40Z

So what's the problem here?

With earlier patch of HBASE-22923, we only covered case of "AssignmentManager taking care of excluding lower version servers for system table assignment" for the thread that keeps track of version differences in the cluster and soon as it finds new version, it tries to move system tables there, so this is auto-migration of system tables use-case. But we missed the case of graceful stop and balancer's plans generation. Graceful stop prepares move commands based on RSGroup but if we don't change the logic of "excluding servers with lower version for system table assignment", system table regions will be in hanging state when we execute graceful stop command.
With the current patch, we have covered all cases that master uses to coordinate region movement, regardless of whether it is auto-migration of regions by AM or whether we execute graceful stop manually.

…d creating regionPlan (ADDENDUM) (#3455) Signed-off-by: David Manning <david.manning@salesforce.com> Signed-off-by: Bharath Vissapragada <bharathv@apache.org>

…d creating regionPlan (ADDENDUM) (#3456) (#3455) Signed-off-by: David Manning <david.manning@salesforce.com> Signed-off-by: Bharath Vissapragada <bharathv@apache.org>

HBASE-22923 Consider minVersionToMoveSysTables while moving region an…
e1596c8
…d creating regionPlan (ADDENDUM)

virajjasani requested review from apurtell and bharathv July 2, 2021 19:25

d-c-manning approved these changes Jul 2, 2021
View reviewed changes

bharathv approved these changes Jul 3, 2021
View reviewed changes

virajjasani merged commit 1883889 into apache:masterJul 3, 2021

virajjasani deleted the HBASE-22923-master-add branch July 3, 2021 17:11

Navigation Menu

Uh oh!

HBASE-22923 Consider minVersionToMoveSysTables while moving region and creating regionPlan (ADDENDUM) - #3455

HBASE-22923 Consider minVersionToMoveSysTables while moving region and creating regionPlan (ADDENDUM)#3455
virajjasani merged 1 commit into
apache:masterfrom
virajjasani:HBASE-22923-master-add

virajjasani commented Jul 2, 2021

Uh oh!

virajjasani commented Jul 2, 2021

Uh oh!

Apache-HBase commented Jul 2, 2021

Uh oh!

d-c-manning left a comment

Uh oh!

Apache-HBase commented Jul 2, 2021

Uh oh!

Apache-HBase commented Jul 2, 2021

Uh oh!

Apache9 commented Jul 3, 2021

Uh oh!

virajjasani commented Jul 3, 2021•
edited
Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

virajjasani commented Jul 2, 2021

Uh oh!

virajjasani commented Jul 2, 2021

Uh oh!

Apache-HBase commented Jul 2, 2021

Uh oh!

d-c-manning left a comment

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Jul 2, 2021

Uh oh!

Apache-HBase commented Jul 2, 2021

Uh oh!

Apache9 commented Jul 3, 2021

Uh oh!

virajjasani commented Jul 3, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

virajjasani commented Jul 3, 2021•
edited
Loading