HBASE-28158 Decouple RIT list management from TRSP by Umeshkumar9414 · Pull Request #7375 · apache/hbase

Umeshkumar9414 · 2025-10-10T15:03:12Z

No description provided.

apurtell · 2025-10-14T23:25:54Z

+ return regionInTransitionTracker.isRegionInTransition(regionInfo);
+ }
+
+ public int getOngoingTRSPCount() {


Call it getInTransitCount()?

"Transit" means something different than "in transition". Javadoc the explaination.

how about scheduledRegionTransition ?

apurtell · 2025-10-14T23:32:44Z

 }

- public boolean isInTransition() {
+ public boolean isOngoingTRSP() {


How about isInTransit?
And javadoc that "transit" means something different than "in transition".

how about changing to isTransitionScheduled?

apurtell · 2025-10-14T23:43:29Z

+ removeRegionInTransition(regionInfo);
+ }
+
+ private List<RegionState.State> getExceptedRegionStates(RegionStateNode regionStateNode) {


I think this method should be renamed and the code should be documented with comments. You are not really "excepting" regions.
You are basically waiting for the assignment state machine to complete and reach the desired terminal state by checking the region's current state against the table's state (ENABLED vs. DISABLED) to determine if the region is in the terminal state. If not, the region is added to the RIT list; otherwise, it is removed.
Your method naming and comments should communicate this theory of operation.
Related, there may be a minor race condition here. Consider if the table's state is changed (e.g., from ENABLED to DISABLING) at the same time a region for that table reports a state change. I think the tracker can momentarily use a stale table state here. However, this is self-correcting. As the region proceeds through its state transitions each subsequent call to the tracker will re-evaluate its status, and it will eventually be removed from RIT correctly.

Correct me if I am wrong but I think this race condition might not occur as before changing the state of any table of region (or running any procedure) we first take a lock (setState using HBCK may be an exception). And this lock might save us from this case.

Umeshkumar9414 · 2025-10-15T11:14:45Z

 static void removeNonDefaultReplicas(MasterProcedureEnv env, Stream<RegionInfo> regions,
 int regionReplication) {
 // Remove from in-memory states
+ // TODO should we not confirm here that replica region are closed or not ?


@apurtell can you help me here? I was curious here why we are not confirming if replica regions are closed or not?

I am not sure about this either.

apurtell

This is good work and looks good to me. There is a checkstyle nit remaining to fix, please attend to that.

Apache9

Still need lots of polishing, but in general I think the approach is OK, it does not change the normal transition logic, just changes the way on how we show regions as in transition.

Apache9 · 2025-11-08T15:21:56Z

 // splittingServersFromWALDir are being actively split -- the directory in the FS ends in
 // '-SPLITTING'. Each splitting server should have a corresponding SCP. Log if not.
- splittingServersFromWALDir.stream().filter(s -> !deadServersFromPE.contains(s))
+ splittingServersFromWALDir.stream().filter(s -> !deadServersFromPE.containsKey(s))


Why we change the parameter from Set to a Map but we only use its containsKey method?

we are passing it to ServerManager where we are adding both the values to deadServermap (ref)

Then better change the parameter name?

@Umeshkumar9414 Here, I've replied some of the comments.

Ping @Umeshkumar9414

Apache9 · 2025-11-08T15:22:34Z

 * @param liveServersFromWALDir the live region servers from wal directory.
 */
- void findDeadServersAndProcess(Set<ServerName> deadServersFromPE,
+ void findDeadServersAndProcess(Map<ServerName, Long> deadServersFromPE,


Why just a parameter type change without chaging any real logic?

I think the value is the crash time? We should use it when setting up dead servers?

yes we should use it. We are using it in line 445 (deadservers::putIfAbsent).
I didn't need to change the code becuase lambda and method overloading.

Let's change the parameter name.

@Umeshkumar9414

ping @Umeshkumar9414

Apache9 · 2025-11-08T15:37:49Z


 private final int forceRegionRetainmentRetries;

+ private final RegionInTransitionTracker regionInTransitionTracker;


Why not put this into RegionStates? In this way we can reduce the code change i think.

There are couple of reasons for this
I wanted regionTransitionTracker to follow meta (RegionStateStore in code) and regionStates doesn't do that. Sometime we update region state in memory and update meta in next step of procedure.
regionInTransitionTracker needed tableStateManager and I doesn't wanted to pass tableStateManager to regionSates. Another option would be create tracker in AssignmentManager itself and then pass that in constructor.
Considering above I feel keeping this in AM is better.

I think we could keep it here and also pass and store it in RegionStates, if we use it in both places.

@Umeshkumar9414

Ping @Umeshkumar9414

We are not using it in RegionStates as of now.
I do thought of a use in RegionStateNode though, I was thinking to include activeTransitProcedureCount in RegionInTransitionTracker only. But then again in that way I had to pass it till RegionStateNode.

Apache-HBase · 2025-11-20T23:01:40Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 12s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 7s		master passed
+1 💚	compile	1m 18s		master passed
+1 💚	javadoc	0m 41s		master passed
+1 💚	shadedjars	6m 16s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 10s		the patch passed
+1 💚	compile	1m 19s		the patch passed
+1 💚	javac	1m 19s		the patch passed
+1 💚	javadoc	0m 41s		the patch passed
+1 💚	shadedjars	6m 13s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	264m 36s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
+1 💚	unit	4m 3s		hbase-testing-util in the patch passed.
		298m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/10/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7375
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux ca11717282bd 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c84ed4d`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/10/testReport/
Max. process+thread count	4127 (vs. ulimit of 30000)
modules	C: hbase-server hbase-testing-util U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/10/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Umeshkumar9414 · 2025-11-21T06:02:51Z

TestSecureIPC test failed in the build, I was not able to run this test successfully in my machine for master branch as well, becuase of keytab issue. Looks like I might need some changes in my machine to run these tests, that I am not aware of.

Umeshkumar9414 · 2025-11-24T17:21:04Z

@Apache9 can you help me with review ?

Apache9

We are almost done, thanks for the patience.

Now I'm OK with not putting RegionInTransitionStateTracker in RegionStates. The only left thing is about the isRegionInRegionStates method, do we really use this method in our code base? Can we just remove it?

Apache9 · 2025-11-27T09:48:12Z

+ // AssignmentManager calls setTableStateManager once hbase:meta is confirmed online, if it is
+ // still null it means confirmation is still pending. One should not access TableStateManger
+ // till the time.
+ if (TableName.isMetaTableName(tableName)) {


Maybe here we could add an assert TableName.isMetaTableName(tableName);, and then return true?
Anyway, throwing a RuntimeException is also acceptable as in production environment we may disable assertion.
Better add something like "CODE-BUG" to tell developers that you may write code wrong.

Somehow I try to avoid if else and liked the idea of assert. Will do that.

…ion of meta online

Apache-HBase · 2025-11-28T17:05:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 13s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 19s		Maven dependency ordering for branch
+1 💚	mvninstall	6m 56s		master passed
+1 💚	compile	6m 5s		master passed
+1 💚	checkstyle	1m 49s		master passed
+1 💚	spotbugs	3m 39s		master passed
+1 💚	spotless	1m 22s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s		Maven dependency ordering for patch
+1 💚	mvninstall	6m 48s		the patch passed
+1 💚	compile	6m 3s		the patch passed
+1 💚	javac	6m 3s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 34s		hbase-server: The patch generated 0 new + 45 unchanged - 2 fixed = 45 total (was 47)
+1 💚	checkstyle	0m 23s		The patch passed checkstyle in hbase-testing-util
+1 💚	spotbugs	5m 28s		the patch passed
+1 💚	hadoopcheck	19m 5s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 💚	spotless	1m 27s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		73m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/11/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7375
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 96c76c7301d1 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4af272b`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	71 (vs. ulimit of 30000)
modules	C: hbase-server hbase-testing-util U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/11/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-11-28T19:56:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 35s		master passed
+1 💚	compile	1m 25s		master passed
+1 💚	javadoc	0m 43s		master passed
+1 💚	shadedjars	6m 23s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 14s		the patch passed
+1 💚	compile	1m 19s		the patch passed
+1 💚	javac	1m 19s		the patch passed
+1 💚	javadoc	0m 42s		the patch passed
+1 💚	shadedjars	6m 13s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	212m 18s		hbase-server in the patch passed.
+1 💚	unit	2m 27s		hbase-testing-util in the patch passed.
		244m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/11/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7375
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux ae37f0c5564f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4af272b`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/11/testReport/
Max. process+thread count	4748 (vs. ulimit of 30000)
modules	C: hbase-server hbase-testing-util U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/11/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-11-28T20:55:13Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	2m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for branch
+1 💚	mvninstall	5m 2s		master passed
+1 💚	compile	4m 51s		master passed
+1 💚	checkstyle	1m 28s		master passed
+1 💚	spotbugs	2m 53s		master passed
+1 💚	spotless	1m 7s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s		Maven dependency ordering for patch
+1 💚	mvninstall	4m 24s		the patch passed
+1 💚	compile	4m 45s		the patch passed
+1 💚	javac	4m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 13s		hbase-server: The patch generated 0 new + 45 unchanged - 2 fixed = 45 total (was 47)
+1 💚	checkstyle	0m 12s		The patch passed checkstyle in hbase-testing-util
+1 💚	spotbugs	3m 4s		the patch passed
+1 💚	hadoopcheck	13m 21s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 💚	spotless	0m 56s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 23s		The patch does not generate ASF License warnings.
		55m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/12/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7375
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 928e2edb867e 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `3218382`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	70 (vs. ulimit of 30000)
modules	C: hbase-server hbase-testing-util U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/12/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-11-29T00:01:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
-0 ⚠️	yetus	0m 2s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 13s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 10s		master passed
+1 💚	compile	1m 17s		master passed
+1 💚	javadoc	0m 40s		master passed
+1 💚	shadedjars	6m 13s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 12s		the patch passed
+1 💚	compile	1m 19s		the patch passed
+1 💚	javac	1m 19s		the patch passed
+1 💚	javadoc	0m 40s		the patch passed
+1 💚	shadedjars	6m 13s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	212m 16s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
+1 💚	unit	2m 9s		hbase-testing-util in the patch passed.
		242m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/12/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7375
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 6b66c6cd5128 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `3218382`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/12/testReport/
Max. process+thread count	4419 (vs. ulimit of 30000)
modules	C: hbase-server hbase-testing-util U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7375/12/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache9

+1.

Thanks @Umeshkumar9414!

Umeshkumar9414 · 2025-11-30T19:02:32Z

Can we merge it @Apache9 ? I think @apurtell is out for some days and I was thinking if it can get merge before any new commit that conflict these changes.
I have opened PR for other branches as well except branch-2.6 where it was clean cherry pick from branch-2

Apache9 · 2025-12-01T15:31:08Z

I'm OK with merging it.

I'm in a business trip in Singapore, so let me see if I can get a device to merge and cherry-pick this to all branches...

Thanks.

Close#7447 Co-authored-by: ukumawat <ukumawat@salesforce.com> Signed-off-by: Andrew Purtell <apurtell@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 7d604d4)

d-c-manning · 2025-12-05T22:01:52Z

+ ServerName crashedServerName = scp.getServerName();
+ for (RegionInfo regionInfo : regionsOnCrashedServer) {
+ RegionStateNode node = regionStates.getOrCreateRegionStateNode(regionInfo);
+ if (node.getRegionLocation() == crashedServerName) {


is this always guaranteed to be comparable via ==? Or do we have to, or would it be safer to, use equals?

Ah, it should be equals, and we should log a warn when they do not equal, as this method is called before we assign any regions on the crashed region server, so the regionsOnCrashedServer should all be on the crashed server.
Let me prepare an addendum. Thanks @d-c-manning !

Thanks @d-c-manning, @Apache9 let me know if I can help here.

Umeshkumar9414 · 2025-12-08T06:35:33Z

Hi @Apache9, @apurtell , @virajjasani while building the phoenix with these changes I got to know that Phoenix uses isRegionInTransition method of RegionStates class that I removed with this change. And this version of pheonix will be facing compile issue after this change. Wanted to know what do we do in such cases ?

 [ERROR] /****/phoenix/phoenix-core/src/it/java/org/apache/phoenix/rpc/PhoenixServerRpcIT.java:[247,79] cannot find symbol
symbol: method isRegionInTransition(org.apache.hadoop.hbase.client.RegionInfo)
location: class org.apache.hadoop.hbase.master.assignment.RegionStates

virajjasani · 2025-12-08T06:40:45Z

@Umeshkumar9414 the usage seems to be in the test code so we can remove it if it is not really necessary, or use JMX metrics to find out RIT count in the test

apurtell · 2025-12-17T16:31:37Z

@Umeshkumar9414 @virajjasani

RegionStates is annotated @InterfaceAudience.Private so Phoenix was wrong to ever use this interface. Phoenix must solve the issue themselves and not rely on Private annotated code. This is well understood by now, it keeps coming up that they do this and get broken.

Umeshkumar9414 · 2025-12-17T16:49:13Z

@Umeshkumar9414 @virajjasani
RegionStates is annotated @InterfaceAudience.Private so Phoenix was wrong to ever use this interface. Phoenix must solve the issue themselves and not rely on Private annotated code. This is well understood by now, it keeps coming up that they do this and get broken.

Raised an JIRA https://issues.apache.org/jira/browse/PHOENIX-7741 and solved it in phoenix.

Umeshkumar9414 force-pushed the HBASE-28158_2 branch from af75dcb to 99caa2cCompare October 10, 2025 15:07