HBASE-29715: AssignmentManager is trying to pick up the active cluster's tables before refreshing meta and hfiles by kgeisz · Pull Request #7474 · apache/hbase

kgeisz · 2025-11-20T03:32:23Z

https://issues.apache.org/jira/browse/HBASE-29715

In a read-replica setup, two or more clusters share the same filesystem. This means the replica cluster can pick up tables created by the active cluster during metrics collection, even when the replica cluster has not had its metadata and HFiles refreshed yet. PR #7304 introduced some logic to the TABLE_TO_REGIONS_COUNT case in HMaster.getClusterMetricsWithoutCoprocessor() that skips metrics collection for foreign meta tables.

This pull request broadens this logic to skip all foreign tables. This change will prevent the replica cluster from logging something like the following:

2025-11-14T03:22:08,098 ERROR [master/hbase-docker-2:16000.Chore.1] master.TableStateManager: Unable to get table testTable1 state
org.apache.hadoop.hbase.TableNotFoundException: No state found for testTable1
at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:141) ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:79) ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at org.apache.hadoop.hbase.master.assignment.AssignmentManager.isTableDisabled(AssignmentManager.java:549) ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at org.apache.hadoop.hbase.master.assignment.AssignmentManager.getRegionStatesCount(AssignmentManager.java:2637) ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at org.apache.hadoop.hbase.master.HMaster.getClusterMetricsWithoutCoprocessor(HMaster.java:3130) ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at org.apache.hadoop.hbase.master.HMaster.getClusterMetricsWithoutCoprocessor(HMaster.java:3017) ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at org.apache.hadoop.hbase.master.balancer.ClusterStatusChore.chore(ClusterStatusChore.java:47) ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:161) ~[hbase-common-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at org.apache.hadoop.hbase.trace.TraceUtil.lambda$tracedRunnable$2(TraceUtil.java:155) ~[hbase-common-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) ~[hbase-common-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]

With this PR's changes, the following will be logged instead:

2025-11-20T04:17:29,535 INFO [master/hbase-docker-2:16000.Chore.1] master.HMaster: Skipping foreign table testTable1 in cluster metrics

Change-Id: I976431d363df438c95feaa2c0d5cc23028858947

Change-Id: I8cebc36512c7f59b437d74dfc376fb49d95bf422

kgeisz · 2025-11-20T03:47:28Z

I noticed PR #7304 also made changes to FSTableDescriptors.java and HbckChore.java via FSUtils.isLocalMetaTable(). Currently, this PR does not make any changes to those files.

anmolnar

lgtm.

anmolnar · 2025-12-09T21:43:43Z

@Kota-SH @kgeisz Are the unit test failures related? Can we move on with merging this PR?

Kota-SH

LGTM, The unit test failures do not seem related to the change, they pass locally.
However, I have added a minor comment about using listTableDescriptors() to avoid duplication.

Kota-SH · 2025-12-10T22:55:25Z

@@ -3117,13 +3117,11 @@ public ClusterMetrics getClusterMetricsWithoutCoprocessor(EnumSet<Option> option
 try {
 Map<TableName, RegionStatesCount> tableRegionStatesCountMap = new HashMap<>();
 Map<String, TableDescriptor> tableDescriptorMap = getTableDescriptors().getAll();


Can we do a single call to HMaster.listTableDescriptors() instead of calculating the tablenames twice?
It has internal filter to check if the tablename is present, so we wouldn't need to manually filter again.
List<TableDescriptor> tableDescriptors = listTableDescriptors(null, null, null, true);

Nice find! This is much cleaner.

I have made changes based on your suggestion. Let me know if it's what you had in mind.

lgtm @kgeisz, thanks.

Change-Id: I2001fd4f8289ee6c74621bd6d78f642e77e78129

taklwu · 2025-12-11T01:10:32Z

+ TableDescriptor foreignTableDescriptor;
+ for (TableName tableName : allTables) {
+ foreignTableDescriptor = TableDescriptorBuilder.newBuilder(tableName)
+ .setColumnFamily(ColumnFamilyDescriptorBuilder.of("cf")).build();
+ master.getTableDescriptors().update(foreignTableDescriptor, true);
+ }


nit: I have a question about this TABLE_TO_REGIONS_COUNT and master.getTableDescriptors().update(foreignTableDescriptor, true); } in this test.
in the read replica cluster, if the newly added table is not synced by meta and hfiles operation (not the meta table only), how does load into the tableDescriptors ? the way you have here is implying the HMaster of read replica cluster is using master.getTableDescriptors().update(foreignTableDescriptor, true); or somehow read from the filesystem directly。
once those table not in the meta of replica cluster loaded and if tableStateManager (the current meta of the replica cluster) does not have this state of this foreignable table , the change in this PR would work.

The table descriptors were coming from the filesystem. The replica cluster was seeing these, and if its meta had not been refreshed, then any foreign table descriptors could not have their state found.

This line in HMaster:
Map<String, TableDescriptor> tableDescriptorMap = getTableDescriptors().getAll();
was using HBaseServerBase.getTableDescriptors() rather than using HMaster.getTableDescriptors().
@Kota-SH Suggested I use HMaster.listTableDescriptors() (which uses HMaster.getTableDescriptors()) so the filtering is done automatically. I have pushed that change recently.

so, I use debug mode to check the unit test, I found that listTableDescriptors is always using the cache of FSTableDescriptors instead of loading from the filesystem, such that we're relying on the tableStateManager via meta to see if the additional loaded tables should be included.
the test itself is right, just I'm wondered practically how newly added table on the primary/write cluster being added to the cache of FSTableDescriptors after master has been initialized ? or does it add from the filesystem after restarting the master (meta is yet synced)?
so, please consider to test it manually and see if that's what you expected in HBASE-29579

kgeisz · 2025-12-11T02:00:16Z

@anmolnar The failed unit tests are passing for me as well when I run them locally.

Apache-HBase · 2025-12-11T02:46:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ HBASE-29081 Compile Tests _
+1 💚	mvninstall	3m 39s		HBASE-29081 passed
+1 💚	compile	3m 38s		HBASE-29081 passed
-0 ⚠️	checkstyle	0m 15s	/buildtool-branch-checkstyle-hbase-server.txt	The patch fails to run checkstyle in hbase-server
+1 💚	spotbugs	1m 40s		HBASE-29081 passed
+1 💚	spotless	0m 52s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 16s		the patch passed
+1 💚	compile	3m 31s		the patch passed
+1 💚	javac	3m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 13s	/buildtool-patch-checkstyle-hbase-server.txt	The patch fails to run checkstyle in hbase-server
+1 💚	spotbugs	1m 42s		the patch passed
+1 💚	hadoopcheck	12m 22s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
-1 ❌	spotless	0m 39s		patch has 26 errors when running spotless:check, run spotless:apply to fix.
			_ Other Tests _
+1 💚	asflicense	0m 12s		The patch does not generate ASF License warnings.
		40m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7474/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7474
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 8e872dc59110 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-29081 / `35e5cba`
Default Java	Eclipse Adoptium-17.0.11+9
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7474/2/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7474/2/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Kota-SH

LGTM

taklwu · 2025-12-11T04:57:53Z

+List<TableDescriptor> tableDescriptors = listTableDescriptors(null, null, null, true);
+ for (TableDescriptor tableDescriptor : tableDescriptors) {


can we have manual test for this change? I'm wondered if this would filter any table (non-metatable that filter by FSUtils.java#isLocalMetaTable) that should be considered as foreign to the primary/writer cluster.
listTableDescriptors should be loading all the tables from the filesystem via FSTableDescriptors.java#getAll(), where this change does not have a way to filter the user/foreign table that should not be synced. (should the chore be always refreshing from filesystem when the meta sync does not trigger from the read-only replica?)
or are we just trying to remove the redundant logic of rechecking the foreign meta table ?
but anyhow, the unit test flow does not match the title and please try to come up a manual test or change the unit test to act as the primary and the read-only replica.

To my understanding this logic has been changed to work only with tables which are present in replica cluster's meta table.
Previously it used HBaseServerBase.getTableDescriptors().getAll() which eventually calls FSTableDescriptors.getAll() method which loads all table from the filesystem. (only once, because it caches the result)
Now, in this patch, we call HMaster.listTableDescriptors() which ends up calling the same FSTableDescriptors.getAll(), but before returning the list it, filters for the tablenames which are present in replica cluster's meta table.
for (TableDescriptordesc : allHtds) { if ( tableStateManager.isTablePresent(desc.getTableName()) && (includeSysTables || !desc.getTableName().isSystemTable()) ) { htds.add(desc); } }
I wonder why don't we just get the list directly from meta, but either way the patch looks good to me.

I wonder why don't we just get the list directly from meta, but either way the patch looks good to me.
It would be a full scan on the meta table VS currently we do a GET on meta for every table that we find on the filesystem.
I'm not sure which one is more efficient.

you probably clean my concern that we're comparing the meta state in tableStateManager vs the table descriptor found on the filesystem.
just again the unit test is doing slightly different, it called master.getTableDescriptors().update(foreignTableDescriptor, true) to update the cache of the tableDescriptors/FSTableDescriptors (not writing the TableDescriptors on actual file system) with the additional tables, and then compare with the tableStateManager when TABLE_TO_REGIONS_COUNT hook is being called.
since FSTableDescriptors.getAll() is always using the cache after the master is initialized (not reloading from the filesystem), the test is partially correct that neither the tableStateManager of meta or FSTableDescriptors.getAll() from the view of filesystem does not have those unsynced table, or tableStateManager does not have those additional loaded table in cache of FSTableDescriptors
I should have used debug mode to clear the confusion that the FSTableDescriptors.getAll() in the unit test are always from cache, thanks again.

I should have used debug mode to clear the confusion that the FSTableDescriptors.getAll() in the unit test are always from cache, thanks again.
Sounds like a reasonable improvement on the testing side that we should do in this patch @kgeisz .

asked clarification about the updated patch

Apache-HBase · 2025-12-11T06:17:49Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 22s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ HBASE-29081 Compile Tests _
+1 💚	mvninstall	2m 47s		HBASE-29081 passed
+1 💚	compile	0m 43s		HBASE-29081 passed
+1 💚	javadoc	0m 23s		HBASE-29081 passed
+1 💚	shadedjars	4m 29s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 15s		the patch passed
+1 💚	compile	0m 44s		the patch passed
+1 💚	javac	0m 44s		the patch passed
+1 💚	javadoc	0m 20s		the patch passed
+1 💚	shadedjars	4m 26s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	229m 47s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
		250m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7474/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7474
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux fee7bb96f70d 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-29081 / `35e5cba`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7474/2/testReport/
Max. process+thread count	6150 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7474/2/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Changes have been made.

anmolnar

Overall lgtm.

taklwu · 2025-12-15T17:08:04Z

@kgeisz please create a follow up to add a test for fresh replica setup that loads from filesystem (instead of the currently one that is loaded from cache.)

…r's tables before refreshing meta and hfiles (#7474) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Signed-off-by: Andor Molnár <andor@apache.org> Reviewed by: Kota-SH <shanmukhaharipriya@gmail.com>

…r's tables before refreshing meta and hfiles (apache#7474) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Signed-off-by: Andor Molnár <andor@apache.org> Reviewed by: Kota-SH <shanmukhaharipriya@gmail.com>

…r's tables before refreshing meta and hfiles (#7474) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Signed-off-by: Andor Molnár <andor@apache.org> Reviewed by: Kota-SH <shanmukhaharipriya@gmail.com>

kgeisz added 2 commits November 19, 2025 19:20

Skip all foreign tables when collecting metrics
ec8d575
Change-Id: I976431d363df438c95feaa2c0d5cc23028858947

Run mvn spotless:apply
7c9c85a
Change-Id: I8cebc36512c7f59b437d74dfc376fb49d95bf422

This comment has been minimized.
Sign in to view

anmolnar previously approved these changes Dec 9, 2025
View reviewed changes

Kota-SH reviewed Dec 10, 2025
View reviewed changes

Use HMaster.listTableDescriptors() to take advantage of internal filter
35e5cba
Change-Id: I2001fd4f8289ee6c74621bd6d78f642e77e78129

taklwu previously approved these changes Dec 11, 2025
View reviewed changes

taklwu self-requested a review December 11, 2025 03:16

Kota-SH approved these changes Dec 11, 2025
View reviewed changes

taklwu reviewed Dec 11, 2025
View reviewed changes

taklwu self-requested a review December 11, 2025 04:58

anmolnar self-requested a review December 11, 2025 15:59

anmolnar approved these changes Dec 11, 2025
View reviewed changes

taklwu approved these changes Dec 11, 2025
View reviewed changes

taklwu merged commit e08c367 into apache:HBASE-29081Dec 15, 2025
1 check failed

kgeisz mentioned this pull request Feb 4, 2026
HBASE-29780: Addendum to HBASE-29715: Add an additional test case that uses the filesystem #7702
Merged

		List<TableDescriptor> tableDescriptors = listTableDescriptors(null, null, null, true);
		for (TableDescriptor tableDescriptor : tableDescriptors) {

Uh oh!

Conversation

kgeisz commented Nov 20, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kgeisz commented Nov 20, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

anmolnar left a comment

Choose a reason for hiding this comment

Uh oh!

anmolnar commented Dec 9, 2025

Uh oh!

Kota-SH left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kgeiszDec 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taklwuDec 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kgeisz commented Dec 11, 2025

Uh oh!

Apache-HBase commented Dec 11, 2025

Uh oh!

Kota-SH left a comment

Choose a reason for hiding this comment

Uh oh!

taklwuDec 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anmolnarDec 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taklwuDec 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Dec 11, 2025

Uh oh!

anmolnar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

taklwu commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kgeisz commented Nov 20, 2025•
edited
Loading

kgeiszDec 11, 2025•
edited
Loading

taklwuDec 11, 2025•
edited
Loading

taklwuDec 11, 2025•
edited
Loading

anmolnarDec 11, 2025•
edited
Loading

taklwuDec 11, 2025•
edited
Loading