HBASE-29831: Fix for NPE in region replication by krconv · Pull Request #7629 · apache/hbase

krconv · 2026-01-15T11:46:08Z

The RegionReplicaSinkWriter.append() method checks table descriptors to determine if a table has region replication enabled (to decide whether to bypass the location cache). When a table is dropped concurrently, tableDescriptors.get(tableName) returns null, and the subsequent call to getRegionReplication() throws a NullPointerException.

This race condition can occur in the following scenario:

WAL entries for a table are queued for replication to region replicas
The table is dropped (via disable + drop or other means)
Before the dropped table is added to the disabledAndDroppedTables cache (which happens when TableNotFoundException is caught during location lookup), the code attempts to read the table descriptor
tableDescriptors.get() returns null for the now-deleted table
NPE crashes the replication endpoint

Since RegionReplicaReplicationEndpoint handles replica updates for all tables on a RegionServer, a single dropped table crashes the entire endpoint. This stops replica updates for all regions (including those from unrelated tables) hosted by that RegionServer until it is restarted.

guluo2016 · 2026-01-16T16:09:56Z

 if (useCache && locations.size() == 1) {
- if (tableDescriptors.get(tableName).getRegionReplication() > 1 && retries <= 3) {
+ TableDescriptor td = tableDescriptors.get(tableName);
+ if (td != null && td.getRegionReplication() > 1 && retries <= 3) {


I noticed through idea's smart suggestions that retries <= 3 seems is unnecessary.
And I analyzed it and it's true
After removing it, there are 3 main cases, and none lead to an infinite loop:
case 1
First loop: useCache && locations.size() == 1 && RegionReplication > 1 is true.
Set useCache = false and continue.
Second loop: The logic will proceed and eventually return or break.
case2
First loop: useCache && locations.size() == 1 is true but RegionReplication > 1 is false.
Go to subsequent logic.
If !Bytes.equals(primaryLocation.getRegionInfo().getEncodedNameAsBytes(), encodedRegionName) is false: break (loop ends).
If !Bytes.equals(primaryLocation.getRegionInfo().getEncodedNameAsBytes(), encodedRegionName) is true and useCache is true: set useCache = false and continue. Second loop will then return or break.
case3
First loop: useCache && locations.size() == 1 is false.
Go to subsequent logic.
If useCache is alread false: return or break.
If useCache is true : similar to case 2, it will either break or retry once (setting useCache=false), then finish.

Thanks for walking through the logic here; I agree it is unneeded, just removed it

guluo2016 · 2026-01-16T16:11:32Z

Is it possible to add a unit test for this? Thanks

chandrasekhar-188k

LGTM

krconv · 2026-01-20T17:16:33Z

Thanks for the reviews! Added a new unit test that catches the original problem, and removed the unneeded retries variable. Also, we encountered this problem on 100s of hosts across all of the data centers where we use HBase last week, hopefully this fix helps others

Apache-HBase · 2026-01-20T22:25:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 15s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ branch-2 Compile Tests _
+1 💚	mvninstall	4m 29s		branch-2 passed
+1 💚	compile	3m 45s		branch-2 passed
+1 💚	checkstyle	0m 46s		branch-2 passed
+1 💚	spotbugs	1m 53s		branch-2 passed
+1 💚	spotless	0m 55s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	4m 6s		the patch passed
+1 💚	compile	3m 49s		the patch passed
+1 💚	javac	3m 49s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 47s		the patch passed
+1 💚	spotbugs	2m 15s		the patch passed
+1 💚	hadoopcheck	24m 35s		Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6 3.4.1.
+1 💚	spotless	1m 2s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 13s		The patch does not generate ASF License warnings.
		52m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7629/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7629
JIRA Issue	HBASE-29831
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux f5e8bc2464c7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `1f43067`
Default Java	Eclipse Adoptium-11.0.23+9
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7629/3/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2026-01-21T01:45:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 48s		Docker mode activated.
-0 ⚠️	yetus	0m 5s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2 Compile Tests _
+1 💚	mvninstall	3m 8s		branch-2 passed
+1 💚	compile	0m 59s		branch-2 passed
+1 💚	javadoc	0m 28s		branch-2 passed
+1 💚	shadedjars	6m 10s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 10s		the patch passed
+1 💚	compile	0m 59s		the patch passed
+1 💚	javac	0m 59s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed
+1 💚	shadedjars	6m 9s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	226m 14s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
		253m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7629/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7629
JIRA Issue	HBASE-29831
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 70295dbf72dd 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `1f43067`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7629/3/testReport/
Max. process+thread count	3430 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7629/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2026-01-21T02:08:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 19s		Docker mode activated.
-0 ⚠️	yetus	0m 6s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2 Compile Tests _
+1 💚	mvninstall	4m 54s		branch-2 passed
+1 💚	compile	1m 11s		branch-2 passed
+1 💚	javadoc	0m 40s		branch-2 passed
+1 💚	shadedjars	7m 41s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	4m 8s		the patch passed
+1 💚	compile	0m 59s		the patch passed
+1 💚	javac	0m 59s		the patch passed
+1 💚	javadoc	0m 30s		the patch passed
+1 💚	shadedjars	7m 24s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	240m 53s		hbase-server in the patch passed.
		275m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7629/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#7629
JIRA Issue	HBASE-29831
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 6941778e670d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `1f43067`
Default Java	Eclipse Adoptium-11.0.23+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7629/3/testReport/
Max. process+thread count	3283 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7629/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2026-01-21T03:19:35Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 11s		Docker mode activated.
-0 ⚠️	yetus	0m 5s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2 Compile Tests _
+1 💚	mvninstall	4m 20s		branch-2 passed
+1 💚	compile	1m 14s		branch-2 passed
+1 💚	javadoc	0m 49s		branch-2 passed
+1 💚	shadedjars	8m 36s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 55s		the patch passed
+1 💚	compile	1m 8s		the patch passed
+1 💚	javac	1m 8s		the patch passed
+1 💚	javadoc	0m 38s		the patch passed
+1 💚	shadedjars	7m 27s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	312m 13s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
		346m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7629/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#7629
JIRA Issue	HBASE-29831
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 46be563ba736 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `1f43067`
Default Java	Temurin-1.8.0_412-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7629/3/testReport/
Max. process+thread count	3389 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7629/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

guluo2016

LGTM, thanks

krconv · 2026-01-23T18:10:22Z

@guluo2016 or @chandrasekhar-188k, would one of you mind merging this for me?

Apache9 · 2026-01-25T14:33:05Z

This does not affect master and branch-3?

krconv · 2026-01-26T16:02:33Z

From what I can tell it only affects HBase 2. It looks like the implementation was rewritten for HBase 3, replacing the RegionReplicaReplicationEndpoint with RegionReplicationSink. The new implementation in branch-3/master doesn't seem to suffer from this same issue, because the RegionReplicationSink has an immutable TableDescriptor (instead of attempting to periodically poll for table descriptor updates like the branch-2 implementation, branch-3/master recreate the sink when the region reopens) and so it can't hit a NPE when a table is dropped.

guluo2016 · 2026-01-28T14:27:33Z

Sorry for the late reply, a bit busy recently.
I manually tested it, it is ok.

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.10 s -- in org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- surefire:3.5.3:test (secondPartTestsExecution) @ hbase-server ---
[INFO] Tests are skipped.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache HBase 2.7.0-SNAPSHOT:
[INFO]
[INFO] Apache HBase ....................................... SUCCESS [ 1.606 s]
[INFO] Apache HBase - Checkstyle .......................... SUCCESS [ 0.484 s]
[INFO] Apache HBase - Annotations ......................... SUCCESS [ 0.330 s]
[INFO] Apache HBase - Build Configuration ................. SUCCESS [ 0.158 s]
[INFO] Apache HBase - Logging ............................. SUCCESS [ 0.501 s]
[INFO] Apache HBase - Shaded Protocol ..................... SUCCESS [ 16.933 s]
[INFO] Apache HBase - Common .............................. SUCCESS [ 3.668 s]
[INFO] Apache HBase - Metrics API ......................... SUCCESS [ 1.372 s]
[INFO] Apache HBase - Hadoop Compatibility ................ SUCCESS [ 1.284 s]
[INFO] Apache HBase - Metrics Implementation .............. SUCCESS [ 0.815 s]
[INFO] Apache HBase - Hadoop Two Compatibility ............ SUCCESS [ 1.927 s]
[INFO] Apache HBase - Protocol ............................ SUCCESS [ 4.494 s]
[INFO] Apache HBase - Client .............................. SUCCESS [ 2.285 s]
[INFO] Apache HBase - Zookeeper ........................... SUCCESS [ 1.339 s]
[INFO] Apache HBase - Replication ......................... SUCCESS [ 1.242 s]
[INFO] Apache HBase - Resource Bundle ..................... SUCCESS [ 0.107 s]
[INFO] Apache HBase - HTTP ................................ SUCCESS [ 0.563 s]
[INFO] Apache HBase - Asynchronous FileSystem ............. SUCCESS [ 2.261 s]
[INFO] Apache HBase - Procedure ........................... SUCCESS [ 1.222 s]
[INFO] Apache HBase - Server .............................. SUCCESS [ 26.601 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Thanks @krconv

Signed-off-by: Chandra Kambham <chandra@apache.org> Signed-off-by: Peng Lu <lupeng@apache.org>

guluo2016 · 2026-01-28T14:53:43Z

@krconv , The patch can not be applied to branch-2.5 cleanly. Please open a new PR for branch-2.5, thanks a lot!

Signed-off-by: Chandra Kambham <chandra@apache.org> Signed-off-by: Peng Lu <lupeng@apache.org>

krconv · 2026-01-28T16:30:40Z

@guluo2016 Thank you! Just put up the PR for branch-2.5 here: #7684

Fix for NPE in region replication
9adf8a9

This comment has been minimized.
Sign in to view

charlesconnell changed the title ~~Fix for NPE in region replication~~HBASE-29831: Fix for NPE in region replicationJan 15, 2026

charlesconnell self-requested a review January 15, 2026 12:56

This comment has been minimized.
Sign in to view

guluo2016 reviewed Jan 16, 2026
View reviewed changes

chandrasekhar-188k approved these changes Jan 20, 2026
View reviewed changes

krconv added 2 commits January 20, 2026 12:11

Remove unneeded retries variable
72024e3

Add a unit test for NPE
1f43067

krconv force-pushed the HBASE-29831-read-replicas-npe branch from d95de47 to 1f43067Compare January 20, 2026 17:15

This comment has been minimized.
Sign in to view

krconv requested a review from guluo2016 January 21, 2026 17:45

guluo2016 approved these changes Jan 22, 2026
View reviewed changes

guluo2016 merged commit 758dc33 into apache:branch-2Jan 28, 2026
0 of 2 checks passed

guluo2016 pushed a commit that referenced this pull request Jan 28, 2026
HBASE-29831Fix for NPE in region replication (#7629)
a2335f8
Signed-off-by: Chandra Kambham <chandra@apache.org> Signed-off-by: Peng Lu <lupeng@apache.org>

krconv added a commit to HubSpot/hbase that referenced this pull request Jan 28, 2026
HBASE-29831 Fix for NPE in region replication (apache#7629)
d89b861
Signed-off-by: Chandra Kambham <chandra@apache.org> Signed-off-by: Peng Lu <lupeng@apache.org>

krconv mentioned this pull request Jan 28, 2026
HBASE-29831 Fix for NPE in region replication (branch-2.5 backport) #7684
Closed

krconv deleted the HBASE-29831-read-replicas-npe branch February 2, 2026 11:56

Uh oh!

Conversation

krconv commented Jan 15, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

guluo2016Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

krconvJan 20, 2026

Choose a reason for hiding this comment

Uh oh!

guluo2016 commented Jan 16, 2026

Uh oh!

chandrasekhar-188k left a comment

Choose a reason for hiding this comment

Uh oh!

krconv commented Jan 20, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented Jan 20, 2026

Uh oh!

Apache-HBase commented Jan 21, 2026

Uh oh!

Apache-HBase commented Jan 21, 2026

Uh oh!

Apache-HBase commented Jan 21, 2026

Uh oh!

guluo2016 left a comment

Choose a reason for hiding this comment

Uh oh!

krconv commented Jan 23, 2026

Uh oh!

Apache9 commented Jan 25, 2026

Uh oh!

krconv commented Jan 26, 2026

Uh oh!

guluo2016 commented Jan 28, 2026

Uh oh!

Uh oh!

guluo2016 commented Jan 28, 2026

Uh oh!

krconv commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

krconv commented Jan 20, 2026•
edited
Loading