HBASE-27093 AsyncNonMetaRegionLocator：put Complete CompletableFuture …

[xiaowangzhixiao]

…outside lock block

[Apache9]

Mind provide some deadlock scenarios?

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 32s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 15s	master passed
+1 💚	compile	0m 17s	master passed
+1 💚	shadedjars	4m 21s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 17s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 3s	the patch passed
+1 💚	compile	0m 17s	the patch passed
+1 💚	javac	0m 17s	the patch passed
+1 💚	shadedjars	4m 21s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 15s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 22s	hbase-client in the patch passed.
		22m 1s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4496/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4496
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 57e2c59b0f87 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 93996bd
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4496/1/testReport/
Max. process+thread count	195 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4496/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	5m 58s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 56s	master passed
+1 💚	compile	0m 18s	master passed
+1 💚	shadedjars	4m 18s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 17s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 44s	the patch passed
+1 💚	compile	0m 19s	the patch passed
+1 💚	javac	0m 19s	the patch passed
+1 💚	shadedjars	4m 17s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 16s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 7s	hbase-client in the patch passed.
		23m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4496/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4496
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 138e1f5d9cc9 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 93996bd
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4496/1/testReport/
Max. process+thread count	160 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4496/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	6m 21s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 29s	master passed
+1 💚	compile	0m 42s	master passed
+1 💚	checkstyle	0m 17s	master passed
+1 💚	spotless	0m 47s	branch has no errors when running spotless:check.
+1 💚	spotbugs	0m 46s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 5s	the patch passed
+1 💚	compile	0m 44s	the patch passed
+1 💚	javac	0m 44s	the patch passed
+1 💚	checkstyle	0m 14s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	15m 7s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotless	0m 46s	patch has no errors when running spotless:check.
+1 💚	spotbugs	0m 54s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		39m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4496/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4496
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 6173070fced5 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 93996bd
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	70 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4496/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[xiaowangzhixiao]

Thank you for attention. @Apache9
For example, when request hbase in a outer lock, process response in this lock, and don't isolate the thread, it will cause deadlock. As the caller, we shouldn't care the lock in the hbase client.

CompletableFuture<Result> getResult(get) {
....
lock.lock();
future = asyncTable.get(get);
lock.unlock();
return future;
...
}
...
getResult(get).handle((v,e)->{
lock.lock();
...
lock.unlock();
});

In getResult(), the order of lock is lock.lock() -> tableCache.lock(). But when the regionLocation response have exception, future.completeExceptionally(err) will trigger this future's handle action, and the order of lock is tableCache.lock() -> lock.lock(). So it will cause deadlock.

Of course we can solve this problem by isolating threads, using handleAsync(xxxx, executor), but I think we can also avoid this in HBase client.

[Apache9]

For example, when request hbase in a outer lock, process response in this lock, and don't isolate the thread, it will cause deadlock. As the caller, we shouldn't care the lock in the hbase client.

CompletableFuture<Result> getResult(get) {
....
lock.lock();
future = asyncTable.get(get);
lock.unlock();
return future;
...
}
...
getResult(get).handle((v,e)->{
lock.lock();
...
lock.unlock();
});

In getResult(), the order of lock is lock.lock() -> tableCache.lock(). But when the regionLocation response have exception, future.completeExceptionally(err) will trigger this future's handle action, and the order of lock is tableCache.lock() -> lock.lock(). So it will cause deadlock.

OK, typical usage, sound reasonable.