HBASE-28792: AsyncTableImpl should call coprocessor callbacks in a defined order

[charlesconnell]

To call a coprocessor endpoint asynchronously, you start by calling AsyncTable#coprocessorService(), which gives you a CoprocessorServiceBuilder, and a few steps later you can talk to your coprocessor over the network. One argument to AsyncTable#coprocessorService() is a CoprocessorCallback object, which contains several methods that will be called during the lifecycle of a coprocessor endpoint call. AsyncTableImpl's implementation of AsyncTable#coprocessorService() wraps your CoprocessorCallback with its own that delegates the work to a thread pool. A snippet of this:

 @Override
public void onRegionComplete(RegionInfo region, R resp) {
pool.execute(context.wrap(() -> callback.onRegionComplete(region, resp)));
}
...
@Override
public void onComplete() {
pool.execute(context.wrap(callback::onComplete));
}

The trouble with this is that your implementations of onRegionComplete() and onComplete() will end up getting called in a random order, and/or at the same time. The tasks of calling them are delegated to a thread pool, and the completion of those tasks is not waited on, so the thread pool can choose any ordering it wants to. Troublingly, onComplete() can be called before the final onRegionComplete(), which is an violation of the contract specified in the CoprocessorCallback#onComplete() javadoc.

This PR eliminates the use of a thread pool for calling the coprocessor callback, and calls them directly on the thread that received the response from the server.

[Apache9]

The intention here is that, in AsyncTableImpl, we will always call any callbacks in a thread pool, in order to prevent blocking operations in callbacks blocking the system thread(like netty event loop). If you want high performance, you can use RawAsyncTable directly.

Here I think we should still use the thread pool, but use some tricks to let onComplete to be called at last.

Thanks.

[charlesconnell]

@Apache9 Understood. I've brought back the thread pool usage and added a barrier to force onComplete() to wait until onRegionComplete() is done.

[ndimiduk]

I agree with Duo about the threadpool bit. I think this Phaser approach works fine. I'm curious if there's a "better" implementation based on chaining CompletableFutures ... I'll need to spend some time with the APIs to see if I have a suggestion.

[ndimiduk]

So unfortunately our CoprocessorCallback API seems to completely preclude use of the CompletableFuture chaining strategy. I think you're on the right path here.

[ndimiduk]

According to the Javadoc on this method, onComplete must implement a happens-after semantic for all onRegionComplete and all onRegionError invocations. This implies that there are multiple regions to complete AND multiple forms of completion, so I think that this can be solved with a phaser ; you need a more general purpose mutex.

[charlesconnell]

Good point that onRegionError should be included in the barrier. I'm guessing you meant to say "so I think that this cannot be solved with a phaser," but I disagree. The phaser can be used in onRegionError just the same as onRegionComplete.

[ndimiduk]

Oh I see. Phaser is more powerful than I initially understood -- very cool! Yes, this looks better to me.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 26s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 38s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 17s		master passed
+1 💚	compile	0m 38s		master passed
+1 💚	javadoc	0m 27s		master passed
+1 💚	shadedjars	5m 34s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 55s		the patch passed
+1 💚	compile	0m 37s		the patch passed
+1 💚	javac	0m 37s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed
+1 💚	shadedjars	5m 36s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	1m 36s		hbase-client in the patch passed.
+1 💚	unit	3m 31s		hbase-endpoint in the patch passed.
		27m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6168/6/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6168
JIRA Issue	HBASE-28792
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 1e9ab3d364db 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 5573a60
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6168/6/testReport/
Max. process+thread count	1383 (vs. ulimit of 30000)
modules	C: hbase-client hbase-endpoint U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6168/6/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 10s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 30s		master passed
+1 💚	compile	1m 27s		master passed
+1 💚	checkstyle	0m 31s		master passed
+1 💚	spotbugs	1m 26s		master passed
+1 💚	spotless	0m 59s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 21s		the patch passed
+1 💚	compile	1m 23s		the patch passed
+1 💚	javac	1m 23s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 29s		the patch passed
+1 💚	spotbugs	1m 35s		the patch passed
+1 💚	hadoopcheck	12m 13s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 53s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 21s		The patch does not generate ASF License warnings.
		36m 49s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6168/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6168
JIRA Issue	HBASE-28792
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 8ada949607ee 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 5573a60
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	82 (vs. ulimit of 30000)
modules	C: hbase-client hbase-endpoint U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6168/6/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[ndimiduk]

Nice fix.

[ndimiduk]

Oh I see. Phaser is more powerful than I initially understood -- very cool! Yes, this looks better to me.

[Apache9]

Ah, just 5 minutes late...

LGTM.

Thanks @charlesconnell and @ndimiduk !

[ndimiduk]

All good. Welcome back @Apache9 !