HBASE-25338 Use ClassLoader of current class instead of thread context ClassLoader

[challenger71498]

Details

When the ServiceLoader::load method is invoked with a single argument, it uses the context ClassLoader of current thread(by calling Thread.currentThread.getContextClassLoader()). (For definition, see here.)

Under normal circumstances, this does not cause issues. But there is a problem if you try to:

• load a class at runtime from a jar file (e.g. call URLClassLoader::loadClass)
  • (the jar file has a dependency with hbase-client)
• from the parent Application.

In my case, I encountered this problem while developing JDBC wrapper for hbase-shell.

If you load the class from the jar file, the ClassLoader of current thread is normally the class loader used to load the parent application.

• This "parent" ClassLoader is not adaquate to ServiceLoader at the SaslClientAuthenticationProviders.
• The ServiceLoader with the "parent" ClassLoader could not access the service info of the jar file(META-INF/services).
• Therefore, IllegalStateException is being thrown.

I also attached the full stacktrace of the exception just in case.

Stacktrace

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=2, exceptions:
2025-04-19T06:02:00.404Z, RpcRetryingCaller{globalStartTime=2025-04-19T06:02:00.241Z, pause=100, maxAttempts=2}, java.io.IOException: Call to address=hbase:16020 failed on local exception: java.io.IOException: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.
2025-04-19T06:02:00.518Z, RpcRetryingCaller{globalStartTime=2025-04-19T06:02:00.241Z, pause=100, maxAttempts=2}, java.io.IOException: Call to address=hbase:16020 failed on local exception: java.io.IOException: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:142)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.IOException: Call to address=hbase:16020 failed on local exception: java.io.IOException: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:270)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:395)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:430)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:116)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:131)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:451)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:94)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:606)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:44040)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:365)
at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:261)
at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:62)
at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:131)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:206)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:437)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:405)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104)
... 4 more
Caused by: java.io.IOException: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.
at org.apache.hadoop.hbase.ipc.IPCUtil.toIOE(IPCUtil.java:190)
... 17 more
Caused by: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.
at org.apache.hadoop.hbase.security.provider.BuiltInProviderSelector.configure(BuiltInProviderSelector.java:93)
at org.apache.hadoop.hbase.security.provider.SaslClientAuthenticationProviders.instantiateSelector(SaslClientAuthenticationProviders.java:114)
at org.apache.hadoop.hbase.security.provider.SaslClientAuthenticationProviders.instantiate(SaslClientAuthenticationProviders.java:187)
at org.apache.hadoop.hbase.security.provider.SaslClientAuthenticationProviders.getInstance(SaslClientAuthenticationProviders.java:76)
at org.apache.hadoop.hbase.ipc.RpcConnection.<init>(RpcConnection.java:138)
at org.apache.hadoop.hbase.ipc.NettyRpcConnection.<init>(NettyRpcConnection.java:108)
at org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:91)
at org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:47)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.lambda$getConnection$0(AbstractRpcClient.java:368)
at org.apache.hadoop.hbase.util.PoolMap.createResource(PoolMap.java:127)
at org.apache.hadoop.hbase.util.PoolMap$RoundRobinPool.getOrCreate(PoolMap.java:211)
at org.apache.hadoop.hbase.util.PoolMap.getOrCreate(PoolMap.java:68)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:368)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:448)
... 16 more

I could reproduce this issue with the repository which I made by myself in here. You could also look into it for more information.

Suggestion

As @AbilashR mentioned at the issue, this can be fixed by adding ClassLoader of current class explicitly to the parameter.

I have run the test locally and had no issues. Please let me know if there are more tests I should run.

I also tried to add a test case for this, but gave up since reproducing the issue is quite complicated.

Any opinions & suggestions are appriciated. Also, I don't have much experiences in Java, so feel free to make any corrections.

Thank you :)

[challenger71498]

Thankfully there is a workaround for this issue.

By setting extra providers option to built-in providers, you may bypass the issue and load the built-in providers.

For example:

<property>
<name>hbase.client.sasl.provider.extras</name>
<value>org.apache.hadoop.hbase.security.provider.SimpleSaslClientAuthenticationProvider,org.apache.hadoop.hbase.security.provider.GssSaslClientAuthenticationProvider,org.apache.hadoop.hbase.security.provider.DigestSaslClientAuthenticationProvider</value>
</property>

See here for more context.

[Apache9]

Please add a new commit which modifies pom.xml so we can trigger tests for the whole project. We can remove it while merging.

Thanks.

[challenger71498]

Please add a new commit which modifies pom.xml so we can trigger tests for the whole project. We can remove it while merging.

Thanks.

You mean the root pom.xml right? No problem, just edited it and pushed it.

[challenger71498]

Just in case, also attaching the test result of my test project mentioned above to be sure that the edit resolves the issue.

I made a custom build of hbase-client from branch-2.5, using hadoop 2, PR applied.

Image 1 shows that the custom build resolves the problem.
Image 2 shows the current behavior of 2.6.2 in contrast, which fails.

[Apache9]

Seems OK, please revert the commit which modifies pom.xml, I will merge it later.

Thanks @challenger71498

[challenger71498]

@Apache9 Will this PR be backported to 2.x? Since the master branch is for 4.0 (which hasn’t been released yet), and I am currently using version 2.6.2, it would be great if this PR could be backported.

[Apache9]

@challenger71498 Could you please reply on the jira so I can assign the jira to you?

I've cherry-picked the patch to all active branches.

Thanks for contributing!

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	1m 15s		Maven dependency ordering for branch
+1 💚	mvninstall	5m 17s		master passed
+1 💚	compile	10m 46s		master passed
+1 💚	checkstyle	1m 21s		master passed
+1 💚	spotbugs	11m 7s		master passed
+1 💚	spotless	1m 20s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 16s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 39s		the patch passed
+1 💚	compile	11m 39s		the patch passed
+1 💚	javac	11m 39s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 4s		the patch passed
+1 💚	xmllint	0m 1s		No new issues.
+1 💚	spotbugs	11m 18s		the patch passed
+1 💚	hadoopcheck	18m 15s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 59s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 23s		The patch does not generate ASF License warnings.
		90m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6918/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6918
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless xmllint
uname	Linux 4831baf87c70 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 5dfa9b6
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	192 (vs. ulimit of 30000)
modules	C: hbase-client . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6918/4/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3 xmllint=20913
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[challenger71498]

Sure, I just requested a Jira account by self-service portal. I will leave a comment after getting one :)

[challenger71498]

@Apache9 Just added the comment at the issue here.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 59s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	1m 14s		Maven dependency ordering for branch
+1 💚	mvninstall	5m 16s		master passed
+1 💚	compile	3m 3s		master passed
+1 💚	javadoc	3m 24s		master passed
+1 💚	shadedjars	7m 47s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 16s		Maven dependency ordering for patch
+1 💚	mvninstall	4m 6s		the patch passed
+1 💚	compile	3m 27s		the patch passed
+1 💚	javac	3m 27s		the patch passed
+1 💚	javadoc	2m 50s		the patch passed
+1 💚	shadedjars	7m 48s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	298m 19s	/patch-unit-root.txt	root in the patch failed.
		345m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6918/4/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6918
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux bfc5df56c70a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 5dfa9b6
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6918/4/testReport/
Max. process+thread count	8290 (vs. ulimit of 30000)
modules	C: hbase-client . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6918/4/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.