HBASE-26546: hbase-shaded-client missing required thirdparty classes under hadoop 3.3.1

[bbeaudreault]

I expected there would be more to this, but this ended up working locally without needing to modify our invariants checks. I'm continuing to look into why, but wanted to get the full build started.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 12s	master passed
+1 💚	compile	0m 29s	master passed
+1 💚	shadedjars	8m 23s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 18s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 54s	the patch passed
+1 💚	compile	0m 30s	the patch passed
+1 💚	javac	0m 30s	the patch passed
+1 💚	shadedjars	8m 24s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 16s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 7s	hbase-shaded in the patch passed.
		29m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3926/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3926
JIRA Issue	HBASE-26546
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c6877e0ec21f 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / ca3ba49
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3926/1/testReport/
Max. process+thread count	482 (vs. ulimit of 30000)
modules	C: hbase-shaded U: hbase-shaded
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3926/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 0s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 50s	master passed
+1 💚	compile	0m 33s	master passed
+1 💚	shadedjars	8m 18s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 19s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 26s	the patch passed
+1 💚	compile	0m 32s	the patch passed
+1 💚	javac	0m 32s	the patch passed
+1 💚	shadedjars	8m 16s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 17s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 2s	hbase-shaded in the patch passed.
		30m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3926/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3926
JIRA Issue	HBASE-26546
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ef39c310b5b8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / ca3ba49
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3926/1/testReport/
Max. process+thread count	476 (vs. ulimit of 30000)
modules	C: hbase-shaded U: hbase-shaded
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3926/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 2s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 31s	master passed
+1 💚	compile	0m 34s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 14s	the patch passed
+1 💚	compile	0m 32s	the patch passed
+1 💚	javac	0m 32s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	hadoopcheck	21m 39s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		41m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3926/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3926
JIRA Issue	HBASE-26546
Optional Tests	dupname asflicense javac hadoopcheck xml compile
uname	Linux f1b749729142 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / ca3ba49
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-shaded U: hbase-shaded
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3926/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[bbeaudreault]

I did some investigation and cannot figure out why this works without tweaking our allow list. In fact, I checked out the original state prior to #3184 and (after applying #3299, which adds a different necessary exclusion) it also succeeds there. For reference, I'm running the following locally to test (which confirms runs and succeeds invariant checks):

mvn clean verify -Phadoop-3.0 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.3.1 -Dcheckstyle.skip -DskipTests

I've also manually verified that the jars look correct:

# should be zero
$ jar -tf hbase-shaded/hbase-shaded-mapreduce/target/hbase-shaded-mapreduce-3.0.0-alpha-2-SNAPSHOT.jar | grep -c "org/apache/hadoop/thirdparty"
0
# should be zero
$ jar -tf hbase-shaded/hbase-shaded-client-byo-hadoop/target/hbase-shaded-client-byo-hadoop-3.0.0-alpha-2-SNAPSHOT.jar | grep -c "org/apache/hadoop/thirdparty"
0
# should be non-zero
$ jar -tf hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-3.0.0-alpha-2-SNAPSHOT.jar | grep -c "org/apache/hadoop/thirdparty"
3092

I also diffed the jar -tf content between master and this branch, and the two byo-hadoop jars were identical while the hbase-shaded-client only included the new thirdparty classes as expected.

Finally, I also pushed this to our internal fork and verified that my end-client which was previously failing is now succeeding.

It seems like this should be good to merge as-is. @saintstack since you originally ran into this failure, what do you think?

[saintstack]

I just tried this on branch-2:

mvn clean install -Phadoop-3.0 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.3.1 -Dcheckstyle.skip -DskipTests assembly:single -Prelease

... which is probably how I produced the fail originally but it works w/ this patch applied. I can't explain why it works now (context is long gone).

Thanks for digging in. Try it. +1.

[ndimiduk]

Is this something we can verify in our nightly build test suite? We already have a stage that goes out of its way to run the client against the pseudo-distributed cluster, maybe this could be extended to use the shaded client?

https://github.com/apache/hbase/blob/master/dev-support/Jenkinsfile#L752
https://github.com/apache/hbase/blob/master/dev-support/hbase_nightly_pseudo-distributed-test.sh

[ndimiduk]

I see. This failure might indeed be detected by our current test case, if but for the fact that we set HADOOP3_VERSION="3.1.1" instead of this newer version.

[bbeaudreault]

@ndimiduk is there something you'd like me to add to this PR? Sorry, I'm not super familiar with our test suite configurations. I could naively change that HADOOP3 version to 3.3.1, but not sure what the effect would be or if there is a more thorough approach to take?

[bbeaudreault]

I took a closer look at the files you referenced. I actually think we wouldn't have caught this, even if the hadoop version were correct. As far as I can tell, hbase_nightly_pseudo-distributed-test.sh is only testing hbase-shaded-mapreduce and hbase-shaded-client-byo-hadoop. I suppose we could add another step to that file which runs the same HBaseClientReadWriteExample.java but using hbase-shaded-client alone.

Would making that change actually show up in these PR checks or only as part of some nightly process? Should we also bump the HADOOP3_VERSION, or run against both 3.1.1/3.3.1?

[ndimiduk]

I think @busbey is most knowledgable about this specific check, so I'll let him weigh in. I would say that if we have a client-consumable artifact that is not currently covered by this check, we should add it. If the upstream Hadoop releases are materially different, we should extend the test coverage to run the full supported matrix, as described in https://hbase.apache.org/book.html#hadoop

The largest obstacles to this level of coverage are test resource capacity in general and time-to-completion specifically.

[ndimiduk]

For an example of the lengths we go to in order to assert basic compile-time compatibility with myriad Hadoop releases during pre-commit, see https://github.com/apache/hbase/blob/master/dev-support/hbase-personality.sh#L562-L634

[bbeaudreault]

Ok, depending on @busbey's input and the level of investment necessary here it might make sense to defer that work to another jira. This PR is fixing a demonstrable bug that is cause hadoop-3.3.1-based clients to not work with the hbase client. I'd be happy to add a quick test in the appropriate place with some guidance, but I'm not sure I want to get embroiled in a larger test coverage initiative at the moment.

[busbey]

I agree that we should not be excluding these classes. there are already checks that allow for hadoop classes to show up in just those artifacts where we allow hadoop, and AFAICT the hadoop third party stuff is properly located.

the failure that HBASE-25792 was in the hbase-shaded-mapreduce artifact, which isn't allowed to have any hadoop classes in the first place. So if there's no complaint after this change it likely just means that whatever was causing a leak of hadoop stuff into that module has since been fixed.