HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat by ndimiduk · Pull Request #1413 · apache/hbase

ndimiduk · 2020-04-01T21:25:53Z

sort lines and cleanup whitespace in hbase-assembly/hadoop-two-compat.xml

Simply cleanup the whitespace and readability of these include, exclude lists before making any subsequent changes. To be squished into the subsequent commit on merge.

HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat

After HBASE-23986, running on JDK8/Hadoop2 from the binary tarball
results in

Exception in thread "main" java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3062)
Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 2 more

The missing class is from com.fasterxml.woodstox:woodstox-core. This
and org.codehaus.woodstox:stax2-api are explicit dependencies of
org.apache.hadoop:hadoop-common:jar:2.10.0. Thus, include them in
the default jars provided under lib.

saintstack

+1 if it works.

Apache-HBase · 2020-04-01T21:32:15Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
		_ Patch Compile Tests _
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		2m 1s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1413
Optional Tests	dupname asflicense xml
uname	Linux 3931444a8385 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `040a0a7`
Max. process+thread count	51 (vs. ulimit of 10000)
modules	C: hbase-assembly U: hbase-assembly
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/console
versions	git=2.17.1 maven=2018-06-17T18:33:14Z)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2020-04-01T21:36:02Z

For reviewer reference (@jatsakthi in particular), the transitive dependency hull of org.apache.hadoop:hadoop-common:jar:2.10.0:compile:

[INFO] +- org.apache.hadoop:hadoop-common:jar:2.10.0:compile
[INFO] | +- asm:asm:jar:3.1:compile
[INFO] | +- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:compile
[INFO] | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
[INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] | +- com.google.guava:guava:jar:11.0.2:compile
[INFO] | +- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile
[INFO] | +- com.jcraft:jsch:jar:0.1.54:compile
[INFO] | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
[INFO] | +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] | +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO] | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] | +- commons-digester:commons-digester:jar:1.8:compile
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- commons-logging:commons-logging:jar:1.2:compile
[INFO] | +- commons-net:commons-net:jar:3.1:compile
[INFO] | +- javax.xml.bind:jaxb-api:jar:2.2.12:compile
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.9.0:compile
[INFO] | +- org.apache.avro:avro:jar:1.7.7:compile
[INFO] | +- org.apache.commons:commons-compress:jar:1.19:compile
[INFO] | +- org.apache.commons:commons-math3:jar:3.6.1:compile
[INFO] | +- org.apache.curator:curator-client:jar:4.2.0:compile
[INFO] | +- org.apache.curator:curator-framework:jar:4.2.0:compile
[INFO] | +- org.apache.curator:curator-recipes:jar:4.2.0:compile
[INFO] | +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] | +- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] | +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] | +- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.10.0:compile
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.10.0:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.3:compile
[INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.13:compile
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.14:compile
[INFO] | +- org.codehaus.jettison:jettison:jar:1.3.8:compile
[INFO] | +- org.codehaus.woodstox:stax2-api:jar:3.1.4:compile
[INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:compile
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.30:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.0.5:compile
[INFO] | +- xmlenc:xmlenc:jar:0.52:compile

And of com.sun.xml.ws:jaxws-ri:pom:2.3.2:compile

[INFO] +- com.sun.xml.ws:jaxws-ri:pom:2.3.2:compile
[INFO] | +- com.sun.istack:istack-commons-runtime:jar:3.0.8:compile
[INFO] | +- com.sun.xml.bind:jaxb-jxc:jar:2.3.2:compile
[INFO] | +- com.sun.xml.bind:jaxb-xjc:jar:2.3.2:compile
[INFO] | +- com.sun.xml.fastinfoset:FastInfoset:jar:1.2.16:compile
[INFO] | +- com.sun.xml.messaging.saaj:saaj-impl:jar:1.5.1:compile
[INFO] | +- com.sun.xml.stream.buffer:streambuffer:jar:1.5.7:compile
[INFO] | +- com.sun.xml.ws:jaxws-eclipselink-plugin:jar:2.3.2:compile
[INFO] | +- com.sun.xml.ws:jaxws-rt:jar:2.3.2:compile
[INFO] | +- com.sun.xml.ws:jaxws-tools:jar:2.3.2:compile
[INFO] | +- com.sun.xml.ws:policy:jar:2.7.6:compile
[INFO] | +- com.sun.xml.ws:release-documentation:zip:docbook:2.3.2:compile
[INFO] | +- com.sun.xml.ws:samples:zip:2.3.2:compile
[INFO] | +- com.sun.xml.ws:sdo-eclipselink-plugin:jar:2.3.2:compile
[INFO] | +- jakarta.activation:jakarta.activation-api:jar:1.2.1:compile
[INFO] | +- jakarta.annotation:jakarta.annotation-api:jar:1.3.4:compile
[INFO] | +- jakarta.jws:jakarta.jws-api:jar:1.1.1:compile
[INFO] | +- jakarta.mail:jakarta.mail-api:jar:1.6.3:compile
[INFO] | +- jakarta.persistence:jakarta.persistence-api:jar:2.2.2:compile
[INFO] | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile
[INFO] | +- jakarta.xml.soap:jakarta.xml.soap-api:jar:1.4.1:compile
[INFO] | +- jakarta.xml.ws:jakarta.xml.ws-api:jar:2.3.2:compile
[INFO] | +- org.eclipse.persistence:commonj.sdo:jar:2.1.1:compile
[INFO] | +- org.eclipse.persistence:org.eclipse.persistence.asm:jar:2.7.4:compile
[INFO] | +- org.eclipse.persistence:org.eclipse.persistence.core:jar:2.7.4:compile
[INFO] | +- org.eclipse.persistence:org.eclipse.persistence.moxy:jar:2.7.4:compile
[INFO] | +- org.eclipse.persistence:org.eclipse.persistence.sdo:jar:2.7.4:compile
[INFO] | +- org.glassfish.external:management-api:jar:3.2.1:compile
[INFO] | +- org.glassfish.gmbal:gmbal:jar:4.0.0:compile
[INFO] | +- org.glassfish.ha:ha-api:jar:3.1.12:compile
[INFO] | +- org.glassfish.jaxb:jaxb-runtime:jar:2.3.2:compile
[INFO] | +- org.glassfish.jaxb:txw2:jar:2.3.2:compile
[INFO] | +- org.glassfish.pfl:pfl-asm:jar:4.0.1:compile
[INFO] | +- org.glassfish.pfl:pfl-basic-tools:jar:4.0.1:compile
[INFO] | +- org.glassfish.pfl:pfl-basic:jar:4.0.1:compile
[INFO] | +- org.glassfish.pfl:pfl-dynamic:jar:4.0.1:compile
[INFO] | +- org.glassfish.pfl:pfl-tf-tools:jar:4.0.1:compile
[INFO] | +- org.glassfish.pfl:pfl-tf:jar:4.0.1:compile
[INFO] | +- org.jvnet.mimepull:mimepull:jar:1.9.11:compile
[INFO] | +- org.jvnet.staxex:stax-ex:jar:1.8.1:compile

Apache-HBase · 2020-04-01T21:50:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	6m 31s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	6m 2s	master passed
+1 💚	javadoc	0m 14s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 35s	the patch passed
+1 💚	javadoc	0m 13s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 29s	hbase-assembly in the patch passed.
		20m 7s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#1413
Optional Tests	javac javadoc unit
uname	Linux f3a5d256449c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `040a0a7`
Default Java	1.8.0_232
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/testReport/
Max. process+thread count	62 (vs. ulimit of 10000)
modules	C: hbase-assembly U: hbase-assembly
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/console
versions	git=2.17.1 maven=2018-06-17T18:33:14Z)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-04-01T21:57:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	5m 20s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	9m 43s	master passed
+1 💚	javadoc	0m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	9m 33s	the patch passed
+1 💚	javadoc	0m 20s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 58s	hbase-assembly in the patch passed.
		27m 25s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#1413
Optional Tests	javac javadoc unit
uname	Linux 94d122c6a2c5 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `040a0a7`
Default Java	2020-01-14
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/testReport/
Max. process+thread count	71 (vs. ulimit of 10000)
modules	C: hbase-assembly U: hbase-assembly
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/console
versions	git=2.17.1 maven=2018-06-17T18:33:14Z)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2020-04-02T16:40:42Z

Yeah seems to work for me locally. All this JDK11 complication can be purged from the hadoop-two profiles since that combination is known to not work at runtime.

…o-compat After HBASE-23986, running on JDK8/Hadoop2 from the binary tarball results in ``` Exception in thread "main" java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3062) Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 2 more ``` The missing class is from `com.fasterxml.woodstox:woodstox-core`. This and `org.codehaus.woodstox:stax2-api` are explicit dependencies of `org.apache.hadoop:hadoop-common:jar:2.10.0`. Thus, include them in the default jars provided under `lib`. Signed-off-by: stack <stack@apache.org>

ndimiduk requested review from busbey, saintstack, the-sakthi and wchevreuil April 1, 2020 21:25

saintstack approved these changes Apr 1, 2020
View reviewed changes

ndimiduk force-pushed the 24084-jdk8-hadoop2-hbase-assembly branch from a0ab2e1 to 704a3a5Compare April 2, 2020 16:50

ndimiduk merged commit 60959df into apache:masterApr 2, 2020

ndimiduk deleted the 24084-jdk8-hadoop2-hbase-assembly branch April 2, 2020 16:51

Navigation Menu

Uh oh!

HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat - #1413

HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat#1413
ndimiduk merged 1 commit into
apache:masterfrom
ndimiduk:24084-jdk8-hadoop2-hbase-assembly

ndimiduk commented Apr 1, 2020

Uh oh!

saintstack left a comment

Uh oh!

Apache-HBase commented Apr 1, 2020

Uh oh!

ndimiduk commented Apr 1, 2020

Uh oh!

Apache-HBase commented Apr 1, 2020

Uh oh!

Apache-HBase commented Apr 1, 2020

Uh oh!

ndimiduk commented Apr 2, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ndimiduk commented Apr 1, 2020

sort lines and cleanup whitespace in hbase-assembly/hadoop-two-compat.xml