HBASE-26907 Update Hadoop3 versions for JEP 223 compliance by ndimiduk · Pull Request #4305 · apache/hbase

ndimiduk · 2022-03-31T16:48:01Z

No description provided.

ndimiduk · 2022-03-31T16:51:36Z

See investigation and discussion on the JIRA.

Maybe we want to drop 2.3 from the matrix in the book?

Our personality makes no distinction in Hadoop3 versions for JDK8 vs. JDK11+. The latter requires 3.2+. We probably should fix that.

ndimiduk · 2022-03-31T18:32:22Z

Well that's interesting. From hadoop-3.3.2:

[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : [INFO] -------------------------------------------------------------
[ERROR] /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4305/yetus-general-check/src/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java:[832,29] package org.apache.htrace.core does not exist
[INFO] 1 error
[INFO] -------------------------------------------------------------

busbey · 2022-03-31T18:45:45Z

we have our hbase-thirdparty no-op stand-in for htrace. We can add a profile for including that when building against some Hadoops?

Apache9

Just a simple nit.

And on the compatibile matrix, IIRC we have a consensus that we will only have ref guide on master branch so it should contain all the release lines instead of only non-EOL release lines. So let's still keep the 2.3.x release line. Let me find the thread on mailing list and try to move forward on this topic.

joshelser

LGTM (agree with Duo's suggestion to bump to the latest 3.3.x)

ndimiduk · 2022-04-13T15:30:52Z

And on the compatibile matrix, IIRC we have a consensus that we will only have ref guide on master branch so it should contain all the release lines instead of only non-EOL release lines. So let's still keep the 2.3.x release line. Let me find the thread on mailing list and try to move forward on this topic.

@Apache9 okay ; I'll take your word for it, I can restore the 2.3 entries. Obviously we don't have all released mentioned, so at what point do we drop them off?

ndimiduk · 2022-04-13T20:54:36Z

I opened HBASE-26954 for the Hadoop 3.3.2 compilation issue.

ndimiduk · 2022-04-13T21:07:47Z

I've bumped our Hadoop 3.2 dependency to 3.2.3, per @Apache9 's suggestion. I've retained the HBase 2.3 rows in the compatibility matrix. I've stepped back our promises about Hadoop 3.3.2, given HBASE-26954.

ndimiduk · 2022-04-13T21:08:15Z

Let me see which branches are impacted by HBASE-26954.

ndimiduk · 2022-04-13T21:25:25Z

HBASE-26954 seems to only affect master branch.

Apache9 · 2022-04-14T02:18:29Z

And on the compatibile matrix, IIRC we have a consensus that we will only have ref guide on master branch so it should contain all the release lines instead of only non-EOL release lines. So let's still keep the 2.3.x release line. Let me find the thread on mailing list and try to move forward on this topic.
@Apache9 okay ; I'll take your word for it, I can restore the 2.3 entries. Obviously we don't have all released mentioned, so at what point do we drop them off?

This has not been well discussed yet, I mean when to drop the EOL versions. For me, I prefer we split the table to two pieces, for EOL versions, after some time(half a year or a whole year?), we move it to the 'archived' pieces, which means we will not update them any more.

Apache9 · 2022-05-22T02:50:52Z

Any updates here?

ndimiduk · 2022-05-22T09:42:24Z

Progress is reported over on the linked ticket where I describe the test failure. There's a ThreadGroup object that goes out of scope during DataNode operation on these newer Hadoop versions. I think that the explicitly managed ThreadGoup serves no purpose, but I'm still looking for a commit that changed the behavior.

I believe is indicates a real and serious bug in the DataNode, which will cause it to become unresponsive after an idle period.

ndimiduk · 2022-08-15T12:49:47Z

I think @saintstack fixed the issue i was seeing in HDFS-16586 / apache/hadoop#4338 . Let me see how widely that patch has propagated.

Apache9 · 2022-08-15T12:58:43Z

Ah, I've replied on the jira, in HBASE-27148 we have already moved up the hadoop 3.2.3.

I also noticed the problem described in HDFS-16586 but I'm afraid we can not only support the hadoop versions which have this patch since it is only a test issue. I added a piece of workaround code in HBTU so we can still use the versions which do not have HDFS-16586 in.

hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtil.java

Line 579 in 9215066

privatestaticfinalclassFsDatasetAsyncDiskServiceFixerextendsThread {

So I think we could close this PR now?

ndimiduk · 2022-08-15T13:07:04Z

Rebased and squashed current patched onto latest master.

ndimiduk · 2022-08-15T13:09:59Z

Ah, I've replied on the jira, in HBASE-27148 we have already moved up the hadoop 3.2.3.

@Apache9 Ah okay, I missed HBASE-27148.

ndimiduk · 2022-08-15T13:12:03Z

It looks like we still have hadoop-3.1.2 in the quick-check list for branch-2.4, I guess we can bump that version. Also, it appears that I have added a column regarding branch-2.5 in this patch.

Let me push an updated based the knowledge of HBASE-27148.

Apache9 · 2022-08-15T13:17:26Z

It looks like we still have hadoop-3.1.2 in the quick-check list for branch-2.4, I guess we can bump that version. Also, it appears that I have added a column regarding branch-2.5 in this patch.
Let me push an updated based the knowledge of HBASE-27148.

Sounds good.

And please see #4692 , where I plan to change the hadoop check versions for our active branches, which also implies what hadoop versions we still wantto support. I think for 2.4.x, we should not remove the support of a hadoop version, but for 2.5+, we can drop the EOL versions and also the versions which have critical CVEs.

- drop 3.1.x - ensure 3.2 >= 3.2.3 - add 3.3.3, 3.3.4

Apache-HBase · 2022-08-15T13:21:56Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/console in case of problems.

Apache-HBase · 2022-08-15T13:22:02Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/console in case of problems.

Apache-HBase · 2022-08-15T13:22:12Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/console in case of problems.

Apache-HBase · 2022-08-15T13:22:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 42s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		1m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4305
Optional Tests
uname	Linux de954c16e7c7 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9215066`
Max. process+thread count	29 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-08-15T13:23:14Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 57s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		2m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4305
Optional Tests
uname	Linux 312e578bb58a 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9215066`
Max. process+thread count	37 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2022-08-15T13:25:41Z

Hmm okay. @Apache9 how about I close HBASE-26907 this PR and let you resolve them on HBASE-27294. Do you might grabbing the update to the book once that ticket is resolved? Thanks.

Apache9 · 2022-08-15T13:37:24Z

Hmm okay. @Apache9 how about I close HBASE-26907 this PR and let you resolve them on HBASE-27294. Do you might grabbing the update to the book once that ticket is resolved? Thanks.

For the book update let's open a separated issue? As part of the branch-2.5 release process.

Apache-HBase · 2022-08-15T13:40:00Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 39s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	shelldocs	0m 0s	Shelldocs was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 46s	master passed
+0 🆗	refguide	3m 22s	branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	spotless	1m 6s	branch has no errors when running spotless:check.
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 21s	the patch passed
+1 💚	shellcheck	0m 1s	There were no new shellcheck issues.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+0 🆗	refguide	2m 46s	patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	spotless	0m 50s	patch has no errors when running spotless:check.
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		18m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4305
Optional Tests	dupname asflicense spotless shellcheck shelldocs refguide
uname	Linux 695c599fd0f6 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9215066`
refguide	https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4305/4/yetus-general-check/output/branch-site/book.html
refguide	https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4305/4/yetus-general-check/output/patch-site/book.html
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/console
versions	git=2.17.1 maven=3.6.3 shellcheck=0.4.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2022-08-15T13:40:36Z

Ah, maybe I misunderstood your comment. You also mean we open a new issue to land the book update? That's fine by me. I could help implementing the PR or reviewing the PR, no problem. Thanks.

Apache-HBase · 2022-08-15T13:45:08Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/console in case of problems.

Apache-HBase · 2022-08-15T13:45:12Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/console in case of problems.

Apache-HBase · 2022-08-15T13:45:29Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/console in case of problems.

Apache-HBase · 2022-08-15T13:46:01Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 41s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		1m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4305
Optional Tests
uname	Linux 24c3d085eac6 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9215066`
Max. process+thread count	29 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-08-15T13:46:05Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 42s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		1m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4305
Optional Tests
uname	Linux 538fb0037040 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9215066`
Max. process+thread count	39 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-08-15T14:02:37Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 29s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+0 🆗	shelldocs	0m 1s	Shelldocs was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 17s	master passed
+0 🆗	refguide	3m 6s	branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	spotless	1m 14s	branch has no errors when running spotless:check.
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 12s	the patch passed
+1 💚	shellcheck	0m 1s	There were no new shellcheck issues.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+0 🆗	refguide	2m 54s	patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	spotless	1m 1s	patch has no errors when running spotless:check.
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		17m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4305
Optional Tests	dupname asflicense spotless shellcheck shelldocs refguide
uname	Linux 51327528fcf9 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9215066`
refguide	https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4305/5/yetus-general-check/output/branch-site/book.html
refguide	https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4305/5/yetus-general-check/output/patch-site/book.html
Max. process+thread count	70 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/console
versions	git=2.17.1 maven=3.6.3 shellcheck=0.4.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2022-08-16T12:26:06Z

Superceded HBASE-27148, HBASE-27294.

ndimiduk requested review from Apache9, apurtell, busbey and joshelser March 31, 2022 16:48

This comment was marked as outdated.
Sign in to view

apurtell approved these changes Apr 2, 2022
View reviewed changes

Apache9 reviewed Apr 4, 2022
View reviewed changes

Comment threaddev-support/hbase-personality.sh Outdated

joshelser approved these changes Apr 4, 2022
View reviewed changes

Apache9 mentioned this pull request Apr 12, 2022
Bump hadoop-common from 3.1.2 to 3.1.4 #4330
Closed

ndimiduk force-pushed the 26907-hadoop3-JEP223-compliance branch from 8a71d7c to 5f8345dCompare April 13, 2022 21:06

This comment was marked as outdated.
Sign in to view

Apache9 approved these changes Apr 14, 2022
View reviewed changes

This comment was marked as outdated.
Sign in to view

ndimiduk mentioned this pull request Apr 28, 2022
YETUS-1060. github status should use htmlreport apache/yetus#261
Merged

HBASE-26907 Update Hadoop3 versions for JEP 223 compliance
a215a0a

ndimiduk force-pushed the 26907-hadoop3-JEP223-compliance branch from 0b9fb7d to a215a0aCompare August 15, 2022 13:06

update after HBASE-27148
16cca4b
- drop 3.1.x - ensure 3.2 >= 3.2.3 - add 3.3.3, 3.3.4

ndimiduk closed this Aug 16, 2022

ndimiduk deleted the 26907-hadoop3-JEP223-compliance branch August 16, 2022 12:26

Uh oh!

Conversation

ndimiduk commented Mar 31, 2022

Uh oh!

ndimiduk commented Mar 31, 2022

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

ndimiduk commented Mar 31, 2022

Uh oh!

busbey commented Mar 31, 2022

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Apache9 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joshelser left a comment

Choose a reason for hiding this comment

Uh oh!

ndimiduk commented Apr 13, 2022

Uh oh!

ndimiduk commented Apr 13, 2022

Uh oh!

ndimiduk commented Apr 13, 2022

Uh oh!

ndimiduk commented Apr 13, 2022

Uh oh!

ndimiduk commented Apr 13, 2022

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Apache9 commented Apr 14, 2022

Uh oh!

This comment was marked as outdated.

Apache9 commented May 22, 2022

Uh oh!

ndimiduk commented May 22, 2022

Uh oh!

ndimiduk commented Aug 15, 2022

Uh oh!

Apache9 commented Aug 15, 2022

Uh oh!

ndimiduk commented Aug 15, 2022

Uh oh!

ndimiduk commented Aug 15, 2022

Uh oh!

ndimiduk commented Aug 15, 2022

Uh oh!

Apache9 commented Aug 15, 2022

Uh oh!

Apache-HBase commented Aug 15, 2022

Uh oh!

Apache-HBase commented Aug 15, 2022

Uh oh!

Apache-HBase commented Aug 15, 2022

Uh oh!

Apache-HBase commented Aug 15, 2022

Uh oh!

Apache-HBase commented Aug 15, 2022

Uh oh!

ndimiduk commented Aug 15, 2022

Uh oh!

Apache9 commented Aug 15, 2022

Uh oh!

Apache-HBase commented Aug 15, 2022

Uh oh!

Apache9 commented Aug 15, 2022

Uh oh!

Apache-HBase commented Aug 15, 2022

Uh oh!

Apache-HBase commented Aug 15, 2022

Uh oh!

Apache-HBase commented Aug 15, 2022

Uh oh!