HBASE-26834 Adapt ConnectionRule for both sync and async connections (branch-2) by ndimiduk · Pull Request #4220 · apache/hbase

ndimiduk · 2022-03-14T15:24:07Z

This is needed primarily for branch-2 codelines where there's no easy bridge from AsyncConnection to Connection.

ndimiduk · 2022-03-14T16:10:07Z

The master forward-port is posted on #4221

ndimiduk · 2022-03-14T16:14:03Z

@busbey do you have any idea why this might be? Is there something missing from the hadoop.three profile in one of the submodules that is required for invoking tests on this module? The failing build is prefixed with this warning.

cd /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4220/yetus-jdk11-hadoop3-check/src
/opt/maven/bin/mvn --batch-mode -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4220/yetus-m2/hbase-branch-2-patch-1 -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true -Dspotbugs.skip=true
[INFO] Scanning for projects...
[WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase-build-configuration:pom:2.6.0-SNAPSHOT
[WARNING] 'profiles.profile[hadoop-3.0].dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-mapreduce-client-app:test-jar -> duplicate declaration of version ${hadoop-three.version} @ org.apache.hbase:hbase:2.6.0-SNAPSHOT, /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4220/yetus-jdk11-hadoop3-check/src/pom.xml, line 3527, column 22
[WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase:pom:2.6.0-SNAPSHOT
[WARNING] 'profiles.profile[hadoop-3.0].dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-mapreduce-client-app:test-jar -> duplicate declaration of version ${hadoop-three.version} @ line 3527, column 22
[WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]

ndimiduk · 2022-03-14T16:59:18Z

@busbey do you have any idea why this might be? Is there something missing from the hadoop.three profile in one of the submodules that is required for invoking tests on this module? The failing build is prefixed with this warning.

cd /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4220/yetus-jdk11-hadoop3-check/src
/opt/maven/bin/mvn --batch-mode -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4220/yetus-m2/hbase-branch-2-patch-1 -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true -Dspotbugs.skip=true
[INFO] Scanning for projects...
[WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase-build-configuration:pom:2.6.0-SNAPSHOT
[WARNING] 'profiles.profile[hadoop-3.0].dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-mapreduce-client-app:test-jar -> duplicate declaration of version ${hadoop-three.version} @ org.apache.hbase:hbase:2.6.0-SNAPSHOT, /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4220/yetus-jdk11-hadoop3-check/src/pom.xml, line 3527, column 22
[WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase:pom:2.6.0-SNAPSHOT
[WARNING] 'profiles.profile[hadoop-3.0].dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-mapreduce-client-app:test-jar -> duplicate declaration of version ${hadoop-three.version} @ line 3527, column 22
[WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]

Oh. That mvn invocation is missing the -Dhadoop.profile=3.0 that is required for building under JDK11.

busbey · 2022-03-15T16:59:53Z

Oh. That mvn invocation is missing the -Dhadoop.profile=3.0 that is required for building under JDK11.

that flag is supposed to get set over here:

https://github.com/apache/hbase/blob/master/dev-support/hbase-personality.sh#L170-L175

so either our branch pattern matching is broken, something else is overriding the setting, or this execution flow somehow skips personality_modules

ndimiduk · 2022-03-15T19:16:47Z

Okay, well. JDK11 aside, the failures in JDK8 build look like worker resource exhaustion and unrelated to this change. None of those test classes use this Rule, so...

ndimiduk · 2022-03-15T19:42:29Z

@busbey my earlier diagnosis was not entirely accurate -- this is not PR's fault. I reproduce the same behavior when I run a single test out of a single module (with JDK11, Hadoop3, branch-2). i.e.,

$ env JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home mvn clean install -Dhadoop.profile=3.0
$ env JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home mvn -pl hbase-server test -Dhadoop.profile=3.0 -Dtest=TestApiV1ClusterMetricsResource

Apache9 · 2022-03-17T00:09:26Z

The failed UT seems because I set java.io.tmpdir to target on the mvn command line...

We should have set it explicitly in surefire arg line, not sure why setting it on mvn command line would effect this...

Let me dig more.

Apache-HBase · 2022-03-17T15:52:06Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 41s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ branch-2 Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for branch
+1 💚	mvninstall	4m 23s	branch-2 passed
+1 💚	compile	4m 14s	branch-2 passed
+1 💚	checkstyle	1m 25s	branch-2 passed
+1 💚	spotbugs	2m 37s	branch-2 passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 20s	the patch passed
+1 💚	compile	5m 5s	the patch passed
+1 💚	javac	5m 5s	the patch passed
+1 💚	checkstyle	1m 37s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	11m 59s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚	spotbugs	3m 27s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 28s	The patch does not generate ASF License warnings.
		48m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4220/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4220
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux e226853a0d84 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `b5d5fe8`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	60 (vs. ulimit of 12500)
modules	C: hbase-server hbase-it U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4220/5/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-03-17T18:29:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 1s	Docker mode activated.
-0 ⚠️	yetus	0m 6s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ branch-2 Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 44s	branch-2 passed
+1 💚	compile	1m 1s	branch-2 passed
+1 💚	shadedjars	0m 23s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	branch-2 passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 14s	the patch passed
+1 💚	compile	0m 59s	the patch passed
+1 💚	javac	0m 59s	the patch passed
+1 💚	shadedjars	0m 22s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	the patch passed
		_ Other Tests _
-1 ❌	unit	193m 9s	hbase-server in the patch failed.
+1 💚	unit	0m 50s	hbase-it in the patch passed.
		206m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4220/5/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#4220
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 3dded531bdcf 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `b5d5fe8`
Default Java	AdoptOpenJDK-1.8.0_282-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4220/5/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4220/5/testReport/
Max. process+thread count	2511 (vs. ulimit of 12500)
modules	C: hbase-server hbase-it U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4220/5/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-03-17T18:46:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 39s	Docker mode activated.
-0 ⚠️	yetus	0m 5s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ branch-2 Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for branch
+1 💚	mvninstall	4m 26s	branch-2 passed
+1 💚	compile	1m 53s	branch-2 passed
+1 💚	shadedjars	0m 29s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 58s	branch-2 passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 32s	the patch passed
+1 💚	compile	1m 32s	the patch passed
+1 💚	javac	1m 32s	the patch passed
+1 💚	shadedjars	0m 36s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 0s	the patch passed
		_ Other Tests _
-1 ❌	unit	202m 31s	hbase-server in the patch failed.
+1 💚	unit	0m 56s	hbase-it in the patch passed.
		222m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4220/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4220
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a87f82c96741 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `b5d5fe8`
Default Java	AdoptOpenJDK-11.0.10+9
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4220/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4220/5/testReport/
Max. process+thread count	2960 (vs. ulimit of 12500)
modules	C: hbase-server hbase-it U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4220/5/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2022-03-21T11:07:07Z

Failure in TestReplicationSource is unrelated.

ndimiduk requested review from Apache9, apurtell, joshelser and taklwu March 14, 2022 15:24

This comment was marked as outdated.
Sign in to view

ndimiduk added 2 commits March 16, 2022 13:30

HBASE-26834 Adapt ConnectionRule for both sync and async connections
eda480b

PR Feedback
742cb4d

ndimiduk force-pushed the 26834-connectionrule-sync-async-branch-2 branch from 945ab47 to 742cb4dCompare March 16, 2022 12:32

This comment was marked as outdated.
Sign in to view

ndimiduk mentioned this pull request Mar 21, 2022
Forward-port "HBASE-26834 Adapt ConnectionRule for both sync and async connections" to master #4221
Merged

ndimiduk merged commit 321c35a into apache:branch-2Mar 21, 2022

ndimiduk deleted the 26834-connectionrule-sync-async-branch-2 branch March 21, 2022 11:42

Uh oh!

Conversation

ndimiduk commented Mar 14, 2022

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

ndimiduk commented Mar 14, 2022

Uh oh!

ndimiduk commented Mar 14, 2022

Uh oh!

ndimiduk commented Mar 14, 2022

Uh oh!

This comment was marked as outdated.

busbey commented Mar 15, 2022

Uh oh!

ndimiduk commented Mar 15, 2022

Uh oh!

ndimiduk commented Mar 15, 2022

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Apache9 commented Mar 17, 2022

Uh oh!

Apache-HBase commented Mar 17, 2022

Uh oh!

Apache-HBase commented Mar 17, 2022

Uh oh!

Apache-HBase commented Mar 17, 2022

Uh oh!

ndimiduk commented Mar 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants