[HBASE-24956] ConnectionManager#locateRegionInMeta waits for user region lock indefinitely. by shahrs87 · Pull Request #2415 · apache/hbase

shahrs87 · 2020-09-17T19:58:30Z

This is just the backport of branch-2 PR. #2322
There are 2 differences compared to branch-2.

LockTimeoutException already exists in branch-1 so reusing that.
In TestMetaCache#testUserRegionLockThrowsException, I have set HBASE_CLIENT_RETRIES_NUMBER value to 1 whereas in branch-2, it was set to 0.
In branch-2, they have attempts and retries notion differently. It is calculating numAttempts = numRetries + 1 but in branch-1, numAttempts = numTries.
@bharathv please review branch-1 PR. Thank you !

…ion lock indefinitely.

bharathv

will merge once the QA bot is green.

virajjasani

+1

virajjasani · 2020-09-18T07:54:58Z

re-trigged the build

Apache-HBase · 2020-09-18T09:58:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ branch-1 Compile Tests _
+0 🆗	mvndep	2m 23s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 2s	branch-1 passed
+1 💚	compile	1m 3s	branch-1 passed with JDK v1.8.0_262
+1 💚	compile	1m 11s	branch-1 passed with JDK v1.7.0_272
+1 💚	checkstyle	2m 14s	branch-1 passed
+1 💚	shadedjars	3m 3s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 0s	branch-1 passed with JDK v1.8.0_262
+1 💚	javadoc	1m 7s	branch-1 passed with JDK v1.7.0_272
+0 🆗	spotbugs	2m 41s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	4m 16s	branch-1 passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 18s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 55s	the patch passed
+1 💚	compile	1m 1s	the patch passed with JDK v1.8.0_262
+1 💚	javac	1m 1s	the patch passed
+1 💚	compile	1m 8s	the patch passed with JDK v1.7.0_272
+1 💚	javac	1m 8s	the patch passed
+1 💚	checkstyle	2m 6s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedjars	2m 55s	patch has no errors when building our shaded downstream artifacts.
+1 💚	hadoopcheck	4m 43s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
+1 💚	javadoc	0m 52s	the patch passed with JDK v1.8.0_262
+1 💚	javadoc	1m 5s	the patch passed with JDK v1.7.0_272
+1 💚	findbugs	4m 59s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 59s	hbase-client in the patch passed.
-1 ❌	unit	69m 29s	hbase-server in the patch failed.
+1 💚	asflicense	0m 46s	The patch does not generate ASF License warnings.
		123m 15s

Reason	Tests
Failed junit tests	hadoop.hbase.regionserver.wal.TestWALReplayCompressed
	hadoop.hbase.regionserver.TestHRegion

Subsystem	Report/Notes
Docker	Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2415/2/artifact/out/Dockerfile
GITHUB PR	#2415
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux 913980ebc570 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2415/out/precommit/personality/provided.sh
git revision	branch-1 / `f0acafc`
Default Java	1.7.0_272
Multi-JDK versions	/usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2415/2/artifact/out/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2415/2/testReport/
Max. process+thread count	4452 (vs. ulimit of 10000)
modules	C: hbase-client hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2415/2/console
versions	git=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

shahrs87 · 2020-09-18T16:08:49Z

hadoop.hbase.regionserver.wal.TestWALReplayCompressed
hadoop.hbase.regionserver.TestHRegion

These 2 tests failed since it couldn't create new thread.
2020-09-18 09:46:47,445 ERROR [nioEventLoopGroup-6-1] logging.Slf4JLogger(181): Failed to submit a listener notification task. Event loop shut down?

java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at io.netty.util.concurrent.GlobalEventExecutor.startThread(GlobalEventExecutor.java:222)
at io.netty.util.concurrent.GlobalEventExecutor.execute(GlobalEventExecutor.java:211)
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428)
at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:95)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:748)

Running the tests locally and both of them succeeded.
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 127.843 s - in org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed
[INFO] Running org.apache.hadoop.hbase.regionserver.TestHRegion
[INFO] Tests run: 109, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.392 s - in org.apache.hadoop.hbase.regionserver.TestHRegion
[INFO] [INFO] Results:
[INFO] [INFO] Tests run: 121, Failures: 0, Errors: 0, Skipped: 0

Cc @bharathv

bharathv · 2020-09-18T16:15:52Z

Ya that looks shady. I kicked off another build, let's see if this is reproducible.

Apache-HBase · 2020-09-18T19:11:00Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ branch-1 Compile Tests _
+0 🆗	mvndep	2m 24s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 0s	branch-1 passed
+1 💚	compile	1m 0s	branch-1 passed with JDK v1.8.0_262
+1 💚	compile	1m 8s	branch-1 passed with JDK v1.7.0_272
+1 💚	checkstyle	2m 17s	branch-1 passed
+1 💚	shadedjars	3m 0s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 2s	branch-1 passed with JDK v1.8.0_262
+1 💚	javadoc	1m 6s	branch-1 passed with JDK v1.7.0_272
+0 🆗	spotbugs	2m 42s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	4m 18s	branch-1 passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 18s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 57s	the patch passed
+1 💚	compile	1m 2s	the patch passed with JDK v1.8.0_262
+1 💚	javac	1m 2s	the patch passed
+1 💚	compile	1m 9s	the patch passed with JDK v1.7.0_272
+1 💚	javac	1m 9s	the patch passed
+1 💚	checkstyle	2m 7s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedjars	2m 46s	patch has no errors when building our shaded downstream artifacts.
+1 💚	hadoopcheck	4m 38s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2.
+1 💚	javadoc	0m 54s	the patch passed with JDK v1.8.0_262
+1 💚	javadoc	1m 7s	the patch passed with JDK v1.7.0_272
+1 💚	findbugs	4m 19s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 44s	hbase-client in the patch passed.
+1 💚	unit	132m 24s	hbase-server in the patch passed.
+1 💚	asflicense	0m 59s	The patch does not generate ASF License warnings.
		185m 21s

Subsystem	Report/Notes
Docker	Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2415/3/artifact/out/Dockerfile
GITHUB PR	#2415
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname	Linux 13909b755aab 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2415/out/precommit/personality/provided.sh
git revision	branch-1 / `f0acafc`
Default Java	1.7.0_272
Multi-JDK versions	/usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2415/3/testReport/
Max. process+thread count	4585 (vs. ulimit of 10000)
modules	C: hbase-client hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2415/3/console
versions	git=1.9.1 maven=3.0.5 findbugs=3.0.1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

[HBASE-24956] ConnectionManager#locateRegionInMeta waits for user reg…
c73b8dd
…ion lock indefinitely.

bharathv approved these changes Sep 17, 2020
View reviewed changes

virajjasani approved these changes Sep 18, 2020
View reviewed changes

bharathv merged commit e1fc3c4 into apache:branch-1Sep 18, 2020

Navigation Menu

Uh oh!

[HBASE-24956] ConnectionManager#locateRegionInMeta waits for user region lock indefinitely. - #2415

[HBASE-24956] ConnectionManager#locateRegionInMeta waits for user region lock indefinitely.#2415
bharathv merged 1 commit into
apache:branch-1from
shahrs87:HBASE-24956-branch-1

shahrs87 commented Sep 17, 2020

Uh oh!

bharathv left a comment

Uh oh!

virajjasani left a comment

Uh oh!

virajjasani commented Sep 18, 2020

Uh oh!

Apache-HBase commented Sep 18, 2020

Uh oh!

shahrs87 commented Sep 18, 2020

Uh oh!

bharathv commented Sep 18, 2020

Uh oh!

Apache-HBase commented Sep 18, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

shahrs87 commented Sep 17, 2020

Uh oh!

bharathv left a comment

Choose a reason for hiding this comment

Uh oh!

virajjasani left a comment

Choose a reason for hiding this comment

Uh oh!

virajjasani commented Sep 18, 2020

Uh oh!

Apache-HBase commented Sep 18, 2020

Uh oh!

shahrs87 commented Sep 18, 2020

Uh oh!

bharathv commented Sep 18, 2020

Uh oh!

Apache-HBase commented Sep 18, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants