HDDS-1451 : SCMBlockManager findPipeline and createPipeline are not lock protected. by avijayanhwx · Pull Request #799 · apache/hadoop

avijayanhwx · 2019-05-07T20:40:01Z

The getPipelines() and createPipeline() already seem to have a lock in their implementation. However, the problem described here involves a race condition between the call to getPipelines and createPipelines in BlockManagerImpl#allocateBlock. The fix is to add another getPipelines check after a failed createPipeline call to get any newly created pipelines.

…ock protected.

hadoop-yetus · 2019-05-07T22:21:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	42	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	415	trunk passed
+1	compile	197	trunk passed
+1	checkstyle	51	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	883	branch has no errors when building and testing our client artifacts.
+1	javadoc	123	trunk passed
0	spotbugs	242	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	425	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	400	the patch passed
+1	compile	202	the patch passed
+1	javac	202	the patch passed
+1	checkstyle	55	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	798	patch has no errors when building and testing our client artifacts.
+1	javadoc	135	the patch passed
+1	findbugs	472	the patch passed
		_ Other Tests _
-1	unit	173	hadoop-hdds in the patch failed.
-1	unit	1435	hadoop-ozone in the patch failed.
-1	asflicense	41	The patch generated 1 ASF License warnings.
		5991

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestWatchForCommit
	hadoop.ozone.om.TestOzoneManagerHA

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-799/1/artifact/out/Dockerfile
GITHUB PR	#799
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 79fd163f71d1 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `7f0e2c6`
Default Java	1.8.0_191
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-799/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-799/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-799/1/testReport/
asflicense	https://builds.apache.org/job/hadoop-multibranch/job/PR-799/1/artifact/out/patch-asflicense-problems.txt
Max. process+thread count	5076 (vs. ulimit of 5500)
modules	C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-799/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

avijayanhwx · 2019-05-08T18:22:16Z

/label ozone

avijayanhwx · 2019-05-15T01:53:52Z

/retest

mukul1987

The patch looks good to me. Please change the log lines to add the type and factor.

hadoop-yetus · 2019-05-17T04:16:05Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	45	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	559	trunk passed
+1	compile	252	trunk passed
+1	checkstyle	67	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	1065	branch has no errors when building and testing our client artifacts.
+1	javadoc	159	trunk passed
0	spotbugs	294	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	529	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	530	the patch passed
+1	compile	253	the patch passed
+1	javac	253	the patch passed
+1	checkstyle	73	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	794	patch has no errors when building and testing our client artifacts.
+1	javadoc	146	the patch passed
+1	findbugs	552	the patch passed
		_ Other Tests _
-1	unit	204	hadoop-hdds in the patch failed.
-1	unit	1915	hadoop-ozone in the patch failed.
+1	asflicense	48	The patch does not generate ASF License warnings.
		7326

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestBlockOutputStream
	hadoop.ozone.TestMiniChaosOzoneCluster
	hadoop.ozone.om.TestOzoneManagerHA
	hadoop.ozone.om.TestOzoneManager
	hadoop.ozone.client.rpc.TestContainerStateMachine
	hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
	hadoop.ozone.client.rpc.TestOzoneRpcClient
	hadoop.ozone.client.rpc.TestBCSID

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-799/2/artifact/out/Dockerfile
GITHUB PR	#799
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux ecd9e7269767 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `c183bd8`
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-799/2/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-799/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-799/2/testReport/
Max. process+thread count	3676 (vs. ulimit of 5500)
modules	C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-799/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

anuengineer · 2019-05-18T09:54:31Z

@avijayanhwx the patch looks good to me. Can you please confirm that the test failures are not due to this patch. Thanks in Advance.

avijayanhwx · 2019-05-20T17:13:19Z

@avijayanhwx the patch looks good to me. Can you please confirm that the test failures are not due to this patch. Thanks in Advance.

@anuengineer The failures seem unrelated. I have rebased with latest trunk. We can see how this run goes and then commit it.

anuengineer · 2019-05-20T20:02:45Z

+1, LGTM. Thanks.

As per subject, table descriptors moved are - LocalTableDescriptor - RemoteTableDescriptor - HybridTableDescriptor - GuavaCacheTableDescriptor - CachingTableDescriptor Author: Wei Song <wsong@linkedin.com> Reviewers: Prateek Maheshwari <pmaheshwari@linkedin.com> Closesapache#799 from weisong44/SAMZA-1981

HDDS-1451 : SCMBlockManager findPipeline and createPipeline are not l…
7f5afef
…ock protected.

elek added the ozone label May 8, 2019

mukul1987 requested changes May 17, 2019
View reviewed changes

Comment threadhadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/BlockManagerImpl.java Outdated
Comment threadhadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/BlockManagerImpl.java Outdated

Aravindan Vijayan added 2 commits May 16, 2019 19:07

Merge remote-tracking branch 'upstream/trunk' into HDDS-1451-trunk
f08d3e6

HDDS-1451 : Address review comments.
f1b8a32

Merge remote-tracking branch 'upstream/trunk' into HDDS-1451-trunk
3ec62af

anuengineer merged commit f633002 into apache:trunkMay 20, 2019

Navigation Menu

Uh oh!

HDDS-1451 : SCMBlockManager findPipeline and createPipeline are not lock protected. - #799

HDDS-1451 : SCMBlockManager findPipeline and createPipeline are not lock protected.#799
anuengineer merged 4 commits into
apache:trunkfrom
avijayanhwx:HDDS-1451-trunk

avijayanhwx commented May 7, 2019

Uh oh!

hadoop-yetus commented May 7, 2019

Uh oh!

avijayanhwx commented May 8, 2019

Uh oh!

avijayanhwx commented May 15, 2019

Uh oh!

mukul1987 left a comment

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented May 17, 2019

Uh oh!

anuengineer commented May 18, 2019

Uh oh!

avijayanhwx commented May 20, 2019

Uh oh!

anuengineer commented May 20, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

avijayanhwx commented May 7, 2019

Uh oh!

hadoop-yetus commented May 7, 2019

Uh oh!

avijayanhwx commented May 8, 2019

Uh oh!

avijayanhwx commented May 15, 2019

Uh oh!

mukul1987 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented May 17, 2019

Uh oh!

anuengineer commented May 18, 2019

Uh oh!

avijayanhwx commented May 20, 2019

Uh oh!

anuengineer commented May 20, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants