HADOOP-16357 TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists by steveloughran · Pull Request #992 · apache/hadoop

steveloughran · 2019-06-19T21:44:28Z

This patch

changes the default for the staging committer to append, as we get for the classic FOF committer
adds a check for the dest path being a file not a dir
adds tests for this
Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present.

Testing: S3A Ireland w/ S3Guard. the Staging test failed, which makes me think that the settings weren't getting through

steveloughran · 2019-06-19T22:32:30Z

Test results (rebased patch): MR tests fail, but not ITestDirectoryCommitProtocol, which is complaining that the dest dir exists in job setup

org.apache.hadoop.fs.PathExistsException: `s3a://hwdev-steve-ireland-new/test/DELAY_LISTING_ME/ITestDirectoryCommitProtocol-testRecoveryAndCleanup': Setting job as Task committer attempt_200707120058_0001_m_000000_0: Destination path exists and committer conflict resolution mode is "fail"
at org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter.failDestinationExists(StagingCommitter.java:878)
at org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter.setupJob(DirectoryStagingCommitter.java:85)
at org.apache.hadoop.fs.s3a.commit.AbstractITCommitProtocol.testRecoveryAndCleanup(AbstractITCommitProtocol.java:587)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:745)

steveloughran · 2019-06-19T23:32:16Z

ITestDirectoryCommitProtocol now working. unsetting per-bucket options made this go away, though I couldn't find out exactly where it was being set (core-site -> auth-keys.xml -> out of hadoop source tree XML)

hadoop-yetus · 2019-06-19T23:54:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	23	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 5 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	74	Maven dependency ordering for branch
+1	mvninstall	1337	trunk passed
+1	compile	1178	trunk passed
+1	checkstyle	158	trunk passed
+1	mvnsite	138	trunk passed
+1	shadedclient	1101	branch has no errors when building and testing our client artifacts.
+1	javadoc	95	trunk passed
0	spotbugs	67	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	195	trunk passed
		_ Patch Compile Tests _
0	mvndep	22	Maven dependency ordering for patch
+1	mvninstall	79	the patch passed
+1	compile	1191	the patch passed
+1	javac	1191	the patch passed
-0	checkstyle	150	root: The patch generated 5 new + 14 unchanged - 0 fixed = 19 total (was 14)
+1	mvnsite	121	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	726	patch has no errors when building and testing our client artifacts.
+1	javadoc	94	the patch passed
+1	findbugs	204	the patch passed
		_ Other Tests _
+1	unit	536	hadoop-common in the patch passed.
-1	unit	283	hadoop-aws in the patch failed.
+1	asflicense	47	The patch does not generate ASF License warnings.
		7728

Reason	Tests
Failed junit tests	hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/1/artifact/out/Dockerfile
GITHUB PR	#992
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux 6e2b3815149e 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 71ecd2e
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/1/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/1/testReport/
Max. process+thread count	1322 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-06-20T00:40:35Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	598	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 5 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	69	Maven dependency ordering for branch
+1	mvninstall	1050	trunk passed
+1	compile	1053	trunk passed
+1	checkstyle	142	trunk passed
+1	mvnsite	131	trunk passed
+1	shadedclient	1014	branch has no errors when building and testing our client artifacts.
+1	javadoc	106	trunk passed
0	spotbugs	68	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	184	trunk passed
		_ Patch Compile Tests _
0	mvndep	24	Maven dependency ordering for patch
+1	mvninstall	80	the patch passed
+1	compile	999	the patch passed
+1	javac	999	the patch passed
-0	checkstyle	185	root: The patch generated 5 new + 14 unchanged - 0 fixed = 19 total (was 14)
+1	mvnsite	132	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	687	patch has no errors when building and testing our client artifacts.
+1	javadoc	106	the patch passed
+1	findbugs	207	the patch passed
		_ Other Tests _
+1	unit	531	hadoop-common in the patch passed.
-1	unit	293	hadoop-aws in the patch failed.
+1	asflicense	53	The patch does not generate ASF License warnings.
		7667

Reason	Tests
Failed junit tests	hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/2/artifact/out/Dockerfile
GITHUB PR	#992
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux b11b4b62f323 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 71ecd2e
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/2/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/2/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/2/testReport/
Max. process+thread count	1376 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-06-20T01:35:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	42	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 7 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	52	Maven dependency ordering for branch
+1	mvninstall	1236	trunk passed
+1	compile	1024	trunk passed
+1	checkstyle	186	trunk passed
+1	mvnsite	130	trunk passed
+1	shadedclient	1103	branch has no errors when building and testing our client artifacts.
+1	javadoc	96	trunk passed
0	spotbugs	64	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	185	trunk passed
		_ Patch Compile Tests _
0	mvndep	21	Maven dependency ordering for patch
+1	mvninstall	76	the patch passed
+1	compile	986	the patch passed
+1	javac	986	the patch passed
-0	checkstyle	148	root: The patch generated 6 new + 14 unchanged - 0 fixed = 20 total (was 14)
+1	mvnsite	122	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	849	patch has no errors when building and testing our client artifacts.
+1	javadoc	94	the patch passed
+1	findbugs	204	the patch passed
		_ Other Tests _
+1	unit	522	hadoop-common in the patch passed.
-1	unit	292	hadoop-aws in the patch failed.
+1	asflicense	46	The patch does not generate ASF License warnings.
		7376

Reason	Tests
Failed junit tests	hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/3/artifact/out/Dockerfile
GITHUB PR	#992
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux d8193d94ac33 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `5bfdf62`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/3/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/3/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/3/testReport/
Max. process+thread count	1504 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-06-20T10:38:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	40	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 7 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	21	Maven dependency ordering for branch
+1	mvninstall	1131	trunk passed
+1	compile	1069	trunk passed
+1	checkstyle	146	trunk passed
+1	mvnsite	123	trunk passed
+1	shadedclient	1064	branch has no errors when building and testing our client artifacts.
+1	javadoc	93	trunk passed
0	spotbugs	67	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	196	trunk passed
		_ Patch Compile Tests _
0	mvndep	25	Maven dependency ordering for patch
+1	mvninstall	79	the patch passed
+1	compile	1026	the patch passed
+1	javac	1026	the patch passed
-0	checkstyle	151	root: The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14)
+1	mvnsite	121	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	743	patch has no errors when building and testing our client artifacts.
+1	javadoc	119	the patch passed
+1	findbugs	259	the patch passed
		_ Other Tests _
+1	unit	631	hadoop-common in the patch passed.
-1	unit	321	hadoop-aws in the patch failed.
+1	asflicense	60	The patch does not generate ASF License warnings.
		7436

Reason	Tests
Failed junit tests	hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/4/artifact/out/Dockerfile
GITHUB PR	#992
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux 70ad53e83e67 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `5bfdf62`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/4/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/4/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/4/testReport/
Max. process+thread count	1490 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran · 2019-06-24T16:01:35Z

checkstyle is confused. This is an enum. There are no values. If I added a default I'd just get told off "all options are already covered"

steveloughran · 2019-06-24T16:10:22Z

reviewing my own patch (and looking @ failure reports of unrelated runs), I plan to change the probes for _SUCCESS of the previous stage to be different: fail with a message "previous stage did not generate output"; always work out and report which stage in the list has the first missing entry, therefore is the first failure to worry about

steveloughran · 2019-06-24T16:26:59Z

also got a failure in the first test case where the source dir still existed. Need to Make sure it is deleted in setup

steveloughran · 2019-06-24T16:28:21Z

[ERROR] test_110_teragen(org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter) Time elapsed: 0.981 s <<< ERROR!org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory s3a://hwdev-steve-ireland-new/terasort-ITestTerasortMagicCommitter/sortin already exists
at org.apache.hadoop.examples.terasort.TeraOutputFormat.checkOutputSpecs(TeraOutputFormat.java:130)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:277)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1576)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1573)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1573)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1594)
at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:304)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.fs.s3a.commit.terasort.AbstractCommitTerasortIT.executeStage(AbstractCommitTerasortIT.java:139)
at org.apache.hadoop.fs.s3a.commit.terasort.AbstractCommitTerasortIT.test_110_teragen(AbstractCommitTerasortIT.java:167)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
``

bgaborg · 2019-06-24T16:48:05Z

Thanks for working on this @steveloughran; this will stabilize the integration tests.
When running the tests I get the following error:

[ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 14.781 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit
[ERROR] testDefault(org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit) Time elapsed: 0.22 s <<< FAILURE!
java.lang.AssertionError: Conflict resolution mode in PartitionedStagingCommitter{StagingCommitter{AbstractS3ACommitter{role=Task committer attempt_job_0001_r_000002_3, name=partitioned, outputPath=s3a://bucket-name/output/path, workPath=file:/var/folders/n_/cnl5_2s56511p7f5842_p8yc0000gp/T/buffer/736a1d32-6626-45bf-ab48-ad50650e9459/_temporary/0/_temporary/attempt_job_0001_r_000002_3}, conflictResolution=APPEND, wrappedCommitter=FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_job_0001}; taskId=attempt_job_0001_r_000002_3, status=''}; org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter@31d48fe3}; outputPath=file:/var/folders/n_/cnl5_2s56511p7f5842_p8yc0000gp/T/cluster/gaborbota/736a1d32-6626-45bf-ab48-ad50650e9459/staging-uploads, workPath=null, algorithmVersion=1, skipCleanup=false, ignoreCleanupFailures=false}}} expected:<FAIL> but was:<APPEND>

Maybe I do something wrong, forgot to set something?

steveloughran · 2019-06-24T21:45:59Z

@bgaborg yes, seeing that too.

steveloughran · 2019-06-24T21:57:04Z

Gabor, fixed that; also rebased to trunk.

I have not done a full itest on this yet*

hadoop-yetus · 2019-06-24T23:55:45Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	91	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 9 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	27	Maven dependency ordering for branch
+1	mvninstall	1294	trunk passed
+1	compile	964	trunk passed
+1	checkstyle	143	trunk passed
+1	mvnsite	123	trunk passed
+1	shadedclient	1016	branch has no errors when building and testing our client artifacts.
+1	javadoc	94	trunk passed
0	spotbugs	65	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	180	trunk passed
		_ Patch Compile Tests _
0	mvndep	20	Maven dependency ordering for patch
+1	mvninstall	77	the patch passed
+1	compile	924	the patch passed
+1	javac	924	the patch passed
-0	checkstyle	146	root: The patch generated 2 new + 15 unchanged - 0 fixed = 17 total (was 15)
+1	mvnsite	118	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	706	patch has no errors when building and testing our client artifacts.
+1	javadoc	92	the patch passed
+1	findbugs	196	the patch passed
		_ Other Tests _
+1	unit	535	hadoop-common in the patch passed.
+1	unit	289	hadoop-aws in the patch passed.
+1	asflicense	43	The patch does not generate ASF License warnings.
		7088

Subsystem	Report/Notes
Docker	Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/7/artifact/out/Dockerfile
GITHUB PR	#992
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux 93bda9c4257e 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `129576f`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/7/artifact/out/diff-checkstyle-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/7/testReport/
Max. process+thread count	1347 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/7/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran · 2019-06-25T13:53:20Z

Tested: S3A ireland

All good except the root dir tests, which Gabor and I are worrying about on the basis that it may be a sign of a regression

hadoop-yetus · 2019-06-25T15:44:04Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	35	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 9 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	66	Maven dependency ordering for branch
+1	mvninstall	1085	trunk passed
+1	compile	1046	trunk passed
+1	checkstyle	135	trunk passed
+1	mvnsite	130	trunk passed
+1	shadedclient	992	branch has no errors when building and testing our client artifacts.
+1	javadoc	92	trunk passed
0	spotbugs	71	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	189	trunk passed
		_ Patch Compile Tests _
0	mvndep	20	Maven dependency ordering for patch
+1	mvninstall	79	the patch passed
+1	compile	997	the patch passed
+1	javac	997	the patch passed
-0	checkstyle	142	root: The patch generated 2 new + 15 unchanged - 0 fixed = 17 total (was 15)
+1	mvnsite	126	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	669	patch has no errors when building and testing our client artifacts.
+1	javadoc	104	the patch passed
+1	findbugs	200	the patch passed
		_ Other Tests _
-1	unit	519	hadoop-common in the patch failed.
+1	unit	295	hadoop-aws in the patch passed.
+1	asflicense	51	The patch does not generate ASF License warnings.
		6995

Reason	Tests
Failed junit tests	hadoop.util.TestReadWriteDiskValidator

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/8/artifact/out/Dockerfile
GITHUB PR	#992
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux 4a6eada0bd8c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `48d7f00`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/8/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/8/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/8/testReport/
Max. process+thread count	1427 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/8/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

This changes the default conflict resolution mode (in code, core-default and docs) to append, to be consistent with FileOutputCommitter. Also checks for all conflict modes that the destination doesn't exist; if not calls mkdirs. If it does exist, verify that it is a directory. Committer test doesn't set partition policy or conflict mode, so verifies the defaults work (the test was changed before the source tests, to verify this was true) Most the change is actually to the mocking tests, because of the switch from FS.exists(path) to getFileStatus() + mkdirs. this is why I hate mocking so much Change-Id: I6fe9a12eea702cf75b35b8c7fb0db3f185d3f40b

…citly go back to append; new defaults aren't being picked up Change-Id: I7837cd100c77eeaafafbb422074b0944282fde82

… clearing bucket options. Somehow locally the commit confict mode fs.s3a.committer.staging.conflict-mode was retaining as fail, even with the core-default.xml set. Extra tests to log the origin of the option on an assert failure implied somehow this was coming via test/resources/core-site.xml and any transitive XMLIncludes from there. resetting the per-bucket options was enough to make this "go away"; the test is retained in case someone else's setting is similar. Change-Id: I3a882919a78438b22fa90e9bceb516a4de640082 Note: if the core-site actually sets it for all buckets, rather than per-bucket, then the new test will fail. Why not clear it? Because we want to verify that the default is now append.

Change-Id: I8da28bce5c5715501fb539e5bafdc247a9c81b62

fix up a test failure Change-Id: I3af7235173b4e35ac51ba1258c4693ad4064b2b0

+ log the terasort CSV file to $TMP rather than the remote FS, where it was being deleted in teardown Change-Id: I5e2e68f7e3d356d8f3dbde784a5fb0796cc888e6

Change-Id: I6224eb5b11de937e6f21d834e5a28a101f1eef89

steveloughran · 2019-07-01T12:57:02Z

rebase to trunk and push up again to kick off a retest

hadoop-yetus · 2019-07-01T14:22:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	89	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 9 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	69	Maven dependency ordering for branch
+1	mvninstall	1213	trunk passed
+1	compile	1066	trunk passed
+1	checkstyle	152	trunk passed
+1	mvnsite	134	trunk passed
+1	shadedclient	1111	branch has no errors when building and testing our client artifacts.
+1	javadoc	105	trunk passed
0	spotbugs	73	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	208	trunk passed
		_ Patch Compile Tests _
0	mvndep	23	Maven dependency ordering for patch
+1	mvninstall	83	the patch passed
+1	compile	993	the patch passed
+1	javac	993	the patch passed
-0	checkstyle	151	root: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15)
+1	mvnsite	126	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	745	patch has no errors when building and testing our client artifacts.
+1	javadoc	97	the patch passed
+1	findbugs	206	the patch passed
		_ Other Tests _
+1	unit	559	hadoop-common in the patch passed.
+1	unit	284	hadoop-aws in the patch passed.
+1	asflicense	46	The patch does not generate ASF License warnings.
		7463

Subsystem	Report/Notes
Docker	Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/9/artifact/out/Dockerfile
GITHUB PR	#992
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux 559b90f4a4a0 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `1e727cf`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/9/artifact/out/diff-checkstyle-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/9/testReport/
Max. process+thread count	1633 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/9/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-07-01T14:57:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	31	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 9 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	62	Maven dependency ordering for branch
+1	mvninstall	1079	trunk passed
+1	compile	1068	trunk passed
+1	checkstyle	140	trunk passed
+1	mvnsite	111	trunk passed
+1	shadedclient	918	branch has no errors when building and testing our client artifacts.
+1	javadoc	86	trunk passed
0	spotbugs	61	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	169	trunk passed
		_ Patch Compile Tests _
0	mvndep	21	Maven dependency ordering for patch
+1	mvninstall	74	the patch passed
+1	compile	996	the patch passed
+1	javac	996	the patch passed
-0	checkstyle	138	root: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15)
+1	mvnsite	115	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	636	patch has no errors when building and testing our client artifacts.
+1	javadoc	108	the patch passed
+1	findbugs	206	the patch passed
		_ Other Tests _
+1	unit	538	hadoop-common in the patch passed.
+1	unit	297	hadoop-aws in the patch passed.
+1	asflicense	53	The patch does not generate ASF License warnings.
		6888

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/10/artifact/out/Dockerfile
GITHUB PR	#992
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux 0e01d3bec442 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `1e727cf`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/10/artifact/out/diff-checkstyle-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/10/testReport/
Max. process+thread count	1361 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-992/10/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran · 2019-07-09T19:08:09Z

checkstyle is mistaken; this patch is ready to go in.

mackrorysd · 2019-07-09T22:53:58Z

I'm gonna give a +0.5 for now. I reviewed the code and it looks good to me, but I don't think I fully understand the practical consequences of changing the default resolution mode just yet.

steveloughran · 2019-07-10T15:20:37Z

we're changing the default mode to == that of FileOutputCommitter, namely "it's ok if the dest exists"

Spark will refuse to execute if the dest exists, so to you use any of the conflict options you actually need to permit overwrite -so you won't change behaviour without noticing

What this does do is make the committer more consistent with the existing one

mackrorysd · 2019-07-10T16:08:43Z

Ok, sounds reasonable enough. +1 from me.

steveloughran · 2019-07-11T17:16:24Z

thanks, committed to trunk.

…erFactory (apache#992)

steveloughran force-pushed the s3/HADOOP-16357-commit-append branch from 55d0ee9 to af06a4aCompare June 19, 2019 22:31

steveloughran commented Jun 24, 2019
View reviewed changes

Comment thread...c/test/java/org/apache/hadoop/fs/s3a/commit/staging/TestStagingDirectoryOutputCommitter.java Outdated

steveloughran force-pushed the s3/HADOOP-16357-commit-append branch 2 times, most recently from 63c1c24 to e27aa70Compare June 24, 2019 21:56

steveloughran changed the title ~~HADOOP-16537 TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists~~HADOOP-16357 TeraSort Job failing on S3 DirectoryStagingCommitter: destination path existsJun 25, 2019

steveloughran added 7 commits July 1, 2019 13:54

HADOOP-16357 ITestTerasortDirectoryCommitter only working if we expli…
f642e2d
…citly go back to append; new defaults aren't being picked up Change-Id: I7837cd100c77eeaafafbb422074b0944282fde82

HADOOP-13657: address checkstyle issues
a30be34
Change-Id: I8da28bce5c5715501fb539e5bafdc247a9c81b62

HADOOP-16357 S3A terasort tests to give more diagnostics
560cf84
fix up a test failure Change-Id: I3af7235173b4e35ac51ba1258c4693ad4064b2b0

HADOOP-13657 fix up the regression in validation of _SUCCESS files
d455990
+ log the terasort CSV file to $TMP rather than the remote FS, where it was being deleted in teardown Change-Id: I5e2e68f7e3d356d8f3dbde784a5fb0796cc888e6

HADOOP-16357: fix a checkstyle
899c36b
Change-Id: I6224eb5b11de937e6f21d834e5a28a101f1eef89

steveloughran force-pushed the s3/HADOOP-16357-commit-append branch from e4afee1 to 899c36bCompare July 1, 2019 12:55

steveloughran added the fs/s3 changes related to hadoop-aws; submitter must declare test endpoint label Jul 10, 2019

steveloughran closed this Jul 11, 2019

steveloughran deleted the s3/HADOOP-16357-commit-append branch July 30, 2019 15:12

shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
SAMZA-2163: Add try-catch to get the version in MetricsSnapshotReport…
b0caee8
…erFactory (apache#992)

brahmareddybattula mentioned this pull request Apr 28, 2022
HADOOP-18079. Upgrade Netty to 4.1.77. #3977
Merged

Uh oh!

Conversation

steveloughran commented Jun 19, 2019

Uh oh!

steveloughran commented Jun 19, 2019

Uh oh!

steveloughran commented Jun 19, 2019

Uh oh!

hadoop-yetus commented Jun 19, 2019

Uh oh!

hadoop-yetus commented Jun 20, 2019

Uh oh!

hadoop-yetus commented Jun 20, 2019

Uh oh!

hadoop-yetus commented Jun 20, 2019

Uh oh!

steveloughran commented Jun 24, 2019

Uh oh!

Uh oh!

steveloughran commented Jun 24, 2019

Uh oh!

steveloughran commented Jun 24, 2019

Uh oh!

steveloughran commented Jun 24, 2019

Uh oh!

bgaborg commented Jun 24, 2019

Uh oh!

steveloughran commented Jun 24, 2019

Uh oh!

steveloughran commented Jun 24, 2019

Uh oh!

hadoop-yetus commented Jun 24, 2019

Uh oh!

steveloughran commented Jun 25, 2019

Uh oh!

hadoop-yetus commented Jun 25, 2019

Uh oh!

steveloughran commented Jul 1, 2019

Uh oh!

hadoop-yetus commented Jul 1, 2019

Uh oh!

hadoop-yetus commented Jul 1, 2019

Uh oh!

steveloughran commented Jul 9, 2019

Uh oh!

mackrorysd commented Jul 9, 2019

Uh oh!

steveloughran commented Jul 10, 2019

Uh oh!

mackrorysd commented Jul 10, 2019

Uh oh!

steveloughran commented Jul 11, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants