HADOOP-17376. ITestS3AContractRename failing against stricter tests

[adoroszlai]

What changes were proposed in this pull request?

Fix ITestS3AContractRename failure:

[ERROR] testRenameFileOverExistingFile[auth=false](org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)  Time elapsed: 2.805 s  <<< FAILURE!
java.lang.AssertionError: expected rename(s3a://rename/test/source-256.txt, s3a://rename/test/dest-512.txt) to be rejected with exception, but got false
	at org.junit.Assert.fail(Assert.java:88)
	at org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileOverExistingFile(AbstractContractRenameTest.java:131)

by declaring rename-returns-false-if-dest-exists=true in the contract.

https://issues.apache.org/jira/browse/HADOOP-17376

How was this patch tested?

Ran ITestS3AContractRename against Ozone S3 Gateway.

1. Download, build and run Ozone:

git clone https://github.com/apache/ozone
cd ozone
git checkout a28cda57e
mvn -DskipTests -Dskip.installnpx -Dskip.npx -DskipShade clean package
cd hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/compose/ozone
docker-compose up -d --scale datanode=3

2. Wait about a minute for Ozone to start.
3. Create bucket for test:

aws s3api create-bucket --bucket rename --endpoint http://localhost:9878

4. Configure test to run against Ozone:

  <property>
    <name>fs.s3a.endpoint</name>
    <value>http://127.0.0.1:9878/</value>
  </property>
  <property>
    <name>test.fs.s3a.name</name>
    <value>s3a://rename/</value>
  </property>

5. Run test in Hadoop source:

cd ~/src/hadoop
mvn -pl :hadoop-aws -Dtest='ITestS3AContractRename' test

Now ITestS3AContractRename#testRenameFileOverExistingFile proceeds beyond previous failure in line 131, and fails due to Ozone-specific incompatibility in line 146:

[ERROR] testRenameFileOverExistingFile[auth=false](org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)  Time elapsed: 3.468 s  <<< ERROR!
org.apache.hadoop.fs.s3a.NoVersionAttributeException: `s3a://rename/test/dest-512.txt': Change detection policy requires ETag
	at org.apache.hadoop.fs.s3a.impl.ChangeTracker.processNewRevision(ChangeTracker.java:275)
	at org.apache.hadoop.fs.s3a.impl.ChangeTracker.processMetadata(ChangeTracker.java:261)
	at org.apache.hadoop.fs.s3a.impl.ChangeTracker.processResponse(ChangeTracker.java:195)
	at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:208)
	at org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:359)
	at org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$3(Invoker.java:223)
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110)
	at org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$5(Invoker.java:347)
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407)
	at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:343)
	at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:221)
	at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:265)
	at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:351)
	at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:464)
	at java.io.DataInputStream.read(DataInputStream.java:149)
	at org.apache.hadoop.fs.contract.ContractTestUtils.readDataset(ContractTestUtils.java:216)
	at org.apache.hadoop.fs.contract.ContractTestUtils.verifyFileContents(ContractTestUtils.java:240)
	at org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileOverExistingFile(AbstractContractRenameTest.java:146)

I expect this to work on real S3.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	26m 3s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 10s		trunk passed
+1 💚	shadedclient	49m 29s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 37s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	shadedclient	16m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 32s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		95m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2462/1/artifact/out/Dockerfile
GITHUB PR	#2462
Optional Tests	dupname asflicense unit xml
uname	Linux a7d152c39402 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / fc961b6
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2462/1/testReport/
Max. process+thread count	535 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2462/1/console
versions	git=2.17.1 maven=3.6.0
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[steveloughran]

LGTM,

+1. merging. (Attempting to do it from IntelliJ, FWIW)

That NoVersionAttributeException means that the code to detect changes in a file while open didn't get an etag in the GET response. Ozone needs to fix that. In the meantime, you can set a different change detection policy for your buckets, so disable that logic

[adoroszlai]

Thanks @steveloughran for reviewing and merging it.

In the meantime, you can set a different change detection policy for your buckets, so disable that logic

Yup, I've found HDDS-4243, which mentions these properties.