TEZ-4305: Invoke hflush, only if the stream supports it by kishendas · Pull Request #120 · apache/tez

kishendas · 2021-04-26T09:15:13Z

Following exception is thrown whenever we do ProtoMessageWriter.hflush on S3, which internally calls S3ABlockOutputStream.hflush which is not supported and it throws java.lang.UnsupportedOperationException.

bdffe22d96ae [mdc@18060 class="yarn.YarnUncaughtExceptionHandler" level="ERROR" thread="HistoryEventHandlingThread"] Thread Thread[HistoryEventHandlingThread, 5,main] threw an Exception.^Mjava.lang.UnsupportedOperationException: S3A streams are not Syncable^M at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.hflush(S3ABlockOutputStream.java:657)^M at org.apache.hadoop.fs.FS DataOutputStream.hflush(FSDataOutputStream.java:136)^M at org.apache.hadoop.io.SequenceFile$Writer.hflush(SequenceFile.java:1367)^M at org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(ProtoMessageWr iter.java:64)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.finishCurrentDag(ProtoHistoryLoggingService.java:239)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.han dleEvent(ProtoHistoryLoggingService.java:198)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.loop(ProtoHistoryLoggingService.java:153)^M at java.lang.Thread.run(Thread.java:748)^M

In order to fix this issue, we should first check if the method is supported or not using StreamCapabilities, before invoking it.

kishendas · 2021-04-26T09:29:41Z

This PR has dependency with apache/hadoop#2949 . Will change the pom.xml to reflect the latest Hadoop version, whenever apache/hadoop#2949 lands.

hadoop-yetus · 2021-04-26T09:48:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	12m 47s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	13m 2s	master passed
+1 💚	compile	0m 30s	master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 29s	master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 4s	master passed
+1 💚	javadoc	0m 37s	master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 26s	master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+0 🆗	spotbugs	1m 8s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 7s	master passed
		_ Patch Compile Tests _
-1 ❌	mvninstall	0m 13s	tez-protobuf-history-plugin in the patch failed.
-1 ❌	compile	0m 14s	tez-protobuf-history-plugin in the patch failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.
-1 ❌	javac	0m 14s	tez-protobuf-history-plugin in the patch failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.
-1 ❌	compile	0m 14s	tez-protobuf-history-plugin in the patch failed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.
-1 ❌	javac	0m 14s	tez-protobuf-history-plugin in the patch failed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.
+1 💚	checkstyle	0m 9s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 13s	the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 13s	the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
-1 ❌	findbugs	0m 14s	tez-protobuf-history-plugin in the patch failed.
		_ Other Tests _
-1 ❌	unit	0m 14s	tez-protobuf-history-plugin in the patch failed.
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		32m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/Dockerfile
GITHUB PR	#120
JIRA Issue	TEZ-4305
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux fa39febd9696 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `0184781`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
mvninstall	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-mvninstall-tez-plugins_tez-protobuf-history-plugin.txt
compile	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-compile-tez-plugins_tez-protobuf-history-plugin-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt
javac	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-compile-tez-plugins_tez-protobuf-history-plugin-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt
compile	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-compile-tez-plugins_tez-protobuf-history-plugin-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt
javac	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-compile-tez-plugins_tez-protobuf-history-plugin-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt
findbugs	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-findbugs-tez-plugins_tez-protobuf-history-plugin.txt
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/artifact/out/patch-unit-tez-plugins_tez-protobuf-history-plugin.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/testReport/
Max. process+thread count	96 (vs. ulimit of 5500)
modules	C: tez-plugins/tez-protobuf-history-plugin U: tez-plugins/tez-protobuf-history-plugin
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/1/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

jteagles · 2021-05-11T18:27:19Z

@kishendas, we will have to rethink this change to not break users who wish to run against other versions of Hadoop. @abstractdog, we may need to consider separate branch for Hadoop 3.3. There are several API changes and dependencies that are making supporting all of Hadoop 3.x difficult

hadoop-yetus · 2021-05-11T19:07:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	12m 57s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	5m 3s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 28s	master passed
+1 💚	compile	2m 32s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	2m 25s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 28s	master passed
+1 💚	javadoc	2m 30s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 1s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	7m 17s	Used deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗	findbugs	7m 12s	root in master has 2 extant findbugs warnings.
		_ Patch Compile Tests _
+0 🆗	mvndep	1m 56s	Maven dependency ordering for patch
-1 ❌	mvninstall	5m 38s	root in the patch failed.
-1 ❌	compile	2m 51s	root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.
-1 ❌	javac	2m 51s	root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.
-1 ❌	compile	2m 29s	root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
-1 ❌	javac	2m 29s	root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
+1 💚	checkstyle	1m 25s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	javadoc	3m 1s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 23s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
-1 ❌	findbugs	6m 50s	root in the patch failed.
		_ Other Tests _
+1 💚	unit	0m 45s	tez-protobuf-history-plugin in the patch passed.
-1 ❌	unit	4m 48s	root in the patch failed.
+1 💚	asflicense	0m 49s	The patch does not generate ASF License warnings.
		82m 48s

Reason	Tests
Failed junit tests	tez.dag.api.client.rpc.TestDAGClient

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/Dockerfile
GITHUB PR	#120
JIRA Issue	TEZ-4305
Optional Tests	dupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle
uname	Linux 4bc6b059830c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `0184781`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
mvninstall	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-mvninstall-root.txt
compile	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
javac	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
compile	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
javac	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
findbugs	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-findbugs-root.txt
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/artifact/out/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/testReport/
Max. process+thread count	404 (vs. ulimit of 5500)
modules	C: tez-plugins/tez-protobuf-history-plugin . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/2/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

kishendas · 2021-05-11T21:14:32Z

@kishendas, we will have to rethink this change to not break users who wish to run against other versions of Hadoop. @abstractdog, we may need to consider separate branch for Hadoop 3.3. There are several API changes and dependencies that are making supporting all of Hadoop 3.x difficult

Please check now. It should now work with older versions of Hadoop as well.

hadoop-yetus · 2021-05-11T22:15:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	5m 9s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 5s	master passed
+1 💚	compile	2m 34s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	2m 25s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 35s	master passed
+1 💚	javadoc	2m 33s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 2s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	5m 34s	Used deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗	findbugs	5m 29s	root in master has 2 extant findbugs warnings.
		_ Patch Compile Tests _
+0 🆗	mvndep	1m 51s	Maven dependency ordering for patch
-1 ❌	mvninstall	4m 3s	root in the patch failed.
-1 ❌	compile	2m 9s	root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.
-1 ❌	javac	2m 9s	root in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.
-1 ❌	compile	1m 56s	root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
-1 ❌	javac	1m 56s	root in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
+1 💚	checkstyle	1m 13s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	javadoc	2m 30s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 9s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
-1 ❌	findbugs	0m 52s	tez-plugins/tez-protobuf-history-plugin generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌	findbugs	5m 2s	root in the patch failed.
		_ Other Tests _
+1 💚	unit	0m 45s	tez-protobuf-history-plugin in the patch passed.
-1 ❌	unit	3m 47s	root in the patch failed.
+1 💚	asflicense	0m 55s	The patch does not generate ASF License warnings.
		61m 53s

Reason	Tests
FindBugs	module:tez-plugins/tez-protobuf-history-plugin
	instanceof will always return true for all non-null values in org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(), since all org.apache.hadoop.io.SequenceFile$Writer are instances of org.apache.hadoop.fs.StreamCapabilities At ProtoMessageWriter.java:for all non-null values in org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(), since all org.apache.hadoop.io.SequenceFile$Writer are instances of org.apache.hadoop.fs.StreamCapabilities At ProtoMessageWriter.java:[line 65]
Failed junit tests	tez.dag.api.client.rpc.TestDAGClient

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/Dockerfile
GITHUB PR	#120
JIRA Issue	TEZ-4305
Optional Tests	dupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle
uname	Linux 97d8c24e3c9d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `0184781`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
mvninstall	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-mvninstall-root.txt
compile	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
javac	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
compile	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
javac	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
findbugs	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/new-findbugs-tez-plugins_tez-protobuf-history-plugin.html
findbugs	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-findbugs-root.txt
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/artifact/out/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/testReport/
Max. process+thread count	444 (vs. ulimit of 5500)
modules	C: tez-plugins/tez-protobuf-history-plugin . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/3/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

kishendas · 2021-05-12T21:45:34Z

Created a ticket for the unit test failure, as it seems to be unrelated -> https://issues.apache.org/jira/browse/TEZ-4306 .

hadoop-yetus · 2021-05-14T20:53:01Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 12s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	5m 14s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 30s	master passed
+1 💚	compile	4m 6s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	3m 45s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	2m 34s	master passed
+1 💚	javadoc	3m 45s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 9s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	6m 15s	Used deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗	findbugs	6m 10s	root in master has 2 extant findbugs warnings.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s	Maven dependency ordering for patch
+1 💚	mvninstall	4m 51s	the patch passed
+1 💚	compile	3m 59s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	3m 59s	the patch passed
+1 💚	compile	3m 41s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	3m 41s	the patch passed
-0 ⚠️	checkstyle	0m 25s	tez-common: The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0)
-0 ⚠️	checkstyle	0m 25s	tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-0 ⚠️	checkstyle	0m 49s	root: The patch generated 11 new + 1 unchanged - 0 fixed = 12 total (was 1)
-1 ❌	whitespace	0m 0s	The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	javadoc	3m 54s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 21s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	findbugs	9m 43s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 48s	tez-common in the patch passed.
+1 💚	unit	4m 32s	tez-dag in the patch passed.
+1 💚	unit	0m 39s	tez-protobuf-history-plugin in the patch passed.
+1 💚	unit	74m 25s	root in the patch passed.
+1 💚	asflicense	1m 46s	The patch does not generate ASF License warnings.
		157m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/artifact/out/Dockerfile
GITHUB PR	#120
JIRA Issue	TEZ-4305
Optional Tests	dupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle
uname	Linux cc82bf444fb3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `0184781`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/artifact/out/diff-checkstyle-tez-common.txt
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/artifact/out/diff-checkstyle-root.txt
whitespace	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/artifact/out/whitespace-eol.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/testReport/
Max. process+thread count	1409 (vs. ulimit of 5500)
modules	C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/5/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-05-14T20:54:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	12m 55s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	5m 19s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 31s	master passed
+1 💚	compile	4m 6s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	3m 47s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	2m 35s	master passed
+1 💚	javadoc	3m 52s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 3s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	6m 16s	Used deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗	findbugs	6m 11s	root in master has 2 extant findbugs warnings.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s	Maven dependency ordering for patch
+1 💚	mvninstall	4m 50s	the patch passed
+1 💚	compile	3m 58s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	3m 58s	the patch passed
+1 💚	compile	3m 39s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	3m 39s	the patch passed
-0 ⚠️	checkstyle	0m 25s	tez-common: The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0)
-0 ⚠️	checkstyle	0m 26s	tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-0 ⚠️	checkstyle	0m 49s	root: The patch generated 11 new + 1 unchanged - 0 fixed = 12 total (was 1)
-1 ❌	whitespace	0m 0s	The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	javadoc	3m 52s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 20s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	findbugs	9m 41s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 48s	tez-common in the patch passed.
+1 💚	unit	4m 32s	tez-dag in the patch passed.
+1 💚	unit	0m 40s	tez-protobuf-history-plugin in the patch passed.
+1 💚	unit	76m 3s	root in the patch passed.
+1 💚	asflicense	1m 56s	The patch does not generate ASF License warnings.
		171m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/Dockerfile
GITHUB PR	#120
JIRA Issue	TEZ-4305
Optional Tests	dupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle
uname	Linux 2d01e63a2bd9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `0184781`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/diff-checkstyle-tez-common.txt
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/diff-checkstyle-root.txt
whitespace	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/whitespace-eol.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/testReport/
Max. process+thread count	1446 (vs. ulimit of 5500)
modules	C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2021-05-15T05:05:56Z

LGTM +1, just two minor asks:

could you please squash the commits for easier later reference?
please address checkstyle/whitespace warnings:
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/whitespace-eol.txt
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/4/artifact/out/diff-checkstyle-root.txt

created TEZ-4307 about Hadoop 3.4.0 to track the fact that this cannot be pushed until we're not upgraded to Hadoop 3.4.0

kishendas · 2021-05-15T08:39:58Z

@abstractdog Please take a look. We can land this change without waiting for Hadoop 3.4.0 upgrade, as this change will work with both Hadoop 3.4.0 and previous versions. It's just that behavior will be different based on the Hadoop version we use with Tez.

hadoop-yetus · 2021-05-15T09:15:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	4m 30s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 29s	master passed
+1 💚	compile	1m 38s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 28s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 41s	master passed
+1 💚	javadoc	1m 40s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 27s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 41s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 53s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 53s	the patch passed
+1 💚	compile	0m 57s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 57s	the patch passed
+1 💚	compile	0m 50s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 50s	the patch passed
-0 ⚠️	checkstyle	0m 9s	tez-common: The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0)
-0 ⚠️	checkstyle	0m 8s	tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 47s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 45s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	findbugs	2m 12s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 30s	tez-common in the patch passed.
+1 💚	unit	4m 17s	tez-dag in the patch passed.
+1 💚	unit	0m 21s	tez-protobuf-history-plugin in the patch passed.
+1 💚	asflicense	0m 29s	The patch does not generate ASF License warnings.
		38m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/6/artifact/out/Dockerfile
GITHUB PR	#120
JIRA Issue	TEZ-4305
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 24744d7bff6d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `0184781`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/6/artifact/out/diff-checkstyle-tez-common.txt
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/6/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/6/testReport/
Max. process+thread count	235 (vs. ulimit of 5500)
modules	C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/6/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2021-05-17T09:43:38Z

@abstractdog Please take a look. We can land this change without waiting for Hadoop 3.4.0 upgrade, as this change will work with both Hadoop 3.4.0 and previous versions. It's just that behavior will be different based on the Hadoop version we use with Tez.

I agree, the latest version is even better because StreamCapabilites is already present in older Hadoop versions
once tez will depend on a hadoop version where HADOOP-17657 present (so writer implements StreamCapabilities), it will behave accordingly

nit: I was wondering if we can place a DEBUG message if hflush is skipped because it's not supported
this codepath is only called from finishCurrentDag, so once per dag, it won't spam the logs...

other than this nit, the patch looks good to me
cc: @harishjp if you have any comments

harishjp · 2021-05-17T11:35:47Z

nit: I was wondering if we can place a DEBUG message if hflush is skipped because it's not supported
this codepath is only called from finishCurrentDag, so once per dag, it won't spam the logs...

other than this nit, the patch looks good to me
cc: @harishjp if you have any comments

This is fine in tez, we use the same code in hive (It has to be copied to hive). And there we write one event per file on s3 since s3 does not support reading a file in which data is still being written. It will write 2 log lines per query there.

kishendas · 2021-05-17T18:27:29Z

@abstractdog I have added the log message. Please commit the PR, if you don't have any other feedback.

hadoop-yetus · 2021-05-17T19:04:09Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	4m 20s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 38s	master passed
+1 💚	compile	1m 39s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 31s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 41s	master passed
+1 💚	javadoc	1m 39s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 26s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 43s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 52s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 52s	the patch passed
+1 💚	compile	0m 56s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 56s	the patch passed
+1 💚	compile	0m 51s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 51s	the patch passed
-0 ⚠️	checkstyle	0m 8s	tez-common: The patch generated 12 new + 0 unchanged - 0 fixed = 12 total (was 0)
-0 ⚠️	checkstyle	0m 9s	tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 47s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 44s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	findbugs	2m 13s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 31s	tez-common in the patch passed.
+1 💚	unit	4m 15s	tez-dag in the patch passed.
+1 💚	unit	0m 22s	tez-protobuf-history-plugin in the patch passed.
+1 💚	asflicense	0m 30s	The patch does not generate ASF License warnings.
		38m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/7/artifact/out/Dockerfile
GITHUB PR	#120
JIRA Issue	TEZ-4305
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux ad740569d412 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `0184781`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/7/artifact/out/diff-checkstyle-tez-common.txt
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/7/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/7/testReport/
Max. process+thread count	207 (vs. ulimit of 5500)
modules	C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/7/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2021-05-18T06:41:45Z

@abstractdog I have added the log message. Please commit the PR, if you don't have any other feedback.

I think the log message should go into an else branch...currently, it's logged every time when StreamCapabilities interface is implemented

kishendas · 2021-05-18T07:02:20Z

@abstractdog My bad. Got tricked by my own comment. Fixed it. Good catch.

hadoop-yetus · 2021-05-18T07:34:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	4m 26s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 44s	master passed
+1 💚	compile	1m 38s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 28s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 41s	master passed
+1 💚	javadoc	1m 36s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 17s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 44s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 3s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 55s	the patch passed
+1 💚	compile	1m 3s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 3s	the patch passed
+1 💚	compile	0m 53s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 53s	the patch passed
-0 ⚠️	checkstyle	0m 9s	tez-common: The patch generated 13 new + 0 unchanged - 0 fixed = 13 total (was 0)
-0 ⚠️	checkstyle	0m 8s	tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 49s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 46s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	findbugs	2m 28s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 32s	tez-common in the patch passed.
-1 ❌	unit	4m 16s	tez-dag in the patch failed.
+1 💚	unit	0m 22s	tez-protobuf-history-plugin in the patch passed.
+1 💚	asflicense	0m 30s	The patch does not generate ASF License warnings.
		39m 11s

Reason	Tests
Failed junit tests	tez.dag.app.rm.TestTaskScheduler

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/8/artifact/out/Dockerfile
GITHUB PR	#120
JIRA Issue	TEZ-4305
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux bd7a09ede325 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `0184781`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/8/artifact/out/diff-checkstyle-tez-common.txt
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/8/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/8/artifact/out/patch-unit-tez-dag.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/8/testReport/
Max. process+thread count	188 (vs. ulimit of 5500)
modules	C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/8/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-05-18T07:37:52Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	4m 27s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 52s	master passed
+1 💚	compile	1m 39s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 30s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 34s	master passed
+1 💚	javadoc	1m 38s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 17s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 39s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 51s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 58s	the patch passed
+1 💚	compile	1m 1s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 1s	the patch passed
+1 💚	compile	0m 54s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 54s	the patch passed
-0 ⚠️	checkstyle	0m 10s	tez-common: The patch generated 13 new + 0 unchanged - 0 fixed = 13 total (was 0)
-0 ⚠️	checkstyle	0m 8s	tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 51s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 48s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	findbugs	2m 24s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 31s	tez-common in the patch passed.
+1 💚	unit	4m 47s	tez-dag in the patch passed.
+1 💚	unit	0m 22s	tez-protobuf-history-plugin in the patch passed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		39m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/9/artifact/out/Dockerfile
GITHUB PR	#120
JIRA Issue	TEZ-4305
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 692d0099bc8e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `0184781`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/9/artifact/out/diff-checkstyle-tez-common.txt
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/9/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/9/testReport/
Max. process+thread count	195 (vs. ulimit of 5500)
modules	C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/9/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2021-05-18T10:33:05Z

@abstractdog My bad. Got tricked by my own comment. Fixed it. Good catch.

thanks!
one more thing, I can still see checkstyle problems, I think you're using 4 spaces instead of 2 spaces which is used in tez project

hadoop-yetus · 2021-05-18T15:51:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	16m 52s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 1s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	4m 12s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 53s	master passed
+1 💚	compile	1m 33s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 19s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 30s	master passed
+1 💚	javadoc	1m 27s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 12s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 40s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 48s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 53s	the patch passed
+1 💚	compile	0m 59s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 59s	the patch passed
+1 💚	compile	0m 50s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 50s	the patch passed
-0 ⚠️	checkstyle	0m 8s	tez-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-0 ⚠️	checkstyle	0m 7s	tez-plugins/tez-protobuf-history-plugin: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	whitespace	0m 1s	The patch has no whitespace issues.
+1 💚	javadoc	0m 50s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 46s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	findbugs	2m 20s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 30s	tez-common in the patch passed.
+1 💚	unit	4m 8s	tez-dag in the patch passed.
+1 💚	unit	0m 21s	tez-protobuf-history-plugin in the patch passed.
+1 💚	asflicense	0m 27s	The patch does not generate ASF License warnings.
		53m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/10/artifact/out/Dockerfile
GITHUB PR	#120
JIRA Issue	TEZ-4305
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 209bf44c28db 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `0184781`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/10/artifact/out/diff-checkstyle-tez-common.txt
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/10/artifact/out/diff-checkstyle-tez-plugins_tez-protobuf-history-plugin.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/10/testReport/
Max. process+thread count	215 (vs. ulimit of 5500)
modules	C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/10/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-05-18T18:51:22Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	13m 40s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	4m 16s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 58s	master passed
+1 💚	compile	1m 42s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 32s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 40s	master passed
+1 💚	javadoc	1m 41s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 28s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 43s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 55s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 54s	the patch passed
+1 💚	compile	0m 57s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 57s	the patch passed
+1 💚	compile	0m 49s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 49s	the patch passed
-0 ⚠️	checkstyle	0m 8s	tez-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 49s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 45s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	findbugs	2m 13s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 31s	tez-common in the patch passed.
+1 💚	unit	4m 15s	tez-dag in the patch passed.
+1 💚	unit	0m 22s	tez-protobuf-history-plugin in the patch passed.
+1 💚	asflicense	0m 30s	The patch does not generate ASF License warnings.
		52m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/11/artifact/out/Dockerfile
GITHUB PR	#120
JIRA Issue	TEZ-4305
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 2acb5d3a8b4a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `0184781`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/11/artifact/out/diff-checkstyle-tez-common.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/11/testReport/
Max. process+thread count	188 (vs. ulimit of 5500)
modules	C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/11/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

make the change backward compatible, if Tez is used with older versions of Hadoop add a utility method do not dependent on Hadoop snapshot Add log Move log message to else block Fix indentation

hadoop-yetus · 2021-05-18T22:40:29Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 29s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	4m 16s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 35s	master passed
+1 💚	compile	1m 38s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 31s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 42s	master passed
+1 💚	javadoc	1m 40s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 26s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 42s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 55s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 54s	the patch passed
+1 💚	compile	0m 56s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 56s	the patch passed
+1 💚	compile	0m 50s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 50s	the patch passed
+1 💚	checkstyle	0m 34s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 48s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 45s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	findbugs	2m 15s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 31s	tez-common in the patch passed.
+1 💚	unit	4m 41s	tez-dag in the patch passed.
+1 💚	unit	0m 23s	tez-protobuf-history-plugin in the patch passed.
+1 💚	asflicense	0m 30s	The patch does not generate ASF License warnings.
		38m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/12/artifact/out/Dockerfile
GITHUB PR	#120
JIRA Issue	TEZ-4305
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 0c7fa85f8ca3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `867e4d2`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/12/testReport/
Max. process+thread count	188 (vs. ulimit of 5500)
modules	C: tez-common tez-dag tez-plugins/tez-protobuf-history-plugin U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-120/12/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

kishendas · 2021-05-18T22:42:27Z

@abstractdog Can you please approve and commit ? Finally tamed the checkstyle beast.

abstractdog · 2021-05-20T14:36:43Z

merged to master, thanks @kishendas for the patch and @harishjp for the review!

kishendas force-pushed the TEZ-4305 branch from b3a8a62 to 7c4a01aCompare May 15, 2021 08:36

kishendas force-pushed the TEZ-4305 branch from 7c4a01a to 5bc3380Compare May 17, 2021 18:25

kishendas force-pushed the TEZ-4305 branch 2 times, most recently from 996235e to b388611Compare May 18, 2021 06:57

kishendas force-pushed the TEZ-4305 branch from b388611 to c0a33bbCompare May 18, 2021 14:57

kishendas force-pushed the TEZ-4305 branch from c0a33bb to 82fc24fCompare May 18, 2021 17:58

invoke hflush, only if the stream supports it
ee5e86c
make the change backward compatible, if Tez is used with older versions of Hadoop add a utility method do not dependent on Hadoop snapshot Add log Move log message to else block Fix indentation

kishendas force-pushed the TEZ-4305 branch from 82fc24f to ee5e86cCompare May 18, 2021 22:01

harishjp approved these changes May 19, 2021
View reviewed changes

abstractdog merged commit 4d37914 into apache:masterMay 20, 2021

Uh oh!

Conversation

kishendas commented Apr 26, 2021

Uh oh!

kishendas commented Apr 26, 2021

Uh oh!

hadoop-yetus commented Apr 26, 2021

Uh oh!

jteagles commented May 11, 2021

Uh oh!

hadoop-yetus commented May 11, 2021

Uh oh!

kishendas commented May 11, 2021

Uh oh!

hadoop-yetus commented May 11, 2021

Uh oh!

kishendas commented May 12, 2021

Uh oh!

hadoop-yetus commented May 14, 2021

Uh oh!

hadoop-yetus commented May 14, 2021

Uh oh!

abstractdog commented May 15, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kishendas commented May 15, 2021

Uh oh!

hadoop-yetus commented May 15, 2021

Uh oh!

abstractdog commented May 17, 2021

Uh oh!

harishjp commented May 17, 2021

Uh oh!

kishendas commented May 17, 2021

Uh oh!

hadoop-yetus commented May 17, 2021

Uh oh!

abstractdog commented May 18, 2021

Uh oh!

kishendas commented May 18, 2021

Uh oh!

hadoop-yetus commented May 18, 2021

Uh oh!

hadoop-yetus commented May 18, 2021

Uh oh!

abstractdog commented May 18, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented May 18, 2021

Uh oh!

hadoop-yetus commented May 18, 2021

Uh oh!

hadoop-yetus commented May 18, 2021

Uh oh!

kishendas commented May 18, 2021

Uh oh!

abstractdog commented May 20, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

abstractdog commented May 15, 2021•
edited
Loading

abstractdog commented May 18, 2021•
edited
Loading