TEZ-4589: Counter for the overall duration of succeeded/failed/killed task attempts

[abstractdog]

tested on cluster with different scenarios, hive LLAP

1. 2 queries running in parallel, leading to task kills, also I killed random nodes during the test

INFO : Run DAG 149.16s
INFO : NUM_FAILED_TASKS: 18
INFO : NUM_KILLED_TASKS: 11078
INFO : NUM_SUCCEEDED_TASKS: 774
INFO : TOTAL_LAUNCHED_TASKS: 11870
INFO : DURATION_FAILED_TASKS_MILLIS: 125559
INFO : DURATION_KILLED_TASKS_MILLIS: 2474797
INFO : DURATION_SUCCEEDED_TASKS_MILLIS: 15740900
ratio of failed to succeeded durations: 0.0080 = 0.80%
ratio of killed to succeeded durations: 0.1575 = 15.75%

low killed/succeeded ratio implies the hive LLAP behavior (task preempted quite fast, 11078 task kills didn't contributed that much)

2. single query, but more aggressive node killing

INFO : Run DAG 392.71s
INFO : NUM_FAILED_TASKS: 210
INFO : NUM_KILLED_TASKS: 334
INFO : NUM_SUCCEEDED_TASKS: 880
INFO : TOTAL_LAUNCHED_TASKS: 1318
INFO : DURATION_FAILED_TASKS_MILLIS: 14433036
INFO : DURATION_KILLED_TASKS_MILLIS: 12815980
INFO : DURATION_SUCCEEDED_TASKS_MILLIS: 31076258
ratio of failed to succeeded durations: 0.4643 = 46.43%
ratio of killed to succeeded durations: 0.4117 = 41.17%

btw, this time, task kills were in the early started Reducer5 tasks, as many Map 4 source tasks have failed (so reducer considered unhealthy most probably):

----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 2 .......... llap SUCCEEDED 1 1 0 0 0 0
Map 1 .......... llap SUCCEEDED 1 1 0 0 0 0
Map 3 .......... llap SUCCEEDED 5 5 0 0 1 0
Map 4 .......... llap SUCCEEDED 675 675 0 0 209 0
Reducer 5 ...... llap SUCCEEDED 92 92 0 0 0 334 ----------------------------------------------------------------------------------------------
VERTICES: 05/05 [==========================>>] 100% ELAPSED TIME: 394.27 s
----------------------------------------------------------------------------------------------

3. normal query, no contention, no node failures

INFO : NUM_SUCCEEDED_TASKS: 774
INFO : TOTAL_LAUNCHED_TASKS: 774
INFO : DURATION_SUCCEEDED_TASKS_MILLIS: 15253902

[ayushtkn]

LGTM

[tez-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 9s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	6m 3s	Maven dependency ordering for branch
+1 💚	mvninstall	11m 6s	master passed
+1 💚	compile	0m 48s	master passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	compile	0m 47s	master passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
+1 💚	checkstyle	1m 10s	master passed
+1 💚	javadoc	0m 56s	master passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 43s	master passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
+0 🆗	spotbugs	0m 45s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 0s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 7s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 32s	the patch passed
+1 💚	compile	0m 31s	the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	javac	0m 31s	the patch passed
+1 💚	compile	0m 28s	the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
+1 💚	javac	0m 28s	the patch passed
+1 💚	checkstyle	0m 18s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 19s	the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 24s	the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
+1 💚	findbugs	1m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 57s	tez-api in the patch passed.
+1 💚	unit	4m 14s	tez-dag in the patch passed.
+1 💚	asflicense	0m 21s	The patch does not generate ASF License warnings.
		35m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-382/9/artifact/out/Dockerfile
GITHUB PR	#382
JIRA Issue	TEZ-4589
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 2b0776be3846 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / 53309ea
Default Java	Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-382/9/testReport/
Max. process+thread count	645 (vs. ulimit of 5500)
modules	C: tez-api tez-dag U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-382/9/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.