TEZ-4725: Fix flaky tests in TestAMRecoveryAggregationBroadcast by maheshrajus · Pull Request #520 · apache/tez

maheshrajus · 2026-07-14T07:18:59Z

Root causes and fixes:

TestAMRecoveryAggregationBroadcast.testMapJoinTemporalFailure (race condition)
Replace fixed Thread.sleep(10s) before AM kill with a deterministic
waitForVertexSucceeded() helper that polls DAGClient.getVertexStatus()
every 500ms (up to 60s) until the target vertices reach SUCCEEDED state.
Each test now waits for only the vertices it logically depends on before
killing the AM, ensuring the recovery log assertions always see the
expected counts.
Make OUT_PATH unique per test run (random suffix) to eliminate cross-test.
DAGClientRPCImpl / TezClientUtils: port out of range:-1 (YARN-808 gap)
YARN sets rpcPort=-1 when an AM container is allocated (state=RUNNING)
but the AM has not yet bound its RPC listener. The existing guard only
checked rpcPort==0 (protobuf default), so rpcPort==-1 reached
NetUtils.createSocketAddrForHost(), which threw
IllegalArgumentException: port out of range:-1.
Fix DAGClientRPCImpl.createAMProxyIfNeeded(): rpcPort == 0 → rpcPort <= 0.
Fix TezClientUtils.getAMProxy(FrameworkClient,...): add the same
rpcPort <= 0 guard

tez-yetus · 2026-07-14T09:10:39Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	4m 11s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 35s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 33s		master passed
+1 💚	compile	3m 59s		master passed
+1 💚	checkstyle	1m 4s		master passed
+1 💚	javadoc	1m 11s		master passed
+0 🆗	spotbugs	1m 23s		tez-api in master has 92 extant spotbugs warnings.
+0 🆗	spotbugs	0m 53s		tez-tests in master has 6 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 51s		the patch passed
+1 💚	codespell	1m 37s		No new issues.
+1 💚	compile	4m 5s		the patch passed
+1 💚	javac	4m 5s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 30s	/results-checkstyle-tez-api.txt	tez-api: The patch generated 2 new + 62 unchanged - 0 fixed = 64 total (was 62)
+1 💚	javadoc	1m 10s		the patch passed
+1 💚	spotbugs	2m 33s		the patch passed
			_ Other Tests _
+1 💚	unit	75m 58s		root in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		111m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/1/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux c15aef73b5a9 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `4bccf3a`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/1/testReport/
Max. process+thread count	1525 (vs. ulimit of 5500)
modules	C: tez-api tez-tests U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/1/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

maheshrajus · 2026-07-28T11:13:44Z

@abstractdog I have addressed your review comments. Could you please review and approve the PR at your convenience? Thanks !

tez-yetus · 2026-07-28T12:09:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	3m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 32s		master passed
+1 💚	compile	4m 3s		master passed
+1 💚	checkstyle	1m 3s		master passed
+1 💚	javadoc	1m 12s		master passed
+0 🆗	spotbugs	1m 23s		tez-api in master has 92 extant spotbugs warnings.
+0 🆗	spotbugs	0m 51s		tez-tests in master has 6 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 40s		the patch passed
+1 💚	codespell	1m 36s		No new issues.
+1 💚	compile	4m 2s		the patch passed
+1 💚	javac	4m 2s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 31s	/results-checkstyle-tez-api.txt	tez-api: The patch generated 2 new + 62 unchanged - 0 fixed = 64 total (was 62)
+1 💚	javadoc	1m 10s		the patch passed
+1 💚	spotbugs	2m 37s		the patch passed
			_ Other Tests _
+1 💚	unit	75m 54s		root in the patch passed.
+1 💚	asflicense	0m 56s		The patch does not generate ASF License warnings.
		110m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/2/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux d02fa7af64e9 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `efab6a1`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/2/testReport/
Max. process+thread count	1479 (vs. ulimit of 5500)
modules	C: tez-api tez-tests U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/2/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

abstractdog

thanks @maheshrajus , only a few minor comments left

abstractdog

1 more thing, otherwise it's okay

tez-yetus · 2026-07-29T15:23:08Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 12s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 31s		master passed
+1 💚	compile	4m 5s		master passed
+1 💚	checkstyle	1m 2s		master passed
+1 💚	javadoc	1m 13s		master passed
+0 🆗	spotbugs	1m 23s		tez-api in master has 92 extant spotbugs warnings.
+0 🆗	spotbugs	0m 53s		tez-tests in master has 6 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s		Maven dependency ordering for patch
+1 💚	mvninstall	4m 0s		the patch passed
+1 💚	codespell	1m 36s		No new issues.
+1 💚	compile	4m 3s		the patch passed
+1 💚	javac	4m 3s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 30s	/results-checkstyle-tez-api.txt	tez-api: The patch generated 2 new + 62 unchanged - 0 fixed = 64 total (was 62)
+1 💚	javadoc	1m 10s		the patch passed
+1 💚	spotbugs	2m 38s		the patch passed
			_ Other Tests _
+1 💚	unit	78m 8s		root in the patch passed.
+1 💚	asflicense	0m 57s		The patch does not generate ASF License warnings.
		109m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/4/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux dc30764bbeb3 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `84e634a`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/4/testReport/
Max. process+thread count	1287 (vs. ulimit of 5500)
modules	C: tez-api tez-tests U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/4/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

abstractdog requested changes Jul 22, 2026
View reviewed changes

maheshrajus added 2 commits July 27, 2026 22:53

TEZ-4725: Fix flaky tests in TestAMRecoveryAggregationBroadcast
82edab6

Review comments fix:1
9cedc91

maheshrajus force-pushed the TEZ-4725_flaky branch from 37f881e to 9cedc91Compare July 28, 2026 10:18

maheshrajus requested a review from abstractdog July 28, 2026 10:18

abstractdog reviewed Jul 29, 2026
View reviewed changes

Comment threadtez-tests/src/test/java/org/apache/tez/test/TestAMRecoveryAggregationBroadcast.java

abstractdog requested changes Jul 29, 2026
View reviewed changes

Comment threadtez-tests/src/test/java/org/apache/tez/test/TestAMRecoveryAggregationBroadcast.java Outdated
Comment threadtez-tests/src/test/java/org/apache/tez/test/TestAMRecoveryAggregationBroadcast.java Outdated
Comment threadtez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java

Review comments fix:2
0329d2e

maheshrajus requested a review from abstractdog July 29, 2026 12:35

abstractdog requested changes Jul 29, 2026
View reviewed changes

Comment threadtez-tests/src/test/java/org/apache/tez/test/TestAMRecoveryAggregationBroadcast.java Outdated

Review comments fix: 3
9b1e494

maheshrajus requested a review from abstractdog July 29, 2026 13:33

abstractdog approved these changes Jul 29, 2026
View reviewed changes

abstractdog merged commit d348bf1 into apache:masterJul 29, 2026
6 checks passed

Uh oh!

Conversation

maheshrajus commented Jul 14, 2026

Root causes and fixes:

Uh oh!

tez-yetus commented Jul 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maheshrajus commented Jul 28, 2026

Uh oh!

tez-yetus commented Jul 28, 2026

Uh oh!

Uh oh!

abstractdog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abstractdog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tez-yetus commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants