Skip to content

TEZ-4723: TestMapProcessor is flaky - #504

Merged
abstractdog merged 4 commits into
apache:masterfrom
abstractdog:TEZ-4723
Jun 8, 2026
Merged

TEZ-4723: TestMapProcessor is flaky#504
abstractdog merged 4 commits into
apache:masterfrom
abstractdog:TEZ-4723

Conversation

@abstractdog

@abstractdogabstractdog commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@abstractdog
abstractdog requested a review from ayushtknJune 2, 2026 09:54
ayushtkn
ayushtkn reviewed Jun 2, 2026
View reviewed changes
Comment threadtez-mapreduce/src/test/java/org/apache/tez/mapreduce/processor/map/TestMapProcessor.java Outdated
output.collect(key, value);
if (counter.incrementAndGet() % N == 0) {
try {
Thread.sleep(1);

@ayushtknayushtknJun 2, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will not fix it I believe, it will increase the probability of it passing I believe. Can we wait in the Test's monitoring thread for some X condition instead? Like maybe using GenericTestUtils.waitFor()

@abstractdogabstractdogJun 3, 2026

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, the current solution is just a sophisticated Thread.sleep :) let me improve it

@tez-yetus

tez-yetus commented Jun 2, 2026

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec5m 38sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗detsecrets0m 1sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 💚mvninstall3m 12smaster passed
+1 💚compile2m 25smaster passed
+1 💚checkstyle0m 20smaster passed
+1 💚javadoc0m 22smaster passed
+0 🆗spotbugs0m 35stez-mapreduce in master has 97 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall2m 8sthe patch passed
+1 💚codespell0m 53sNo new issues.
+1 💚compile2m 24sthe patch passed
+1 💚javac2m 24sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 14sthe patch passed
+1 💚javadoc0m 18sthe patch passed
+1 💚spotbugs0m 39sthe patch passed
_ Other Tests _
+1 💚unit58m 27sroot in the patch passed.
+1 💚asflicense0m 20sThe patch does not generate ASF License warnings.
78m 56s
SubsystemReport/Notes
DockerClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/1/artifact/out/Dockerfile
Optional Testsdupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
unameLinux 0518a0fadd0b 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitytez-personality.sh
git revisionmaster / 330fdc8
Default JavaEclipse Adoptium-21.0.11+10-LTS
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/1/testReport/
Max. process+thread count1344 (vs. ulimit of 5500)
modulesC: tez-mapreduce U: tez-mapreduce
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/1/console
versionsgit=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered byApache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

@abstractdog
abstractdogforce-pushed the TEZ-4723 branch 2 times, most recently from 5a07af8 to 8521d5dCompareJune 3, 2026 11:39
@abstractdog

abstractdog commented Jun 3, 2026

Copy link
Copy Markdown
ContributorAuthor

@ayushtkn : created LatchMapper, please let me know if it's fine

@tez-yetus

tez-yetus commented Jun 3, 2026

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec2m 20sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 💚mvninstall3m 3smaster passed
+1 💚compile2m 27smaster passed
+1 💚checkstyle0m 23smaster passed
+1 💚javadoc0m 24smaster passed
+0 🆗spotbugs0m 35stez-mapreduce in master has 97 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall2m 11sthe patch passed
+1 💚codespell0m 48sNo new issues.
+1 💚compile2m 31sthe patch passed
+1 💚javac2m 31sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 18stez-mapreduce: The patch generated 0 new + 2 unchanged - 2 fixed = 2 total (was 4)
+1 💚javadoc0m 22sthe patch passed
+1 💚spotbugs0m 43sthe patch passed
_ Other Tests _
+1 💚unit58m 48sroot in the patch passed.
+1 💚asflicense0m 21sThe patch does not generate ASF License warnings.
76m 17s
SubsystemReport/Notes
DockerClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/5/artifact/out/Dockerfile
Optional Testsdupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
unameLinux 15ea1b7fa1d1 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitytez-personality.sh
git revisionmaster / 330fdc8
Default JavaEclipse Adoptium-21.0.11+10-LTS
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/5/testReport/
Max. process+thread count2127 (vs. ulimit of 5500)
modulesC: tez-mapreduce U: tez-mapreduce
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/5/console
versionsgit=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered byApache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

ayushtkn
ayushtkn approved these changes Jun 8, 2026
View reviewed changes

@ayushtknayushtkn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Minor suggestions, feel free to commit once you handle it

Comment threadtez-mapreduce/src/test/java/org/apache/tez/mapreduce/processor/map/TestMapProcessor.java Outdated
Comment on lines 286 to 288
scheduler.shutdownNow();
task.close();
sharedExecutor.shutdownNow();

@ayushtknayushtknJun 8, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe this could be in finally block, if the above assertion fails, resources would get leaked

@abstractdogabstractdogJun 8, 2026

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack, fixed in 384c066
also made TezSharedExecutorAutoCloseable

@tez-yetus

tez-yetus commented Jun 8, 2026

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec3m 2sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep0m 29sMaven dependency ordering for branch
+1 💚mvninstall2m 49smaster passed
+1 💚compile2m 30smaster passed
+1 💚checkstyle0m 38smaster passed
+1 💚javadoc0m 40smaster passed
+0 🆗spotbugs0m 33stez-common in master has 13 extant spotbugs warnings.
+0 🆗spotbugs0m 36stez-mapreduce in master has 97 extant spotbugs warnings.
_ Patch Compile Tests _
+0 🆗mvndep0m 6sMaven dependency ordering for patch
+1 💚mvninstall2m 13sthe patch passed
+1 💚codespell0m 49sNo new issues.
+1 💚compile2m 32sthe patch passed
+1 💚javac2m 32sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
-0 ⚠️checkstyle0m 19s/results-checkstyle-tez-mapreduce.txttez-mapreduce: The patch generated 1 new + 1 unchanged - 3 fixed = 2 total (was 4)
+1 💚javadoc0m 38sthe patch passed
+1 💚spotbugs1m 24sthe patch passed
_ Other Tests _
+1 💚unit59m 25sroot in the patch passed.
+1 💚asflicense0m 40sThe patch does not generate ASF License warnings.
80m 53s
SubsystemReport/Notes
DockerClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/6/artifact/out/Dockerfile
Optional Testsdupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
unameLinux e4b05018e27e 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitytez-personality.sh
git revisionmaster / 330fdc8
Default JavaEclipse Adoptium-21.0.11+10-LTS
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/6/testReport/
Max. process+thread count1392 (vs. ulimit of 5500)
modulesC: tez-common tez-mapreduce U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/6/console
versionsgit=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered byApache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

@tez-yetus

tez-yetus commented Jun 8, 2026

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 8sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep0m 29sMaven dependency ordering for branch
+1 💚mvninstall2m 49smaster passed
+1 💚compile2m 27smaster passed
+1 💚checkstyle0m 38smaster passed
+1 💚javadoc0m 40smaster passed
+0 🆗spotbugs0m 35stez-common in master has 13 extant spotbugs warnings.
+0 🆗spotbugs0m 37stez-mapreduce in master has 97 extant spotbugs warnings.
_ Patch Compile Tests _
+0 🆗mvndep0m 6sMaven dependency ordering for patch
+1 💚mvninstall2m 10sthe patch passed
+1 💚codespell0m 48sNo new issues.
+1 💚compile2m 29sthe patch passed
+1 💚javac2m 29sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 17sThe patch passed checkstyle in tez-common
+1 💚checkstyle0m 18stez-mapreduce: The patch generated 0 new + 1 unchanged - 3 fixed = 1 total (was 4)
+1 💚javadoc0m 40sthe patch passed
+1 💚spotbugs1m 17sthe patch passed
_ Other Tests _
-1 ❌unit58m 58s/patch-unit-root.txtroot in the patch passed.
+1 💚asflicense0m 39sThe patch does not generate ASF License warnings.
77m 22s
ReasonTests
Failed junit teststez.test.TestAMRecoveryAggregationBroadcast
SubsystemReport/Notes
DockerClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/7/artifact/out/Dockerfile
Optional Testsdupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
unameLinux b63fdbbbcd47 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitytez-personality.sh
git revisionmaster / 330fdc8
Default JavaEclipse Adoptium-21.0.11+10-LTS
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/7/testReport/
Max. process+thread count2129 (vs. ulimit of 5500)
modulesC: tez-common tez-mapreduce U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/7/console
versionsgit=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered byApache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

@tez-yetus

tez-yetus commented Jun 8, 2026

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 7sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep0m 30sMaven dependency ordering for branch
+1 💚mvninstall2m 45smaster passed
+1 💚compile2m 26smaster passed
+1 💚checkstyle0m 38smaster passed
+1 💚javadoc0m 40smaster passed
+0 🆗spotbugs0m 34stez-common in master has 13 extant spotbugs warnings.
+0 🆗spotbugs0m 35stez-mapreduce in master has 97 extant spotbugs warnings.
_ Patch Compile Tests _
+0 🆗mvndep0m 6sMaven dependency ordering for patch
+1 💚mvninstall2m 10sthe patch passed
+1 💚codespell0m 48sNo new issues.
+1 💚compile2m 28sthe patch passed
+1 💚javac2m 28sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 16sThe patch passed checkstyle in tez-common
+1 💚checkstyle0m 17stez-mapreduce: The patch generated 0 new + 1 unchanged - 3 fixed = 1 total (was 4)
+1 💚javadoc0m 39sthe patch passed
+1 💚spotbugs1m 22sthe patch passed
_ Other Tests _
+1 💚unit60m 34sroot in the patch passed.
+1 💚asflicense0m 40sThe patch does not generate ASF License warnings.
78m 55s
SubsystemReport/Notes
DockerClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/8/artifact/out/Dockerfile
Optional Testsdupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
unameLinux 278c82cff98d 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitytez-personality.sh
git revisionmaster / 330fdc8
Default JavaEclipse Adoptium-21.0.11+10-LTS
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/8/testReport/
Max. process+thread count1452 (vs. ulimit of 5500)
modulesC: tez-common tez-mapreduce U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/8/console
versionsgit=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered byApache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

@abstractdog
abstractdog merged commit cba5689 into apache:masterJun 8, 2026
4 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ayushtknayushtknayushtkn approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@abstractdog@tez-yetus@ayushtkn