Skip to content

TEZ-4569: SCATTER_GATHER + BROADCAST hangs on DAG Recovery - #361

Merged
abstractdog merged 5 commits into
apache:masterfrom
okumin:TEZ-4569-hang
Dec 23, 2024
Merged

TEZ-4569: SCATTER_GATHER + BROADCAST hangs on DAG Recovery#361
abstractdog merged 5 commits into
apache:masterfrom
okumin:TEZ-4569-hang

Conversation

@okumin

@okuminokumin commented Jun 12, 2024

Copy link
Copy Markdown
Contributor

Let an AM correctly restore its state and restart tasks.
https://issues.apache.org/jira/browse/TEZ-4569

@tez-yetus

This comment was marked as outdated.

@tez-yetus

This comment was marked as outdated.

@okuminokumin changed the title [WIP] TEZ-4569: SCATTER_GATHER + BROADCAST hangs on DAG RecoveryTEZ-4569: SCATTER_GATHER + BROADCAST hangs on DAG RecoveryAug 23, 2024
Comment threadtez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java Outdated
Comment threadtez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java Outdated
@okumin

Copy link
Copy Markdown
ContributorAuthor

I rebased this branch and also added two cosmetic changes.

@abstractdog
abstractdog self-requested a review December 23, 2024 11:26

@abstractdogabstractdog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1
pending tests

@tez-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec28m 3sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep2m 58sMaven dependency ordering for branch
+1 💚mvninstall10m 56smaster passed
+1 💚compile1m 24smaster passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚compile1m 9smaster passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
+1 💚checkstyle1m 16smaster passed
+1 💚javadoc0m 51smaster passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚javadoc0m 41smaster passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
+0 🆗spotbugs1m 3sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs2m 57smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 10sMaven dependency ordering for patch
+1 💚mvninstall0m 51sthe patch passed
+1 💚compile0m 55sthe patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚javac0m 55sthe patch passed
+1 💚compile0m 48sthe patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
+1 💚javac0m 48sthe patch passed
-0 ⚠️checkstyle0m 11stez-tests: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 19sthe patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚javadoc0m 19sthe patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
+1 💚findbugs2m 0sthe patch passed
_ Other Tests _
-1 ❌unit5m 2stez-dag in the patch failed.
+1 💚unit43m 19stez-tests in the patch passed.
+1 💚asflicense0m 21sThe patch does not generate ASF License warnings.
106m 23s
ReasonTests
Failed junit teststez.dag.app.dag.impl.TestDAGRecovery
SubsystemReport/Notes
DockerClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-361/3/artifact/out/Dockerfile
GITHUB PR#361
JIRA IssueTEZ-4569
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
unameLinux 1c2c562e42b5 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / 9efa6f1
Default JavaPrivate Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
checkstylehttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-361/3/artifact/out/diff-checkstyle-tez-tests.txt
unithttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-361/3/artifact/out/patch-unit-tez-dag.txt
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-361/3/testReport/
Max. process+thread count1134 (vs. ulimit of 5500)
modulesC: tez-dag tez-tests U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-361/3/console
versionsgit=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@okumin

Copy link
Copy Markdown
ContributorAuthor

I'm checking why TestDAGRecovery failed

@abstractdog

Copy link
Copy Markdown
Contributor

I'm checking why TestDAGRecovery failed

sometimes recovery tests are flaky, I simply restarted the last precommit job

@okumin

Copy link
Copy Markdown
ContributorAuthor

@abstractdog Sorry, my hand-made refactoring included a mistake. I copy-pasted the method names used in the original condition. I appreciate it if you could double-check it.
bdece70

@abstractdog

Copy link
Copy Markdown
Contributor

@abstractdog Sorry, my hand-made refactoring included a mistake. I copy-pasted the method names used in the original condition. I appreciate it if you could double-check it. bdece70

missed it, I saw it's fixed, glad to see that unit tests revealed the problem
+1 still holds if tests will pass

@tez-yetus

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 15sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep2m 48sMaven dependency ordering for branch
+1 💚mvninstall13m 59smaster passed
+1 💚compile1m 19smaster passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚compile1m 11smaster passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
+1 💚checkstyle1m 31smaster passed
+1 💚javadoc0m 57smaster passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚javadoc0m 42smaster passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
+0 🆗spotbugs0m 50sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs2m 37smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 10sMaven dependency ordering for patch
+1 💚mvninstall0m 48sthe patch passed
+1 💚compile0m 51sthe patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚javac0m 51sthe patch passed
+1 💚compile0m 44sthe patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
+1 💚javac0m 44sthe patch passed
-0 ⚠️checkstyle0m 12stez-tests: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 19sthe patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚javadoc0m 19sthe patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
+1 💚findbugs2m 0sthe patch passed
_ Other Tests _
+1 💚unit5m 15stez-dag in the patch passed.
+1 💚unit46m 32stez-tests in the patch passed.
+1 💚asflicense0m 22sThe patch does not generate ASF License warnings.
84m 40s
SubsystemReport/Notes
DockerClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-361/5/artifact/out/Dockerfile
GITHUB PR#361
JIRA IssueTEZ-4569
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
unameLinux 7e9468acddb5 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / 9efa6f1
Default JavaPrivate Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu222.04-ga
checkstylehttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-361/5/artifact/out/diff-checkstyle-tez-tests.txt
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-361/5/testReport/
Max. process+thread count1187 (vs. ulimit of 5500)
modulesC: tez-dag tez-tests U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-361/5/console
versionsgit=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@abstractdog
abstractdog merged commit 44c4f1e into apache:masterDec 23, 2024
@okumin
okumin deleted the TEZ-4569-hang branch December 24, 2024 01:16
@okumin

Copy link
Copy Markdown
ContributorAuthor

Thanks!

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@okumin@tez-yetus@abstractdog