Skip to content

TEZ-4553. Default task scheduler to DagAwareTaskScheduler to avoid hang in TEZ-3535 - #347

Merged
jteagles merged 1 commit into
apache:masterfrom
jteagles:TEZ-4553
May 1, 2024
Merged

TEZ-4553. Default task scheduler to DagAwareTaskScheduler to avoid hang in TEZ-3535#347
jteagles merged 1 commit into
apache:masterfrom
jteagles:TEZ-4553

Conversation

@jteagles

@jteaglesjteagles commented Apr 16, 2024

Copy link
Copy Markdown
Contributor

No description provided.

@tez-yetus

tez-yetus commented Apr 16, 2024

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec26m 18sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗mvndep6m 42sMaven dependency ordering for branch
+1 💚mvninstall11m 43smaster passed
+1 💚compile3m 46smaster passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚compile3m 30smaster passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚checkstyle2m 0smaster passed
+1 💚javadoc2m 21smaster passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚javadoc2m 0smaster passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+0 🆗spotbugs6m 3sUsed deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗findbugs0m 45sbranch/tez-ui no findbugs output file (findbugsXml.xml)
_ Patch Compile Tests _
+0 🆗mvndep0m 15sMaven dependency ordering for patch
+1 💚mvninstall5m 35sthe patch passed
+1 💚compile3m 49sthe patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚javac3m 49sthe patch passed
+1 💚compile3m 33sthe patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚javac3m 33sthe patch passed
+1 💚checkstyle1m 35sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚xml0m 1sThe patch has no ill-formed XML file.
+1 💚javadoc2m 9sthe patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚javadoc2m 4sthe patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+0 🆗findbugs0m 48stez-ui has no data from findbugs
_ Other Tests _
+1 💚unit2m 30stez-api in the patch passed.
+1 💚unit1m 45stez-ui in the patch passed.
-1 ❌unit10m 2sroot in the patch failed.
+1 💚asflicense1m 12sThe patch does not generate ASF License warnings.
112m 2s
ReasonTests
Failed junit teststez.runtime.library.conf.TestUnorderedPartitionedKVEdgeConfig
SubsystemReport/Notes
DockerClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/1/artifact/out/Dockerfile
GITHUB PR#347
Optional Testsdupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle
unameLinux 95a339b00ea8 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / f080031
Default JavaPrivate Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
unithttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/1/artifact/out/patch-unit-root.txt
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/1/testReport/
Max. process+thread count2101 (vs. ulimit of 5500)
modulesC: tez-api tez-ui . U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/1/console
versionsgit=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn
ayushtkn reviewed Apr 18, 2024
View reviewed changes

@ayushtknayushtkn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @jteagles

  • Is the maven change intentional? related to this?
  • UI changes?
  • How come changing the default be a solution to a problem, like avoiding Hang.
  • Can you describe the problem in the description, X -> Y -> Hence hang
  • You mentioned: TEZ-3535, it has ticket linked which talks about tez job getting hang, by any chance you aren't missing that? TEZ-3491

@jteagles

jteagles commented Apr 24, 2024

Copy link
Copy Markdown
ContributorAuthor

Hi @jteagles

  • Is the maven change intentional? related to this?
  • UI changes?
  • How come changing the default be a solution to a problem, like avoiding Hang.
  • Can you describe the problem in the description, X -> Y -> Hence hang
  • You mentioned: TEZ-3535, it has ticket linked which talks about tez job getting hang, by any chance you aren't missing that? TEZ-3491

This jira is changing the default scheduler away from the one with the known hang in TEZ-3535 and to the one added by TEZ-3491. I've been running the TEZ-3491 scheduler DagAwareTaskScheduler for 8 years.
Fixed the extra commit changes

@tez-yetus

tez-yetus commented Apr 24, 2024

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec24m 13sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall14m 36smaster passed
+1 💚compile0m 33smaster passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚compile0m 32smaster passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚checkstyle1m 18smaster passed
+1 💚javadoc0m 48smaster passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚javadoc0m 37smaster passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+0 🆗spotbugs1m 42sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs1m 40smaster passed
_ Patch Compile Tests _
+1 💚mvninstall0m 23sthe patch passed
+1 💚compile0m 25sthe patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚javac0m 25sthe patch passed
+1 💚compile0m 22sthe patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚javac0m 22sthe patch passed
+1 💚checkstyle0m 13sthe patch passed
+1 💚whitespace0m 1sThe patch has no whitespace issues.
+1 💚javadoc0m 24sthe patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚javadoc0m 27sthe patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚findbugs1m 4sthe patch passed
_ Other Tests _
+1 💚unit2m 19stez-api in the patch passed.
+1 💚asflicense0m 16sThe patch does not generate ASF License warnings.
51m 10s
SubsystemReport/Notes
DockerClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/2/artifact/out/Dockerfile
GITHUB PR#347
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
unameLinux e9a2ad51aff2 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / f080031
Default JavaPrivate Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/2/testReport/
Max. process+thread count335 (vs. ulimit of 5500)
modulesC: tez-api U: tez-api
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-347/2/console
versionsgit=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn
ayushtkn approved these changes Apr 24, 2024
View reviewed changes

@ayushtknayushtkn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jteagles
jteagles merged commit a77f8ef into apache:masterMay 1, 2024
@jteagles

jteagles commented May 1, 2024

Copy link
Copy Markdown
ContributorAuthor

Thanks, @ayushtkn.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ayushtknayushtknayushtkn approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jteagles@tez-yetus@ayushtkn