Skip to content

TEZ-4339: Expose real-time memory consumption of AM and task containers via DagClient - #157

Merged
abstractdog merged 2 commits into
apache:masterfrom
abstractdog:TEZ-4339
Nov 2, 2021
Merged

TEZ-4339: Expose real-time memory consumption of AM and task containers via DagClient#157
abstractdog merged 2 commits into
apache:masterfrom
abstractdog:TEZ-4339

Conversation

@abstractdog

@abstractdogabstractdog commented Oct 24, 2021

Copy link
Copy Markdown
Contributor

No description provided.

@tez-yetus

This comment has been minimized.

Sign in to view
@tez-yetus

This comment has been minimized.

Sign in to view
rbalamohan
rbalamohan reviewed Oct 25, 2021
View reviewed changes
Comment threadtez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TaskReporter.java Outdated
rbalamohan
rbalamohan reviewed Oct 25, 2021
View reviewed changes
Comment threadtez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TaskReporter.java Outdated
@tez-yetus

tez-yetus commented Oct 26, 2021

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec16m 49sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗prototool0m 0sprototool was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 2 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep4m 18sMaven dependency ordering for branch
+1 💚mvninstall8m 59smaster passed
+1 💚compile2m 20smaster passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚compile2m 10smaster passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚checkstyle2m 10smaster passed
+1 💚javadoc2m 20smaster passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚javadoc2m 4smaster passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗spotbugs0m 40sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs3m 57smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 9sMaven dependency ordering for patch
+1 💚mvninstall1m 23sthe patch passed
+1 💚compile1m 28sthe patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚cc1m 28sthe patch passed
+1 💚javac1m 28sthe patch passed
+1 💚compile1m 17sthe patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚cc1m 17sthe patch passed
+1 💚javac1m 17sthe patch passed
+1 💚checkstyle0m 14stez-api: The patch generated 0 new + 78 unchanged - 5 fixed = 78 total (was 83)
-0 ⚠️checkstyle0m 9stez-runtime-internals: The patch generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4)
-0 ⚠️checkstyle0m 23stez-dag: The patch generated 1 new + 183 unchanged - 0 fixed = 184 total (was 183)
+1 💚checkstyle0m 12sThe patch passed checkstyle in tez-tests
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc1m 14sthe patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚javadoc1m 8sthe patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚findbugs3m 24sthe patch passed
_ Other Tests _
+1 💚unit2m 1stez-api in the patch passed.
+1 💚unit0m 33stez-runtime-internals in the patch passed.
+1 💚unit4m 18stez-dag in the patch passed.
-1 ❌unit42m 53stez-tests in the patch failed.
+1 💚asflicense0m 43sThe patch does not generate ASF License warnings.
108m 26s
ReasonTests
Failed junit teststez.test.TestRecovery
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-157/3/artifact/out/Dockerfile
GITHUB PR#157
JIRA IssueTEZ-4339
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile cc prototool
unameLinux b8562b6e5388 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / 3326978
Default JavaPrivate Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
checkstylehttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-157/3/artifact/out/diff-checkstyle-tez-runtime-internals.txt
checkstylehttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-157/3/artifact/out/diff-checkstyle-tez-dag.txt
unithttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-157/3/artifact/out/patch-unit-tez-tests.txt
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-157/3/testReport/
Max. process+thread count1381 (vs. ulimit of 5500)
modulesC: tez-api tez-runtime-internals tez-dag tez-tests U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-157/3/console
versionsgit=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

rbalamohan
rbalamohan reviewed Oct 31, 2021
View reviewed changes
Comment threadtez-dag/src/main/java/org/apache/tez/dag/api/client/DAGStatusBuilder.java
getBuilder().addVertexProgress(builder.build());
}

//TODO: let this be a map of values in protobuf 3.x

@rbalamohanrbalamohanOct 31, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this todo for? or am i missing something here?

@abstractdogabstractdogNov 1, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to put these memory related stuff into a map, which can be dinamically extended later with other metrics if needed, but unfortunately the protoc compiler which is used in protobuf 2.5 doesn't support maps

rbalamohan
rbalamohan approved these changes Nov 2, 2021
View reviewed changes

@rbalamohanrbalamohan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. +1

@abstractdog
abstractdog merged commit f39a51e into apache:masterNov 2, 2021
@tez-yetustez-yetus mentioned this pull request Aug 7, 2025
Open
@tez-yetustez-yetus mentioned this pull request Aug 22, 2025
Merged
This was referenced Sep 15, 2025
Closed
Merged
@tez-yetustez-yetus mentioned this pull request Oct 14, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rbalamohanrbalamohanrbalamohan approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@abstractdog@tez-yetus@rbalamohan