Skip to content

[DRAFT] TEZ-4505: Create counters about time intervals spent in certain states in StateMachineTez - #304

Open
abstractdog wants to merge 1 commit into
apache:masterfrom
abstractdog:TEZ-4505
Open

[DRAFT] TEZ-4505: Create counters about time intervals spent in certain states in StateMachineTez#304
abstractdog wants to merge 1 commit into
apache:masterfrom
abstractdog:TEZ-4505

Conversation

@abstractdog

Copy link
Copy Markdown
Contributor

No description provided.

@tez-yetus

This comment was marked as outdated.

@tez-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 18sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗mvndep5m 40sMaven dependency ordering for branch
+1 💚mvninstall8m 54smaster passed
+1 💚compile1m 25smaster passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚compile1m 24smaster passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu122.04-b09
+1 💚checkstyle1m 31smaster passed
+1 💚javadoc1m 22smaster passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚javadoc1m 7smaster passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu122.04-b09
+0 🆗spotbugs1m 22sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs2m 56smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for patch
+1 💚mvninstall0m 50sthe patch passed
+1 💚compile0m 56sthe patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚javac0m 56sthe patch passed
+1 💚compile0m 47sthe patch passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu122.04-b09
+1 💚javac0m 47sthe patch passed
-0 ⚠️checkstyle0m 26stez-dag: The patch generated 8 new + 206 unchanged - 0 fixed = 214 total (was 206)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 35sthe patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚javadoc0m 36sthe patch passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu122.04-b09
+1 💚findbugs2m 17sthe patch passed
_ Other Tests _
+1 💚unit2m 19stez-api in the patch passed.
-1 ❌unit5m 40stez-dag in the patch failed.
+1 💚asflicense0m 23sThe patch does not generate ASF License warnings.
41m 27s
ReasonTests
Failed junit teststez.dag.app.TestSpeculation
tez.dag.app.dag.impl.TestDAGImpl
tez.dag.app.TestPreemption
tez.dag.app.dag.impl.TestCommit
tez.dag.app.TestMockDAGAppMaster
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-304/2/artifact/out/Dockerfile
GITHUB PR#304
JIRA IssueTEZ-4505
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
unameLinux 745bdcc7db9f 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / 5beab4c
Default JavaPrivate Build-1.8.0_362-8u372-gaus1-0ubuntu122.04-b09
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-gaus1-0ubuntu122.04-b09
checkstylehttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-304/2/artifact/out/diff-checkstyle-tez-dag.txt
unithttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-304/2/artifact/out/patch-unit-tez-dag.txt
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-304/2/testReport/
Max. process+thread count690 (vs. ulimit of 5500)
modulesC: tez-api tez-dag U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-304/2/console
versionsgit=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@ayushtknayushtkn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds cool to me, I think you need to add some test around it as well


private boolean isStateIntervalMonitorEnabled = false;
private long lastStateChangedTime = Time.monotonicNow();
private Map<String, Long> intervalSpentInStatesMs = new HashMap<>();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be final

private final StateMachine<STATE, EVENTTYPE, EVENT> realStatemachine;

private boolean isStateIntervalMonitorEnabled = false;
private long lastStateChangedTime = Time.monotonicNow();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be set as part of enableStateIntervalMonitor?

}
}

private void maybeMergeAllTaskAttemptCounters(TezCounters taskCounters) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit
I think we can ditch maybe from the name & just have mergeAllTaskAttemptCounters(, if stateIntervalMontior is enabled it does merge always

*/
package org.apache.tez.common.counters;

public class TaskAttemptCounter {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't decode the usage of this 'empty' class, we are only using the name of this class, if only name is required can't we have a string constant or an enum

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is just a leftover, actually I'm in the middle of this, full of bugs, let me set this PR to [DRAFT]

@abstractdogabstractdog changed the title TEZ-4505: Create counters about time intervals spent in certain states in StateMachineTez[DRAFT] TEZ-4505: Create counters about time intervals spent in certain states in StateMachineTezSep 14, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@abstractdog@tez-yetus@ayushtkn