Skip to content

TEZ-4038: Add a /prof profiler endpoint like HiveServer2 has - #212

Merged
abstractdog merged 1 commit into
apache:masterfrom
johnnyCake1:TEZ-4038
Jun 10, 2022
Merged

TEZ-4038: Add a /prof profiler endpoint like HiveServer2 has#212
abstractdog merged 1 commit into
apache:masterfrom
johnnyCake1:TEZ-4038

Conversation

@johnnyCake1

@johnnyCake1johnnyCake1 commented May 30, 2022

Copy link
Copy Markdown
Contributor

Add a /prof profiler endpoint like HiveServer2 has

  • added 2 new endpoints to webuiservice:
    • /prof runs async-profiler
    • /prof-output serves the output file
  • added corresponding tests in TestAM
  • checked manually in browser

@johnnyCake1
johnnyCake1 marked this pull request as draft May 30, 2022 14:37
@johnnyCake1
johnnyCake1 marked this pull request as ready for review May 30, 2022 14:38
@tez-yetus

This comment was marked as outdated.

Sign in to view
abstractdog
abstractdog reviewed May 31, 2022
View reviewed changes
Comment threadtez-common/src/main/java/org/apache/tez/common/web/ProfileOutputServlet.java Outdated
abstractdog
abstractdog reviewed May 31, 2022
View reviewed changes
Comment threadtez-common/src/main/java/org/apache/tez/common/web/ProfileServlet.java Outdated
abstractdog
abstractdog reviewed May 31, 2022
View reviewed changes
Comment threadtez-tests/src/test/java/org/apache/tez/test/TestAM.java Outdated
abstractdog
abstractdog reviewed May 31, 2022
View reviewed changes
Comment threadtez-tests/src/test/java/org/apache/tez/test/TestAM.java Outdated
abstractdog
abstractdog reviewed May 31, 2022
View reviewed changes
Comment threadtez-common/src/main/java/org/apache/tez/common/web/ProfileServlet.java Outdated
abstractdog
abstractdog reviewed May 31, 2022
View reviewed changes
Comment threadtez-common/src/main/java/org/apache/tez/common/web/ProfileServlet.java Outdated
@tez-yetus

This comment was marked as outdated.

Sign in to view
@tez-yetus

This comment was marked as outdated.

Sign in to view
@tez-yetus

This comment was marked as outdated.

Sign in to view
@tez-yetus

This comment was marked as outdated.

Sign in to view
@tez-yetus

This comment was marked as outdated.

Sign in to view
@tez-yetus

This comment was marked as outdated.

Sign in to view
@johnnyCake1
johnnyCake1force-pushed the TEZ-4038 branch 2 times, most recently from a02ea00 to 35d4eefCompareJune 7, 2022 22:40
@tez-yetus

tez-yetus commented Jun 8, 2022

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec20m 55sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep6m 4sMaven dependency ordering for branch
+1 💚mvninstall13m 21smaster passed
+1 💚compile3m 24smaster passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚compile3m 14smaster passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚checkstyle3m 4smaster passed
+1 💚javadoc3m 12smaster passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚javadoc2m 53smaster passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗spotbugs1m 8sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs5m 24smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 10sMaven dependency ordering for patch
+1 💚mvninstall1m 38sthe patch passed
+1 💚compile1m 52sthe patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚javac1m 52sthe patch passed
+1 💚compile1m 39sthe patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚javac1m 39sthe patch passed
+1 💚checkstyle1m 13sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc1m 23sthe patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚javadoc1m 17sthe patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚findbugs3m 48sthe patch passed
_ Other Tests _
+1 💚unit0m 42stez-common in the patch passed.
+1 💚unit5m 59stez-dag in the patch passed.
-1 ❌unit19m 24stez-tests in the patch failed.
+0 🆗asflicense1m 8sASF License check generated no output?
103m 29s
ReasonTests
Failed junit teststez.test.TestAMRecovery
tez.test.TestDAGRecovery2
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-212/8/artifact/out/Dockerfile
GITHUB PR#212
JIRA IssueTEZ-4038
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
unameLinux 48e0ffa6e135 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / e5a5578
Default JavaPrivate Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
unithttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-212/8/artifact/out/patch-unit-tez-tests.txt
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-212/8/testReport/
Max. process+thread count808 (vs. ulimit of 5500)
modulesC: tez-common tez-dag tez-tests U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-212/8/console
versionsgit=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

abstractdog
abstractdog reviewed Jun 9, 2022
View reviewed changes
Comment threadtez-common/src/main/java/org/apache/tez/common/TezUtilsInternal.java Outdated
abstractdog
abstractdog reviewed Jun 9, 2022
View reviewed changes
Comment threadtez-common/src/main/java/org/apache/tez/common/TezUtilsInternal.java Outdated
abstractdog
abstractdog reviewed Jun 9, 2022
View reviewed changes
Comment threadtez-common/src/main/java/org/apache/tez/common/web/ProfileOutputServlet.java Outdated
abstractdog
abstractdog reviewed Jun 9, 2022
View reviewed changes
Comment threadtez-tests/src/test/java/org/apache/tez/test/TestAM.java Outdated
@abstractdog

abstractdog commented Jun 9, 2022

Copy link
Copy Markdown
Contributor

recovery failures are most probably unrelated
left very minor comments, other than that, looks good to me, this is very close to be committed

@johnnyCake1
johnnyCake1force-pushed the TEZ-4038 branch 2 times, most recently from 6b83720 to edf920fCompareJune 9, 2022 14:16
@tez-yetus

This comment was marked as outdated.

Sign in to view
@tez-yetus

This comment was marked as outdated.

Sign in to view
@tez-yetus

tez-yetus commented Jun 9, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec16m 36sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep5m 57sMaven dependency ordering for branch
+1 💚mvninstall10m 40smaster passed
+1 💚compile2m 8smaster passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚compile1m 58smaster passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚checkstyle2m 0smaster passed
+1 💚javadoc2m 3smaster passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚javadoc1m 49smaster passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗spotbugs0m 56sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs3m 27smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 10sMaven dependency ordering for patch
+1 💚mvninstall1m 3sthe patch passed
+1 💚compile1m 9sthe patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚javac1m 9sthe patch passed
+1 💚compile0m 59sthe patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚javac0m 59sthe patch passed
+1 💚checkstyle0m 43sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 55sthe patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚javadoc0m 48sthe patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚findbugs2m 34sthe patch passed
_ Other Tests _
+1 💚unit0m 32stez-common in the patch passed.
+1 💚unit5m 2stez-dag in the patch passed.
+1 💚unit37m 0stez-tests in the patch passed.
+1 💚asflicense0m 41sThe patch does not generate ASF License warnings.
99m 38s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-212/10/artifact/out/Dockerfile
GITHUB PR#212
JIRA IssueTEZ-4038
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
unameLinux eeeef1bf9c31 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / e5a5578
Default JavaPrivate Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-212/10/testReport/
Max. process+thread count1195 (vs. ulimit of 5500)
modulesC: tez-common tez-dag tez-tests U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-212/10/console
versionsgit=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

abstractdog
abstractdog approved these changes Jun 10, 2022
View reviewed changes

@abstractdogabstractdog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

@abstractdog
abstractdog merged commit 86dc4fd into apache:masterJun 10, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@abstractdogabstractdogabstractdog approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@johnnyCake1@tez-yetus@abstractdog