Skip to content

TEZ-4407: Misleading split info in TezSplitGrouper logs when adjusting small splits - #202

Merged
abstractdog merged 4 commits into
apache:masterfrom
zabetak:TEZ-4407
Dec 23, 2024
Merged

TEZ-4407: Misleading split info in TezSplitGrouper logs when adjusting small splits#202
abstractdog merged 4 commits into
apache:masterfrom
zabetak:TEZ-4407

Conversation

@zabetak

Copy link
Copy Markdown
Member

No description provided.

@tez-yetus

This comment was marked as outdated.

" Total length: " + totalLength +
" Original splits: " + originalSplits.size());
desiredNumSplits = newDesiredNumSplits;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about adding an else branch here with some useful log message describing whatever happens (or doesn't happen) when allSplitsHaveLocalhost=true
otherwise LGTM
cc: @rbalamohan

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a log message here c715c0b and also made a small refactoring to gather logging in one place and always log if splitLength bounds are exceeded.

Anyways the most important point in this patch is to be able to see the original desiredNumSplits in every case; especially when it is different from newDesiredNumSplits or not.

@tez-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall16m 11smaster passed
+1 💚compile0m 33smaster passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚compile0m 30smaster passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu122.04-b09
+1 💚checkstyle1m 14smaster passed
+1 💚javadoc0m 42smaster passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚javadoc0m 26smaster passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu122.04-b09
+0 🆗spotbugs1m 19sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs1m 17smaster passed
_ Patch Compile Tests _
+1 💚mvninstall0m 19sthe patch passed
+1 💚compile0m 20sthe patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚javac0m 20sthe patch passed
+1 💚compile0m 18sthe patch passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu122.04-b09
+1 💚javac0m 18sthe patch passed
+1 💚checkstyle0m 11sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 17sthe patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚javadoc0m 16sthe patch passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu122.04-b09
+1 💚findbugs0m 44sthe patch passed
_ Other Tests _
+1 💚unit1m 26stez-mapreduce in the patch passed.
+1 💚asflicense0m 15sThe patch does not generate ASF License warnings.
26m 40s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-202/2/artifact/out/Dockerfile
GITHUB PR#202
JIRA IssueTEZ-4407
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
unameLinux 53b947cf7a6f 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / 7a56e9b
Default JavaPrivate Build-1.8.0_362-8u372-gaus1-0ubuntu122.04-b09
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-gaus1-0ubuntu122.04-b09
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-202/2/testReport/
Max. process+thread count222 (vs. ulimit of 5500)
modulesC: tez-mapreduce U: tez-mapreduce
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-202/2/console
versionsgit=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@abstractdog
abstractdog self-requested a review December 22, 2024 12:28
@abstractdog
abstractdog merged commit b95defc into apache:masterDec 23, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@zabetak@tez-yetus@abstractdog