Skip to content

Incorrect desired split count printed in TezSplitGrouper when run on S3 - #76

Open
Darkness303 wants to merge 1 commit into
apache:masterfrom
Darkness303:master
Open

Incorrect desired split count printed in TezSplitGrouper when run on S3#76
Darkness303 wants to merge 1 commit into
apache:masterfrom
Darkness303:master

Conversation

@Darkness303

Copy link
Copy Markdown

I was a bit confused when i saw the log print below .Through calculation and analysis ,it is found that the information is not printed correctly because the parameter assignment time is too early.

image

After modification:

image

PS: Desired splits * Desired splitLength = totalLength

@jteaglesjteagles changed the title Solved the problem of incorrect printing information.Incorrect desired split count printed in TezSplitGrouperJun 30, 2021
@jteagles

Copy link
Copy Markdown
Contributor

@rbalamohan, could you take a look at this PR related to TezSplitGrouper when run on S3

@jteaglesjteagles changed the title Incorrect desired split count printed in TezSplitGrouperIncorrect desired split count printed in TezSplitGrouper when run on S3Jun 30, 2021
@jteagles

Copy link
Copy Markdown
Contributor

@Darkness303, I have updated the PR title to better reach a reviewer. Can you confirm the title is correct and whether this occurs or S3 or another file system?

@tez-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec16m 13sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall13m 41smaster passed
+1 💚compile0m 28smaster passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚compile0m 26smaster passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚checkstyle0m 54smaster passed
+1 💚javadoc0m 36smaster passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚javadoc0m 23smaster passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗spotbugs1m 10sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs1m 8smaster passed
_ Patch Compile Tests _
+1 💚mvninstall0m 19sthe patch passed
+1 💚compile0m 19sthe patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚javac0m 19sthe patch passed
+1 💚compile0m 17sthe patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚javac0m 17sthe patch passed
+1 💚checkstyle0m 10sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 16sthe patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚javadoc0m 14sthe patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚findbugs0m 45sthe patch passed
_ Other Tests _
+1 💚unit1m 16stez-mapreduce in the patch passed.
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
38m 27s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-76/1/artifact/out/Dockerfile
GITHUB PR#76
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
unameLinux 77286d283734 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / 6d7ef20
Default JavaPrivate Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-76/1/testReport/
Max. process+thread count222 (vs. ulimit of 5500)
modulesC: tez-mapreduce U: tez-mapreduce
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-76/1/console
versionsgit=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Darkness303

Darkness303 commented Apr 12, 2022

Copy link
Copy Markdown
Author

@jteagles THS,The title is better than before, but it was running xfs file System and using ORC columnar Storage.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Darkness303@jteagles@tez-yetus