Skip to content

TEZ-4397: Open Tez Input splits asynchronously - #263

Merged
abstractdog merged 1 commit into
apache:masterfrom
shameersss1:TEZ-4397
Feb 13, 2023
Merged

TEZ-4397: Open Tez Input splits asynchronously#263
abstractdog merged 1 commit into
apache:masterfrom
shameersss1:TEZ-4397

Conversation

@shameersss1

@shameersss1shameersss1 commented Jan 31, 2023

Copy link
Copy Markdown
Contributor

Tez input splits can be opened asynchronously. This will reduce the amount of time spent for s3 to prepare the connection and opening the object.

The changes are taken from the original PR: #195 with some additional changes

  1. Disable the feature by default (to avoid regression in dependent engines)
  2. Minor refactoring

@tez-yetus

This comment was marked as outdated.

@tez-yetus

This comment was marked as outdated.

@tez-yetus

This comment was marked as outdated.

@tez-yetus

This comment was marked as outdated.

@abstractdogabstractdog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very minor nits, otherwise it looks good
precommit is stable now, once this passes, I'm ready to merge

@tez-yetus

This comment was marked as outdated.

@tez-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 38sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall15m 29smaster passed
+1 💚compile0m 32smaster passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04
+1 💚compile0m 31smaster passed with JDK Private Build-1.8.0_352-8u352-ga-1~22.04-b08
+1 💚checkstyle1m 1smaster passed
+1 💚javadoc0m 39smaster passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04
+1 💚javadoc0m 28smaster passed with JDK Private Build-1.8.0_352-8u352-ga-1~22.04-b08
+0 🆗spotbugs1m 11sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs1m 8smaster passed
_ Patch Compile Tests _
+1 💚mvninstall0m 18sthe patch passed
+1 💚compile0m 18sthe patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04
+1 💚javac0m 18sthe patch passed
+1 💚compile0m 17sthe patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~22.04-b08
+1 💚javac0m 17sthe patch passed
+1 💚checkstyle0m 12stez-mapreduce: The patch generated 0 new + 32 unchanged - 1 fixed = 32 total (was 33)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 15sthe patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04
+1 💚javadoc0m 14sthe patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~22.04-b08
+1 💚findbugs0m 41sthe patch passed
_ Other Tests _
+1 💚unit1m 27stez-mapreduce in the patch passed.
+1 💚asflicense0m 13sThe patch does not generate ASF License warnings.
25m 21s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-263/6/artifact/out/Dockerfile
GITHUB PR#263
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
unameLinux 2257340f0786 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / e236f51
Default JavaPrivate Build-1.8.0_352-8u352-ga-1~22.04-b08
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~22.04-b08
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-263/6/testReport/
Max. process+thread count238 (vs. ulimit of 5500)
modulesC: tez-mapreduce U: tez-mapreduce
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-263/6/console
versionsgit=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@abstractdogabstractdog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@abstractdog
abstractdog merged commit e3e91a1 into apache:masterFeb 13, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@shameersss1@tez-yetus@abstractdog