Skip to content

measured input stream - #262

Open
abstractdog wants to merge 1 commit into
apache:masterfrom
abstractdog:TEZ-4466
Open

measured input stream#262
abstractdog wants to merge 1 commit into
apache:masterfrom
abstractdog:TEZ-4466

Conversation

@abstractdog

Copy link
Copy Markdown
Contributor

No description provided.

@tez-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec32m 22sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall15m 26smaster passed
+1 💚compile0m 36smaster passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚compile0m 34smaster passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚checkstyle1m 6smaster passed
+1 💚javadoc0m 41smaster passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚javadoc0m 31smaster passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+0 🆗spotbugs1m 21sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs1m 18smaster passed
_ Patch Compile Tests _
+1 💚mvninstall0m 21sthe patch passed
+1 💚compile0m 23sthe patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚javac0m 23sthe patch passed
+1 💚compile0m 21sthe patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚javac0m 21sthe patch passed
+1 💚checkstyle0m 16sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 18sthe patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚javadoc0m 17sthe patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚findbugs0m 54sthe patch passed
_ Other Tests _
+1 💚unit5m 40stez-runtime-library in the patch passed.
+1 💚asflicense0m 14sThe patch does not generate ASF License warnings.
62m 13s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-262/1/artifact/out/Dockerfile
GITHUB PR#262
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
unameLinux fd729b9c5142 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / 39e5a8e
Default JavaPrivate Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-262/1/testReport/
Max. process+thread count2100 (vs. ulimit of 5500)
modulesC: tez-runtime-library U: tez-runtime-library
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-262/1/console
versionsgit=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

protected void setupConnectionInternal(String host, Collection<InputAttemptIdentifier> attempts)
throws IOException, InterruptedException {
input = httpConnection.getInputStream();
input = new MeasuredDataInputStream(httpConnection.getInputStream());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-1 for making it the default. Note that System.currentTimeMillis may turn out to be expensive when number of calls are higher. Especially in cloud nodes where hypervisors are involved.

Would suggest to make it optional.

@abstractdogabstractdogJan 16, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the input @rbalamohan, yes, absolutely, this is a work-in-progress patch, I was thinking about making it configurable too...working on that
I hope the basic idea makes sense to you: counting the time spent in InputStream.read to reflect the actual time spent with network traffic

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't realize that it was WIP patch. sure, +1 on the idea.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@abstractdog@tez-yetus@rbalamohan