Skip to content

TEZ-4388: Asynchttpclient can cause stuck TezChild processes - #189

Merged
abstractdog merged 2 commits into
apache:masterfrom
abstractdog:TEZ-4388
Feb 22, 2022
Merged

TEZ-4388: Asynchttpclient can cause stuck TezChild processes#189
abstractdog merged 2 commits into
apache:masterfrom
abstractdog:TEZ-4388

Conversation

@abstractdog

Copy link
Copy Markdown
Contributor

No description provided.

@tez-yetus

This comment was marked as outdated.

@abstractdog

Copy link
Copy Markdown
ContributorAuthor

could you review this @rbalamohan ? I'm hoping that you're still familiar with async http client

@abstractdog

Copy link
Copy Markdown
ContributorAuthor

I've just realized this patch NULLs out the static client and forces the next async http connection to create one...I'm not sure this is the perfect solution, let me investigate this further

@abstractdogabstractdog changed the title TEZ-4388: TestSecureShuffle: TezChild processes keep running after testTEZ-4388: Asynchttpclient can cause stuck TezChild processesFeb 17, 2022
@tez-yetus

This comment was marked as outdated.

TezRuntimeShutdownHandler.addShutdownTask(new Thread(() -> {
try {
if (httpAsyncClient != null) {
httpAsyncClient.close();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should httpAsyncClient be set to "null" explicitly as well? Without it, it may create issues during container reuse. i.e httpAsyncClient will not be null, but in closed state & system would try to use it leading to exceptions.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @rbalamohan, makes sense, fixed

@rbalamohanrbalamohan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left minor comment.

@abstractdog
abstractdogforce-pushed the TEZ-4388 branch 2 times, most recently from 8ee557a to 27d3f7eCompareFebruary 22, 2022 07:42
@rbalamohan

Copy link
Copy Markdown
Contributor

LGTM. +1

@tez-yetus

This comment was marked as outdated.

@tez-yetus

This comment was marked as outdated.

@tez-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 7sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗mvndep5m 6sMaven dependency ordering for branch
+1 💚mvninstall9m 15smaster passed
+1 💚compile1m 33smaster passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚compile1m 29smaster passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚checkstyle1m 32smaster passed
+1 💚javadoc1m 36smaster passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚javadoc1m 23smaster passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗spotbugs1m 0sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs2m 42smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 10sMaven dependency ordering for patch
+1 💚mvninstall0m 50sthe patch passed
+1 💚compile0m 52sthe patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚javac0m 52sthe patch passed
+1 💚compile0m 48sthe patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚javac0m 48sthe patch passed
-0 ⚠️checkstyle0m 8stez-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 46sthe patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚javadoc0m 42sthe patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚findbugs2m 4sthe patch passed
_ Other Tests _
+1 💚unit0m 31stez-common in the patch passed.
+1 💚unit0m 34stez-runtime-internals in the patch passed.
+1 💚unit4m 33stez-runtime-library in the patch passed.
+1 💚asflicense0m 33sThe patch does not generate ASF License warnings.
40m 3s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-189/6/artifact/out/Dockerfile
GITHUB PR#189
JIRA IssueTEZ-4388
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
unameLinux 4d37eae13121 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / 5241f58
Default JavaPrivate Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
checkstylehttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-189/6/artifact/out/diff-checkstyle-tez-common.txt
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-189/6/testReport/
Max. process+thread count1099 (vs. ulimit of 5500)
modulesC: tez-common tez-runtime-internals tez-runtime-library U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-189/6/console
versionsgit=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@abstractdog@tez-yetus@rbalamohan