Skip to content

TEZ-4617 prevent resource leaks for Closeables - #402

Merged
abstractdog merged 5 commits into
apache:masterfrom
dk2k:dk_resource_mgmt
Jan 22, 2026
Merged

TEZ-4617 prevent resource leaks for Closeables#402
abstractdog merged 5 commits into
apache:masterfrom
dk2k:dk_resource_mgmt

Conversation

@dk2k

@dk2kdk2k commented Apr 1, 2025

Copy link
Copy Markdown
Contributor

No description provided.

@tez-yetus

tez-yetus commented Apr 1, 2025

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec13m 49sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗mvndep1m 56sMaven dependency ordering for branch
+1 💚mvninstall7m 45smaster passed
+1 💚compile0m 45smaster passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚compile0m 47smaster passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu122.04-b06
+1 💚checkstyle1m 1smaster passed
+1 💚javadoc0m 40smaster passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚javadoc0m 29smaster passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu122.04-b06
+0 🆗spotbugs0m 30sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs1m 34smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 8sMaven dependency ordering for patch
+1 💚mvninstall0m 31sthe patch passed
+1 💚compile0m 29sthe patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚javac0m 29sthe patch passed
+1 💚compile0m 28sthe patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu122.04-b06
+1 💚javac0m 28sthe patch passed
+1 💚checkstyle0m 11stez-dag: The patch generated 0 new + 16 unchanged - 2 fixed = 16 total (was 18)
-0 ⚠️checkstyle0m 16stez-tests: The patch generated 3 new + 570 unchanged - 11 fixed = 573 total (was 581)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 14sthe patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚javadoc0m 13sthe patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu122.04-b06
+1 💚findbugs1m 4sthe patch passed
_ Other Tests _
+1 💚unit4m 16stez-dag in the patch passed.
+1 💚unit38m 49stez-tests in the patch passed.
+1 💚asflicense0m 21sThe patch does not generate ASF License warnings.
76m 40s
SubsystemReport/Notes
DockerClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/1/artifact/out/Dockerfile
GITHUB PR#402
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
unameLinux a4ee3ca29fc9 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/tez.sh
git revisionmaster / 7c24f25
Default JavaPrivate Build-1.8.0_442-8u442-b06us1-0ubuntu122.04-b06
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu122.04-b06
checkstylehttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/1/artifact/out/diff-checkstyle-tez-tests.txt
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/1/testReport/
Max. process+thread count1221 (vs. ulimit of 5500)
modulesC: tez-dag tez-tests U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/1/console
versionsgit=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

abstractdog
abstractdog reviewed Jan 16, 2026
View reviewed changes
Comment threadtez-tests/src/main/java/org/apache/tez/mapreduce/examples/Join.java
abstractdog
abstractdog reviewed Jan 16, 2026
View reviewed changes
Comment threadtez-dag/src/main/java/org/apache/tez/dag/app/web/AMWebController.java Outdated
abstractdog
abstractdog reviewed Jan 16, 2026
View reviewed changes
Comment threadtez-dag/src/main/java/org/apache/tez/dag/app/web/AMWebController.java
@abstractdog

abstractdog commented Jan 16, 2026

Copy link
Copy Markdown
Contributor

thanks for this patch @dk2k, left some comments

@dk2k
TEZ-4617 reacted upon comments
3fca95f
@dk2k
Merge branch 'master' into dk_resource_mgmt
323e289
# Conflicts:
#	tez-dag/src/main/java/org/apache/tez/dag/app/web/AMWebController.java
@dk2k

dk2k commented Jan 17, 2026

Copy link
Copy Markdown
ContributorAuthor

@abstractdog Please review, I fixed comments and merge conflict

abstractdog
abstractdog reviewed Jan 17, 2026
View reviewed changes
Comment threadtez-tests/src/main/java/org/apache/tez/mapreduce/examples/RandomTextWriter.java Outdated
Comment on lines +186 to +192
long numBytesToWritePerMap = conf.getLong(BYTES_PER_MAP,
1*1024*1024*1024);
1 * 1024 * 1024 * 1024);

@abstractdogabstractdogJan 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we need this line break here, it could be:

 long numBytesToWritePerMap = conf.getLong(BYTES_PER_MAP, 1*1024*1024*1024);

the same applies to:

 numBytesToWrite = conf.getLong(BYTES_PER_MAP, 1*1024*1024*1024);

@abstractdog

abstractdog commented Jan 17, 2026

Copy link
Copy Markdown
Contributor

@abstractdog Please review, I fixed comments and merge conflict

thanks @dk2k , I left 1 more minor, otherwise it looks good to me +1

@abstractdog
abstractdog self-requested a review January 17, 2026 11:13
@tez-yetus

tez-yetus commented Jan 17, 2026

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec14m 20sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗mvndep1m 46sMaven dependency ordering for branch
+1 💚mvninstall7m 53smaster passed
+1 💚compile0m 48smaster passed
+1 💚checkstyle0m 47smaster passed
+1 💚javadoc0m 39smaster passed
-1 ❌spotbugs1m 2s/branch-spotbugs-tez-dag.txttez-dag in master failed.
-1 ❌spotbugs0m 23s/branch-spotbugs-tez-tests.txttez-tests in master failed.
-0 ⚠️patch3m 13sUsed diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗mvndep0m 5sMaven dependency ordering for patch
+1 💚mvninstall0m 36sthe patch passed
+1 💚codespell0m 15sNo new issues.
+1 💚compile0m 33sthe patch passed
+1 💚javac0m 33sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
-0 ⚠️checkstyle0m 7s/results-checkstyle-tez-tests.txttez-tests: The patch generated 6 new + 573 unchanged - 8 fixed = 579 total (was 581)
+1 💚javadoc0m 18sthe patch passed
-1 ❌spotbugs0m 20s/patch-spotbugs-tez-dag.txttez-dag in the patch failed.
-1 ❌spotbugs0m 14s/patch-spotbugs-tez-tests.txttez-tests in the patch failed.
_ Other Tests _
+1 💚unit4m 58stez-dag in the patch passed.
+1 💚unit33m 39stez-tests in the patch passed.
+1 💚asflicense0m 19sThe patch does not generate ASF License warnings.
70m 8s
SubsystemReport/Notes
DockerClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/3/artifact/out/Dockerfile
GITHUB PR#402
Optional Testsdupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile
unameLinux 1ef178e8020c 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-402/src/.yetus/personality.sh
git revisionmaster / 323e289
Default JavaUbuntu-21.0.9+10-Ubuntu-124.04
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/3/testReport/
Max. process+thread count1032 (vs. ulimit of 5500)
modulesC: tez-dag tez-tests U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/3/console
versionsgit=2.43.0 maven=3.8.7 codespell=2.0.0
Powered byApache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

@abstractdog

abstractdog commented Jan 17, 2026

Copy link
Copy Markdown
Contributor

spotbugs is broken at the moment (TEZ-4669), there some checkstyle problems in the affected code, I would appreciate if you can handle those
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/3/artifact/out/results-checkstyle-tez-tests.txt

@tez-yetus

tez-yetus commented Jan 17, 2026

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 22sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗mvndep1m 43sMaven dependency ordering for branch
+1 💚mvninstall8m 27smaster passed
+1 💚compile0m 45smaster passed
+1 💚checkstyle0m 52smaster passed
+1 💚javadoc0m 36smaster passed
-1 ❌spotbugs1m 1s/branch-spotbugs-tez-dag.txttez-dag in master failed.
-1 ❌spotbugs0m 20s/branch-spotbugs-tez-tests.txttez-tests in master failed.
-0 ⚠️patch3m 13sUsed diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗mvndep0m 5sMaven dependency ordering for patch
+1 💚mvninstall0m 33sthe patch passed
+1 💚codespell0m 18sNo new issues.
+1 💚compile0m 33sthe patch passed
+1 💚javac0m 33sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
-0 ⚠️checkstyle0m 7s/results-checkstyle-tez-tests.txttez-tests: The patch generated 6 new + 573 unchanged - 8 fixed = 579 total (was 581)
+1 💚javadoc0m 16sthe patch passed
-1 ❌spotbugs0m 17s/patch-spotbugs-tez-dag.txttez-dag in the patch failed.
-1 ❌spotbugs0m 13s/patch-spotbugs-tez-tests.txttez-tests in the patch failed.
_ Other Tests _
+1 💚unit4m 45stez-dag in the patch passed.
+1 💚unit34m 32stez-tests in the patch passed.
+1 💚asflicense0m 20sThe patch does not generate ASF License warnings.
57m 15s
SubsystemReport/Notes
DockerClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/4/artifact/out/Dockerfile
GITHUB PR#402
Optional Testsdupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile
unameLinux a17a31d9e2f9 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-402@2/src/.yetus/personality.sh
git revisionmaster / 27a4828
Default JavaUbuntu-21.0.9+10-Ubuntu-124.04
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/4/testReport/
Max. process+thread count1057 (vs. ulimit of 5500)
modulesC: tez-dag tez-tests U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/4/console
versionsgit=2.43.0 maven=3.8.7 codespell=2.0.0
Powered byApache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

@tez-yetus

tez-yetus commented Jan 17, 2026

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 22sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗mvndep1m 49sMaven dependency ordering for branch
+1 💚mvninstall8m 17smaster passed
+1 💚compile0m 46smaster passed
+1 💚checkstyle0m 49smaster passed
+1 💚javadoc0m 37smaster passed
-1 ❌spotbugs1m 3s/branch-spotbugs-tez-dag.txttez-dag in master failed.
-1 ❌spotbugs0m 23s/branch-spotbugs-tez-tests.txttez-tests in master failed.
-0 ⚠️patch3m 13sUsed diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗mvndep0m 6sMaven dependency ordering for patch
+1 💚mvninstall0m 36sthe patch passed
+1 💚codespell0m 21sNo new issues.
+1 💚compile0m 33sthe patch passed
+1 💚javac0m 33sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 9sThe patch passed checkstyle in tez-dag
+1 💚checkstyle0m 9stez-tests: The patch generated 0 new + 573 unchanged - 8 fixed = 573 total (was 581)
+1 💚javadoc0m 19sthe patch passed
-1 ❌spotbugs0m 20s/patch-spotbugs-tez-dag.txttez-dag in the patch failed.
-1 ❌spotbugs0m 14s/patch-spotbugs-tez-tests.txttez-tests in the patch failed.
_ Other Tests _
+1 💚unit5m 4stez-dag in the patch passed.
+1 💚unit31m 55stez-tests in the patch passed.
+1 💚asflicense0m 19sThe patch does not generate ASF License warnings.
55m 11s
SubsystemReport/Notes
DockerClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/5/artifact/out/Dockerfile
GITHUB PR#402
Optional Testsdupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile
unameLinux 179edd0110f8 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-402/src/.yetus/personality.sh
git revisionmaster / 2c61e30
Default JavaUbuntu-21.0.9+10-Ubuntu-124.04
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/5/testReport/
Max. process+thread count1125 (vs. ulimit of 5500)
modulesC: tez-dag tez-tests U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-402/5/console
versionsgit=2.43.0 maven=3.8.7 codespell=2.0.0
Powered byApache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

@abstractdog
abstractdog merged commit 6aed5f2 into apache:masterJan 22, 2026
4 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@abstractdogabstractdogabstractdog approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dk2k@tez-yetus@abstractdog