[TEZ-4019] Modify Tez shuffle handler to use AuxiliaryLocalPathHandler instead of LocalDirAllocator

[jteagles]

No description provided.

[jteagles]

@abstractdog , do you know who might be good to review this ShuffleHandler change? Handles cases where drives go bad or are added back after being fixed

[tez-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	30m 35s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ master Compile Tests _
+1 💚	mvninstall	19m 0s	master passed
+1 💚	compile	0m 29s	master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚	compile	0m 27s	master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚	checkstyle	1m 30s	master passed
+1 💚	javadoc	0m 35s	master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚	javadoc	0m 21s	master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+0 🆗	spotbugs	1m 16s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 14s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s	the patch passed
+1 💚	compile	0m 18s	the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚	javac	0m 18s	the patch passed
+1 💚	compile	0m 16s	the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚	javac	0m 16s	the patch passed
+1 💚	checkstyle	0m 9s	tez-plugins/tez-aux-services: The patch generated 0 new + 65 unchanged - 1 fixed = 65 total (was 66)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 9s	the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚	javadoc	0m 9s	the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚	findbugs	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 55s	tez-aux-services in the patch passed.
+1 💚	asflicense	0m 16s	The patch does not generate ASF License warnings.
		60m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-355/3/artifact/out/Dockerfile
GITHUB PR	#355
JIRA Issue	TEZ-4019
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 49defce6224b 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / 9a6757b
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-355/3/testReport/
Max. process+thread count	1498 (vs. ulimit of 5500)
modules	C: tez-plugins/tez-aux-services U: tez-plugins/tez-aux-services
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-355/3/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[jteagles]

Checkstyle issues have been addressed.

[abstractdog]

@abstractdog , do you know who might be good to review this ShuffleHandler change? Handles cases where drives go bad or are added back after being fixed

thanks for this patch @jteagles!
basically I'm fine to switch from LocalDirAllocator to AuxiliaryLocalPathHandler, by looking at YARN-7244 I feel that AuxiliaryLocalPathHandler is kind of the successor of the LocalDirAllocator

what's the default used in hadoop nowadays?

[jteagles]

@abstractdog , do you know who might be good to review this ShuffleHandler change? Handles cases where drives go bad or are added back after being fixed

thanks for this patch @jteagles! basically I'm fine to switch from LocalDirAllocator to AuxiliaryLocalPathHandler, by looking at YARN-7244 I feel that AuxiliaryLocalPathHandler is kind of the successor of the LocalDirAllocator

what's the default used in hadoop nowadays?

AuxiliaryLocalPathHandler is the default for some time in Hadoop and it gives MapReduce shuffle handler the same insight into the disk that the node manager has, meaning the shuffle handler will have the same view of disks as the jobs launched

[abstractdog]

LGTM