TEZ-4737: Session DAG submission fails with FileAlreadyExistsException when the session outlives the TezClient

[deniskuzZ]

The serialized DAG plan file (tez-dag.pb) written for plans exceeding the IPC limit is named by a per-TezClient counter and never deleted. When the session and its staging directory outlive the TezClient instances, a new client generation recomputes the same file name and fails with FileAlreadyExistsException, permanently blocking oversized submissions on that session.

Create the plan file with overwrite=true (a file at that path can only be an already consumed leftover), and delete it in the AM once the submitDAG request has been parsed.

Reproduced by the new unit test: TestTezClient#testSessionLargeDAGPlanWithLeftoverPlanFile:

org.apache.hadoop.fs.FileAlreadyExistsException: File already exists: file:/.../tez-api/target/tmp/org.apache.tez.client.TestTezClient/.tez/application_0_0001/tez-dag.pb1 at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:622) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:660) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:692) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:780) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:759) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1234) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1211) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1092) at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:695) at org.apache.tez.client.TezClient.submitDAG(TezClient.java:614) at org.apache.tez.client.TestTezClient.testSessionLargeDAGPlanWithLeftoverPlanFile(TestTezClient.java:328) 

[tez-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 49s		master passed
+1 💚	compile	3m 57s		master passed
+1 💚	checkstyle	1m 10s		master passed
+1 💚	javadoc	1m 10s		master passed
+0 🆗	spotbugs	1m 24s		tez-api in master has 92 extant spotbugs warnings.
+0 🆗	spotbugs	1m 43s		tez-dag in master has 537 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 9s		Maven dependency ordering for patch
+1 💚	mvninstall	4m 1s		the patch passed
+1 💚	codespell	1m 48s		No new issues.
+1 💚	compile	3m 57s		the patch passed
+1 💚	javac	3m 57s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 4s		the patch passed
+1 💚	javadoc	1m 9s		the patch passed
+1 💚	spotbugs	3m 32s		the patch passed
			_ Other Tests _
+1 💚	unit	72m 43s		root in the patch passed.
+1 💚	asflicense	0m 59s		The patch does not generate ASF License warnings.
		106m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-523/1/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux 84a506d533d2 5.15.0-185-generic #195-Ubuntu SMP Fri Jun 19 17:11:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / 4bccf3a
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-523/1/testReport/
Max. process+thread count	1477 (vs. ulimit of 5500)
modules	C: tez-api tez-dag U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-523/1/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

[abstractdog]

thanks for the fix @deniskuzZ so far, left minor comments

[abstractdog]

LGTM

[deniskuzZ]

thank you, @abstractdog !