TEZ-4729: Improve ShuffleHandler

[ayushtkn]

No description provided.

[tez-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 11s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 36s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 39s		master passed
+1 💚	compile	4m 3s		master passed
+1 💚	checkstyle	1m 9s		master passed
+1 💚	javadoc	0m 58s		master passed
+0 🆗	spotbugs	1m 41s		tez-dag in master has 537 extant spotbugs warnings.
+0 🆗	spotbugs	0m 46s		tez-plugins/tez-aux-services in master has 5 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 52s		the patch passed
+1 💚	codespell	1m 36s		No new issues.
+1 💚	compile	4m 4s		the patch passed
+1 💚	javac	4m 4s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 40s	/results-checkstyle-tez-dag.txt	tez-dag: The patch generated 1 new + 187 unchanged - 1 fixed = 188 total (was 188)
+1 💚	javadoc	0m 57s		the patch passed
+1 💚	spotbugs	2m 54s		the patch passed
			_ Other Tests _
+1 💚	unit	74m 48s		root in the patch passed.
+1 💚	asflicense	0m 59s		The patch does not generate ASF License warnings.
		106m 30s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-512/5/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux dc444b35b4a8 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / 61cc636
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-512/5/testReport/
Max. process+thread count	2075 (vs. ulimit of 5500)
modules	C: tez-dag tez-plugins/tez-aux-services U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-512/5/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

[abstractdog]

not sure if we need a separate secret manager, as the sessionToken is provided for jobTokenSecretManager already:

tez/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java

Line 528 in 61cc636

jobTokenSecretManager.addTokenForJob(

[ayushtkn]

JobTokenSecretManager has two different key paths:

1. computeHash(byte[] msg) (instance method) — uses mac.doFinal(msg) with the MAC master key (random at construction)
2. retrieveTokenSecret(String jobId) — returns the per-job key from the currentJobTokens map (populated by addTokenForJob)

The client-side AsyncHttpConnection.computeEncHash() calls:
encHash = SecureShuffleUtils.hashFromString(msgToEncode, jobTokenSecretMgr);
// → mgr.computeHash(msg) ← uses MAC master key, NOT the per-job token from the map

The server-side verifyRequest calls:
SecretKey tokenSecret = getSecretManager().retrieveTokenSecret(appid); // ← uses the map
SecureShuffleUtils.verifyReply(urlHashStr, enc_str, tokenSecret);

So addTokenForJob populates the map (server path), but the client hash computation uses the MAC field (instance method). They're completely separate code paths. The MAC must be initialized
with the token password for the client-side hash to match what the server verifies.

It was leading to test failure here: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-512/3/testReport/org.apache.tez.auxservices/TestShuffleHandlerJobs/testOrderedWordCount/

Did I decode it wrong?

[abstractdog]

that's right, thanks for clarifying, I tend to forget SecretManager internals
so the code comment indeed explains the situation well: client side uses the instance method, so we must initialize the instance MAC with the sessionToken, which is used from the map on the server/shufflehandler side