Skip to content

TEZ-4650: Remove useless logic from AM plugin management: processSchedulerDescriptors - #433

Merged
abstractdog merged 2 commits into
apache:masterfrom
abstractdog:TEZ-4650
Oct 13, 2025
Merged

TEZ-4650: Remove useless logic from AM plugin management: processSchedulerDescriptors#433
abstractdog merged 2 commits into
apache:masterfrom
abstractdog:TEZ-4650

Conversation

@abstractdog

@abstractdogabstractdog commented Sep 22, 2025

Copy link
Copy Markdown
Contributor

See: jira description

Additional local testing:

  1. removed the method
  2. rebuilt tez 0.10.5
  3. ran hive on tez unit test locally:
mvn install -Dtest.output.overwrite=true -Pitests -nsu -pl itests/qtest -Dtest=TestMiniTezCliDriver -Dqfile=acid_vectorization_original_tez.q

Also tez has unit test coverage in this area: in case it's green, this patch is safe.

@tez-yetus

This comment was marked as outdated.

@tez-yetus

This comment was marked as outdated.

@abstractdog

abstractdog commented Sep 23, 2025

Copy link
Copy Markdown
ContributorAuthor

apparently, there is a failing case:

[ERROR] TestDAGAppMaster.testParseAllPluginsOnlyCustomSpecified:275->verifyDescAndMap:413 expected:<2> but was:<1>

I'm going to investigate this: it seems like this unit test asserts that the yarn plugin descriptor is hacked to the entities even when enableYarn:false, so the application expects it to be present in non-local mode, even when containersEnabled=false...which still seems quite hacky, need to understand what use case this unit test covers

EDIT: in the only 1 known custom plugin use-case, which is Hive LLAP, Hive sets the descriptor to enableYarn:true:
https://github.com/apache/hive/blob/c338904b80d4dc6df08981738b69fa136bf6c624/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L341
because this calls enableContainers=true case:

returnnewServicePluginsDescriptor(true, enableUber, null, null, null);

so, this unit test doesn't cover a known use-case
I'm about to remove it or change the asserted values with an explanation in a code comment

@tez-yetus

This comment was marked as outdated.

@tez-yetus

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 39sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 💚mvninstall13m 49smaster passed
+1 💚compile0m 47smaster passed
+1 💚checkstyle1m 19smaster passed
+1 💚javadoc0m 37smaster passed
+0 🆗spotbugs2m 35stez-dag in master has 785 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall0m 31sthe patch passed
+1 💚codespell0m 26sNo new issues.
+1 💚compile0m 30sthe patch passed
+1 💚javac0m 30sthe patch passed
+1 💚blanks0m 1sThe patch has no blanks issues.
+1 💚checkstyle0m 16sthe patch passed
+1 💚javadoc0m 14sthe patch passed
+1 💚spotbugs1m 45sthe patch passed
_ Other Tests _
+1 💚unit5m 11stez-dag in the patch passed.
+1 💚asflicense0m 14sThe patch does not generate ASF License warnings.
30m 9s
SubsystemReport/Notes
DockerClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-433/4/artifact/out/Dockerfile
GITHUB PR#433
Optional Testsdupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile
unameLinux eda0a00b806e 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-433/src/.yetus/personality.sh
git revisionmaster / d5bfce2
Default JavaUbuntu-21.0.8+9-Ubuntu-0ubuntu124.04.1
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-433/4/testReport/
Max. process+thread count221 (vs. ulimit of 5500)
modulesC: tez-dag U: tez-dag
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-433/4/console
versionsgit=2.43.0 maven=3.8.7 spotbugs=4.9.3 codespell=2.0.0
Powered byApache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

@ayushtknayushtkn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@abstractdog
abstractdog merged commit 8be8008 into apache:masterOct 13, 2025
4 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@abstractdog@tez-yetus@ayushtkn