Skip to content

TEZ-4014: Allow DAGAppMaster to read configuration from plaintext (1/3) - #408

Merged
abstractdog merged 2 commits into
apache:masterfrom
abstractdog:TEZ-4014
Sep 2, 2025
Merged

TEZ-4014: Allow DAGAppMaster to read configuration from plaintext (1/3)#408
abstractdog merged 2 commits into
apache:masterfrom
abstractdog:TEZ-4014

Conversation

@abstractdog

Copy link
Copy Markdown
Contributor

No description provided.

@tez-yetus

This comment was marked as outdated.

@tez-yetus

This comment was marked as outdated.

@okuminokumin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@abstractdog

Copy link
Copy Markdown
ContributorAuthor

rebased on top of current master
thanks a lot @okumin for your review so far!
@ayushtkn , wdyt, could we go ahead with this?

@tez-yetus

This comment was marked as outdated.

Comment on lines +308 to +328
assertEquals(conf.get("tez.lib.uris"), "tez.tar.gz");
}

@Test(timeout = 5000)
public void testPluginsDescriptorFromJSON() throws IOException {
InputStream is = ClassLoader.getSystemResourceAsStream(TezConstants.SERVICE_PLUGINS_DESCRIPTOR_JSON);
ServicePluginsDescriptor spd = TezClientUtils.createPluginsDescriptorFromJSON(is);
TaskSchedulerDescriptor tsd = spd.getTaskSchedulerDescriptors()[0];
ContainerLauncherDescriptor cld = spd.getContainerLauncherDescriptors()[0];
TaskCommunicatorDescriptor tcd = spd.getTaskCommunicatorDescriptors()[0];

assertFalse(spd.areContainersEnabled());
assertTrue(spd.isUberEnabled());
assertEquals(tsd.getClassName(), "testScheduler0_class");
assertEquals(tsd.getEntityName(), "testScheduler0");
assertEquals(cld.getClassName(), "testLauncher0_class");
assertEquals(cld.getEntityName(), "testLauncher0");
assertEquals(tcd.getClassName(), "testComm0_class");
assertEquals(tcd.getEntityName(), "testComm0");
assertEquals(tcd.getUserPayload().getVersion(), 1);
assertArrayEquals(tcd.getUserPayload().deepCopyAsArray(), new byte[] {0, 0, 0, 1});

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you check the order of arguments in assertEquals, the expected in first argument actual is second, I think it is opposite here

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, let me fix

}
}

public static Configuration readTezConfigurationXml(InputStream is) throws IOException {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't throw IOE

return configuration;
}

public static ConfigurationProto loadConfProtoFromText() throws IOException {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which method call this? this shows as unused. Removing it doesn't make any test fail either

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, this is supposed to be added by TEZ-4007 or TEZ-4008, removing it now

@tez-yetus

This comment was marked as outdated.

@tez-yetus

This comment was marked as outdated.

@tez-yetus

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 16sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+0 🆗xmllint0m 0sxmllint was not available.
+0 🆗jsonlint0m 0sjsonlint was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 3 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep2m 31sMaven dependency ordering for branch
+1 💚mvninstall9m 33smaster passed
+1 💚compile3m 27smaster passed
+1 💚checkstyle1m 49smaster passed
+1 💚javadoc2m 34smaster passed
+0 🆗spotbugs2m 9stez-api in master has 610 extant spotbugs warnings.
+0 🆗spotbugs0m 45stez-common in master has 13 extant spotbugs warnings.
+0 🆗spotbugs6m 53sroot in master has 2066 extant spotbugs warnings.
_ Patch Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for patch
+1 💚mvninstall5m 5sthe patch passed
+1 💚codespell0m 50sNo new issues.
+1 💚compile3m 26sthe patch passed
+1 💚javac3m 26sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 26stez-api: The patch generated 0 new + 63 unchanged - 4 fixed = 63 total (was 67)
+1 💚checkstyle0m 24sThe patch passed checkstyle in tez-common
+1 💚checkstyle0m 41sroot: The patch generated 0 new + 127 unchanged - 4 fixed = 127 total (was 131)
+1 💚javadoc2m 26sthe patch passed
+1 💚spotbugs9m 53sthe patch passed
_ Other Tests _
+1 💚unit2m 28stez-api in the patch passed.
+1 💚unit0m 43stez-common in the patch passed.
+1 💚unit71m 57sroot in the patch passed.
+1 💚asflicense1m 20sThe patch does not generate ASF License warnings.
133m 14s
SubsystemReport/Notes
DockerClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-408/7/artifact/out/Dockerfile
GITHUB PR#408
Optional Testsdupname asflicense javac javadoc unit codespell detsecrets xmllint compile spotbugs checkstyle jsonlint
unameLinux 1f350f03d919 5.15.0-152-generic #162-Ubuntu SMP Wed Jul 23 09:48:42 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-408/src/.yetus/personality.sh
git revisionmaster / 1c21f1f
Default JavaUbuntu-21.0.8+9-Ubuntu-0ubuntu124.04.1
Test Resultshttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-408/7/testReport/
Max. process+thread count1322 (vs. ulimit of 5500)
modulesC: tez-api tez-common . U: .
Console outputhttps://ci-hadoop.apache.org/job/tez-multibranch/job/PR-408/7/console
versionsgit=2.43.0 maven=3.8.7 spotbugs=4.9.3 codespell=2.0.0
Powered byApache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

@ayushtknayushtkn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@abstractdog
abstractdog merged commit 6e357a0 into apache:masterSep 2, 2025
4 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@abstractdog@tez-yetus@okumin@ayushtkn