MAPREDUCE-7486. Add fast-fail guidance when cluster storage capacity is exceeded. by deepujain · Pull Request #8388 · apache/hadoop

deepujain · 2026-03-27T06:52:04Z

Summary

This patch improves the logging path in YarnChild.reportError() when ClusterStorageCapacityExceededException is raised and fast fail is disabled. The task now emits a warning that explains why the job is not failing immediately and points users to the configuration that enables fast fail.

Change

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java: add a warning log message for the non-fast-fail capacity-exceeded path.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestYarnChild.java: extend the unit coverage to assert the warning path and preserve the existing fatal-error behavior checks.

Tests

JAVA_HOME=/opt/homebrew/opt/openjdk /opt/homebrew/bin/mvn -Dmaven.repo.local=/tmp/codex-m2 test -pl hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app -am -Dtest=TestYarnChild -DskipTests=false

JIRA

Fixes MAPREDUCE-7486

deepujain · 2026-03-27T06:52:22Z

Added a warning when cluster storage capacity is exceeded but fast fail is disabled, so users get guidance on enabling it. TestYarnChild passes locally with mvn -Dmaven.repo.local=/tmp/codex-m2 test -pl hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app -am -Dtest=TestYarnChild -DskipTests=false. Ready for CI.

hadoop-yetus · 2026-03-27T08:17:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	30m 15s		trunk passed
+1 💚	compile	0m 28s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	compile	0m 29s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	checkstyle	0m 30s		trunk passed
+1 💚	mvnsite	0m 32s		trunk passed
+1 💚	javadoc	0m 28s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	0m 26s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	0m 47s		trunk passed
+1 💚	shadedclient	20m 0s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javac	0m 19s		the patch passed
+1 💚	compile	0m 22s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	javac	0m 22s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 14s		the patch passed
+1 💚	mvnsite	0m 17s		the patch passed
+1 💚	javadoc	0m 15s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	0m 46s		the patch passed
+1 💚	shadedclient	19m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	6m 27s		hadoop-mapreduce-client-app in the patch passed.
+1 💚	asflicense	0m 20s		The patch does not generate ASF License warnings.
		84m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8388/1/artifact/out/Dockerfile
GITHUB PR	#8388
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 39900e9a0038 5.15.0-171-generic #181-Ubuntu SMP Fri Feb 6 22:44:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a8e254c`
Default Java	Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8388/1/testReport/
Max. process+thread count	718 (vs. ulimit of 10000)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8388/1/console
versions	git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

deepujain · 2026-05-27T15:59:47Z

CI is green on the current head, and this one still has not had a human review. @adoroszlai, when you have a moment, could you please take a look? Happy to make any follow-up changes.

deepujain · 2026-06-14T16:17:37Z

CI is still green on the current head, and this one is waiting on a MapReduce/YARN review. @wangda @sunilg, when you have a moment, could you please take a look? Happy to make any follow-up changes.

deepujain · 2026-06-26T15:00:15Z

CI is still green on the current head, and this one is still waiting on a MapReduce/YARN review. @wangda @sunilg, when you have a moment, could you please take a look? Happy to make any follow-up changes.

deepujain · 2026-07-06T10:36:46Z

CI is still green on the current head, and this one is still waiting on a MapReduce/YARN review. @wangda @sunilg, when you have a moment, could you please take a look? Happy to make any follow-up changes.

deepujain · 2026-07-10T03:04:13Z

CI is still green on the current head, and this one is still waiting on a MapReduce/YARN review. @wangda @sunilg, when you have a moment, could you please take a look? Happy to make any follow-up changes.

deepujain · 2026-07-20T11:33:20Z

CI is still green on the current head, and this one is still waiting on a MapReduce/YARN review as of July 20, 2026. @wangda @sunilg, when you have a moment, could you please take a look? Happy to make any follow-up changes.

deepujain · 2026-07-26T05:06:39Z

CI is still green on the current head as of July 26, 2026, and this one is still waiting on a MapReduce/YARN review. @wangda @sunilg, when you have a moment, could you please take a look? Happy to make any follow-up changes.

deepujain · 2026-07-29T06:09:26Z

CI is still green on the current head as of July 29, 2026, and this one is still waiting on a MapReduce/YARN review. @wangda @sunilg, when you have a moment, could you please take a look? Happy to make any follow-up changes.

deepujain · 2026-08-04T07:34:59Z

CI is still green on the current head as of August 4, 2026, and this one is still waiting on a MapReduce/YARN review. @wangda @sunilg, when you have a moment, could you please take a look? Happy to make any follow-up changes.

deepujain · 2026-08-07T12:42:42Z

CI is still green on the current head as of August 7, 2026, and this one is still waiting on a MapReduce/YARN review. @wangda @sunilg, when you have a moment, could you please take a look? Happy to make any follow-up changes.

deepujain · 2026-08-10T05:09:03Z

CI is still green on the current head as of August 10, 2026, and this one is still waiting on a MapReduce/YARN review. @wangda @sunilg, when you have a moment, could you please take a look? Happy to make any follow-up changes.

hadoop-yetus · 2026-09-01T17:26:42Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	28m 50s		trunk passed
+1 💚	compile	0m 31s		trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	compile	0m 31s		trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	checkstyle	0m 29s		trunk passed
+1 💚	mvnsite	0m 34s		trunk passed
+1 💚	javadoc	0m 35s		trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	javadoc	0m 33s		trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	spotbugs	1m 8s		trunk passed
+1 💚	shadedclient	18m 59s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	19m 16s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	javac	0m 17s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	javac	0m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 13s		the patch passed
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	javadoc	0m 16s		the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	spotbugs	0m 45s		the patch passed
+1 💚	shadedclient	17m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	6m 44s		hadoop-mapreduce-client-app in the patch passed.
+1 💚	asflicense	0m 22s		The patch does not generate ASF License warnings.
		80m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8388/2/artifact/out/Dockerfile
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux bf04b3b00272 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `abbc78d`
Default Java	Ubuntu-17.0.20+8-1-24.04-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12+8-1-24.04-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20+8-1-24.04-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8388/2/testReport/
Max. process+thread count	747 (vs. ulimit of 10000)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8388/2/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

deepujain · 2026-09-01T21:30:06Z

Hi @steveloughran @slfan1989 @pan3793, MAPREDUCE-7486 has been open since March. Small YarnChild change plus a test, rebased on trunk with Yetus +1. Could someone review or merge if it looks good?

hadoop-yetus · 2026-09-03T19:20:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 23s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	29m 0s		trunk passed
+1 💚	compile	0m 32s		trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	compile	0m 32s		trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	checkstyle	0m 29s		trunk passed
+1 💚	mvnsite	0m 32s		trunk passed
+1 💚	javadoc	0m 33s		trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	javadoc	0m 27s		trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	spotbugs	1m 3s		trunk passed
+1 💚	shadedclient	18m 25s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	javac	0m 18s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	javac	0m 18s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
+1 💚	checkstyle	0m 14s		the patch passed
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	javadoc	0m 15s		the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	spotbugs	0m 44s		the patch passed
+1 💚	shadedclient	17m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	7m 31s		hadoop-mapreduce-client-app in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		81m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8388/3/artifact/out/Dockerfile
GITHUB PR	#8388
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 2c6990306a52 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `80036f5`
Default Java	Ubuntu-17.0.20+8-1-24.04-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12+8-1-24.04-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20+8-1-24.04-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8388/3/testReport/
Max. process+thread count	744 (vs. ulimit of 10000)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8388/3/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

…is exceeded.

deepujain · 2026-09-04T16:01:37Z

Rebased cleanly onto current trunk at 706d4dcb. git diff --check apache/trunk...origin/MAPREDUCE-7486-enhanced-storage-logging passes. This host currently has no Java runtime or Maven, so I could not rerun the focused Maven test here; fresh Build and Jenkins checks are queued on 3e7310de. Ready for CI.

hadoop-yetus · 2026-09-04T17:15:08Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	27m 59s		trunk passed
+1 💚	compile	0m 34s		trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	compile	0m 38s		trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	checkstyle	0m 34s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 39s		trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	javadoc	0m 38s		trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	spotbugs	0m 59s		trunk passed
+1 💚	shadedclient	16m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 21s		the patch passed
+1 💚	compile	0m 20s		the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	javac	0m 20s		the patch passed
+1 💚	compile	0m 20s		the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	javac	0m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 15s		the patch passed
+1 💚	mvnsite	0m 23s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	javadoc	0m 16s		the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	spotbugs	0m 47s		the patch passed
+1 💚	shadedclient	15m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	6m 27s		hadoop-mapreduce-client-app in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		76m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8388/4/artifact/out/Dockerfile
GITHUB PR	#8388
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux c1727888fa66 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3e7310d`
Default Java	Ubuntu-17.0.20+8-1-24.04-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12+8-1-24.04-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20+8-1-24.04-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8388/4/testReport/
Max. process+thread count	716 (vs. ulimit of 10000)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8388/4/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

github-actions Bot added MapReduce trunk labels Mar 27, 2026

deepujain force-pushed the MAPREDUCE-7486-enhanced-storage-logging branch from a8e254c to da3325d Compare September 1, 2026 16:04

deepujain force-pushed the MAPREDUCE-7486-enhanced-storage-logging branch from da3325d to 80036f5 Compare September 3, 2026 17:58

MAPREDUCE-7486. Add fast-fail guidance when cluster storage capacity …

3e7310d

…is exceeded.

deepujain force-pushed the MAPREDUCE-7486-enhanced-storage-logging branch from 80036f5 to 3e7310d Compare September 4, 2026 15:58

Conversation

deepujain commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change

Tests

JIRA

Uh oh!

deepujain commented Mar 27, 2026

Uh oh!

hadoop-yetus commented Mar 27, 2026

Uh oh!

deepujain commented May 27, 2026

Uh oh!

deepujain commented Jun 14, 2026

Uh oh!

deepujain commented Jun 26, 2026

Uh oh!

deepujain commented Jul 6, 2026

Uh oh!

deepujain commented Jul 10, 2026

Uh oh!

deepujain commented Jul 20, 2026

Uh oh!

deepujain commented Jul 26, 2026

Uh oh!

deepujain commented Jul 29, 2026

Uh oh!

deepujain commented Aug 4, 2026

Uh oh!

deepujain commented Aug 7, 2026

Uh oh!

deepujain commented Aug 10, 2026

Uh oh!

hadoop-yetus commented Sep 1, 2026

Uh oh!

deepujain commented Sep 1, 2026

Uh oh!

hadoop-yetus commented Sep 3, 2026

Uh oh!

deepujain commented Sep 4, 2026

Uh oh!

hadoop-yetus commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

deepujain commented Mar 27, 2026 •

edited

Loading