HDFS-17970. Exclude failed EC checksum target from reconstruction sources

[zouhuajian]

Description of PR

This PR fixes HDFS-17970.

During striped block checksum reconstruction, the DataNode that failed the original checksum request could still be selected as a reconstruction source. This may cause reconstruction to read from the same failed target and prevent the file checksum from being calculated.

This change:

• excludes every location belonging to the failed checksum target from the reconstruction source candidates;
• keeps source block indices aligned with their corresponding DataNode locations during source selection;
• preserves valid non-target replicas when selecting the required sources;
• adds a deterministic regression test that triggers the checksum reconstruction fallback with an invalid child block token.

Contains content generated by OpenAI Codex.

How was this patch tested?

• Ran the focused regression test: TestFileChecksum#testStripedFileChecksumReconstructionExcludesFailedTarget (2 tests passed).
• Ran the complete TestFileChecksum test class (72 tests passed).
• Ran git diff --check.

For code changes:

• Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation? Note: Automated CI testing doesn't cover all cases so manual testing with cloud storage is still required.
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

The unchecked object storage, dependency, and license-file items are not applicable to this change.

AI Tooling

If an AI tool was used:

• The PR includes the phrase "Contains content generated by OpenAI Codex."
• My use of AI contributions follows the ASF legal policy https://www.apache.org/legal/generative-tooling.html

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	19m 46s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	47m 54s		trunk passed
+1 💚	compile	1m 50s		trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	compile	1m 49s		trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	checkstyle	1m 50s		trunk passed
+1 💚	mvnsite	1m 59s		trunk passed
+1 💚	javadoc	1m 30s		trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	javadoc	1m 30s		trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	spotbugs	4m 23s		trunk passed
+1 💚	shadedclient	42m 50s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 24s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	javac	1m 17s		the patch passed
+1 💚	compile	1m 22s		the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	javac	1m 22s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 17s		the patch passed
+1 💚	mvnsite	1m 32s		the patch passed
+1 💚	javadoc	0m 59s		the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu
+1 💚	javadoc	1m 3s		the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu
+1 💚	spotbugs	4m 5s		the patch passed
+1 💚	shadedclient	36m 36s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	264m 21s		hadoop-hdfs in the patch passed.
-1 ❌	asflicense	0m 55s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		438m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8709/1/artifact/out/Dockerfile
GITHUB PR	#8709
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux ae8fd63dc7b3 5.15.0-185-generic #195-Ubuntu SMP Fri Jun 19 17:11:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 2339461
Default Java	Ubuntu-17.0.20+8-1-24.04-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12+8-1-24.04-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20+8-1-24.04-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8709/1/testReport/
Max. process+thread count	2468 (vs. ulimit of 10000)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8709/1/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

[zouhuajian]

Hi @Hexiaoqiao , could you please take a look when you have time? Thanks!

[qijiale76]

Hi @zouhuajian , thanks for this fix. We are evaluating HDFS-17970 for production and have two questions:

1. Have you deployed this patch in production? If so, have you observed any issues with EC checksum reconstruction, particularly during DataNode decommissioning or when duplicate internal-block replicas are present?

2. Have you considered duplicate replicas being counted as independent sources during reconstruction fallback?

We reproduced the following case on trunk 3f440d242312 plus this PR’s commit ac4a2ce42984, using JDK 17.

For RS-6-3, reconstructing index 0, the source indices are ordered as the patch allows:

[1, 2, 3, 4, 5, 6, 1, 6]

If opening the first index-6 reader fails, StripedReader.initReaders() selects [1, 2, 3, 4, 5, 1] and stops after six successful readers. The healthy index-6 replica at the end is never tried.

getInputBuffers() then places both index-1 buffers into the same slot, leaving only five distinct inputs. The Java RS decoder throws:

No enough valid inputs are provided, not recoverable

Our targeted test uses the actual trunk StripedReader selection logic and Java RS decoder, with mocked readers to inject the connection failure. The control case with all primary sources available decodes successfully. The PR’s existing MD5 and Composite CRC regression tests also pass. We have not reproduced this with real DataNode failures in a cluster.

The underlying reader behavior predates this PR. However, retaining duplicate replicas as fallbacks appears to require counting distinct internal-block indices, both during initialization and when replacing failed or slow reads.

Does this match your understanding? Have you encountered this case, or is there an existing issue or follow-up patch? Please let us know if there is an invariant we have missed that would prevent it in practice.