MR: Keep computed block locations in IcebergSplit.getLocations - #18036
Open
thswlsqls wants to merge 1 commit into
Open
MR: Keep computed block locations in IcebergSplit.getLocations#18036thswlsqls wants to merge 1 commit into
thswlsqls wants to merge 1 commit into
Conversation
The transient locations field and the locations == null check make getLocations() a one-time computation that is cached on the split, but the else branch reassigned locations to ANYWHERE on every later call. Calling getLocations() twice on the same split therefore returned the computed block locations first and "*" afterwards, which breaks the Hadoop InputSplit contract that repeated calls return the same value. The else branch was added in apache#1582 to avoid an NPE when getLocations() runs on a worker node, where the deserialized split has a null conf. Folding the conf null check into localityPreferred keeps that guard: with a null conf the split still resolves to ANYWHERE and never calls Util.blockLocations. Extend testLocality to assert that a second call returns the same locations. Generated-by: Claude Code
uros-b
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #18032
Summary
IcebergSplitdeclarestransient String[] locationsand checkslocations == null, so the value is meant to be computed once and cached. Theelsebranch overwrites it withANYWHEREon every later call, so two calls on an unchanged split return different values — Hadoop'sInputSplit.getLocations()contract expects them to match.elsebranch came in with #1582, which fixed an NPE whengetLocations()runs on a worker node whose deserialized split has a nullconf. The original implementation, #1192, hadif (locations == null)alone. This change restores that shape and keeps the guard by folding the null check intolocalityPreferred, so a nullconfstill resolves toANYWHEREand never callsUtil.blockLocations.getLocations()once per split, so the case rests on the contract and on the caching the class already declares.Testing done
TestIcebergInputFormats#testLocalityto callgetLocations()twice on the same split and assert both calls return the computed locations. It fails before the fix — all 6 parameterized variants return["*"]on the second call — and passes after../gradlew :iceberg-mr:check— 158 tests, 0 failures.AI Disclosure
IcebergSplit.getLocations()so a cache hit returns the previously computed block locations instead of overwriting them withANYWHERE, keep the null-confNPE guard added in MR: Fix NPE when InputSplit.getLocations is called on mappers #1582, and extendtestLocalityto cover repeated calls.