fix: decide libhdfs routing from the scheme as written - #5825
Merged
Conversation
`prepare_object_store_with_configs` computed `is_hdfs_scheme` on the URL returned by `normalize_object_store_url`, that is, after the `s3a`/alias rewrite to `s3://`. With `fs.comet.libhdfs.schemes=s3` -- which asks for `s3://` to go through libhdfs and says nothing about `s3a` -- an `s3a://` scan normalized to `s3://` and was then captured by the libhdfs list on the second look, handing an S3 read to `create_hdfs_object_store` with a name node of `s3://bucket`. Schemes opted in via `fs.comet.s3Compliant.schemes` normalize onto `s3` too, so they hit the same trap. Before apache#5314 the flag was computed on the original URL and the rewrite was guarded by it, so the rewrite could not flip the decision. apache#5314 replaced that inline rewrite with a call to `normalize_object_store_url` and moved the check after it. The JVM gate matches `fs.comet.libhdfs.schemes` against the scheme the user wrote (`CometScanRule.classifyRootPaths`), so it classifies such roots as object_store-native and admits the scan. The recompute therefore also desynced the planner from the executor. `normalize_object_store_url` now returns `NormalizedObjectStoreUrl`, carrying the libhdfs decision it already takes on the URL as written, so callers cannot re-derive it from the rewritten URL. Closes apache#5816
sunchao
reviewed
Sep 10, 2026
sunchao
approved these changes
Sep 10, 2026
sunchao
left a comment
Member
There was a problem hiding this comment.
Looks good. The routing decision now stays tied to the original scheme.
This was referenced Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #5816
Rationale for this change
fs.comet.libhdfs.schemesis matched against the scheme after thes3a/alias rewrite, so listings3also routess3a://reads through libhdfs.prepare_object_store_with_configsreads:normalize_object_store_urlguards its own early return withis_hdfs_scheme, so ans3a://URL whose scheme is not in the list falls through torewrite_alias_to_s3and comes back ass3://. The next line then seess3, matches the list, and selects the libhdfs backend with a name node ofs3://bucket. Schemes opted in viafs.comet.s3Compliant.schemesnormalize ontos3too, so they hit the same trap.Before #5314 the flag was computed on the original URL and the rewrite was guarded by it, so the rewrite could not flip the decision. #5314 replaced that inline rewrite with a call to
normalize_object_store_urland moved the check after it.The JVM gate matches
fs.comet.libhdfs.schemesagainst the scheme the user wrote (CometScanRule.classifyRootPaths) and so admits these scans as object_store-native, which means the recompute also desyncs the planner from the executor.Only an asymmetric list triggers it. Listing both
s3ands3a, or leaving the config unset (defaulthdfs), behaves identically before and after.What changes are included in this PR?
normalize_object_store_urlreturnsNormalizedObjectStoreUrl { url, is_hdfs }, carrying the libhdfs decision it already takes on the URL as written. Callers read the flag rather than re-deriving it from the rewritten URL, which makes the recompute structurally impossible instead of merely discouraged.prepare_object_store_with_configsdestructures that flag.planner.rstakes.url.No behavior change for any other configuration.
How are these changes tested?
Two new Rust unit tests:
test_libhdfs_routing_uses_the_scheme_as_writtenins3_blob_fs_support.rscoverss3aand an opted-inblobalias underfs.comet.libhdfs.schemes=s3(must not route to libhdfs), the legitimatefs.comet.libhdfs.schemes=s3aopt-in (must still route there, with the URL left unrewritten), and the unset default. It also pins the trap explicitly:is_hdfs_schemeon the normalized URL still returnstrue, which is why the flag is carried.test_prepare_object_store_keeps_s3a_off_libhdfs_when_only_s3_is_listedinparquet_support.rscovers the real dispatch, asserting an S3 store is built for both spellings and that an explicitly listeds3astill reaches the libhdfs backend.Verified as regression guards: with the defect reintroduced, both fail on the exact assertion, while the five pre-existing tests in the same two modules pass, confirming the gap was previously uncovered.
The full
datafusion-cometlib suite passes under both feature configurations, 337 tests with--no-default-featuresand 334 with default features (hdfs-opendal).cargo fmt --checkandcargo clippy --all-targets -- -D warningsare clean under both.Per #5816, the
isolates_backends_even_when_s3_alias_and_configs_matchtest in #5503 covers this same pair and was failing on merge withmain. This should unblock it.