Uh oh!
There was an error while loading. Please reload this page.
[Fix](nereids) Freeze sortedPartitionRanges in SelectedPartitions to prevent TOCTOU NPE during partition pruning - #65659
Merged
morrySnow merged 5 commits intoJul 20, 2026
Conversation
bosswnx
requested review from
924060929, englefly, morrySnow and starocean999
as code ownersJuly 15, 2026 10:45
hello-stephen
commented
Jul 15, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
bosswnx
commented
Jul 15, 2026
ContributorAuthor
/review |
bosswnx
commented
Jul 15, 2026
ContributorAuthor
run buildall |
bosswnxforce-pushed
the
fix/freeze-sorted-partition-ranges-in-selected-partitions
branch
from
July 15, 2026 11:23
a1ec31a to
28d79aaComparebosswnx
commented
Jul 15, 2026
ContributorAuthor
run buildall |
1 similar comment
bosswnx
commented
Jul 15, 2026
ContributorAuthor
run buildall |
bosswnx
commented
Jul 15, 2026
ContributorAuthor
run buildall |
hello-stephen
commented
Jul 15, 2026
Contributor
TPC-H: Total hot run time: 29730 ms |
hello-stephen
commented
Jul 15, 2026
Contributor
TPC-DS: Total hot run time: 177670 ms |
hello-stephen
commented
Jul 15, 2026
Contributor
ClickBench: Total hot run time: 25.23 s |
morrySnow
commented
Jul 15, 2026
Contributor
/review |
Uh oh!
There was an error while loading. Please reload this page.
hello-stephen
commented
Jul 15, 2026
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
There was a problem hiding this comment.
Automated review completed for PR 65659.
I found two issues that should be addressed before merge:
- The new null guard in file-scan partition pruning silently drops a pruner-returned partition instead of failing on an invariant breach.
- External sorted partition ranges are now built eagerly before the binary-search session-variable gate, including preload paths that discard the result.
Checkpoint summary:
- Goal/test: the PR moves the sorted partition range snapshot into
SelectedPartitionsto avoid cache/snapshot divergence, and adds a focused unit test. - Scope/parallel paths: the change is small, but it touches shared external-table initialization and the Hudi-specific initializer.
- Concurrency/lifecycle: the original TOCTOU shape is addressed by freezing the map/range pair, but invariant failure handling should not silently continue.
- Configuration/performance:
enable_binary_search_filtering_partitionsno longer avoids range construction cost for external scans/preload. - Compatibility/persistence/FE-BE/data writes: no storage format, persistence, or FE-BE protocol compatibility issue found.
- Test coverage: no tests were run in this review environment per the review prompt; the added test was statically reviewed.
- User focus: no additional user-provided focus was supplied.
Round 2 convergence completed: both convergence subagents returned NO_NEW_VALUABLE_FINDINGS against the current comment set.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…fix TOCTOU NPE PruneFileScanPartition froze nameToPartitionItem in SelectedPartitions at plan construction (T1) but re-read sortedPartitionRanges from the cache at pruning time (T2). A concurrent ADD/DROP PARTITION between T1 and T2 made the two snapshots diverge: binary search returned partitions present only in the new snapshot, nameToPartitionItem.get() returned null for them, and ImmutableMap.copyOf() threw NullPointerException. Fix: freeze both views from a single snapshot. - SelectedPartitions now carries an Optional<SortedPartitionRanges> field. - HMSExternalTable.initSelectedPartitions reads the cached HivePartitionValues once and freezes both the partition map and the cached sorted ranges together (reuses cache, no just-in-time rebuild). - Hudi has no cached ranges, so PruneFileScanPartition builds them lazily from the frozen map only when binary search filtering is enabled. - A missing partition in the lookup loop is now an invariant failure (Preconditions.checkState) instead of being silently skipped, which previously produced a partial scan over fewer partitions.
bosswnx
commented
Jul 16, 2026
ContributorAuthor
/review |
bosswnx
commented
Jul 16, 2026
ContributorAuthor
run buildall |
hello-stephen
commented
Jul 16, 2026
Contributor
TPC-H: Total hot run time: 29567 ms |
hello-stephen
commented
Jul 16, 2026
Contributor
TPC-DS: Total hot run time: 177139 ms |
hello-stephen
commented
Jul 16, 2026
Contributor
ClickBench: Total hot run time: 25.02 s |
Uh oh!
There was an error while loading. Please reload this page.
github-actionsBot
pushed a commit
that referenced
this pull request
Jul 20, 2026
…prevent TOCTOU NPE during partition pruning (#65659) ### What problem does this PR solve? Issue Number: #64800 Related PR: #58877 Problem Summary: Fix a TOCTOU (Time-of-Check Time-of-Use) race condition that causes `NullPointerException` during partition pruning on external tables. **Root cause:** In `PruneFileScanPartition.pruneExternalPartitions()`: 1. `nameToPartitionItem` — frozen at T1 inside `LogicalFileScan.SelectedPartitions` when the plan node is constructed (via `initSelectedPartitions()`) 2. `sortedPartitionRanges` — re-read from the `HivePartitionValues` cache at T2 when the pruning rule executes (via `externalTable.getSortedPartitionRanges()`) If the cache is refreshed between T1 and T2 (e.g. concurrent `ALTER TABLE ADD/DROP PARTITION`), the two snapshots diverge. `binarySearchFiltering` uses the new snapshot to decide which partitions match the predicate, but the caller looks them up in the old snapshot: ```java for (String name : prunedPartitions) { selectedPartitionItems.put(name, nameToPartitionItem.get(name)); // nameToPartitionItem.get(name) returns null for partitions that were added after T1 } // => ImmutableMap.copyOf() throws NPE: "null value in entry: dt=2026-06-22=null" ``` **Concrete example:** A Hive table has 3 partitions `dt=2026-06-20/21/23`. Session A runs `SELECT * FROM t WHERE dt='2026-06-22'`: ``` T1 BindRelation: LogicalFileScan freezes nameToPartitionItem from cache → {2026-06-20, 2026-06-21, 2026-06-23} (no 2026-06-22) [Session B runs ALTER TABLE ADD PARTITION (dt='2026-06-22')] [cache is refreshed → now has 4 partitions including 2026-06-22] T2 PruneFileScanPartition: re-reads sortedPartitionRanges from cache → {2026-06-20, 2026-06-21, 2026-06-22, 2026-06-23} (new snapshot) binarySearchFiltering matches dt=2026-06-22 → returns "dt=2026-06-22" nameToPartitionItem.get("dt=2026-06-22") → null (old snapshot has no such key) → NPE: "null value in entry: dt=2026-06-22=null" ``` **Fix:** freeze both views from a single snapshot so T2 never re-reads the cache. - `SelectedPartitions` now carries an `Optional<SortedPartitionRanges>` field. - `HMSExternalTable.initSelectedPartitions` reads the cached `HivePartitionValues` once and freezes both the partition map and the cached sorted ranges together (reuses the cache, no just-in-time rebuild). - Hudi has no cached ranges, so `PruneFileScanPartition` builds them lazily from the frozen map only when binary search filtering is enabled. - A missing partition in the lookup loop is now an invariant failure (`Preconditions.checkState`) instead of being silently skipped, which previously produced a partial scan over fewer partitions. ### Release note Fix `NullPointerException` in partition pruning when external table partitions are modified concurrently during query optimization (TOCTOU race in binary search partition filtering).
morningman added a commit
that referenced
this pull request
Jul 21, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 21, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 22, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 22, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 23, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 23, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
3 tasks
morningman added a commit
that referenced
this pull request
Jul 24, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 24, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 24, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 25, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 26, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 27, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 27, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 27, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 28, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 28, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 29, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
morningman added a commit
that referenced
this pull request
Jul 29, 2026
…rived views (#65829) ### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
wyxxxcat pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
…prevent TOCTOU NPE during partition pruning (apache#65659) ### What problem does this PR solve? Issue Number: apache#64800 Related PR: apache#58877 Problem Summary: Fix a TOCTOU (Time-of-Check Time-of-Use) race condition that causes `NullPointerException` during partition pruning on external tables. **Root cause:** In `PruneFileScanPartition.pruneExternalPartitions()`: 1. `nameToPartitionItem` — frozen at T1 inside `LogicalFileScan.SelectedPartitions` when the plan node is constructed (via `initSelectedPartitions()`) 2. `sortedPartitionRanges` — re-read from the `HivePartitionValues` cache at T2 when the pruning rule executes (via `externalTable.getSortedPartitionRanges()`) If the cache is refreshed between T1 and T2 (e.g. concurrent `ALTER TABLE ADD/DROP PARTITION`), the two snapshots diverge. `binarySearchFiltering` uses the new snapshot to decide which partitions match the predicate, but the caller looks them up in the old snapshot: ```java for (String name : prunedPartitions) { selectedPartitionItems.put(name, nameToPartitionItem.get(name)); // nameToPartitionItem.get(name) returns null for partitions that were added after T1 } // => ImmutableMap.copyOf() throws NPE: "null value in entry: dt=2026-06-22=null" ``` **Concrete example:** A Hive table has 3 partitions `dt=2026-06-20/21/23`. Session A runs `SELECT * FROM t WHERE dt='2026-06-22'`: ``` T1 BindRelation: LogicalFileScan freezes nameToPartitionItem from cache → {2026-06-20, 2026-06-21, 2026-06-23} (no 2026-06-22) [Session B runs ALTER TABLE ADD PARTITION (dt='2026-06-22')] [cache is refreshed → now has 4 partitions including 2026-06-22] T2 PruneFileScanPartition: re-reads sortedPartitionRanges from cache → {2026-06-20, 2026-06-21, 2026-06-22, 2026-06-23} (new snapshot) binarySearchFiltering matches dt=2026-06-22 → returns "dt=2026-06-22" nameToPartitionItem.get("dt=2026-06-22") → null (old snapshot has no such key) → NPE: "null value in entry: dt=2026-06-22=null" ``` **Fix:** freeze both views from a single snapshot so T2 never re-reads the cache. - `SelectedPartitions` now carries an `Optional<SortedPartitionRanges>` field. - `HMSExternalTable.initSelectedPartitions` reads the cached `HivePartitionValues` once and freezes both the partition map and the cached sorted ranges together (reuses the cache, no just-in-time rebuild). - Hudi has no cached ranges, so `PruneFileScanPartition` builds them lazily from the frozen map only when binary search filtering is enabled. - A missing partition in the lookup loop is now an invariant failure (`Preconditions.checkState`) instead of being silently skipped, which previously produced a partial scan over fewer partitions. ### Release note Fix `NullPointerException` in partition pruning when external table partitions are modified concurrently during query optimization (TOCTOU race in binary search partition filtering).
morrySnow pushed a commit
that referenced
this pull request
Aug 31, 2026
…prevent TOCTOU NPE during partition pruning (#65659) ### What problem does this PR solve? Issue Number: #64800 Related PR: #58877 Problem Summary: Fix a TOCTOU (Time-of-Check Time-of-Use) race condition that causes `NullPointerException` during partition pruning on external tables. **Root cause:** In `PruneFileScanPartition.pruneExternalPartitions()`: 1. `nameToPartitionItem` — frozen at T1 inside `LogicalFileScan.SelectedPartitions` when the plan node is constructed (via `initSelectedPartitions()`) 2. `sortedPartitionRanges` — re-read from the `HivePartitionValues` cache at T2 when the pruning rule executes (via `externalTable.getSortedPartitionRanges()`) If the cache is refreshed between T1 and T2 (e.g. concurrent `ALTER TABLE ADD/DROP PARTITION`), the two snapshots diverge. `binarySearchFiltering` uses the new snapshot to decide which partitions match the predicate, but the caller looks them up in the old snapshot: ```java for (String name : prunedPartitions) { selectedPartitionItems.put(name, nameToPartitionItem.get(name)); // nameToPartitionItem.get(name) returns null for partitions that were added after T1 } // => ImmutableMap.copyOf() throws NPE: "null value in entry: dt=2026-06-22=null" ``` **Concrete example:** A Hive table has 3 partitions `dt=2026-06-20/21/23`. Session A runs `SELECT * FROM t WHERE dt='2026-06-22'`: ``` T1 BindRelation: LogicalFileScan freezes nameToPartitionItem from cache → {2026-06-20, 2026-06-21, 2026-06-23} (no 2026-06-22) [Session B runs ALTER TABLE ADD PARTITION (dt='2026-06-22')] [cache is refreshed → now has 4 partitions including 2026-06-22] T2 PruneFileScanPartition: re-reads sortedPartitionRanges from cache → {2026-06-20, 2026-06-21, 2026-06-22, 2026-06-23} (new snapshot) binarySearchFiltering matches dt=2026-06-22 → returns "dt=2026-06-22" nameToPartitionItem.get("dt=2026-06-22") → null (old snapshot has no such key) → NPE: "null value in entry: dt=2026-06-22=null" ``` **Fix:** freeze both views from a single snapshot so T2 never re-reads the cache. - `SelectedPartitions` now carries an `Optional<SortedPartitionRanges>` field. - `HMSExternalTable.initSelectedPartitions` reads the cached `HivePartitionValues` once and freezes both the partition map and the cached sorted ranges together (reuses the cache, no just-in-time rebuild). - Hudi has no cached ranges, so `PruneFileScanPartition` builds them lazily from the frozen map only when binary search filtering is enabled. - A missing partition in the lookup loop is now an invariant failure (`Preconditions.checkState`) instead of being silently skipped, which previously produced a partial scan over fewer partitions. ### Release note Fix `NullPointerException` in partition pruning when external table partitions are modified concurrently during query optimization (TOCTOU race in binary search partition filtering).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: #64800
Related PR: #58877
Problem Summary:
Fix a TOCTOU (Time-of-Check Time-of-Use) race condition that causes
NullPointerExceptionduring partition pruning on external tables.Root cause: In
PruneFileScanPartition.pruneExternalPartitions():nameToPartitionItem— frozen at T1 insideLogicalFileScan.SelectedPartitionswhen the plan node is constructed (viainitSelectedPartitions())sortedPartitionRanges— re-read from theHivePartitionValuescache at T2 when the pruning rule executes (viaexternalTable.getSortedPartitionRanges())If the cache is refreshed between T1 and T2 (e.g. concurrent
ALTER TABLE ADD/DROP PARTITION), the two snapshots diverge.binarySearchFilteringuses the new snapshot to decide which partitions match the predicate, but the caller looks them up in the old snapshot:Concrete example: A Hive table has 3 partitions
dt=2026-06-20/21/23. Session A runsSELECT * FROM t WHERE dt='2026-06-22':Fix: freeze both views from a single snapshot so T2 never re-reads the cache.
SelectedPartitionsnow carries anOptional<SortedPartitionRanges>field.HMSExternalTable.initSelectedPartitionsreads the cachedHivePartitionValuesonce and freezes both the partition map and the cached sorted ranges together (reuses the cache, no just-in-time rebuild).PruneFileScanPartitionbuilds them lazily from the frozen map only when binary search filtering is enabled.Preconditions.checkState) instead of being silently skipped, which previously produced a partial scan over fewer partitions.Release note
Fix
NullPointerExceptionin partition pruning when external table partitions are modified concurrently during query optimization (TOCTOU race in binary search partition filtering).Check List (For Author)