Uh oh!
There was an error while loading. Please reload this page.
[opt](hive) use binary search to prune hive partitions - #58877
Merged
Conversation
hello-stephen
commented
Dec 9, 2025
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
zy-kkk
commented
Dec 9, 2025
MemberAuthor
run buildall |
doris-robot
commented
Dec 9, 2025
TPC-H: Total hot run time: 36473 ms |
doris-robot
commented
Dec 9, 2025
TPC-DS: Total hot run time: 182268 ms |
doris-robot
commented
Dec 9, 2025
ClickBench: Total hot run time: 27.37 s |
zy-kkk
commented
Dec 9, 2025
MemberAuthor
run buildall |
doris-robot
commented
Dec 9, 2025
TPC-H: Total hot run time: 36315 ms |
doris-robot
commented
Dec 9, 2025
TPC-DS: Total hot run time: 182033 ms |
doris-robot
commented
Dec 9, 2025
ClickBench: Total hot run time: 27.63 s |
hello-stephen
commented
Dec 9, 2025
Contributor
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Dec 9, 2025
Contributor
FE Regression Coverage ReportIncrement line coverage |
zy-kkk
commented
Dec 10, 2025
MemberAuthor
run buildall |
| }); | ||
| return new SortedPartitionRanges<>(sortedRanges, defaultPartitions); | ||
| } |
Contributor
There was a problem hiding this comment.
I think you should extract code from NereidsSortedPartitionsCacheManager.loadCache and reuse the same utility function
doris-robot
commented
Dec 10, 2025
TPC-H: Total hot run time: 36931 ms |
doris-robot
commented
Dec 10, 2025
TPC-DS: Total hot run time: 181153 ms |
doris-robot
commented
Dec 10, 2025
ClickBench: Total hot run time: 27.33 s |
924060929
approved these changes
Dec 10, 2025
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
zy-kkk
commented
Dec 10, 2025
MemberAuthor
run buildall |
doris-robot
commented
Dec 10, 2025
TPC-H: Total hot run time: 36566 ms |
doris-robot
commented
Dec 10, 2025
TPC-DS: Total hot run time: 180827 ms |
doris-robot
commented
Dec 10, 2025
ClickBench: Total hot run time: 27.41 s |
doris-robot
commented
Dec 10, 2025
ClickBench: Total hot run time: 27.64 s |
hello-stephen
commented
Dec 10, 2025
Contributor
FE Regression Coverage ReportIncrement line coverage |
Uh oh!
There was an error while loading. Please reload this page.
github-actionsBot
pushed a commit
that referenced
this pull request
Dec 10, 2025
Followup #44586 Enable binary search partition pruning optimization for Hive external tables. This PR adds binary search partition pruning support for Hive tables by: - Adding `getSortedPartitionRanges()` method to `ExternalTable` base class - Maintaining sorted partition ranges directly in `HivePartitionValues` for cache lifecycle consistency - Overriding `getSortedPartitionRanges()` in `HMSExternalTable` to provide sorted ranges **Performance improvement (20000 partitions, 1000 queries):** - Binary search enabled: **4.548 seconds** - Binary search disabled: **12.849 seconds** - **~2.8x faster**
zy-kkk added a commit
to zy-kkk/doris
that referenced
this pull request
Dec 11, 2025
Followup apache#44586 Enable binary search partition pruning optimization for Hive external tables. This PR adds binary search partition pruning support for Hive tables by: - Adding `getSortedPartitionRanges()` method to `ExternalTable` base class - Maintaining sorted partition ranges directly in `HivePartitionValues` for cache lifecycle consistency - Overriding `getSortedPartitionRanges()` in `HMSExternalTable` to provide sorted ranges **Performance improvement (20000 partitions, 1000 queries):** - Binary search enabled: **4.548 seconds** - Binary search disabled: **12.849 seconds** - **~2.8x faster**
starocean999 pushed a commit
that referenced
this pull request
Dec 11, 2025
Followup #44586 Enable binary search partition pruning optimization for Hive external tables. This PR adds binary search partition pruning support for Hive tables by: - Adding `getSortedPartitionRanges()` method to `ExternalTable` base class - Maintaining sorted partition ranges directly in `HivePartitionValues` for cache lifecycle consistency - Overriding `getSortedPartitionRanges()` in `HMSExternalTable` to provide sorted ranges **Performance improvement (20000 partitions, 1000 queries):** - Binary search enabled: **4.548 seconds** - Binary search disabled: **12.849 seconds** - **~2.8x faster**
yiguolei pushed a commit
that referenced
this pull request
Dec 12, 2025
nagisa-kunhah pushed a commit
to nagisa-kunhah/doris
that referenced
this pull request
Dec 14, 2025
Followup apache#44586 Enable binary search partition pruning optimization for Hive external tables. This PR adds binary search partition pruning support for Hive tables by: - Adding `getSortedPartitionRanges()` method to `ExternalTable` base class - Maintaining sorted partition ranges directly in `HivePartitionValues` for cache lifecycle consistency - Overriding `getSortedPartitionRanges()` in `HMSExternalTable` to provide sorted ranges **Performance improvement (20000 partitions, 1000 queries):** - Binary search enabled: **4.548 seconds** - Binary search disabled: **12.849 seconds** - **~2.8x faster**
zy-kkk added a commit
to zy-kkk/doris
that referenced
this pull request
Dec 16, 2025
morrySnow pushed a commit
that referenced
this pull request
Dec 17, 2025
4 tasks
morrySnow pushed a commit
that referenced
this pull request
Jul 20, 2026
…prevent TOCTOU NPE during partition pruning (#65659) ### What problem does this PR solve? Issue Number: #64800 Related PR: #58877 Problem Summary: Fix a TOCTOU (Time-of-Check Time-of-Use) race condition that causes `NullPointerException` during partition pruning on external tables. **Root cause:** In `PruneFileScanPartition.pruneExternalPartitions()`: 1. `nameToPartitionItem` — frozen at T1 inside `LogicalFileScan.SelectedPartitions` when the plan node is constructed (via `initSelectedPartitions()`) 2. `sortedPartitionRanges` — re-read from the `HivePartitionValues` cache at T2 when the pruning rule executes (via `externalTable.getSortedPartitionRanges()`) If the cache is refreshed between T1 and T2 (e.g. concurrent `ALTER TABLE ADD/DROP PARTITION`), the two snapshots diverge. `binarySearchFiltering` uses the new snapshot to decide which partitions match the predicate, but the caller looks them up in the old snapshot: ```java for (String name : prunedPartitions) { selectedPartitionItems.put(name, nameToPartitionItem.get(name)); // nameToPartitionItem.get(name) returns null for partitions that were added after T1 } // => ImmutableMap.copyOf() throws NPE: "null value in entry: dt=2026-06-22=null" ``` **Concrete example:** A Hive table has 3 partitions `dt=2026-06-20/21/23`. Session A runs `SELECT * FROM t WHERE dt='2026-06-22'`: ``` T1 BindRelation: LogicalFileScan freezes nameToPartitionItem from cache → {2026-06-20, 2026-06-21, 2026-06-23} (no 2026-06-22) [Session B runs ALTER TABLE ADD PARTITION (dt='2026-06-22')] [cache is refreshed → now has 4 partitions including 2026-06-22] T2 PruneFileScanPartition: re-reads sortedPartitionRanges from cache → {2026-06-20, 2026-06-21, 2026-06-22, 2026-06-23} (new snapshot) binarySearchFiltering matches dt=2026-06-22 → returns "dt=2026-06-22" nameToPartitionItem.get("dt=2026-06-22") → null (old snapshot has no such key) → NPE: "null value in entry: dt=2026-06-22=null" ``` **Fix:** freeze both views from a single snapshot so T2 never re-reads the cache. - `SelectedPartitions` now carries an `Optional<SortedPartitionRanges>` field. - `HMSExternalTable.initSelectedPartitions` reads the cached `HivePartitionValues` once and freezes both the partition map and the cached sorted ranges together (reuses the cache, no just-in-time rebuild). - Hudi has no cached ranges, so `PruneFileScanPartition` builds them lazily from the frozen map only when binary search filtering is enabled. - A missing partition in the lookup loop is now an invariant failure (`Preconditions.checkState`) instead of being silently skipped, which previously produced a partial scan over fewer partitions. ### Release note Fix `NullPointerException` in partition pruning when external table partitions are modified concurrently during query optimization (TOCTOU race in binary search partition filtering).
github-actionsBot
pushed a commit
that referenced
this pull request
Jul 20, 2026
…prevent TOCTOU NPE during partition pruning (#65659) ### What problem does this PR solve? Issue Number: #64800 Related PR: #58877 Problem Summary: Fix a TOCTOU (Time-of-Check Time-of-Use) race condition that causes `NullPointerException` during partition pruning on external tables. **Root cause:** In `PruneFileScanPartition.pruneExternalPartitions()`: 1. `nameToPartitionItem` — frozen at T1 inside `LogicalFileScan.SelectedPartitions` when the plan node is constructed (via `initSelectedPartitions()`) 2. `sortedPartitionRanges` — re-read from the `HivePartitionValues` cache at T2 when the pruning rule executes (via `externalTable.getSortedPartitionRanges()`) If the cache is refreshed between T1 and T2 (e.g. concurrent `ALTER TABLE ADD/DROP PARTITION`), the two snapshots diverge. `binarySearchFiltering` uses the new snapshot to decide which partitions match the predicate, but the caller looks them up in the old snapshot: ```java for (String name : prunedPartitions) { selectedPartitionItems.put(name, nameToPartitionItem.get(name)); // nameToPartitionItem.get(name) returns null for partitions that were added after T1 } // => ImmutableMap.copyOf() throws NPE: "null value in entry: dt=2026-06-22=null" ``` **Concrete example:** A Hive table has 3 partitions `dt=2026-06-20/21/23`. Session A runs `SELECT * FROM t WHERE dt='2026-06-22'`: ``` T1 BindRelation: LogicalFileScan freezes nameToPartitionItem from cache → {2026-06-20, 2026-06-21, 2026-06-23} (no 2026-06-22) [Session B runs ALTER TABLE ADD PARTITION (dt='2026-06-22')] [cache is refreshed → now has 4 partitions including 2026-06-22] T2 PruneFileScanPartition: re-reads sortedPartitionRanges from cache → {2026-06-20, 2026-06-21, 2026-06-22, 2026-06-23} (new snapshot) binarySearchFiltering matches dt=2026-06-22 → returns "dt=2026-06-22" nameToPartitionItem.get("dt=2026-06-22") → null (old snapshot has no such key) → NPE: "null value in entry: dt=2026-06-22=null" ``` **Fix:** freeze both views from a single snapshot so T2 never re-reads the cache. - `SelectedPartitions` now carries an `Optional<SortedPartitionRanges>` field. - `HMSExternalTable.initSelectedPartitions` reads the cached `HivePartitionValues` once and freezes both the partition map and the cached sorted ranges together (reuses the cache, no just-in-time rebuild). - Hudi has no cached ranges, so `PruneFileScanPartition` builds them lazily from the frozen map only when binary search filtering is enabled. - A missing partition in the lookup loop is now an invariant failure (`Preconditions.checkState`) instead of being silently skipped, which previously produced a partial scan over fewer partitions. ### Release note Fix `NullPointerException` in partition pruning when external table partitions are modified concurrently during query optimization (TOCTOU race in binary search partition filtering).
wyxxxcat pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
…prevent TOCTOU NPE during partition pruning (apache#65659) ### What problem does this PR solve? Issue Number: apache#64800 Related PR: apache#58877 Problem Summary: Fix a TOCTOU (Time-of-Check Time-of-Use) race condition that causes `NullPointerException` during partition pruning on external tables. **Root cause:** In `PruneFileScanPartition.pruneExternalPartitions()`: 1. `nameToPartitionItem` — frozen at T1 inside `LogicalFileScan.SelectedPartitions` when the plan node is constructed (via `initSelectedPartitions()`) 2. `sortedPartitionRanges` — re-read from the `HivePartitionValues` cache at T2 when the pruning rule executes (via `externalTable.getSortedPartitionRanges()`) If the cache is refreshed between T1 and T2 (e.g. concurrent `ALTER TABLE ADD/DROP PARTITION`), the two snapshots diverge. `binarySearchFiltering` uses the new snapshot to decide which partitions match the predicate, but the caller looks them up in the old snapshot: ```java for (String name : prunedPartitions) { selectedPartitionItems.put(name, nameToPartitionItem.get(name)); // nameToPartitionItem.get(name) returns null for partitions that were added after T1 } // => ImmutableMap.copyOf() throws NPE: "null value in entry: dt=2026-06-22=null" ``` **Concrete example:** A Hive table has 3 partitions `dt=2026-06-20/21/23`. Session A runs `SELECT * FROM t WHERE dt='2026-06-22'`: ``` T1 BindRelation: LogicalFileScan freezes nameToPartitionItem from cache → {2026-06-20, 2026-06-21, 2026-06-23} (no 2026-06-22) [Session B runs ALTER TABLE ADD PARTITION (dt='2026-06-22')] [cache is refreshed → now has 4 partitions including 2026-06-22] T2 PruneFileScanPartition: re-reads sortedPartitionRanges from cache → {2026-06-20, 2026-06-21, 2026-06-22, 2026-06-23} (new snapshot) binarySearchFiltering matches dt=2026-06-22 → returns "dt=2026-06-22" nameToPartitionItem.get("dt=2026-06-22") → null (old snapshot has no such key) → NPE: "null value in entry: dt=2026-06-22=null" ``` **Fix:** freeze both views from a single snapshot so T2 never re-reads the cache. - `SelectedPartitions` now carries an `Optional<SortedPartitionRanges>` field. - `HMSExternalTable.initSelectedPartitions` reads the cached `HivePartitionValues` once and freezes both the partition map and the cached sorted ranges together (reuses the cache, no just-in-time rebuild). - Hudi has no cached ranges, so `PruneFileScanPartition` builds them lazily from the frozen map only when binary search filtering is enabled. - A missing partition in the lookup loop is now an invariant failure (`Preconditions.checkState`) instead of being silently skipped, which previously produced a partial scan over fewer partitions. ### Release note Fix `NullPointerException` in partition pruning when external table partitions are modified concurrently during query optimization (TOCTOU race in binary search partition filtering).
morrySnow pushed a commit
that referenced
this pull request
Aug 31, 2026
…prevent TOCTOU NPE during partition pruning (#65659) ### What problem does this PR solve? Issue Number: #64800 Related PR: #58877 Problem Summary: Fix a TOCTOU (Time-of-Check Time-of-Use) race condition that causes `NullPointerException` during partition pruning on external tables. **Root cause:** In `PruneFileScanPartition.pruneExternalPartitions()`: 1. `nameToPartitionItem` — frozen at T1 inside `LogicalFileScan.SelectedPartitions` when the plan node is constructed (via `initSelectedPartitions()`) 2. `sortedPartitionRanges` — re-read from the `HivePartitionValues` cache at T2 when the pruning rule executes (via `externalTable.getSortedPartitionRanges()`) If the cache is refreshed between T1 and T2 (e.g. concurrent `ALTER TABLE ADD/DROP PARTITION`), the two snapshots diverge. `binarySearchFiltering` uses the new snapshot to decide which partitions match the predicate, but the caller looks them up in the old snapshot: ```java for (String name : prunedPartitions) { selectedPartitionItems.put(name, nameToPartitionItem.get(name)); // nameToPartitionItem.get(name) returns null for partitions that were added after T1 } // => ImmutableMap.copyOf() throws NPE: "null value in entry: dt=2026-06-22=null" ``` **Concrete example:** A Hive table has 3 partitions `dt=2026-06-20/21/23`. Session A runs `SELECT * FROM t WHERE dt='2026-06-22'`: ``` T1 BindRelation: LogicalFileScan freezes nameToPartitionItem from cache → {2026-06-20, 2026-06-21, 2026-06-23} (no 2026-06-22) [Session B runs ALTER TABLE ADD PARTITION (dt='2026-06-22')] [cache is refreshed → now has 4 partitions including 2026-06-22] T2 PruneFileScanPartition: re-reads sortedPartitionRanges from cache → {2026-06-20, 2026-06-21, 2026-06-22, 2026-06-23} (new snapshot) binarySearchFiltering matches dt=2026-06-22 → returns "dt=2026-06-22" nameToPartitionItem.get("dt=2026-06-22") → null (old snapshot has no such key) → NPE: "null value in entry: dt=2026-06-22=null" ``` **Fix:** freeze both views from a single snapshot so T2 never re-reads the cache. - `SelectedPartitions` now carries an `Optional<SortedPartitionRanges>` field. - `HMSExternalTable.initSelectedPartitions` reads the cached `HivePartitionValues` once and freezes both the partition map and the cached sorted ranges together (reuses the cache, no just-in-time rebuild). - Hudi has no cached ranges, so `PruneFileScanPartition` builds them lazily from the frozen map only when binary search filtering is enabled. - A missing partition in the lookup loop is now an invariant failure (`Preconditions.checkState`) instead of being silently skipped, which previously produced a partial scan over fewer partitions. ### Release note Fix `NullPointerException` in partition pruning when external table partitions are modified concurrently during query optimization (TOCTOU race in binary search partition filtering).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Followup #44586
Enable binary search partition pruning optimization for Hive external tables.
This PR adds binary search partition pruning support for Hive tables by:
getSortedPartitionRanges()method toExternalTablebase classHivePartitionValuesfor cache lifecycle consistencygetSortedPartitionRanges()inHMSExternalTableto provide sorted rangesPerformance improvement (20000 partitions, 1000 queries):