Allow Range partitioned inputs to PartitionedTopK by stuhood · Pull Request #23355 · apache/datafusion

stuhood · 2026-07-06T23:41:29Z

Which issue does this PR close?

ClosesAllow Partitioning::Range to satisfy PartitionedTopK Distribution::KeyPartitioned requirements #23290.

What changes are included in this PR?

In two commits:

Improve the robustness of WindowTopN to intermediate RepartitionExec nodes, to ensure that PartitionedTopK can still be used when target_partitions mismatches physical partitions.
Use required_input_distributions to declare required partitioning for PartitionedTopKExec, BoundedWindowAggExec, and WindowAggExec.

Are these changes tested?

Yes, new slt tests are added for both the WindowTopN robustness fix, and for the use of required_input_distributions in PartitionedTopK.

Are there any user-facing changes?

No: only plan changes.

gene-bordegaray · 2026-07-09T10:11:30Z

This is in my queue, just waiting for #23184 to land 🙇

gene-bordegaray

Looks goodl, siilar to #23416 lets keep the testing set in slt leaner and not stress the full preserve_file_partitions and subset_repartition_threshold matrix. I will make a PR to have this matrix tested in the unit 👍

I think for slt tests keeping it to a simple positive / negative, subset positive / negative is good, lmk what you think 🙇

Thank you

stuhood · 2026-07-13T21:06:46Z

Thanks @gene-bordegaray : have rebased atop #23416 (only the top 3 commits are relevant) and applied your feedback. I'll put this back in draft until #23416 has landed.

…for Window/TopK

…rtitioned requirements (apache#23416) ## Which issue does this PR close? - Closesapache#23289. Rebased on main now that apache#23184 has merged. Overlaps with apache#23355, which also opts the window execs into range satisfaction as part of its PartitionedTopK work — happy to rebase whichever lands second. ## What changes are included in this PR? In one commit: * Opt `WindowAggExec` and `BoundedWindowAggExec` with partition keys into range satisfaction via `InputDistributionRequirements::allow_range_satisfaction_for_key_partitioning`, mirroring `AggregateExec`. Compatible range-partitioned inputs then satisfy the window key requirement without a hash repartition; subset satisfaction and the hash fallback for incompatible keys come from the existing satisfaction machinery. Windows without `PARTITION BY` keep requiring a single partition. ## Are these changes tested? Yes: new `slt` tests in `range_partitioning.slt` (exact and subset reuse, rehash on incompatible keys, `subset_repartition_threshold` / `preserve_file_partitions` / `target_partitions` behavior, `WindowAggExec` via an unbounded frame, and no-`PARTITION BY`), plus plan-shape tests in `enforce_distribution.rs` and acceptance/rejection tests in `sanity_checker.rs`. ## Are there any user-facing changes? No: only plan changes.

stuhood · 2026-07-14T19:26:49Z

@gene-bordegaray: Ready for another look! Thanks.

gene-bordegaray

Great, thanks. Just some small nits 👍

cc: @gabotechs

gabotechs

Thanks @stuhood and @gene-bordegaray!

…rtitioned requirements (apache#23416) ## Which issue does this PR close? - Closesapache#23289. Rebased on main now that apache#23184 has merged. Overlaps with apache#23355, which also opts the window execs into range satisfaction as part of its PartitionedTopK work — happy to rebase whichever lands second. ## What changes are included in this PR? In one commit: * Opt `WindowAggExec` and `BoundedWindowAggExec` with partition keys into range satisfaction via `InputDistributionRequirements::allow_range_satisfaction_for_key_partitioning`, mirroring `AggregateExec`. Compatible range-partitioned inputs then satisfy the window key requirement without a hash repartition; subset satisfaction and the hash fallback for incompatible keys come from the existing satisfaction machinery. Windows without `PARTITION BY` keep requiring a single partition. ## Are these changes tested? Yes: new `slt` tests in `range_partitioning.slt` (exact and subset reuse, rehash on incompatible keys, `subset_repartition_threshold` / `preserve_file_partitions` / `target_partitions` behavior, `WindowAggExec` via an unbounded frame, and no-`PARTITION BY`), plus plan-shape tests in `enforce_distribution.rs` and acceptance/rejection tests in `sanity_checker.rs`. ## Are there any user-facing changes? No: only plan changes.

- Closesapache#23290. In two commits: * Improve the robustness of `WindowTopN` to intermediate `RepartitionExec` nodes, to ensure that `PartitionedTopK` can still be used when `target_partitions` mismatches physical partitions. * Use `required_input_distributions` to declare required partitioning for `PartitionedTopKExec`, `BoundedWindowAggExec`, and `WindowAggExec`. Yes, new `slt` tests are added for both the `WindowTopN` robustness fix, and for the use of `required_input_distributions` in `PartitionedTopK`. No: only plan changes.

…rtitioned requirements (apache#23416) ## Which issue does this PR close? - Closesapache#23289. Rebased on main now that apache#23184 has merged. Overlaps with apache#23355, which also opts the window execs into range satisfaction as part of its PartitionedTopK work — happy to rebase whichever lands second. ## What changes are included in this PR? In one commit: * Opt `WindowAggExec` and `BoundedWindowAggExec` with partition keys into range satisfaction via `InputDistributionRequirements::allow_range_satisfaction_for_key_partitioning`, mirroring `AggregateExec`. Compatible range-partitioned inputs then satisfy the window key requirement without a hash repartition; subset satisfaction and the hash fallback for incompatible keys come from the existing satisfaction machinery. Windows without `PARTITION BY` keep requiring a single partition. ## Are these changes tested? Yes: new `slt` tests in `range_partitioning.slt` (exact and subset reuse, rehash on incompatible keys, `subset_repartition_threshold` / `preserve_file_partitions` / `target_partitions` behavior, `WindowAggExec` via an unbounded frame, and no-`PARTITION BY`), plus plan-shape tests in `enforce_distribution.rs` and acceptance/rejection tests in `sanity_checker.rs`. ## Are there any user-facing changes? No: only plan changes.

- Closesapache#23290. In two commits: * Improve the robustness of `WindowTopN` to intermediate `RepartitionExec` nodes, to ensure that `PartitionedTopK` can still be used when `target_partitions` mismatches physical partitions. * Use `required_input_distributions` to declare required partitioning for `PartitionedTopKExec`, `BoundedWindowAggExec`, and `WindowAggExec`. Yes, new `slt` tests are added for both the `WindowTopN` robustness fix, and for the use of `required_input_distributions` in `PartitionedTopK`. No: only plan changes.

…rtitioned requirements (apache#23416) ## Which issue does this PR close? - Closesapache#23289. Rebased on main now that apache#23184 has merged. Overlaps with apache#23355, which also opts the window execs into range satisfaction as part of its PartitionedTopK work — happy to rebase whichever lands second. ## What changes are included in this PR? In one commit: * Opt `WindowAggExec` and `BoundedWindowAggExec` with partition keys into range satisfaction via `InputDistributionRequirements::allow_range_satisfaction_for_key_partitioning`, mirroring `AggregateExec`. Compatible range-partitioned inputs then satisfy the window key requirement without a hash repartition; subset satisfaction and the hash fallback for incompatible keys come from the existing satisfaction machinery. Windows without `PARTITION BY` keep requiring a single partition. ## Are these changes tested? Yes: new `slt` tests in `range_partitioning.slt` (exact and subset reuse, rehash on incompatible keys, `subset_repartition_threshold` / `preserve_file_partitions` / `target_partitions` behavior, `WindowAggExec` via an unbounded frame, and no-`PARTITION BY`), plus plan-shape tests in `enforce_distribution.rs` and acceptance/rejection tests in `sanity_checker.rs`. ## Are there any user-facing changes? No: only plan changes.

## Which issue does this PR close? - Closesapache#23290. ## What changes are included in this PR? In two commits: * Improve the robustness of `WindowTopN` to intermediate `RepartitionExec` nodes, to ensure that `PartitionedTopK` can still be used when `target_partitions` mismatches physical partitions. * Use `required_input_distributions` to declare required partitioning for `PartitionedTopKExec`, `BoundedWindowAggExec`, and `WindowAggExec`. ## Are these changes tested? Yes, new `slt` tests are added for both the `WindowTopN` robustness fix, and for the use of `required_input_distributions` in `PartitionedTopK`. ## Are there any user-facing changes? No: only plan changes.

github-actionsBot added physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Jul 6, 2026

stuhood changed the title ~~Stuhood.range partitioned top k~~Allow Range partitioned inputs to PartitionedTopKJul 6, 2026

stuhood mentioned this pull request Jul 6, 2026
Allow Partitioning::Range to satisfy PartitionedTopK Distribution::KeyPartitioned requirements #23290
Closed

github-actionsBot added the auto detected api change Auto detected API change label Jul 6, 2026

stuhood force-pushed the stuhood.range-partitioned-top-k branch from 1b1c490 to e236f7bCompare July 8, 2026 19:30

github-actionsBot added the datasource Changes to the datasource crate label Jul 8, 2026

stuhood force-pushed the stuhood.range-partitioned-top-k branch 3 times, most recently from eb2c841 to 3ab476dCompare July 8, 2026 23:10

mithuncy mentioned this pull request Jul 9, 2026
feat: allow Partitioning::Range to satisfy window Distribution::KeyPartitioned requirements #23416
Merged

stuhood force-pushed the stuhood.range-partitioned-top-k branch from 3ab476d to 7d0f380Compare July 10, 2026 18:10

github-actionsBot removed physical-expr Changes to the physical-expr crates core Core DataFusion crate datasource Changes to the datasource crate auto detected api change Auto detected API change labels Jul 10, 2026

stuhood marked this pull request as ready for review July 10, 2026 20:55

gene-bordegaray reviewed Jul 13, 2026
View reviewed changes

stuhood force-pushed the stuhood.range-partitioned-top-k branch from 7d0f380 to ab90d5cCompare July 13, 2026 21:04

github-actionsBot added the core Core DataFusion crate label Jul 13, 2026

stuhood marked this pull request as draft July 13, 2026 21:06

mithuncy mentioned this pull request Jul 14, 2026
Add config-matrix tests in enforce_distribution.rs for range-satisfaction settings #23572
Closed

stuhood added 3 commits July 14, 2026 12:22

fix: make WindowTopN robust to intermediate RepartitionExec nodes
3457fb7

feat: allow Range partitioning to satisfy KeyPartitioned requirement …
33b3267
…for Window/TopK

Review feedback.
c92419e

stuhood force-pushed the stuhood.range-partitioned-top-k branch from ab90d5c to c92419eCompare July 14, 2026 19:25

github-actionsBot removed the core Core DataFusion crate label Jul 14, 2026

stuhood marked this pull request as ready for review July 14, 2026 19:25

stuhood requested a review from gene-bordegaray July 14, 2026 19:27

gene-bordegaray approved these changes Jul 14, 2026
View reviewed changes

Comment threaddatafusion/sqllogictest/test_files/range_partitioning.slt Outdated
Comment threaddatafusion/physical-optimizer/src/window_topn.rs
Comment threaddatafusion/sqllogictest/test_files/range_partitioning.slt Outdated

Review feedback.
34c5995

gabotechs approved these changes Jul 15, 2026
View reviewed changes

gabotechs added this pull request to the merge queue Jul 15, 2026

Merged via the queue into apache:main with commit e123e6fJul 15, 2026
40 checks passed

Navigation Menu

Uh oh!

Allow Range partitioned inputs to PartitionedTopK - #23355

Allow Range partitioned inputs to PartitionedTopK#23355
gabotechs merged 4 commits into
apache:mainfrom
paradedb:stuhood.range-partitioned-top-k

stuhood commented Jul 6, 2026

Uh oh!

gene-bordegaray commented Jul 9, 2026

Uh oh!

gene-bordegaray left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stuhood commented Jul 13, 2026

Uh oh!

stuhood commented Jul 14, 2026

Uh oh!

gene-bordegaray left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gabotechs left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

stuhood commented Jul 6, 2026

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

gene-bordegaray commented Jul 9, 2026

Uh oh!

gene-bordegaray left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stuhood commented Jul 13, 2026

Uh oh!

stuhood commented Jul 14, 2026

Uh oh!

gene-bordegaray left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gabotechs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants