Uh oh!
There was an error while loading. Please reload this page.
fix: preserve aggregate filter pushdown order - #22926
Conversation
discord9
commented
Jul 1, 2026
Hi, this PR fixes #22925 and is still waiting for review. The only failing check is Thanks! |
05e5a77 to
3626a96CompareCodecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@## main #22926 +/- ##
==========================================
+ Coverage 80.70% 80.75% +0.04%
==========================================
Files 1095 1096 +1 Lines 372554 373588 +1034 Branches 372554 373588 +1034 ==========================================
+ Hits 300665 301677 +1012 + Misses 53921 53906 -15 - Partials 17968 18005 +37 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
neilconway
left a comment
There was a problem hiding this comment.
Thank you for this PR! Overall, this is excellent work: clear diagnosis, minimal and correct fix, and good test coverage. I'm sorry it didn't get reviewed sooner -- a few minor suggestions but it should be good to land shortly.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
3626a96 to
4b0e6c6Compare
neilconway
left a comment
There was a problem hiding this comment.
Looking good! Can you update the PR description, please? Currently refers to the old implementation; would also be good to elaborate on the pushdown change for global aggregates?
Uh oh!
There was an error while loading. Please reload this page.
discord9
commented
Jul 29, 2026
Updated the PR description to reflect the current |
Uh oh!
There was an error while loading. Please reload this page.
## Which issue does this PR close? - Closesapache#22925. ## Rationale for this change `FilterPushdown` maps each child pushdown result back to its original parent filter by position. `AggregateExec::gather_filters_for_pushdown` previously split parent filters into safe and unsafe buckets and then concatenated the results, which changed their order. For example, in `cnt@2 = 1 AND b@1 = bar`, the grouping-column predicate on `b` can cross the aggregate, while the predicate on aggregate output `cnt` must remain above it. Reordering the returned results could make the optimizer associate the pushed-down `b` result with the `cnt` predicate and incorrectly remove the latter. Aggregate pushdown also needs to account for empty-input semantics. A global aggregate, or a grouping-sets aggregate containing `()`, can emit a row even when its input is empty. Moving any parent predicate below such an aggregate — including a column-free predicate such as `false` — can therefore change the result. ## What changes are included in this PR? - Preserve parent-filter order by constructing each child description once with `ChildFilterDescription::from_child_with_allowed_indices`. - Allow only grouping-output columns that are present in every grouping set; aggregate-result columns remain above the aggregate. - Mark all parent filters unsupported for global aggregates and grouping sets containing an empty grouping set, while retaining aggregate-generated dynamic filters. - Validate that every child returns one parent-filter result per input filter before positional remapping. - Add physical optimizer and SQL regression coverage for mixed filter order, global-aggregate name collisions, constant predicates, and grouping sets. ## Are these changes tested? Yes: - `cargo test -p datafusion --test core_integration physical_optimizer::filter_pushdown` - `cargo fmt --all -- --check` - `cargo check -p datafusion-physical-plan -p datafusion-physical-optimizer` - `git diff --check upstream/main` - Full GitHub CI, including Rust tests, clippy, and sqllogictests ## Are there any user-facing changes? There are no public API changes. Filter pushdown now preserves mixed aggregate predicates correctly and avoids moving predicates across aggregates when doing so could change empty-input results.
Which issue does this PR close?
Rationale for this change
FilterPushdownmaps each child pushdown result back to its original parentfilter by position.
AggregateExec::gather_filters_for_pushdownpreviouslysplit parent filters into safe and unsafe buckets and then concatenated the
results, which changed their order.
For example, in
cnt@2 = 1 AND b@1 = bar, the grouping-column predicate onbcan cross the aggregate, while the predicate on aggregate outputcntmust remain above it. Reordering the returned results could make the optimizer
associate the pushed-down
bresult with thecntpredicate and incorrectlyremove the latter.
Aggregate pushdown also needs to account for empty-input semantics. A global
aggregate, or a grouping-sets aggregate containing
(), can emit a row evenwhen its input is empty. Moving any parent predicate below such an aggregate —
including a column-free predicate such as
false— can therefore change theresult.
What changes are included in this PR?
with
ChildFilterDescription::from_child_with_allowed_indices.aggregate-result columns remain above the aggregate.
containing an empty grouping set, while retaining aggregate-generated
dynamic filters.
before positional remapping.
global-aggregate name collisions, constant predicates, and grouping sets.
Are these changes tested?
Yes:
cargo test -p datafusion --test core_integration physical_optimizer::filter_pushdowncargo fmt --all -- --checkcargo check -p datafusion-physical-plan -p datafusion-physical-optimizergit diff --check upstream/mainAre there any user-facing changes?
There are no public API changes. Filter pushdown now preserves mixed aggregate
predicates correctly and avoids moving predicates across aggregates when doing
so could change empty-input results.