Uh oh!
There was an error while loading. Please reload this page.
feat: Set NDV to Exact(1) for numeric equality filter predicates - #21077
Merged
xudong963 merged 5 commits intoApr 7, 2026
Conversation
asolimandoforce-pushed
the
asolimando/ndv-single-value-filter
branch
from
March 20, 2026 12:24
061df5a to
0613f1aCompareasolimando
commented
Mar 20, 2026
MemberAuthor
cc: @jonathanc-n |
| @@ -59,7 +59,7 @@ query TT | |||
| EXPLAIN SELECT * FROM test_table WHERE column1 = 1; | |||
Contributor
There was a problem hiding this comment.
Can we add some tests with e.g. floating point numbers, strings? I think it should work just the same but....
MemberAuthor
There was a problem hiding this comment.
Thanks for the suggestion! I added tests for additional numeric types (Int8, Int64, Float32) beyond the original Int32/Float64, all producing NDV=Exact(1) as expected. Also added a reversed predicate test (42 = a instead of a = 42) to confirm operand order doesn't matter.
I investigated string (Utf8) and temporal types (Timestamp, Interval) as well: neither collapses to a single-value interval today due to limitations in the constraint propagation solver. I filed separate issues to track those:
- feat: Extend single-value NDV optimization to string types #21109 (strings)
- feat: Extend single-value NDV optimization to timestamp and interval types #21111 (timestamps/intervals)
The test coverage now includes:
- Per-type: Int8, Int32, Int64, Float32, Float64
- Predicate shapes: simple equality, OR, AND (multi-column), reversed operands, absent bounds
Contributor
There was a problem hiding this comment.
Can you make the tests in 1531e5c SLT tests please? We often find them more valuable than unit tests. Sorry I should have clarified in my comment above...
MemberAuthor
There was a problem hiding this comment.
No worries, re-reading your comment it was pretty clear already, I think this is on me!
I have added tests to the same file in 0613f1a, but over a different (single file) table, the use case for single table/multiple files being covered by the first test, I hope I have captured your intention.
I took dictionary.slt#L34 as a reference on how to cast to different numeric types in SLT tests.
I haven't dropped 1531e5c as I feel it's valuable to have unit tests co-located with the code, but this feels redundant I am OK removing them.
MemberAuthor
When a filter predicate collapses a column's interval to a single value (e.g. d_qoy = 1), the output has exactly 1 distinct value. Previously the original Parquet NDV was propagated, inflating GROUP BY output estimates for CTE self-join patterns like Q31.
Cover Int8, Int64, Float32 equality predicates and reversed operand order (literal = column) for NDV single-value optimization.
Cover Int8, Int64, Float32 equality predicates and reversed operand order (literal = column) as sqllogictests in parquet_statistics.slt.
asolimandoforce-pushed
the
asolimando/ndv-single-value-filter
branch
from
April 1, 2026 12:38
a98ead3 to
ab09466Compareadriangb
approved these changes
Apr 1, 2026
Dandandan
commented
Apr 4, 2026
Contributor
Run benchmarks |
adriangbot
commented
Apr 4, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing asolimando/ndv-single-value-filter (ab09466) to bc2b36c (merge-base) diff using: tpch File an issue against this benchmark runner |
adriangbot
commented
Apr 4, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing asolimando/ndv-single-value-filter (ab09466) to bc2b36c (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
adriangbot
commented
Apr 4, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing asolimando/ndv-single-value-filter (ab09466) to bc2b36c (merge-base) diff using: tpcds File an issue against this benchmark runner |
adriangbot
commented
Apr 4, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
adriangbot
commented
Apr 4, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
adriangbot
commented
Apr 4, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
adriangb
commented
Apr 4, 2026
Contributor
run benchmark clickbench_partitioned |
adriangbot
commented
Apr 4, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing asolimando/ndv-single-value-filter (ab09466) to bc2b36c (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
adriangbot
commented
Apr 4, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
adriangb
commented
Apr 4, 2026
Contributor
Huh that's quite a bit of variability on Q32 |
adriangb
commented
Apr 4, 2026
Contributor
run benchmark clickbench_partitioned |
adriangbot
commented
Apr 4, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing asolimando/ndv-single-value-filter (ab09466) to bc2b36c (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
adriangbot
commented
Apr 4, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
The benchmark runners are not usually this noisy - not sure what's going on there. But nothing egregious, I don't think it's related to this PR. I think we can merge this. |
adriangb
enabled auto-merge
April 4, 2026 18:50
xudong963
approved these changes
Apr 7, 2026
xudong963
disabled auto-merge
April 7, 2026 08:15
xudong963
enabled auto-merge
April 7, 2026 08:15
Uh oh!
There was an error while loading. Please reload this page.
Dandandan pushed a commit
to Dandandan/arrow-datafusion
that referenced
this pull request
Apr 8, 2026
…che#21077) ## Which issue does this PR close? - Part of apache#20766 Related: apache#20789 (uses NDV for equality filter selectivity, complementary - this PR improves the NDV output stats, that PR consumes them) ## Rationale for this change When a filter predicate collapses a column interval to a single value (e.g. `d_qoy = 1`), the output column can only have one distinct value. Currently `distinct_count` is always demoted to `Inexact`, losing this information. This matters for downstream optimizers that rely on `distinct_count`, such as join cardinality estimation in `estimate_inner_join_cardinality`. ## What changes are included in this PR? In `collect_new_statistics` (filter.rs), when the post-filter interval has `lower == upper` (both non-null), set `distinct_count` to `Precision::Exact(1)` instead of demoting the input NDV to `Inexact`. ## Are these changes tested? Yes, 4 unit tests: - Equality predicate (`a = 42`) -> NDV becomes `Exact(1)` - OR predicate (`a = 42 OR a = 22`) -> interval does not collapse, NDV stays `Inexact` - AND with mixed predicates (`a = 42 AND b > 10 AND c = 7`) -> `a` and `c` get `Exact(1)`, `b` stays `Inexact` - Equality with absent bounds (`a = 42`, no min/max) -> interval analysis still resolves to `Exact(1)` ## Are there any user-facing changes? No breaking changes. Statistics consumers will now see `Exact(1)` for `distinct_count` on columns constrained to a single value by filter predicates. Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding. --------- Co-authored-by: xudong.w <wxd963996380@gmail.com>
Rich-T-kid pushed a commit
to Rich-T-kid/datafusion
that referenced
this pull request
Apr 21, 2026
…che#21077) ## Which issue does this PR close? - Part of apache#20766 Related: apache#20789 (uses NDV for equality filter selectivity, complementary - this PR improves the NDV output stats, that PR consumes them) ## Rationale for this change When a filter predicate collapses a column interval to a single value (e.g. `d_qoy = 1`), the output column can only have one distinct value. Currently `distinct_count` is always demoted to `Inexact`, losing this information. This matters for downstream optimizers that rely on `distinct_count`, such as join cardinality estimation in `estimate_inner_join_cardinality`. ## What changes are included in this PR? In `collect_new_statistics` (filter.rs), when the post-filter interval has `lower == upper` (both non-null), set `distinct_count` to `Precision::Exact(1)` instead of demoting the input NDV to `Inexact`. ## Are these changes tested? Yes, 4 unit tests: - Equality predicate (`a = 42`) -> NDV becomes `Exact(1)` - OR predicate (`a = 42 OR a = 22`) -> interval does not collapse, NDV stays `Inexact` - AND with mixed predicates (`a = 42 AND b > 10 AND c = 7`) -> `a` and `c` get `Exact(1)`, `b` stays `Inexact` - Equality with absent bounds (`a = 42`, no min/max) -> interval analysis still resolves to `Exact(1)` ## Are there any user-facing changes? No breaking changes. Statistics consumers will now see `Exact(1)` for `distinct_count` on columns constrained to a single value by filter predicates. Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding. --------- Co-authored-by: xudong.w <wxd963996380@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related: #20789 (uses NDV for equality filter selectivity, complementary - this PR improves the NDV output stats, that PR consumes them)
Rationale for this change
When a filter predicate collapses a column interval to a single value (e.g.
d_qoy = 1), the output column can only have one distinct value. Currentlydistinct_countis always demoted toInexact, losing this information.This matters for downstream optimizers that rely on
distinct_count, such as join cardinality estimation inestimate_inner_join_cardinality.What changes are included in this PR?
In
collect_new_statistics(filter.rs), when the post-filter interval haslower == upper(both non-null), setdistinct_counttoPrecision::Exact(1)instead of demoting the input NDV toInexact.Are these changes tested?
Yes, 4 unit tests:
a = 42) -> NDV becomesExact(1)a = 42 OR a = 22) -> interval does not collapse, NDV staysInexacta = 42 AND b > 10 AND c = 7) ->aandcgetExact(1),bstaysInexacta = 42, no min/max) -> interval analysis still resolves toExact(1)Are there any user-facing changes?
No breaking changes. Statistics consumers will now see
Exact(1)fordistinct_counton columns constrained to a single value by filter predicates.Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.