Add support for dictionary for approx_distinct by mkleen · Pull Request #24646 · apache/datafusion

mkleen · 2026-08-25T07:36:16Z

Which issue does this PR close?

Relates to Support more types for approx_distinct function #22989 but does not close it. More types are coming.

Rationale for this change

Support the Arrow type Dictionary for approx_distinct

What changes are included in this PR?

Enable HLLAccumulator and HllGroupsAccumulator to support Dictionary
Tests

Are these changes tested?

Yes

Are there any user-facing changes?

Yes, approx_distinct supports now Dictionary but no breaking changes.

codecov-commenter · 2026-08-25T08:02:10Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.59%. Comparing base (5980374) to head (e5132c7).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #24646 +/- ##
==========================================
- Coverage 81.63% 81.59% -0.05% 
==========================================
Files 1123 1123 Lines 410298 410970 +672 Branches 410298 410970 +672 ==========================================
+ Hits 334965 335333 +368 - Misses 55642 55893 +251 - Partials 19691 19744 +53

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mkleen · 2026-08-25T09:12:04Z

@Jefffrey

kumarUjjawal

Thank you @mkleen this looks good overall. I had two comments please check.

kumarUjjawal · 2026-08-25T12:10:01Z

 | DataType::Map(_, _)
 | DataType::Struct(_)
 | DataType::Union(_, _)
+ | DataType::Dictionary(_, _)


Would it work to match on the value type instead, so a dictionary is supported exactly when its values would be? Something like DataType::Dictionary(_, value_type)

Good point. Floats will be unsupported see #23084, So we should exclude them in all container types.

i still feel we can support floats here if we already support it in regular distinct count 🤔
also for reference seems duckdb supports:
memory D select approx_count_distinct(a) fromvalues (1.5::double), (1.5), ('nan'::double), (null), (0) t(a); ┌──────────────────────────┐ │ approx_count_distinct(a) │ │ int64 │ ├──────────────────────────┤ │ 3 │ └──────────────────────────┘

Ok makes sense. I will look into this.

The unconditional arm accepts dictionaries whose value type remains intentionally unsupported.
For example, Dictionary(Int32, Float64) reaches HLLAccumulator, whose hashing path supports floats, although plain Float64 returns NotImplemented. Grouped aggregation also bypasses the restriction through GroupsAccumulatorAdapter.
We can validate the value type recursively with a shared supported-type predicate and add a negative float-dictionary regression.

kumarUjjawal · 2026-08-25T12:31:48Z

 | DataType::Map(_, _)
 | DataType::Struct(_)
 | DataType::Union(_, _)
+ | DataType::Dictionary(_, _)


would it make sense to have both dictionary arms on the value type, so Dictionary(_, value_type) is accepted only when is_hll_groups_type(value_type) is true?

this is a good point

That's a good idea. Thank you!

Rich-T-kid

you beat me to it #24731 😆.

kumarUjjawal · 2026-09-04T03:36:01Z

 /// [`HllGroupsAccumulator`]. The fixed-domain types (booleans / small ints) and
 /// `Null` fall back to the per-group [`Accumulator`] path.
 fn is_hll_groups_type(data_type: &DataType) -> bool {
+ if let DataType::Dictionary(_, value_type) = data_type {


Recursing here returns false for dictionaries containing Boolean, small integers, or Null, because those plain types use specialized scalar accumulators. The grouped executor then creates one HLLAccumulator per group through GroupsAccumulatorAdapter; each accumulator embeds a 16 KiB sketch. A dictionary-encoded Boolean with 100,000 groups can therefore consume about 1.5 GiB for sketches alone. Please route every supported dictionary through HllGroupsAccumulator, or provide a compact dictionary-aware fallback, and add a path-sensitive regression for a fixed-domain dictionary.

kumarUjjawal · 2026-09-04T03:38:36Z

 | DataType::Map(_, _)
 | DataType::Struct(_)
 | DataType::Union(_, _)
+ | DataType::Dictionary(_, _)


The unconditional arm accepts dictionaries whose value type remains intentionally unsupported.
For example, Dictionary(Int32, Float64) reaches HLLAccumulator, whose hashing path supports floats, although plain Float64 returns NotImplemented. Grouped aggregation also bypasses the restriction through GroupsAccumulatorAdapter.
We can validate the value type recursively with a shared supported-type predicate and add a negative float-dictionary regression.

mkleen · 2026-09-04T10:00:05Z

@kumarUjjawal Thanks for the review. Could you do one more pass please?

kumarUjjawal

Thank you @mkleen for all the iterations.

…24646) Resolves merge conflict between HEAD's type-specific HLL accumulators and the upstream dictionary support commit. Implements dictionary handling via a DictionaryAccumulator wrapper that casts to the value type before delegating to the appropriate inner accumulator. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…anch [cherry-pick] Add support for dictionary for approx_distinct (apache#24646) Co-authored-by: Rich-T-kid <richard.baah@datadoghq.com> Co-authored-by: mkleen <mkleen@gmail.com>

…anch-55 [Cherry-pick] Add support for dictionary for approx_distinct (apache#24646)

github-actionsBot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Aug 25, 2026

mkleen marked this pull request as ready for review August 25, 2026 07:58

kumarUjjawal reviewed Aug 25, 2026
View reviewed changes

nuno-faria mentioned this pull request Aug 27, 2026
feat: support dictionary in approx_distinct #24731
Closed

Rich-T-kid approved these changes Aug 27, 2026
View reviewed changes

mkleen force-pushed the dictionary_approx_distinct branch 2 times, most recently from f2593d7 to a3520c7Compare September 4, 2026 01:45

mkleen added 2 commits September 4, 2026 03:46

Add support for dictionary for approx_distinct
d21556c

Check if value_type is group type for dictionary
5f60729

mkleen force-pushed the dictionary_approx_distinct branch from a3520c7 to 5f60729Compare September 4, 2026 01:46

mkleen requested review from Jefffrey and kumarUjjawal September 4, 2026 01:53

Fix fmt
9f9aa29

mkleen force-pushed the dictionary_approx_distinct branch from c898a95 to 9f9aa29Compare September 4, 2026 02:39

kumarUjjawal reviewed Sep 4, 2026
View reviewed changes

Validate dictionary value type for approx_distinct
e5132c7

mkleen requested a review from kumarUjjawal September 4, 2026 10:00

kumarUjjawal approved these changes Sep 4, 2026
View reviewed changes

mkleen mentioned this pull request Sep 4, 2026
Support more types for approx_distinct function #22989
Open

kumarUjjawal added this pull request to the merge queue Sep 4, 2026

Merged via the queue into apache:main with commit 39d36f0Sep 4, 2026
38 checks passed

Rich-T-kid mentioned this pull request Sep 4, 2026
[cherry-pick] Add support for dictionary for approx_distinct (#24646) DataDog/datafusion#175
Merged

Rich-T-kid added a commit to DataDog/datafusion that referenced this pull request Sep 4, 2026
Merge pull request#177from DataDog/richard.baah/cherry-pick-dict-br…
3a95ddd
…anch-55 [Cherry-pick] Add support for dictionary for approx_distinct (apache#24646)

Uh oh!

Conversation

mkleen commented Aug 25, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Aug 25, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mkleen commented Aug 25, 2026

Uh oh!

kumarUjjawal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkleenAug 25, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rich-T-kid left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkleen commented Sep 4, 2026

Uh oh!

kumarUjjawal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Aug 25, 2026•
edited
Loading

mkleenAug 25, 2026•
edited
Loading