Feat: add dictionaries as a supported group column type by Rich-T-kid · Pull Request #23187 · apache/datafusion

Rich-T-kid · 2026-06-25T13:52:52Z

Which issue does this PR close?

works towards closing feat: add GroupColumn support for List / Struct / Map / FixedSizeList in multi-column GROUP BY #22682.
replacement for Optimize Dictionary groupings #21765
- Optimize Dictionary groupings #21765 (review)

Rationale for this change

This PR introduces a specialized GroupColumn implementation for dictionary-typed columns inside GroupValuesColumn, allowing dictionary columns to participate in the columnar, vectorized aggregation path instead of the row-based fallback.

The Implementation is only about 175+ lines of code. the remaining LOC is adding extensive test at the GroupColumn trait level as well as testing the GroupValuesColumn GroupValues trait and how it inter-opts with multi-dictionary group by's.

What changes are included in this PR?

Adds a DictionaryGroupValueBuilder struct implementing the GroupColumn trait for Dictionary-typed group-by columns, supporting a configurable subset of value types
Extends the type-check gate in GroupValuesColumn::try_new (the matches! block) to accept Dictionary(_, value_type) where value_type is already supported.
Adds schema-level support so emitted dictionary group key columns round-trip through the output schema correctly
removes casting thats done for each dictionary array in emit

Are these changes tested?

yes. a majority of this PR is test

Are there any user-facing changes?

no. this is a pure perf boost for users.

Rich-T-kid · 2026-07-01T04:28:17Z

@kumarUjjawal could you run the dictionary benchmarks on this PR? Thx

Rich-T-kid · 2026-07-01T05:08:03Z

 }
 }
+ DataType::Dictionary(key_dt, value_dt) => {
+ let new_field = Field::new("", *value_dt.clone(), true);


Since this field is never read again it may be fine to ignore the name field.
should be weary of similar issues to #21765 (comment)

Kind of annoying that make_group_column takes a field instead of a DataType. Maybe we can change that in a follow up PR?

Rich-T-kid · 2026-07-01T13:33:07Z

@kumarUjjawal wanted to bump this 😄

geoffreyclaude · 2026-07-01T15:06:25Z

run benchmark dictionary_group_values

adriangbot · 2026-07-01T15:09:30Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4856707692-778-smmmp 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing rich-T-kid/dictionary-groupValuesColumn-impl (eb41915) to 01bf68c (merge-base) diff using: dictionary_group_values
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-07-01T15:21:31Z

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

group HEAD rich-T-kid_dictionary-groupValuesColumn-impl
----- ---- --------------------------------------------
dict_intern_emit/intern_emit/size_65536_card_1000_null_0.00 1.15 869.8±5.08µs 71.9 MElem/sec 1.00 756.8±139.44µs 82.6 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_20_null_0.00 1.22 779.9±5.80µs 80.1 MElem/sec 1.00 636.7±43.49µs 98.2 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_300_null_0.00 1.17 809.2±5.84µs 77.2 MElem/sec 1.00 692.8±108.32µs 90.2 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_65536_null_0.00 2.45 6.3±0.01ms 10.0 MElem/sec 1.00 2.6±0.04ms 24.5 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_75_null_0.00 1.24 800.2±6.35µs 78.1 MElem/sec 1.00 645.5±6.09µs 96.8 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_1000_null_0.00 1.42 160.1±0.87µs 48.8 MElem/sec 1.00 112.7±15.66µs 69.3 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_20_null_0.00 1.28 104.0±1.39µs 75.1 MElem/sec 1.00 81.5±0.45µs 95.8 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_300_null_0.00 1.30 121.2±1.05µs 64.5 MElem/sec 1.00 92.9±9.63µs 84.1 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_75_null_0.00 1.17 109.7±1.26µs 71.2 MElem/sec 1.00 94.1±13.82µs 83.0 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_8192_null_0.00 2.66 678.0±2.09µs 11.5 MElem/sec 1.00 255.3±12.37µs 30.6 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_1000_null_0.10 1.04 4.4±0.02ms 57.4 MElem/sec 1.00 4.2±0.01ms 59.7 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_20_null_0.10 1.02 4.1±0.02ms 60.5 MElem/sec 1.00 4.1±0.01ms 61.5 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_300_null_0.10 1.01 4.2±0.02ms 59.2 MElem/sec 1.00 4.2±0.01ms 60.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_65536_null_0.10 1.54 16.5±0.05ms 15.2 MElem/sec 1.00 10.7±0.06ms 23.4 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_75_null_0.10 1.01 4.2±0.02ms 59.3 MElem/sec 1.00 4.2±0.01ms 60.2 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_1000_null_0.10 1.13 618.7±3.27µs 50.5 MElem/sec 1.00 549.9±6.40µs 56.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_20_null_0.10 1.00 512.1±3.06µs 61.0 MElem/sec 1.00 511.2±3.10µs 61.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_300_null_0.10 1.05 550.0±3.63µs 56.8 MElem/sec 1.00 522.5±2.60µs 59.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_75_null_0.10 1.02 526.6±2.85µs 59.3 MElem/sec 1.00 515.0±2.45µs 60.7 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_8192_null_0.10 1.74 1479.0±6.20µs 21.1 MElem/sec 1.00 847.9±5.26µs 36.9 MElem/sec

Resource Usage

dictionary_group_values — base (merge-base)

Metric	Value
Wall time	380.1s
Peak memory	592.6 MiB
Avg memory	94.8 MiB
CPU user	227.9s
CPU sys	14.8s
Peak spill	0 B

dictionary_group_values — branch

Metric	Value
Wall time	330.1s
Peak memory	464.6 MiB
Avg memory	67.9 MiB
CPU user	225.8s
CPU sys	19.2s
Peak spill	0 B

File an issue against this benchmark runner

Rich-T-kid · 2026-07-01T18:55:54Z

@codex review

kumarUjjawal · 2026-07-02T03:17:29Z

@kumarUjjawal wanted to bump this 😄

@Rich-T-kid Thank you! I have been sick so I won't be available for review. I will probably get back next week.

Rich-T-kid · 2026-07-02T04:31:26Z

@Rich-T-kid Thank you! I have been sick so I won't be available for review. I will probably get back next week.

@kumarUjjawal Sorry to hear that. I hope you feel better! no rush on the review!

Rich-T-kid

Revision #2

Rich-T-kid

Revision #3

Rich-T-kid · 2026-07-06T20:06:26Z

Latest PR speeds up the low cardinality case

Rich-T-kid · 2026-07-06T20:07:27Z

@geoffreyclaude could you run the benchmarks command again when you get a chance. Thanks 🚀

geoffreyclaude · 2026-07-06T20:12:42Z

run benchmark dictionary_group_values

adriangbot · 2026-07-06T20:15:36Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4897199089-877-zms2m 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing rich-T-kid/dictionary-groupValuesColumn-impl (3d1e1c9) to 01bf68c (merge-base) diff using: dictionary_group_values
Results will be posted here when complete

File an issue against this benchmark runner

Rich-T-kid

Revision #4 Last revision before making this open for review

adriangbot · 2026-07-06T20:25:58Z

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

group HEAD rich-T-kid_dictionary-groupValuesColumn-impl
----- ---- --------------------------------------------
dict_intern_emit/intern_emit/size_65536_card_1000_null_0.00 1.23 871.4±8.87µs 71.7 MElem/sec 1.00 707.5±5.34µs 88.3 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_20_null_0.00 1.30 790.0±9.68µs 79.1 MElem/sec 1.00 608.0±7.50µs 102.8 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_300_null_0.00 1.23 815.1±10.24µs 76.7 MElem/sec 1.00 665.2±5.90µs 94.0 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_65536_null_0.00 2.51 6.2±0.02ms 10.0 MElem/sec 1.00 2.5±0.01ms 25.1 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_75_null_0.00 1.23 804.8±8.79µs 77.7 MElem/sec 1.00 655.6±9.93µs 95.3 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_1000_null_0.00 1.35 161.5±1.39µs 48.4 MElem/sec 1.00 119.8±17.59µs 65.2 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_20_null_0.00 1.31 104.8±1.68µs 74.5 MElem/sec 1.00 80.0±0.85µs 97.7 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_300_null_0.00 1.34 123.0±1.59µs 63.5 MElem/sec 1.00 92.0±10.05µs 84.9 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_75_null_0.00 1.29 112.0±6.33µs 69.7 MElem/sec 1.00 86.6±30.70µs 90.2 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_8192_null_0.00 2.70 681.0±3.01µs 11.5 MElem/sec 1.00 252.0±1.80µs 31.0 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_1000_null_0.10 1.45 4.4±0.03ms 57.1 MElem/sec 1.00 3.0±0.01ms 82.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_20_null_0.10 1.58 4.2±0.02ms 60.1 MElem/sec 1.00 2.6±0.02ms 94.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_300_null_0.10 1.48 4.2±0.02ms 58.9 MElem/sec 1.00 2.9±0.01ms 87.4 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_65536_null_0.10 1.40 17.0±0.13ms 14.7 MElem/sec 1.00 12.2±0.13ms 20.5 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_75_null_0.10 1.51 4.2±0.03ms 59.1 MElem/sec 1.00 2.8±0.02ms 89.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_1000_null_0.10 1.29 620.6±3.17µs 50.4 MElem/sec 1.00 480.4±1.96µs 65.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_20_null_0.10 1.54 513.2±3.32µs 60.9 MElem/sec 1.00 332.5±5.61µs 94.0 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_300_null_0.10 1.48 549.7±3.56µs 56.8 MElem/sec 1.00 370.5±1.52µs 84.3 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_75_null_0.10 1.53 528.9±3.45µs 59.1 MElem/sec 1.00 344.6±4.98µs 90.7 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_8192_null_0.10 1.52 1486.9±5.41µs 21.0 MElem/sec 1.00 975.5±4.94µs 32.0 MElem/sec

Resource Usage

dictionary_group_values — base (merge-base)

Metric	Value
Wall time	290.1s
Peak memory	587.7 MiB
Avg memory	124.8 MiB
CPU user	227.7s
CPU sys	15.5s
Peak spill	0 B

dictionary_group_values — branch

Metric	Value
Wall time	320.1s
Peak memory	486.4 MiB
Avg memory	71.3 MiB
CPU user	210.2s
CPU sys	15.2s
Peak spill	0 B

File an issue against this benchmark runner

Rich-T-kid · 2026-07-06T20:50:20Z

🤔 benchmarks show good improvement in every case but it should be much larger. Currently every intern() call causes the cache to be rebuilt from the bottom up. Instead of a flat vector it may make sense to use a hashtable that last in-between calls.

Rich-T-kid · 2026-08-04T04:15:49Z

commit 44ad7d1 should address all of your comments except for the take_n() optimization. I think that would make sense as a follow up @kumarUjjawal, happy to hear your thoughts

kumarUjjawal · 2026-08-04T07:11:01Z

run benchmark dictionary_group_values

adriangbot · 2026-08-04T07:13:02Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5175755158-1373-7r6tf 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing rich-T-kid/dictionary-groupValuesColumn-impl (63d4021) to 47ca490 (merge-base) diff

Run configuration

run benchmark dictionary_group_values

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-08-04T07:25:59Z

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/dictionary-groupValuesColumn-impl (63d4021) to 47ca490 (merge-base) diff

Run configuration

run benchmark dictionary_group_values

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

group HEAD rich-T-kid_dictionary-groupValuesColumn-impl
----- ---- --------------------------------------------
dict_intern_emit/intern_emit/size_65536_card_1000_null_0.00 1.73 877.4±8.70µs 71.2 MElem/sec 1.00 506.7±14.15µs 123.3 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_20_null_0.00 1.83 780.9±5.99µs 80.0 MElem/sec 1.00 426.5±6.24µs 146.5 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_300_null_0.00 1.79 817.3±7.97µs 76.5 MElem/sec 1.00 457.4±5.18µs 136.6 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_65536_null_0.00 1.01 6.2±0.06ms 10.1 MElem/sec 1.00 6.1±0.01ms 10.2 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_75_null_0.00 1.81 802.4±5.79µs 77.9 MElem/sec 1.00 444.4±4.42µs 140.7 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_1000_null_0.00 1.26 159.8±1.36µs 48.9 MElem/sec 1.00 127.2±11.00µs 61.4 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_20_null_0.00 1.81 104.4±1.25µs 74.8 MElem/sec 1.00 57.7±0.76µs 135.5 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_300_null_0.00 1.55 121.6±1.27µs 64.3 MElem/sec 1.00 78.3±7.45µs 99.8 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_75_null_0.00 1.75 111.2±6.16µs 70.2 MElem/sec 1.00 63.4±0.62µs 123.2 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_8192_null_0.00 1.27 660.9±2.95µs 11.8 MElem/sec 1.00 519.4±2.52µs 15.0 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_1000_null_0.10 1.80 4.3±0.01ms 58.2 MElem/sec 1.00 2.4±0.00ms 104.9 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_20_null_0.10 1.84 4.1±0.01ms 61.2 MElem/sec 1.00 2.2±0.00ms 112.6 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_300_null_0.10 1.80 4.2±0.01ms 59.9 MElem/sec 1.00 2.3±0.00ms 107.6 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_65536_null_0.10 1.06 17.8±0.13ms 14.0 MElem/sec 1.00 16.8±0.15ms 14.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_75_null_0.10 1.81 4.1±0.01ms 60.3 MElem/sec 1.00 2.3±0.00ms 109.2 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_1000_null_0.10 1.46 599.2±2.48µs 52.2 MElem/sec 1.00 410.6±1.88µs 76.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_20_null_0.10 1.81 498.5±2.66µs 62.7 MElem/sec 1.00 275.1±2.33µs 113.6 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_300_null_0.10 1.67 531.3±3.11µs 58.8 MElem/sec 1.00 317.6±2.21µs 98.4 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_75_null_0.10 1.76 507.7±2.26µs 61.5 MElem/sec 1.00 289.2±1.86µs 108.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_8192_null_0.10 1.13 1467.4±10.66µs 21.3 MElem/sec 1.00 1294.1±6.57µs 24.1 MElem/sec

Resource Usage

dictionary_group_values — base (merge-base)

Metric	Value
Wall time	380.1s
Peak memory	593.5 MiB
Avg memory	87.7 MiB
CPU user	225.7s
CPU sys	15.2s
Peak spill	0 B

dictionary_group_values — branch

Metric	Value
Wall time	385.1s
Peak memory	886.3 MiB
Avg memory	82.9 MiB
CPU user	227.2s
CPU sys	13.4s
Peak spill	0 B

File an issue against this benchmark runner

kumarUjjawal

Thank you @Rich-T-kid

This looks good. Please create a separate issue for the optimization as you mentioned.

Rich-T-kid · 2026-08-04T16:22:35Z

created a follow up issue here #24089

thank you for the reviews @kumarUjjawal @zhuqi-lucas @jayshrivastava 🚀

kumarUjjawal · 2026-08-05T03:48:29Z

I will merge this in 1-2 days in no one else has any objections.

alamb

Thank you for this @Rich-T-kid and @kumarUjjawal

I think it would be really great to show that this works end to end (and improves performance) -- I suspect it does but it would be good to verify

One way we could do this is run TPCH Q1 https://github.com/apache/datafusion/blob/main/benchmarks/queries/q1.sql

You could test this locally like

Make data

tpcgen-cli tpch parquet --scale-factor=10 --tables=lineitem
lineitem [==================] (100%)

Then convert the relevant columns to dictioanry

> COPY (select arrow_cast(l_returnflag, 'Dictionary(Int32, Utf8)') as l_returnflag, arrow_cast(l_linestatus, 'Dictionary(Int32, Utf8)') as l_linestatus, l_quantity, l_extendedprice, l_discount, l_shipdate, l_tax from 'lineitem.parquet') to 'lineitem_dict.parquet';
+----------+
| count |
+----------+
| 59986052 |
+----------+
1 row(s) fetched.
Elapsed 1.767 seconds.

Then run query

>select
l_returnflag,
l_linestatus,
sum(l_quantity) as sum_qty,
sum(l_extendedprice) as sum_base_price,
sum(l_extendedprice * (1- l_discount)) as sum_disc_price,
sum(l_extendedprice * (1- l_discount) * (1+ l_tax)) as sum_charge,
avg(l_quantity) as avg_qty,
avg(l_extendedprice) as avg_price,
avg(l_discount) as avg_disc,
count(*) as count_order
from'lineitem_dict.parquet'where
l_shipdate <=date'1998-09-02'group by
l_returnflag,
l_linestatus
order by
l_returnflag,
l_linestatus;
+--------------+--------------+--------------+------------------+--------------------+----------------------+-----------+--------------+----------+-------------+
| l_returnflag | l_linestatus | sum_qty | sum_base_price | sum_disc_price | sum_charge | avg_qty | avg_price | avg_disc | count_order |
+--------------+--------------+--------------+------------------+--------------------+----------------------+-----------+--------------+----------+-------------+
| A | F | 377518399.00 | 566065727797.25 | 537759104278.0656 | 559276670892.116819 | 25.500975 | 38237.151008 | 0.050006 | 14804077 |
| N | F | 9851614.00 | 14767438399.17 | 14028805792.2114 | 14590490998.366737 | 25.522448 | 38257.810660 | 0.049973 | 385998 |
| N | O | 743124873.00 | 1114302286901.88 | 1058580922144.9638 | 1100937000170.591854 | 25.498075 | 38233.902923 | 0.050000 | 29144351 |
| R | F | 377732830.00 | 566431054976.00 | 538110922664.7677 | 559634780885.086257 | 25.508384 | 38251.219273 | 0.049996 | 14808183 |
+--------------+--------------+--------------+------------------+--------------------+----------------------+-----------+--------------+----------+-------------+4 row(s) fetched.
Elapsed 0.370 seconds.

alamb · 2026-08-05T10:13:49Z

 }
 };

- // TODO: Materialize dictionaries in group keys (#7647)


this is pretty nice

alamb · 2026-08-05T10:24:13Z

Another thing that might be valuable is to make a variant of the tpch benchmarks, where you tell the parquet reader that the low cardinality columns (that would be good for dictionary encoding) should be read as Dictionary... Arrays -- and then run the queries and see how much improvement this PR gives

I actually think that would be helpful in general if your goal is to improve the overall performance of the dictionaries in DataFusion. I will file a ticket with more detail

Rich-T-kid · 2026-08-05T13:51:01Z

@alamb I think the second approach is easier to work with and less error-prone. But from my understanding, this isn't currently possible, or at least isn't exposed in DataFusion. The only config that currently exists is for writing: https://docs.rs/datafusion/latest/datafusion/config/struct.ParquetOptions.html#structfield.dictionary_enabled

here you tell the parquet reader that the low cardinality columns (that would be good for dictionary encoding) should be read as Dictionary... Arrays

I think it'd be nice to expose this as a session prop so users can configure it with SET notation in SQL:

SET datafusion.execution.batch_size = 4096;
SET datafusion.parquet.dictionary_read_enable;
....

The only caveat that comes to mind is this may cause the Parquet reader to read all string columns into dictionary arrays, which isn't what we want for precision.

the concept your referring to is actually quite similar to what I raised in this issue: #22891. Ideally we would be able to determine which columns are low-cardinality and change their type dictionary at physical planning time. with #23187 this should provide a nice perf boost. the only issue would be determining what threshold counts as "low cardinality".

cc @asolimando

Rich-T-kid · 2026-08-05T14:26:13Z

So I think some nice follow ups would be

expose a parquet dictionary encoding read flag to users
allow a way to read only specific string columns as dictionary encoded (this may already be possible I need to check)

by hand we can experiment with transforming low-card string group keys into Dict<k,v> and then #22891 will be in a position to be implemented. it may be possible to use column statistics as a guide for "low cardinality" columns but due to the state of statistics in datafusion this may or may not be accurate. ideally this will be tunable as either a hard set value or a percentage. There are some other issues that come to mind like schema management but I think its best to leave that in #22891's thread

Rich-T-kid · 2026-08-05T15:31:56Z

I ran TPCH Q4 against main and these were my results

tpchgen-cli -s 10 --tables orders --format=parquet

main

datafusion-cli -c "
SELECT o_orderpriority, count(*) AS order_count
FROM 'orders.parquet'
WHERE o_orderdate >= date '1993-07-01'
AND o_orderdate < date '1993-10-01'
AND EXISTS (
SELECT 1 FROM 'lineitem.parquet'
WHERE l_orderkey = o_orderkey AND l_commitdate < l_receiptdate
)
GROUP BY o_orderpriority
ORDER BY o_orderpriority;
"
DataFusion CLI v54.0.0
+-----------------+-------------+
| o_orderpriority | order_count |
+-----------------+-------------+
| 1-URGENT | 105214 |
| 2-HIGH | 104821 |
| 3-MEDIUM | 105227 |
| 4-NOT SPECIFIED | 105422 |
| 5-LOW | 105356 |
+-----------------+-------------+
5 row(s) fetched. Elapsed 0.161 seconds.

cast the columns to dictionary using

datafusion-cli -c "
COPY (
SELECT o_orderkey, o_orderdate,
arrow_cast(o_orderpriority, 'Dictionary(Int32, Utf8)') AS o_orderpriority
FROM 'orders.parquet'
) TO 'orders_dict.parquet';
"

this branch

datafusion-cli -c "
SELECT o_orderpriority, count(*) AS order_count
FROM 'orders_dict.parquet'
WHERE o_orderdate >= date '1993-07-01'
AND o_orderdate < date '1993-10-01'
AND EXISTS (
SELECT 1 FROM 'lineitem.parquet'
WHERE l_orderkey = o_orderkey AND l_commitdate < l_receiptdate
)
GROUP BY o_orderpriority
ORDER BY o_orderpriority;
"
DataFusion CLI v54.0.0
+-----------------+-------------+
| o_orderpriority | order_count |
+-----------------+-------------+
| 1-URGENT | 105214 |
| 2-HIGH | 104821 |
| 3-MEDIUM | 105227 |
| 4-NOT SPECIFIED | 105422 |
| 5-LOW | 105356 |
+-----------------+-------------+
5 row(s) fetched. Elapsed 0.095 seconds.

Rich-T-kid · 2026-08-09T21:44:09Z

just wanted to bump this PR for merge since #24111 , #24089 & #23921 are blocked on this

kumarUjjawal · 2026-08-10T03:38:16Z

just wanted to bump this PR for merge since #24111 , #24089 & #23921 are blocked on this

cc @alamb

Rich-T-kid · 2026-08-15T17:50:52Z

just wanted to bump this PR for merge since #24111 , #24089 & #23921 are blocked on this
cc @alamb

@kumarUjjawal @alamb @zhuqi-lucas

github-actionsBot added the physical-plan Changes to the physical-plan crate label Jun 25, 2026

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch 2 times, most recently from 3f7ff57 to e6b6dceCompare July 1, 2026 04:25

Rich-T-kid commented Jul 1, 2026
View reviewed changes

Comment threaddatafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs

Rich-T-kid commented Jul 1, 2026
View reviewed changes

Comment threaddatafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs Outdated

Rich-T-kid mentioned this pull request Jul 1, 2026
Test: add more aggregation focused dictionary sql logic test #23280
Merged

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch 2 times, most recently from 770abfe to 243a557Compare July 1, 2026 15:32

Rich-T-kid changed the title ~~[Draft][22715] introduce dictionarys as a supported group column type~~Feat:[Draft][22715] add dictionaries as a supported group column typeJul 1, 2026

Rich-T-kid commented Jul 2, 2026
View reviewed changes

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from 4ee52da to 7af7080Compare July 2, 2026 19:18

Rich-T-kid commented Jul 4, 2026
View reviewed changes

Comment threaddatafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs
Comment threaddatafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs Outdated
Comment threaddatafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from 152c1f0 to f3387c5Compare July 5, 2026 03:59

Rich-T-kid mentioned this pull request Jul 6, 2026
feat: add GroupColumn support for List / Struct / Map / FixedSizeList in multi-column GROUP BY #22682
Open

Rich-T-kid commented Jul 6, 2026
View reviewed changes

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from 41192e5 to 44ad7d1Compare August 4, 2026 03:49

kumarUjjawal approved these changes Aug 4, 2026
View reviewed changes

Rich-T-kid mentioned this pull request Aug 4, 2026
Optimize take_n on DictionaryGroupValuesColumn<K> #24089
Open

alamb reviewed Aug 5, 2026
View reviewed changes

This was referenced Aug 5, 2026
Epic: Cost-Based Dictionary Encoding #24111
Open
Optimize hash operations and byte-wise row comparisons for dictionary encoding #21466
Open

Rich-T-kid added 7 commits August 11, 2026 11:20

resolve merge conflicts
edbee62

add test, avoid extra hashes, fix i16 test to use i8
e696b5e

address Pr comments and add test
d43a264

reduce comment bloat
68f8bdc

Fix clippy
f64b4c7

fix null handling & take_n
cb9af06

fix merge conflicts, add test for dictionary overflow, trim LOC
d83d1da

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from 63d4021 to aa8addeCompare August 11, 2026 15:21

fix ci
daaae0b

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from aa8adde to daaae0bCompare August 11, 2026 16:06

trigger CI
7c21620

Uh oh!

Conversation

Rich-T-kid commented Jun 25, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Rich-T-kid commented Jul 1, 2026

Uh oh!

Rich-T-kidJul 1, 2026

Choose a reason for hiding this comment

Uh oh!

jayshrivastavaJul 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Rich-T-kid commented Jul 1, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geoffreyclaude commented Jul 1, 2026

Uh oh!

adriangbot commented Jul 1, 2026

Uh oh!

adriangbot commented Jul 1, 2026

Uh oh!

Rich-T-kid commented Jul 1, 2026

Uh oh!

kumarUjjawal commented Jul 2, 2026

Uh oh!

Rich-T-kid commented Jul 2, 2026

Uh oh!

Rich-T-kid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rich-T-kid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rich-T-kid commented Jul 6, 2026

Uh oh!

Rich-T-kid commented Jul 6, 2026

Uh oh!

geoffreyclaude commented Jul 6, 2026

Uh oh!

adriangbot commented Jul 6, 2026

Uh oh!

Rich-T-kid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adriangbot commented Jul 6, 2026

Uh oh!

Rich-T-kid commented Jul 6, 2026

Uh oh!

Rich-T-kid commented Aug 4, 2026

Uh oh!

kumarUjjawal commented Aug 4, 2026

Uh oh!

adriangbot commented Aug 4, 2026

Uh oh!

adriangbot commented Aug 4, 2026

Uh oh!

kumarUjjawal left a comment

Choose a reason for hiding this comment

Uh oh!

Rich-T-kid commented Aug 4, 2026

Uh oh!

Rich-T-kid commented Jun 25, 2026•
edited
Loading

Rich-T-kid commented Jul 1, 2026•
edited
Loading

Rich-T-kid commented Aug 5, 2026•
edited
Loading