Add FixedSizeBinary support for MultiGroupBy by maxburke · Pull Request #23646 · apache/datafusion

maxburke · 2026-07-16T17:11:03Z

ClosesMulti-Group-By does not support FixedSizeBinary types #23645
part of EPIC: complete GroupValuesColumn type coverage (nested types + remaining primitives) #22715
ClosesEPIC: complete GroupValuesColumn type coverage (nested types + remaining primitives) #22715

Rationale for this change

Multi-Group-By has cases for regular Binary/LargeBinary types, but not FixedSizeBinary

Are these changes tested?

Yes.

Are there any user-facing changes?

No

codecov-commenter · 2026-07-16T17:39:44Z

Codecov Report

❌ Patch coverage is 98.92761% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.07%. Comparing base (24483db) to head (128ef37).
⚠️ Report is 44 commits behind head on main.

Files with missing lines	Patch %	Lines
...s/group_values/multi_group_by/fixed_size_binary.rs	98.71%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #23646 +/- ##
==========================================
+ Coverage 81.05% 81.07% +0.01% 
==========================================
Files 1106 1107 +1 Lines 380556 380929 +373 Branches 380556 380929 +373 ==========================================
+ Hits 308477 308842 +365 - Misses 53861 53865 +4 - Partials 18218 18222 +4

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

alamb

Thanks @maxburke -- what is the use case for this query? Is it that you are grouping on multiple columns and if that also includes a fixed size binary the performance drops off?

alamb · 2026-07-16T18:42:47Z

Related discussion

feat(physical-plan): generic Rows-backed GroupColumn keeps mixed schemas on the column-wise path #23523

alamb · 2026-07-16T18:46:00Z

There is also this PR

feat(physical-plan): generic Rows-backed GroupColumn keeps mixed schemas on the column-wise path #23523

However I don't think that will handle FixedSizeBinary

Rich-T-kid

This PR looks very reasonable. I noticed that this is very similar to the ByteGroupValueBuilder implementation.
I think it would be worth it to try and see if we can modify ByteGroupValueBuilder to support similar optimizations such as batched writes. besides that vectorized/equal_to and the other trait methods are 1-1.
If ByteGroupValueBuilder cant be expanded on to include fixed size binary without extensive branching that hurts readability thats also fine.

Rich-T-kid · 2026-07-16T19:14:05Z

+
+impl GroupColumn for FixedSizeBinaryGroupValueBuilder {
+ fn equal_to(&self, lhs_row: usize, array: &ArrayRef, rhs_row: usize) -> bool {
+ debug_assert!(matches!(array.data_type(), DataType::FixedSizeBinary(_)));


the array input is guaranteed to be the correct type. you can remove the assertion here

Rich-T-kid · 2026-07-16T19:17:05Z

+ rhs_rows: &[usize],
+ equal_to_results: &mut BooleanBufferBuilder,
+ ) {
+ debug_assert!(matches!(array.data_type(), DataType::FixedSizeBinary(_)));


similar point to above

alamb · 2026-07-16T19:35:09Z

This PR looks very reasonable. I noticed that this is very similar to the ByteGroupValueBuilder implementation. I think it would be worth it to try and see if we can modify ByteGroupValueBuilder to support similar optimizations such as batched writes. besides that vectorized/equal_to and the other trait methods are 1-1. If ByteGroupValueBuilder cant be expanded on to include fixed size binary without extensive branching that hurts readability thats also fine.

This sounds like a good thing to explore in a follow on / parallel PR

alamb · 2026-07-16T19:35:48Z

Since I think the point of doing this PR is to improve performance it would be nice to have some benchmark results

I created a PR with some benchmarks here

bench: add FixedSizeBinary coverage to multi_group_by benchmark #23650

Hopefully we can merge that and then use it to validate that this PR improves things

maxburke · 2026-07-16T20:47:25Z

Thanks @maxburke -- what is the use case for this query? Is it that you are grouping on multiple columns and if that also includes a fixed size binary the performance drops off?

More so that if we're grouping by a FixedSizeBinary column we run out of memory in the GroupValuesRows interning and crash :)

…he#23650) ## Which issue does this PR close? - Related to apache#23646 - Related to apache#23645. ## Rationale for this change The point of a specialized FixedSizeBinary group values is performance but we have no performance benchmark for it. ## What changes are included in this PR? Adds a `fixed_size_binary` experiment to `datafusion/physical-plan/benches/multi_group_by.rs` Run with: ```bash cargo bench -p datafusion-physical-plan --bench multi_group_by --features test_utils -- fixed_size_binary ``` ## Are these changes tested? This is benchmark-only. The benchmark compiles on `main` and runs end-to-end on top of apache#23646. No product code changes. ## Are there any user-facing changes? No. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

alamb · 2026-07-17T18:44:11Z

Can you please merge this PR up from main (so I can run the benchmarks using the bot?) I can't do it myself b/c I don't have write permissions int he urbanlogiq repo

maxburke · 2026-07-17T18:47:57Z

Can you please merge this PR up from main (so I can run the benchmarks using the bot?) I can't do it myself b/c I don't have write permissions int he urbanlogiq repo

done!

alamb · 2026-08-03T20:49:56Z

run benchmark multi_group_by

adriangbot · 2026-08-03T20:52:31Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5171560183-1360-vphx4 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing multi-group-by-fsb (5eb8979) to 4957f5d (merge-base) diff

Run configuration

run benchmark multi_group_bu

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-08-03T20:52:53Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5171561354-1361-qb869 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing multi-group-by-fsb (5eb8979) to 4957f5d (merge-base) diff

Run configuration

run benchmark multi_group_by

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-08-03T20:53:04Z

Benchmark for this request failed.

Run configuration

run benchmark multi_group_by

Last 20 lines of output:

Click to expand

 Downloaded hmac v0.13.0
Downloaded sysinfo v0.39.6
Downloaded serde_json v1.0.150
Downloaded icu_collections v2.2.0
Downloaded substrait v0.63.0
Downloaded parquet v59.1.0
Downloaded zstd-sys v2.0.16+zstd.1.5.7
Downloaded time v0.3.47
Downloaded unicode-width v0.2.2
Downloaded typify-impl v0.5.0
Downloaded aws-lc-sys v0.40.0
Downloaded tokio v1.52.3
Downloaded ring v0.17.14
Downloaded liblzma-sys v0.4.6
Downloaded linux-raw-sys v0.12.1
Downloaded sha2-const-stable v0.1.0
error: target `multi_group_by` in package `datafusion-physical-plan` requires the features: `test_utils`
Consider enabling them by passing, e.g., `--features="test_utils"`
error: target `multi_group_by` in package `datafusion-physical-plan` requires the features: `test_utils`
Consider enabling them by passing, e.g., `--features="test_utils"`

File an issue against this benchmark runner

adriangbot · 2026-08-03T21:09:30Z

Benchmark for this request failed.

Run configuration

run benchmark multi_group_bu

Last 20 lines of output:

Click to expand

# Micro-Benchmarks (specific operators and features)
cancellation: How long cancelling a query takes
nlj: Benchmark for simple nested loop joins, testing various join scenarios
hj: Benchmark for simple hash joins, testing various join scenarios
smj: Benchmark for simple sort merge joins, testing various join scenarios
dict: Benchmark for dictionary-encoded group-by scenarios
compile_profile: Compile and execute TPC-H across selected Cargo profiles, reporting timing and binary size
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Supported Configuration (Environment Variables)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
DATA_DIR directory to store datasets
CARGO_COMMAND command that runs the benchmark binary
DATAFUSION_DIR directory to use (default /workspace/datafusion-base)
RESULTS_NAME folder where the benchmark files are stored
PREFER_HASH_JOIN Prefer hash join algorithm (default true)
SIMULATE_LATENCY Simulate object store latency to mimic S3 (default false)
DATAFUSION_* Set the given datafusion configuration

File an issue against this benchmark runner

alamb · 2026-08-03T21:22:22Z

run benchmark multi_group_by

adriangbot · 2026-08-03T21:25:42Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5171878692-1362-wpk62 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing multi-group-by-fsb (5eb8979) to 4957f5d (merge-base) diff

Run configuration

run benchmark multi_group_by

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-08-03T21:25:53Z

Benchmark for this request failed.

Run configuration

run benchmark multi_group_by

Last 20 lines of output:

Click to expand

 Downloaded rustls v0.23.39
Downloaded tokio-stream v0.1.18
Downloaded zerocopy v0.8.48
Downloaded regex-syntax v0.8.11
Downloaded csv v1.4.0
Downloaded regex-automata v0.4.14
Downloaded time v0.3.47
Downloaded rustix v1.1.4
Downloaded tokio v1.52.3
Downloaded liblzma-sys v0.4.6
Downloaded zstd-sys v2.0.16+zstd.1.5.7
Downloaded toml_datetime v1.1.1+spec-1.1.0
Downloaded regress v0.10.5
Downloaded linux-raw-sys v0.12.1
Downloaded aws-lc-sys v0.40.0
Downloaded sha2-const-stable v0.1.0
error: target `multi_group_by` in package `datafusion-physical-plan` requires the features: `test_utils`
Consider enabling them by passing, e.g., `--features="test_utils"`
error: target `multi_group_by` in package `datafusion-physical-plan` requires the features: `test_utils`
Consider enabling them by passing, e.g., `--features="test_utils"`

File an issue against this benchmark runner

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

alamb

Thanks @maxburke and @Rich-T-kid

I reviewed the code and it looks good to me

I also did some ad-hoc performance testing with it like this:

Made a datafile like this

COPY (
SELECT
arrow_cast(decode(md5((value % 5000000)::varchar), 'hex'), 'FixedSizeBinary(16)') AS uuid,
(value % 10) AS id
FROM generate_series(1, 200000000)
) TO 'uuids.parquet';

This looks like this

andrewlamb@Andrews-MacBook-Pro-3:~/Downloads$ du -s -h uuids.parquet
3.0G	uuids.parquet

>select*from'uuids.parquet'limit10;
+----------------------------------+----+
| uuid | id |
+----------------------------------+----+
| ae3fcf8bfebbc45af3bf12f70d9d3acd | 5 |
| c94adce9d37cb5fde98966e278815c64 | 6 |
| 2bc049eddc1314ec3bee9509f935741b | 7 |
| 76b22168b7e87edca43468290be51218 | 8 |
| aa59f82ba5a34a3d3c25773d8a948e99 | 9 |
| 657aa03184c164a30af703a02e2a687b | 0 |
| 491f0cbcf7b88ac92d2adff2a0528d81 | 1 |
| 17da0cbc4ffef7f04b4e3a2283160fd7 | 2 |
| 81a1f57e643d79dfcee6cd4117d43c30 | 3 |
| f0b301dad7ddd5eb3ea3d854823cc798 | 4 |
+----------------------------------+----+10 row(s) fetched.
Elapsed 0.021 seconds.

And then ran this query:

SELECTcount(*) FROM (SELECT uuid, id FROM'uuids.parquet'GROUP BY uuid, id);

On main: 1.132 seconds.
With this PR: 1.073 seconds.

(this was a very noisy system)

alamb · 2026-08-10T19:14:48Z

+ ) {
+ let array = array.as_fixed_size_binary();
+
+ for (idx, (&lhs_row, &rhs_row)) in


As a follow on, this can likely be optimized more -- for example, we could have a special case loop for when the inputs are known to have no nulls (likely a common case for things like UUIDs)
We could also move to using get_unchecked to skip the bounds check and try to make this lookup loop even more performant

I filed this ticket to track
Improved performance for FixedSizeBinary grouping #24267

alamb · 2026-08-10T19:15:38Z

+ Nulls::Some
+ };
+
+ match all_null_or_non_null {


this is nice

alamb · 2026-08-11T15:04:34Z

Thanks again @maxburke

maxburke · 2026-08-11T15:40:16Z

You're welcome!

…he#23650) ## Which issue does this PR close? - Related to apache#23646 - Related to apache#23645. ## Rationale for this change The point of a specialized FixedSizeBinary group values is performance but we have no performance benchmark for it. ## What changes are included in this PR? Adds a `fixed_size_binary` experiment to `datafusion/physical-plan/benches/multi_group_by.rs` Run with: ```bash cargo bench -p datafusion-physical-plan --bench multi_group_by --features test_utils -- fixed_size_binary ``` ## Are these changes tested? This is benchmark-only. The benchmark compiles on `main` and runs end-to-end on top of apache#23646. No product code changes. ## Are there any user-facing changes? No. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Closesapache#23645 - part of apache#22715 ## Rationale for this change Multi-Group-By has cases for regular Binary/LargeBinary types, but not FixedSizeBinary ## Are these changes tested? Yes. ## Are there any user-facing changes? No Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

github-actionsBot added sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Jul 16, 2026

alamb reviewed Jul 16, 2026
View reviewed changes

alamb mentioned this pull request Jul 16, 2026
bench: add FixedSizeBinary coverage to multi_group_by benchmark #23650
Merged

Rich-T-kid reviewed Jul 16, 2026
View reviewed changes

maxburke force-pushed the multi-group-by-fsb branch from e843517 to f8ebf73Compare July 16, 2026 20:49

Implement GroupColumn for fixed size binary types
5eb8979

maxburke force-pushed the multi-group-by-fsb branch from 3970070 to 5eb8979Compare July 17, 2026 18:54

This comment was marked as outdated.
Sign in to view

maxburkeand others added 2 commits August 5, 2026 11:40

Merge remote-tracking branch 'origin/main' into multi-group-by-fsb
1b27f34
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Merge branch 'main' into multi-group-by-fsb
128ef37

alamb mentioned this pull request Aug 10, 2026
EPIC: complete GroupValuesColumn type coverage (nested types + remaining primitives) #22715
Closed
7 tasks

alamb approved these changes Aug 10, 2026
View reviewed changes

alamb mentioned this pull request Aug 11, 2026
Improved performance for FixedSizeBinary grouping #24267
Open

alamb added this pull request to the merge queue Aug 11, 2026

Merged via the queue into apache:main with commit 3f0a953Aug 11, 2026
42 checks passed

Uh oh!

Conversation

maxburke commented Jul 16, 2026• edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Are these changes tested?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Jul 16, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

Rich-T-kid left a comment

Choose a reason for hiding this comment

Uh oh!

Rich-T-kidJul 16, 2026

Choose a reason for hiding this comment

Uh oh!

maxburkeJul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Rich-T-kidJul 16, 2026

Choose a reason for hiding this comment

Uh oh!

maxburkeJul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

maxburke commented Jul 16, 2026

Uh oh!

alamb commented Jul 17, 2026

Uh oh!

maxburke commented Jul 17, 2026

Uh oh!

This comment was marked as outdated.

alamb commented Aug 3, 2026

Uh oh!

adriangbot commented Aug 3, 2026

Uh oh!

adriangbot commented Aug 3, 2026

Uh oh!

adriangbot commented Aug 3, 2026

Uh oh!

adriangbot commented Aug 3, 2026

Uh oh!

alamb commented Aug 3, 2026

Uh oh!

adriangbot commented Aug 3, 2026

Uh oh!

adriangbot commented Aug 3, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alambAug 10, 2026

Choose a reason for hiding this comment

Uh oh!

alambAug 11, 2026

Choose a reason for hiding this comment

Uh oh!

alambAug 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb commented Aug 11, 2026

Uh oh!

maxburke commented Aug 11, 2026

Uh oh!

Reviewers

maxburke commented Jul 16, 2026•
edited by alamb
Loading

codecov-commenter commented Jul 16, 2026•
edited
Loading