IN LIST: centralize primitive filters and optimize Decimal128 by geoffreyclaude · Pull Request #24283 · apache/datafusion

geoffreyclaude · 2026-08-12T10:07:26Z

Which issue does this PR close?

Part of Further improve performance of IN list evaluation #19241.
Stacked on #23014.
Next in stack: #24102.

Rationale for this change

#23014 adds fast direct comparisons for short primitive IN lists. For longer lists, DataFusion uses a bitmap, a hash set, or another specialized filter when one exists, and otherwise uses the general filter.

#24102 needs to make that same choice after reading FixedSizeBinary values as same-width primitive keys. Today, the choice is part of the native primitive path, so #24102 would have to repeat its limits and fallback rules.

This PR moves that choice beside the primitive filters and makes it reusable. The limits count only non-null list values. A separate commit uses one PrimitiveHashSetFilter<T, K> for the existing integer and floating-point hash-set paths. Integers keep their native keys; Float32 and Float64 keep their bitwise keys, including distinct signed zeros and NaN encodings. Filter selection and SQL behavior are unchanged.

The only new lookup path is for Decimal128. Lists with at most four non-null values still use direct comparisons. Larger lists now use a hash set over the stored i128 values instead of the general Arrow filter. Values are not rescaled, and the existing precision, scale, and null rules are unchanged. #24102 also uses this path for 16-byte FixedSizeBinary values.

What changes are included in this PR?

Makes primitive filter selection reusable by IN LIST: reuse primitive filters for FixedSizeBinary #24102.
Uses one primitive hash-set implementation for integer and floating-point values, while preserving their existing key semantics.
Uses PrimitiveHashSetFilter<Decimal128Type> for Decimal128 lists with more than four non-null values.
Adds four Decimal128 benchmark cases at list sizes 5 and 64.

Are these changes tested?

Tests cover the direct-comparison boundary, an all-null list, and bitwise float hash keys, including signed zero and distinct NaN encodings. Existing end-to-end Decimal tests cover hits, misses, nulls, IN, and NOT IN.

Are there any user-facing changes?

No. SQL results and public APIs are unchanged.

Benchmark snapshot

The benchmark uses list size 5, the first size above the direct-comparison limit, and a larger size of 64. Each size is measured with 0% and 50% matches. Filter construction is outside the timed loop. Lower is better.

An automated run compared this PR's benchmark-only first commit with the completed implementation. The benchmark source is identical in both states, so the comparison isolates the production changes.

All four cases improved, with a 34.2% geometric-mean reduction in execution time.

Benchmark	Before	After	Change
`primitive/decimal128/large_list/list=5/match=0%`	20.8 us	13.5 us	-35.1% (1.54x faster)
`primitive/decimal128/large_list/list=5/match=50%`	34.3 us	26.6 us	-22.4% (1.29x faster)
`primitive/decimal128/large_list/list=64/match=0%`	20.3 us	13.2 us	-35.0% (1.54x faster)
`primitive/decimal128/large_list/list=64/match=50%`	34.8 us	19.9 us	-42.8% (1.75x faster)

codecov-commenter · 2026-08-12T10:45:59Z

Codecov Report

❌ Patch coverage is 73.33333% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.14%. Comparing base (426b351) to head (510bf3e).
⚠️ Report is 106 commits behind head on main.

Files with missing lines	Patch %	Lines
...l-expr/src/expressions/in_list/primitive_filter.rs	73.79%	23 Missing and 15 partials ⚠️
.../physical-expr/src/expressions/in_list/strategy.rs	0.00%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #24283 +/- ##
==========================================
+ Coverage 80.91% 81.14% +0.23% 
==========================================
Files 1102 1112 +10 Lines 377102 386940 +9838 Branches 377102 386940 +9838 ==========================================
+ Hits 305143 313998 +8855 - Misses 53769 54439 +670 - Partials 18190 18503 +313

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Use PrimitiveHashSetFilter<T, K> for integer and floating-point arrays. Integer filters keep their native key type by default; Float32 and Float64 use their existing bitwise wrapper keys, preserving signed-zero and NaN-payload behavior. The key type and conversion are selected statically, with no function pointer or dynamic dispatch in the lookup loop. Filter selection, dictionary handling, null handling, IN, and NOT IN behavior remain unchanged. Decimal128 routing remains in the following commit.

Route Decimal128 lists above the branchless cutoff through PrimitiveHashSetFilter, replacing the ArrayStaticFilter fallback.

geoffreyclaude · 2026-08-14T12:43:36Z

run benchmark in_list_strategy
baseline:
ref: 8649e80

adriangbot · 2026-08-14T12:46:32Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5293417054-1597-nk866 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing codex/in-list-primitive-selector-decimal (510bf3e) to 8649e80diff

Run configuration

run benchmark in_list_strategybaseline:
ref: "8649e80c239c68d141af4ec655c5132ea2660169"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-08-14T13:35:35Z

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing codex/in-list-primitive-selector-decimal (510bf3e) to 8649e80diff

Run configuration

run benchmark in_list_strategybaseline:
ref: "8649e80c239c68d141af4ec655c5132ea2660169"

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

group HEAD codex_in-list-primitive-selector-decimal
----- ---- ----------------------------------------
dictionary/i32/dict=10/list=16 1.00 7.8±0.01µs ? ?/sec 1.00 7.8±0.01µs ? ?/sec
dictionary/i32/dict=100/list=16 1.00 7.9±0.00µs ? ?/sec 1.00 7.9±0.01µs ? ?/sec
dictionary/i32/dict=100/list=16/NOT_IN 1.00 8.0±0.01µs ? ?/sec 1.00 8.1±0.01µs ? ?/sec
dictionary/i32/dict=100/list=4 1.00 7.9±0.02µs ? ?/sec 1.00 7.9±0.01µs ? ?/sec
dictionary/i32/dict=100/list=64 1.00 7.9±0.00µs ? ?/sec 1.00 7.9±0.00µs ? ?/sec
dictionary/i32/dict=1000/list=16 1.00 8.9±0.01µs ? ?/sec 1.00 8.9±0.01µs ? ?/sec
dictionary/utf8_long/dict=100/list=16 1.00 8.4±0.01µs ? ?/sec 1.00 8.4±0.01µs ? ?/sec
dictionary/utf8_short/dict=50/list=32 1.00 8.2±0.01µs ? ?/sec 1.00 8.3±0.01µs ? ?/sec
dictionary/utf8_short/dict=50/list=8 1.00 8.1±0.01µs ? ?/sec 1.00 8.1±0.01µs ? ?/sec
dictionary/utf8_short/dict=500/list=20 1.00 9.9±0.01µs ? ?/sec 1.00 9.9±0.01µs ? ?/sec
f32/large_list/list=64/match=0% 1.01 16.1±0.01µs ? ?/sec 1.00 16.0±0.01µs ? ?/sec
f32/large_list/list=64/match=50% 1.45 26.5±0.54µs ? ?/sec 1.00 18.3±0.13µs ? ?/sec
f32/small_list/list=32/match=0% 1.00 16.4±0.01µs ? ?/sec 1.00 16.4±0.01µs ? ?/sec
f32/small_list/list=32/match=50% 1.00 16.4±0.02µs ? ?/sec 1.00 16.4±0.01µs ? ?/sec
f32/small_list/list=4/match=0% 1.00 3.5±0.00µs ? ?/sec 1.00 3.5±0.00µs ? ?/sec
f32/small_list/list=4/match=50% 1.00 3.6±0.00µs ? ?/sec 1.00 3.5±0.00µs ? ?/sec
fixed_size_binary/fsb16/list=10000/match=0% 1.00 27.6±0.03µs ? ?/sec 1.01 28.1±0.05µs ? ?/sec
fixed_size_binary/fsb16/list=10000/match=50% 1.00 54.4±0.40µs ? ?/sec 1.04 56.8±0.21µs ? ?/sec
fixed_size_binary/fsb16/list=256/match=0% 1.00 26.5±0.02µs ? ?/sec 1.01 26.8±0.06µs ? ?/sec
fixed_size_binary/fsb16/list=256/match=50% 1.00 49.0±0.44µs ? ?/sec 1.06 51.9±0.47µs ? ?/sec
fixed_size_binary/fsb16/list=4/match=0% 1.00 26.0±0.04µs ? ?/sec 1.02 26.4±0.09µs ? ?/sec
fixed_size_binary/fsb16/list=4/match=50% 1.00 54.4±0.72µs ? ?/sec 1.01 55.1±0.79µs ? ?/sec
fixed_size_binary/fsb16/list=64/match=0% 1.00 25.8±0.02µs ? ?/sec 1.02 26.2±0.03µs ? ?/sec
fixed_size_binary/fsb16/list=64/match=50% 1.02 55.3±0.59µs ? ?/sec 1.00 54.0±0.45µs ? ?/sec
interval_month_day_nano/small_list/list=4/match=0% 1.21 14.7±0.01µs ? ?/sec 1.00 12.2±0.01µs ? ?/sec
interval_month_day_nano/small_list/list=4/match=50% 1.09 21.0±0.08µs ? ?/sec 1.00 19.2±0.11µs ? ?/sec
narrow_integer/f16/list=256/match=0% 1.00 5.3±0.00µs ? ?/sec 1.00 5.3±0.00µs ? ?/sec
narrow_integer/f16/list=256/match=50% 1.00 5.3±0.00µs ? ?/sec 1.00 5.3±0.00µs ? ?/sec
narrow_integer/f16/list=4/match=0% 1.00 3.7±0.00µs ? ?/sec 1.00 3.7±0.00µs ? ?/sec
narrow_integer/f16/list=4/match=50% 1.00 3.7±0.00µs ? ?/sec 1.00 3.7±0.00µs ? ?/sec
narrow_integer/f16/list=64/match=0% 1.00 5.3±0.00µs ? ?/sec 1.00 5.3±0.00µs ? ?/sec
narrow_integer/f16/list=64/match=50% 1.00 5.3±0.00µs ? ?/sec 1.00 5.3±0.00µs ? ?/sec
narrow_integer/i16/list=256/match=0% 1.00 5.3±0.00µs ? ?/sec 1.00 5.3±0.00µs ? ?/sec
narrow_integer/i16/list=256/match=50% 1.00 5.3±0.00µs ? ?/sec 1.00 5.3±0.01µs ? ?/sec
narrow_integer/i16/list=4/match=0% 1.00 3.7±0.01µs ? ?/sec 1.00 3.7±0.00µs ? ?/sec
narrow_integer/i16/list=4/match=50% 1.00 3.7±0.00µs ? ?/sec 1.00 3.7±0.00µs ? ?/sec
narrow_integer/i16/list=64/match=0% 1.00 5.3±0.00µs ? ?/sec 1.00 5.3±0.00µs ? ?/sec
narrow_integer/i16/list=64/match=50% 1.00 5.3±0.00µs ? ?/sec 1.00 5.3±0.00µs ? ?/sec
narrow_integer/u8/list=16/match=0% 1.00 4.7±0.00µs ? ?/sec 1.00 4.7±0.01µs ? ?/sec
narrow_integer/u8/list=16/match=50% 1.00 4.7±0.00µs ? ?/sec 1.00 4.7±0.00µs ? ?/sec
narrow_integer/u8/list=4/match=0% 1.00 3.4±0.01µs ? ?/sec 1.00 3.4±0.02µs ? ?/sec
narrow_integer/u8/list=4/match=50% 1.00 3.4±0.01µs ? ?/sec 1.00 3.4±0.02µs ? ?/sec
nulls/narrow_integer/u8/list=16/match=50%/nulls=20% 1.00 4.8±0.00µs ? ?/sec 1.00 4.8±0.01µs ? ?/sec
nulls/primitive/i32/large_list/list=64/match=50%/nulls=20% 1.00 18.5±2.61µs ? ?/sec 1.18 21.9±0.55µs ? ?/sec
nulls/primitive/i32/small_list/list=16/match=50%/nulls=20% 1.00 9.2±0.02µs ? ?/sec 1.00 9.2±0.02µs ? ?/sec
nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN 1.00 9.3±0.02µs ? ?/sec 1.00 9.3±0.02µs ? ?/sec
nulls/primitive/i32/small_list/list=16/match=50%/nulls=50% 1.00 9.3±0.02µs ? ?/sec 1.00 9.2±0.02µs ? ?/sec
nulls/utf8/long_24b/list=16/match=50%/nulls=20% 1.00 70.9±0.40µs ? ?/sec 1.01 71.5±0.37µs ? ?/sec
nulls/utf8/short_8b/list=16/match=50%/nulls=20% 1.00 61.6±0.41µs ? ?/sec 1.01 62.5±0.59µs ? ?/sec
nulls/utf8view/long_24b/list=16/match=50%/nulls=20% 1.00 86.4±0.18µs ? ?/sec 1.02 88.1±0.13µs ? ?/sec
nulls/utf8view/short_8b/list=16/match=50%/nulls=20% 1.01 38.1±0.23µs ? ?/sec 1.00 37.7±0.14µs ? ?/sec
nulls/utf8view/short_8b/list=16/match=50%/nulls=20%/NOT_IN 1.03 39.1±1.51µs ? ?/sec 1.00 38.0±0.18µs ? ?/sec
nulls/utf8view/short_8b/list=16/match=50%/nulls=50% 1.00 28.9±0.09µs ? ?/sec 1.03 29.8±0.22µs ? ?/sec
primitive/decimal128/large_list/list=5/match=0% 1.54 20.8±0.02µs ? ?/sec 1.00 13.5±0.09µs ? ?/sec
primitive/decimal128/large_list/list=5/match=50% 1.29 34.3±0.23µs ? ?/sec 1.00 26.6±0.23µs ? ?/sec
primitive/decimal128/large_list/list=64/match=0% 1.54 20.3±0.01µs ? ?/sec 1.00 13.2±0.07µs ? ?/sec
primitive/decimal128/large_list/list=64/match=50% 1.75 34.8±0.21µs ? ?/sec 1.00 19.9±3.36µs ? ?/sec
primitive/i32/large_list/list=256/match=0% 1.00 12.3±0.02µs ? ?/sec 1.23 15.1±0.09µs ? ?/sec
primitive/i32/large_list/list=256/match=50% 1.00 15.6±0.07µs ? ?/sec 1.47 23.0±0.30µs ? ?/sec
primitive/i32/large_list/list=64/match=0% 1.00 12.7±0.02µs ? ?/sec 1.19 15.0±0.01µs ? ?/sec
primitive/i32/large_list/list=64/match=50% 1.28 22.3±0.64µs ? ?/sec 1.00 17.4±0.25µs ? ?/sec
primitive/i32/small_list/list=16/match=50%/NOT_IN 1.00 9.2±0.02µs ? ?/sec 1.00 9.2±0.01µs ? ?/sec
primitive/i32/small_list/list=32/match=0% 1.00 16.4±0.01µs ? ?/sec 1.00 16.4±0.02µs ? ?/sec
primitive/i32/small_list/list=32/match=50% 1.00 16.4±0.01µs ? ?/sec 1.00 16.4±0.02µs ? ?/sec
primitive/i32/small_list/list=4/match=0% 1.00 3.5±0.00µs ? ?/sec 1.00 3.5±0.00µs ? ?/sec
primitive/i32/small_list/list=4/match=50% 1.00 3.5±0.00µs ? ?/sec 1.00 3.5±0.00µs ? ?/sec
primitive/i64/large_list/list=128/match=0% 1.10 15.3±0.02µs ? ?/sec 1.00 13.8±0.08µs ? ?/sec
primitive/i64/large_list/list=128/match=50% 1.00 23.4±2.00µs ? ?/sec 1.01 23.6±0.17µs ? ?/sec
primitive/i64/large_list/list=32/match=0% 1.23 15.5±0.03µs ? ?/sec 1.00 12.6±0.03µs ? ?/sec
primitive/i64/large_list/list=32/match=50% 1.47 24.4±0.45µs ? ?/sec 1.00 16.6±0.43µs ? ?/sec
primitive/i64/small_list/list=16/match=0% 1.00 16.7±0.05µs ? ?/sec 1.00 16.7±0.02µs ? ?/sec
primitive/i64/small_list/list=16/match=50% 1.00 16.7±0.02µs ? ?/sec 1.00 16.7±0.02µs ? ?/sec
primitive/i64/small_list/list=4/match=0% 1.00 4.4±0.00µs ? ?/sec 1.00 4.4±0.00µs ? ?/sec
primitive/i64/small_list/list=4/match=50% 1.00 4.4±0.01µs ? ?/sec 1.00 4.4±0.00µs ? ?/sec
timestamp_ns/large_list/list=32/match=0% 1.00 20.3±0.02µs ? ?/sec 1.00 20.3±0.01µs ? ?/sec
timestamp_ns/large_list/list=32/match=50% 1.11 34.0±0.37µs ? ?/sec 1.00 30.7±0.15µs ? ?/sec
timestamp_ns/small_list/list=16/match=0% 1.00 16.6±0.02µs ? ?/sec 1.00 16.7±0.01µs ? ?/sec
timestamp_ns/small_list/list=16/match=50% 1.00 16.6±0.01µs ? ?/sec 1.00 16.6±0.01µs ? ?/sec
timestamp_ns/small_list/list=4/match=0% 1.00 4.4±0.01µs ? ?/sec 1.00 4.4±0.00µs ? ?/sec
timestamp_ns/small_list/list=4/match=50% 1.00 4.4±0.00µs ? ?/sec 1.00 4.4±0.00µs ? ?/sec
utf8/long_24b/list=256/match=0% 1.00 36.4±0.02µs ? ?/sec 1.00 36.2±0.02µs ? ?/sec
utf8/long_24b/list=256/match=50% 1.00 69.9±0.23µs ? ?/sec 1.02 71.6±0.20µs ? ?/sec
utf8/long_24b/list=4/match=0% 1.04 38.0±0.62µs ? ?/sec 1.00 36.5±0.38µs ? ?/sec
utf8/long_24b/list=4/match=50% 1.00 71.5±0.37µs ? ?/sec 1.03 73.3±0.31µs ? ?/sec
utf8/long_24b/list=64/match=0% 1.00 36.5±0.03µs ? ?/sec 1.00 36.6±0.36µs ? ?/sec
utf8/long_24b/list=64/match=50% 1.00 70.1±0.25µs ? ?/sec 1.03 72.3±0.57µs ? ?/sec
utf8/mixed_len/list=16/match=0% 1.00 35.8±0.54µs ? ?/sec 1.09 38.8±0.12µs ? ?/sec
utf8/mixed_len/list=16/match=50% 1.00 94.9±2.28µs ? ?/sec 1.03 97.6±0.62µs ? ?/sec
utf8/mixed_len/list=64/match=0% 1.00 40.1±0.11µs ? ?/sec 1.02 40.7±0.08µs ? ?/sec
utf8/mixed_len/list=64/match=50% 1.00 102.7±0.87µs ? ?/sec 1.01 103.9±0.82µs ? ?/sec
utf8/shared_prefix/pfx=12/list=32/match=50% 1.00 70.7±0.23µs ? ?/sec 1.02 71.9±0.25µs ? ?/sec
utf8/short_8b/list=16/match=50%/NOT_IN 1.00 64.6±0.58µs ? ?/sec 1.01 65.2±0.45µs ? ?/sec
utf8/short_8b/list=256/match=0% 1.00 29.0±0.02µs ? ?/sec 1.00 29.1±0.03µs ? ?/sec
utf8/short_8b/list=256/match=50% 1.03 66.3±0.29µs ? ?/sec 1.00 64.1±0.52µs ? ?/sec
utf8/short_8b/list=4/match=0% 1.00 29.6±0.03µs ? ?/sec 1.00 29.5±0.02µs ? ?/sec
utf8/short_8b/list=4/match=50% 1.01 65.6±0.64µs ? ?/sec 1.00 65.2±0.36µs ? ?/sec
utf8/short_8b/list=64/match=0% 1.02 29.6±0.55µs ? ?/sec 1.00 29.2±0.26µs ? ?/sec
utf8/short_8b/list=64/match=50% 1.05 66.2±0.55µs ? ?/sec 1.00 63.1±0.68µs ? ?/sec
utf8view/len_12b/list=16/match=0% 1.00 21.1±0.08µs ? ?/sec 1.00 21.2±0.08µs ? ?/sec
utf8view/len_12b/list=16/match=50% 1.03 41.0±0.23µs ? ?/sec 1.00 39.9±0.34µs ? ?/sec
utf8view/len_12b/list=64/match=0% 1.00 21.1±0.06µs ? ?/sec 1.01 21.4±0.05µs ? ?/sec
utf8view/len_12b/list=64/match=50% 1.02 39.8±0.17µs ? ?/sec 1.00 38.9±0.24µs ? ?/sec
utf8view/long_24b/list=16/match=0% 1.00 43.4±0.04µs ? ?/sec 1.01 43.7±0.04µs ? ?/sec
utf8view/long_24b/list=16/match=50% 1.00 85.2±0.28µs ? ?/sec 1.00 85.5±0.22µs ? ?/sec
utf8view/long_24b/list=256/match=0% 1.00 43.1±0.13µs ? ?/sec 1.00 42.9±0.02µs ? ?/sec
utf8view/long_24b/list=256/match=50% 1.00 85.0±0.31µs ? ?/sec 1.00 85.4±0.32µs ? ?/sec
utf8view/long_24b/list=4/match=0% 1.00 44.5±0.20µs ? ?/sec 1.00 44.6±0.17µs ? ?/sec
utf8view/long_24b/list=4/match=50% 1.00 85.4±0.36µs ? ?/sec 1.01 86.1±0.23µs ? ?/sec
utf8view/long_24b/list=64/match=0% 1.00 43.2±0.04µs ? ?/sec 1.00 43.1±0.08µs ? ?/sec
utf8view/long_24b/list=64/match=50% 1.00 85.0±0.17µs ? ?/sec 1.00 84.9±0.42µs ? ?/sec
utf8view/mixed_len/list=16/match=0% 1.00 34.2±0.10µs ? ?/sec 1.06 36.2±0.13µs ? ?/sec
utf8view/mixed_len/list=16/match=50% 1.00 65.5±0.28µs ? ?/sec 1.28 84.1±0.78µs ? ?/sec
utf8view/mixed_len/list=64/match=0% 1.05 37.8±0.15µs ? ?/sec 1.00 36.2±0.15µs ? ?/sec
utf8view/mixed_len/list=64/match=50% 1.00 80.2±0.42µs ? ?/sec 1.16 92.7±1.09µs ? ?/sec
utf8view/shared_prefix/pfx=12/list=32/match=0% 1.00 45.2±0.04µs ? ?/sec 1.02 46.0±0.10µs ? ?/sec
utf8view/shared_prefix/pfx=12/list=32/match=50% 1.00 84.2±0.48µs ? ?/sec 1.04 87.5±0.46µs ? ?/sec
utf8view/shared_prefix/pfx=16/list=64/match=0% 1.00 43.0±0.08µs ? ?/sec 1.00 43.2±0.04µs ? ?/sec
utf8view/shared_prefix/pfx=16/list=64/match=50% 1.01 85.6±0.28µs ? ?/sec 1.00 85.0±0.16µs ? ?/sec
utf8view/shared_prefix/pfx=8/list=16/match=0% 1.00 32.6±0.03µs ? ?/sec 1.02 33.3±0.20µs ? ?/sec
utf8view/shared_prefix/pfx=8/list=16/match=50% 1.00 72.6±0.49µs ? ?/sec 1.01 73.6±0.14µs ? ?/sec
utf8view/short_8b/list=16/match=0% 1.00 20.7±0.02µs ? ?/sec 1.01 20.8±0.07µs ? ?/sec
utf8view/short_8b/list=16/match=50% 1.05 40.1±0.26µs ? ?/sec 1.00 38.3±0.43µs ? ?/sec
utf8view/short_8b/list=256/match=0% 1.01 20.9±0.37µs ? ?/sec 1.00 20.7±0.01µs ? ?/sec
utf8view/short_8b/list=256/match=50% 1.02 38.8±0.32µs ? ?/sec 1.00 37.9±0.25µs ? ?/sec
utf8view/short_8b/list=4/match=0% 1.00 21.0±0.12µs ? ?/sec 1.00 21.1±0.03µs ? ?/sec
utf8view/short_8b/list=4/match=50% 1.02 40.0±0.18µs ? ?/sec 1.00 39.2±0.22µs ? ?/sec
utf8view/short_8b/list=64/match=0% 1.00 20.9±0.06µs ? ?/sec 1.00 20.9±0.02µs ? ?/sec
utf8view/short_8b/list=64/match=50% 1.04 39.1±0.36µs ? ?/sec 1.00 37.5±0.21µs ? ?/sec

Resource Usage

in_list_strategy — base (merge-base)

Metric	Value
Wall time	1475.3s
Peak memory	41.9 MiB
Avg memory	30.3 MiB
CPU user	1598.2s
CPU sys	1.4s
Peak spill	0 B

in_list_strategy — branch

Metric	Value
Wall time	1455.3s
Peak memory	40.9 MiB
Avg memory	29.7 MiB
CPU user	1580.9s
CPU sys	1.4s
Peak spill	0 B

File an issue against this benchmark runner

Add Decimal128 IN LIST benchmarks
8649e80

github-actionsBot added the physical-expr Changes to the physical-expr crates label Aug 12, 2026

This was referenced Aug 12, 2026
IN LIST: reuse primitive filters for FixedSizeBinary #24102
Draft
IN LIST: optimize Utf8View and BinaryView filters #24088
Draft
IN LIST: add branchless filter for small primitive lists #23014
Merged
Further improve performance of IN list evaluation #19241
Open

geoffreyclaude force-pushed the codex/in-list-primitive-selector-decimal branch 7 times, most recently from 00885e0 to 65b1e9fCompare August 13, 2026 10:14

geoffreyclaude added 4 commits August 13, 2026 12:40

Reuse shared IN LIST result construction
fd2fed3

Move primitive IN LIST selection beside primitive filters
bfe1394

Optimize large Decimal128 IN LIST filters
510bf3e
Route Decimal128 lists above the branchless cutoff through PrimitiveHashSetFilter, replacing the ArrayStaticFilter fallback.

geoffreyclaude force-pushed the codex/in-list-primitive-selector-decimal branch from 65b1e9f to 510bf3eCompare August 13, 2026 10:46

geoffreyclaude marked this pull request as ready for review August 14, 2026 11:15

alamb added the performance Make DataFusion faster label Aug 14, 2026

Navigation Menu

Uh oh!

IN LIST: centralize primitive filters and optimize Decimal128 - #24283

IN LIST: centralize primitive filters and optimize Decimal128#24283
geoffreyclaude wants to merge 5 commits into
apache:mainfrom
geoffreyclaude:codex/in-list-primitive-selector-decimal

geoffreyclaude commented Aug 12, 2026•
edited
Loading

Uh oh!

codecov-commenter commented Aug 12, 2026•
edited
Loading

Uh oh!

geoffreyclaude commented Aug 14, 2026

Uh oh!

adriangbot commented Aug 14, 2026

Uh oh!

adriangbot commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

geoffreyclaude commented Aug 12, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Benchmark snapshot

Uh oh!

codecov-commenter commented Aug 12, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

geoffreyclaude commented Aug 14, 2026

Uh oh!

adriangbot commented Aug 14, 2026

Uh oh!

adriangbot commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

geoffreyclaude commented Aug 12, 2026•
edited
Loading

codecov-commenter commented Aug 12, 2026•
edited
Loading