IN LIST: add branchless filter for small primitive lists by geoffreyclaude · Pull Request #23014 · apache/datafusion

geoffreyclaude · 2026-06-18T07:31:47Z

Which issue does this PR close?

Part of Further improve performance of IN list evaluation #19241.
Stacked on #23311.
Next in stack: #24283.
Extracted from Optimize IN performance with specialized implementations #19390.

Rationale for this change

For very small IN lists, building or probing a hash table can be more work than just comparing the input value with each constant.

For example, for x IN (10, 20, 30), the fast path can behave like:

x == 10 OR x == 20 OR x == 30

Because the list is tiny, those comparisons are cheap. The implementation stores the constants in a fixed-size array and checks them with a compact comparison chain.

“Branchless” here means the comparisons are combined without stopping at the first match. That can be faster for these small fixed-width lists because the CPU gets a predictable sequence of simple operations instead of hash-table setup and probe logic.

For primitive values that are not already plain unsigned integers, this PR keeps the logical Arrow type explicit and uses a matching same-width comparison representation only inside the branchless filter. For example, Float16 uses UInt16 storage, Float32 uses UInt32 storage, and TimestampNanosecond uses UInt64 storage. Decimal128 and IntervalMonthDayNano use their own 16-byte native representation. This preserves bit-pattern equality while relying on Arrow's native primitive compatibility rules: timestamp timezone metadata and Decimal128 precision/scale metadata may differ, while incompatible primitive representations remain rejected.

What changes are included in this PR?

Adds a const-generic BranchlessFilter for small primitive IN lists.
Adds thresholds for when this path is used:
- up to 16 values for 1-byte types
- up to 8 values for 2-byte types
- up to 32 values for 4-byte types
- up to 16 values for 8-byte types
- up to 4 values for 16-byte types
Keeps dispatch concrete and explicit in strategy.rs.
Maps each optimized logical type to the comparison representation used by the branchless filter:
- Int8 -> UInt8
- Int16, Float16 -> UInt16
- Int32, Float32, Date32, Time32 -> UInt32
- Int64, Float64, Date64, Time64, Timestamp, Duration -> UInt64
- Decimal128, IntervalMonthDayNano -> their native 16-byte representation
Leaves larger 1-byte and 2-byte lists on the existing bitmap filters.
Leaves larger 4-byte and 8-byte lists on the existing hash/generic paths.
Leaves wider primitive types such as Decimal256 and unsupported complex types on the generic path.
Keeps the same IN / NOT IN null behavior as the rest of the stack.
Adds tests for branchless null handling, signed boundary values, slices, Float16/Float32/Float64 bit patterns, compatible timestamp/Decimal128 metadata, incompatible timestamp units, IntervalMonthDayNano values, and same-width wrong-type probe rejection.

Are these changes tested?

Yes.

cargo fmt --all -- --check
cargo test -p datafusion-physical-expr expressions::in_list --lib
cargo test -p datafusion-physical-expr --bench in_list_strategy --no-run
cargo clippy --all-targets --all-features -- -D warnings

Are there any user-facing changes?

No. SQL results and public APIs are unchanged.

Local benchmark snapshot

Built and run with release-nonlto; results below include only small primitive-list rows:

cargo bench -p datafusion-physical-expr --profile release-nonlto --bench in_list_strategy -- <filter> --save-baseline <baseline>

Filters used: narrow_integer, primitive/i32/small_list, primitive/i64/small_list, f32/small_list, timestamp_ns/small_list, and interval_month_day_nano/small_list.

Method: directly compared Criterion's raw sample minima (min(time / iterations)) from sample.json. Lower is better; changes within +/-5% are treated as noise.

Compared baselines: #23311 -> #23014

Summary: 28 of 39 rows were faster, none were slower, and 11 were within +/-5%.

Largest changes:

Benchmark	Before	After	Change
`timestamp_ns/small_list/list=4/match=50%`	46.55 us	3.17 us	-93.2% (14.69x faster)
`f32/small_list/list=4/match=50%`	33.93 us	3.04 us	-91.0% (11.15x faster)
`primitive/i32/small_list/list=4/match=50%`	32.63 us	3.08 us	-90.5% (10.58x faster)
`primitive/i64/small_list/list=4/match=50%`	33.55 us	3.18 us	-90.5% (10.54x faster)
`timestamp_ns/small_list/list=4/match=0%`	19.57 us	3.18 us	-83.8% (6.16x faster)
`f32/small_list/list=4/match=0%`	18.14 us	3.05 us	-83.2% (5.95x faster)
`primitive/i32/small_list/list=4/match=0%`	17.00 us	3.04 us	-82.1% (5.59x faster)
`primitive/i64/small_list/list=4/match=0%`	17.12 us	3.22 us	-81.2% (5.31x faster)
`primitive/i32/small_list/list=16/match=50%/NOT_IN`	31.98 us	7.26 us	-77.3% (4.41x faster)
`nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%`	29.35 us	7.32 us	-75.1% (4.01x faster)
`timestamp_ns/small_list/list=16/match=50%`	45.32 us	11.79 us	-74.0% (3.84x faster)
`nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%`	25.89 us	7.31 us	-71.8% (3.54x faster)
`nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN`	26.05 us	7.42 us	-71.5% (3.51x faster)
`interval_month_day_nano/small_list/list=4/match=50%`	52.94 us	15.52 us	-70.7% (3.41x faster)
`f32/small_list/list=32/match=50%`	38.78 us	13.27 us	-65.8% (2.92x faster)
`primitive/i64/small_list/list=16/match=50%`	29.46 us	11.76 us	-60.1% (2.50x faster)

Full table (39 rows)

Benchmark	Before	After	Change
`narrow_integer/u8/list=4/match=0%`	3.86 us	2.79 us	-27.8% (1.38x faster)
`narrow_integer/u8/list=4/match=50%`	3.84 us	2.78 us	-27.7% (1.38x faster)
`narrow_integer/u8/list=16/match=0%`	3.88 us	3.85 us	-0.8% (within noise)
`narrow_integer/u8/list=16/match=50%`	3.84 us	3.86 us	+0.5% (within noise)
`narrow_integer/i16/list=4/match=0%`	3.93 us	3.18 us	-19.1% (1.24x faster)
`narrow_integer/i16/list=4/match=50%`	3.92 us	3.16 us	-19.5% (1.24x faster)
`narrow_integer/i16/list=64/match=0%`	3.96 us	3.82 us	-3.5% (within noise)
`narrow_integer/i16/list=64/match=50%`	3.91 us	3.80 us	-2.9% (within noise)
`narrow_integer/i16/list=256/match=0%`	3.90 us	3.81 us	-2.5% (within noise)
`narrow_integer/i16/list=256/match=50%`	3.97 us	3.81 us	-4.1% (within noise)
`narrow_integer/f16/list=4/match=0%`	3.87 us	3.16 us	-18.5% (1.23x faster)
`narrow_integer/f16/list=4/match=50%`	3.94 us	3.15 us	-20.2% (1.25x faster)
`narrow_integer/f16/list=64/match=0%`	3.87 us	3.84 us	-0.6% (within noise)
`narrow_integer/f16/list=64/match=50%`	3.93 us	3.85 us	-1.9% (within noise)
`narrow_integer/f16/list=256/match=0%`	3.90 us	3.84 us	-1.5% (within noise)
`narrow_integer/f16/list=256/match=50%`	3.87 us	3.91 us	+1.2% (within noise)
`nulls/narrow_integer/u8/list=16/match=50%/nulls=20%`	3.92 us	4.02 us	+2.5% (within noise)
`primitive/i32/small_list/list=4/match=0%`	17.00 us	3.04 us	-82.1% (5.59x faster)
`primitive/i32/small_list/list=4/match=50%`	32.63 us	3.08 us	-90.5% (10.58x faster)
`primitive/i32/small_list/list=32/match=0%`	16.34 us	13.33 us	-18.5% (1.23x faster)
`primitive/i32/small_list/list=32/match=50%`	31.17 us	13.31 us	-57.3% (2.34x faster)
`primitive/i32/small_list/list=16/match=50%/NOT_IN`	31.98 us	7.26 us	-77.3% (4.41x faster)
`nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%`	29.35 us	7.32 us	-75.1% (4.01x faster)
`nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN`	26.05 us	7.42 us	-71.5% (3.51x faster)
`nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%`	25.89 us	7.31 us	-71.8% (3.54x faster)
`primitive/i64/small_list/list=4/match=0%`	17.12 us	3.22 us	-81.2% (5.31x faster)
`primitive/i64/small_list/list=4/match=50%`	33.55 us	3.18 us	-90.5% (10.54x faster)
`primitive/i64/small_list/list=16/match=0%`	16.34 us	11.93 us	-27.0% (1.37x faster)
`primitive/i64/small_list/list=16/match=50%`	29.46 us	11.76 us	-60.1% (2.50x faster)
`f32/small_list/list=4/match=0%`	18.14 us	3.05 us	-83.2% (5.95x faster)
`f32/small_list/list=4/match=50%`	33.93 us	3.05 us	-91.0% (11.15x faster)
`f32/small_list/list=32/match=0%`	22.05 us	13.43 us	-39.1% (1.64x faster)
`f32/small_list/list=32/match=50%`	38.78 us	13.27 us	-65.8% (2.92x faster)
`timestamp_ns/small_list/list=4/match=0%`	19.57 us	3.18 us	-83.8% (6.16x faster)
`timestamp_ns/small_list/list=4/match=50%`	46.55 us	3.17 us	-93.2% (14.69x faster)
`timestamp_ns/small_list/list=16/match=0%`	19.73 us	12.07 us	-38.8% (1.63x faster)
`timestamp_ns/small_list/list=16/match=50%`	45.32 us	11.79 us	-74.0% (3.84x faster)
`interval_month_day_nano/small_list/list=4/match=0%`	20.12 us	13.20 us	-34.4% (1.52x faster)
`interval_month_day_nano/small_list/list=4/match=50%`	52.94 us	15.52 us	-70.7% (3.41x faster)

alamb · 2026-07-22T15:10:02Z

I will review this today

alamb · 2026-07-22T16:25:40Z

run benchmark in_list_strategy

adriangbot · 2026-07-22T16:27:45Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5048702402-1228-grpkv 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing perf/in_list_branchless_filter (06f2b6b) to f33dcec (merge-base) diff using: in_list_strategy
Results will be posted here when complete

File an issue against this benchmark runner

alamb

Thank you @geoffreyclaude and @kosiew

My main concern about this Pr is that it generates a LOT of new code (by instantiating several hundred BranchlessFilter) -- by Claude Code's estiamte it is "~507 distinct BranchlessFilter<T, N> types, e" and by my measurement increases the code size by 4-6.7 percent

I think if we templated on the physical type (e.g. i32) rather than all the logical types we could reduce the number of distinct types substantially, but fundamentally creating many specialized code paths for the different sizes seems like a very expensive way to increase performance

Release binary size is 5MB (6.7%) larger and debug binary is 17MB (4%) larger

One way we could potentially reduce the code size would be to minimize the parts of the code that really need specializations. For example, I think the actual storage values could be a runtime value, and then just optimize the inner compare loop:

Compare the values 8 at a time or something using chunks_exact and then just generate a special comparison for 7,6,5,4,3,2,1 values to catch the last value.

That could also make this optimization more applicable as well, to any size where we wanted implement branchless (brute force comparison)

Details

Release

# release
cargo build --release --bin datafusion-cli
# debug
cargo build --bin datafusion-cli

Size of datafusion-cli on main

 74M	/Users/andrewlamb/Downloads/datafusion-main-base-in-list

Size on this branch

 79M	/Users/andrewlamb/Downloads/datafusion-in-list

Debug size

main

412M	target/debug/datafusion-cli

this branch

429M	target/debug/datafusion-cli

alamb · 2026-07-22T16:34:47Z

+branchless_filter_type!(UInt16Type, UInt16Type, BRANCHLESS_MAX_2B);
+branchless_filter_type!(Float16Type, UInt16Type, BRANCHLESS_MAX_2B);
+
+branchless_filter_type!(Int32Type, UInt32Type, BRANCHLESS_MAX_4B);


const BRANCHLESS_MAX_4B: usize = 32;
do I read the code above that this is creating 32 specializations (just) for UInt32 IN lists

Yes, that was correct for the previous version: UInt32 alone generated 32 copies of the full BranchlessFilter 😬.
The follow-up removes the list length from BranchlessFilter, so there is now only one full filter per type. Only the small comparison function still varies by length. I put the detailed size and performance results in my reply to the overall review.

alamb · 2026-07-22T16:38:34Z

+ };
+ }
+
+ match in_array.data_type() {


i found it somewhat hard to follow by reading the code to understand what cases would use the branchless filter and what would use the bitmap filter for small integer types -- because the cases that are covered by branchless filters are behind some macros and then the bitmap filtering happens only if there isn't a matchess branchless filter
I am not sure there is a concrete action to take from this observation

Good point. The latest refactor removed the per-length dispatch from this module, but it's still not super clear, I admit.
For the small types, Int8/UInt8 use direct comparisons for up to 16 non-null values, and Int16/UInt16/Float16 for up to 8. Longer lists fall back to the bitmap filter. I added a short comment to make that explicit...

alamb · 2026-07-22T16:45:10Z

+ };
+}
+
+branchless_filter_type!(Int8Type, UInt8Type, BRANCHLESS_MAX_1B);


Given we now have fast bitmap filter implementations for 8 and 16 bit integer types, what extra value do branchless filters give over those bitmaps?
UNless the performance win is compelling, I suggest we simply use the bitmap implementation for those types

Branchless is still faster (1.5x) for small lists (eg, 4 values). Since the latest commits resolves the compilation size issue, I think the branchless path is worth keeping for short 8 and 16 bit type lists, with the bitmap filter remaining the fallback for larger ones.

alamb · 2026-07-22T16:49:53Z

+ // The branchless filter reads the same Arrow value buffer as the
+ // comparison type. That is only valid when both native types have the
+ // same width, so catch any bad mapping here at compile time.
+ const _: () = assert!(


my reading of this loop is that it instantiates max_len copies of the branchless filter

Yes, exactly: this loop was the main source of the code-size increase.
After the follow-up it still creates one small comparison function per supported length, but it no longer creates max_len copies of the whole filter. The rest of the filter is shared now; I included the resulting size and benchmark numbers in my reply to the overall review.

adriangbot · 2026-07-22T17:18:48Z

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

group HEAD perf_in_list_branchless_filter
----- ---- ------------------------------
dictionary/i32/dict=10/list=16 1.00 7.6±0.01µs ? ?/sec 1.02 7.7±0.01µs ? ?/sec
dictionary/i32/dict=100/list=16 1.00 7.7±0.01µs ? ?/sec 1.01 7.8±0.01µs ? ?/sec
dictionary/i32/dict=100/list=16/NOT_IN 1.00 7.7±0.02µs ? ?/sec 1.02 7.9±0.01µs ? ?/sec
dictionary/i32/dict=100/list=4 1.00 7.8±0.01µs ? ?/sec 1.00 7.8±0.01µs ? ?/sec
dictionary/i32/dict=100/list=64 1.00 7.8±0.01µs ? ?/sec 1.00 7.7±0.00µs ? ?/sec
dictionary/i32/dict=1000/list=16 1.03 9.1±0.01µs ? ?/sec 1.00 8.8±0.01µs ? ?/sec
dictionary/utf8_long/dict=100/list=16 1.00 8.3±0.01µs ? ?/sec 1.00 8.3±0.01µs ? ?/sec
dictionary/utf8_short/dict=50/list=32 1.00 8.1±0.01µs ? ?/sec 1.00 8.1±0.01µs ? ?/sec
dictionary/utf8_short/dict=50/list=8 1.00 8.0±0.01µs ? ?/sec 1.00 8.0±0.01µs ? ?/sec
dictionary/utf8_short/dict=500/list=20 1.00 9.6±0.01µs ? ?/sec 1.00 9.6±0.01µs ? ?/sec
f32/large_list/list=64/match=0% 1.00 15.2±0.01µs ? ?/sec 1.06 16.2±0.01µs ? ?/sec
f32/large_list/list=64/match=50% 1.03 24.7±0.20µs ? ?/sec 1.00 24.1±0.42µs ? ?/sec
f32/small_list/list=32/match=0% 1.00 15.4±0.01µs ? ?/sec 1.00 15.5±0.01µs ? ?/sec
f32/small_list/list=32/match=50% 1.50 23.2±0.38µs ? ?/sec 1.00 15.5±0.01µs ? ?/sec
f32/small_list/list=4/match=0% 4.38 15.3±0.01µs ? ?/sec 1.00 3.5±0.00µs ? ?/sec
f32/small_list/list=4/match=50% 7.79 27.2±0.33µs ? ?/sec 1.00 3.5±0.00µs ? ?/sec
fixed_size_binary/fsb16/list=10000/match=0% 1.01 25.6±0.37µs ? ?/sec 1.00 25.3±0.05µs ? ?/sec
fixed_size_binary/fsb16/list=10000/match=50% 1.00 53.1±0.14µs ? ?/sec 1.02 54.3±0.32µs ? ?/sec
fixed_size_binary/fsb16/list=256/match=0% 1.00 23.9±0.21µs ? ?/sec 1.00 23.9±0.03µs ? ?/sec
fixed_size_binary/fsb16/list=256/match=50% 1.00 48.7±0.35µs ? ?/sec 1.01 49.2±0.32µs ? ?/sec
fixed_size_binary/fsb16/list=4/match=0% 1.00 22.8±0.02µs ? ?/sec 1.00 22.8±0.03µs ? ?/sec
fixed_size_binary/fsb16/list=4/match=50% 1.00 53.5±0.47µs ? ?/sec 1.04 55.4±0.43µs ? ?/sec
fixed_size_binary/fsb16/list=64/match=0% 1.00 22.8±0.02µs ? ?/sec 1.00 22.8±0.02µs ? ?/sec
fixed_size_binary/fsb16/list=64/match=50% 1.00 54.2±0.51µs ? ?/sec 1.04 56.1±0.27µs ? ?/sec
interval_month_day_nano/small_list/list=4/match=0% 1.00 10.4±0.01µs ? ?/sec
interval_month_day_nano/small_list/list=4/match=50% 1.00 10.4±0.01µs ? ?/sec
narrow_integer/f16/list=256/match=0% 1.00 5.2±0.00µs ? ?/sec 1.00 5.2±0.01µs ? ?/sec
narrow_integer/f16/list=256/match=50% 1.00 5.2±0.01µs ? ?/sec 1.00 5.2±0.00µs ? ?/sec
narrow_integer/f16/list=4/match=0% 1.43 5.2±0.01µs ? ?/sec 1.00 3.7±0.00µs ? ?/sec
narrow_integer/f16/list=4/match=50% 1.43 5.2±0.01µs ? ?/sec 1.00 3.7±0.00µs ? ?/sec
narrow_integer/f16/list=64/match=0% 1.00 5.2±0.02µs ? ?/sec 1.00 5.2±0.00µs ? ?/sec
narrow_integer/f16/list=64/match=50% 1.00 5.2±0.01µs ? ?/sec 1.00 5.2±0.00µs ? ?/sec
narrow_integer/i16/list=256/match=0% 1.00 5.2±0.00µs ? ?/sec 1.00 5.2±0.00µs ? ?/sec
narrow_integer/i16/list=256/match=50% 1.00 5.2±0.01µs ? ?/sec 1.00 5.2±0.01µs ? ?/sec
narrow_integer/i16/list=4/match=0% 1.43 5.2±0.00µs ? ?/sec 1.00 3.7±0.00µs ? ?/sec
narrow_integer/i16/list=4/match=50% 1.43 5.2±0.00µs ? ?/sec 1.00 3.7±0.00µs ? ?/sec
narrow_integer/i16/list=64/match=0% 1.00 5.2±0.00µs ? ?/sec 1.00 5.2±0.00µs ? ?/sec
narrow_integer/i16/list=64/match=50% 1.00 5.2±0.00µs ? ?/sec 1.00 5.2±0.00µs ? ?/sec
narrow_integer/u8/list=16/match=0% 1.12 5.2±0.00µs ? ?/sec 1.00 4.7±0.00µs ? ?/sec
narrow_integer/u8/list=16/match=50% 1.11 5.2±0.00µs ? ?/sec 1.00 4.7±0.00µs ? ?/sec
narrow_integer/u8/list=4/match=0% 1.56 5.2±0.00µs ? ?/sec 1.00 3.3±0.01µs ? ?/sec
narrow_integer/u8/list=4/match=50% 1.56 5.2±0.00µs ? ?/sec 1.00 3.3±0.01µs ? ?/sec
nulls/narrow_integer/u8/list=16/match=50%/nulls=20% 1.11 5.3±0.00µs ? ?/sec 1.00 4.8±0.00µs ? ?/sec
nulls/primitive/i32/large_list/list=64/match=50%/nulls=20% 1.00 22.0±0.30µs ? ?/sec 1.10 24.3±0.31µs ? ?/sec
nulls/primitive/i32/small_list/list=16/match=50%/nulls=20% 2.54 24.0±0.10µs ? ?/sec 1.00 9.4±0.01µs ? ?/sec
nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN 2.37 22.6±0.10µs ? ?/sec 1.00 9.5±0.01µs ? ?/sec
nulls/primitive/i32/small_list/list=16/match=50%/nulls=50% 1.92 18.1±0.06µs ? ?/sec 1.00 9.5±0.01µs ? ?/sec
nulls/utf8/long_24b/list=16/match=50%/nulls=20% 1.00 69.4±0.22µs ? ?/sec 1.02 70.7±0.29µs ? ?/sec
nulls/utf8/short_8b/list=16/match=50%/nulls=20% 1.00 58.2±0.45µs ? ?/sec 1.02 59.3±0.53µs ? ?/sec
nulls/utf8view/long_24b/list=16/match=50%/nulls=20% 1.00 78.9±0.67µs ? ?/sec 1.10 86.7±0.18µs ? ?/sec
nulls/utf8view/short_8b/list=16/match=50%/nulls=20% 1.13 47.6±0.20µs ? ?/sec 1.00 42.1±0.18µs ? ?/sec
nulls/utf8view/short_8b/list=16/match=50%/nulls=20%/NOT_IN 1.14 47.7±0.21µs ? ?/sec 1.00 41.7±0.15µs ? ?/sec
nulls/utf8view/short_8b/list=16/match=50%/nulls=50% 1.15 34.3±0.11µs ? ?/sec 1.00 29.7±0.12µs ? ?/sec
primitive/i32/large_list/list=256/match=0% 1.00 12.2±0.11µs ? ?/sec 1.04 12.6±0.02µs ? ?/sec
primitive/i32/large_list/list=256/match=50% 1.06 20.4±0.17µs ? ?/sec 1.00 19.2±0.14µs ? ?/sec
primitive/i32/large_list/list=64/match=0% 1.06 12.4±0.05µs ? ?/sec 1.00 11.8±0.01µs ? ?/sec
primitive/i32/large_list/list=64/match=50% 1.00 19.6±0.13µs ? ?/sec 1.22 23.9±0.36µs ? ?/sec
primitive/i32/small_list/list=16/match=50%/NOT_IN 2.48 23.4±0.24µs ? ?/sec 1.00 9.4±0.00µs ? ?/sec
primitive/i32/small_list/list=32/match=0% 1.00 12.3±0.01µs ? ?/sec 1.25 15.5±0.01µs ? ?/sec
primitive/i32/small_list/list=32/match=50% 1.24 19.1±0.18µs ? ?/sec 1.00 15.5±0.01µs ? ?/sec
primitive/i32/small_list/list=4/match=0% 3.56 12.4±0.01µs ? ?/sec 1.00 3.5±0.00µs ? ?/sec
primitive/i32/small_list/list=4/match=50% 6.76 23.6±0.40µs ? ?/sec 1.00 3.5±0.00µs ? ?/sec
primitive/i64/large_list/list=128/match=0% 1.05 12.7±0.02µs ? ?/sec 1.00 12.1±0.06µs ? ?/sec
primitive/i64/large_list/list=128/match=50% 1.01 17.8±0.14µs ? ?/sec 1.00 17.7±0.17µs ? ?/sec
primitive/i64/large_list/list=32/match=0% 1.00 12.2±0.06µs ? ?/sec 1.00 12.2±0.03µs ? ?/sec
primitive/i64/large_list/list=32/match=50% 1.00 18.9±0.20µs ? ?/sec 1.23 23.2±0.46µs ? ?/sec
primitive/i64/small_list/list=16/match=0% 1.00 12.2±0.02µs ? ?/sec 1.32 16.0±0.01µs ? ?/sec
primitive/i64/small_list/list=16/match=50% 1.37 22.0±0.88µs ? ?/sec 1.00 16.0±0.01µs ? ?/sec
primitive/i64/small_list/list=4/match=0% 2.81 12.2±0.01µs ? ?/sec 1.00 4.4±0.01µs ? ?/sec
primitive/i64/small_list/list=4/match=50% 4.94 21.5±0.28µs ? ?/sec 1.00 4.4±0.01µs ? ?/sec
timestamp_ns/large_list/list=32/match=0% 1.03 18.0±0.56µs ? ?/sec 1.00 17.4±0.02µs ? ?/sec
timestamp_ns/large_list/list=32/match=50% 1.00 29.2±0.31µs ? ?/sec 1.05 30.6±0.68µs ? ?/sec
timestamp_ns/small_list/list=16/match=0% 1.08 17.4±0.01µs ? ?/sec 1.00 16.0±0.01µs ? ?/sec
timestamp_ns/small_list/list=16/match=50% 1.81 29.1±0.59µs ? ?/sec 1.00 16.0±0.02µs ? ?/sec
timestamp_ns/small_list/list=4/match=0% 4.01 17.4±0.39µs ? ?/sec 1.00 4.3±0.01µs ? ?/sec
timestamp_ns/small_list/list=4/match=50% 6.97 30.2±0.35µs ? ?/sec 1.00 4.3±0.00µs ? ?/sec
utf8/long_24b/list=256/match=0% 1.01 34.0±0.02µs ? ?/sec 1.00 33.6±0.03µs ? ?/sec
utf8/long_24b/list=256/match=50% 1.00 71.3±0.31µs ? ?/sec 1.01 72.2±0.43µs ? ?/sec
utf8/long_24b/list=4/match=0% 1.00 34.1±0.04µs ? ?/sec 1.00 33.9±0.02µs ? ?/sec
utf8/long_24b/list=4/match=50% 1.00 72.2±0.37µs ? ?/sec 1.01 72.7±0.44µs ? ?/sec
utf8/long_24b/list=64/match=0% 1.00 33.6±0.03µs ? ?/sec 1.03 34.7±0.28µs ? ?/sec
utf8/long_24b/list=64/match=50% 1.00 71.3±0.30µs ? ?/sec 1.00 71.6±0.69µs ? ?/sec
utf8/mixed_len/list=16/match=0% 1.00 36.6±0.10µs ? ?/sec 1.02 37.2±0.10µs ? ?/sec
utf8/mixed_len/list=16/match=50% 1.00 102.2±0.65µs ? ?/sec 1.02 104.5±0.79µs ? ?/sec
utf8/mixed_len/list=64/match=0% 1.00 37.4±0.09µs ? ?/sec 1.03 38.5±0.12µs ? ?/sec
utf8/mixed_len/list=64/match=50% 1.00 115.1±0.53µs ? ?/sec 1.00 115.3±0.43µs ? ?/sec
utf8/shared_prefix/pfx=12/list=32/match=50% 1.00 71.6±0.49µs ? ?/sec 1.01 72.5±0.43µs ? ?/sec
utf8/short_8b/list=16/match=50%/NOT_IN 1.00 61.2±0.46µs ? ?/sec 1.03 62.9±0.22µs ? ?/sec
utf8/short_8b/list=256/match=0% 1.00 26.2±0.04µs ? ?/sec 1.00 26.2±0.02µs ? ?/sec
utf8/short_8b/list=256/match=50% 1.00 61.1±0.33µs ? ?/sec 1.05 64.3±0.39µs ? ?/sec
utf8/short_8b/list=4/match=0% 1.00 26.3±0.07µs ? ?/sec 1.00 26.4±0.05µs ? ?/sec
utf8/short_8b/list=4/match=50% 1.00 62.1±0.34µs ? ?/sec 1.04 64.5±0.17µs ? ?/sec
utf8/short_8b/list=64/match=0% 1.00 26.3±0.02µs ? ?/sec 1.00 26.3±0.03µs ? ?/sec
utf8/short_8b/list=64/match=50% 1.00 61.3±0.52µs ? ?/sec 1.03 63.3±0.86µs ? ?/sec
utf8view/len_12b/list=16/match=0% 1.00 17.8±0.02µs ? ?/sec 1.00 17.8±0.02µs ? ?/sec
utf8view/len_12b/list=16/match=50% 1.00 44.5±0.16µs ? ?/sec 1.03 46.0±0.31µs ? ?/sec
utf8view/len_12b/list=64/match=0% 1.00 17.8±0.02µs ? ?/sec 1.00 17.9±0.02µs ? ?/sec
utf8view/len_12b/list=64/match=50% 1.00 44.0±0.28µs ? ?/sec 1.02 45.1±0.23µs ? ?/sec
utf8view/long_24b/list=16/match=0% 1.00 40.2±0.03µs ? ?/sec 1.01 40.7±0.23µs ? ?/sec
utf8view/long_24b/list=16/match=50% 1.02 88.2±0.36µs ? ?/sec 1.00 86.6±0.17µs ? ?/sec
utf8view/long_24b/list=256/match=0% 1.00 40.2±0.03µs ? ?/sec 1.00 40.1±0.04µs ? ?/sec
utf8view/long_24b/list=256/match=50% 1.01 85.2±0.18µs ? ?/sec 1.00 84.6±0.18µs ? ?/sec
utf8view/long_24b/list=4/match=0% 1.01 40.6±0.10µs ? ?/sec 1.00 40.3±0.04µs ? ?/sec
utf8view/long_24b/list=4/match=50% 1.00 86.2±0.19µs ? ?/sec 1.01 86.8±0.23µs ? ?/sec
utf8view/long_24b/list=64/match=0% 1.00 40.3±0.05µs ? ?/sec 1.00 40.2±0.04µs ? ?/sec
utf8view/long_24b/list=64/match=50% 1.01 84.5±0.13µs ? ?/sec 1.00 83.7±0.18µs ? ?/sec
utf8view/mixed_len/list=16/match=0% 1.09 32.3±0.15µs ? ?/sec 1.00 29.7±0.08µs ? ?/sec
utf8view/mixed_len/list=16/match=50% 1.13 82.0±0.63µs ? ?/sec 1.00 72.8±0.52µs ? ?/sec
utf8view/mixed_len/list=64/match=0% 1.00 32.6±0.15µs ? ?/sec 1.03 33.4±0.07µs ? ?/sec
utf8view/mixed_len/list=64/match=50% 1.08 91.1±0.50µs ? ?/sec 1.00 84.3±0.52µs ? ?/sec
utf8view/shared_prefix/pfx=12/list=32/match=0% 1.00 42.3±0.07µs ? ?/sec 1.02 42.9±0.14µs ? ?/sec
utf8view/shared_prefix/pfx=12/list=32/match=50% 1.03 85.5±0.40µs ? ?/sec 1.00 82.7±0.27µs ? ?/sec
utf8view/shared_prefix/pfx=16/list=64/match=0% 1.00 40.5±0.03µs ? ?/sec 1.00 40.4±0.22µs ? ?/sec
utf8view/shared_prefix/pfx=16/list=64/match=50% 1.00 83.4±0.18µs ? ?/sec 1.01 84.0±0.28µs ? ?/sec
utf8view/shared_prefix/pfx=8/list=16/match=0% 1.00 30.0±0.06µs ? ?/sec 1.00 29.9±0.05µs ? ?/sec
utf8view/shared_prefix/pfx=8/list=16/match=50% 1.03 71.6±0.32µs ? ?/sec 1.00 69.8±0.12µs ? ?/sec
utf8view/short_8b/list=16/match=0% 1.00 17.8±0.02µs ? ?/sec 1.00 17.8±0.03µs ? ?/sec
utf8view/short_8b/list=16/match=50% 1.00 41.9±0.17µs ? ?/sec 1.01 42.3±0.28µs ? ?/sec
utf8view/short_8b/list=256/match=0% 1.07 19.0±0.03µs ? ?/sec 1.00 17.8±0.08µs ? ?/sec
utf8view/short_8b/list=256/match=50% 1.00 40.6±0.18µs ? ?/sec 1.02 41.5±0.18µs ? ?/sec
utf8view/short_8b/list=4/match=0% 1.00 17.8±0.02µs ? ?/sec 1.00 17.8±0.07µs ? ?/sec
utf8view/short_8b/list=4/match=50% 1.00 44.0±0.25µs ? ?/sec 1.02 45.0±0.35µs ? ?/sec
utf8view/short_8b/list=64/match=0% 1.00 17.9±0.01µs ? ?/sec 1.00 17.9±0.02µs ? ?/sec
utf8view/short_8b/list=64/match=50% 1.00 39.6±0.19µs ? ?/sec 1.02 40.4±0.38µs ? ?/sec

Resource Usage

in_list_strategy — base (merge-base)

Metric	Value
Wall time	1485.3s
Peak memory	40.8 MiB
Avg memory	26.7 MiB
CPU user	1495.3s
CPU sys	1.3s
Peak spill	0 B

in_list_strategy — branch

Metric	Value
Wall time	1565.4s
Peak memory	42.0 MiB
Avg memory	27.2 MiB
CPU user	1531.0s
CPU sys	1.3s
Peak spill	0 B

File an issue against this benchmark runner

geoffreyclaude · 2026-07-23T08:44:02Z

One way we could potentially reduce the code size would be to minimize the parts of the code that really need specializations.

@alamb I agree, and I pushed a new remediation commit along those lines.

The IN-list values are now stored at runtime, and the filter chooses a comparison function for that list length when it is built. That function is called once per input array, while the rest of the filter is shared.

I didn't use chunks_exact exactly: keeping one small comparison function per list length was the simplest option that preserved performance. Looking at the release artifacts, the number of copies of BranchlessFilter::try_new and contains drops from 507 to 27. About 86 small comparison functions remain.

I also rebuilt release datafusion-cli for the previous and current versions on the same base, toolchain and arm64 machine:

executable: 82,316,496 → 77,773,840 bytes (-5.5%)
Mach-O __TEXT: 78,479,360 → 73,990,144 bytes (-5.7%)
Mach-O __text: 65,119,604 → 60,957,044 bytes (-6.4%)

That brings the release binary back to roughly the 74M main baseline from your measurement.

For performance, I compared the two versions across 28 focused Criterion cases. The worst median regression was 2.45%; the next two were 1.58% and 1.57%. None reached 3%, and many cases were flat or faster.

Thanks for pushing on this! I think this follow-up addresses the main concern without giving up the performance gains. WDYT?

alamb

Thank you @geoffreyclaude -- I went through this PR again and I think it looks great.

I double checked this the code size and it seems like it is much smaller (a few kb I think)

I personally think it would help to have some more slt coverage, but I will propose that as a follow on PR

Code size measurements

(venv) andrewlamb@Andrews-MacBook-Pro-3:/tmp/branchless_in$ du -s -h target/release/datafusion-cli target/debug/datafusion-cli
74M	target/release/datafusion-cli
414M	target/debug/datafusion-cli
andrewlamb@Andrews-MacBook-Pro-3:/tmp/branchless_in_merge_base$ du -s -h target/release/datafusion-cli target/debug/datafusion-cli
74M	target/release/datafusion-cli
413M	target/debug/datafusion-cli

alamb · 2026-07-24T20:17:13Z

+ let non_null_count = in_array.len() - in_array.null_count();
+ // `try_new` can be called on its own, so check the limit here too.
+ if non_null_count > T::MAX_LIST_LEN {
+ return Err(exec_datafusion_err!(


This is probably more correctly an internal error rather than an execution error as it isn't really based on user input (it would signal a bug in the code I think)

alamb · 2026-07-24T20:21:28Z

+ 4 => choose!(0, 1, 2, 3, 4),
+ 8 => choose!(0, 1, 2, 3, 4, 5, 6, 7, 8),
+ 16 => choose!(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16),
+ 32 => choose!(


this is cool -- it makes much more sense to isolate the code repetition to just the comparison function (which is where it is most helpful)

alamb · 2026-07-26T01:29:56Z

🚀

alamb · 2026-07-26T01:30:21Z

Thanks again @geoffreyclaude and @kosiew for the iteration -- IN lists are getting quite cool

…types (apache#23875) ## Which issue does this PR close? - Part of apache#23307. ## Rationale for this change IN lists over temporal, `Decimal128` and `Interval` columns had no SQL level coverage at all, and the specializations being added in apache#23014 route those types onto new code paths. ## What changes are included in this PR? Adds `in_list.slt` coverage for `Date32`, `Date64`, `Time32(Second)`, `Time64(Nanosecond)`, `Timestamp(Nanosecond, None)`, `Timestamp(Second, "UTC")`, `Duration(Second)`, `Decimal128` and `Interval(MonthDayNano)` ## Are these changes tested? This PR is only tests; they pass on `main` and against the head of apache#23014. ## Are there any user-facing changes? No. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

## Which issue does this PR close? - Part of apache#19241. - Follow-up to apache#23014. ## Rationale for this change apache#23014 added `BranchlessFilter` for small primitive `IN` lists. That implementation currently sits in `primitive_filter.rs` alongside the bitmap and hash-based filters. This PR moves the branchless-specific types, functions, and tests into `branchless_filter.rs`. The module docs explain the direct-comparison path and its per-width limits. They also cover why narrow types switch to bitmap filters and how nulls are handled. `BranchlessFilter::try_new` also returns an execution error when the list is over its limit. Strategy selection rejects such lists before constructing the filter, so hitting that guard means the caller violated an internal invariant. This PR reports it as an internal error instead. ## What changes are included in this PR? - Moves `BranchlessFilter`, its type mappings, size limits, and tests into `branchless_filter.rs`. - Adds module documentation for the comparison strategy, thresholds, narrow-type bitmap fallback, and null handling. - Changes the oversized-list guard from an execution error to an internal error. - Leaves routing, thresholds, comparison behavior, and null semantics unchanged. ## Are these changes tested? Yes. The existing branchless-filter tests were moved unchanged and continue to pass. - `cargo fmt --all` - `cargo test -p datafusion-physical-expr expressions::in_list --lib` - `cargo clippy --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? No. This only reorganizes internal code and corrects an internal error classification.

## Which issue does this PR close? - Part of apache#19241. - Stacked on [apache#23311](apache#23311). - Next in stack: apache#23015. - Extracted from apache#19390. ## Rationale for this change For very small `IN` lists, building or probing a hash table can be more work than just comparing the input value with each constant. For example, for `x IN (10, 20, 30)`, the fast path can behave like: ```text x == 10 OR x == 20 OR x == 30 ``` Because the list is tiny, those comparisons are cheap. The implementation stores the constants in a fixed-size array and checks them with a compact comparison chain. “Branchless” here means the comparisons are combined without stopping at the first match. That can be faster for these small fixed-width lists because the CPU gets a predictable sequence of simple operations instead of hash-table setup and probe logic. For primitive values that are not already plain unsigned integers, this PR keeps the logical Arrow type explicit and uses a matching same-width comparison representation only inside the branchless filter. For example, `Float16` uses `UInt16` storage, `Float32` uses `UInt32` storage, and `TimestampNanosecond` uses `UInt64` storage. `Decimal128` and `IntervalMonthDayNano` use their own 16-byte native representation. This preserves bit-pattern equality while relying on Arrow's native primitive compatibility rules: timestamp timezone metadata and Decimal128 precision/scale metadata may differ, while incompatible primitive representations remain rejected. ## What changes are included in this PR? - Adds a const-generic `BranchlessFilter` for small primitive `IN` lists. - Adds thresholds for when this path is used: - up to 16 values for 1-byte types - up to 8 values for 2-byte types - up to 32 values for 4-byte types - up to 16 values for 8-byte types - up to 4 values for 16-byte types - Keeps dispatch concrete and explicit in `strategy.rs`. - Maps each optimized logical type to the comparison representation used by the branchless filter: - `Int8` -> `UInt8` - `Int16`, `Float16` -> `UInt16` - `Int32`, `Float32`, `Date32`, `Time32` -> `UInt32` - `Int64`, `Float64`, `Date64`, `Time64`, `Timestamp`, `Duration` -> `UInt64` - `Decimal128`, `IntervalMonthDayNano` -> their native 16-byte representation - Leaves larger 1-byte and 2-byte lists on the existing bitmap filters. - Leaves larger 4-byte and 8-byte lists on the existing hash/generic paths. - Leaves wider primitive types such as `Decimal256` and unsupported complex types on the generic path. - Keeps the same `IN` / `NOT IN` null behavior as the rest of the stack. - Adds focused coverage for branchless null handling, signed boundary values, slices, Float16/Float32/Float64 bit patterns, compatible timestamp/Decimal128 metadata, incompatible timestamp units, IntervalMonthDayNano values, and same-width wrong-type probe rejection. ## Are these changes tested? Yes. - `cargo fmt --all -- --check` - `cargo test -p datafusion-physical-expr expressions::in_list --lib` - `cargo test -p datafusion-physical-expr --bench in_list_strategy --no-run` - `cargo clippy --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? No. This is an internal performance optimization only. ## Local benchmark snapshot Built and run with `release-nonlto`, filtered to the relevant small primitive-list rows: ```bash cargo bench -p datafusion-physical-expr --profile release-nonlto --bench in_list_strategy -- <filter> --save-baseline <baseline> ``` Filters used: `narrow_integer`, `primitive/i32/small_list`, `primitive/i64/small_list`, `f32/small_list`, `timestamp_ns/small_list`, and `interval_month_day_nano/small_list`. Method: directly compared Criterion's raw sample minima (`min(time / iterations)`) from `sample.json`. Lower is better; changes within +/-5% are treated as noise. Compared baselines: [apache#23311](apache#23311) -> [apache#23014](apache#23014) Relevant scope: small primitive-list rows. Summary: 39 relevant rows, 28 faster, 0 slower, 11 within +/-5%. Largest relevant deltas: | Benchmark | Before | After | Change | |---|---:|---:|---:| | `timestamp_ns/small_list/list=4/match=50%` | 46.55 us | 3.17 us | -93.2% (14.69x faster) | | `f32/small_list/list=4/match=50%` | 33.93 us | 3.04 us | -91.0% (11.15x faster) | | `primitive/i32/small_list/list=4/match=50%` | 32.63 us | 3.08 us | -90.5% (10.58x faster) | | `primitive/i64/small_list/list=4/match=50%` | 33.55 us | 3.18 us | -90.5% (10.54x faster) | | `timestamp_ns/small_list/list=4/match=0%` | 19.57 us | 3.18 us | -83.8% (6.16x faster) | | `f32/small_list/list=4/match=0%` | 18.14 us | 3.05 us | -83.2% (5.95x faster) | | `primitive/i32/small_list/list=4/match=0%` | 17.00 us | 3.04 us | -82.1% (5.59x faster) | | `primitive/i64/small_list/list=4/match=0%` | 17.12 us | 3.22 us | -81.2% (5.31x faster) | | `primitive/i32/small_list/list=16/match=50%/NOT_IN` | 31.98 us | 7.26 us | -77.3% (4.41x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%` | 29.35 us | 7.32 us | -75.1% (4.01x faster) | | `timestamp_ns/small_list/list=16/match=50%` | 45.32 us | 11.79 us | -74.0% (3.84x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%` | 25.89 us | 7.31 us | -71.8% (3.54x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN` | 26.05 us | 7.42 us | -71.5% (3.51x faster) | | `interval_month_day_nano/small_list/list=4/match=50%` | 52.94 us | 15.52 us | -70.7% (3.41x faster) | | `f32/small_list/list=32/match=50%` | 38.78 us | 13.27 us | -65.8% (2.92x faster) | | `primitive/i64/small_list/list=16/match=50%` | 29.46 us | 11.76 us | -60.1% (2.50x faster) | <details> <summary>Full relevant table (39 rows)</summary> | Benchmark | Before | After | Change | |---|---:|---:|---:| | `narrow_integer/u8/list=4/match=0%` | 3.86 us | 2.79 us | -27.8% (1.38x faster) | | `narrow_integer/u8/list=4/match=50%` | 3.84 us | 2.78 us | -27.7% (1.38x faster) | | `narrow_integer/u8/list=16/match=0%` | 3.88 us | 3.85 us | -0.8% (within noise) | | `narrow_integer/u8/list=16/match=50%` | 3.84 us | 3.86 us | +0.5% (within noise) | | `narrow_integer/i16/list=4/match=0%` | 3.93 us | 3.18 us | -19.1% (1.24x faster) | | `narrow_integer/i16/list=4/match=50%` | 3.92 us | 3.16 us | -19.5% (1.24x faster) | | `narrow_integer/i16/list=64/match=0%` | 3.96 us | 3.82 us | -3.5% (within noise) | | `narrow_integer/i16/list=64/match=50%` | 3.91 us | 3.80 us | -2.9% (within noise) | | `narrow_integer/i16/list=256/match=0%` | 3.90 us | 3.81 us | -2.5% (within noise) | | `narrow_integer/i16/list=256/match=50%` | 3.97 us | 3.81 us | -4.1% (within noise) | | `narrow_integer/f16/list=4/match=0%` | 3.87 us | 3.16 us | -18.5% (1.23x faster) | | `narrow_integer/f16/list=4/match=50%` | 3.94 us | 3.15 us | -20.2% (1.25x faster) | | `narrow_integer/f16/list=64/match=0%` | 3.87 us | 3.84 us | -0.6% (within noise) | | `narrow_integer/f16/list=64/match=50%` | 3.93 us | 3.85 us | -1.9% (within noise) | | `narrow_integer/f16/list=256/match=0%` | 3.90 us | 3.84 us | -1.5% (within noise) | | `narrow_integer/f16/list=256/match=50%` | 3.87 us | 3.91 us | +1.2% (within noise) | | `nulls/narrow_integer/u8/list=16/match=50%/nulls=20%` | 3.92 us | 4.02 us | +2.5% (within noise) | | `primitive/i32/small_list/list=4/match=0%` | 17.00 us | 3.04 us | -82.1% (5.59x faster) | | `primitive/i32/small_list/list=4/match=50%` | 32.63 us | 3.08 us | -90.5% (10.58x faster) | | `primitive/i32/small_list/list=32/match=0%` | 16.34 us | 13.33 us | -18.5% (1.23x faster) | | `primitive/i32/small_list/list=32/match=50%` | 31.17 us | 13.31 us | -57.3% (2.34x faster) | | `primitive/i32/small_list/list=16/match=50%/NOT_IN` | 31.98 us | 7.26 us | -77.3% (4.41x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%` | 29.35 us | 7.32 us | -75.1% (4.01x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN` | 26.05 us | 7.42 us | -71.5% (3.51x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%` | 25.89 us | 7.31 us | -71.8% (3.54x faster) | | `primitive/i64/small_list/list=4/match=0%` | 17.12 us | 3.22 us | -81.2% (5.31x faster) | | `primitive/i64/small_list/list=4/match=50%` | 33.55 us | 3.18 us | -90.5% (10.54x faster) | | `primitive/i64/small_list/list=16/match=0%` | 16.34 us | 11.93 us | -27.0% (1.37x faster) | | `primitive/i64/small_list/list=16/match=50%` | 29.46 us | 11.76 us | -60.1% (2.50x faster) | | `f32/small_list/list=4/match=0%` | 18.14 us | 3.05 us | -83.2% (5.95x faster) | | `f32/small_list/list=4/match=50%` | 33.93 us | 3.04 us | -91.0% (11.15x faster) | | `f32/small_list/list=32/match=0%` | 22.05 us | 13.43 us | -39.1% (1.64x faster) | | `f32/small_list/list=32/match=50%` | 38.78 us | 13.27 us | -65.8% (2.92x faster) | | `timestamp_ns/small_list/list=4/match=0%` | 19.57 us | 3.18 us | -83.8% (6.16x faster) | | `timestamp_ns/small_list/list=4/match=50%` | 46.55 us | 3.17 us | -93.2% (14.69x faster) | | `timestamp_ns/small_list/list=16/match=0%` | 19.73 us | 12.07 us | -38.8% (1.63x faster) | | `timestamp_ns/small_list/list=16/match=50%` | 45.32 us | 11.79 us | -74.0% (3.84x faster) | | `interval_month_day_nano/small_list/list=4/match=0%` | 20.12 us | 13.20 us | -34.4% (1.52x faster) | | `interval_month_day_nano/small_list/list=4/match=50%` | 52.94 us | 15.52 us | -70.7% (3.41x faster) | </details> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…types (apache#23875) ## Which issue does this PR close? - Part of apache#23307. ## Rationale for this change IN lists over temporal, `Decimal128` and `Interval` columns had no SQL level coverage at all, and the specializations being added in apache#23014 route those types onto new code paths. ## What changes are included in this PR? Adds `in_list.slt` coverage for `Date32`, `Date64`, `Time32(Second)`, `Time64(Nanosecond)`, `Timestamp(Nanosecond, None)`, `Timestamp(Second, "UTC")`, `Duration(Second)`, `Decimal128` and `Interval(MonthDayNano)` ## Are these changes tested? This PR is only tests; they pass on `main` and against the head of apache#23014. ## Are there any user-facing changes? No. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

## Which issue does this PR close? - Part of apache#19241. - Follow-up to apache#23014. ## Rationale for this change apache#23014 added `BranchlessFilter` for small primitive `IN` lists. That implementation currently sits in `primitive_filter.rs` alongside the bitmap and hash-based filters. This PR moves the branchless-specific types, functions, and tests into `branchless_filter.rs`. The module docs explain the direct-comparison path and its per-width limits. They also cover why narrow types switch to bitmap filters and how nulls are handled. `BranchlessFilter::try_new` also returns an execution error when the list is over its limit. Strategy selection rejects such lists before constructing the filter, so hitting that guard means the caller violated an internal invariant. This PR reports it as an internal error instead. ## What changes are included in this PR? - Moves `BranchlessFilter`, its type mappings, size limits, and tests into `branchless_filter.rs`. - Adds module documentation for the comparison strategy, thresholds, narrow-type bitmap fallback, and null handling. - Changes the oversized-list guard from an execution error to an internal error. - Leaves routing, thresholds, comparison behavior, and null semantics unchanged. ## Are these changes tested? Yes. The existing branchless-filter tests were moved unchanged and continue to pass. - `cargo fmt --all` - `cargo test -p datafusion-physical-expr expressions::in_list --lib` - `cargo clippy --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? No. This only reorganizes internal code and corrects an internal error classification.

geoffreyclaude mentioned this pull request Jun 18, 2026
IN LIST: add keyed frozen set for large primitive lists #23015
Closed

github-actionsBot added the physical-expr Changes to the physical-expr crates label Jun 18, 2026

geoffreyclaude mentioned this pull request Jun 18, 2026
Further improve performance of IN list evaluation #19241
Open
10 tasks

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch 3 times, most recently from 428e3cd to eae4046Compare June 18, 2026 09:03

This was referenced Jun 18, 2026
IN LIST: reuse primitive filters for FixedSizeBinary #23018
Closed
Optimize IN performance with specialized implementations #19390
Closed

geoffreyclaude changed the title ~~Implement Branchless Filter for small primitive lists~~IN LIST: add branchless filter for small primitive listsJun 18, 2026

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch from eae4046 to 3e3651bCompare June 19, 2026 05:11

github-actionsBot added the auto detected api change Auto detected API change label Jun 19, 2026

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch from 3e3651b to 6a1869fCompare June 19, 2026 05:35

github-actionsBot removed the auto detected api change Auto detected API change label Jun 19, 2026

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch 5 times, most recently from bf69ed2 to e040466Compare June 24, 2026 20:49

geoffreyclaude mentioned this pull request Jun 25, 2026
IN LIST: reinterpret small-width types for bitmap filters #23013
Closed

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch 8 times, most recently from 04ca744 to f89d3b6Compare June 30, 2026 07:08

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch from f89d3b6 to 409b0cbCompare July 2, 2026 08:54

geoffreyclaude mentioned this pull request Jul 3, 2026
IN LIST: add Float16 bitmap filter #23311
Merged

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch from 409b0cb to 78e3631Compare July 6, 2026 11:53

Merge branch 'main' into perf/in_list_branchless_filter
06f2b6b

alamb reviewed Jul 22, 2026
View reviewed changes

Reduce branchless IN-list filter code size
5af3108

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch 5 times, most recently from 5af3108 to 08277d2Compare July 23, 2026 08:54

alamb approved these changes Jul 24, 2026
View reviewed changes

alamb mentioned this pull request Jul 24, 2026
test: add IN list slt coverage for temporal, Decimal128 and Interval types #23875
Merged

alamb added this pull request to the merge queue Jul 26, 2026

Merged via the queue into apache:main with commit e3e2cb2Jul 26, 2026
40 checks passed

geoffreyclaude mentioned this pull request Jul 26, 2026
IN LIST: isolate branchless filter implementation #23907
Merged

This was referenced Aug 5, 2026
IN LIST: reuse primitive filters for FixedSizeBinary #24102
Draft
IN LIST: optimize Utf8View and BinaryView filters #24088
Draft
IN LIST: index 32- and 64-bit integers by byte #24181
Closed

geoffreyclaude mentioned this pull request Aug 12, 2026
IN LIST: centralize primitive filters and optimize Decimal128 #24283
Open

Uh oh!

Conversation

geoffreyclaude commented Jun 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Local benchmark snapshot

Uh oh!

alamb commented Jul 22, 2026

Uh oh!

alamb commented Jul 22, 2026

Uh oh!

adriangbot commented Jul 22, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangbot commented Jul 22, 2026

Uh oh!

geoffreyclaude commented Jul 23, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 26, 2026

Uh oh!

alamb commented Jul 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

geoffreyclaude commented Jun 18, 2026•
edited
Loading