IN LIST: add keyed frozen set for large primitive lists

[geoffreyclaude]

Which issue does this PR close?

• Part of Further improve performance of IN list evaluation #19241.
• Stacked on #23014.
• Next in stack: #23016.
• Extracted from Optimize IN performance with specialized implementations #19390.

Rationale for this change

#23014 handles small primitive IN lists with direct comparisons. For larger 4- and 8-byte lists, integers and floats already use direct-value hashbrown::HashSets; temporal values fall back to the generic table of Arrow array indices and comparator calls.

This PR adds a keyed immutable FrozenSet for fixed-width values. Each bucket stores two values directly; only buckets with additional collisions consult a native hashbrown::HashTable overflow table. The first real member is also the empty-slot sentinel, so the primary table stores values without Option wrappers.

This keeps the common probe small and predictable while retaining exact collision handling. Existing optimizer thresholds are unchanged; any threshold change belongs in a dedicated follow-up PR.

What changes are included in this PR?

• Adds a reusable keyed FrozenSet with two inline values per bucket and native HashTable overflow handling.
• Routes larger 4- and 8-byte primitive IN lists through PrimitiveFrozenFilter.
• Consolidates primitive routing into one type dispatch and removes six legacy integer/float HashSet filters superseded by branchless/frozen routing.
• Uses Arrow physical compatibility when accepting primitive input arrays.
• Supports u128 values for later representation-based optimizations without enabling general 128-bit primitive routing here.
• Covers empty sets, duplicates, the sentinel member, forced collisions, slices, dictionaries, nulls, and physical compatibility.

Are these changes tested?

Yes.

• cargo fmt --all
• cargo test -p datafusion-physical-expr --lib expressions::in_list
• cargo clippy --all-targets --all-features -- -D warnings
• ./dev/rust_lint.sh
• RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption

Are there any user-facing changes?

No. This is an internal performance optimization only.

Local benchmark snapshot

Built and run with release-nonlto, filtered to the relevant large primitive-list rows:

cargo bench -p datafusion-physical-expr --profile release-nonlto --bench in_list_strategy -- <filter> --save-baseline <baseline>

Filters used: primitive/i32/large_list and primitive/i64/large_list.

Method: directly compared Criterion slope point estimates from isolated saved baselines. The expression is constructed outside the timed loop. Lower is better; changes within +/-5% are treated as noise.

Compared baselines: #23014 -> #23015

Relevant scope: the 12 large primitive-list rows shared with #23014.

Summary: 12 relevant rows, 12 faster, 0 slower, 0 within +/-5%. Geometric-mean time decreased by 45.4%.

A separate extension run covered eight added 1K/10K-list rows. All eight improved, with a 43.6% geometric-mean decrease.

Largest relevant deltas:

Benchmark	Before	After	Change
primitive/i64/large_list/list=32/match=50%	35.01 us	11.86 us	-66.1% (2.95x faster)
primitive/i64/large_list/list=17/match=50%	35.23 us	12.06 us	-65.8% (2.92x faster)
primitive/i64/large_list/list=128/match=50%	32.66 us	11.72 us	-64.1% (2.79x faster)
primitive/i32/large_list/list=33/match=50%	38.13 us	15.30 us	-59.9% (2.49x faster)
primitive/i32/large_list/list=64/match=50%	36.89 us	16.26 us	-55.9% (2.27x faster)
primitive/i32/large_list/list=256/match=50%	34.67 us	15.90 us	-54.1% (2.18x faster)

Full relevant table (12 rows)

Benchmark	Before	After	Change
primitive/i32/large_list/list=33/match=0%	17.19 us	13.14 us	-23.6% (1.31x faster)
primitive/i32/large_list/list=33/match=50%	38.13 us	15.30 us	-59.9% (2.49x faster)
primitive/i32/large_list/list=64/match=0%	17.61 us	13.72 us	-22.1% (1.28x faster)
primitive/i32/large_list/list=64/match=50%	36.89 us	16.26 us	-55.9% (2.27x faster)
primitive/i32/large_list/list=256/match=0%	18.00 us	13.31 us	-26.1% (1.35x faster)
primitive/i32/large_list/list=256/match=50%	34.67 us	15.90 us	-54.1% (2.18x faster)
primitive/i64/large_list/list=17/match=0%	20.20 us	11.47 us	-43.2% (1.76x faster)
primitive/i64/large_list/list=17/match=50%	35.23 us	12.06 us	-65.8% (2.92x faster)
primitive/i64/large_list/list=32/match=0%	16.96 us	15.85 us	-6.5% (1.07x faster)
primitive/i64/large_list/list=32/match=50%	35.01 us	11.86 us	-66.1% (2.95x faster)
primitive/i64/large_list/list=128/match=0%	17.99 us	16.04 us	-10.8% (1.12x faster)
primitive/i64/large_list/list=128/match=50%	32.66 us	11.72 us	-64.1% (2.79x faster)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 90.03115% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.90%. Comparing base (26fbf4d) to head (48c32f6).
⚠️ Report is 32 commits behind head on main.

Files with missing lines	Patch %	Lines
...l-expr/src/expressions/in_list/primitive_filter.rs	83.96%	6 Missing and 11 partials ⚠️
.../physical-expr/src/expressions/in_list/strategy.rs	86.81%	1 Missing and 11 partials ⚠️
...hysical-expr/src/expressions/in_list/frozen_set.rs	97.58%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #23015 +/- ##
==========================================
+ Coverage 80.89% 80.90% +0.01% 
==========================================
Files 1102 1103 +1 Lines 376093 376322 +229 Branches 376093 376322 +229 ==========================================
+ Hits 304239 304482 +243 + Misses 53748 53714 -34 - Partials 18106 18126 +20 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.