Uh oh!
There was an error while loading. Please reload this page.
IN LIST: optimize Utf8View and BinaryView filters - #24088
Draft
geoffreyclaude wants to merge 8 commits into
Draft
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@## main #24088 +/- ##
==========================================
+ Coverage 80.91% 81.15% +0.23%
==========================================
Files 1102 1114 +12 Lines 377102 387354 +10252 Branches 377102 387354 +10252 ==========================================
+ Hits 305143 314345 +9202 - Misses 53769 54461 +692 - Partials 18190 18548 +358 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
geoffreyclaudeforce-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 5, 2026 07:02
ec9cb2e to
65b67baComparegeoffreyclaude
commented
Aug 5, 2026
ContributorAuthor
run benchmark in_list |
adriangbot
commented
Aug 5, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_listResults will be posted here when complete File an issue against this benchmark runner |
adriangbot
commented
Aug 5, 2026
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_listCPU Details (lscpu)DetailsResource Usagein_list — base (merge-base)
in_list — branch
File an issue against this benchmark runner |
geoffreyclaude
commented
Aug 5, 2026
ContributorAuthor
run benchmark in_list_strategy |
adriangbot
commented
Aug 5, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_list_strategyResults will be posted here when complete File an issue against this benchmark runner |
adriangbot
commented
Aug 5, 2026
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_list_strategyCPU Details (lscpu)DetailsResource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
geoffreyclaudeforce-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 6, 2026 16:01
65b67ba to
a44ee0dComparegeoffreyclaudeforce-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 9, 2026 06:21
a44ee0d to
b0443b6Compare10 tasks
geoffreyclaudeforce-pushed
the
codex/in-list-byte-view-hashset
branch
10 times, most recently
from
August 11, 2026 16:29
61cc7a6 to
156d76bComparegeoffreyclaudeforce-pushed
the
codex/in-list-byte-view-hashset
branch
2 times, most recently
from
August 12, 2026 10:18
6a5284e to
f8ae3adComparegeoffreyclaude
commented
Aug 12, 2026
ContributorAuthor
run benchmark in_list_strategy |
adriangbot
commented
Aug 12, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-byte-view-hashset (f8ae3ad) to 426b351 (merge-base) diff Run configurationrun benchmark in_list_strategyResults will be posted here when complete File an issue against this benchmark runner |
adriangbot
commented
Aug 12, 2026
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-byte-view-hashset (f8ae3ad) to 426b351 (merge-base) diff Run configurationrun benchmark in_list_strategyCPU Details (lscpu)DetailsResource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
alamb
commented
Aug 12, 2026
Contributor
I noticed this is marked as draft -- please ping me when ready to review |
geoffreyclaudeforce-pushed
the
codex/in-list-byte-view-hashset
branch
7 times, most recently
from
August 13, 2026 10:14
854fa0b to
dab222aCompareUse PrimitiveHashSetFilter<T, K> for integer and floating-point arrays. Integer filters keep their native key type by default; Float32 and Float64 use their existing bitwise wrapper keys, preserving signed-zero and NaN-payload behavior. The key type and conversion are selected statically, with no function pointer or dynamic dispatch in the lookup loop. Filter selection, dictionary handling, null handling, IN, and NOT IN behavior remain unchanged. Decimal128 routing remains in the following commit.
Route Decimal128 lists above the branchless cutoff through PrimitiveHashSetFilter, replacing the ArrayStaticFilter fallback.
geoffreyclaudeforce-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 13, 2026 10:46
dab222a to
cc8aacfCompareTreat inline Utf8View and BinaryView values as 128-bit primitive keys and pass them to the shared primitive filter selector. This reuses direct comparisons for short lists and the primitive hash-set path for larger lists without reading backing buffers. Lists containing a non-inline value continue to use the general filter. Exact view types, dictionaries, slices, nulls, IN, and NOT IN keep their existing behavior.
geoffreyclaudeforce-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 13, 2026 11:42
cc8aacf to
00dd083Comparegeoffreyclaude
commented
Aug 13, 2026
ContributorAuthor
adriangbot
commented
Aug 13, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 00dd083 (00dd083) to 7ff49f9diff Run configurationrun benchmark in_list_strategybaseline:
ref: "7ff49f94d14f73151a55f59314247c4c223073f2"changed:
ref: "00dd083e02da959679b1b5996f9b64cafe2cff75"Results will be posted here when complete File an issue against this benchmark runner |
adriangbot
commented
Aug 13, 2026
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 00dd083 (00dd083) to 7ff49f9diff Run configurationrun benchmark in_list_strategybaseline:
ref: "7ff49f94d14f73151a55f59314247c4c223073f2"changed:
ref: "00dd083e02da959679b1b5996f9b64cafe2cff75"CPU Details (lscpu)DetailsResource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
geoffreyclaude
commented
Aug 14, 2026
ContributorAuthor
@alamb thank you! I'm reworked a bit the ordering of PRs, the next logical one to me should be #24283 , and it should now be ready for review. The next two are pretty much ready as well, just keeping them in draft so we can rebase over main for easier review as we go. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Note
Until #24102 merges, GitHub's Files changed tab also shows the earlier PRs in the stack. The direct diff from #24102 to this PR shows only this PR's byte-view changes.
Rationale for this change
Arrow represents every
Utf8ViewandBinaryViewvalue with a 16-byte view. When a value is at most 12 bytes long, the complete value is stored directly in that view:Equal short values therefore have the same 128-bit view. DataFusion can look up that view directly, without reading another buffer or using Arrow's general value comparison.
This PR uses that faster path when every non-null value in the
INlist is at most 12 bytes long. The byte-view adapter treats each inline view as aDecimal128key and passes it to the shared primitive selector from #24283, the same selector used by #24102:Decimal128is only a 16-byte container here; no decimal operations are performed.A longer input value cannot match a short list value because its encoded length is different, so it can be rejected without reading its backing bytes. If the list itself contains a value longer than 12 bytes, DataFusion keeps using the general filter.
Null list entries do not affect this choice, but are still recorded for SQL null behavior. After unwrapping any dictionary, the input and list must use the same view type:
Utf8Viewis not mixed withBinaryView, and regularUtf8andBinarykeep their existing paths.What changes are included in this PR?
Utf8ViewandBinaryView.IN, andNOT IN.Are these changes tested?
Tests cover both view types and both faster paths; the 4/5-value and 12/13-byte boundaries; long inputs and lists that require the general filter; exact-type routing; slices and dictionaries; input and list nulls; and
INandNOT IN.Are there any user-facing changes?
No. SQL results and public APIs are unchanged.
Automated benchmark snapshot
Automated run on an aarch64 Neoverse V2 runner, comparing #24102 with this PR at exact commit
00dd083e. The benchmark evaluates 8,192-row batches after filter construction. Lower is better.All 15 reported rows improved, with a 35.9% geometric-mean time reduction.
utf8view/short_8b/list=4/match=0%utf8view/short_8b/list=4/match=50%utf8view/short_8b/list=16/match=0%utf8view/short_8b/list=16/match=50%utf8view/short_8b/list=64/match=0%utf8view/short_8b/list=64/match=50%utf8view/short_8b/list=256/match=0%utf8view/short_8b/list=256/match=50%utf8view/len_12b/list=16/match=0%utf8view/len_12b/list=16/match=50%utf8view/len_12b/list=64/match=0%utf8view/len_12b/list=64/match=50%nulls/utf8view/short_8b/list=16/match=50%/nulls=20%nulls/utf8view/short_8b/list=16/match=50%/nulls=20%/NOT_INnulls/utf8view/short_8b/list=16/match=50%/nulls=50%