perf: Optimize semi-, anti-join implementation using existence probes by neilconway · Pull Request #22914 · apache/datafusion

neilconway · 2026-06-11T16:08:25Z

Which issue does this PR close?

ClosesOptimize RightSemi, RightAnti joins with existence-probes #22854

Rationale for this change

Suppose we have A semijoin B, and the optimizer chooses to implement that as as a RightSemi join. That means we build B and stream A, doing lookups against the B hash table. The current implementation does the following:

Get the hash values for a batch of A rows
Lookup all the B hash table entries with matching hash values
Filter the candidate pairs by true key equality, discarding hash collisions; for each match, produce a (probe_idx, build_idx) pair.
Apply the join filter if there's a non-equijoin filter
Remove duplicates from the list of all probe_idx values
Materialize the output RecordBatch using the distinct probe_idx values

Some of this work is redundant for RightSemi joins (and RightAnti joins as well):

We can stop once we hit the first matching B value, rather than walking the rest of the chain. This means we can both do less hash table traversal and do fewer equality comparisons.
We can skip removing duplicates from the probe_idx array, since we never produced them in the first place

This PR adds an existence-probe fast path that emits each probe row at most once and stops each row's hash-chain search at its first key-equal match, while keeping key comparison fully vectorized.

Benchmarks: (Arm64)

right_semi_d100_h100 4.977 ms 0.981 ms -80.3%
right_anti_d100_h100 1.892 ms 1.132 ms -40.2%
right_semi_d100_h10 0.886 ms 0.438 ms -50.6%
right_anti_d100_h10 3.281 ms 0.765 ms -76.7%
right_semi_d100_h50 2.754 ms 0.751 ms -72.7%
right_anti_d100_h50 2.739 ms 0.981 ms -64.2%
right_semi_d50_h100 5.062 ms 0.999 ms -80.3%
right_anti_d50_h100 1.921 ms 1.128 ms -41.3%
right_semi_d50_h10 0.890 ms 0.441 ms -50.5%
right_anti_d50_h10 3.311 ms 0.767 ms -76.8%
right_semi_d10_h100 7.314 ms 3.840 ms -47.5%
right_anti_d10_h100 4.260 ms 3.932 ms -7.7%
right_semi_d10_h10 7.604 ms 7.253 ms -4.6%
right_anti_d10_h10 10.111 ms 7.573 ms -25.1%
right_semi_d10_h50 7.234 ms 4.705 ms -35.0%
right_anti_d10_h50 7.305 ms 4.943 ms -32.3%
right_semi_fanout100_h1 2.903 ms 0.414 ms -85.7%
right_anti_fanout100_h1 6.070 ms 0.935 ms -84.6%
right_semi_fanout10_h50 11.007 ms 0.921 ms -91.6%
right_anti_fanout10_h50 10.840 ms 1.118 ms -89.7%
right_semi_fanout10_h50_hashmap 17.153 ms 2.622 ms -84.7%
right_anti_fanout10_h50_hashmap 16.945 ms 2.810 ms -83.4%
right_semi_skewed_h50_hashmap 8.621 ms 7.428 ms -13.8%
right_anti_skewed_h50_hashmap 8.752 ms 7.765 ms -11.3%
right_semi_utf8_h50 11.757 ms 8.427 ms -28.3%
right_anti_utf8_h50 11.789 ms 8.690 ms -26.3%
right_semi_utf8_fanout10_h50 53.618 ms 6.959 ms -87.0%
right_anti_utf8_fanout10_h50 53.025 ms 7.064 ms -86.7%

As expected, this is a massive win for high fanout scenarios (in which we would previously have produced many intermediate duplicates), but it's still a win across the board.

What changes are included in this PR?

Core change

Introduce a new "existence probe" code path, which we use for RightSemi and RightAnti, provided the join has no join filter and the build side is non-empty.
For HashMap, the existence probe walks candidate hash chains in lockstep. In each round, we gather the next build-side candidates for every not-yet-matched probe row and do a single vectorized comparison. Any rows that did not match are advanced to the next round; since hash collisions will be very rare in practice, this will involve a single round almost always.
For ArrayMap, we can do a single bucket probe per row.
Fast paths for several reasonably common scenarios: (1) every row in the probe chunk matches, (2) all HashMap chains are of length 1, and (3) emitted probe indices form a contiguous run.

Supporting changes

Refactor and share is_contiguous_range from SMJ
Add an ExistenceProbe trait. JoinHashMapU32/U64 implement it; PruningJoinHashMap does not
Add a ProbeScratch abstraction to reuse scratch space between probe chunks
Refactor equal_rows_arr so we can reuse the vectorized comparison logic
Significantly broaden semi/anti join benchmark coverage
Add unit tests

Are these changes tested?

Yes; new tests added.

Are there any user-facing changes?

No.

neilconway · 2026-06-11T16:14:46Z

This PR includes a fix for ArrayMap probe key -> bucket mapping logic, which I've sent as a separate PR (fix: Reject out-of-range ArrayMap probe keys on 32-bit targets #22911); I'll adjust this PR once that lands
There's also a concurrent PR to fix HashMap behavior for build/probe rows with NULLs in their join keys (fix: Omit NULL values from build side of hash joins #22893); I'll adjust this PR once that lands

Future work:

I'd expect this approach should also be a win for mark joins; I'll implement that in a follow-on PR (Optimize mark joins using existence probes #22880)
Optimizer should bias toward RightSemi: Optimizer should bias to prefer RightSemi over LeftSemi #22931
Consider dup-elim on RightSemi build side: Consider optimizing RightSemi to eliminate dups from build side #22930

neilconway · 2026-06-11T16:14:55Z

run benchmarks

adriangbot · 2026-06-11T16:17:31Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4682608054-535-kgwth 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing neilc/perf-semi-join-existence-gather (46c5e62) to b8998c7 (merge-base) diff using: tpcds
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-06-11T16:17:38Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4682608054-534-gdm7k 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing neilc/perf-semi-join-existence-gather (46c5e62) to b8998c7 (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-06-11T16:18:36Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4682608054-536-8d8jr 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing neilc/perf-semi-join-existence-gather (46c5e62) to b8998c7 (merge-base) diff using: tpch
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-06-11T16:33:43Z

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

Comparing HEAD and neilc_perf-semi-join-existence-gather
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ neilc_perf-semi-join-existence-gather ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1 │ 38.07 / 39.67 ±1.29 / 41.52 ms │ 38.32 / 39.34 ±1.31 / 41.89 ms │ no change │
│ QQuery 2 │ 18.37 / 19.18 ±0.57 / 19.99 ms │ 18.49 / 19.16 ±0.52 / 19.96 ms │ no change │
│ QQuery 3 │ 30.70 / 31.71 ±1.12 / 33.22 ms │ 30.45 / 31.66 ±1.16 / 33.07 ms │ no change │
│ QQuery 4 │ 16.97 / 17.27 ±0.25 / 17.59 ms │ 17.01 / 17.47 ±0.64 / 18.75 ms │ no change │
│ QQuery 5 │ 37.54 / 39.35 ±1.39 / 40.84 ms │ 37.88 / 39.17 ±1.01 / 40.04 ms │ no change │
│ QQuery 6 │ 15.83 / 16.65 ±0.48 / 17.32 ms │ 15.99 / 16.12 ±0.11 / 16.27 ms │ no change │
│ QQuery 7 │ 44.98 / 47.40 ±1.94 / 50.60 ms │ 43.39 / 46.19 ±2.48 / 50.42 ms │ no change │
│ QQuery 8 │ 42.16 / 42.76 ±0.72 / 44.17 ms │ 42.50 / 42.90 ±0.34 / 43.50 ms │ no change │
│ QQuery 9 │ 48.62 / 49.82 ±0.89 / 51.17 ms │ 48.84 / 50.34 ±1.48 / 53.13 ms │ no change │
│ QQuery 10 │ 41.46 / 42.19 ±1.16 / 44.49 ms │ 41.67 / 41.77 ±0.06 / 41.86 ms │ no change │
│ QQuery 11 │ 12.82 / 13.14 ±0.27 / 13.51 ms │ 12.75 / 13.23 ±0.52 / 14.20 ms │ no change │
│ QQuery 12 │ 23.68 / 24.02 ±0.28 / 24.37 ms │ 23.90 / 24.11 ±0.21 / 24.47 ms │ no change │
│ QQuery 13 │ 31.93 / 33.48 ±0.81 / 34.31 ms │ 32.89 / 35.28 ±2.14 / 38.85 ms │ 1.05x slower │
│ QQuery 14 │ 23.32 / 23.51 ±0.13 / 23.71 ms │ 23.53 / 23.64 ±0.07 / 23.73 ms │ no change │
│ QQuery 15 │ 30.49 / 31.93 ±0.89 / 32.95 ms │ 30.93 / 31.81 ±0.81 / 33.03 ms │ no change │
│ QQuery 16 │ 13.83 / 14.07 ±0.30 / 14.65 ms │ 13.99 / 14.26 ±0.30 / 14.73 ms │ no change │
│ QQuery 17 │ 72.29 / 72.82 ±0.51 / 73.79 ms │ 72.50 / 73.36 ±0.77 / 74.31 ms │ no change │
│ QQuery 18 │ 57.78 / 59.44 ±1.06 / 60.77 ms │ 57.41 / 58.27 ±0.51 / 58.92 ms │ no change │
│ QQuery 19 │ 32.93 / 34.03 ±0.77 / 35.04 ms │ 32.91 / 33.33 ±0.44 / 34.11 ms │ no change │
│ QQuery 20 │ 31.36 / 31.61 ±0.19 / 31.87 ms │ 31.72 / 32.12 ±0.31 / 32.49 ms │ no change │
│ QQuery 21 │ 54.35 / 55.62 ±1.59 / 58.69 ms │ 52.87 / 54.85 ±1.37 / 57.07 ms │ no change │
│ QQuery 22 │ 13.47 / 13.78 ±0.28 / 14.20 ms │ 13.49 / 13.94 ±0.55 / 15.03 ms │ no change │
└───────────┴────────────────────────────────┴───────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 753.44ms │
│ Total Time (neilc_perf-semi-join-existence-gather) │ 752.32ms │
│ Average Time (HEAD) │ 34.25ms │
│ Average Time (neilc_perf-semi-join-existence-gather) │ 34.20ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 21 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────────────────┴──────────┘

Resource Usage

tpch — base (merge-base)

Metric	Value
Wall time	5.0s
Peak memory	1.2 GiB
Avg memory	530.8 MiB
CPU user	21.7s
CPU sys	1.6s
Peak spill	0 B

tpch — branch

Metric	Value
Wall time	5.0s
Peak memory	1.3 GiB
Avg memory	533.2 MiB
CPU user	21.7s
CPU sys	1.6s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-06-11T16:35:34Z

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

Comparing HEAD and neilc_perf-semi-join-existence-gather
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ neilc_perf-semi-join-existence-gather ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 5.36 / 5.89 ±0.96 / 7.82 ms │ 5.58 / 6.07 ±0.93 / 7.93 ms │ no change │
│ QQuery 2 │ 81.94 / 82.27 ±0.34 / 82.84 ms │ 80.97 / 81.79 ±0.42 / 82.09 ms │ no change │
│ QQuery 3 │ 28.66 / 29.10 ±0.27 / 29.38 ms │ 28.48 / 28.80 ±0.20 / 29.02 ms │ no change │
│ QQuery 4 │ 476.60 / 481.70 ±2.63 / 483.63 ms │ 472.55 / 478.78 ±4.66 / 486.14 ms │ no change │
│ QQuery 5 │ 51.78 / 54.82 ±5.17 / 65.14 ms │ 50.96 / 51.77 ±0.54 / 52.67 ms │ +1.06x faster │
│ QQuery 6 │ 35.68 / 36.21 ±0.46 / 36.94 ms │ 36.13 / 36.48 ±0.30 / 37.02 ms │ no change │
│ QQuery 7 │ 95.14 / 95.80 ±0.66 / 96.94 ms │ 94.51 / 95.29 ±0.47 / 95.77 ms │ no change │
│ QQuery 8 │ 36.37 / 36.66 ±0.20 / 36.90 ms │ 36.86 / 38.97 ±2.70 / 44.19 ms │ 1.06x slower │
│ QQuery 9 │ 53.10 / 54.70 ±1.89 / 58.40 ms │ 53.29 / 55.87 ±2.29 / 59.99 ms │ no change │
│ QQuery 10 │ 62.29 / 62.83 ±0.27 / 63.05 ms │ 61.98 / 62.14 ±0.12 / 62.33 ms │ no change │
│ QQuery 11 │ 290.96 / 294.68 ±3.45 / 300.91 ms │ 293.49 / 297.32 ±3.20 / 303.23 ms │ no change │
│ QQuery 12 │ 28.76 / 28.95 ±0.17 / 29.19 ms │ 28.10 / 28.53 ±0.31 / 29.02 ms │ no change │
│ QQuery 13 │ 118.96 / 119.28 ±0.39 / 119.97 ms │ 118.60 / 118.85 ±0.17 / 119.10 ms │ no change │
│ QQuery 14 │ 414.95 / 416.84 ±1.23 / 418.20 ms │ 394.96 / 399.56 ±4.07 / 404.78 ms │ no change │
│ QQuery 15 │ 59.37 / 60.98 ±1.99 / 64.90 ms │ 58.53 / 59.77 ±1.24 / 62.16 ms │ no change │
│ QQuery 16 │ 6.66 / 6.72 ±0.08 / 6.87 ms │ 6.59 / 6.72 ±0.10 / 6.89 ms │ no change │
│ QQuery 17 │ 80.21 / 81.13 ±0.94 / 82.86 ms │ 80.19 / 81.00 ±0.84 / 82.52 ms │ no change │
│ QQuery 18 │ 123.81 / 125.37 ±1.14 / 127.21 ms │ 124.63 / 126.30 ±1.56 / 128.16 ms │ no change │
│ QQuery 19 │ 41.14 / 41.69 ±0.32 / 42.15 ms │ 40.55 / 41.07 ±0.32 / 41.53 ms │ no change │
│ QQuery 20 │ 35.08 / 35.62 ±0.34 / 36.12 ms │ 34.85 / 35.07 ±0.19 / 35.39 ms │ no change │
│ QQuery 21 │ 17.32 / 17.95 ±1.07 / 20.08 ms │ 17.29 / 17.41 ±0.09 / 17.51 ms │ no change │
│ QQuery 22 │ 62.56 / 62.88 ±0.33 / 63.36 ms │ 60.76 / 61.17 ±0.42 / 61.86 ms │ no change │
│ QQuery 23 │ 344.61 / 348.99 ±3.13 / 353.91 ms │ 340.74 / 343.99 ±4.50 / 352.89 ms │ no change │
│ QQuery 24 │ 224.26 / 226.15 ±2.04 / 230.02 ms │ 224.93 / 226.71 ±1.40 / 229.00 ms │ no change │
│ QQuery 25 │ 109.29 / 110.01 ±0.70 / 111.08 ms │ 109.38 / 110.75 ±1.30 / 113.21 ms │ no change │
│ QQuery 26 │ 57.42 / 59.10 ±2.62 / 64.32 ms │ 56.72 / 57.25 ±0.30 / 57.57 ms │ no change │
│ QQuery 27 │ 5.99 / 6.11 ±0.15 / 6.40 ms │ 6.24 / 8.07 ±3.35 / 14.76 ms │ 1.32x slower │
│ QQuery 28 │ 55.54 / 60.48 ±2.73 / 63.97 ms │ 56.49 / 59.39 ±2.26 / 61.53 ms │ no change │
│ QQuery 29 │ 96.05 / 97.21 ±1.32 / 99.74 ms │ 96.84 / 97.54 ±0.95 / 99.41 ms │ no change │
│ QQuery 30 │ 31.85 / 33.63 ±2.37 / 38.21 ms │ 31.44 / 31.85 ±0.31 / 32.39 ms │ +1.06x faster │
│ QQuery 31 │ 110.73 / 111.41 ±0.67 / 112.59 ms │ 110.80 / 112.72 ±2.72 / 118.09 ms │ no change │
│ QQuery 32 │ 19.85 / 20.26 ±0.41 / 21.00 ms │ 19.84 / 20.26 ±0.42 / 20.93 ms │ no change │
│ QQuery 33 │ 37.32 / 37.63 ±0.23 / 38.02 ms │ 36.49 / 37.36 ±0.55 / 37.98 ms │ no change │
│ QQuery 34 │ 9.53 / 9.77 ±0.22 / 10.16 ms │ 9.64 / 9.76 ±0.15 / 10.06 ms │ no change │
│ QQuery 35 │ 71.83 / 73.30 ±1.41 / 75.75 ms │ 70.58 / 73.01 ±3.05 / 78.81 ms │ no change │
│ QQuery 36 │ 5.66 / 5.80 ±0.15 / 6.08 ms │ 5.69 / 5.82 ±0.18 / 6.17 ms │ no change │
│ QQuery 37 │ 6.73 / 6.77 ±0.05 / 6.88 ms │ 6.67 / 6.98 ±0.26 / 7.45 ms │ no change │
│ QQuery 38 │ 62.76 / 63.05 ±0.25 / 63.34 ms │ 62.31 / 63.15 ±0.70 / 64.41 ms │ no change │
│ QQuery 39 │ 456.58 / 459.02 ±2.09 / 462.13 ms │ 452.97 / 456.43 ±2.34 / 459.89 ms │ no change │
│ QQuery 40 │ 23.47 / 25.84 ±4.06 / 33.94 ms │ 22.82 / 23.12 ±0.23 / 23.47 ms │ +1.12x faster │
│ QQuery 41 │ 11.30 / 11.46 ±0.18 / 11.78 ms │ 11.50 / 13.42 ±3.60 / 20.61 ms │ 1.17x slower │
│ QQuery 42 │ 24.26 / 24.75 ±0.61 / 25.91 ms │ 23.48 / 24.22 ±0.89 / 25.93 ms │ no change │
│ QQuery 43 │ 4.91 / 5.01 ±0.12 / 5.24 ms │ 5.04 / 5.12 ±0.06 / 5.20 ms │ no change │
│ QQuery 44 │ 9.16 / 9.33 ±0.17 / 9.65 ms │ 9.28 / 9.41 ±0.08 / 9.51 ms │ no change │
│ QQuery 45 │ 37.92 / 38.20 ±0.26 / 38.62 ms │ 37.49 / 38.77 ±1.07 / 40.73 ms │ no change │
│ QQuery 46 │ 11.47 / 11.79 ±0.39 / 12.56 ms │ 11.49 / 11.74 ±0.17 / 11.96 ms │ no change │
│ QQuery 47 │ 227.61 / 231.20 ±3.71 / 238.23 ms │ 223.79 / 226.70 ±3.41 / 233.29 ms │ no change │
│ QQuery 48 │ 96.78 / 98.91 ±2.77 / 104.36 ms │ 95.39 / 97.99 ±4.37 / 106.70 ms │ no change │
│ QQuery 49 │ 76.44 / 77.24 ±0.59 / 78.03 ms │ 76.22 / 76.94 ±0.59 / 77.99 ms │ no change │
│ QQuery 50 │ 58.15 / 58.65 ±0.43 / 59.40 ms │ 58.11 / 59.04 ±0.66 / 59.91 ms │ no change │
│ QQuery 51 │ 92.41 / 93.77 ±1.05 / 95.17 ms │ 92.35 / 96.38 ±3.94 / 103.82 ms │ no change │
│ QQuery 52 │ 23.44 / 23.83 ±0.27 / 24.29 ms │ 23.73 / 24.37 ±0.59 / 25.39 ms │ no change │
│ QQuery 53 │ 29.16 / 29.41 ±0.17 / 29.61 ms │ 29.08 / 29.65 ±0.53 / 30.45 ms │ no change │
│ QQuery 54 │ 53.60 / 54.08 ±0.51 / 55.03 ms │ 53.49 / 53.89 ±0.24 / 54.19 ms │ no change │
│ QQuery 55 │ 23.15 / 25.36 ±3.51 / 32.35 ms │ 22.76 / 22.95 ±0.18 / 23.27 ms │ +1.10x faster │
│ QQuery 56 │ 38.49 / 39.55 ±0.64 / 40.40 ms │ 37.58 / 38.16 ±0.44 / 38.78 ms │ no change │
│ QQuery 57 │ 177.17 / 180.71 ±3.77 / 187.77 ms │ 175.89 / 177.17 ±1.49 / 180.00 ms │ no change │
│ QQuery 58 │ 115.42 / 117.70 ±1.43 / 119.25 ms │ 115.05 / 117.01 ±3.24 / 123.44 ms │ no change │
│ QQuery 59 │ 119.46 / 121.37 ±2.45 / 126.08 ms │ 117.76 / 118.53 ±0.54 / 119.06 ms │ no change │
│ QQuery 60 │ 39.18 / 39.86 ±0.87 / 41.48 ms │ 38.82 / 39.86 ±0.84 / 40.96 ms │ no change │
│ QQuery 61 │ 11.62 / 11.75 ±0.15 / 12.01 ms │ 11.82 / 12.06 ±0.24 / 12.48 ms │ no change │
│ QQuery 62 │ 46.66 / 47.02 ±0.25 / 47.44 ms │ 46.44 / 46.60 ±0.14 / 46.82 ms │ no change │
│ QQuery 63 │ 29.09 / 29.53 ±0.30 / 29.88 ms │ 29.14 / 29.47 ±0.35 / 30.12 ms │ no change │
│ QQuery 64 │ 407.43 / 415.51 ±7.55 / 428.82 ms │ 405.76 / 412.41 ±4.70 / 419.32 ms │ no change │
│ QQuery 65 │ 146.57 / 149.51 ±1.63 / 151.43 ms │ 151.57 / 155.80 ±2.62 / 158.69 ms │ no change │
│ QQuery 66 │ 79.37 / 81.24 ±2.24 / 85.45 ms │ 79.36 / 79.91 ±0.57 / 80.96 ms │ no change │
│ QQuery 67 │ 241.41 / 247.40 ±4.53 / 254.47 ms │ 236.99 / 244.30 ±6.27 / 254.92 ms │ no change │
│ QQuery 68 │ 11.56 / 11.89 ±0.22 / 12.13 ms │ 11.72 / 12.10 ±0.55 / 13.17 ms │ no change │
│ QQuery 69 │ 56.63 / 57.29 ±0.53 / 58.20 ms │ 55.80 / 56.17 ±0.31 / 56.56 ms │ no change │
│ QQuery 70 │ 104.62 / 109.59 ±6.72 / 122.68 ms │ 104.04 / 108.87 ±7.78 / 124.37 ms │ no change │
│ QQuery 71 │ 35.25 / 35.59 ±0.30 / 36.04 ms │ 34.63 / 35.11 ±0.27 / 35.41 ms │ no change │
│ QQuery 72 │ 2185.55 / 2219.20 ±35.22 / 2272.08 ms │ 1938.58 / 2050.93 ±68.15 / 2148.50 ms │ +1.08x faster │
│ QQuery 73 │ 9.73 / 10.04 ±0.26 / 10.44 ms │ 9.37 / 9.59 ±0.35 / 10.29 ms │ no change │
│ QQuery 74 │ 169.00 / 173.29 ±4.85 / 182.68 ms │ 167.25 / 168.82 ±1.86 / 172.40 ms │ no change │
│ QQuery 75 │ 147.93 / 152.36 ±8.11 / 168.57 ms │ 147.14 / 151.05 ±4.58 / 159.96 ms │ no change │
│ QQuery 76 │ 35.29 / 36.62 ±0.80 / 37.57 ms │ 34.94 / 35.74 ±0.52 / 36.52 ms │ no change │
│ QQuery 77 │ 60.16 / 61.11 ±0.76 / 62.12 ms │ 60.08 / 60.46 ±0.44 / 61.31 ms │ no change │
│ QQuery 78 │ 184.99 / 189.42 ±3.98 / 196.50 ms │ 185.18 / 186.70 ±1.03 / 188.34 ms │ no change │
│ QQuery 79 │ 66.76 / 69.79 ±5.71 / 81.20 ms │ 65.49 / 65.75 ±0.26 / 66.26 ms │ +1.06x faster │
│ QQuery 80 │ 98.93 / 101.01 ±1.82 / 103.82 ms │ 97.62 / 100.85 ±3.57 / 107.15 ms │ no change │
│ QQuery 81 │ 25.15 / 25.60 ±0.26 / 25.86 ms │ 25.16 / 25.41 ±0.18 / 25.63 ms │ no change │
│ QQuery 82 │ 16.04 / 16.29 ±0.18 / 16.60 ms │ 16.29 / 16.43 ±0.15 / 16.72 ms │ no change │
│ QQuery 83 │ 39.23 / 39.47 ±0.12 / 39.58 ms │ 39.38 / 40.01 ±0.92 / 41.83 ms │ no change │
│ QQuery 84 │ 29.98 / 31.53 ±2.59 / 36.70 ms │ 29.49 / 29.96 ±0.28 / 30.28 ms │ no change │
│ QQuery 85 │ 107.64 / 108.23 ±0.53 / 109.16 ms │ 107.33 / 110.26 ±4.28 / 118.58 ms │ no change │
│ QQuery 86 │ 24.67 / 24.98 ±0.28 / 25.38 ms │ 24.08 / 24.67 ±0.67 / 25.96 ms │ no change │
│ QQuery 87 │ 63.04 / 65.19 ±3.08 / 71.22 ms │ 62.55 / 63.18 ±0.73 / 64.35 ms │ no change │
│ QQuery 88 │ 61.54 / 62.57 ±1.15 / 64.82 ms │ 59.51 / 62.13 ±3.31 / 68.26 ms │ no change │
│ QQuery 89 │ 35.42 / 35.89 ±0.43 / 36.54 ms │ 35.20 / 35.83 ±0.49 / 36.39 ms │ no change │
│ QQuery 90 │ 16.18 / 16.35 ±0.17 / 16.63 ms │ 16.34 / 16.48 ±0.12 / 16.62 ms │ no change │
│ QQuery 91 │ 45.27 / 45.78 ±0.30 / 46.17 ms │ 44.78 / 45.23 ±0.40 / 45.87 ms │ no change │
│ QQuery 92 │ 28.52 / 30.74 ±3.78 / 38.29 ms │ 28.55 / 29.31 ±0.79 / 30.64 ms │ no change │
│ QQuery 93 │ 50.09 / 50.97 ±0.76 / 52.34 ms │ 50.17 / 51.69 ±1.48 / 54.52 ms │ no change │
│ QQuery 94 │ 37.62 / 38.19 ±0.56 / 39.19 ms │ 37.67 / 38.69 ±1.29 / 40.89 ms │ no change │
│ QQuery 95 │ 80.37 / 80.85 ±0.41 / 81.45 ms │ 80.66 / 81.69 ±0.72 / 82.58 ms │ no change │
│ QQuery 96 │ 23.50 / 23.78 ±0.25 / 24.20 ms │ 23.31 / 23.60 ±0.25 / 23.97 ms │ no change │
│ QQuery 97 │ 45.91 / 47.50 ±2.48 / 52.44 ms │ 44.06 / 44.69 ±0.40 / 45.21 ms │ +1.06x faster │
│ QQuery 98 │ 42.71 / 44.29 ±1.58 / 46.67 ms │ 42.39 / 43.05 ±0.68 / 44.22 ms │ no change │
│ QQuery 99 │ 70.88 / 71.04 ±0.14 / 71.29 ms │ 69.76 / 70.31 ±0.62 / 71.51 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 10347.15ms │
│ Total Time (neilc_perf-semi-join-existence-gather) │ 10109.54ms │
│ Average Time (HEAD) │ 104.52ms │
│ Average Time (neilc_perf-semi-join-existence-gather) │ 102.12ms │
│ Queries Faster │ 7 │
│ Queries Slower │ 3 │
│ Queries with No Change │ 89 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────────────────┴────────────┘

Resource Usage

tpcds — base (merge-base)

Metric	Value
Wall time	55.0s
Peak memory	2.0 GiB
Avg memory	1.4 GiB
CPU user	237.1s
CPU sys	5.9s
Peak spill	0 B

tpcds — branch

Metric	Value
Wall time	55.0s
Peak memory	2.2 GiB
Avg memory	1.5 GiB
CPU user	225.6s
CPU sys	5.8s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-06-11T16:38:47Z

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

Comparing HEAD and neilc_perf-semi-join-existence-gather
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ neilc_perf-semi-join-existence-gather ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.20 / 3.31 ±4.13 / 11.56 ms │ 1.16 / 3.30 ±4.19 / 11.68 ms │ no change │
│ QQuery 1 │ 12.35 / 12.69 ±0.22 / 13.05 ms │ 12.65 / 13.00 ±0.36 / 13.63 ms │ no change │
│ QQuery 2 │ 35.66 / 35.91 ±0.25 / 36.26 ms │ 35.28 / 35.60 ±0.22 / 35.95 ms │ no change │
│ QQuery 3 │ 29.81 / 30.62 ±0.78 / 31.95 ms │ 30.07 / 30.32 ±0.27 / 30.82 ms │ no change │
│ QQuery 4 │ 220.38 / 223.56 ±2.70 / 227.66 ms │ 219.66 / 224.22 ±3.86 / 228.55 ms │ no change │
│ QQuery 5 │ 268.45 / 271.82 ±3.65 / 278.55 ms │ 267.99 / 270.63 ±1.71 / 273.20 ms │ no change │
│ QQuery 6 │ 1.22 / 1.38 ±0.22 / 1.81 ms │ 1.21 / 1.36 ±0.22 / 1.79 ms │ no change │
│ QQuery 7 │ 13.38 / 13.48 ±0.07 / 13.57 ms │ 13.73 / 13.87 ±0.19 / 14.24 ms │ no change │
│ QQuery 8 │ 315.40 / 319.80 ±3.32 / 325.43 ms │ 317.26 / 320.15 ±2.17 / 323.53 ms │ no change │
│ QQuery 9 │ 444.70 / 451.05 ±4.78 / 459.47 ms │ 442.00 / 451.72 ±6.28 / 461.74 ms │ no change │
│ QQuery 10 │ 68.36 / 69.07 ±0.60 / 70.01 ms │ 67.06 / 68.26 ±0.94 / 69.92 ms │ no change │
│ QQuery 11 │ 79.45 / 82.15 ±4.40 / 90.94 ms │ 78.29 / 78.97 ±0.64 / 80.18 ms │ no change │
│ QQuery 12 │ 263.57 / 269.72 ±5.13 / 276.75 ms │ 261.98 / 272.93 ±9.07 / 288.30 ms │ no change │
│ QQuery 13 │ 350.77 / 365.13 ±10.94 / 377.62 ms │ 362.54 / 378.01 ±15.27 / 400.75 ms │ no change │
│ QQuery 14 │ 274.93 / 278.86 ±3.84 / 285.15 ms │ 277.33 / 280.81 ±3.25 / 285.61 ms │ no change │
│ QQuery 15 │ 265.15 / 273.08 ±8.88 / 288.14 ms │ 264.71 / 275.32 ±14.58 / 304.22 ms │ no change │
│ QQuery 16 │ 598.39 / 606.06 ±7.24 / 618.09 ms │ 596.98 / 605.39 ±5.52 / 614.27 ms │ no change │
│ QQuery 17 │ 598.02 / 608.07 ±5.66 / 615.56 ms │ 605.10 / 613.97 ±10.59 / 634.80 ms │ no change │
│ QQuery 18 │ 1212.29 / 1250.00 ±25.81 / 1280.86 ms │ 1212.84 / 1238.05 ±20.25 / 1261.54 ms │ no change │
│ QQuery 19 │ 27.52 / 28.44 ±0.91 / 29.79 ms │ 27.45 / 28.08 ±0.70 / 29.44 ms │ no change │
│ QQuery 20 │ 515.68 / 528.45 ±9.14 / 538.79 ms │ 516.42 / 522.90 ±5.03 / 530.07 ms │ no change │
│ QQuery 21 │ 513.06 / 519.04 ±9.64 / 538.20 ms │ 514.03 / 521.56 ±5.15 / 528.52 ms │ no change │
│ QQuery 22 │ 977.79 / 992.03 ±11.35 / 1012.23 ms │ 986.95 / 993.17 ±7.80 / 1007.91 ms │ no change │
│ QQuery 23 │ 3181.19 / 3217.57 ±31.34 / 3252.34 ms │ 3164.77 / 3195.82 ±20.37 / 3213.90 ms │ no change │
│ QQuery 24 │ 41.72 / 44.83 ±3.44 / 49.72 ms │ 40.81 / 42.12 ±1.86 / 45.80 ms │ +1.06x faster │
│ QQuery 25 │ 109.50 / 111.15 ±1.11 / 112.34 ms │ 111.15 / 113.41 ±2.73 / 118.11 ms │ no change │
│ QQuery 26 │ 41.54 / 43.40 ±2.07 / 46.67 ms │ 41.25 / 41.52 ±0.36 / 42.22 ms │ no change │
│ QQuery 27 │ 658.63 / 667.09 ±7.17 / 680.10 ms │ 664.97 / 670.13 ±5.06 / 677.91 ms │ no change │
│ QQuery 28 │ 2998.16 / 3025.46 ±14.97 / 3039.60 ms │ 3019.78 / 3044.77 ±17.96 / 3075.11 ms │ no change │
│ QQuery 29 │ 40.37 / 44.82 ±5.30 / 53.31 ms │ 40.08 / 42.56 ±2.98 / 48.24 ms │ +1.05x faster │
│ QQuery 30 │ 296.09 / 298.34 ±1.76 / 300.11 ms │ 295.77 / 299.38 ±2.68 / 303.05 ms │ no change │
│ QQuery 31 │ 274.25 / 288.19 ±7.19 / 294.17 ms │ 287.43 / 297.14 ±8.97 / 311.68 ms │ no change │
│ QQuery 32 │ 914.06 / 949.09 ±28.42 / 993.50 ms │ 944.97 / 968.09 ±18.89 / 993.15 ms │ no change │
│ QQuery 33 │ 1412.74 / 1439.28 ±18.19 / 1462.52 ms │ 1416.85 / 1459.87 ±29.12 / 1488.28 ms │ no change │
│ QQuery 34 │ 1455.54 / 1488.50 ±20.09 / 1514.92 ms │ 1431.17 / 1465.87 ±24.55 / 1505.82 ms │ no change │
│ QQuery 35 │ 271.24 / 306.18 ±38.39 / 372.40 ms │ 270.84 / 277.79 ±7.79 / 290.79 ms │ +1.10x faster │
│ QQuery 36 │ 65.45 / 74.22 ±7.37 / 87.35 ms │ 65.09 / 73.14 ±5.59 / 80.78 ms │ no change │
│ QQuery 37 │ 35.32 / 37.96 ±2.63 / 41.93 ms │ 34.86 / 36.30 ±1.75 / 39.65 ms │ no change │
│ QQuery 38 │ 43.54 / 47.68 ±4.35 / 53.62 ms │ 39.46 / 43.47 ±3.67 / 49.08 ms │ +1.10x faster │
│ QQuery 39 │ 136.26 / 147.99 ±6.81 / 154.89 ms │ 139.93 / 148.06 ±4.51 / 151.62 ms │ no change │
│ QQuery 40 │ 14.27 / 17.12 ±2.89 / 22.23 ms │ 13.47 / 14.88 ±2.19 / 19.22 ms │ +1.15x faster │
│ QQuery 41 │ 13.36 / 14.02 ±0.89 / 15.70 ms │ 13.17 / 13.41 ±0.21 / 13.68 ms │ no change │
│ QQuery 42 │ 12.48 / 15.75 ±6.05 / 27.85 ms │ 12.56 / 13.62 ±1.42 / 16.44 ms │ +1.16x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 19512.34ms │
│ Total Time (neilc_perf-semi-join-existence-gather) │ 19502.92ms │
│ Average Time (HEAD) │ 453.78ms │
│ Average Time (neilc_perf-semi-join-existence-gather) │ 453.56ms │
│ Queries Faster │ 6 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 37 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	100.0s
Peak memory	12.0 GiB
Avg memory	4.7 GiB
CPU user	997.0s
CPU sys	68.0s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	100.0s
Peak memory	11.0 GiB
Avg memory	4.4 GiB
CPU user	996.0s
CPU sys	70.4s
Peak spill	0 B

File an issue against this benchmark runner

Dandandan · 2026-06-12T14:15:15Z

run benchmark tpch10

adriangbot · 2026-06-12T14:18:05Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4692083606-560-kqdjz 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing neilc/perf-semi-join-existence-gather (262c59c) to b8998c7 (merge-base) diff using: tpch10
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-06-12T14:32:27Z

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

Comparing HEAD and neilc_perf-semi-join-existence-gather
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ neilc_perf-semi-join-existence-gather ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 310.92 / 312.21 ±1.52 / 314.73 ms │ 312.13 / 313.92 ±1.23 / 315.71 ms │ no change │
│ QQuery 2 │ 105.19 / 108.17 ±3.98 / 115.95 ms │ 103.94 / 107.48 ±2.80 / 110.59 ms │ no change │
│ QQuery 3 │ 234.39 / 237.18 ±2.02 / 240.00 ms │ 236.11 / 241.31 ±5.82 / 251.86 ms │ no change │
│ QQuery 4 │ 110.85 / 113.87 ±2.74 / 118.20 ms │ 113.81 / 115.07 ±1.07 / 117.04 ms │ no change │
│ QQuery 5 │ 347.83 / 352.45 ±3.70 / 357.82 ms │ 355.35 / 361.44 ±6.64 / 373.63 ms │ no change │
│ QQuery 6 │ 122.68 / 125.18 ±2.11 / 129.08 ms │ 123.70 / 125.16 ±2.02 / 129.13 ms │ no change │
│ QQuery 7 │ 452.53 / 458.10 ±4.53 / 464.56 ms │ 464.74 / 472.00 ±6.97 / 483.84 ms │ no change │
│ QQuery 8 │ 384.46 / 387.52 ±2.82 / 392.84 ms │ 378.43 / 386.39 ±7.38 / 396.89 ms │ no change │
│ QQuery 9 │ 554.99 / 562.49 ±6.93 / 575.05 ms │ 546.12 / 558.53 ±7.45 / 567.11 ms │ no change │
│ QQuery 10 │ 301.14 / 306.88 ±8.09 / 322.57 ms │ 300.11 / 310.23 ±9.34 / 322.96 ms │ no change │
│ QQuery 11 │ 82.35 / 90.87 ±9.41 / 109.08 ms │ 85.71 / 92.27 ±8.37 / 107.74 ms │ no change │
│ QQuery 12 │ 179.35 / 183.55 ±6.81 / 197.11 ms │ 179.24 / 181.77 ±2.82 / 185.25 ms │ no change │
│ QQuery 13 │ 285.29 / 299.21 ±9.57 / 313.35 ms │ 292.35 / 296.34 ±3.78 / 301.95 ms │ no change │
│ QQuery 14 │ 169.71 / 173.95 ±5.62 / 184.20 ms │ 170.33 / 174.84 ±5.45 / 185.49 ms │ no change │
│ QQuery 15 │ 300.86 / 302.29 ±1.22 / 304.13 ms │ 302.21 / 303.72 ±1.91 / 307.26 ms │ no change │
│ QQuery 16 │ 64.89 / 68.99 ±3.03 / 73.77 ms │ 65.13 / 67.68 ±1.85 / 70.73 ms │ no change │
│ QQuery 17 │ 624.76 / 637.05 ±6.68 / 642.28 ms │ 625.06 / 640.80 ±10.47 / 657.10 ms │ no change │
│ QQuery 18 │ 671.60 / 687.73 ±10.76 / 704.40 ms │ 678.84 / 704.88 ±16.00 / 721.25 ms │ no change │
│ QQuery 19 │ 244.77 / 261.50 ±15.51 / 285.75 ms │ 243.98 / 257.03 ±14.68 / 281.44 ms │ no change │
│ QQuery 20 │ 275.78 / 283.20 ±5.81 / 290.30 ms │ 277.83 / 290.48 ±7.52 / 298.52 ms │ no change │
│ QQuery 21 │ 653.29 / 669.81 ±13.50 / 685.70 ms │ 653.37 / 667.49 ±10.59 / 682.10 ms │ no change │
│ QQuery 22 │ 59.54 / 63.62 ±4.63 / 72.53 ms │ 60.53 / 63.75 ±3.55 / 70.59 ms │ no change │
└───────────┴────────────────────────────────────┴───────────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 6685.81ms │
│ Total Time (neilc_perf-semi-join-existence-gather) │ 6732.60ms │
│ Average Time (HEAD) │ 303.90ms │
│ Average Time (neilc_perf-semi-join-existence-gather) │ 306.03ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 22 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────────────────┴───────────┘

Resource Usage

tpch10 — base (merge-base)

Metric	Value
Wall time	35.0s
Peak memory	4.9 GiB
Avg memory	1.5 GiB
CPU user	340.1s
CPU sys	17.6s
Peak spill	0 B

tpch10 — branch

Metric	Value
Wall time	35.0s
Peak memory	4.6 GiB
Avg memory	1.4 GiB
CPU user	342.6s
CPU sys	18.0s
Peak spill	0 B

File an issue against this benchmark runner

neilconway · 2026-06-16T16:21:36Z

@Dandandan Do you have any availability to take a look at this PR?

…existence-gather # Conflicts: # datafusion/physical-plan/src/joins/array_map.rs # datafusion/physical-plan/src/joins/hash_join/stream.rs # datafusion/physical-plan/src/joins/join_hash_map.rs

…existence-gather # Conflicts: # datafusion/physical-plan/src/joins/utils.rs

codecov-commenter · 2026-07-21T17:56:48Z

Codecov Report

❌ Patch coverage is 75.19582% with 190 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.69%. Comparing base (bdd8238) to head (66cf583).
⚠️ Report is 213 commits behind head on main.

Files with missing lines	Patch %	Lines
...fusion/physical-plan/src/joins/hash_join/stream.rs	62.44%	75 Missing and 8 partials ⚠️
...atafusion/physical-plan/src/joins/join_hash_map.rs	61.58%	68 Missing ⚠️
...tafusion/physical-plan/src/joins/hash_join/exec.rs	90.87%	0 Missing and 23 partials ⚠️
datafusion/physical-plan/src/joins/array_map.rs	83.82%	9 Missing and 2 partials ⚠️
datafusion/physical-plan/src/joins/utils.rs	89.58%	3 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #22914 +/- ##
==========================================
- Coverage 80.71% 80.69% -0.02% 
==========================================
Files 1089 1089 Lines 368748 369472 +724 Branches 368748 369472 +724 ==========================================
+ Hits 297641 298160 +519 - Misses 53364 53532 +168 - Partials 17743 17780 +37

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

alamb · 2026-08-10T18:48:15Z

@avantgardnerio do you have some time to help review this PR?

neilconway · 2026-08-10T19:04:37Z

@alamb @avantgardnerio I've been thinking that I should revisit this and split it into smaller pieces that are easier to review. I think the idea fundamentally has merit; I'm happily accepting code reviews now, of course 😊, but I can refactor it first if that's easier to review.

alamb · 2026-08-12T18:25:53Z

@alamb @avantgardnerio I've been thinking that I should revisit this and split it into smaller pieces that are easier to review. I think the idea fundamentally has merit; I'm happily accepting code reviews now, of course 😊, but I can refactor it first if that's easier to review.

Thank you -- that would be helpful.

However I don't want to lead you on -- I am not super familiar with the DataFusion join code and thus it will take me a while to review changes to it (and therefore given my limited time I will likely be a huge bottleneck / not reliable reviewer)

If we want to make real progress on joins (which I do think is strategically quite important) maybe we can find a group of committers / reviewers who are interested in working together to push things along

I tried to organize such a team before in an issue, but nothing came of it:

[DISCUSSION] JOIN "task force" / project team #15885

avantgardnerio · 2026-08-12T18:30:18Z

I am also not super familiar, but I'd like to become so - which is why this review has been sitting in my inbox for a while, sorry about that. I'm still trying to find the time for a proper review.

.
46c5e62

github-actionsBot added the physical-plan Changes to the physical-plan crate label Jun 11, 2026

Fix docs CI
262c59c

neilconway mentioned this pull request Jun 11, 2026
Optimize RightSemi, RightAnti joins with existence-probes #22854
Open

neilconway mentioned this pull request Jun 12, 2026
Consider optimizing RightSemi to eliminate dups from build side #22930
Open

neilconway added 2 commits July 13, 2026 11:13

Merge remote-tracking branch 'origin/main' into neilc/perf-semi-join-…
678fba0
…existence-gather # Conflicts: # datafusion/physical-plan/src/joins/array_map.rs # datafusion/physical-plan/src/joins/hash_join/stream.rs # datafusion/physical-plan/src/joins/join_hash_map.rs

Merge remote-tracking branch 'origin/main' into neilc/perf-semi-join-…
66cf583
…existence-gather # Conflicts: # datafusion/physical-plan/src/joins/utils.rs

Uh oh!

Conversation

neilconway commented Jun 11, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

neilconway commented Jun 11, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neilconway commented Jun 11, 2026

Uh oh!

adriangbot commented Jun 11, 2026

Uh oh!

adriangbot commented Jun 11, 2026

Uh oh!

adriangbot commented Jun 11, 2026

Uh oh!

adriangbot commented Jun 11, 2026

Uh oh!

adriangbot commented Jun 11, 2026

Uh oh!

adriangbot commented Jun 11, 2026

Uh oh!

Dandandan commented Jun 12, 2026

Uh oh!

adriangbot commented Jun 12, 2026

Uh oh!

adriangbot commented Jun 12, 2026

Uh oh!

neilconway commented Jun 16, 2026

Uh oh!

codecov-commenter commented Jul 21, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alamb commented Aug 10, 2026

Uh oh!

neilconway commented Aug 10, 2026

Uh oh!

alamb commented Aug 12, 2026

Uh oh!

avantgardnerio commented Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

neilconway commented Jun 11, 2026•
edited
Loading

neilconway commented Jun 11, 2026•
edited
Loading

codecov-commenter commented Jul 21, 2026•
edited
Loading