fix: correlated NOT IN with a non-equality correlation returns wrong results by adriangb · Pull Request #25339 · apache/datafusion

adriangb · 2026-09-15T17:49:13Z

Which issue does this PR close?

Closes Wrong results: correlated NOT IN with a non-equality correlation returns no rows (null-aware LeftAnti join ignores the residual filter) #25336.

Rationale for this change

A correlated NOT IN subquery gives wrong results when the correlation is not an equality and the subquery column contains NULL. There is no error and no warning.

CREATE TABLE t1(id INT, z INT) AS VALUES (1,10), (2,20), (NULL,30), (4,40);
CREATE TABLE t2(id INT, z INT) AS VALUES (1,5), (NULL,50);

SELECT id FROM t1 WHERE id NOT IN (SELECT t2.id FROM t2 WHERE t2.z < t1.z) ORDER BY id;
-- main: (no rows)    this PR, DuckDB, PostgreSQL: 2, 4

SELECT id FROM t1 WHERE NOT (id IN (SELECT t2.id FROM t2 WHERE t2.z < t1.z)) OR id = 4 ORDER BY id;
-- main: 2, 4, NULL   this PR, DuckDB, PostgreSQL: 2, 4

The same gap also makes an equality-correlated NOT IN in a WHERE clause fail to plan:

SELECT id FROM t1 WHERE id NOT IN (SELECT t2.id FROM t2 WHERE t2.z = t1.z);
-- main: Error during planning: null_aware LeftAnti joins only support single column join key, got 2 columns

The fix sketch in the issue (turn off null_aware for a LeftAnti join that has a join filter) does not work. I tried it: the plain anti join ignores NULLs completely, so queries that are correct today start to return rows. For example, id NOT IN (SELECT t2.id FROM t2 WHERE t2.z > t1.z) must return no rows, and returns 1, 2, 4, NULL with that change.

What changes are included in this PR?

The hash join already had the right mechanism for correlated NOT IN mark joins with equality correlation keys: a per-build-row bitmap that records "this row's NOT IN is UNKNOWN". This PR uses that mechanism for every correlated null-aware join and makes it apply the join filter. The commits are split for review:

HashJoinExec: a null-aware LeftAnti or LeftMark join is correlated when it has correlation scope keys or a join filter. For a NULL value on either side, the join finds the candidate (build, probe) row pairs through the scope key hash map, or takes all pairs when there are no scope keys. The join filter then decides which pairs make a build row UNKNOWN. The LeftAnti final stage drops those rows. The extra work is only for rows that have a NULL value key, so it is zero when the data has no NULLs. JoinSelection swaps a null-aware LeftAnti only when it has a single key and no filter.
DecorrelatePredicateSubquery: a NOT IN mark join with a non-equality correlation is now planned as null-aware. Four EXPLAIN results in subquery.slt change at this commit. Commit 6 takes three of them back, so only one EXPLAIN changes in the end.
Tests.
Benchmarks: correctness asserts for Q05 to Q08 of the null_aware_join suite (bench: SQL benchmark suite for null-aware (NOT IN) joins #25386), and expect_plan null_aware: true for Q08. These checks fail on main and pass with this PR.
Performance, from the review: a build row stays UNKNOWN after it is marked. Thus the join now skips candidate pairs whose build row is already marked, and it does not evaluate the join filter for those pairs. Without scope keys, the pairing also drops the marked build rows after each chunk of pairs, and it stops when no unmarked build row is left. At the default benchmark sizes this takes Q06 from 20.6M candidate pairs to 240K, and Q07 from 20.6M to 169K. A consequence is that the filter is evaluated for fewer pairs, so a filter that gives an error on a skipped pair no longer causes an error. This is the same as the short-circuit behavior of AND.
Correctness of the plan choice, from the review: only a mark that can tell NULL from FALSE needs the null-aware join. A Filter keeps a row only when the predicate is TRUE, and AND and OR give TRUE only when an operand is TRUE. A non-negated IN or EXISTS that a WHERE conjunct reaches only through AND/OR thus keeps the same rows with a FALSE mark as with a NULL mark, and it keeps its plain mark join. Without this, commit 2 made such a query pay for a join that pins the outer table as the CollectLeft build side, cannot be swapped, and evaluates the join filter for every pair of a build row and a NULL probe row. NOT IN, NOT (x IN ...), (x IN ...) IS NULL, CASE and a mark that goes into a projection stay null-aware, so the fix of this pull request is not affected. On the query of the review (100k x 100k rows, same result 9900, release build, Apple M4 Pro) this takes the run time from 7.4 s at commit 5 to 2-5 ms, which is what main costs. Three of the four subquery.slt plans of commit 2 go back to their earlier form; the fourth is a NOT IN and keeps null_aware.

null_aware_join suite, release build, Apple M4 Pro. Each number is the median of 40 iterations, taken as 4 interleaved rounds. "Before" is commit 4 of this PR and "after" is commit 5. Q01 to Q03 do not use this code path.

Query	before	after
Q01	19.4 ms	17.7 ms
Q02	16.6 ms	15.0 ms
Q03	16.2 ms	14.7 ms
Q04	0.8 ms	0.8 ms	1.00x
Q05	2.7 ms	1.0 ms	2.70x
Q06	94.8 ms	1.0 ms	95x
Q07	93.2 ms	1.0 ms	93x
Q08	14.7 ms	12.0 ms	1.22x

Q06 and Q07 now cost about the same as Q04, which is the same shape with no NULL. Thus the correct result for the non-equality correlation is now nearly free.

Q08 is the remaining case. It has an equality correlation, so the candidate pairs come from the scope-key hash lookup, and the marked build rows are removed only after that lookup. At the default sizes it still enumerates 6.25M pairs, of which 86K survive. #25438 tracks narrowing the lookup itself.

The benchmark bot compares this PR with main (results). On main, Q05 to Q08 give wrong results, so the main column for those rows is the time to calculate an incorrect result (see #25386).

Query	`main`	this PR
Q01	21.9 ms	22.6 ms
Q02	21.6 ms	22.3 ms
Q03	21.1 ms	21.5 ms
Q04	1.43 ms	1.45 ms	1.01x
Q05	1.43 ms (wrong result)	1.93 ms	1.35x
Q06	1.42 ms (wrong result)	1.86 ms	1.32x
Q07	1.39 ms (wrong result)	1.77 ms	1.27x
Q08	1.75 ms (wrong result)	21.2 ms	12.1x

For Q05 to Q07, a correct result costs about 30% more than the incorrect fast result on main.

For an absolute reference, the same eight queries in datafusion-cli (this PR) and in DuckDB 1.5.2, on the same tables and the default sizes, Apple M4 Pro, median of 3 EXPLAIN ANALYZE runs. Both engines give the same result for all eight queries.

Query	DataFusion	DuckDB
Q01	5 ms	14 ms
Q02	3 ms	13 ms
Q03	2 ms	13 ms
Q04	< 1 ms	117 ms
Q05	1 ms	116 ms
Q06	1 ms	114 ms
Q07	1 ms	3540 ms
Q08	12 ms	22 ms

DataFusion is faster than DuckDB for every query of this suite. Thus the cost of the correct result is small in absolute terms, and Q08 is 12 ms against DuckDB's 22 ms.

What is the testing strategy for this PR?

New sqllogictest cases in null_aware_anti_join.slt and null_aware_mark_join.slt: the queries from the issue, NULL outer values with empty and non-empty subquery results, equality plus non-equality correlation, a filter on the subquery value itself, the positive IN form, the mark column through IS NULL / IS TRUE / IS FALSE / NOT ... OR and directly in a SELECT list, and runs with batch_size = 1. I checked all expected results with DuckDB 1.5.2 and PostgreSQL 17.11. 14 of these cases fail on main.
New HashJoinExec unit tests for a null-aware LeftAnti and LeftMark join that has a join filter and no scope keys, at all batch sizes.
Two EXPLAIN tests in null_aware_mark_join.slt that pin the plan of a correlated IN under OR as not null-aware and the plan of its NOT IN counterpart as null-aware, plus the matching DecorrelatePredicateSubquery unit tests.
The null_aware_join benchmark suite (bench: SQL benchmark suite for null-aware (NOT IN) joins #25386) now checks the result of all eight queries. Each assert compares the NOT IN count with a reference count that does not use NOT IN, so it holds for all values of NAJ_ROWS and NAJ_LARGE_ROWS. The suite passes at the default sizes and at -r 3000 -l 1001. On main, the asserts for Q05 to Q08 fail. See bench: SQL benchmark suite for null-aware (NOT IN) joins #25386 for the performance numbers.

Are there any user-facing changes?

Queries that returned wrong results now return correct results, and correlated NOT IN with an equality correlation in a WHERE clause no longer fails to plan. There are no public API changes.

🤖 Generated with Claude Code

codecov-commenter · 2026-09-15T18:18:41Z

Codecov Report

❌ Patch coverage is 90.83333% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.42%. Comparing base (edc936f) to head (0324155).
⚠️ Report is 42 commits behind head on main.

Files with missing lines	Patch %	Lines
...fusion/physical-plan/src/joins/hash_join/stream.rs	93.40%	6 Missing and 7 partials ⚠️
...on/optimizer/src/decorrelate_predicate_subquery.rs	86.07%	3 Missing and 8 partials ⚠️
...tafusion/physical-plan/src/joins/hash_join/exec.rs	89.15%	1 Missing and 8 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25339      +/-   ##
==========================================
+ Coverage   82.33%   82.42%   +0.08%     
==========================================
  Files        1137     1138       +1     
  Lines      432498   435479    +2981     
  Branches   432498   435479    +2981     
==========================================
+ Hits       356116   358937    +2821     
- Misses      54843    54847       +4     
- Partials    21539    21695     +156

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

kosiew

@adriangb,

Thanks for working on this. The approach looks good to me, especially keeping the residual join filter involved when determining the per-build-row UNKNOWN state for correlated NOT IN. I also like the added coverage for both anti and mark joins.

I left one non-blocking performance suggestion below. Nothing that needs to hold up the PR.

kosiew · 2026-09-16T03:57:16Z

+            None => {
+                let probe_rows =
+                    UInt32Array::from_iter_values(0..state.batch.num_rows() as u32);
+                for_each_cross_product(


One potential performance concern here: when there are no scope keys, we evaluate the residual filter for every NULL build row × probe row pair. The symmetric path below does the same for NULL probe rows. For a nullable non-equality-correlated NOT IN, that could add quadratic work, including for build rows that have already been marked UNKNOWN.

Would it be worth adding a small bounded benchmark or targeted performance regression test for this path? As a follow-up optimization, we might also be able to skip build rows that are already marked UNKNOWN, although we'd need to be careful about volatile or erroring filter expressions.

jayzhan211

Thanks @adriangb , 2 non-blocking suggestions

jayzhan211 · 2026-09-16T15:04:31Z

+            None => {
+                let build_rows =
+                    UInt64Array::from_iter_values(0..left_data.batch().num_rows() as u64);
+                for_each_cross_product(


The case with no scope keys re-checks build rows that are already marked UNKNOWN

Without scope keys, case 2 evaluates the filter for every (build row × NULL probe row) pair in every probe batch, including build rows already set in null_indices_bitmap. Those bits never clear, so that work is wasted. 20K outer × 10K NULL inner with i.z < o.z spends 8.85s in join_time (debug build). Skip marked rows and stop once none are left (same for case 1 at :1466):

None => { let num_build_rows = left_data.batch().num_rows(); for probe_rows in null_probe_rows.values().chunks(batch_size.max(1)) { let build_rows = { let bitmap = left_data.null_indices_bitmap().lock(); UInt64Array::from_iter_values( (0..num_build_rows) .filter(|i| !bitmap.get_bit(*i)) .map(|i| i as u64), ) }; if build_rows.is_empty() { break; } let probe_rows = UInt32Array::from(probe_rows.to_vec()); for_each_cross_product(&build_rows, &probe_rows, batch_size, &mut mark)?; } }

Fine to handle in a follow-up

jayzhan211 · 2026-09-16T15:06:40Z

        num_keys: usize,
        has_filter: bool,
    ) -> Result<Self> {
+        let correlated = num_keys > 1 || has_filter;


correlated = num_keys > 1 || has_filter assumes on[0] is the NOT IN value key. When the value has no outer columns, 1 = i.id is pushed into the subquery, so on[0] becomes the correlation key o.g = i.g, and a NULL o.g is marked UNKNOWN:

CREATE TABLE o(id INT, g INT, z INT) AS VALUES (1,1,10),(2,NULL,10),(3,2,10); CREATE TABLE i(id INT, g INT, z INT) AS VALUES (1,1,5),(5,2,5),(NULL,3,5); SELECT id FROM o WHERE 1 NOT IN (SELECT i.id FROM i WHERE i.g = o.g AND i.z < o.z); -- expected 2, 3; returns 3

The form without AND i.z < o.z is also wrong and doesn't go through the new code, so this predates the PR. Fine as a follow-up: in build_join, only set null_aware when the in-predicate's outer side references a left column.

Agreed, this is a bug in main:

main: R1 (with i.z < o.z) → 1 row R2 (equality only) → 1 row PR: R1 → 1 row R2 → 1 row

adriangb · 2026-09-16T20:02:17Z

@jayzhan211 @kosiew I opened #25386 w/ benchmarks for this change. Could we merge that first so we can look at before/afterS?

HashJoinExec's Debug output did not include null_aware, so `expect_plan HashJoinExec` also passed for a plain anti join. Add the field to Debug and require `null_aware: true` on Q02-Q07. Q01 has non-nullable keys, so it is not null-aware. Q08 plans as a plain mark join on main until apache#25339 lands, so it keeps only the HashJoinExec check here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Each assert compares the NOT IN count with a reference count that does not use NOT IN, so it holds at every NAJ_ROWS / NAJ_LARGE_ROWS value. Q05-Q08 give wrong results on main (apache#25336), so their asserts go in apache#25339 together with the fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

## Which issue does this PR close? - N/A. This PR adds benchmarks only. It is split out of apache#25339 so that the suite is on `main` first, and that PR can then be measured against it. ## Rationale for this change A `NOT IN` subquery becomes a null-aware join. An outer row that finds no match is TRUE only when neither side has a NULL in scope. If a NULL is in scope, the result is UNKNOWN. This decision is cheap for an uncorrelated `NOT IN`. For a correlated `NOT IN`, the correlation predicate stays behind as a join filter. The join must then evaluate that filter for each candidate (build row x probe row) pair, to find which rows the NULLs reach. A non-equality correlation gives no equality key, so there is no scope key to reduce the number of pairs. The cost then grows with the NULL count multiplied by the size of the opposite table. No benchmark measured this shape, so there was no way to see the cost, or to tell a change from noise. Review on apache#25339 asked for this benchmark. These are the measured results for apache#25339. Each number is the median of 60 iterations, taken as 6 interleaved rounds of 10 iterations on an Apple M4 Pro in release mode. The two sides are the base commit of apache#25339 and its head commit, each with this suite applied, so the comparison isolates the change in that PR. | Query | Shape | base | apache#25339 | | |---|---|---|---|---| | Q01 | uncorrelated, non-nullable keys | 17.9 ms | 17.7 ms | 0.99x | | Q02 | uncorrelated, 1% NULL subquery side | 14.9 ms | 14.9 ms | 1.00x | | Q03 | uncorrelated, 50% NULL outer side | 14.6 ms | 14.6 ms | 1.00x | | Q04 | correlated, nullable keys, no NULL present | 0.9 ms | 0.9 ms | 0.96x | | Q05 | correlated, 1% NULL outer side | 0.9 ms | 2.9 ms | 3.1x | | Q06 | correlated, 50% NULL outer side | 0.9 ms | 96.2 ms | 109x | | Q07 | correlated, 50% NULL subquery side | 0.8 ms | 94.4 ms | 112x | | Q08 | as Q06, with an equality correlation | 1.1 ms | 15.1 ms | 14x | Q01 to Q04 are the comparable rows, and they show no change. The base gives wrong results for Q05 to Q08, which is the bug that apache#25339 corrects. Thus the base numbers for those four rows are the time to calculate an incorrect result. They show the cost of correct results, not a regression. These are the results at the default sizes. DuckDB agrees with the "correct" column. | Query | correct (apache#25339) | base | |---|---|---| | Q05 | 7460 | 7450 | | Q06 | 5010 | 5000 | | Q07 | 10 | 0 | | Q08 | 5530 | 10000 | Q06 and Q07 are the rows that the review of apache#25339 asked about. They also give the baseline to measure any later optimization of that path against. Q08 has the same NULL fraction as Q06 and is 6 times cheaper, which is the value of the equality correlation. ## What changes are included in this PR? A `null_aware_join` SQL benchmark suite. There are no Rust changes. The runner finds suites in `benchmarks/sql_benchmarks/`, and the load SQL makes each table from `range()`, so there is no data generation step. - Q01 to Q03 are uncorrelated `NOT IN` at different NULL fractions. Their cost is linear with the table size. They are the regression guard for the plain null-aware path. - Q04 is the correlated shape with nullable keys that hold no NULL. It separates the baseline cost of the shape from the per-pair filter work. - Q05 to Q07 are the same correlation at 1% and 50% NULL on each side. This is where that work becomes visible. - Q08 has the same NULL fraction as Q06, but adds an equality correlation. The candidate pairs then come from a hash lookup. The difference between Q06 and Q08 shows the value of the scope key. Both table sizes are knobs. `NAJ_ROWS` (default 10000) sets the size for the correlated queries, whose cost grows with its square. `NAJ_LARGE_ROWS` (default 1000000) sets the size for the uncorrelated queries. ```bash ./bench.sh run null_aware_join # One query, with more rows for the correlated shape NAJ_ROWS=20000 ./bench.sh run null_aware_join 6 ``` This PR also adds the suite to `bench.sh` (including `all`) and documents it in `benchmarks/README.md` and `benchmarks/sql_benchmarks/README.md`. There is one Rust change: the `Debug` output of `HashJoinExec` now includes `null_aware`. The suite's `expect_plan` directive matches that output, so Q02 to Q07 can require `null_aware: true`. Before this change, `expect_plan HashJoinExec` also passed for a plain anti join. ## What is the testing strategy for this PR? This PR adds benchmarks, so it adds no new tests. The existing `checked_in_suites_cover_benchmark_directories` test in `benchmarks/src/sql_benchmark_suite.rs` covers suite discovery, and it passes with the new directory. Each query has these checks: - `expect_plan HashJoinExec`. Q02 to Q07 also require `expect_plan null_aware: true`. Q01 has non-nullable keys, so it is not null-aware. Q08 plans as a plain mark join on `main`, so apache#25339 adds its `null_aware: true` check. - Q01 to Q04 have an `assert` correctness canary. The assert compares the `NOT IN` count with a reference count that does not use `NOT IN`, so it is correct for all values of `NAJ_ROWS` and `NAJ_LARGE_ROWS`. I checked each reference against the `NOT IN` result in DuckDB at six pairs of sizes. The same asserts for Q05 to Q08 fail on `main`, so apache#25339 adds them together with the fix. I ran them on this branch merged with apache#25339, and all eight queries pass at the default sizes and at `-r 3000 -l 1001`. I ran all eight queries on this branch at the default sizes and at `-r 1500 -l 101`. As a counterfactual check on `main`, the Q05 to Q08 asserts fail, and `null_aware: true` fails on Q01 and Q08. Each query also runs on `main` as written. Q08 uses the mark join form on purpose. The plain `WHERE ... NOT IN` form with an equality correlation does not plan on `main`, and a query that runs on only one branch cannot compare two branches. ## Are there any user-facing changes? No. This PR changes benchmarks and documentation only. It does not change library code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>

adriangb · 2026-09-17T16:31:55Z

run benchmark null_aware_join

adriangbot · 2026-09-17T16:36:41Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5717854886-2404-6n7np 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing claude/datafusion-issue-25336-ba2e61 (7162cd7) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-17T17:04:33Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: null_aware_join

Runner log (last 40 lines)

2026-09-17T16:35:50.774765Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/25339, benchmarks=null_aware_join
2026-09-17T16:35:50.845987Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-09-17T16:35:50.846113Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-09-17T16:35:55.861661Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/25339/head:claude/datafusion-issue-25336-ba2e61", "main"], cwd="/workspace/datafusion-branch"
2026-09-17T16:36:00.863032Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "claude/datafusion-issue-25336-ba2e61"], cwd="/workspace/datafusion-branch"
2026-09-17T16:36:05.864827Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-09-17T16:36:10.867319Z  INFO benchmark_controller::runner::bench_datafusion === Cloning merge-base ===
2026-09-17T16:36:10.867334Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-base"], cwd="/"
2026-09-17T16:36:15.868761Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["-c", "advice.detachedHead=false", "checkout", "c4f72bad34249798182ac7de605eb63ab134f26e"], cwd="/workspace/datafusion-base"
2026-09-17T16:36:20.871452Z  INFO benchmark_controller::runner::shell running command cmd=rustc, args=["--version"], cwd="/"
2026-09-17T16:36:25.873992Z  INFO benchmark_controller::runner::shell running command cmd=cargo, args=["metadata", "--no-deps", "--format-version", "1"], cwd="/workspace/datafusion-branch/benchmarks"
2026-09-17T16:36:30.878134Z  INFO benchmark_controller::runner::bench_datafusion === Compiling dfbench for PR branch and merge-base in parallel ===
2026-09-17T16:36:30.907956Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["rev-parse", "HEAD"], cwd="/workspace/datafusion-branch"
2026-09-17T16:36:35.913246Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["rev-parse", "HEAD"], cwd="/workspace/datafusion-base"
2026-09-17T16:36:40.923635Z  INFO benchmark_controller::runner::controller_client post_comment _repo=apache/datafusion, _pr_number=25339, job_id=2404
2026-09-17T16:36:41.732143Z  INFO benchmark_controller::runner::bench_datafusion === Waiting for builds ===
2026-09-17T16:51:12.728542Z  INFO benchmark_controller::runner::bench_datafusion === Builds complete ===
2026-09-17T16:51:12.728553Z  INFO benchmark_controller::runner::bench_datafusion === Setting up bench runner ===
2026-09-17T16:51:12.728563Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-bench"], cwd="/"
2026-09-17T16:51:17.730532Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["-c", "advice.detachedHead=false", "checkout", "origin/main"], cwd="/workspace/datafusion-bench"
2026-09-17T16:51:22.733337Z  INFO benchmark_controller::runner::shell running command cmd=cp, args=["-r", "/data/tpch-answers/.", "/workspace/datafusion-bench/benchmarks/data/tpch_sf1/answers"], cwd="/"
2026-09-17T16:51:27.737470Z  INFO benchmark_controller::runner::shell running command cmd=cp, args=["-r", "/data/tpch-answers/.", "/workspace/datafusion-bench/benchmarks/data/tpch_sf10/answers"], cwd="/"
2026-09-17T16:51:32.739061Z  INFO benchmark_controller::runner::bench_datafusion ** Creating data if needed for null_aware_join **
2026-09-17T16:51:32.739652Z  INFO benchmark_controller::runner::shell running command cmd=/scripts/cache_data.sh, args=["null_aware_join", "/workspace/datafusion-bench/benchmarks"], cwd="/workspace/datafusion-bench/benchmarks"
2026-09-17T16:51:37.742609Z  INFO benchmark_controller::runner::bench_datafusion ** Running null_aware_join baseline **
2026-09-17T16:51:37.742726Z  INFO benchmark_controller::runner::shell running command (monitored) cmd=env, args=["DATAFUSION_DIR=/workspace/datafusion-base", "RESULTS_NAME=HEAD", "DATAFUSION_RUNTIME_TEMP_DIRECTORY=/workspace/spill-base-null_aware_join", "SQL_CARGO_COMMAND=cargo bench --bench sql -- --save-baseline HEAD", "./bench.sh", "run", "null_aware_join"], cwd="/workspace/datafusion-bench/benchmarks"

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

adriangb · 2026-09-17T17:06:46Z

run benchmark null_aware_join

env:
CARGO_PROFILE_BENCH_LTO: "thin"

adriangbot · 2026-09-17T17:09:43Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5718298180-2405-v2mjl 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing claude/datafusion-issue-25336-ba2e61 (7162cd7) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join
env:
  CARGO_PROFILE_BENCH_LTO: "thin"

Results will be posted here when complete

File an issue against this benchmark runner

adriangb · 2026-09-17T17:20:27Z

run benchmark null_aware_join

env:
  CARGO_PROFILE_BENCH_LTO: "thin"
  CARGO_BUILD_JOBS: "3"

adriangbot · 2026-09-17T17:23:32Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5718476115-2406-hj6rf 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing claude/datafusion-issue-25336-ba2e61 (7162cd7) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join
env:
  CARGO_BUILD_JOBS: "3"
  CARGO_PROFILE_BENCH_LTO: "thin"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-17T17:55:25Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing claude/datafusion-issue-25336-ba2e61 (7162cd7) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join
env:
  CARGO_BUILD_JOBS: "3"
  CARGO_PROFILE_BENCH_LTO: "thin"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                  HEAD                                    claude_datafusion-issue-25336-ba2e61
-----                  ----                                    ------------------------------------
null_aware_join/Q01    1.00     22.9±0.81ms        ? ?/sec     1.01     23.1±0.28ms        ? ?/sec
null_aware_join/Q02    1.00     22.4±0.61ms        ? ?/sec     1.03     23.1±0.56ms        ? ?/sec
null_aware_join/Q03    1.00     21.6±0.28ms        ? ?/sec     1.03     22.3±0.76ms        ? ?/sec
null_aware_join/Q04    1.08  1548.9±115.27µs        ? ?/sec    1.00  1437.0±17.62µs        ? ?/sec
null_aware_join/Q05    1.00  1605.8±130.29µs        ? ?/sec    1.60      2.6±0.17ms        ? ?/sec
null_aware_join/Q06    1.00  1450.2±14.87µs        ? ?/sec     12.14    17.6±0.08ms        ? ?/sec
null_aware_join/Q07    1.00  1484.9±100.60µs        ? ?/sec    11.69    17.4±0.09ms        ? ?/sec
null_aware_join/Q08    1.00      2.0±0.18ms        ? ?/sec     10.78    21.7±0.24ms        ? ?/sec

Resource Usage

null_aware_join — base (merge-base)

Metric	Value
Wall time	470.2s
Peak memory	312.2 MiB
Avg memory	50.1 MiB
CPU user	130.0s
CPU sys	6.8s
Peak spill	0 B

null_aware_join — branch

Metric	Value
Wall time	810.2s
Peak memory	365.8 MiB
Avg memory	30.2 MiB
CPU user	141.1s
CPU sys	3.9s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-09-17T17:55:39Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing claude/datafusion-issue-25336-ba2e61 (7162cd7) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join
env:
  CARGO_PROFILE_BENCH_LTO: "thin"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                  HEAD                                   claude_datafusion-issue-25336-ba2e61
-----                  ----                                   ------------------------------------
null_aware_join/Q01    1.00     23.0±0.47ms        ? ?/sec    1.01     23.1±0.36ms        ? ?/sec
null_aware_join/Q02    1.00     22.8±0.84ms        ? ?/sec    1.02     23.1±0.30ms        ? ?/sec
null_aware_join/Q03    1.00     22.1±0.95ms        ? ?/sec    1.03     22.7±0.43ms        ? ?/sec
null_aware_join/Q04    1.00  1446.1±25.01µs        ? ?/sec    1.01  1458.8±22.26µs        ? ?/sec
null_aware_join/Q05    1.00  1440.2±25.64µs        ? ?/sec    1.72      2.5±0.03ms        ? ?/sec
null_aware_join/Q06    1.00  1448.0±30.19µs        ? ?/sec    12.39    17.9±0.15ms        ? ?/sec
null_aware_join/Q07    1.00  1424.6±36.45µs        ? ?/sec    12.41    17.7±0.06ms        ? ?/sec
null_aware_join/Q08    1.00  1836.8±31.67µs        ? ?/sec    11.75    21.6±0.05ms        ? ?/sec

Resource Usage

null_aware_join — base (merge-base)

Metric	Value
Wall time	850.4s
Peak memory	328.5 MiB
Avg memory	29.2 MiB
CPU user	127.7s
CPU sys	6.4s
Peak spill	0 B

null_aware_join — branch

Metric	Value
Wall time	850.3s
Peak memory	341.5 MiB
Avg memory	29.4 MiB
CPU user	143.5s
CPU sys	4.2s
Peak spill	0 B

File an issue against this benchmark runner

adriangb · 2026-09-17T19:07:14Z

run benchmark null_aware_join

env:
CARGO_PROFILE_BENCH_LTO: "thin"

adriangbot · 2026-09-17T19:15:40Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5719775194-2408-7h469 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing claude/datafusion-issue-25336-ba2e61 (439274b) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join
env:
  CARGO_PROFILE_BENCH_LTO: "thin"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-17T19:45:35Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing claude/datafusion-issue-25336-ba2e61 (439274b) to c4f72ba (merge-base) diff

Run configuration

run benchmark null_aware_join
env:
  CARGO_PROFILE_BENCH_LTO: "thin"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                  HEAD                                   claude_datafusion-issue-25336-ba2e61
-----                  ----                                   ------------------------------------
null_aware_join/Q01    1.00     21.9±0.46ms        ? ?/sec    1.03     22.6±0.51ms        ? ?/sec
null_aware_join/Q02    1.00     21.6±0.41ms        ? ?/sec    1.03     22.3±0.40ms        ? ?/sec
null_aware_join/Q03    1.00     21.1±0.28ms        ? ?/sec    1.02     21.5±0.38ms        ? ?/sec
null_aware_join/Q04    1.00  1433.9±26.32µs        ? ?/sec    1.01  1448.6±29.92µs        ? ?/sec
null_aware_join/Q05    1.00  1431.3±26.44µs        ? ?/sec    1.35  1927.8±67.09µs        ? ?/sec
null_aware_join/Q06    1.00  1415.2±12.03µs        ? ?/sec    1.32  1863.6±12.18µs        ? ?/sec
null_aware_join/Q07    1.00  1390.7±19.90µs        ? ?/sec    1.27  1769.3±27.81µs        ? ?/sec
null_aware_join/Q08    1.00  1751.5±19.14µs        ? ?/sec    12.11    21.2±0.16ms        ? ?/sec

Resource Usage

null_aware_join — base (merge-base)

Metric	Value
Wall time	400.2s
Peak memory	346.6 MiB
Avg memory	57.8 MiB
CPU user	122.8s
CPU sys	5.9s
Peak spill	0 B

null_aware_join — branch

Metric	Value
Wall time	595.2s
Peak memory	311.3 MiB
Avg memory	41.1 MiB
CPU user	126.1s
CPU sys	5.7s
Peak spill	0 B

File an issue against this benchmark runner

adriangb · 2026-09-17T21:34:42Z

DuckDB comparison, and what the Q08 number means

To find out whether the remaining cost of this PR is acceptable, I measured the suite's eight queries in datafusion-cli (this branch, release) and in DuckDB 1.5.2 (EXPLAIN ANALYZE, three runs, median, Apple M4 Pro, same tables from the suite's load.sql). The two engines give the same result for all eight queries.

At the suite's default sizes

Query	Shape	DataFusion	DuckDB	DuckDB / DF
Q01	uncorrelated, non-nullable	5 ms	14 ms	2.8x
Q02	uncorrelated, 1% NULL subquery	3 ms	13 ms	4.2x
Q03	uncorrelated, 50% NULL outer	2 ms	13 ms	6.6x
Q04	correlated, no NULL	< 1 ms	117 ms	> 100x
Q05	correlated, 1% NULL outer	1 ms	116 ms	116x
Q06	correlated, 50% NULL outer	1 ms	114 ms	114x
Q07	correlated, 50% NULL subquery	1 ms	3540 ms	~3500x
Q08	as Q06, plus an equality correlation	12 ms	22 ms	1.8x

datafusion-cli reports milliseconds, so the sub-millisecond queries show as 1 ms.

Each engine against itself

Q06 and Q08 have the same NULL fraction. Q08 adds an equality correlation, which gives the join a scope key. Thus the ratio of the two shows what each engine gets from that key.

`NAJ_ROWS`	DF Q06	DF Q08	DF Q08/Q06	DuckDB Q06	DuckDB Q08	DuckDB Q08/Q06
10,000	1 ms	12 ms	12x	114 ms	22 ms	0.19x
30,000	1 ms	38.5 ms	38x	329 ms	49.7 ms	0.15x
100,000	13 ms	418.5 ms	32x	1110 ms	147 ms	0.13x

The equality correlation makes Q08 about 7x cheaper than Q06 in DuckDB. In DataFusion it makes Q08 more than 10x more expensive than Q06. So the Q08 number is not a property of the shape. It is a property of our scope-key path.

The cause is visible in the pair counts. Without a scope key, the join now drops the build rows that are already UNKNOWN after each chunk of candidate pairs, so Q06 evaluates 240K pairs instead of 20.6M. With a scope key, the candidate pairs come from the hash lookup, and the marked rows are removed only after that lookup, so Q08 still enumerates 6.25M pairs and keeps 86K of them. Q08's cost grows with the square of the table size, while DuckDB's grows about linearly. DuckDB is faster than DataFusion for Q08 from about 100,000 rows.

Conclusion

For the non-equality correlation, which is the bug this PR closes, the correct result is now nearly free: Q06 and Q07 cost about the same as Q04 (the same shape with no NULL), and about 30% more than the incorrect fast result on main.
For the uncorrelated shapes (Q01 to Q03) this PR changes nothing.
Q08 is a real gap, not a measurement artifact. It is pre-existing in the scope-key path, and this PR makes it visible because the other correlated queries are now fast. I suggest we track it as a follow-up: narrow the scope-key lookup with the UNKNOWN bitmap, so that the equality correlation reduces work instead of adding it.

Filed as #25438.

adriangb · 2026-09-17T21:40:31Z

Thanks for review @kosiew and @jayzhan211. Since I've made some implementation changes and posted new benches I'll give you an opportunity to re-review before we merge this, but it's looking ready from my end.

…s UNKNOWN A null-aware LeftAnti join with a join filter ignored the filter for NULL keys: one NULL probe key removed every build row, even when the filter excluded that NULL row for every build row. A null-aware LeftAnti join with correlation scope keys failed to plan. Treat a null-aware LeftAnti or LeftMark join as correlated when it has scope keys or a join filter. Correlated joins record the UNKNOWN decision per build row in the null-indices bitmap: the candidate (build, probe) pairs come from the scope map, or from all pairs when there are no scope keys, and the join filter decides which pairs count. The LeftAnti final stage drops the rows marked UNKNOWN. JoinSelection only swaps an uncorrelated null-aware LeftAnti. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ware The hash join now applies the join filter when it marks UNKNOWN rows, so a NOT IN mark join no longer needs to fall back to a non null-aware join when a non-equality correlation stays behind as a join filter. The fallback gave FALSE instead of NULL, so NOT (x IN (...)) returned extra rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Adds sqllogictest regression tests for apache#25336 (expected results checked with DuckDB and PostgreSQL) and HashJoinExec unit tests for null-aware LeftAnti and LeftMark joins that have a join filter and no scope keys. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ll_aware These asserts fail on main (apache#25336) and pass with the fix. Q08 is a null-aware mark join once the fix lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… joins A build row stays UNKNOWN once it is marked. The candidate pairing now removes pairs whose build row is already marked before it evaluates the join filter. Without correlation scope keys, the pairing also drops the marked build rows after each chunk of pairs, and it stops when no unmarked build row is left. For `NAJ_ROWS=10000` of the `null_aware_join` benchmark, this takes Q06 from 20.6M candidate pairs to 240K, and Q07 from 20.6M to 169K. A consequence is that the join filter is evaluated for fewer pairs, so a filter that gives an error only for a skipped pair no longer gives that error. This is the same as the short-circuit behavior of `AND`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

kosiew

@adriangb,

Thanks for the follow-up. I went through the changes again and don't see any new blocking issues.

The residual-filter-aware UNKNOWN handling for correlated null-aware NOT IN anti/mark hash joins looks good, and the follow-up in 3b38f6be6c addresses the concern about re-checking build rows that are already UNKNOWN by using retain_unmarked / for_each_unmarked_cross_product.

On the performance side, the null_aware_join correctness and benchmark coverage, together with the candidate-pruning follow-up, addresses the earlier benchmark request. The follow-up may evaluate fewer residual-filter pairs once an UNKNOWN result is found, but I wasn't able to establish a deterministic contract violation from that behavior.

Looks good to me. Thanks for working through the review feedback!

jayzhan211 · 2026-09-18T12:23:38Z

I will take another look

jayzhan211

I also noticed a few issues that already exist on main, so those can be follow-ups.

Correlated NOT IN with a constant value is still wrong, and now violates the on[0] convention · decorrelate_predicate_subquery.rs:476

5 NOT IN (SELECT t2.id FROM t2 WHERE t2.z = t1.z) plans as on=[(z, z)], filter: 5 = t2.id, null_aware. HashJoinExec then treats z, the scope key, as the NOT IN value key.
The PR returns 0 rows and DuckDB returns 5. Base is wrong too, so this is not a regression.
The constant projection from #25348 only runs when join_filter_opt is None. Extend it to the correlated case, or bail out with Ok(None) for that shape.

NOT IN correlated on the value column itself is still wrong · decorrelate_predicate_subquery.rs:476

id NOT IN (SELECT t2.id FROM t2 WHERE t2.id = t1.id) dedups to a single key, so it is planned as an uncorrelated null-aware RightAnti.
It returns 0 rows and DuckDB returns 3. Base is wrong too.
The mark form of the same query is correct.

jayzhan211 · 2026-09-18T12:57:33Z

+        // nullable mark column. A non-equality correlation stays behind as a
+        // join filter, which the hash join also applies when it decides
+        // whether a NULL makes the mark UNKNOWN.
        let null_aware = join_type == JoinType::LeftMark


This change also makes plain IN (not just NOT IN) null-aware when the subquery has a non-equality correlation; see the four changed plans in subquery.slt. In WHERE a OR x IN (...) the filter drops the row whether the mark is NULL or FALSE, so base was already correct for these queries. The null-aware plan only adds cost: it pins the outer table as the CollectLeft build side, cannot swap, and evaluates the join filter for every (build row × NULL probe row) pair.

Measured (release-nonlto, 100k × 100k, same result 9900 on both): base 1–2 ms → PR 7.4–8.1 s. At 30k rows it takes 0.57 s, so it is quadratic. The plan changes from RightMark to LeftMark ... null_aware.

Repro:

CREATE TABLE so AS SELECT value AS id_n0, value % 1000 AS z FROM range(0, 100000); CREATE TABLE si AS SELECT CASE WHEN value % 2 = 0 THEN NULL ELSE value * 2 END AS id_n50, value % 1000 AS z FROM range(0, 100000); SELECT count(*) FROM so o WHERE o.z > 900 OR o.id_n0 IN (SELECT i.id_n50 FROM si i WHERE i.z > o.z + 990);

Fix: only make the mark join null-aware when a NULL mark can behave differently from FALSE. That is never the case when the subquery is a non-negated IN/EXISTS sitting directly under AND/OR in a WHERE conjunct. A helper to detect that:

/// True when every subquery in `expr` is a non-negated `IN`/`EXISTS` reached /// only through AND/OR. A filter treats a NULL mark like FALSE there, so the /// mark join does not need to be null-aware. fn subqueries_only_positive(expr: &Expr) -> bool { match expr { Expr::BinaryExpr(BinaryExpr { left, op: Operator::And | Operator::Or, right, }) => subqueries_only_positive(left) && subqueries_only_positive(right), Expr::InSubquery(InSubquery { negated, .. }) => !negated, Expr::Exists(Exists { negated, .. }) => !negated, other => !has_subquery(other), } }

Call it on each conjunct in the SubqueryPredicate::Embedded arm and pass the result down rewrite_inner_subqueries → mark_join → build_join as a new bool (say needs_null_aware_mark). Callers outside a Filter pass true. Then:

let null_aware = join_type == JoinType::LeftMark && in_predicate_opt.is_some() + && needs_null_aware_mark && join_keys_may_be_null(

NOT IN, NOT (x IN ...), (x IN ...) IS NULL, CASE and projected marks stay null-aware, so the fix in this PR is unaffected. The four subquery.slt plan diffs should revert; please add an EXPLAIN test that pins the positive-IN plan as non-null-aware

The previous commits make a correlated `IN` mark join null-aware whenever a join key can be NULL. That is necessary only when a NULL mark can give a different answer than a FALSE mark. A `Filter` keeps a row only when its predicate is TRUE, and `AND` and `OR` give TRUE only when an operand is TRUE. A non-negated `IN` or `EXISTS` that a `WHERE` conjunct reaches only through `AND`/`OR` thus keeps the same rows with a FALSE mark as with a NULL mark. There the null-aware join only costs more: it pins the outer table as the `CollectLeft` build side, it cannot be swapped, and it evaluates the join filter for every pair of a build row and a NULL probe row. `subqueries_only_positive` finds that shape. The `Filter` path passes the result down to `build_join`, and every other caller asks for the null-aware mark. `NOT IN`, `NOT (x IN ...)`, `(x IN ...) IS NULL`, `CASE` and a mark that goes into a projection thus stay null-aware, and the fix of this pull request is not affected. Three of the four `subquery.slt` plans that the previous commits changed go back to their earlier form. The fourth is a `NOT IN` and keeps `null_aware`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…25441) ## Which issue does this PR close? - N/A. This is a benchmark harness fix found while measuring apache#25339. ## Rationale for this change A benchmark file can assert a string in the physical plan with the `expect_plan` directive. That check ran on every iteration, inside the measured region, and it rendered the plan with `{:#?}`. A derived `Debug` for an in-memory source prints every `RecordBatch` the source holds. Thus, for a benchmark whose tables come from `CREATE TABLE ... AS SELECT`, the check builds a multi-megabyte string per iteration and then searches it. For the `null_aware_join` suite, whose tables are built from `range()`, that string is 1.85 MB and the check is most of the measured time: the suite reports 17.6 ms for Q01, while `datafusion-cli` runs the same query in 4 ms. This inflates absolute numbers and, more importantly, dilutes the A/B difference the benchmark exists to show. ## What changes are included in this PR? The check now renders the plan as `EXPLAIN` displays it, through `DisplayableExecutionPlan`. It also runs on the first iteration only, because a plan does not change between iterations. `null_aware_join`, release build, both sides at the same commit (`c4f5a9e0f2`), median of 30 iterations: | Query | before | after | |---|---|---| | Q01 | 17.6 ms | 4.6 ms | | Q02 | 15.3 ms | 2.6 ms | | Q03 | 14.7 ms | 2.1 ms | | Q04 | 0.9 ms | 0.4 ms | | Q05 | 0.9 ms | 0.4 ms | | Q06 | 0.8 ms | 0.3 ms | | Q07 | 0.9 ms | 0.3 ms | | Q08 | 1.2 ms | 0.5 ms | A failed check now also prints an 8-line plan instead of a 1.8 MB dump. There are 96 `expect_plan` strings in the suites. 89 are operator names, which both forms print. The 6 `null_aware_join` strings that read `null_aware: true` become `null_aware`, which is how `HashJoinExec` displays the flag; this PR changes those 6 lines. The one remaining string, h2o's `output_ordering=[pk@0 ASC NULLS LAST, ob@1 DESC]`, holds in the display form as well. ## What is the testing strategy for this PR? - New unit test `run_checks_expect_plan_once_per_benchmark`: the first run checks the plan, and a later run does not repeat the check. The existing tests still cover the accept and reject paths of a first run. - I ran the suites that need no downloaded data, and all their `expect_plan` strings still hold: `null_aware_join` (8 queries), `smj` (26 queries, which include the 3 `LeftMark` checks), `nlj` (4 queries) and `array_agg_distinct`. - For the h2o `output_ordering` string, which needs data I do not have, I built an equivalent case: a Parquet table with `WITH ORDER`, a window query over it, and that `expect_plan` line. It passes. - Counterfactual check: with a string that is not in the plan, the first iteration still fails, and the message names the string. - `cargo test -p datafusion-benchmarks --lib` passes (143 tests). `cargo fmt` and clippy are clean. ## Are there any user-facing changes? No. This changes the benchmark harness and its documentation only. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

jayzhan211 · 2026-09-19T01:52:15Z

Constant-projection rewrite only fires when the join filter is the bare IN predicate. With a correlation, 1 = sq.id is not an equi-key → on[0] is the scope key, 1 = t4.id lands in the join filter, and the exec applies value-key NULL rules to z. Same output on main, so not a regression, but this PR extends the positional on[0] convention to LeftAnti and the comment here ("null-aware hash joins accept only a single key") is now false.

Repro (returns 7, 10; expected 7, NULL):

CREATE TABLE t3(z INT) AS VALUES (10), (NULL), (7);
CREATE TABLE t4(id INT, z INT) AS VALUES (NULL, 10), (1, 99), (2, 7);
SELECT z FROM t3 WHERE 1 NOT IN (SELECT t4.id FROM t4 WHERE t4.z = t3.z) ORDER BY z;

Fix: project the constant as a left column when join_filter_opt is Some too, so the IN equality is always on[0].

## Which issue does this PR close? - N/A. This PR adds benchmarks only. It is split out of apache#25339 so that the suite is on `main` first, and that PR can then be measured against it. ## Rationale for this change A `NOT IN` subquery becomes a null-aware join. An outer row that finds no match is TRUE only when neither side has a NULL in scope. If a NULL is in scope, the result is UNKNOWN. This decision is cheap for an uncorrelated `NOT IN`. For a correlated `NOT IN`, the correlation predicate stays behind as a join filter. The join must then evaluate that filter for each candidate (build row x probe row) pair, to find which rows the NULLs reach. A non-equality correlation gives no equality key, so there is no scope key to reduce the number of pairs. The cost then grows with the NULL count multiplied by the size of the opposite table. No benchmark measured this shape, so there was no way to see the cost, or to tell a change from noise. Review on apache#25339 asked for this benchmark. These are the measured results for apache#25339. Each number is the median of 60 iterations, taken as 6 interleaved rounds of 10 iterations on an Apple M4 Pro in release mode. The two sides are the base commit of apache#25339 and its head commit, each with this suite applied, so the comparison isolates the change in that PR. | Query | Shape | base | apache#25339 | | |---|---|---|---|---| | Q01 | uncorrelated, non-nullable keys | 17.9 ms | 17.7 ms | 0.99x | | Q02 | uncorrelated, 1% NULL subquery side | 14.9 ms | 14.9 ms | 1.00x | | Q03 | uncorrelated, 50% NULL outer side | 14.6 ms | 14.6 ms | 1.00x | | Q04 | correlated, nullable keys, no NULL present | 0.9 ms | 0.9 ms | 0.96x | | Q05 | correlated, 1% NULL outer side | 0.9 ms | 2.9 ms | 3.1x | | Q06 | correlated, 50% NULL outer side | 0.9 ms | 96.2 ms | 109x | | Q07 | correlated, 50% NULL subquery side | 0.8 ms | 94.4 ms | 112x | | Q08 | as Q06, with an equality correlation | 1.1 ms | 15.1 ms | 14x | Q01 to Q04 are the comparable rows, and they show no change. The base gives wrong results for Q05 to Q08, which is the bug that apache#25339 corrects. Thus the base numbers for those four rows are the time to calculate an incorrect result. They show the cost of correct results, not a regression. These are the results at the default sizes. DuckDB agrees with the "correct" column. | Query | correct (apache#25339) | base | |---|---|---| | Q05 | 7460 | 7450 | | Q06 | 5010 | 5000 | | Q07 | 10 | 0 | | Q08 | 5530 | 10000 | Q06 and Q07 are the rows that the review of apache#25339 asked about. They also give the baseline to measure any later optimization of that path against. Q08 has the same NULL fraction as Q06 and is 6 times cheaper, which is the value of the equality correlation. ## What changes are included in this PR? A `null_aware_join` SQL benchmark suite. There are no Rust changes. The runner finds suites in `benchmarks/sql_benchmarks/`, and the load SQL makes each table from `range()`, so there is no data generation step. - Q01 to Q03 are uncorrelated `NOT IN` at different NULL fractions. Their cost is linear with the table size. They are the regression guard for the plain null-aware path. - Q04 is the correlated shape with nullable keys that hold no NULL. It separates the baseline cost of the shape from the per-pair filter work. - Q05 to Q07 are the same correlation at 1% and 50% NULL on each side. This is where that work becomes visible. - Q08 has the same NULL fraction as Q06, but adds an equality correlation. The candidate pairs then come from a hash lookup. The difference between Q06 and Q08 shows the value of the scope key. Both table sizes are knobs. `NAJ_ROWS` (default 10000) sets the size for the correlated queries, whose cost grows with its square. `NAJ_LARGE_ROWS` (default 1000000) sets the size for the uncorrelated queries. ```bash ./bench.sh run null_aware_join # One query, with more rows for the correlated shape NAJ_ROWS=20000 ./bench.sh run null_aware_join 6 ``` This PR also adds the suite to `bench.sh` (including `all`) and documents it in `benchmarks/README.md` and `benchmarks/sql_benchmarks/README.md`. There is one Rust change: the `Debug` output of `HashJoinExec` now includes `null_aware`. The suite's `expect_plan` directive matches that output, so Q02 to Q07 can require `null_aware: true`. Before this change, `expect_plan HashJoinExec` also passed for a plain anti join. ## What is the testing strategy for this PR? This PR adds benchmarks, so it adds no new tests. The existing `checked_in_suites_cover_benchmark_directories` test in `benchmarks/src/sql_benchmark_suite.rs` covers suite discovery, and it passes with the new directory. Each query has these checks: - `expect_plan HashJoinExec`. Q02 to Q07 also require `expect_plan null_aware: true`. Q01 has non-nullable keys, so it is not null-aware. Q08 plans as a plain mark join on `main`, so apache#25339 adds its `null_aware: true` check. - Q01 to Q04 have an `assert` correctness canary. The assert compares the `NOT IN` count with a reference count that does not use `NOT IN`, so it is correct for all values of `NAJ_ROWS` and `NAJ_LARGE_ROWS`. I checked each reference against the `NOT IN` result in DuckDB at six pairs of sizes. The same asserts for Q05 to Q08 fail on `main`, so apache#25339 adds them together with the fix. I ran them on this branch merged with apache#25339, and all eight queries pass at the default sizes and at `-r 3000 -l 1001`. I ran all eight queries on this branch at the default sizes and at `-r 1500 -l 101`. As a counterfactual check on `main`, the Q05 to Q08 asserts fail, and `null_aware: true` fails on Q01 and Q08. Each query also runs on `main` as written. Q08 uses the mark join form on purpose. The plain `WHERE ... NOT IN` form with an equality correlation does not plan on `main`, and a query that runs on only one branch cannot compare two branches. ## Are there any user-facing changes? No. This PR changes benchmarks and documentation only. It does not change library code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>

…25441) ## Which issue does this PR close? - N/A. This is a benchmark harness fix found while measuring apache#25339. ## Rationale for this change A benchmark file can assert a string in the physical plan with the `expect_plan` directive. That check ran on every iteration, inside the measured region, and it rendered the plan with `{:#?}`. A derived `Debug` for an in-memory source prints every `RecordBatch` the source holds. Thus, for a benchmark whose tables come from `CREATE TABLE ... AS SELECT`, the check builds a multi-megabyte string per iteration and then searches it. For the `null_aware_join` suite, whose tables are built from `range()`, that string is 1.85 MB and the check is most of the measured time: the suite reports 17.6 ms for Q01, while `datafusion-cli` runs the same query in 4 ms. This inflates absolute numbers and, more importantly, dilutes the A/B difference the benchmark exists to show. ## What changes are included in this PR? The check now renders the plan as `EXPLAIN` displays it, through `DisplayableExecutionPlan`. It also runs on the first iteration only, because a plan does not change between iterations. `null_aware_join`, release build, both sides at the same commit (`c4f5a9e0f2`), median of 30 iterations: | Query | before | after | |---|---|---| | Q01 | 17.6 ms | 4.6 ms | | Q02 | 15.3 ms | 2.6 ms | | Q03 | 14.7 ms | 2.1 ms | | Q04 | 0.9 ms | 0.4 ms | | Q05 | 0.9 ms | 0.4 ms | | Q06 | 0.8 ms | 0.3 ms | | Q07 | 0.9 ms | 0.3 ms | | Q08 | 1.2 ms | 0.5 ms | A failed check now also prints an 8-line plan instead of a 1.8 MB dump. There are 96 `expect_plan` strings in the suites. 89 are operator names, which both forms print. The 6 `null_aware_join` strings that read `null_aware: true` become `null_aware`, which is how `HashJoinExec` displays the flag; this PR changes those 6 lines. The one remaining string, h2o's `output_ordering=[pk@0 ASC NULLS LAST, ob@1 DESC]`, holds in the display form as well. ## What is the testing strategy for this PR? - New unit test `run_checks_expect_plan_once_per_benchmark`: the first run checks the plan, and a later run does not repeat the check. The existing tests still cover the accept and reject paths of a first run. - I ran the suites that need no downloaded data, and all their `expect_plan` strings still hold: `null_aware_join` (8 queries), `smj` (26 queries, which include the 3 `LeftMark` checks), `nlj` (4 queries) and `array_agg_distinct`. - For the h2o `output_ordering` string, which needs data I do not have, I built an equivalent case: a Parquet table with `WITH ORDER`, a window query over it, and that `expect_plan` line. It passes. - Counterfactual check: with a string that is not in the plan, the first iteration still fails, and the message names the string. - `cargo test -p datafusion-benchmarks --lib` passes (143 tests). `cargo fmt` and clippy are clean. ## Are there any user-facing changes? No. This changes the benchmark harness and its documentation only. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

… IN` A null-aware hash join reads `on[0]` as the `NOT IN` value key and `on[1..]` as correlation scope keys. Decorrelation projected a constant value expression as an outer column only for an uncorrelated subquery, so a correlated one kept `<constant> = __sq.col` in the join filter and gave `on[0]` to the correlation. The join then applied the value-key NULL rules to the correlation key and returned wrong rows. CREATE TABLE t3(z INT) AS VALUES (10), (NULL), (7); CREATE TABLE t4(id INT, z INT) AS VALUES (NULL, 10), (1, 99), (2, 7); SELECT z FROM t3 WHERE 1 NOT IN (SELECT t4.id FROM t4 WHERE t4.z = t3.z) ORDER BY z; returned `7, 10`; DuckDB and the SQL standard give `7, NULL`. Project the constant for a correlated subquery too, and keep the `IN` equality as the leading conjunct so that it stays `on[0]`. This also removes the planning gap for a constant value with a non-equality correlation, which had no equi-join key at all. The projection is now taken only when the join really ends up null-aware, so a positive `IN` mark join keeps pushing the equality into the subquery. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

adriangb · 2026-09-19T23:08:35Z

Constant-projection rewrite only fires when the join filter is the bare IN predicate. With a correlation, 1 = sq.id is not an equi-key → on[0] is the scope key, 1 = t4.id lands in the join filter, and the exec applies value-key NULL rules to z. Same output on main, so not a regression, but this PR extends the positional on[0] convention to LeftAnti and the comment here ("null-aware hash joins accept only a single key") is now false.

Repro (returns 7, 10; expected 7, NULL):
CREATE TABLE t3(z INT) AS VALUES (10), (NULL), (7);
CREATE TABLE t4(id INT, z INT) AS VALUES (NULL, 10), (1, 99), (2, 7);
SELECT z FROM t3 WHERE 1 NOT IN (SELECT t4.id FROM t4 WHERE t4.z = t3.z) ORDER BY z;
Fix: project the constant as a left column when join_filter_opt is Some too, so the IN equality is always on[0].

Addressed in 78b49ba. This commit also:

Removes the planning error for a constant value with a non-equality correlation (null_aware LeftAnti join requires equi-join keys, but the join has none). That shape now has an equi-join key, so the query error in null_aware_anti_join.slt is replaced with result assertions.
The projection now fires only when the join really ends up null-aware. Without that guard a positive IN mark join under AND/OR loses the = __sq.col pushdown into the subquery.

Checked with a DuckDB differential fuzz over constant/column values × equality/inequality/no correlation: 61/200 wrong on 283b8ca, 0/200 after.

jayzhan211 · 2026-09-20T07:20:30Z

+            op: Operator::And | Operator::Or,
+            right,
+        }) => subqueries_only_positive(left) && subqueries_only_positive(right),
+        Expr::InSubquery(InSubquery { negated, .. }) => !negated,


I found another issue

InSubquery arm returns !negated without checking expr → a subquery inside the IN value gets a non-null-aware mark, but its NULL/FALSE is observable there. Regression from commit 6: base edc936f38 returns 0 rows, this PR returns 2, 5, NULL, NULL.

Repro:

CREATE TABLE t1(id INT, z INT, w INT) AS VALUES (1,10,1), (2,20,1), (NULL,30,2), (4,40,2), (5,NULL,1), (NULL,NULL,2); CREATE TABLE t2(id INT, z INT, w INT) AS VALUES (1,5,1), (NULL,50,1), (4,NULL,2), (NULL,NULL,2), (2,20,3); CREATE TABLE tb(b BOOLEAN) AS VALUES (false); -- expected: no rows SELECT id FROM t1 WHERE ((id IN (SELECT t2.id FROM t2 WHERE t2.w = t1.w)) IN (SELECT b FROM tb)) OR id = -1 ORDER BY id;

Fix (verified locally, touched slt files still pass); please add the repro to null_aware_mark_join.slt:

- Expr::InSubquery(InSubquery { negated, .. }) => !negated, + Expr::InSubquery(InSubquery { expr, negated, .. }) => { + !negated && !has_subquery(expr) + }

`SessionConfig::batch_size()` returns `ConfigNonZeroUsize::get()`, so the value is never 0 and `batch_size.max(1)` can never change it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A mark join needs null-aware semantics only where a NULL mark can behave differently from a FALSE mark. That test was computed by walking a whole `Filter` conjunct and producing one bool for every subquery in it, and the shape caused two defects: - the walk had to re-implement "which sub-expressions are truth contexts" as its own recursive enumeration, separate from the traversal that visits the occurrences, so any arm it treated as a leaf but which had children was a hole. `InSubquery.expr` was one, and a subquery there returned wrong rows because a mark in an `IN` value is compared rather than used as a filter truth value (reported in apache#25339 (comment)); - one bool for the conjunct let a sibling's shape change a different subquery's join, so a `NOT EXISTS` sibling forced null-awareness on an unrelated `IN`. `EXISTS` is two-valued, so that was never needed. Decide at the occurrence instead, in a recursion that only knows `AND`/`OR`. The permissive outcome lives in one arm whose pattern is its own proof: a non-negated `IN` whose value holds no subquery, reached through nothing but `AND`/`OR` frames of this recursion. Everything else, including every future `Expr` variant, takes the null-aware branch. `a IN (sq1) OR b NOT IN (sq2)` now keeps sq1 on the plain join and the `RightMark` swap, which the conjunct-level test denied it. In `build_join`, compute `null_aware` once instead of re-deriving the same conditions for the constant projection, the mark branch and the anti branch. The left projection only adds a non-nullable literal column and the right projection only drops unreferenced columns, so neither changes the nullability the decision looks at. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Results alone cannot fail on a needless null-aware join: it is still correct, only slower. Mutation testing showed the gap directly, so the decision is pinned at the plan level. Plan pins (`null_aware_mark_join.slt`): - a non-negated `IN` under `OR` is not null-aware, and is free to swap to `RightMark`; - a `NOT EXISTS` sibling does not change that; - a constant value expression is not projected as an outer column for a mark join that is not null-aware, which keeps the constant pushed into the subquery. New cases: - a subquery inside the `IN` value, both spellings of the outer `IN`; - `IS NOT NULL` over a subquery predicate, and a comparison between two marks, the two contexts that can tell a NULL mark from a FALSE mark and had no coverage; - a correlation that names only outer columns, which cannot become an equi-join key and stays a residual filter on a single anti join (`null_aware_anti_join.slt`). Every other correlated case here names a subquery column, so this shape was untested. Four Rust plan assertions in `decorrelate_predicate_subquery.rs` move here. Three are now covered by the pins above. The fourth is ported to an `EXPLAIN` beside the `naconst_corr_t3/t4` results, where it is the stronger assertion: the unit test pinned the order of the logical join filter and trusted `ExtractEquijoinPredicate` to preserve it, while the `EXPLAIN` pins the physical keys the executor actually reads, on=[(__correlated_sq_1_value@1, naconst_corr_t4.id@0), (z@0, z@1)] which is the positional `on[0]` invariant itself. Expected results verified with DuckDB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Q01-Q08 all cover shapes that need null-aware handling, so they only fail when null-awareness is lost. The opposite direction, taking a null-aware join where a plain one is correct, costs nothing in results and so is invisible to every assertion in the suite. That is the direction that regressed. Q09 is Q08's correlated shape with a non-negated `IN`. Removing the guard keeps the result identical and moves the timing onto the null-aware path: rows guard no guard 10000 7.6 ms 153.6 ms (20x) 20000 9.2 ms 290.4 ms (32x) For reference Q08, which does need null-aware handling, runs at 147 ms, so the unguarded Q09 lands on exactly that cost. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Sep 15, 2026

adriangb mentioned this pull request Sep 15, 2026

Wrong results: constant NOT IN (subquery) in a WHERE clause ignores NULLs in the subquery #25340

Closed

adriangb requested review from kosiew and neilconway and removed request for neilconway September 15, 2026 20:17

adriangb mentioned this pull request Sep 15, 2026

Wrong results: correlated NOT IN (subquery) ignores NULLs in the subquery #25347

Open

kosiew approved these changes Sep 16, 2026

View reviewed changes

adriangb mentioned this pull request Sep 16, 2026

perf: use one null-aware mark join for hashable IN subqueries in projections #25338

Open

jayzhan211 approved these changes Sep 16, 2026

View reviewed changes

adriangb mentioned this pull request Sep 16, 2026

bench: SQL benchmark suite for null-aware (NOT IN) joins #25386

Merged

adriangb force-pushed the claude/datafusion-issue-25336-ba2e61 branch from e9a4ea3 to 31f4dcd Compare September 17, 2026 15:14

adriangb mentioned this pull request Sep 17, 2026

Force thin LTO and a job cap on every build the runner starts adriangb/datafusion-benchmarking#35

Merged

adriangb force-pushed the claude/datafusion-issue-25336-ba2e61 branch from 7162cd7 to 439274b Compare September 17, 2026 19:06

adriangb mentioned this pull request Sep 17, 2026

Null-aware join: an equality correlation makes a correlated NOT IN slower, not faster #25438

Open

adriangb mentioned this pull request Sep 17, 2026

bench: keep the expect_plan check out of the measured region #25441

Merged

adriangb and others added 5 commits September 17, 2026 18:08

bench: add correctness asserts to null_aware_join Q05-Q08, pin Q08 nu…

f6d74b4

…ll_aware These asserts fail on main (apache#25336) and pass with the fix. Q08 is a null-aware mark join once the fix lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

adriangb force-pushed the claude/datafusion-issue-25336-ba2e61 branch from 439274b to 3b38f6b Compare September 17, 2026 23:11

kosiew approved these changes Sep 18, 2026

View reviewed changes

jayzhan211 reviewed Sep 18, 2026

View reviewed changes

adriangb mentioned this pull request Sep 18, 2026

Wrong results: correlated NOT IN loses the correlation when it is an equality on the IN value column #25480

Open

jayzhan211 reviewed Sep 20, 2026

View reviewed changes

adriangb and others added 4 commits September 20, 2026 14:55

refactor: drop a dead batch size guard in the null-aware probe loop

f5eba18

`SessionConfig::batch_size()` returns `ConfigNonZeroUsize::get()`, so the value is never 0 and `batch_size.max(1)` can never change it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Conversation

adriangb commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kosiew left a comment

Choose a reason for hiding this comment

Uh oh!

kosiew Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Uh oh!

jayzhan211 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

jayzhan211 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb commented Sep 16, 2026

Uh oh!

adriangb commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangb commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangb commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangb commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangbot commented Sep 17, 2026

Uh oh!

adriangb commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DuckDB comparison, and what the Q08 number means

At the suite's default sizes

Each engine against itself

Conclusion

Uh oh!

adriangb commented Sep 17, 2026

Uh oh!

kosiew left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jayzhan211 commented Sep 18, 2026

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Correlated NOT IN with a constant value is still wrong, and now violates the on[0] convention · decorrelate_predicate_subquery.rs:476

NOT IN correlated on the value column itself is still wrong · decorrelate_predicate_subquery.rs:476

Uh oh!

jayzhan211 Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb commented Sep 15, 2026 •

edited

Loading

codecov-commenter commented Sep 15, 2026 •

edited

Loading

adriangb commented Sep 17, 2026 •

edited

Loading

kosiew left a comment •

edited

Loading

adriangb commented Sep 19, 2026 •

edited

Loading