bench: add predicate_eval SQL micro-benchmark suite for conjunctive filter evaluation

[adriangb]

Which issue does this PR close?

This PR does not close an issue. It adds a benchmark suite to support work and
discussion around predicate ordering in filter evaluation (e.g. the static
reordering in #22343 and the runtime/statistics-based reordering explored in
#22698). It deliberately benchmarks no specific implementation — see below.

Rationale for this change

Conjunctive (AND) filter evaluation in FilterExec is a left-deep
BinaryExpr(And) chain, and the order conjuncts are evaluated in can change
runtime by large factors: once a leading conjunct passes few enough rows the
batch is physically compacted before the rest, so a cheap-and-selective
predicate evaluated early saves later predicates work. Predicate ordering is
therefore an active area (static heuristics, runtime/adaptive schemes, cost
models).

There is currently no benchmark suite that isolates the dimensions that drive
this. Existing macro-benchmarks (TPC-H/DS, ClickBench) only incidentally
exercise filter ordering, so they can't show why a change to ordering helped
or hurt, or guard the order-insensitive case against regressions.

What changes are included in this PR?

A new SQL benchmark suite, benchmarks/sql_benchmarks/predicate_eval, built on
the existing .benchmark template framework (no engine code, no new Rust). It
sets no engine config of its own and measures DataFusion's built-in short-circuit
by default; a system under test is toggled purely via its native
DATAFUSION_EXECUTION_* env var (the bench harness builds its SessionContext
with SessionConfig::from_env), so the same scenarios can characterise the
baseline, a static heuristic, an adaptive scheme, or a cost model and be
compared apples-to-apples.

It is organised into 10 subgroups (select with BENCH_SUBGROUP), each varying
one property of conjunctive filter evaluation while holding the others fixed:

Subgroup	What it varies (others held fixed)
costsel	cost and selectivity point in different directions (expensive predicate is the selective one)
cost	per-predicate cost, at equal selectivity
selectivity	per-predicate selectivity, at equal cost
cardinality	conjunct count k = 2/4/8/16
width	string-column width (PRED_FILL = 2 / 30 / 170 chars)
scale	row count 5k / 100k / 5M / 50M
neutral	predicates are interchangeable (equal cost, none selective) — an order-insensitive control
correlation	conditional vs marginal selectivity (independent / positively / anti-correlated)
drift	selectivity that changes across the scan
nulls	null density (two- vs three-valued predicate results)

Each query's comment notes the per-predicate cost/selectivity that the data
generation hides from the SQL. Data is synthetic and generated inline by each
subgroup's load SQL (no external files); PRED_ROWS sizes it and PRED_FILL
sets string width. Wired into bench.sh (./bench.sh run predicate_eval) and
documented in benchmarks/sql_benchmarks/README.md.

The design was informed by surveying how Velox drives the analogous decision
(it ranks by cycles-per-row-eliminated, time / (rows_in - rows_out)).

Note: the scale subgroup's q52/q53 build 5M / 50M-row tables (the latter
~9 GB); run a single point with BENCH_QUERY if that is too heavy.

Are these changes tested?

These are benchmark definitions, not engine code. Each .benchmark includes an
assert that the generated table is non-empty, and every subgroup was run
locally at small PRED_ROWS to confirm the suite parses, loads, asserts, and
executes end-to-end. The queries are order-invariant (SELECT count(*) ...), so
any predicate-ordering system can also be checked for correctness by diffing
counts with the optimization on vs. off.

Are there any user-facing changes?

No. This only adds an opt-in benchmark suite and its documentation; no public
API, engine behavior, or default configuration changes.

[adriangb]

Results

Run config (defaults):BENCH_NAME=predicate_eval PRED_ROWS=1000000 (all 10 subgroups, scale subgroup at its own 5k/100k/5M/50M sizes), criterion 10 samples/query, no engine config set (measures DataFusion's built-in left-deep AND short-circuit).

This was run on a noisy laptop, noise is expected.

costsel — cost & selectivity point opposite ways (expensive predicate is the selective one)

Benchmark	Median	Mean	Mean 95% CI
costsel_q01_regexp_selective_last	5.113 ms	5.184 ms	[5.097 ms, 5.287 ms]
costsel_q02_regexp_selective_first	1.642 ms	1.654 ms	[1.636 ms, 1.675 ms]
costsel_q03_cheap_unselective_then_expensive_selective	1.264 ms	1.265 ms	[1.250 ms, 1.283 ms]

cost — per-predicate cost, at equal selectivity

Benchmark	Median	Mean	Mean 95% CI
cost_q10_expensive_first	2.261 ms	2.545 ms	[2.062 ms, 3.152 ms]
cost_q11_cheap_first	653.311 µs	660.827 µs	[648.549 µs, 678.636 µs]

selectivity — per-predicate selectivity, at equal cost

Benchmark	Median	Mean	Mean 95% CI
selectivity_q20_unselective_first	454.933 µs	455.685 µs	[449.862 µs, 462.258 µs]
selectivity_q21_selective_first	420.744 µs	422.432 µs	[418.718 µs, 426.746 µs]

cardinality — conjunct count k = 2/4/8/16

Benchmark	Median	Mean	Mean 95% CI
cardinality_q30_k2	424.973 µs	429.211 µs	[414.203 µs, 447.021 µs]
cardinality_q31_k4	608.709 µs	613.319 µs	[590.725 µs, 637.308 µs]
cardinality_q32_k8	3.018 ms	2.874 ms	[2.195 ms, 3.556 ms]
cardinality_q33_k16	3.996 ms	4.392 ms	[3.787 ms, 5.123 ms]

width — string-column width (PRED_FILL = 2/30/170 chars)

Benchmark	Median	Mean	Mean 95% CI
width_q40_narrow	8.387 ms	7.571 ms	[6.594 ms, 8.428 ms]
width_q41_wide	5.822 ms	6.005 ms	[5.740 ms, 6.326 ms]
width_q42_xwide	30.149 ms	30.270 ms	[29.570 ms, 31.062 ms]

scale — row count 5k/100k/5M/50M

Benchmark	Median	Mean	Mean 95% CI
scale_q50_5k	195.848 µs	194.429 µs	[190.381 µs, 197.835 µs]
scale_q51_100k	300.590 µs	301.308 µs	[299.721 µs, 303.113 µs]
scale_q52_5m	5.157 ms	5.231 ms	[5.114 ms, 5.361 ms]
scale_q53_50m	48.703 ms	48.883 ms	[48.221 ms, 49.680 ms]

neutral — order-insensitive control (equal cost, none selective)

Benchmark	Median	Mean	Mean 95% CI
neutral_q60_cheap_uniform	797.491 µs	816.330 µs	[768.721 µs, 869.888 µs]
neutral_q61_expensive_uniform	4.263 ms	4.463 ms	[4.072 ms, 4.905 ms]

correlation — conditional vs marginal selectivity (indep/pos/anti-correlated)

Benchmark	Median	Mean	Mean 95% CI
correlation_q70_independent	443.514 µs	455.220 µs	[442.026 µs, 472.727 µs]
correlation_q71_positive	353.478 µs	371.662 µs	[349.020 µs, 401.121 µs]
correlation_q72_anti	370.724 µs	372.424 µs	[356.075 µs, 389.509 µs]

drift — selectivity changes across the scan

Benchmark	Median	Mean	Mean 95% CI
drift_q80_a_then_b	355.596 µs	358.509 µs	[351.533 µs, 365.869 µs]
drift_q81_b_then_a	323.365 µs	323.517 µs	[319.436 µs, 327.678 µs]

nulls — null density (two- vs three-valued predicate results)

Benchmark	Median	Mean	Mean 95% CI
nulls_q90_no_nulls_control	415.150 µs	434.152 µs	[412.615 µs, 463.365 µs]
nulls_q91_half_null	382.150 µs	391.865 µs	[375.653 µs, 409.933 µs]

Total runtime is ~5 min, but it is dominated by criterion overhead (warmup, etc.).

[alamb]

I ran it locally like

andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ ./benchmarks/bench.sh run predicate_eval
***************************
DataFusion Benchmark Script
COMMAND: run
BENCHMARK: predicate_eval
QUERY: All
DATAFUSION_DIR: /Users/andrewlamb/Software/datafusion/benchmarks/..
BRANCH_NAME: predicate-eval-benchmarks
DATA_DIR: /Users/andrewlamb/Software/datafusion/benchmarks/data
RESULTS_DIR: /Users/andrewlamb/Software/datafusion/benchmarks/results/predicate-eval-benchmarks
CARGO_COMMAND: cargo run --release
PREFER_HASH_JOIN: true
SIMULATE_LATENCY: false***************************
Running predicate_eval benchmark (subgroup=all, rows=1000000)...
+ env BENCH_NAME=predicate_eval PRED_ROWS=1000000 bash -c 'cargo bench --bench sql'
Finished `bench` profile [optimized] target(s) in 0.46s
Running benches/sql.rs (/Users/andrewlamb/Software/datafusion/target/release/deps/sql-b0eb418b4acab8cd)
Gnuplot not found, using plotters backend
Loading benchmarks...
Loaded benchmarks in 7 ms ...
predicate_eval/cardinality_q30_k2_cardinality
time: [465.56 µs 468.56 µs 471.54 µs]
change: [−4.9556% −3.6923% −2.3399%] (p = 0.00 < 0.05)
Performance has improved.
predicate_eval/cardinality_q31_k4_cardinality
time: [699.03 µs 713.94 µs 732.46 µs]
change: [+0.2133% +2.5165% +5.6037%] (p = 0.10 > 0.05)
No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
Benchmarking predicate_eval/cardinality_q32_k8_cardinality: Collecting 10 samples in estimated 5.0021 s (4330 iterations)
...

I tried to verify with instruments

BENCH_NAME=predicate_eval PRED_ROWS=1000000 bash -c 'cargo bench --profile=profiling --bench sql'
Compiling datafusion-benchmarks v53.1.0 (/Users/andrewlamb/Software/datafusion/benchmarks)
^C Building [=======================> ] 450/457: mem_profile(bin), external_aggr(bin), sql(bench), benchmark_runner(bin), gen_wide_data(bin), imdb(bin), dfbench(bin)

But building the --profiling was building a bunch of binaries all at once. 😢

[alamb]

@Omega359 -- it is cool to see the new benchmark runners in action

[alamb]

i found this description somewhat hard to parse. I think it might be better to keep it short explaining it tests differnt predicate patterns that test for adaptivity and perhaps linking to the relevant tickets for more context.

[adriangb]

addressed, thanks!

[alamb]

Same comment from the description also applies here

[2010YOUY01]

Subgroup What it varies (others held fixed)
costsel cost and selectivity point in different directions (expensive predicate is the selective one)
cost per-predicate cost, at equal selectivity
selectivity per-predicate selectivity, at equal cost
cardinality conjunct count k = 2/4/8/16
width string-column width (PRED_FILL = 2 / 30 / 170 chars)
scale row count 5k / 100k / 5M / 50M
neutral predicates are interchangeable (equal cost, none selective) — an order-insensitive control
correlation conditional vs marginal selectivity (independent / positively / anti-correlated)
drift selectivity that changes across the scan
nulls null density (two- vs three-valued predicate results)

Regarding the design: overall, I think it's great. In particular, I think it's important to test the drift dimension.

One suggestion: nulls seem redundant here. Efficiently handling nulls is the responsibility of the expr evaluator. Within the scope of filter reordering, I think null density is at the same dimension as selectivity or cost, so we could remove it for simplicity.