Uh oh!
There was an error while loading. Please reload this page.
exp: add --pin-threads option to dfbench for CPU affinity; thread-local IO - #20912
exp: add --pin-threads option to dfbench for CPU affinity; thread-local IO#20912Dandandan wants to merge 4 commits into
Conversation
Pin each tokio worker thread to a distinct CPU core for more stable and reproducible benchmark results. Enabled via PIN_THREADS=true in bench.sh or --pin-threads flag directly on dfbench. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dandandan
commented
Mar 12, 2026
run benchmarks |
alamb-ghbot
commented
Mar 12, 2026
🤖 |
alamb-ghbot
commented
Mar 12, 2026
🤖: Benchmark completed Details |
Dandandan
commented
Mar 12, 2026
So it seems to have some effect on But I guess it matters more when we also have control over the IO threads, which perhaps fits good in the morsel work. |
adriangb
commented
Mar 12, 2026
run benchmarks |
adriangbot
commented
Mar 12, 2026
Hi @adriangb, your benchmark configuration could not be parsed (#20912 (comment)). Error: Supported benchmarks:
Usage: Per-side configuration ( env:
SHARED_SETTING: enabledbaseline:
ref: v45.0.0env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1Gchanged:
ref: v46.0.0env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G |
alamb-ghbot
commented
Mar 12, 2026
🤖 |
adriangb
commented
Mar 12, 2026
run benchmarks env:
PIN_THREADS: true |
adriangbot
commented
Mar 12, 2026
Hi @adriangb, your benchmark configuration could not be parsed (#20912 (comment)). Error: Supported benchmarks:
Usage: Per-side configuration ( env:
SHARED_SETTING: enabledbaseline:
ref: v45.0.0env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1Gchanged:
ref: v46.0.0env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G |
Dandandan
commented
Mar 12, 2026
Lol 2 bots competing!! |
adriangb
commented
Mar 12, 2026
run benchmarks env:
PIN_THREADS: true |
adriangbot
commented
Mar 12, 2026
Benchmark job started for this request (job |
adriangbot
commented
Mar 12, 2026
Benchmark job started for this request (job |
adriangbot
commented
Mar 12, 2026
Benchmark job started for this request (job |
adriangb
commented
Mar 12, 2026
adriangbot
commented
Mar 12, 2026
adriangbot
commented
Mar 12, 2026
adriangbot
commented
Mar 12, 2026
adriangbot
commented
Mar 12, 2026
🤖 Benchmark completed (GKE) | trigger DetailsResource Usagetpch — base (merge-base)
tpch — branch
|
adriangbot
commented
Mar 12, 2026
🤖 Benchmark completed (GKE) | trigger DetailsResource Usagetpcds — base (merge-base)
tpcds — branch
|
adriangbot
commented
Mar 12, 2026
🤖 Benchmark completed (GKE) | trigger DetailsResource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
|
alamb-ghbot
commented
Mar 12, 2026
🤖: Benchmark completed Details |
alamb-ghbot
commented
Mar 12, 2026
🤖 |
alamb-ghbot
commented
Mar 12, 2026
🤖: Benchmark completed Details |
alamb-ghbot
commented
Mar 12, 2026
🤖 |
alamb-ghbot
commented
Mar 12, 2026
🤖: Benchmark completed Details |
Dandandan
commented
Mar 12, 2026
run benchmarks |
adriangbot
commented
Mar 12, 2026
Benchmark job started for this request (job |
adriangbot
commented
Mar 12, 2026
adriangbot
commented
Mar 12, 2026
Dandandan
commented
Mar 12, 2026
run benchmarks env:
PIN_THREADS: true |
adriangbot
commented
Mar 12, 2026
Benchmark job started for this request (job |
adriangbot
commented
Mar 12, 2026
Benchmark job started for this request (job |
adriangbot
commented
Mar 12, 2026
Benchmark job started for this request (job |
adriangbot
commented
Mar 12, 2026
adriangbot
commented
Mar 12, 2026
adriangbot
commented
Mar 12, 2026
…O thread for large Small reads (<1MB) use block_in_place for L1/L2 cache locality with zero coordination overhead. Large reads (>=1MB) dispatch to the per-core IO thread to keep the tokio worker free, since the data won't fit in cache anyway. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dandandan
commented
Mar 12, 2026
run benchmarks env:
PIN_THREADS: true |
adriangbot
commented
Mar 12, 2026
Benchmark job started for this request (job |
adriangbot
commented
Mar 12, 2026
Benchmark job started for this request (job |
adriangbot
commented
Mar 12, 2026
Benchmark job started for this request (job |
adriangbot
commented
Mar 12, 2026
adriangbot
commented
Mar 12, 2026
🤖 Benchmark completed (GKE) | trigger DetailsResource Usagetpch — base (merge-base)
tpch — branch
|
adriangbot
commented
Mar 12, 2026
adriangbot
commented
Mar 12, 2026
adriangbot
commented
Mar 12, 2026
🤖 Benchmark completed (GKE) | trigger DetailsResource Usagetpch — base (merge-base)
tpch — branch
|
adriangbot
commented
Mar 12, 2026
🤖 Benchmark completed (GKE) | trigger DetailsResource Usagetpcds — base (merge-base)
tpcds — branch
|
adriangbot
commented
Mar 12, 2026
🤖 Benchmark completed (GKE) | trigger DetailsResource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
|
adriangbot
commented
Mar 12, 2026
🤖 Benchmark completed (GKE) | trigger DetailsResource Usagetpch — base (merge-base)
tpch — branch
|
adriangbot
commented
Mar 12, 2026
🤖 Benchmark completed (GKE) | trigger DetailsResource Usagetpcds — base (merge-base)
tpcds — branch
|
adriangbot
commented
Mar 12, 2026
🤖 Benchmark completed (GKE) | trigger DetailsResource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
|
adriangbot
commented
Mar 12, 2026
🤖 Benchmark completed (GKE) | trigger DetailsResource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
|
adriangbot
commented
Mar 12, 2026
🤖 Benchmark completed (GKE) | trigger DetailsResource Usagetpcds — base (merge-base)
tpcds — branch
|
alamb-ghbot
commented
Mar 21, 2026
🤖 |

Pin each tokio worker thread to a distinct CPU core to allow for improved. Enabled via PIN_THREADS=true in bench.sh or --pin-threads flag directly on dfbench.
Ideally we should do it both for IO (e.g. those created by
spawn_blocking) threads to make sure more data from IO reads are in CPU cache once we start reading (but that would be a future step).Which issue does this PR close?
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?