feat: make sort pushdown BufferExec capacity configurable, default 1GB by zhuqi-lucas · Pull Request #21426 · apache/datafusion

zhuqi-lucas · 2026-04-07T02:40:27Z

Which issue does this PR close?

Rationale for this change

#21182 introduced BufferExec between SortPreservingMergeExec and DataSourceExec when sort elimination removes a SortExec. The buffer capacity was hardcoded to 64MB, which can cause I/O stalls for wide-row full scans.

What changes are included in this PR?

Add datafusion.execution.sort_pushdown_buffer_capacity config option (default 1GB)
Replace hardcoded BUFFER_CAPACITY_AFTER_SORT_ELIMINATION constant with the config value
Update SLT test expectations for new default capacity

How are these changes justified?

Why 1GB default:

This is a maximum, not pre-allocated — actual usage is bounded by partition data size
Strictly less memory than the SortExec it replaces (which buffers entire partition)
BufferExec integrates with MemoryPool, so global memory limits are respected
64MB was too small for wide-row scans (16-column TPC-H SELECT * queries showed I/O stalls)

Why configurable:

Different workloads have different optimal buffer sizes
Users with memory-constrained environments can reduce it
Users with wide tables or large row groups can increase it

Are these changes tested?

Existing SLT Test G verifies BufferExec appears in plan with correct capacity
Config integration tested via existing config framework

Are there any user-facing changes?

New config option: datafusion.execution.sort_pushdown_buffer_capacity (default: 1GB)

Replace hardcoded 64MB BUFFER_CAPACITY_AFTER_SORT_ELIMINATION with configurable `datafusion.execution.sort_pushdown_buffer_capacity`. Default 1GB — large enough to hold wide-row data without I/O stalls. This is strictly less memory than the SortExec it replaces, and actual usage is bounded by partition size and global memory pool limits. Closesapache#21417

Copilot

Pull request overview

This PR makes the per-partition BufferExec capacity used during sort-pushdown sort elimination configurable (defaulting from 64MB to 1GiB) to reduce I/O stalls for wide-row scans.

Changes:

Add new config option datafusion.execution.sort_pushdown_buffer_capacity (default 1GiB).
Replace the hardcoded buffer capacity in PushdownSort with the new config value.
Update SLT plan expectations to reflect the new default capacity.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
`datafusion/common/src/config.rs`	Introduces `sort_pushdown_buffer_capacity` execution config with documentation and default (1GiB).
`datafusion/physical-optimizer/src/pushdown_sort.rs`	Uses `ConfigOptions` value when inserting `BufferExec` after sort elimination.
`datafusion/sqllogictest/test_files/sort_pushdown.slt`	Updates plan assertions to expect 1GiB buffer capacity.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

zhuqi-lucas · 2026-04-07T08:04:42Z

run benchmarks

zhuqi-lucas · 2026-04-07T08:05:25Z

run benchmark sort_pushdown_sorted

adriangbot · 2026-04-07T08:06:42Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4197472241-920-qzgrm 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing feat/configurable-buffer-capacity (f944ac6) to cdfade5 (merge-base) diff using: tpch
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-04-07T08:07:21Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4197472241-918-gwttj 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing feat/configurable-buffer-capacity (f944ac6) to cdfade5 (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-04-07T08:07:24Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4197475681-921-758vh 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing feat/configurable-buffer-capacity (f944ac6) to cdfade5 (merge-base) diff using: sort_pushdown_sorted
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-04-07T08:08:13Z

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4197472241-919-ppn7g 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing feat/configurable-buffer-capacity (f944ac6) to cdfade5 (merge-base) diff using: tpcds
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-04-07T08:20:11Z

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

Comparing HEAD and feat_configurable-buffer-capacity
--------------------
Benchmark sort_pushdown_sorted.json
--------------------
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_configurable-buffer-capacity ┃ Change ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ Q1 │ 124.55 / 124.90 ±0.30 / 125.34 ms │ 123.01 / 124.19 ±0.84 / 125.21 ms │ no change │
│ Q2 │ 2.60 / 3.00 ±0.54 / 4.03 ms │ 2.61 / 2.82 ±0.25 / 3.30 ms │ +1.07x faster │
│ Q3 │ 332.38 / 338.61 ±4.15 / 344.25 ms │ 309.52 / 311.42 ±2.00 / 315.22 ms │ +1.09x faster │
│ Q4 │ 5.65 / 6.16 ±0.86 / 7.88 ms │ 5.56 / 5.91 ±0.60 / 7.11 ms │ no change │
└───────┴───────────────────────────────────┴───────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 472.67ms │
│ Total Time (feat_configurable-buffer-capacity) │ 444.34ms │
│ Average Time (HEAD) │ 118.17ms │
│ Average Time (feat_configurable-buffer-capacity) │ 111.09ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 2 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────────────┴──────────┘

Resource Usage

sort_pushdown_sorted — base (merge-base)

Metric	Value
Wall time	2.7s
Peak memory	4.7 GiB
Avg memory	3.8 GiB
CPU user	5.7s
CPU sys	0.8s
Peak spill	0 B

sort_pushdown_sorted — branch

Metric	Value
Wall time	2.5s
Peak memory	5.3 GiB
Avg memory	4.1 GiB
CPU user	5.8s
CPU sys	0.7s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-04-07T08:25:20Z

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

Comparing HEAD and feat_configurable-buffer-capacity
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_configurable-buffer-capacity ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 6.59 / 7.05 ±0.83 / 8.72 ms │ 6.70 / 7.15 ±0.83 / 8.80 ms │ no change │
│ QQuery 2 │ 146.66 / 147.67 ±0.68 / 148.47 ms │ 145.14 / 146.99 ±1.47 / 148.46 ms │ no change │
│ QQuery 3 │ 114.55 / 115.28 ±0.50 / 116.06 ms │ 113.76 / 114.35 ±0.72 / 115.74 ms │ no change │
│ QQuery 4 │ 1314.53 / 1350.41 ±25.62 / 1377.77 ms │ 1316.77 / 1327.69 ±11.38 / 1349.70 ms │ no change │
│ QQuery 5 │ 173.96 / 175.08 ±0.78 / 176.00 ms │ 173.28 / 174.00 ±0.63 / 175.15 ms │ no change │
│ QQuery 6 │ 844.19 / 864.36 ±15.61 / 888.20 ms │ 814.93 / 848.81 ±28.46 / 891.32 ms │ no change │
│ QQuery 7 │ 351.92 / 353.61 ±1.65 / 356.66 ms │ 348.30 / 350.80 ±2.09 / 353.48 ms │ no change │
│ QQuery 8 │ 115.89 / 117.03 ±0.68 / 117.87 ms │ 114.19 / 115.74 ±1.46 / 118.25 ms │ no change │
│ QQuery 9 │ 101.24 / 105.25 ±3.32 / 109.81 ms │ 101.49 / 108.87 ±9.83 / 128.14 ms │ no change │
│ QQuery 10 │ 109.58 / 110.37 ±1.02 / 112.29 ms │ 107.18 / 108.89 ±1.00 / 110.25 ms │ no change │
│ QQuery 11 │ 922.34 / 934.11 ±8.41 / 945.41 ms │ 904.53 / 920.24 ±12.33 / 940.80 ms │ no change │
│ QQuery 12 │ 44.61 / 46.93 ±1.19 / 47.75 ms │ 44.56 / 46.16 ±1.55 / 48.80 ms │ no change │
│ QQuery 13 │ 397.65 / 400.45 ±2.07 / 402.54 ms │ 396.35 / 399.34 ±2.09 / 402.44 ms │ no change │
│ QQuery 14 │ 1015.15 / 1024.87 ±7.41 / 1035.53 ms │ 1007.96 / 1017.16 ±5.02 / 1022.85 ms │ no change │
│ QQuery 15 │ 16.39 / 17.35 ±0.87 / 18.89 ms │ 15.63 / 17.55 ±1.28 / 19.29 ms │ no change │
│ QQuery 16 │ 6.96 / 7.27 ±0.23 / 7.50 ms │ 7.12 / 7.59 ±0.58 / 8.71 ms │ no change │
│ QQuery 17 │ 236.46 / 237.77 ±1.12 / 239.86 ms │ 238.84 / 239.96 ±1.26 / 241.53 ms │ no change │
│ QQuery 18 │ 129.47 / 130.61 ±1.03 / 132.07 ms │ 128.58 / 129.44 ±0.91 / 130.78 ms │ no change │
│ QQuery 19 │ 155.89 / 157.39 ±0.88 / 158.35 ms │ 156.10 / 156.85 ±0.64 / 157.91 ms │ no change │
│ QQuery 20 │ 14.48 / 14.68 ±0.16 / 14.90 ms │ 13.90 / 14.15 ±0.29 / 14.71 ms │ no change │
│ QQuery 21 │ 19.56 / 20.23 ±0.36 / 20.58 ms │ 19.11 / 19.33 ±0.20 / 19.62 ms │ no change │
│ QQuery 22 │ 488.17 / 493.07 ±3.94 / 500.25 ms │ 479.24 / 485.50 ±5.07 / 494.11 ms │ no change │
│ QQuery 23 │ 884.64 / 899.21 ±8.50 / 907.86 ms │ 865.24 / 888.82 ±12.31 / 899.62 ms │ no change │
│ QQuery 24 │ 395.40 / 396.86 ±0.94 / 398.05 ms │ 389.34 / 392.63 ±2.85 / 397.82 ms │ no change │
│ QQuery 25 │ 346.12 / 351.06 ±3.09 / 354.96 ms │ 348.68 / 350.34 ±1.50 / 352.87 ms │ no change │
│ QQuery 26 │ 82.49 / 84.66 ±1.39 / 86.41 ms │ 80.10 / 82.18 ±1.83 / 84.90 ms │ no change │
│ QQuery 27 │ 6.92 / 7.45 ±0.59 / 8.56 ms │ 6.86 / 7.47 ±0.56 / 8.38 ms │ no change │
│ QQuery 28 │ 147.96 / 150.16 ±1.76 / 153.16 ms │ 148.93 / 150.26 ±1.24 / 152.34 ms │ no change │
│ QQuery 29 │ 292.24 / 295.19 ±2.78 / 300.05 ms │ 291.83 / 308.16 ±25.57 / 359.17 ms │ no change │
│ QQuery 30 │ 43.55 / 46.05 ±1.34 / 47.13 ms │ 43.53 / 45.90 ±1.30 / 47.22 ms │ no change │
│ QQuery 31 │ 170.03 / 173.25 ±1.73 / 175.06 ms │ 169.85 / 171.16 ±1.76 / 174.63 ms │ no change │
│ QQuery 32 │ 57.21 / 57.54 ±0.35 / 58.20 ms │ 56.84 / 58.83 ±1.71 / 61.67 ms │ no change │
│ QQuery 33 │ 140.86 / 142.72 ±1.85 / 146.14 ms │ 143.06 / 143.95 ±0.96 / 145.81 ms │ no change │
│ QQuery 34 │ 6.91 / 7.18 ±0.30 / 7.60 ms │ 6.80 / 7.74 ±1.04 / 9.48 ms │ 1.08x slower │
│ QQuery 35 │ 111.14 / 111.75 ±0.97 / 113.68 ms │ 106.56 / 108.64 ±1.38 / 110.30 ms │ no change │
│ QQuery 36 │ 6.51 / 6.61 ±0.12 / 6.84 ms │ 6.57 / 7.19 ±0.81 / 8.72 ms │ 1.09x slower │
│ QQuery 37 │ 8.42 / 8.81 ±0.33 / 9.22 ms │ 8.43 / 8.83 ±0.24 / 9.11 ms │ no change │
│ QQuery 38 │ 85.82 / 90.10 ±4.31 / 98.19 ms │ 82.44 / 87.04 ±4.49 / 95.45 ms │ no change │
│ QQuery 39 │ 125.38 / 130.14 ±3.61 / 136.55 ms │ 122.85 / 127.77 ±3.01 / 132.21 ms │ no change │
│ QQuery 40 │ 112.43 / 118.91 ±7.96 / 134.34 ms │ 112.13 / 118.37 ±6.51 / 130.29 ms │ no change │
│ QQuery 41 │ 14.38 / 15.29 ±1.05 / 17.31 ms │ 14.07 / 14.44 ±0.27 / 14.74 ms │ +1.06x faster │
│ QQuery 42 │ 110.97 / 112.10 ±0.86 / 113.61 ms │ 107.88 / 109.86 ±1.51 / 112.56 ms │ no change │
│ QQuery 43 │ 5.92 / 6.08 ±0.19 / 6.45 ms │ 6.09 / 6.49 ±0.53 / 7.51 ms │ 1.07x slower │
│ QQuery 44 │ 11.69 / 11.95 ±0.21 / 12.32 ms │ 11.80 / 12.43 ±0.55 / 13.30 ms │ no change │
│ QQuery 45 │ 51.87 / 53.08 ±1.70 / 56.45 ms │ 52.56 / 53.26 ±0.58 / 53.94 ms │ no change │
│ QQuery 46 │ 8.49 / 8.79 ±0.18 / 9.05 ms │ 8.80 / 9.04 ±0.23 / 9.43 ms │ no change │
│ QQuery 47 │ 700.84 / 715.15 ±8.65 / 724.89 ms │ 695.04 / 707.70 ±8.13 / 716.70 ms │ no change │
│ QQuery 48 │ 283.09 / 287.46 ±3.18 / 292.38 ms │ 281.21 / 286.70 ±3.51 / 291.38 ms │ no change │
│ QQuery 49 │ 255.81 / 258.53 ±2.43 / 262.20 ms │ 254.41 / 256.47 ±1.99 / 259.29 ms │ no change │
│ QQuery 50 │ 229.82 / 235.73 ±4.92 / 242.66 ms │ 229.96 / 234.35 ±4.56 / 242.79 ms │ no change │
│ QQuery 51 │ 183.74 / 187.44 ±2.30 / 190.19 ms │ 182.06 / 186.12 ±3.03 / 190.58 ms │ no change │
│ QQuery 52 │ 108.42 / 109.71 ±0.74 / 110.53 ms │ 106.23 / 108.46 ±1.47 / 110.65 ms │ no change │
│ QQuery 53 │ 103.75 / 104.77 ±1.02 / 106.74 ms │ 101.76 / 102.44 ±0.59 / 103.23 ms │ no change │
│ QQuery 54 │ 146.61 / 147.29 ±0.60 / 148.18 ms │ 144.74 / 147.09 ±1.46 / 148.64 ms │ no change │
│ QQuery 55 │ 108.15 / 110.28 ±2.47 / 114.82 ms │ 106.21 / 107.77 ±1.10 / 109.04 ms │ no change │
│ QQuery 56 │ 141.98 / 143.10 ±1.18 / 145.21 ms │ 140.18 / 142.59 ±1.79 / 144.74 ms │ no change │
│ QQuery 57 │ 174.66 / 176.04 ±1.12 / 177.58 ms │ 171.82 / 174.01 ±1.11 / 174.89 ms │ no change │
│ QQuery 58 │ 296.34 / 304.12 ±8.15 / 318.88 ms │ 294.06 / 298.55 ±4.02 / 305.76 ms │ no change │
│ QQuery 59 │ 196.35 / 200.61 ±2.78 / 204.31 ms │ 196.30 / 197.73 ±1.43 / 199.56 ms │ no change │
│ QQuery 60 │ 142.52 / 146.35 ±5.57 / 157.37 ms │ 143.26 / 144.63 ±0.90 / 145.58 ms │ no change │
│ QQuery 61 │ 12.79 / 13.11 ±0.24 / 13.50 ms │ 12.88 / 13.31 ±0.31 / 13.78 ms │ no change │
│ QQuery 62 │ 901.94 / 950.85 ±34.25 / 1001.65 ms │ 901.80 / 924.51 ±20.63 / 955.94 ms │ no change │
│ QQuery 63 │ 105.58 / 108.28 ±3.00 / 114.13 ms │ 104.76 / 106.44 ±1.96 / 110.17 ms │ no change │
│ QQuery 64 │ 696.25 / 700.26 ±5.59 / 711.22 ms │ 691.36 / 701.70 ±5.93 / 708.51 ms │ no change │
│ QQuery 65 │ 252.80 / 254.98 ±1.94 / 258.35 ms │ 252.34 / 255.71 ±2.98 / 260.27 ms │ no change │
│ QQuery 66 │ 249.46 / 257.78 ±5.53 / 264.48 ms │ 252.07 / 286.41 ±46.98 / 379.56 ms │ 1.11x slower │
│ QQuery 67 │ 310.99 / 318.11 ±9.09 / 335.64 ms │ 305.53 / 314.93 ±9.00 / 331.24 ms │ no change │
│ QQuery 68 │ 8.64 / 9.75 ±0.67 / 10.52 ms │ 8.77 / 9.91 ±1.03 / 11.39 ms │ no change │
│ QQuery 69 │ 104.50 / 106.24 ±1.51 / 108.77 ms │ 102.49 / 104.99 ±2.70 / 108.77 ms │ no change │
│ QQuery 70 │ 341.95 / 364.38 ±16.34 / 390.97 ms │ 338.34 / 355.49 ±11.70 / 374.43 ms │ no change │
│ QQuery 71 │ 135.67 / 140.52 ±5.67 / 151.30 ms │ 135.09 / 137.55 ±1.88 / 140.22 ms │ no change │
│ QQuery 72 │ 713.22 / 720.92 ±7.65 / 734.91 ms │ 714.35 / 729.99 ±10.08 / 744.21 ms │ no change │
│ QQuery 73 │ 7.19 / 8.26 ±0.77 / 9.19 ms │ 7.03 / 8.22 ±1.13 / 10.15 ms │ no change │
│ QQuery 74 │ 578.03 / 588.31 ±5.92 / 594.55 ms │ 599.33 / 614.01 ±10.29 / 631.53 ms │ no change │
│ QQuery 75 │ 275.34 / 278.25 ±2.53 / 282.33 ms │ 276.41 / 278.30 ±1.85 / 280.97 ms │ no change │
│ QQuery 76 │ 131.13 / 132.80 ±1.27 / 134.81 ms │ 136.83 / 137.18 ±0.25 / 137.46 ms │ no change │
│ QQuery 77 │ 189.90 / 191.08 ±1.16 / 193.13 ms │ 189.83 / 191.65 ±1.68 / 194.26 ms │ no change │
│ QQuery 78 │ 350.13 / 353.64 ±3.22 / 358.87 ms │ 351.97 / 357.25 ±4.82 / 365.78 ms │ no change │
│ QQuery 79 │ 234.60 / 236.69 ±1.60 / 238.70 ms │ 233.64 / 236.99 ±2.06 / 240.09 ms │ no change │
│ QQuery 80 │ 333.00 / 334.40 ±1.76 / 337.86 ms │ 326.76 / 329.93 ±3.51 / 335.72 ms │ no change │
│ QQuery 81 │ 26.36 / 27.75 ±1.16 / 29.14 ms │ 27.04 / 28.43 ±1.63 / 31.63 ms │ no change │
│ QQuery 82 │ 200.15 / 202.58 ±2.21 / 205.29 ms │ 197.45 / 203.07 ±2.91 / 205.78 ms │ no change │
│ QQuery 83 │ 39.96 / 41.20 ±1.29 / 43.52 ms │ 39.66 / 40.67 ±0.90 / 42.30 ms │ no change │
│ QQuery 84 │ 48.52 / 48.84 ±0.18 / 49.04 ms │ 47.43 / 48.71 ±0.71 / 49.60 ms │ no change │
│ QQuery 85 │ 148.77 / 149.75 ±0.68 / 150.78 ms │ 147.75 / 150.62 ±1.86 / 152.80 ms │ no change │
│ QQuery 86 │ 39.02 / 39.58 ±0.55 / 40.50 ms │ 38.82 / 40.38 ±0.96 / 41.67 ms │ no change │
│ QQuery 87 │ 87.35 / 91.02 ±3.15 / 96.10 ms │ 86.16 / 89.56 ±3.31 / 95.82 ms │ no change │
│ QQuery 88 │ 100.06 / 101.65 ±1.22 / 103.59 ms │ 99.24 / 100.40 ±0.95 / 101.78 ms │ no change │
│ QQuery 89 │ 117.97 / 118.70 ±0.96 / 120.54 ms │ 118.81 / 120.42 ±1.07 / 121.72 ms │ no change │
│ QQuery 90 │ 23.36 / 24.80 ±1.77 / 28.27 ms │ 23.69 / 24.39 ±0.50 / 25.01 ms │ no change │
│ QQuery 91 │ 63.78 / 65.27 ±1.11 / 66.56 ms │ 64.45 / 65.85 ±1.20 / 67.91 ms │ no change │
│ QQuery 92 │ 58.20 / 58.92 ±0.67 / 59.82 ms │ 57.11 / 58.99 ±1.12 / 60.49 ms │ no change │
│ QQuery 93 │ 189.96 / 193.99 ±2.39 / 197.04 ms │ 194.48 / 196.26 ±1.89 / 199.74 ms │ no change │
│ QQuery 94 │ 60.59 / 61.86 ±0.77 / 62.58 ms │ 62.16 / 62.90 ±0.75 / 64.31 ms │ no change │
│ QQuery 95 │ 135.32 / 137.32 ±1.60 / 139.46 ms │ 134.14 / 136.85 ±1.54 / 138.24 ms │ no change │
│ QQuery 96 │ 74.68 / 75.54 ±0.92 / 77.18 ms │ 73.80 / 74.85 ±0.91 / 76.36 ms │ no change │
│ QQuery 97 │ 132.90 / 134.61 ±1.17 / 136.08 ms │ 131.13 / 133.65 ±2.80 / 137.20 ms │ no change │
│ QQuery 98 │ 153.22 / 154.98 ±1.43 / 157.59 ms │ 157.08 / 159.11 ±1.60 / 160.73 ms │ no change │
│ QQuery 99 │ 10746.85 / 10773.58 ±30.99 / 10832.84 ms │ 10807.86 / 10904.81 ±129.70 / 11158.90 ms │ no change │
└───────────┴──────────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 31812.94ms │
│ Total Time (feat_configurable-buffer-capacity) │ 31856.34ms │
│ Average Time (HEAD) │ 321.34ms │
│ Average Time (feat_configurable-buffer-capacity) │ 321.78ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 4 │
│ Queries with No Change │ 94 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────────────┴────────────┘

Resource Usage

tpcds — base (merge-base)

Metric	Value
Wall time	159.4s
Peak memory	5.5 GiB
Avg memory	4.6 GiB
CPU user	260.3s
CPU sys	17.3s
Peak spill	0 B

tpcds — branch

Metric	Value
Wall time	159.6s
Peak memory	5.5 GiB
Avg memory	4.5 GiB
CPU user	260.3s
CPU sys	17.1s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-04-07T08:26:17Z

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

Comparing HEAD and feat_configurable-buffer-capacity
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_configurable-buffer-capacity ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.20 / 4.45 ±6.37 / 17.20 ms │ 1.20 / 4.41 ±6.34 / 17.08 ms │ no change │
│ QQuery 1 │ 14.42 / 14.59 ±0.13 / 14.78 ms │ 14.12 / 14.35 ±0.18 / 14.55 ms │ no change │
│ QQuery 2 │ 43.68 / 43.94 ±0.15 / 44.08 ms │ 44.41 / 44.95 ±0.40 / 45.65 ms │ no change │
│ QQuery 3 │ 42.91 / 44.66 ±1.39 / 45.97 ms │ 41.13 / 43.94 ±2.57 / 47.61 ms │ no change │
│ QQuery 4 │ 291.08 / 301.54 ±8.03 / 313.49 ms │ 291.26 / 303.20 ±9.70 / 318.92 ms │ no change │
│ QQuery 5 │ 346.89 / 352.17 ±5.11 / 360.43 ms │ 348.30 / 354.25 ±4.71 / 359.81 ms │ no change │
│ QQuery 6 │ 5.39 / 6.17 ±0.79 / 7.37 ms │ 5.58 / 7.52 ±1.93 / 10.43 ms │ 1.22x slower │
│ QQuery 7 │ 16.67 / 17.12 ±0.66 / 18.42 ms │ 16.63 / 17.74 ±1.60 / 20.88 ms │ no change │
│ QQuery 8 │ 412.83 / 426.31 ±8.10 / 436.08 ms │ 421.23 / 443.07 ±25.91 / 493.40 ms │ no change │
│ QQuery 9 │ 638.29 / 647.05 ±5.32 / 654.66 ms │ 631.67 / 640.43 ±8.77 / 655.81 ms │ no change │
│ QQuery 10 │ 93.30 / 96.17 ±3.03 / 101.80 ms │ 93.81 / 98.63 ±4.92 / 107.13 ms │ no change │
│ QQuery 11 │ 105.93 / 106.60 ±0.85 / 108.27 ms │ 104.42 / 105.53 ±0.82 / 106.87 ms │ no change │
│ QQuery 12 │ 345.45 / 355.90 ±7.30 / 367.56 ms │ 343.36 / 351.15 ±4.68 / 356.59 ms │ no change │
│ QQuery 13 │ 457.23 / 478.03 ±20.14 / 506.20 ms │ 453.99 / 466.72 ±10.22 / 485.19 ms │ no change │
│ QQuery 14 │ 352.09 / 356.97 ±3.72 / 361.71 ms │ 353.74 / 358.14 ±2.76 / 362.23 ms │ no change │
│ QQuery 15 │ 371.93 / 388.03 ±16.92 / 420.25 ms │ 362.43 / 374.91 ±16.08 / 405.32 ms │ no change │
│ QQuery 16 │ 723.68 / 731.92 ±5.14 / 736.43 ms │ 722.58 / 728.21 ±4.81 / 734.62 ms │ no change │
│ QQuery 17 │ 717.49 / 726.88 ±5.85 / 732.88 ms │ 719.97 / 733.32 ±6.84 / 739.06 ms │ no change │
│ QQuery 18 │ 1431.01 / 1478.81 ±37.94 / 1546.92 ms │ 1409.96 / 1474.90 ±43.02 / 1523.93 ms │ no change │
│ QQuery 19 │ 34.47 / 43.79 ±16.62 / 76.97 ms │ 34.98 / 70.90 ±49.94 / 165.84 ms │ 1.62x slower │
│ QQuery 20 │ 719.02 / 735.95 ±18.65 / 761.24 ms │ 710.30 / 729.76 ±27.91 / 784.72 ms │ no change │
│ QQuery 21 │ 765.01 / 772.08 ±8.06 / 786.27 ms │ 753.67 / 758.02 ±3.42 / 763.34 ms │ no change │
│ QQuery 22 │ 1134.16 / 1147.41 ±13.25 / 1169.42 ms │ 1121.46 / 1126.71 ±4.01 / 1133.87 ms │ no change │
│ QQuery 23 │ 3076.64 / 3105.98 ±23.98 / 3147.07 ms │ 3065.74 / 3084.36 ±18.01 / 3115.10 ms │ no change │
│ QQuery 24 │ 101.27 / 102.63 ±0.92 / 103.67 ms │ 100.03 / 103.93 ±2.89 / 108.89 ms │ no change │
│ QQuery 25 │ 140.25 / 140.57 ±0.29 / 140.98 ms │ 137.80 / 139.95 ±1.84 / 143.23 ms │ no change │
│ QQuery 26 │ 100.70 / 102.89 ±1.51 / 104.67 ms │ 101.05 / 102.18 ±0.79 / 102.95 ms │ no change │
│ QQuery 27 │ 846.09 / 850.94 ±5.48 / 861.03 ms │ 843.24 / 846.20 ±3.43 / 851.93 ms │ no change │
│ QQuery 28 │ 7761.90 / 7784.48 ±17.43 / 7802.01 ms │ 7687.56 / 7737.65 ±25.06 / 7751.23 ms │ no change │
│ QQuery 29 │ 50.58 / 56.12 ±6.25 / 68.24 ms │ 50.29 / 54.96 ±4.86 / 63.03 ms │ no change │
│ QQuery 30 │ 370.38 / 372.59 ±2.63 / 377.76 ms │ 367.69 / 374.34 ±5.32 / 382.82 ms │ no change │
│ QQuery 31 │ 370.53 / 374.90 ±4.75 / 383.92 ms │ 369.19 / 386.36 ±10.34 / 399.30 ms │ no change │
│ QQuery 32 │ 1272.45 / 1317.77 ±28.37 / 1353.12 ms │ 1018.25 / 1042.96 ±21.97 / 1084.06 ms │ +1.26x faster │
│ QQuery 33 │ 1549.75 / 1585.99 ±45.67 / 1672.28 ms │ 1447.97 / 1473.20 ±19.49 / 1505.27 ms │ +1.08x faster │
│ QQuery 34 │ 1470.94 / 1543.46 ±44.24 / 1608.76 ms │ 1433.55 / 1468.01 ±34.14 / 1524.21 ms │ no change │
│ QQuery 35 │ 384.77 / 392.41 ±8.51 / 408.99 ms │ 380.88 / 385.32 ±2.35 / 387.24 ms │ no change │
│ QQuery 36 │ 112.49 / 121.64 ±5.73 / 129.79 ms │ 111.75 / 120.85 ±5.56 / 128.30 ms │ no change │
│ QQuery 37 │ 47.35 / 48.46 ±0.95 / 50.14 ms │ 47.51 / 48.51 ±0.94 / 50.08 ms │ no change │
│ QQuery 38 │ 75.56 / 76.71 ±0.83 / 78.10 ms │ 75.55 / 76.95 ±1.16 / 78.62 ms │ no change │
│ QQuery 39 │ 218.30 / 221.74 ±2.94 / 226.46 ms │ 206.85 / 216.13 ±5.26 / 221.47 ms │ no change │
│ QQuery 40 │ 22.71 / 26.01 ±1.89 / 28.52 ms │ 23.73 / 24.80 ±0.78 / 26.08 ms │ no change │
│ QQuery 41 │ 19.56 / 21.02 ±1.19 / 23.17 ms │ 18.91 / 20.50 ±1.13 / 22.10 ms │ no change │
│ QQuery 42 │ 18.66 / 19.39 ±0.71 / 20.58 ms │ 18.47 / 19.87 ±1.24 / 21.90 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 27542.26ms │
│ Total Time (feat_configurable-buffer-capacity) │ 26977.78ms │
│ Average Time (HEAD) │ 640.52ms │
│ Average Time (feat_configurable-buffer-capacity) │ 627.39ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 39 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	138.8s
Peak memory	40.3 GiB
Avg memory	28.5 GiB
CPU user	1289.1s
CPU sys	107.4s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	135.9s
Peak memory	38.2 GiB
Avg memory	27.8 GiB
CPU user	1278.7s
CPU sys	89.5s
Peak spill	0 B

File an issue against this benchmark runner

zhuqi-lucas · 2026-04-07T08:35:44Z

@adriangbot The regression above seems invalid for clickbench because this query has no sort?

│ QQuery 19 │ 34.47 / 43.79 ±16.62 / 76.97 ms │ 34.98 / 70.90 ±49.94 / 165.84 ms │ 1.62x slower │

SELECT"UserID"FROM hits WHERE"UserID" = 435090932899640449;

Dandandan · 2026-04-07T10:49:21Z

@adriangbot The regression above seems invalid for clickbench because this query has no sort?
│ QQuery 19 │ 34.47 / 43.79 ±16.62 / 76.97 ms │ 34.98 / 70.90 ±49.94 / 165.84 ms │ 1.62x slower │
SELECT"UserID"FROM hits WHERE"UserID" = 435090932899640449;

This query is just noisy sometimes.

zhuqi-lucas · 2026-04-07T11:00:12Z

@adriangbot The regression above seems invalid for clickbench because this query has no sort?
│ QQuery 19 │ 34.47 / 43.79 ±16.62 / 76.97 ms │ 34.98 / 70.90 ±49.94 / 165.84 ms │ 1.62x slower │
SELECT"UserID"FROM hits WHERE"UserID" = 435090932899640449;
This query is just noisy sometimes.

Good to know, thanks @Dandandan !

zhuqi-lucas · 2026-04-07T11:03:24Z

Faster for the real test:

Benchmark sort_pushdown_sorted.json
--------------------
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_configurable-buffer-capacity ┃ Change ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ Q1 │ 124.55 / 124.90 ±0.30 / 125.34 ms │ 123.01 / 124.19 ±0.84 / 125.21 ms │ no change │
│ Q2 │ 2.60 / 3.00 ±0.54 / 4.03 ms │ 2.61 / 2.82 ±0.25 / 3.30 ms │ +1.07x faster │
│ Q3 │ 332.38 / 338.61 ±4.15 / 344.25 ms │ 309.52 / 311.42 ±2.00 / 315.22 ms │ +1.09x faster │
│ Q4 │ 5.65 / 6.16 ±0.86 / 7.88 ms │ 5.56 / 5.91 ±0.60 / 7.11 ms │ no change │
└───────┴───────────────────────────────────┴───────────────────────────────────┴───────────────┘

alamb

Makes sense to me -- thank you @zhuqi-lucas and @xudong963

alamb · 2026-04-07T12:39:52Z

+ /// buffers the entire partition). The buffer respects the global memory
+ /// pool limit. Setting this to a large value is safe — actual memory
+ /// usage is bounded by partition size and global memory limits.
+ pub sort_pushdown_buffer_capacity: usize, default = 1024 * 1024 * 1024


This PR increases the size buffer because
64MB was too small for wide-row scans (16-column TPC-H SELECT * queries showed I/O stalls)
To be clear to anyone reading this, what will be on main is still better than 53.0.0 because prior to #21182 DataFusion would have sorted the entire thing (rather than just buffering it)
A fixed size like this is likely to buffer more than required for narrow cases
I suspect a better solution than a fixed size buffer would be some calculation based on the actual size of the data (e.g. the number of rows to buffer). However, that is tricky to compute / constrain memory when large strings are involved.
We probably would need to have both a row limit and a memory cap and pick the smaller of the two.
We can perhaps do this as a follow on issue/PR

Good point! A dynamic approach with both a row limit and a memory cap (whichever is reached first) would adapt better to different row widths. Created #21440 to track this.

adriangb · 2026-04-07T13:00:56Z

This query is just noisy sometimes.

Yeah anything <50ms is going to be noisy

adriangb · 2026-04-07T13:01:38Z

Thanks @zhuqi-lucas !

zhuqi-lucas · 2026-04-07T13:46:45Z

Thanks @alamb @adriangb @Dandandan @xudong963 for review!

apache#21426) ## Which issue does this PR close? Closesapache#21417 ## Rationale for this change apache#21182 introduced `BufferExec` between `SortPreservingMergeExec` and `DataSourceExec` when sort elimination removes a `SortExec`. The buffer capacity was hardcoded to 64MB, which can cause I/O stalls for wide-row full scans. ## What changes are included in this PR? - Add `datafusion.execution.sort_pushdown_buffer_capacity` config option (default 1GB) - Replace hardcoded `BUFFER_CAPACITY_AFTER_SORT_ELIMINATION` constant with the config value - Update SLT test expectations for new default capacity ## How are these changes justified? **Why 1GB default:** - This is a maximum, not pre-allocated — actual usage is bounded by partition data size - Strictly less memory than the `SortExec` it replaces (which buffers entire partition) - `BufferExec` integrates with `MemoryPool`, so global memory limits are respected - 64MB was too small for wide-row scans (16-column TPC-H `SELECT *` queries showed I/O stalls) **Why configurable:** - Different workloads have different optimal buffer sizes - Users with memory-constrained environments can reduce it - Users with wide tables or large row groups can increase it ## Are these changes tested? - Existing SLT Test G verifies `BufferExec` appears in plan with correct capacity - Config integration tested via existing config framework ## Are there any user-facing changes? New config option: `datafusion.execution.sort_pushdown_buffer_capacity` (default: 1GB)

CopilotAI review requested due to automatic review settings April 7, 2026 02:40

github-actionsBot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Apr 7, 2026

Copilotstarted reviewing on behalf of zhuqi-lucas April 7, 2026 02:41 View session

CopilotAI reviewed Apr 7, 2026
View reviewed changes

Comment threaddatafusion/physical-optimizer/src/pushdown_sort.rs
Comment threaddatafusion/common/src/config.rs
Comment threaddatafusion/common/src/config.rs

docs: update configs.md for sort_pushdown_buffer_capacity
130acd4

github-actionsBot added the documentation Improvements or additions to documentation label Apr 7, 2026

zhuqi-lucas added 2 commits April 7, 2026 14:32

ci: trigger CI
626c9ff

fix: add sort_pushdown_buffer_capacity to information_schema SLT
f944ac6

xudong963 approved these changes Apr 7, 2026
View reviewed changes

alamb approved these changes Apr 7, 2026
View reviewed changes

adriangb added this pull request to the merge queue Apr 7, 2026

Merged via the queue into apache:main with commit 727022dApr 7, 2026
36 checks passed

zhuqi-lucas mentioned this pull request Apr 7, 2026
Dynamic BufferExec sizing: row limit + memory cap for sort pushdown #21440
Open

zhuqi-lucas mentioned this pull request May 14, 2026
Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion apache/datafusion-site#186
Merged
4 tasks

zhuqi-lucas mentioned this pull request Jun 19, 2026
[EPIC] Sort Pushdown: skip sorts and skip IO / decode for ORDER BY / TopK queries #23036
Closed

Uh oh!

Conversation

zhuqi-lucas commented Apr 7, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes justified?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhuqi-lucas commented Apr 7, 2026

Uh oh!

zhuqi-lucas commented Apr 7, 2026

Uh oh!

adriangbot commented Apr 7, 2026

Uh oh!

adriangbot commented Apr 7, 2026

Uh oh!

adriangbot commented Apr 7, 2026

Uh oh!

adriangbot commented Apr 7, 2026

Uh oh!

adriangbot commented Apr 7, 2026

Uh oh!

adriangbot commented Apr 7, 2026

Uh oh!

adriangbot commented Apr 7, 2026

Uh oh!

zhuqi-lucas commented Apr 7, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dandandan commented Apr 7, 2026

Uh oh!

zhuqi-lucas commented Apr 7, 2026

Uh oh!

zhuqi-lucas commented Apr 7, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alambApr 7, 2026

Choose a reason for hiding this comment

Uh oh!

zhuqi-lucasApr 7, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb commented Apr 7, 2026

Uh oh!

adriangb commented Apr 7, 2026

Uh oh!

Uh oh!

zhuqi-lucas commented Apr 7, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zhuqi-lucas commented Apr 7, 2026•
edited
Loading

zhuqi-lucas commented Apr 7, 2026•
edited
Loading