Remove file prefetching from FileStream by Dandandan · Pull Request #20916 · apache/datafusion

Dandandan · 2026-03-12T21:53:42Z

Simplify the FileStream state machine by removing the mechanism that opens the next file in parallel while scanning the current one. Files are now opened sequentially (Scan -> Idle -> Open) instead of prefetching.

Which issue does this PR close?

ClosesRemove file prefetching from FileStream #20984

Rationale for this change

Current prefetching is ineffective or even counterproductive, see e.g. query 6 when testing against simulated S3 latency:

│ QQuery 6 │ 709.93 / 744.83 ±21.29 / 769.15 ms │ 304.17 / 436.76 ±90.12 / 542.71 ms │ +1.71x faster │

Let's remove it so we can make room for better prefetching taking care of pruning / late materilization better, cache behavior and IO/CPU split.

What changes are included in this PR?

Remove this API / prefetching to make room for more impactful prefetching,

Are these changes tested?

Existing tests.

Are there any user-facing changes?

Yes, NextOpennext: Option<NextOpen>, are removed of pub members.

Simplify the FileStream state machine by removing the mechanism that opens the next file in parallel while scanning the current one. Files are now opened sequentially (Scan -> Idle -> Open) instead of prefetching. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Dandandan · 2026-03-12T21:53:58Z

run benchmarks

adriangbot · 2026-03-12T21:54:01Z

Benchmark job started for this request (job bench-c4050432820-168). Results will be posted here when complete.

adriangbot · 2026-03-12T21:54:02Z

Benchmark job started for this request (job bench-c4050432820-169). Results will be posted here when complete.

adriangbot · 2026-03-12T21:54:03Z

Benchmark job started for this request (job bench-c4050432820-170). Results will be posted here when complete.

adriangbot · 2026-03-12T21:56:04Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4050432820-170-fgr64 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (70bc7d0) to 57b275a (merge-base) diff using: tpch
Results will be posted here when complete

adriangbot · 2026-03-12T21:56:09Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4050432820-168-xqc2n 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (70bc7d0) to 57b275a (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete

adriangbot · 2026-03-12T21:56:56Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4050432820-169-sn5f2 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (70bc7d0) to 57b275a (merge-base) diff using: tpcds
Results will be posted here when complete

This reverts commit 38fe60a.

adriangbot · 2026-03-12T22:06:50Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and remove_prefetch
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 47.63 / 48.41 ±0.79 / 49.64 ms │ 46.74 / 47.22 ±0.66 / 48.53 ms │ no change │
│ QQuery 2 │ 26.14 / 26.40 ±0.23 / 26.75 ms │ 24.65 / 25.03 ±0.46 / 25.93 ms │ +1.05x faster │
│ QQuery 3 │ 34.66 / 35.18 ±0.27 / 35.43 ms │ 34.46 / 34.77 ±0.19 / 34.98 ms │ no change │
│ QQuery 4 │ 22.21 / 23.13 ±0.54 / 23.67 ms │ 22.84 / 23.86 ±0.87 / 24.98 ms │ no change │
│ QQuery 5 │ 53.66 / 54.72 ±1.05 / 56.42 ms │ 51.23 / 54.11 ±1.59 / 56.12 ms │ no change │
│ QQuery 6 │ 18.16 / 18.43 ±0.20 / 18.77 ms │ 18.02 / 18.55 ±0.56 / 19.55 ms │ no change │
│ QQuery 7 │ 59.01 / 60.73 ±1.32 / 62.27 ms │ 56.95 / 58.01 ±0.91 / 59.44 ms │ no change │
│ QQuery 8 │ 53.77 / 54.89 ±0.82 / 56.09 ms │ 52.94 / 53.82 ±0.75 / 55.18 ms │ no change │
│ QQuery 9 │ 59.51 / 60.34 ±1.11 / 62.49 ms │ 57.03 / 58.86 ±1.68 / 60.97 ms │ no change │
│ QQuery 10 │ 74.50 / 75.83 ±0.80 / 76.83 ms │ 74.30 / 74.90 ±0.69 / 75.96 ms │ no change │
│ QQuery 11 │ 18.00 / 18.65 ±0.89 / 20.41 ms │ 17.00 / 17.66 ±0.62 / 18.50 ms │ +1.06x faster │
│ QQuery 12 │ 29.73 / 30.20 ±0.39 / 30.84 ms │ 29.63 / 29.97 ±0.34 / 30.53 ms │ no change │
│ QQuery 13 │ 41.05 / 41.35 ±0.26 / 41.74 ms │ 40.30 / 41.26 ±0.98 / 43.09 ms │ no change │
│ QQuery 14 │ 30.28 / 30.52 ±0.23 / 30.85 ms │ 29.40 / 29.62 ±0.18 / 29.92 ms │ no change │
│ QQuery 15 │ 36.92 / 37.34 ±0.33 / 37.87 ms │ 36.81 / 37.36 ±0.56 / 38.38 ms │ no change │
│ QQuery 16 │ 18.38 / 18.99 ±0.54 / 19.80 ms │ 17.72 / 18.11 ±0.28 / 18.60 ms │ no change │
│ QQuery 17 │ 78.77 / 80.72 ±1.51 / 83.12 ms │ 75.36 / 77.98 ±2.29 / 81.08 ms │ no change │
│ QQuery 18 │ 82.62 / 84.34 ±1.49 / 86.78 ms │ 82.10 / 83.46 ±1.07 / 84.75 ms │ no change │
│ QQuery 19 │ 39.95 / 41.55 ±1.13 / 42.85 ms │ 38.74 / 39.77 ±1.42 / 42.58 ms │ no change │
│ QQuery 20 │ 45.38 / 46.94 ±1.10 / 48.02 ms │ 43.16 / 44.38 ±0.98 / 45.49 ms │ +1.06x faster │
│ QQuery 21 │ 74.85 / 76.81 ±2.46 / 81.62 ms │ 70.51 / 74.36 ±2.61 / 78.36 ms │ no change │
│ QQuery 22 │ 21.38 / 22.30 ±0.82 / 23.79 ms │ 19.72 / 20.20 ±0.49 / 21.12 ms │ +1.10x faster │
└───────────┴────────────────────────────────┴────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 987.77ms │
│ Total Time (remove_prefetch) │ 963.25ms │
│ Average Time (HEAD) │ 44.90ms │
│ Average Time (remove_prefetch) │ 43.78ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 18 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴──────────┘

Resource Usage

tpch — base (merge-base)

Metric	Value
Wall time	5.3s
Peak memory	3.9 GiB
Avg memory	3.5 GiB
CPU user	35.1s
CPU sys	3.4s
Disk read	0 B
Disk write	140.0 KiB

tpch — branch

Metric	Value
Wall time	5.1s
Peak memory	4.0 GiB
Avg memory	3.5 GiB
CPU user	34.4s
CPU sys	3.3s
Disk read	0 B
Disk write	68.0 KiB

adriangbot · 2026-03-12T22:12:42Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and remove_prefetch
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.56 / 4.89 ±6.48 / 17.86 ms │ 1.58 / 4.87 ±6.46 / 17.78 ms │ no change │
│ QQuery 1 │ 15.04 / 15.41 ±0.34 / 16.00 ms │ 15.39 / 15.68 ±0.16 / 15.85 ms │ no change │
│ QQuery 2 │ 56.99 / 57.35 ±0.30 / 57.72 ms │ 57.80 / 58.23 ±0.24 / 58.50 ms │ no change │
│ QQuery 3 │ 49.37 / 51.69 ±1.84 / 54.33 ms │ 49.57 / 51.45 ±1.00 / 52.43 ms │ no change │
│ QQuery 4 │ 311.41 / 313.95 ±2.10 / 316.96 ms │ 311.67 / 319.43 ±7.87 / 334.43 ms │ no change │
│ QQuery 5 │ 358.08 / 359.77 ±1.16 / 361.25 ms │ 356.46 / 360.69 ±2.28 / 363.11 ms │ no change │
│ QQuery 6 │ 5.28 / 6.02 ±0.52 / 6.52 ms │ 5.38 / 5.90 ±0.54 / 6.86 ms │ no change │
│ QQuery 7 │ 17.38 / 17.59 ±0.20 / 17.95 ms │ 17.88 / 18.61 ±0.56 / 19.27 ms │ 1.06x slower │
│ QQuery 8 │ 453.34 / 463.44 ±11.07 / 482.86 ms │ 447.20 / 452.07 ±3.84 / 456.23 ms │ no change │
│ QQuery 9 │ 688.59 / 703.51 ±8.81 / 712.89 ms │ 691.69 / 697.84 ±6.04 / 708.12 ms │ no change │
│ QQuery 10 │ 92.48 / 96.51 ±2.75 / 100.60 ms │ 96.51 / 98.96 ±2.75 / 104.12 ms │ no change │
│ QQuery 11 │ 106.37 / 108.25 ±1.13 / 109.37 ms │ 110.28 / 112.46 ±1.79 / 115.24 ms │ no change │
│ QQuery 12 │ 354.99 / 357.32 ±2.05 / 360.86 ms │ 351.05 / 358.58 ±4.41 / 364.27 ms │ no change │
│ QQuery 13 │ 471.58 / 475.12 ±3.46 / 481.33 ms │ 480.65 / 487.12 ±4.17 / 492.20 ms │ no change │
│ QQuery 14 │ 361.09 / 367.63 ±5.56 / 373.81 ms │ 364.27 / 371.21 ±3.80 / 375.22 ms │ no change │
│ QQuery 15 │ 376.35 / 390.71 ±14.41 / 415.07 ms │ 390.30 / 414.74 ±20.78 / 447.61 ms │ 1.06x slower │
│ QQuery 16 │ 754.10 / 788.09 ±31.65 / 831.92 ms │ 755.05 / 783.62 ±36.08 / 853.84 ms │ no change │
│ QQuery 17 │ 743.81 / 754.92 ±7.89 / 765.01 ms │ 752.21 / 763.04 ±6.60 / 771.84 ms │ no change │
│ QQuery 18 │ 1515.94 / 1535.69 ±19.15 / 1566.24 ms │ 1450.83 / 1555.30 ±62.17 / 1629.20 ms │ no change │
│ QQuery 19 │ 36.65 / 62.87 ±30.55 / 103.89 ms │ 37.10 / 37.97 ±0.72 / 39.20 ms │ +1.66x faster │
│ QQuery 20 │ 716.08 / 743.34 ±25.50 / 782.06 ms │ 718.24 / 728.56 ±14.04 / 756.21 ms │ no change │
│ QQuery 21 │ 760.86 / 768.71 ±5.35 / 775.82 ms │ 762.44 / 766.00 ±3.08 / 770.31 ms │ no change │
│ QQuery 22 │ 1129.94 / 1136.80 ±7.32 / 1150.10 ms │ 1130.23 / 1132.90 ±1.85 / 1134.91 ms │ no change │
│ QQuery 23 │ 3181.58 / 3207.70 ±18.69 / 3237.53 ms │ 3111.37 / 3119.08 ±8.32 / 3134.98 ms │ no change │
│ QQuery 24 │ 99.07 / 104.48 ±4.14 / 110.86 ms │ 101.81 / 104.82 ±3.81 / 112.05 ms │ no change │
│ QQuery 25 │ 139.99 / 144.04 ±2.07 / 145.65 ms │ 140.60 / 142.64 ±1.55 / 145.04 ms │ no change │
│ QQuery 26 │ 104.29 / 105.34 ±1.02 / 106.81 ms │ 103.45 / 106.14 ±1.82 / 108.63 ms │ no change │
│ QQuery 27 │ 850.25 / 859.64 ±6.34 / 869.75 ms │ 855.67 / 858.43 ±2.57 / 861.87 ms │ no change │
│ QQuery 28 │ 7728.91 / 7780.90 ±42.67 / 7845.96 ms │ 7746.89 / 7782.02 ±21.10 / 7806.54 ms │ no change │
│ QQuery 29 │ 331.63 / 335.62 ±6.92 / 349.44 ms │ 332.05 / 337.84 ±9.76 / 357.33 ms │ no change │
│ QQuery 30 │ 374.95 / 385.00 ±10.90 / 405.61 ms │ 371.27 / 381.56 ±7.62 / 390.95 ms │ no change │
│ QQuery 31 │ 379.61 / 392.95 ±9.96 / 410.36 ms │ 374.86 / 388.67 ±12.85 / 411.48 ms │ no change │
│ QQuery 32 │ 1089.46 / 1107.08 ±21.09 / 1148.01 ms │ 1076.32 / 1108.73 ±24.99 / 1152.89 ms │ no change │
│ QQuery 33 │ 1522.09 / 1531.88 ±10.93 / 1548.16 ms │ 1515.12 / 1532.60 ±12.80 / 1547.00 ms │ no change │
│ QQuery 34 │ 1503.73 / 1518.36 ±15.57 / 1548.42 ms │ 1569.49 / 1635.79 ±43.89 / 1690.21 ms │ 1.08x slower │
│ QQuery 35 │ 410.92 / 417.68 ±5.67 / 427.93 ms │ 450.48 / 471.07 ±14.12 / 486.86 ms │ 1.13x slower │
│ QQuery 36 │ 115.52 / 125.70 ±5.77 / 133.31 ms │ 117.49 / 126.25 ±4.65 / 130.09 ms │ no change │
│ QQuery 37 │ 50.42 / 51.44 ±1.20 / 53.69 ms │ 51.32 / 52.56 ±0.79 / 53.81 ms │ no change │
│ QQuery 38 │ 77.50 / 78.84 ±0.91 / 80.27 ms │ 77.68 / 81.15 ±2.75 / 85.18 ms │ no change │
│ QQuery 39 │ 223.06 / 233.12 ±6.98 / 242.35 ms │ 222.27 / 235.35 ±7.54 / 242.25 ms │ no change │
│ QQuery 40 │ 26.59 / 27.77 ±0.68 / 28.37 ms │ 27.03 / 28.99 ±1.90 / 32.08 ms │ no change │
│ QQuery 41 │ 21.39 / 22.88 ±0.79 / 23.63 ms │ 21.65 / 22.29 ±0.50 / 23.09 ms │ no change │
│ QQuery 42 │ 21.24 / 22.09 ±0.60 / 22.69 ms │ 21.21 / 21.71 ±0.27 / 21.98 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 28032.06ms │
│ Total Time (remove_prefetch) │ 28132.92ms │
│ Average Time (HEAD) │ 651.91ms │
│ Average Time (remove_prefetch) │ 654.25ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 4 │
│ Queries with No Change │ 38 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	141.4s
Peak memory	42.9 GiB
Avg memory	31.8 GiB
CPU user	1327.7s
CPU sys	89.0s
Disk read	0 B
Disk write	4.0 GiB

clickbench_partitioned — branch

Metric	Value
Wall time	141.5s
Peak memory	39.8 GiB
Avg memory	32.2 GiB
CPU user	1328.8s
CPU sys	98.2s
Disk read	0 B
Disk write	608.0 KiB

adriangbot · 2026-03-12T22:12:52Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and remove_prefetch
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 50.31 / 50.97 ±0.46 / 51.76 ms │ 50.26 / 51.14 ±0.64 / 51.90 ms │ no change │
│ QQuery 2 │ 155.41 / 155.68 ±0.32 / 156.25 ms │ 154.08 / 154.87 ±0.46 / 155.40 ms │ no change │
│ QQuery 3 │ 120.29 / 121.51 ±0.65 / 122.15 ms │ 119.61 / 120.30 ±0.53 / 121.24 ms │ no change │
│ QQuery 4 │ 1449.85 / 1472.40 ±12.83 / 1484.86 ms │ 1438.16 / 1463.14 ±16.87 / 1481.93 ms │ no change │
│ QQuery 5 │ 186.69 / 187.85 ±1.14 / 189.86 ms │ 186.49 / 187.20 ±0.88 / 188.93 ms │ no change │
│ QQuery 6 │ 1073.36 / 1083.22 ±7.53 / 1092.84 ms │ 1024.31 / 1045.61 ±13.53 / 1064.04 ms │ no change │
│ QQuery 7 │ 362.31 / 366.34 ±3.97 / 373.61 ms │ 363.73 / 366.98 ±3.83 / 374.24 ms │ no change │
│ QQuery 8 │ 124.93 / 125.31 ±0.43 / 126.11 ms │ 124.03 / 124.97 ±1.15 / 127.15 ms │ no change │
│ QQuery 9 │ 113.35 / 116.24 ±2.36 / 119.06 ms │ 107.39 / 114.73 ±4.03 / 118.45 ms │ no change │
│ QQuery 10 │ 115.40 / 115.98 ±0.59 / 116.79 ms │ 114.47 / 116.20 ±0.97 / 117.00 ms │ no change │
│ QQuery 11 │ 1030.73 / 1040.00 ±5.27 / 1046.49 ms │ 1021.22 / 1032.65 ±8.92 / 1045.83 ms │ no change │
│ QQuery 12 │ 48.43 / 49.86 ±1.07 / 51.47 ms │ 48.60 / 50.52 ±1.68 / 53.57 ms │ no change │
│ QQuery 13 │ 415.57 / 421.64 ±5.35 / 430.69 ms │ 413.63 / 415.08 ±1.39 / 417.09 ms │ no change │
│ QQuery 14 │ 1094.06 / 1103.64 ±6.10 / 1110.71 ms │ 1095.79 / 1098.09 ±2.25 / 1101.78 ms │ no change │
│ QQuery 15 │ 17.91 / 18.80 ±0.85 / 20.35 ms │ 18.87 / 19.42 ±0.60 / 20.20 ms │ no change │
│ QQuery 16 │ 46.01 / 47.36 ±1.07 / 48.64 ms │ 46.30 / 46.87 ±0.53 / 47.79 ms │ no change │
│ QQuery 17 │ 257.06 / 258.02 ±0.65 / 258.75 ms │ 253.12 / 255.15 ±1.37 / 256.96 ms │ no change │
│ QQuery 18 │ 135.32 / 136.67 ±0.97 / 138.05 ms │ 133.14 / 135.49 ±1.74 / 137.79 ms │ no change │
│ QQuery 19 │ 166.79 / 168.43 ±0.88 / 169.43 ms │ 165.46 / 166.33 ±1.24 / 168.77 ms │ no change │
│ QQuery 20 │ 16.19 / 16.67 ±0.50 / 17.55 ms │ 16.24 / 17.25 ±0.51 / 17.55 ms │ no change │
│ QQuery 21 │ 24.41 / 24.72 ±0.35 / 25.27 ms │ 23.95 / 24.85 ±0.63 / 25.68 ms │ no change │
│ QQuery 22 │ 528.92 / 533.24 ±4.12 / 540.85 ms │ 529.92 / 537.36 ±11.47 / 560.20 ms │ no change │
│ QQuery 23 │ 981.75 / 995.79 ±9.36 / 1009.24 ms │ 986.27 / 993.12 ±6.78 / 1005.06 ms │ no change │
│ QQuery 24 │ 446.50 / 448.15 ±1.98 / 451.90 ms │ 443.31 / 444.47 ±0.80 / 445.45 ms │ no change │
│ QQuery 25 │ 373.47 / 375.79 ±1.60 / 377.87 ms │ 369.62 / 371.94 ±1.48 / 374.23 ms │ no change │
│ QQuery 26 │ 85.38 / 87.18 ±1.13 / 88.72 ms │ 86.93 / 88.53 ±1.70 / 91.10 ms │ no change │
│ QQuery 27 │ 359.42 / 361.96 ±1.62 / 364.49 ms │ 358.28 / 359.72 ±1.52 / 361.70 ms │ no change │
│ QQuery 28 │ 156.87 / 157.75 ±0.73 / 158.97 ms │ 155.31 / 156.62 ±1.31 / 159.06 ms │ no change │
│ QQuery 29 │ 312.30 / 315.84 ±2.42 / 319.73 ms │ 312.92 / 315.27 ±1.34 / 316.64 ms │ no change │
│ QQuery 30 │ 49.68 / 51.96 ±1.73 / 54.15 ms │ 48.65 / 51.66 ±2.18 / 55.25 ms │ no change │
│ QQuery 31 │ 187.04 / 187.77 ±0.74 / 189.07 ms │ 184.46 / 186.13 ±1.06 / 187.54 ms │ no change │
│ QQuery 32 │ 62.55 / 63.96 ±1.80 / 67.49 ms │ 63.08 / 63.41 ±0.35 / 63.88 ms │ no change │
│ QQuery 33 │ 151.93 / 154.56 ±1.54 / 155.90 ms │ 150.31 / 152.11 ±1.15 / 153.78 ms │ no change │
│ QQuery 34 │ 111.70 / 113.57 ±1.12 / 115.10 ms │ 112.01 / 112.76 ±0.57 / 113.68 ms │ no change │
│ QQuery 35 │ 117.93 / 120.45 ±1.29 / 121.53 ms │ 119.81 / 120.66 ±0.57 / 121.50 ms │ no change │
│ QQuery 36 │ 225.13 / 230.46 ±3.14 / 233.77 ms │ 225.52 / 231.40 ±3.03 / 233.86 ms │ no change │
│ QQuery 37 │ 192.94 / 194.75 ±1.63 / 197.12 ms │ 191.33 / 192.52 ±1.43 / 195.09 ms │ no change │
│ QQuery 38 │ 95.23 / 96.16 ±0.92 / 97.64 ms │ 92.64 / 96.29 ±3.24 / 101.80 ms │ no change │
│ QQuery 39 │ 139.12 / 143.17 ±2.54 / 146.88 ms │ 141.37 / 142.88 ±1.34 / 145.18 ms │ no change │
│ QQuery 40 │ 117.67 / 125.20 ±4.76 / 132.29 ms │ 123.16 / 127.40 ±3.81 / 134.55 ms │ no change │
│ QQuery 41 │ 18.26 / 19.75 ±1.09 / 21.29 ms │ 18.28 / 19.43 ±1.22 / 21.60 ms │ no change │
│ QQuery 42 │ 113.19 / 114.32 ±0.93 / 115.78 ms │ 114.69 / 116.11 ±0.75 / 116.83 ms │ no change │
│ QQuery 43 │ 90.37 / 91.11 ±0.52 / 91.72 ms │ 91.08 / 91.88 ±0.42 / 92.30 ms │ no change │
│ QQuery 44 │ 17.51 / 18.07 ±0.36 / 18.52 ms │ 17.63 / 18.15 ±0.33 / 18.61 ms │ no change │
│ QQuery 45 │ 56.79 / 58.33 ±0.93 / 59.58 ms │ 57.64 / 58.20 ±0.70 / 59.57 ms │ no change │
│ QQuery 46 │ 245.19 / 248.23 ±2.00 / 250.80 ms │ 248.16 / 251.12 ±1.55 / 252.67 ms │ no change │
│ QQuery 47 │ 795.42 / 802.85 ±4.59 / 807.84 ms │ 796.51 / 806.68 ±5.79 / 813.99 ms │ no change │
│ QQuery 48 │ 297.34 / 299.99 ±2.08 / 303.53 ms │ 298.15 / 304.06 ±4.63 / 311.89 ms │ no change │
│ QQuery 49 │ 265.50 / 268.14 ±2.49 / 272.47 ms │ 264.54 / 267.32 ±1.81 / 270.20 ms │ no change │
│ QQuery 50 │ 244.29 / 250.81 ±4.15 / 256.73 ms │ 242.75 / 250.02 ±3.91 / 253.69 ms │ no change │
│ QQuery 51 │ 193.47 / 196.96 ±2.20 / 199.64 ms │ 196.01 / 197.47 ±0.98 / 198.78 ms │ no change │
│ QQuery 52 │ 113.83 / 114.96 ±1.17 / 116.50 ms │ 113.29 / 115.15 ±1.36 / 117.25 ms │ no change │
│ QQuery 53 │ 109.69 / 111.35 ±1.62 / 113.80 ms │ 109.20 / 110.26 ±0.74 / 111.23 ms │ no change │
│ QQuery 54 │ 156.12 / 157.95 ±1.16 / 159.79 ms │ 158.40 / 159.55 ±0.87 / 160.77 ms │ no change │
│ QQuery 55 │ 111.91 / 113.48 ±1.05 / 115.22 ms │ 111.68 / 112.73 ±0.68 / 113.64 ms │ no change │
│ QQuery 56 │ 153.37 / 155.15 ±1.84 / 158.40 ms │ 152.80 / 154.11 ±1.16 / 156.13 ms │ no change │
│ QQuery 57 │ 191.66 / 193.09 ±1.16 / 194.49 ms │ 188.64 / 192.15 ±2.64 / 196.40 ms │ no change │
│ QQuery 58 │ 311.03 / 317.66 ±5.94 / 325.89 ms │ 306.95 / 319.20 ±12.87 / 341.59 ms │ no change │
│ QQuery 59 │ 212.62 / 215.27 ±2.47 / 219.45 ms │ 210.16 / 211.43 ±1.25 / 213.66 ms │ no change │
│ QQuery 60 │ 156.02 / 157.21 ±1.60 / 160.15 ms │ 154.01 / 155.53 ±1.22 / 157.67 ms │ no change │
│ QQuery 61 │ 185.80 / 188.48 ±2.19 / 191.43 ms │ 183.37 / 186.03 ±1.53 / 187.87 ms │ no change │
│ QQuery 62 │ 936.57 / 971.23 ±27.28 / 998.73 ms │ 920.73 / 934.61 ±15.62 / 964.80 ms │ no change │
│ QQuery 63 │ 111.20 / 113.90 ±2.08 / 115.99 ms │ 109.50 / 112.70 ±3.97 / 120.24 ms │ no change │
│ QQuery 64 │ 735.91 / 740.26 ±3.44 / 744.43 ms │ 729.37 / 734.78 ±3.81 / 739.93 ms │ no change │
│ QQuery 65 │ 274.95 / 277.39 ±2.90 / 282.81 ms │ 275.80 / 277.78 ±1.81 / 280.23 ms │ no change │
│ QQuery 66 │ 248.76 / 262.56 ±9.14 / 274.64 ms │ 256.15 / 260.65 ±3.40 / 265.71 ms │ no change │
│ QQuery 67 │ 357.12 / 363.63 ±5.58 / 371.16 ms │ 349.89 / 358.73 ±5.11 / 365.37 ms │ no change │
│ QQuery 68 │ 297.91 / 301.35 ±4.29 / 309.81 ms │ 297.28 / 300.25 ±2.39 / 304.04 ms │ no change │
│ QQuery 69 │ 113.63 / 115.55 ±1.56 / 117.86 ms │ 114.22 / 116.29 ±2.22 / 120.34 ms │ no change │
│ QQuery 70 │ 357.93 / 367.26 ±7.95 / 380.91 ms │ 343.26 / 359.42 ±8.55 / 368.79 ms │ no change │
│ QQuery 71 │ 145.01 / 145.99 ±1.25 / 148.35 ms │ 140.51 / 143.49 ±1.51 / 144.71 ms │ no change │
│ QQuery 72 │ 742.22 / 754.07 ±11.76 / 773.75 ms │ 738.53 / 755.76 ±8.99 / 763.84 ms │ no change │
│ QQuery 73 │ 108.53 / 111.40 ±1.83 / 114.06 ms │ 109.42 / 110.35 ±0.88 / 111.84 ms │ no change │
│ QQuery 74 │ 655.78 / 662.11 ±4.86 / 669.71 ms │ 653.33 / 659.19 ±4.84 / 667.18 ms │ no change │
│ QQuery 75 │ 297.06 / 299.35 ±1.78 / 301.84 ms │ 294.27 / 296.13 ±2.00 / 299.70 ms │ no change │
│ QQuery 76 │ 139.93 / 141.62 ±0.93 / 142.55 ms │ 139.94 / 141.11 ±1.25 / 143.05 ms │ no change │
│ QQuery 77 │ 205.20 / 206.98 ±1.79 / 210.26 ms │ 205.45 / 207.84 ±1.93 / 210.01 ms │ no change │
│ QQuery 78 │ 374.50 / 375.48 ±0.82 / 376.52 ms │ 364.97 / 371.30 ±3.45 / 375.06 ms │ no change │
│ QQuery 79 │ 249.74 / 252.21 ±3.06 / 258.12 ms │ 247.67 / 252.00 ±3.15 / 255.74 ms │ no change │
│ QQuery 80 │ 345.41 / 349.45 ±2.76 / 352.94 ms │ 342.27 / 346.05 ±2.80 / 350.84 ms │ no change │
│ QQuery 81 │ 31.89 / 32.45 ±0.72 / 33.83 ms │ 31.41 / 32.52 ±1.00 / 34.21 ms │ no change │
│ QQuery 82 │ 210.57 / 212.24 ±2.07 / 216.18 ms │ 208.03 / 211.46 ±2.26 / 215.02 ms │ no change │
│ QQuery 83 │ 47.20 / 48.82 ±1.12 / 49.75 ms │ 47.82 / 50.07 ±1.96 / 53.73 ms │ no change │
│ QQuery 84 │ 53.38 / 54.77 ±1.42 / 57.13 ms │ 52.65 / 53.57 ±0.82 / 54.78 ms │ no change │
│ QQuery 85 │ 154.03 / 156.03 ±1.58 / 158.88 ms │ 153.84 / 156.22 ±1.70 / 158.39 ms │ no change │
│ QQuery 86 │ 42.23 / 43.27 ±0.81 / 44.28 ms │ 42.47 / 43.05 ±0.67 / 43.91 ms │ no change │
│ QQuery 87 │ 94.63 / 98.11 ±4.54 / 107.00 ms │ 94.50 / 96.91 ±2.65 / 101.85 ms │ no change │
│ QQuery 88 │ 114.85 / 115.61 ±0.50 / 116.37 ms │ 113.91 / 114.97 ±0.72 / 115.85 ms │ no change │
│ QQuery 89 │ 127.79 / 128.64 ±0.47 / 129.16 ms │ 128.32 / 129.02 ±0.47 / 129.56 ms │ no change │
│ QQuery 90 │ 29.37 / 29.88 ±0.29 / 30.25 ms │ 28.73 / 29.69 ±0.86 / 31.30 ms │ no change │
│ QQuery 91 │ 67.28 / 68.66 ±1.82 / 72.23 ms │ 68.15 / 68.67 ±0.40 / 69.29 ms │ no change │
│ QQuery 92 │ 63.94 / 64.29 ±0.28 / 64.70 ms │ 63.28 / 64.67 ±1.45 / 67.32 ms │ no change │
│ QQuery 93 │ 200.20 / 203.37 ±2.00 / 206.38 ms │ 200.71 / 201.81 ±1.41 / 204.55 ms │ no change │
│ QQuery 94 │ 68.69 / 69.36 ±0.97 / 71.28 ms │ 67.21 / 67.69 ±0.45 / 68.49 ms │ no change │
│ QQuery 95 │ 144.68 / 147.87 ±2.90 / 152.62 ms │ 144.70 / 146.27 ±1.75 / 149.52 ms │ no change │
│ QQuery 96 │ 79.87 / 81.16 ±1.09 / 83.08 ms │ 77.22 / 80.51 ±1.86 / 82.90 ms │ no change │
│ QQuery 97 │ 135.33 / 139.54 ±3.40 / 144.86 ms │ 137.20 / 138.82 ±1.37 / 140.60 ms │ no change │
│ QQuery 98 │ 157.41 / 159.68 ±1.77 / 162.78 ms │ 158.35 / 159.87 ±1.32 / 161.79 ms │ no change │
│ QQuery 99 │ 10826.27 / 10871.32 ±27.52 / 10907.96 ms │ 10825.99 / 10878.61 ±38.07 / 10934.04 ms │ no change │
└───────────┴──────────────────────────────────────────┴──────────────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 35469.08ms │
│ Total Time (remove_prefetch) │ 35310.53ms │
│ Average Time (HEAD) │ 358.27ms │
│ Average Time (remove_prefetch) │ 356.67ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 99 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴────────────┘

Resource Usage

tpcds — base (merge-base)

Metric	Value
Wall time	177.7s
Peak memory	5.6 GiB
Avg memory	4.6 GiB
CPU user	283.2s
CPU sys	21.1s
Disk read	0 B
Disk write	636.1 MiB

tpcds — branch

Metric	Value
Wall time	176.9s
Peak memory	5.6 GiB
Avg memory	4.4 GiB
CPU user	281.3s
CPU sys	20.9s
Disk read	0 B
Disk write	148.0 KiB

Dandandan · 2026-03-12T22:13:31Z

- .metrics
- .clone();
- let _timer = scanning_total_metric.timer();
- self.start_next_file().transpose()


So this @alamb is what I was mostly talking about. It will read the footer (what we want) but AFAIK also:
build the pruning predicate (I think this is suboptimal, too early)
prune row groups
optionally load the page index
return the stream (without driving that forward)
We should be able to do this much better with the IO / CPU separation.

This makes sense to me 👍🏻

alamb-ghbot · 2026-03-12T22:30:05Z

🤖 ./gh_compare_branch.shgh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing remove_prefetch (ba2863c) to 57b275adiff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-03-12T23:06:54Z

🤖: Benchmark completed

Details

Comparing HEAD and remove_prefetch
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2312.38 ms │ 2333.66 ms │ no change │
│ QQuery 1 │ 865.55 ms │ 910.31 ms │ 1.05x slower │
│ QQuery 2 │ 1637.13 ms │ 1827.34 ms │ 1.12x slower │
│ QQuery 3 │ 998.37 ms │ 1053.16 ms │ 1.05x slower │
│ QQuery 4 │ 2190.67 ms │ 2183.89 ms │ no change │
│ QQuery 5 │ 26723.95 ms │ 27084.99 ms │ no change │
│ QQuery 6 │ 3602.82 ms │ 3898.27 ms │ 1.08x slower │
│ QQuery 7 │ 2873.65 ms │ 2771.68 ms │ no change │
└──────────┴─────────────┴─────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 41204.52ms │
│ Total Time (remove_prefetch) │ 42063.30ms │
│ Average Time (HEAD) │ 5150.56ms │
│ Average Time (remove_prefetch) │ 5257.91ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 4 │
│ Queries with No Change │ 4 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.57 ms │ 2.67 ms │ no change │
│ QQuery 1 │ 46.55 ms │ 47.72 ms │ no change │
│ QQuery 2 │ 150.47 ms │ 152.99 ms │ no change │
│ QQuery 3 │ 154.27 ms │ 155.31 ms │ no change │
│ QQuery 4 │ 971.25 ms │ 1013.94 ms │ no change │
│ QQuery 5 │ 1179.82 ms │ 1281.79 ms │ 1.09x slower │
│ QQuery 6 │ 6.22 ms │ 6.10 ms │ no change │
│ QQuery 7 │ 52.83 ms │ 52.45 ms │ no change │
│ QQuery 8 │ 1343.22 ms │ 1388.13 ms │ no change │
│ QQuery 9 │ 1677.53 ms │ 1752.61 ms │ no change │
│ QQuery 10 │ 306.18 ms │ 340.67 ms │ 1.11x slower │
│ QQuery 11 │ 353.68 ms │ 376.69 ms │ 1.07x slower │
│ QQuery 12 │ 1114.31 ms │ 1202.33 ms │ 1.08x slower │
│ QQuery 13 │ 1815.48 ms │ 1925.39 ms │ 1.06x slower │
│ QQuery 14 │ 1147.34 ms │ 1237.05 ms │ 1.08x slower │
│ QQuery 15 │ 1120.69 ms │ 1245.88 ms │ 1.11x slower │
│ QQuery 16 │ 2310.92 ms │ 2547.52 ms │ 1.10x slower │
│ QQuery 17 │ 2312.17 ms │ 2535.81 ms │ 1.10x slower │
│ QQuery 18 │ 5359.85 ms │ 4874.00 ms │ +1.10x faster │
│ QQuery 19 │ 115.10 ms │ 122.23 ms │ 1.06x slower │
│ QQuery 20 │ 1755.46 ms │ 1843.45 ms │ 1.05x slower │
│ QQuery 21 │ 1952.38 ms │ 2075.71 ms │ 1.06x slower │
│ QQuery 22 │ 3441.48 ms │ 3584.14 ms │ no change │
│ QQuery 23 │ 19325.01 ms │ 11321.87 ms │ +1.71x faster │
│ QQuery 24 │ 180.60 ms │ 170.86 ms │ +1.06x faster │
│ QQuery 25 │ 411.26 ms │ 423.73 ms │ no change │
│ QQuery 26 │ 183.28 ms │ 174.56 ms │ no change │
│ QQuery 27 │ 2523.11 ms │ 2620.70 ms │ no change │
│ QQuery 28 │ 23892.38 ms │ 22992.31 ms │ no change │
│ QQuery 29 │ 949.94 ms │ 972.85 ms │ no change │
│ QQuery 30 │ 1190.65 ms │ 1200.17 ms │ no change │
│ QQuery 31 │ 1256.26 ms │ 1242.72 ms │ no change │
│ QQuery 32 │ 4385.95 ms │ 4454.00 ms │ no change │
│ QQuery 33 │ 5332.41 ms │ 5323.05 ms │ no change │
│ QQuery 34 │ 6245.86 ms │ 5877.98 ms │ +1.06x faster │
│ QQuery 35 │ 1117.34 ms │ 1106.56 ms │ no change │
│ QQuery 36 │ 183.08 ms │ 182.77 ms │ no change │
│ QQuery 37 │ 68.71 ms │ 70.99 ms │ no change │
│ QQuery 38 │ 106.54 ms │ 115.68 ms │ 1.09x slower │
│ QQuery 39 │ 344.32 ms │ 334.93 ms │ no change │
│ QQuery 40 │ 40.77 ms │ 39.27 ms │ no change │
│ QQuery 41 │ 34.68 ms │ 35.08 ms │ no change │
│ QQuery 42 │ 31.52 ms │ 29.11 ms │ +1.08x faster │
└───────────┴─────────────┴─────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 96493.46ms │
│ Total Time (remove_prefetch) │ 88453.75ms │
│ Average Time (HEAD) │ 2244.03ms │
│ Average Time (remove_prefetch) │ 2057.06ms │
│ Queries Faster │ 5 │
│ Queries Slower │ 13 │
│ Queries with No Change │ 25 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 100.79 ms │ 100.69 ms │ no change │
│ QQuery 2 │ 30.73 ms │ 30.59 ms │ no change │
│ QQuery 3 │ 38.11 ms │ 38.62 ms │ no change │
│ QQuery 4 │ 30.35 ms │ 30.31 ms │ no change │
│ QQuery 5 │ 80.35 ms │ 80.73 ms │ no change │
│ QQuery 6 │ 20.56 ms │ 20.12 ms │ no change │
│ QQuery 7 │ 143.92 ms │ 148.17 ms │ no change │
│ QQuery 8 │ 39.10 ms │ 40.46 ms │ no change │
│ QQuery 9 │ 101.63 ms │ 96.09 ms │ +1.06x faster │
│ QQuery 10 │ 66.20 ms │ 63.35 ms │ no change │
│ QQuery 11 │ 19.23 ms │ 17.84 ms │ +1.08x faster │
│ QQuery 12 │ 55.72 ms │ 55.92 ms │ no change │
│ QQuery 13 │ 48.23 ms │ 48.71 ms │ no change │
│ QQuery 14 │ 14.07 ms │ 14.03 ms │ no change │
│ QQuery 15 │ 29.53 ms │ 29.51 ms │ no change │
│ QQuery 16 │ 26.86 ms │ 27.41 ms │ no change │
│ QQuery 17 │ 138.23 ms │ 142.63 ms │ no change │
│ QQuery 18 │ 267.59 ms │ 263.72 ms │ no change │
│ QQuery 19 │ 42.80 ms │ 43.50 ms │ no change │
│ QQuery 20 │ 55.13 ms │ 56.41 ms │ no change │
│ QQuery 21 │ 186.70 ms │ 192.19 ms │ no change │
│ QQuery 22 │ 22.32 ms │ 22.29 ms │ no change │
└───────────┴───────────┴─────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1558.18ms │
│ Total Time (remove_prefetch) │ 1563.29ms │
│ Average Time (HEAD) │ 70.83ms │
│ Average Time (remove_prefetch) │ 71.06ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 20 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴───────────┘

Dandandan · 2026-03-13T07:00:44Z

run benchmarks

adriangbot · 2026-03-13T07:00:47Z

Benchmark job started for this request (job bench-c4053198456-187). Results will be posted here when complete.

adriangbot · 2026-03-13T07:00:48Z

Benchmark job started for this request (job bench-c4053198456-188). Results will be posted here when complete.

adriangbot · 2026-03-13T07:00:48Z

Benchmark job started for this request (job bench-c4053198456-189). Results will be posted here when complete.

adriangbot · 2026-03-13T07:02:10Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4053198456-188-tv5q9 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (ba2863c) to 57b275a (merge-base) diff using: tpcds
Results will be posted here when complete

adriangbot · 2026-03-13T07:03:29Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4053198456-189-f98rr 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (ba2863c) to 57b275a (merge-base) diff using: tpch
Results will be posted here when complete

adriangbot · 2026-03-13T07:03:32Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4053198456-187-4tnh7 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (ba2863c) to 57b275a (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete

thinkharderdev

Makes sense. The original goal here was to pipeline the IO but current file opening blends too much IO and CPU work for this to be effective I think

Dandandan · 2026-03-17T15:01:44Z

run benchmark supported?

adriangbot · 2026-03-17T15:01:45Z

Hi @Dandandan, thanks for the request (#20916 (comment)).

Supported benchmarks:

Standard: clickbench_1, clickbench_extended, clickbench_partitioned, clickbench_pushdown, external_aggr, tpcds, tpch, tpch10, tpch_mem, tpch_mem10
Criterion: (any)

Usage:

run benchmark <name> # run specific benchmark(s)
run benchmarks # run default suite
run benchmarks <name1> <name2> # run specific benchmarks

Per-side configuration (run benchmark tpch followed by):

env:
SHARED_SETTING: enabledbaseline:
ref: v45.0.0env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1Gchanged:
ref: v46.0.0env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G

Dandandan · 2026-03-17T15:02:54Z

run benchmark tpch10 clickbench_partitioned clickbench_1 clickbench_extended

env: SIMULATE_LATENCY: true

adriangbot · 2026-03-17T15:05:58Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4075667797-358-n5rw2 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (344513d) to 11b9693 (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete

adriangbot · 2026-03-17T15:05:58Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4075667797-360-zf5dv 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (344513d) to 11b9693 (merge-base) diff using: clickbench_extended
Results will be posted here when complete

adriangbot · 2026-03-17T15:06:02Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4075667797-357-5hwzs 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (344513d) to 11b9693 (merge-base) diff using: tpch10
Results will be posted here when complete

adriangbot · 2026-03-17T15:06:08Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4075667797-359-fd99f 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (344513d) to 11b9693 (merge-base) diff using: clickbench_1
Results will be posted here when complete

Dandandan · 2026-03-17T15:07:57Z

Thanks all, let's wait a couple of days for further feedback - I launched some more benchmarks to check them all

Dandandan · 2026-03-17T15:16:22Z

run benchmark tpch10 clickbench_partitioned clickbench_1 clickbench_extended

adriangbot · 2026-03-17T15:19:13Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4075772779-363-wwn86 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (344513d) to 11b9693 (merge-base) diff using: clickbench_1
Results will be posted here when complete

adriangbot · 2026-03-17T15:19:14Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4075772779-362-sxgvg 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (344513d) to 11b9693 (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete

adriangbot · 2026-03-17T15:19:24Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4075772779-364-zwc88 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (344513d) to 11b9693 (merge-base) diff using: clickbench_extended
Results will be posted here when complete

adriangbot · 2026-03-17T15:19:27Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4075772779-361-dc9xr 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing remove_prefetch (344513d) to 11b9693 (merge-base) diff using: tpch10
Results will be posted here when complete

adriangbot · 2026-03-17T15:29:26Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and remove_prefetch
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 370.29 / 372.33 ±1.06 / 373.28 ms │ 370.34 / 374.20 ±2.77 / 378.17 ms │ no change │
│ QQuery 2 │ 138.30 / 141.73 ±1.98 / 143.88 ms │ 139.50 / 140.90 ±0.72 / 141.55 ms │ no change │
│ QQuery 3 │ 287.27 / 295.04 ±4.02 / 298.45 ms │ 290.91 / 298.78 ±5.06 / 305.95 ms │ no change │
│ QQuery 4 │ 153.17 / 154.91 ±1.67 / 157.00 ms │ 155.68 / 157.26 ±1.46 / 160.01 ms │ no change │
│ QQuery 5 │ 429.13 / 438.40 ±6.16 / 446.03 ms │ 435.86 / 438.21 ±2.38 / 442.07 ms │ no change │
│ QQuery 6 │ 134.72 / 136.71 ±2.55 / 141.68 ms │ 134.09 / 135.17 ±0.97 / 136.90 ms │ no change │
│ QQuery 7 │ 552.76 / 562.94 ±5.37 / 567.39 ms │ 565.05 / 569.97 ±5.67 / 580.75 ms │ no change │
│ QQuery 8 │ 481.22 / 484.12 ±4.55 / 493.17 ms │ 476.95 / 483.67 ±7.14 / 493.97 ms │ no change │
│ QQuery 9 │ 671.32 / 685.60 ±11.60 / 705.51 ms │ 680.58 / 692.52 ±7.94 / 700.12 ms │ no change │
│ QQuery 10 │ 328.09 / 338.68 ±7.09 / 346.69 ms │ 327.99 / 337.53 ±5.64 / 343.93 ms │ no change │
│ QQuery 11 │ 109.35 / 113.83 ±3.32 / 119.63 ms │ 106.67 / 112.45 ±3.95 / 116.85 ms │ no change │
│ QQuery 12 │ 205.35 / 208.56 ±2.30 / 210.64 ms │ 204.77 / 207.54 ±2.13 / 210.05 ms │ no change │
│ QQuery 13 │ 307.66 / 320.83 ±7.16 / 328.10 ms │ 304.51 / 315.67 ±7.19 / 326.57 ms │ no change │
│ QQuery 14 │ 183.32 / 186.86 ±3.05 / 192.45 ms │ 184.26 / 186.95 ±2.95 / 192.66 ms │ no change │
│ QQuery 15 │ 337.34 / 338.84 ±1.52 / 341.72 ms │ 337.91 / 341.13 ±3.12 / 346.33 ms │ no change │
│ QQuery 16 │ 89.64 / 90.22 ±0.66 / 91.04 ms │ 86.15 / 88.84 ±1.66 / 91.19 ms │ no change │
│ QQuery 17 │ 737.16 / 748.09 ±7.15 / 758.99 ms │ 739.36 / 745.44 ±5.48 / 753.57 ms │ no change │
│ QQuery 18 │ 857.17 / 870.60 ±11.43 / 887.93 ms │ 812.40 / 859.64 ±26.90 / 891.36 ms │ no change │
│ QQuery 19 │ 272.03 / 278.02 ±7.51 / 292.70 ms │ 271.42 / 276.47 ±8.20 / 292.72 ms │ no change │
│ QQuery 20 │ 325.63 / 330.07 ±4.27 / 335.47 ms │ 324.21 / 331.91 ±4.29 / 336.18 ms │ no change │
│ QQuery 21 │ 837.42 / 842.69 ±3.50 / 847.28 ms │ 837.45 / 851.90 ±10.30 / 869.55 ms │ no change │
│ QQuery 22 │ 83.21 / 86.76 ±3.10 / 92.46 ms │ 82.78 / 84.44 ±1.51 / 86.73 ms │ no change │
└───────────┴────────────────────────────────────┴────────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 8025.86ms │
│ Total Time (remove_prefetch) │ 8030.60ms │
│ Average Time (HEAD) │ 364.81ms │
│ Average Time (remove_prefetch) │ 365.03ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 22 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴───────────┘

Resource Usage

tpch10 — base (merge-base)

Metric	Value
Wall time	40.5s
Peak memory	10.0 GiB
Avg memory	7.5 GiB
CPU user	418.5s
CPU sys	30.8s
Disk read	0 B
Disk write	2.9 GiB

tpch10 — branch

Metric	Value
Wall time	40.5s
Peak memory	10.1 GiB
Avg memory	7.6 GiB
CPU user	420.0s
CPU sys	30.0s
Disk read	0 B
Disk write	80.0 KiB

adriangbot · 2026-03-17T15:30:34Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and remove_prefetch
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 2218.79 / 2309.58 ±61.74 / 2367.48 ms │ 2266.66 / 2332.75 ±77.29 / 2473.20 ms │ no change │
│ QQuery 1 │ 2037.89 / 2208.24 ±180.95 / 2521.92 ms │ 1968.70 / 2185.68 ±117.59 / 2327.07 ms │ no change │
│ QQuery 2 │ 2259.41 / 2374.52 ±70.00 / 2476.28 ms │ 2155.62 / 2426.83 ±149.92 / 2565.57 ms │ no change │
│ QQuery 3 │ 2111.26 / 2347.96 ±244.27 / 2768.48 ms │ 2046.51 / 2254.74 ±193.09 / 2620.03 ms │ no change │
│ QQuery 4 │ 2231.96 / 2326.11 ±62.73 / 2414.27 ms │ 2117.37 / 2279.44 ±91.70 / 2402.11 ms │ no change │
│ QQuery 5 │ 10251.85 / 10464.49 ±152.65 / 10655.99 ms │ 10335.12 / 10690.31 ±282.24 / 11162.82 ms │ no change │
│ QQuery 6 │ 2840.86 / 2936.88 ±71.44 / 3033.10 ms │ 2693.64 / 2897.68 ±137.02 / 3090.04 ms │ no change │
│ QQuery 7 │ 2192.91 / 2331.01 ±122.68 / 2519.52 ms │ 2108.99 / 2406.81 ±226.21 / 2663.61 ms │ no change │
└──────────┴───────────────────────────────────────────┴───────────────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 27298.78ms │
│ Total Time (remove_prefetch) │ 27474.23ms │
│ Average Time (HEAD) │ 3412.35ms │
│ Average Time (remove_prefetch) │ 3434.28ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴────────────┘

Resource Usage

clickbench_extended — base (merge-base)

Metric	Value
Wall time	137.2s
Peak memory	32.0 GiB
Avg memory	25.8 GiB
CPU user	707.5s
CPU sys	27.5s
Disk read	0 B
Disk write	22.4 MiB

clickbench_extended — branch

Metric	Value
Wall time	138.1s
Peak memory	32.0 GiB
Avg memory	25.6 GiB
CPU user	710.6s
CPU sys	27.9s
Disk read	0 B
Disk write	724.0 KiB

adriangbot · 2026-03-17T15:34:06Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and remove_prefetch
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.60 / 4.78 ±6.26 / 17.30 ms │ 1.69 / 5.08 ±6.53 / 18.14 ms │ 1.06x slower │
│ QQuery 1 │ 14.61 / 15.00 ±0.25 / 15.33 ms │ 16.14 / 16.48 ±0.29 / 16.89 ms │ 1.10x slower │
│ QQuery 2 │ 55.45 / 55.86 ±0.30 / 56.28 ms │ 57.87 / 58.07 ±0.26 / 58.57 ms │ no change │
│ QQuery 3 │ 49.30 / 51.58 ±1.28 / 52.98 ms │ 49.56 / 53.37 ±2.81 / 56.68 ms │ no change │
│ QQuery 4 │ 287.02 / 295.13 ±8.70 / 309.55 ms │ 314.13 / 346.64 ±19.16 / 363.13 ms │ 1.17x slower │
│ QQuery 5 │ 342.52 / 347.18 ±3.51 / 351.90 ms │ 347.54 / 351.79 ±4.06 / 356.56 ms │ no change │
│ QQuery 6 │ 5.04 / 5.72 ±0.56 / 6.52 ms │ 5.02 / 7.90 ±1.90 / 10.02 ms │ 1.38x slower │
│ QQuery 7 │ 17.12 / 17.53 ±0.49 / 18.49 ms │ 17.84 / 18.01 ±0.12 / 18.19 ms │ no change │
│ QQuery 8 │ 421.18 / 434.03 ±12.39 / 455.44 ms │ 441.23 / 454.35 ±19.83 / 493.64 ms │ no change │
│ QQuery 9 │ 653.50 / 673.42 ±23.44 / 717.89 ms │ 656.28 / 697.09 ±33.72 / 737.13 ms │ no change │
│ QQuery 10 │ 91.71 / 94.55 ±1.98 / 97.69 ms │ 97.61 / 102.25 ±3.30 / 107.89 ms │ 1.08x slower │
│ QQuery 11 │ 104.40 / 105.48 ±0.92 / 106.88 ms │ 110.55 / 111.65 ±0.91 / 113.14 ms │ 1.06x slower │
│ QQuery 12 │ 336.41 / 346.32 ±6.72 / 355.16 ms │ 347.10 / 366.07 ±20.16 / 403.55 ms │ 1.06x slower │
│ QQuery 13 │ 457.59 / 463.12 ±4.80 / 470.35 ms │ 469.28 / 486.84 ±15.77 / 510.30 ms │ 1.05x slower │
│ QQuery 14 │ 351.70 / 358.92 ±8.89 / 375.98 ms │ 355.00 / 363.08 ±5.90 / 372.20 ms │ no change │
│ QQuery 15 │ 388.02 / 400.18 ±12.97 / 424.48 ms │ 361.16 / 370.80 ±7.09 / 379.29 ms │ +1.08x faster │
│ QQuery 16 │ 707.05 / 732.56 ±29.34 / 790.00 ms │ 730.59 / 761.06 ±20.12 / 783.60 ms │ no change │
│ QQuery 17 │ 706.01 / 727.28 ±12.29 / 739.18 ms │ 714.72 / 722.92 ±5.66 / 732.46 ms │ no change │
│ QQuery 18 │ 1435.89 / 1493.33 ±43.75 / 1563.85 ms │ 1468.70 / 1512.19 ±25.39 / 1541.63 ms │ no change │
│ QQuery 19 │ 35.99 / 37.81 ±1.93 / 41.51 ms │ 35.80 / 37.81 ±1.11 / 39.19 ms │ no change │
│ QQuery 20 │ 704.58 / 727.13 ±21.69 / 758.25 ms │ 712.24 / 730.86 ±17.81 / 763.89 ms │ no change │
│ QQuery 21 │ 751.53 / 756.01 ±2.82 / 759.60 ms │ 776.94 / 781.97 ±3.63 / 786.65 ms │ no change │
│ QQuery 22 │ 1126.55 / 1131.69 ±6.56 / 1144.32 ms │ 1126.80 / 1137.06 ±12.59 / 1161.78 ms │ no change │
│ QQuery 23 │ 3197.84 / 3246.44 ±41.40 / 3310.66 ms │ 3051.33 / 3104.73 ±57.18 / 3212.88 ms │ no change │
│ QQuery 24 │ 102.80 / 104.74 ±2.22 / 109.05 ms │ 102.32 / 105.00 ±1.83 / 107.38 ms │ no change │
│ QQuery 25 │ 140.00 / 141.69 ±0.93 / 142.59 ms │ 139.88 / 141.97 ±1.36 / 143.38 ms │ no change │
│ QQuery 26 │ 100.19 / 103.18 ±1.67 / 104.91 ms │ 98.16 / 102.05 ±2.34 / 104.64 ms │ no change │
│ QQuery 27 │ 843.09 / 844.67 ±1.96 / 848.43 ms │ 846.27 / 853.93 ±5.50 / 861.21 ms │ no change │
│ QQuery 28 │ 7679.53 / 7747.29 ±38.93 / 7791.96 ms │ 7717.22 / 7785.87 ±44.53 / 7833.84 ms │ no change │
│ QQuery 29 │ 56.65 / 63.92 ±7.54 / 77.61 ms │ 62.41 / 104.04 ±23.46 / 126.09 ms │ 1.63x slower │
│ QQuery 30 │ 380.70 / 390.02 ±6.41 / 399.63 ms │ 368.63 / 385.34 ±15.38 / 412.33 ms │ no change │
│ QQuery 31 │ 378.84 / 384.23 ±4.61 / 389.65 ms │ 402.67 / 417.33 ±16.05 / 446.66 ms │ 1.09x slower │
│ QQuery 32 │ 1038.77 / 1070.33 ±29.20 / 1108.69 ms │ 1216.61 / 1266.78 ±54.11 / 1368.80 ms │ 1.18x slower │
│ QQuery 33 │ 1454.84 / 1526.54 ±54.42 / 1585.51 ms │ 1573.64 / 1637.50 ±73.42 / 1768.80 ms │ 1.07x slower │
│ QQuery 34 │ 1451.35 / 1464.16 ±12.53 / 1486.21 ms │ 1546.67 / 1620.09 ±49.48 / 1702.32 ms │ 1.11x slower │
│ QQuery 35 │ 391.01 / 397.40 ±9.86 / 416.92 ms │ 395.22 / 427.55 ±32.19 / 468.10 ms │ 1.08x slower │
│ QQuery 36 │ 120.56 / 125.19 ±3.30 / 130.61 ms │ 122.79 / 136.13 ±9.98 / 152.42 ms │ 1.09x slower │
│ QQuery 37 │ 48.72 / 51.05 ±1.39 / 52.74 ms │ 48.55 / 56.74 ±13.06 / 82.60 ms │ 1.11x slower │
│ QQuery 38 │ 75.46 / 77.90 ±1.97 / 80.94 ms │ 78.87 / 83.73 ±2.87 / 87.68 ms │ 1.07x slower │
│ QQuery 39 │ 219.61 / 223.63 ±3.34 / 227.97 ms │ 213.04 / 228.25 ±8.57 / 236.11 ms │ no change │
│ QQuery 40 │ 24.04 / 27.52 ±3.04 / 31.94 ms │ 23.57 / 27.33 ±2.89 / 32.33 ms │ no change │
│ QQuery 41 │ 20.67 / 22.27 ±0.88 / 23.40 ms │ 20.72 / 21.74 ±0.69 / 22.42 ms │ no change │
│ QQuery 42 │ 21.64 / 21.83 ±0.12 / 21.97 ms │ 20.40 / 21.47 ±1.25 / 23.87 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 27308.58ms │
│ Total Time (remove_prefetch) │ 28020.91ms │
│ Average Time (HEAD) │ 635.08ms │
│ Average Time (remove_prefetch) │ 651.65ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 17 │
│ Queries with No Change │ 25 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	137.7s
Peak memory	42.0 GiB
Avg memory	29.8 GiB
CPU user	1291.8s
CPU sys	92.1s
Disk read	0 B
Disk write	4.0 GiB

clickbench_partitioned — branch

Metric	Value
Wall time	140.6s
Peak memory	38.8 GiB
Avg memory	28.0 GiB
CPU user	1296.2s
CPU sys	117.3s
Disk read	0 B
Disk write	116.0 KiB

adriangbot · 2026-03-17T15:35:49Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and remove_prefetch
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 885.93 / 896.59 ±12.23 / 914.96 ms │ 868.20 / 886.58 ±11.28 / 902.09 ms │ no change │
│ QQuery 1 │ 206.93 / 207.63 ±0.63 / 208.71 ms │ 208.16 / 208.88 ±0.77 / 210.24 ms │ no change │
│ QQuery 2 │ 499.85 / 501.83 ±2.08 / 505.75 ms │ 499.86 / 501.43 ±0.92 / 502.37 ms │ no change │
│ QQuery 3 │ 315.56 / 317.48 ±1.09 / 318.88 ms │ 320.21 / 322.01 ±1.18 / 323.26 ms │ no change │
│ QQuery 4 │ 690.72 / 699.15 ±8.89 / 714.55 ms │ 687.02 / 697.19 ±13.89 / 724.05 ms │ no change │
│ QQuery 5 │ 9800.93 / 10081.34 ±247.90 / 10507.45 ms │ 10223.93 / 10357.82 ±148.06 / 10605.90 ms │ no change │
│ QQuery 6 │ 997.37 / 1010.22 ±12.83 / 1033.35 ms │ 1007.87 / 1013.52 ±4.60 / 1019.32 ms │ no change │
│ QQuery 7 │ 801.83 / 818.27 ±18.58 / 851.43 ms │ 817.96 / 841.06 ±27.69 / 892.71 ms │ no change │
└──────────┴──────────────────────────────────────────┴───────────────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 14532.49ms │
│ Total Time (remove_prefetch) │ 14828.49ms │
│ Average Time (HEAD) │ 1816.56ms │
│ Average Time (remove_prefetch) │ 1853.56ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴────────────┘

Resource Usage

clickbench_extended — base (merge-base)

Metric	Value
Wall time	73.3s
Peak memory	31.8 GiB
Avg memory	27.3 GiB
CPU user	725.2s
CPU sys	24.7s
Disk read	0 B
Disk write	21.8 MiB

clickbench_extended — branch

Metric	Value
Wall time	74.7s
Peak memory	30.3 GiB
Avg memory	25.3 GiB
CPU user	738.2s
CPU sys	27.7s
Disk read	0 B
Disk write	76.0 KiB

adriangbot · 2026-03-17T15:37:48Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and remove_prefetch
--------------------
Benchmark clickbench_1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.08 / 1.42 ±0.63 / 2.68 ms │ 1.08 / 1.40 ±0.62 / 2.64 ms │ no change │
│ QQuery 1 │ 14.68 / 15.14 ±0.50 / 16.11 ms │ 15.27 / 16.02 ±1.02 / 18.05 ms │ 1.06x slower │
│ QQuery 2 │ 40.94 / 41.13 ±0.10 / 41.25 ms │ 41.94 / 42.12 ±0.15 / 42.32 ms │ no change │
│ QQuery 3 │ 37.99 / 38.85 ±1.10 / 41.00 ms │ 37.82 / 38.04 ±0.16 / 38.24 ms │ no change │
│ QQuery 4 │ 267.56 / 271.97 ±4.02 / 279.30 ms │ 269.63 / 275.73 ±3.54 / 280.70 ms │ no change │
│ QQuery 5 │ 450.82 / 454.09 ±1.76 / 455.89 ms │ 447.74 / 464.34 ±9.05 / 474.54 ms │ no change │
│ QQuery 6 │ 4.74 / 6.60 ±0.97 / 7.56 ms │ 6.83 / 7.25 ±0.32 / 7.56 ms │ 1.10x slower │
│ QQuery 7 │ 16.74 / 16.81 ±0.06 / 16.89 ms │ 17.64 / 17.98 ±0.36 / 18.56 ms │ 1.07x slower │
│ QQuery 8 │ 385.56 / 397.89 ±6.85 / 404.46 ms │ 407.03 / 418.87 ±9.64 / 436.35 ms │ 1.05x slower │
│ QQuery 9 │ 607.75 / 617.33 ±6.90 / 628.82 ms │ 619.02 / 633.34 ±12.08 / 651.05 ms │ no change │
│ QQuery 10 │ 92.83 / 96.78 ±5.45 / 107.44 ms │ 95.23 / 97.92 ±3.74 / 105.30 ms │ no change │
│ QQuery 11 │ 106.24 / 106.90 ±0.49 / 107.67 ms │ 107.90 / 109.08 ±1.07 / 111.10 ms │ no change │
│ QQuery 12 │ 466.12 / 471.75 ±5.03 / 480.98 ms │ 466.65 / 475.30 ±5.86 / 484.90 ms │ no change │
│ QQuery 13 │ 530.44 / 544.84 ±10.62 / 558.47 ms │ 522.37 / 543.54 ±13.10 / 561.94 ms │ no change │
│ QQuery 14 │ 467.29 / 484.01 ±10.28 / 499.16 ms │ 477.79 / 487.46 ±8.79 / 498.54 ms │ no change │
│ QQuery 15 │ 351.39 / 372.44 ±16.16 / 395.11 ms │ 373.37 / 386.16 ±9.95 / 403.02 ms │ no change │
│ QQuery 16 │ 699.44 / 728.80 ±36.09 / 799.93 ms │ 709.10 / 715.99 ±7.86 / 730.58 ms │ no change │
│ QQuery 17 │ 700.40 / 710.30 ±13.58 / 737.07 ms │ 704.75 / 720.10 ±12.46 / 735.97 ms │ no change │
│ QQuery 18 │ 1447.75 / 1481.10 ±36.25 / 1547.34 ms │ 1382.78 / 1436.36 ±42.75 / 1491.07 ms │ no change │
│ QQuery 19 │ 38.60 / 54.61 ±18.21 / 82.87 ms │ 36.27 / 38.21 ±2.72 / 43.52 ms │ +1.43x faster │
│ QQuery 20 │ 616.80 / 635.78 ±22.19 / 674.97 ms │ 617.73 / 626.38 ±11.29 / 648.47 ms │ no change │
│ QQuery 21 │ 708.00 / 714.36 ±10.25 / 734.69 ms │ 699.56 / 705.88 ±5.71 / 716.33 ms │ no change │
│ QQuery 22 │ 1346.18 / 1363.01 ±19.49 / 1396.85 ms │ 1346.90 / 1350.08 ±2.46 / 1354.19 ms │ no change │
│ QQuery 23 │ 3681.55 / 3699.35 ±14.49 / 3712.25 ms │ 3704.17 / 3714.29 ±13.99 / 3741.08 ms │ no change │
│ QQuery 24 │ 217.22 / 220.19 ±3.23 / 225.98 ms │ 193.74 / 216.07 ±11.51 / 226.78 ms │ no change │
│ QQuery 25 │ 181.70 / 185.45 ±2.45 / 189.24 ms │ 183.42 / 186.85 ±3.16 / 192.18 ms │ no change │
│ QQuery 26 │ 215.35 / 216.79 ±1.42 / 218.53 ms │ 216.81 / 218.23 ±1.07 / 219.80 ms │ no change │
│ QQuery 27 │ 746.57 / 751.94 ±5.94 / 762.70 ms │ 743.33 / 747.87 ±4.79 / 756.99 ms │ no change │
│ QQuery 28 │ 8433.36 / 8491.31 ±34.79 / 8528.57 ms │ 8433.36 / 8520.38 ±48.96 / 8579.86 ms │ no change │
│ QQuery 29 │ 47.17 / 49.86 ±2.76 / 54.95 ms │ 48.71 / 51.92 ±5.96 / 63.83 ms │ no change │
│ QQuery 30 │ 437.75 / 456.36 ±11.31 / 473.08 ms │ 450.55 / 464.60 ±11.49 / 483.34 ms │ no change │
│ QQuery 31 │ 432.45 / 440.01 ±10.27 / 460.16 ms │ 439.68 / 453.19 ±10.24 / 464.95 ms │ no change │
│ QQuery 32 │ 1223.56 / 1270.22 ±53.79 / 1375.27 ms │ 1127.32 / 1252.63 ±80.37 / 1372.42 ms │ no change │
│ QQuery 33 │ 1544.50 / 1592.34 ±44.46 / 1670.90 ms │ 1493.52 / 1505.37 ±11.64 / 1520.01 ms │ +1.06x faster │
│ QQuery 34 │ 1522.01 / 1570.55 ±31.97 / 1608.73 ms │ 1511.28 / 1536.38 ±18.09 / 1558.78 ms │ no change │
│ QQuery 35 │ 363.92 / 367.18 ±2.34 / 371.13 ms │ 369.48 / 374.52 ±3.59 / 378.83 ms │ no change │
│ QQuery 36 │ 126.19 / 129.74 ±2.65 / 133.06 ms │ 121.21 / 129.86 ±5.10 / 136.96 ms │ no change │
│ QQuery 37 │ 54.58 / 57.20 ±1.62 / 59.01 ms │ 57.22 / 60.89 ±2.75 / 63.92 ms │ 1.06x slower │
│ QQuery 38 │ 84.74 / 85.79 ±0.70 / 86.57 ms │ 85.72 / 86.90 ±1.26 / 89.12 ms │ no change │
│ QQuery 39 │ 222.92 / 231.04 ±6.37 / 241.18 ms │ 219.50 / 231.61 ±9.43 / 242.69 ms │ no change │
│ QQuery 40 │ 25.15 / 28.01 ±2.12 / 31.12 ms │ 26.28 / 27.65 ±1.13 / 29.69 ms │ no change │
│ QQuery 41 │ 20.62 / 22.45 ±1.35 / 24.34 ms │ 21.13 / 22.04 ±0.57 / 22.90 ms │ no change │
│ QQuery 42 │ 20.63 / 21.06 ±0.23 / 21.25 ms │ 20.95 / 21.65 ±0.80 / 23.21 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 29512.75ms │
│ Total Time (remove_prefetch) │ 29430.47ms │
│ Average Time (HEAD) │ 686.34ms │
│ Average Time (remove_prefetch) │ 684.43ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 5 │
│ Queries with No Change │ 36 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴────────────┘

Resource Usage

clickbench_1 — base (merge-base)

Metric	Value
Wall time	148.6s
Peak memory	35.4 GiB
Avg memory	25.9 GiB
CPU user	1343.5s
CPU sys	107.2s
Disk read	0 B
Disk write	21.8 MiB

clickbench_1 — branch

Metric	Value
Wall time	148.1s
Peak memory	37.7 GiB
Avg memory	27.4 GiB
CPU user	1346.9s
CPU sys	99.7s
Disk read	0 B
Disk write	112.0 KiB

adriangbot · 2026-03-17T15:42:01Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and remove_prefetch
--------------------
Benchmark clickbench_1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 30.53 / 97.87 ±64.47 / 202.50 ms │ 30.47 / 97.79 ±64.46 / 202.43 ms │ no change │
│ QQuery 1 │ 2004.87 / 2097.48 ±76.27 / 2209.61 ms │ 1878.92 / 2054.89 ±95.81 / 2152.78 ms │ no change │
│ QQuery 2 │ 1964.16 / 2038.57 ±78.86 / 2147.59 ms │ 1970.59 / 2053.83 ±75.73 / 2196.47 ms │ no change │
│ QQuery 3 │ 1808.58 / 1952.09 ±97.72 / 2064.33 ms │ 1901.81 / 2021.07 ±101.42 / 2204.30 ms │ no change │
│ QQuery 4 │ 1991.77 / 2262.68 ±205.05 / 2558.41 ms │ 2116.17 / 2252.80 ±124.41 / 2474.00 ms │ no change │
│ QQuery 5 │ 2046.17 / 2250.46 ±159.45 / 2485.76 ms │ 2001.07 / 2156.15 ±167.30 / 2446.93 ms │ no change │
│ QQuery 6 │ 300.43 / 337.41 ±28.12 / 378.01 ms │ 299.55 / 357.03 ±45.38 / 435.98 ms │ 1.06x slower │
│ QQuery 7 │ 1756.32 / 2052.10 ±217.99 / 2421.67 ms │ 1787.15 / 1952.10 ±162.32 / 2249.81 ms │ no change │
│ QQuery 8 │ 2216.67 / 2289.45 ±78.01 / 2404.65 ms │ 2258.07 / 2383.83 ±84.00 / 2486.55 ms │ no change │
│ QQuery 9 │ 2421.08 / 2524.44 ±92.97 / 2690.54 ms │ 2237.14 / 2508.84 ±260.74 / 2980.16 ms │ no change │
│ QQuery 10 │ 1951.69 / 2166.30 ±178.82 / 2494.38 ms │ 2059.90 / 2163.96 ±94.81 / 2292.13 ms │ no change │
│ QQuery 11 │ 2160.46 / 2294.23 ±103.23 / 2402.68 ms │ 1990.89 / 2081.77 ±58.34 / 2158.92 ms │ +1.10x faster │
│ QQuery 12 │ 2076.99 / 2318.95 ±170.06 / 2569.39 ms │ 2073.77 / 2246.47 ±150.09 / 2469.10 ms │ no change │
│ QQuery 13 │ 2248.44 / 2376.25 ±77.71 / 2477.12 ms │ 2216.94 / 2348.96 ±85.18 / 2427.72 ms │ no change │
│ QQuery 14 │ 2103.56 / 2268.92 ±116.72 / 2407.16 ms │ 2109.75 / 2287.92 ±140.89 / 2489.93 ms │ no change │
│ QQuery 15 │ 2088.02 / 2162.80 ±96.39 / 2344.71 ms │ 2070.05 / 2210.29 ±127.51 / 2418.96 ms │ no change │
│ QQuery 16 │ 2407.65 / 2531.96 ±84.35 / 2672.56 ms │ 2382.70 / 2555.53 ±111.91 / 2683.20 ms │ no change │
│ QQuery 17 │ 2360.99 / 2557.85 ±149.43 / 2706.94 ms │ 2224.80 / 2490.19 ±244.24 / 2937.66 ms │ no change │
│ QQuery 18 │ 3085.46 / 3185.80 ±84.76 / 3307.63 ms │ 2952.54 / 3033.82 ±86.85 / 3184.88 ms │ no change │
│ QQuery 19 │ 1838.92 / 1988.74 ±170.03 / 2277.25 ms │ 1710.55 / 2032.94 ±240.27 / 2390.35 ms │ no change │
│ QQuery 20 │ 2293.47 / 2463.84 ±188.68 / 2812.70 ms │ 2381.55 / 2497.96 ±63.38 / 2559.71 ms │ no change │
│ QQuery 21 │ 2408.35 / 2633.23 ±174.18 / 2925.40 ms │ 2364.76 / 2532.64 ±106.12 / 2665.80 ms │ no change │
│ QQuery 22 │ 2968.41 / 3205.62 ±142.35 / 3348.80 ms │ 3159.86 / 3306.84 ±125.99 / 3465.28 ms │ no change │
│ QQuery 23 │ 5557.45 / 5727.34 ±116.97 / 5871.08 ms │ 5338.23 / 5463.80 ±99.03 / 5562.60 ms │ no change │
│ QQuery 24 │ 1874.59 / 2117.00 ±191.24 / 2419.05 ms │ 2093.01 / 2225.00 ±97.14 / 2371.40 ms │ 1.05x slower │
│ QQuery 25 │ 1982.83 / 2134.67 ±156.60 / 2387.44 ms │ 1927.77 / 2078.26 ±117.80 / 2247.47 ms │ no change │
│ QQuery 26 │ 1939.84 / 2186.79 ±129.83 / 2321.69 ms │ 1987.02 / 2280.12 ±181.57 / 2535.62 ms │ no change │
│ QQuery 27 │ 2578.43 / 2662.80 ±85.39 / 2823.80 ms │ 2612.01 / 2674.05 ±40.20 / 2720.80 ms │ no change │
│ QQuery 28 │ 9699.79 / 9887.61 ±157.29 / 10087.67 ms │ 9772.25 / 10072.43 ±204.66 / 10357.22 ms │ no change │
│ QQuery 29 │ 1943.10 / 2096.74 ±120.02 / 2225.13 ms │ 1793.92 / 2029.52 ±169.24 / 2279.56 ms │ no change │
│ QQuery 30 │ 2080.96 / 2338.40 ±153.07 / 2555.56 ms │ 2146.61 / 2325.49 ±96.48 / 2420.99 ms │ no change │
│ QQuery 31 │ 2066.65 / 2315.42 ±155.30 / 2491.88 ms │ 2112.61 / 2268.92 ±121.59 / 2427.54 ms │ no change │
│ QQuery 32 │ 2381.64 / 2634.40 ±144.27 / 2790.34 ms │ 2453.44 / 2537.32 ±83.63 / 2682.92 ms │ no change │
│ QQuery 33 │ 2994.75 / 3259.15 ±174.13 / 3453.99 ms │ 3023.38 / 3125.38 ±85.61 / 3283.38 ms │ no change │
│ QQuery 34 │ 3057.39 / 3166.16 ±69.77 / 3269.41 ms │ 3038.36 / 3261.72 ±135.63 / 3411.24 ms │ no change │
│ QQuery 35 │ 2175.55 / 2333.15 ±82.05 / 2398.50 ms │ 2138.05 / 2362.12 ±118.34 / 2471.96 ms │ no change │
│ QQuery 36 │ 307.25 / 363.40 ±30.93 / 402.41 ms │ 272.49 / 381.91 ±93.80 / 505.96 ms │ 1.05x slower │
│ QQuery 37 │ 235.42 / 279.30 ±30.68 / 317.17 ms │ 168.96 / 311.02 ±86.63 / 435.01 ms │ 1.11x slower │
│ QQuery 38 │ 225.86 / 322.07 ±52.98 / 385.19 ms │ 249.31 / 337.99 ±68.72 / 459.82 ms │ no change │
│ QQuery 39 │ 474.78 / 497.39 ±29.42 / 552.54 ms │ 337.56 / 415.48 ±62.10 / 522.44 ms │ +1.20x faster │
│ QQuery 40 │ 206.74 / 268.99 ±43.04 / 342.07 ms │ 145.72 / 280.06 ±94.66 / 405.80 ms │ no change │
│ QQuery 41 │ 198.60 / 258.34 ±47.74 / 339.99 ms │ 144.15 / 274.96 ±82.39 / 396.23 ms │ 1.06x slower │
│ QQuery 42 │ 175.51 / 262.78 ±58.63 / 335.71 ms │ 181.51 / 273.26 ±71.92 / 392.48 ms │ no change │
└───────────┴─────────────────────────────────────────┴──────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 95168.95ms │
│ Total Time (remove_prefetch) │ 94606.42ms │
│ Average Time (HEAD) │ 2213.23ms │
│ Average Time (remove_prefetch) │ 2200.15ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 5 │
│ Queries with No Change │ 36 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴────────────┘

Resource Usage

clickbench_1 — base (merge-base)

Metric	Value
Wall time	476.9s
Peak memory	34.5 GiB
Avg memory	22.6 GiB
CPU user	1353.2s
CPU sys	130.8s
Disk read	0 B
Disk write	22.4 MiB

clickbench_1 — branch

Metric	Value
Wall time	474.1s
Peak memory	36.7 GiB
Avg memory	27.9 GiB
CPU user	1350.6s
CPU sys	120.8s
Disk read	0 B
Disk write	820.0 KiB

adriangbot · 2026-03-17T15:42:27Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and remove_prefetch
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.69 / 5.06 ±6.52 / 18.11 ms │ 1.95 / 5.36 ±6.61 / 18.57 ms │ 1.06x slower │
│ QQuery 1 │ 2090.56 / 2393.04 ±185.36 / 2663.10 ms │ 2234.33 / 2496.85 ±153.11 / 2644.46 ms │ no change │
│ QQuery 2 │ 2637.29 / 2838.48 ±130.83 / 3048.29 ms │ 2469.91 / 2716.81 ±210.40 / 3042.53 ms │ no change │
│ QQuery 3 │ 2498.33 / 2798.70 ±163.36 / 2976.23 ms │ 2616.42 / 2943.00 ±226.61 / 3327.18 ms │ 1.05x slower │
│ QQuery 4 │ 2564.73 / 2864.96 ±289.77 / 3342.66 ms │ 2678.54 / 2937.53 ±169.48 / 3137.33 ms │ no change │
│ QQuery 5 │ 2691.72 / 2806.42 ±70.33 / 2895.75 ms │ 2807.60 / 3024.94 ±187.08 / 3274.64 ms │ 1.08x slower │
│ QQuery 6 │ 288.64 / 328.39 ±26.83 / 370.67 ms │ 206.01 / 219.38 ±14.71 / 246.15 ms │ +1.50x faster │
│ QQuery 7 │ 2126.00 / 2378.82 ±131.89 / 2500.54 ms │ 2270.60 / 2495.12 ±200.37 / 2752.25 ms │ no change │
│ QQuery 8 │ 2757.70 / 3156.84 ±216.29 / 3398.14 ms │ 2834.99 / 2978.01 ±128.14 / 3166.75 ms │ +1.06x faster │
│ QQuery 9 │ 3222.51 / 3349.47 ±100.62 / 3508.31 ms │ 3073.54 / 3347.15 ±187.66 / 3611.35 ms │ no change │
│ QQuery 10 │ 2371.95 / 2717.44 ±258.35 / 3170.40 ms │ 2615.66 / 2923.25 ±198.12 / 3236.70 ms │ 1.08x slower │
│ QQuery 11 │ 2525.49 / 2792.88 ±194.49 / 3029.00 ms │ 2641.07 / 2808.14 ±123.52 / 2988.39 ms │ no change │
│ QQuery 12 │ 2740.47 / 2944.36 ±255.08 / 3438.02 ms │ 2737.62 / 2845.63 ±103.64 / 3023.33 ms │ no change │
│ QQuery 13 │ 2903.77 / 3093.10 ±141.78 / 3237.96 ms │ 2833.37 / 2996.69 ±164.40 / 3306.82 ms │ no change │
│ QQuery 14 │ 2652.70 / 2838.62 ±167.66 / 3079.71 ms │ 2792.70 / 2899.19 ±77.22 / 3029.76 ms │ no change │
│ QQuery 15 │ 2805.75 / 3046.26 ±177.50 / 3348.74 ms │ 2756.86 / 2952.42 ±119.55 / 3129.42 ms │ no change │
│ QQuery 16 │ 3070.04 / 3197.94 ±137.40 / 3458.67 ms │ 2999.88 / 3260.48 ±227.97 / 3678.74 ms │ no change │
│ QQuery 17 │ 3297.66 / 3417.56 ±108.83 / 3574.28 ms │ 3021.01 / 3101.54 ±80.64 / 3247.34 ms │ +1.10x faster │
│ QQuery 18 │ 3613.16 / 3814.86 ±141.47 / 4053.52 ms │ 3589.55 / 3788.33 ±154.09 / 3964.95 ms │ no change │
│ QQuery 19 │ 2101.84 / 2418.54 ±183.88 / 2671.08 ms │ 2211.85 / 2504.64 ±249.15 / 2890.26 ms │ no change │
│ QQuery 20 │ 2931.88 / 3168.65 ±165.82 / 3433.88 ms │ 2844.63 / 3187.70 ±208.99 / 3442.05 ms │ no change │
│ QQuery 21 │ 2965.39 / 3229.71 ±142.28 / 3356.70 ms │ 3009.28 / 3190.29 ±245.63 / 3671.18 ms │ no change │
│ QQuery 22 │ 3556.24 / 3717.98 ±178.94 / 3989.18 ms │ 3614.05 / 3834.93 ±216.12 / 4214.26 ms │ no change │
│ QQuery 23 │ 5413.31 / 5565.97 ±99.16 / 5692.35 ms │ 5154.42 / 5287.65 ±190.34 / 5660.82 ms │ +1.05x faster │
│ QQuery 24 │ 1161.41 / 1463.82 ±224.05 / 1723.20 ms │ 1127.82 / 1251.88 ±119.79 / 1466.71 ms │ +1.17x faster │
│ QQuery 25 │ 2712.84 / 2841.99 ±134.75 / 3071.49 ms │ 2451.29 / 2710.94 ±193.63 / 2990.72 ms │ no change │
│ QQuery 26 │ 1178.74 / 1335.59 ±115.80 / 1531.60 ms │ 1111.99 / 1347.02 ±192.15 / 1639.65 ms │ no change │
│ QQuery 27 │ 3060.06 / 3196.53 ±159.26 / 3497.99 ms │ 3140.68 / 3333.04 ±186.83 / 3588.23 ms │ no change │
│ QQuery 28 │ 9951.11 / 10302.33 ±226.77 / 10536.26 ms │ 9740.89 / 10385.78 ±392.21 / 10938.17 ms │ no change │
│ QQuery 29 │ 2619.39 / 2797.13 ±152.91 / 2976.17 ms │ 2585.06 / 2813.86 ±231.08 / 3104.62 ms │ no change │
│ QQuery 30 │ 2718.22 / 2854.08 ±105.22 / 3004.89 ms │ 2840.27 / 3011.61 ±149.63 / 3267.07 ms │ 1.06x slower │
│ QQuery 31 │ 2684.19 / 2962.79 ±228.62 / 3244.38 ms │ 2651.64 / 2846.48 ±132.62 / 2995.23 ms │ no change │
│ QQuery 32 │ 2908.71 / 3070.46 ±124.64 / 3293.98 ms │ 2977.88 / 3105.72 ±88.40 / 3246.69 ms │ no change │
│ QQuery 33 │ 3776.89 / 3891.28 ±104.48 / 4080.48 ms │ 3848.26 / 4072.17 ±144.82 / 4245.05 ms │ no change │
│ QQuery 34 │ 3761.25 / 3979.51 ±171.34 / 4237.17 ms │ 3805.85 / 3925.40 ±90.54 / 4079.58 ms │ no change │
│ QQuery 35 │ 2753.08 / 3022.23 ±223.15 / 3399.20 ms │ 2685.70 / 2995.42 ±166.04 / 3168.42 ms │ no change │
│ QQuery 36 │ 230.20 / 317.84 ±62.53 / 393.20 ms │ 231.47 / 391.57 ±95.42 / 506.12 ms │ 1.23x slower │
│ QQuery 37 │ 195.74 / 250.54 ±46.50 / 318.69 ms │ 144.55 / 308.43 ±97.60 / 429.05 ms │ 1.23x slower │
│ QQuery 38 │ 211.93 / 290.55 ±69.09 / 404.95 ms │ 170.58 / 334.77 ±97.00 / 452.37 ms │ 1.15x slower │
│ QQuery 39 │ 389.45 / 483.72 ±65.99 / 588.65 ms │ 334.03 / 505.00 ±100.66 / 623.48 ms │ no change │
│ QQuery 40 │ 165.89 / 262.95 ±72.58 / 351.71 ms │ 118.88 / 283.21 ±96.56 / 394.04 ms │ 1.08x slower │
│ QQuery 41 │ 190.95 / 278.55 ±53.65 / 345.42 ms │ 117.27 / 277.02 ±95.12 / 391.60 ms │ no change │
│ QQuery 42 │ 257.19 / 277.51 ±22.32 / 318.45 ms │ 115.36 / 275.09 ±95.66 / 391.71 ms │ no change │
└───────────┴──────────────────────────────────────────┴──────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 112763.45ms │
│ Total Time (remove_prefetch) │ 112919.43ms │
│ Average Time (HEAD) │ 2622.41ms │
│ Average Time (remove_prefetch) │ 2626.03ms │
│ Queries Faster │ 5 │
│ Queries Slower │ 9 │
│ Queries with No Change │ 29 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴─────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	565.2s
Peak memory	35.7 GiB
Avg memory	27.0 GiB
CPU user	1352.8s
CPU sys	137.0s
Disk read	0 B
Disk write	4.0 GiB

clickbench_partitioned — branch

Metric	Value
Wall time	565.9s
Peak memory	35.4 GiB
Avg memory	27.4 GiB
CPU user	1353.5s
CPU sys	135.1s
Disk read	0 B
Disk write	832.0 KiB

adriangbot · 2026-03-17T15:42:34Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and remove_prefetch
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ remove_prefetch ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 3944.74 / 4141.27 ±125.39 / 4306.00 ms │ 4096.18 / 4162.00 ±78.05 / 4310.23 ms │ no change │
│ QQuery 2 │ 3507.19 / 3736.62 ±152.30 / 3964.48 ms │ 3580.29 / 3816.42 ±164.35 / 4091.05 ms │ no change │
│ QQuery 3 │ 6021.55 / 6171.88 ±92.76 / 6302.47 ms │ 5578.63 / 5791.35 ±194.28 / 6155.05 ms │ +1.07x faster │
│ QQuery 4 │ 5078.67 / 5304.12 ±134.68 / 5456.13 ms │ 5301.87 / 5546.09 ±185.30 / 5730.13 ms │ no change │
│ QQuery 5 │ 5997.10 / 6238.13 ±189.30 / 6571.39 ms │ 6010.02 / 6114.28 ±73.14 / 6222.99 ms │ no change │
│ QQuery 6 │ 3685.65 / 3915.99 ±237.51 / 4342.20 ms │ 3898.43 / 4111.60 ±204.15 / 4491.76 ms │ no change │
│ QQuery 7 │ 5945.67 / 6112.87 ±99.20 / 6225.24 ms │ 6068.97 / 6168.42 ±85.12 / 6272.22 ms │ no change │
│ QQuery 8 │ 6805.34 / 6966.72 ±137.77 / 7217.22 ms │ 6564.19 / 6848.09 ±195.67 / 7171.42 ms │ no change │
│ QQuery 9 │ 7364.64 / 7670.47 ±231.71 / 8054.28 ms │ 7454.96 / 7589.75 ±86.94 / 7681.67 ms │ no change │
│ QQuery 10 │ 5785.99 / 6307.08 ±284.78 / 6629.74 ms │ 6088.37 / 6236.51 ±135.93 / 6471.62 ms │ no change │
│ QQuery 11 │ 3057.61 / 3254.73 ±227.96 / 3695.75 ms │ 3073.63 / 3315.59 ±187.22 / 3651.24 ms │ no change │
│ QQuery 12 │ 5337.02 / 5616.78 ±203.95 / 5817.27 ms │ 5081.95 / 5478.88 ±396.57 / 6197.14 ms │ no change │
│ QQuery 13 │ 1723.78 / 1983.68 ±131.02 / 2071.24 ms │ 1802.05 / 1909.23 ±82.50 / 2053.78 ms │ no change │
│ QQuery 14 │ 4350.31 / 4527.32 ±148.09 / 4761.25 ms │ 4171.99 / 4403.63 ±162.54 / 4654.88 ms │ no change │
│ QQuery 15 │ 8247.15 / 8394.28 ±193.51 / 8753.73 ms │ 8195.43 / 8567.96 ±281.69 / 8945.69 ms │ no change │
│ QQuery 16 │ 1851.32 / 1905.63 ±53.52 / 1998.78 ms │ 1695.90 / 1858.80 ±133.97 / 2100.92 ms │ no change │
│ QQuery 17 │ 7920.71 / 8364.33 ±230.52 / 8563.32 ms │ 8190.24 / 8541.79 ±294.79 / 9054.55 ms │ no change │
│ QQuery 18 │ 9963.63 / 10284.84 ±196.82 / 10499.85 ms │ 9444.97 / 10269.59 ±512.32 / 10770.88 ms │ no change │
│ QQuery 19 │ 4083.91 / 4363.77 ±236.69 / 4719.17 ms │ 4063.76 / 4317.76 ±223.66 / 4703.75 ms │ no change │
│ QQuery 20 │ 6052.61 / 6377.80 ±235.16 / 6780.20 ms │ 5995.42 / 6401.75 ±205.21 / 6553.31 ms │ no change │
│ QQuery 21 │ 13818.11 / 14369.32 ±320.16 / 14764.29 ms │ 13997.89 / 14262.45 ±262.64 / 14743.22 ms │ no change │
│ QQuery 22 │ 2204.06 / 2293.21 ±68.18 / 2387.60 ms │ 2038.67 / 2263.07 ±140.62 / 2471.26 ms │ no change │
└───────────┴───────────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 128300.83ms │
│ Total Time (remove_prefetch) │ 127975.02ms │
│ Average Time (HEAD) │ 5831.86ms │
│ Average Time (remove_prefetch) │ 5817.05ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 21 │
│ Queries with Failure │ 0 │
└────────────────────────────────┴─────────────┘

Resource Usage

tpch10 — base (merge-base)

Metric	Value
Wall time	643.7s
Peak memory	9.7 GiB
Avg memory	7.2 GiB
CPU user	464.4s
CPU sys	70.5s
Disk read	0 B
Disk write	2.8 GiB

tpch10 — branch

Metric	Value
Wall time	642.1s
Peak memory	10.0 GiB
Avg memory	7.2 GiB
CPU user	463.4s
CPU sys	72.1s
Disk read	0 B
Disk write	800.0 KiB

Dandandan · 2026-03-20T15:03:00Z

Let's go forward with this!

Simplify the FileStream state machine by removing the mechanism that opens the next file in parallel while scanning the current one. Files are now opened sequentially (Scan -> Idle -> Open) instead of prefetching. ## Which issue does this PR close?  - Closesapache#20984 ## Rationale for this change Current prefetching is ineffective or even counterproductive, see e.g. query 6 when testing against simulated S3 latency: ``` │ QQuery 6 │ 709.93 / 744.83 ±21.29 / 769.15 ms │ 304.17 / 436.76 ±90.12 / 542.71 ms │ +1.71x faster │ ``` Let's remove it so we can make room for better prefetching taking care of pruning / late materilization better, cache behavior and IO/CPU split. ## What changes are included in this PR? Remove this API / prefetching to make room for more impactful prefetching, ## Are these changes tested? Existing tests. ## Are there any user-facing changes? Yes, `NextOpen` `next: Option<NextOpen>,` are removed of `pub` members. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Dandandan changed the title ~~Remove file prefetching from FileStream~~[bench] Remove file prefetching from FileStreamMar 12, 2026

github-actionsBot added the datasource Changes to the datasource crate label Mar 12, 2026

More coalesce
38fe60a

github-actionsBot added the core Core DataFusion crate label Mar 12, 2026

Revert "More coalesce"
e2f0106
This reverts commit 38fe60a.

github-actionsBot removed the core Core DataFusion crate label Mar 12, 2026

Dandandan commented Mar 12, 2026
View reviewed changes

Metric
ba2863c

Dandandan changed the title ~~[bench] Remove file prefetching from FileStream~~Remove file prefetching from FileStreamMar 13, 2026

Dandandan mentioned this pull request Mar 13, 2026
Sketch out a Morselize API #20820
Closed
5 tasks

thinkharderdev approved these changes Mar 17, 2026
View reviewed changes

xudong963 approved these changes Mar 18, 2026
View reviewed changes

Dandandan added this pull request to the merge queue Mar 20, 2026

Merged via the queue into apache:main with commit 78d5ac6Mar 20, 2026
11 checks passed

Dandandan deleted the remove_prefetch branch March 20, 2026 15:03

Uh oh!

Conversation

Dandandan commented Mar 12, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Dandandan commented Mar 12, 2026

Uh oh!

adriangbot commented Mar 12, 2026

Uh oh!

adriangbot commented Mar 12, 2026

Uh oh!

adriangbot commented Mar 12, 2026

Uh oh!

adriangbot commented Mar 12, 2026

Uh oh!

adriangbot commented Mar 12, 2026

Uh oh!

adriangbot commented Mar 12, 2026

Uh oh!

adriangbot commented Mar 12, 2026

Uh oh!

adriangbot commented Mar 12, 2026

Uh oh!

adriangbot commented Mar 12, 2026

Uh oh!

DandandanMar 12, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangbMar 13, 2026

Choose a reason for hiding this comment

Uh oh!

alamb-ghbot commented Mar 12, 2026

Uh oh!

alamb-ghbot commented Mar 12, 2026

Uh oh!

Dandandan commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

thinkharderdev left a comment

Choose a reason for hiding this comment

Uh oh!

Dandandan commented Mar 17, 2026

Uh oh!

adriangbot commented Mar 17, 2026

Uh oh!

Dandandan commented Mar 17, 2026

Uh oh!

adriangbot commented Mar 17, 2026

Uh oh!

adriangbot commented Mar 17, 2026

Uh oh!

adriangbot commented Mar 17, 2026

Uh oh!

adriangbot commented Mar 17, 2026

Uh oh!

Dandandan commented Mar 17, 2026

Uh oh!

Dandandan commented Mar 17, 2026

Uh oh!

adriangbot commented Mar 17, 2026

Uh oh!

adriangbot commented Mar 17, 2026

Uh oh!

Dandandan commented Mar 12, 2026•
edited
Loading

DandandanMar 12, 2026•
edited
Loading