Skip to content

Add --simulate-latency / SIMULATE_LATENCY option to dfbench / ./bench.sh - #20954

Merged
Dandandan merged 13 commits into
apache:mainfrom
Dandandan:throttle_dfbench
Mar 16, 2026
Merged

Add --simulate-latency / SIMULATE_LATENCY option to dfbench / ./bench.sh#20954
Dandandan merged 13 commits into
apache:mainfrom
Dandandan:throttle_dfbench

Conversation

@Dandandan

@DandandanDandandan commented Mar 15, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Measure IO / CPU pipelining (prefetching).

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Dandandanand others added 3 commits March 15, 2026 22:03
@Dandandan@claude
Add --simulate-latency option to dfbench to simulate S3-like object s…
d486889
…tore latency
Wraps the local filesystem with ThrottledStore (100ms get, 200ms list) when enabled,
allowing benchmarks to measure performance under realistic remote storage conditions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Dandandan
fmt
508f1a3
@Dandandan@claude
Add SIMULATE_LATENCY env var to bench.sh
1aec572
Passes --simulate-latency to all dfbench/imdb benchmark invocations when
SIMULATE_LATENCY=true, e.g.: SIMULATE_LATENCY=true ./bench.sh run tpch
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@DandandanDandandan mentioned this pull request Mar 15, 2026
Merged
adriangb
adriangb approved these changes Mar 15, 2026
View reviewed changes

@adriangbadriangb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pending CI fixes and successful test w/ benchmark runner

Dandandanand others added 2 commits March 15, 2026 22:35
@Dandandan@claude
Fix missing simulate_latency field in IMDB test CommonOpt initializers
581ac01
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Dandandan@claude
Enable object_store "fs" feature for ThrottledStore LocalFileSystem s…
a28e638
…upport
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Dandandan

Dandandan commented Mar 15, 2026

Copy link
Copy Markdown
ContributorAuthor

Hmm seems throttledobjectstore doesn't support files yet

@adriangb

adriangb commented Mar 15, 2026

Copy link
Copy Markdown
Contributor

We could vendor + submit a patch to arrow-rs?

@Dandandan

Dandandan commented Mar 16, 2026

Copy link
Copy Markdown
ContributorAuthor

We could vendor + submit a patch to arrow-rs?

Yeah or wrap our own, should be quite easy. Then we can also better customize it to "simulate" s3.

@adriangb

adriangb commented Mar 16, 2026

Copy link
Copy Markdown
Contributor

Up to you. Could make sense either way, it's not terrible to have 2 test object stores if they serve slightly different purposes.

Dandandanand others added 7 commits March 16, 2026 08:37
@Dandandan@claude
Replace ThrottledStore with LatencyObjectStore for realistic S3-like …
fda2a32
…latency simulation
Instead of fixed latency via ThrottledStore, cycle through a distribution
inspired by real S3 performance (P50: ~30ms, P75-P90: 100ms+, P99: 150-200ms).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Dandandan
Merge branch 'main' into throttle_dfbench
b1f0c6c
@Dandandan
Merge remote-tracking branch 'upstream/main' into throttle_dfbench
aeb953a
@Dandandan
Fixes
f7c7350
@Dandandan
Merge branch 'throttle_dfbench' of github.com:Dandandan/arrow-datafus…
0ed1923
…ion into throttle_dfbench
@Dandandan@claude
Shuffle latency distributions and use separate GET/LIST lists
f894ac1
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Dandandan@claude
Fix latency distributions to match target percentiles
edd6625
GET: P50≈32ms, P90≈110ms, P99≈200ms
LIST: P50≈65ms, P90≈180ms, P99≈400ms
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@DandandanDandandan changed the title [Minor] Add --simulate-latency / SIMULATE_LATENCY option to dfbench / ./bench.shAdd --simulate-latency / SIMULATE_LATENCY option to dfbench / ./bench.shMar 16, 2026
@Dandandan

Dandandan commented Mar 16, 2026

Copy link
Copy Markdown
ContributorAuthor
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ main ┃ throttle_dfbench ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 73.63 ms │ 110.89 ms │ 1.51x slower │
│ QQuery 2 │ 40.22 ms │ 489.39 ms │ 12.17x slower │
│ QQuery 3 │ 46.89 ms │ 274.80 ms │ 5.86x slower │
│ QQuery 4 │ 33.77 ms │ 175.22 ms │ 5.19x slower │
│ QQuery 5 │ 61.84 ms │ 390.31 ms │ 6.31x slower │
│ QQuery 6 │ 25.06 ms │ 56.65 ms │ 2.26x slower │
│ QQuery 7 │ 69.15 ms │ 358.44 ms │ 5.18x slower │
│ QQuery 8 │ 68.21 ms │ 829.40 ms │ 12.16x slower │
│ QQuery 9 │ 88.12 ms │ 540.30 ms │ 6.13x slower │
│ QQuery 10 │ 81.51 ms │ 295.32 ms │ 3.62x slower │
│ QQuery 11 │ 26.28 ms │ 248.17 ms │ 9.44x slower │
│ QQuery 12 │ 50.29 ms │ 177.37 ms │ 3.53x slower │
│ QQuery 13 │ 118.20 ms │ 190.37 ms │ 1.61x slower │
│ QQuery 14 │ 35.35 ms │ 172.84 ms │ 4.89x slower │
│ QQuery 15 │ 40.01 ms │ 337.29 ms │ 8.43x slower │
│ QQuery 16 │ 26.88 ms │ 115.27 ms │ 4.29x slower │
│ QQuery 17 │ 97.46 ms │ 455.69 ms │ 4.68x slower │
│ QQuery 18 │ 115.96 ms │ 469.73 ms │ 4.05x slower │
│ QQuery 19 │ 50.44 ms │ 245.72 ms │ 4.87x slower │
│ QQuery 20 │ 51.23 ms │ 264.92 ms │ 5.17x slower │
│ QQuery 21 │ 87.16 ms │ 622.37 ms │ 7.14x slower │
│ QQuery 22 │ 17.93 ms │ 103.60 ms │ 5.78x slower │
└───────────┴───────────┴──────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (main) │ 1305.59ms │
│ Total Time (throttle_dfbench) │ 6924.04ms │
│ Average Time (main) │ 59.34ms │
│ Average Time (throttle_dfbench) │ 314.73ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 22 │
│ Queries with No Change │ 0 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴───────────┘

🤣

@Dandandan

Dandandan commented Mar 16, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@alamb curious how better prefetching plays out on this (also for filter pushdown).

@Dandandan

Dandandan commented Mar 16, 2026

Copy link
Copy Markdown
ContributorAuthor

run benchmark clickbench_partitioned clickbench_pushdown

env: SIMULATE_LATENCY: true

@adriangbot

adriangbot commented Mar 16, 2026

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Linux bench-c4065985160-289-fj957 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing throttle_dfbench (edd6625) to ab28234 (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete

@adriangbot

adriangbot commented Mar 16, 2026

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Linux bench-c4065985160-290-js4c8 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing throttle_dfbench (edd6625) to ab28234 (merge-base) diff using: clickbench_pushdown
Results will be posted here when complete

@Dandandan@claude
Revert object_store feature flag to workspace default
4bab068
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@adriangbot

adriangbot commented Mar 16, 2026

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and throttle_dfbench
--------------------
Benchmark clickbench_pushdown.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ throttle_dfbench ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.59 / 4.92 ±6.46 / 17.85 ms │ 1.61 / 4.86 ±6.36 / 17.59 ms │ no change │
│ QQuery 1 │ 16.26 / 16.69 ±0.24 / 16.98 ms │ 16.10 / 16.55 ±0.26 / 16.87 ms │ no change │
│ QQuery 2 │ 57.27 / 57.95 ±0.43 / 58.43 ms │ 56.02 / 56.33 ±0.20 / 56.57 ms │ no change │
│ QQuery 3 │ 48.39 / 52.08 ±3.38 / 58.27 ms │ 48.68 / 50.29 ±1.32 / 51.68 ms │ no change │
│ QQuery 4 │ 311.17 / 318.58 ±3.96 / 322.43 ms │ 311.09 / 321.58 ±7.02 / 333.23 ms │ no change │
│ QQuery 5 │ 360.16 / 372.41 ±12.16 / 394.64 ms │ 356.59 / 362.48 ±5.01 / 369.79 ms │ no change │
│ QQuery 6 │ 11.30 / 11.67 ±0.31 / 12.05 ms │ 11.76 / 14.50 ±3.28 / 19.50 ms │ 1.24x slower │
│ QQuery 7 │ 23.33 / 23.90 ±0.53 / 24.78 ms │ 23.00 / 24.98 ±2.08 / 28.99 ms │ no change │
│ QQuery 8 │ 447.94 / 455.57 ±8.80 / 466.73 ms │ 446.86 / 455.81 ±9.25 / 467.31 ms │ no change │
│ QQuery 9 │ 696.68 / 714.07 ±10.59 / 725.34 ms │ 687.44 / 713.05 ±14.02 / 727.95 ms │ no change │
│ QQuery 10 │ 120.33 / 125.41 ±4.12 / 131.30 ms │ 121.47 / 124.63 ±5.12 / 134.84 ms │ no change │
│ QQuery 11 │ 134.10 / 136.35 ±1.82 / 138.87 ms │ 134.88 / 138.20 ±1.83 / 139.95 ms │ no change │
│ QQuery 12 │ 389.63 / 399.65 ±7.75 / 408.12 ms │ 390.51 / 403.66 ±11.12 / 421.61 ms │ no change │
│ QQuery 13 │ 507.06 / 517.98 ±6.68 / 524.45 ms │ 507.57 / 512.65 ±3.58 / 517.14 ms │ no change │
│ QQuery 14 │ 400.28 / 407.59 ±7.51 / 417.15 ms │ 399.75 / 407.94 ±5.63 / 415.60 ms │ no change │
│ QQuery 15 │ 375.99 / 392.38 ±8.39 / 398.36 ms │ 383.84 / 397.40 ±14.81 / 421.05 ms │ no change │
│ QQuery 16 │ 756.91 / 768.00 ±7.65 / 780.25 ms │ 764.72 / 786.75 ±17.36 / 811.79 ms │ no change │
│ QQuery 17 │ 769.48 / 795.46 ±21.92 / 833.01 ms │ 759.68 / 766.95 ±5.36 / 775.27 ms │ no change │
│ QQuery 18 │ 1493.83 / 1536.69 ±42.47 / 1600.42 ms │ 1488.57 / 1529.82 ±30.54 / 1582.59 ms │ no change │
│ QQuery 19 │ 36.44 / 37.41 ±1.56 / 40.52 ms │ 36.15 / 64.54 ±54.53 / 173.60 ms │ 1.73x slower │
│ QQuery 20 │ 712.01 / 731.34 ±18.77 / 759.80 ms │ 713.17 / 729.30 ±18.96 / 761.70 ms │ no change │
│ QQuery 21 │ 718.23 / 739.76 ±11.06 / 749.06 ms │ 734.51 / 752.41 ±16.16 / 781.78 ms │ no change │
│ QQuery 22 │ 1156.63 / 1184.23 ±24.82 / 1226.52 ms │ 1166.55 / 1188.55 ±15.24 / 1214.47 ms │ no change │
│ QQuery 23 │ 389.98 / 394.69 ±3.69 / 399.15 ms │ 378.58 / 391.51 ±10.60 / 409.32 ms │ no change │
│ QQuery 24 │ 115.41 / 119.00 ±2.41 / 122.05 ms │ 113.15 / 117.68 ±4.60 / 126.23 ms │ no change │
│ QQuery 25 │ 189.86 / 197.14 ±6.06 / 207.85 ms │ 190.21 / 194.00 ±2.41 / 196.72 ms │ no change │
│ QQuery 26 │ 159.36 / 165.51 ±3.64 / 170.56 ms │ 158.85 / 165.89 ±4.28 / 170.17 ms │ no change │
│ QQuery 27 │ 976.67 / 984.49 ±5.72 / 990.45 ms │ 962.38 / 974.48 ±7.23 / 983.89 ms │ no change │
│ QQuery 28 │ 7919.34 / 7963.24 ±34.69 / 8013.36 ms │ 7861.65 / 7927.18 ±42.08 / 7975.73 ms │ no change │
│ QQuery 29 │ 57.30 / 61.74 ±7.66 / 77.02 ms │ 56.75 / 61.91 ±5.06 / 71.19 ms │ no change │
│ QQuery 30 │ 376.33 / 384.28 ±5.95 / 391.65 ms │ 372.21 / 380.96 ±6.98 / 388.42 ms │ no change │
│ QQuery 31 │ 355.59 / 365.16 ±10.49 / 385.52 ms │ 348.75 / 369.01 ±11.58 / 384.46 ms │ no change │
│ QQuery 32 │ 1104.31 / 1178.51 ±85.88 / 1326.13 ms │ 1084.94 / 1109.92 ±27.93 / 1162.92 ms │ +1.06x faster │
│ QQuery 33 │ 1537.19 / 1555.65 ±16.82 / 1585.37 ms │ 1523.95 / 1547.27 ±12.54 / 1561.67 ms │ no change │
│ QQuery 34 │ 1516.07 / 1534.54 ±9.90 / 1545.60 ms │ 1543.05 / 1629.77 ±66.39 / 1744.32 ms │ 1.06x slower │
│ QQuery 35 │ 420.00 / 425.98 ±3.14 / 428.87 ms │ 428.22 / 454.38 ±31.32 / 503.22 ms │ 1.07x slower │
│ QQuery 36 │ 109.46 / 116.17 ±4.74 / 121.54 ms │ 108.53 / 116.14 ±5.56 / 123.86 ms │ no change │
│ QQuery 37 │ 58.73 / 61.63 ±2.26 / 65.11 ms │ 62.00 / 64.21 ±2.10 / 67.57 ms │ no change │
│ QQuery 38 │ 58.71 / 60.94 ±1.62 / 63.73 ms │ 55.24 / 59.28 ±3.11 / 62.86 ms │ no change │
│ QQuery 39 │ 178.54 / 194.97 ±9.85 / 205.69 ms │ 186.19 / 196.99 ±7.16 / 205.74 ms │ no change │
│ QQuery 40 │ 34.80 / 38.34 ±2.70 / 41.83 ms │ 34.97 / 38.43 ±3.18 / 44.12 ms │ no change │
│ QQuery 41 │ 32.67 / 34.29 ±1.30 / 35.90 ms │ 30.39 / 32.72 ±1.30 / 33.99 ms │ no change │
│ QQuery 42 │ 25.27 / 26.27 ±0.82 / 27.67 ms │ 24.99 / 25.55 ±0.48 / 26.28 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 25662.62ms │
│ Total Time (throttle_dfbench) │ 25680.52ms │
│ Average Time (HEAD) │ 596.81ms │
│ Average Time (throttle_dfbench) │ 597.22ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 4 │
│ Queries with No Change │ 38 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘

Resource Usage

clickbench_pushdown — base (merge-base)

MetricValue
Wall time129.5s
Peak memory40.3 GiB
Avg memory30.0 GiB
CPU user1180.9s
CPU sys97.5s
Disk read0 B
Disk write4.0 GiB

clickbench_pushdown — branch

MetricValue
Wall time129.3s
Peak memory39.8 GiB
Avg memory31.0 GiB
CPU user1177.0s
CPU sys100.3s
Disk read0 B
Disk write116.0 KiB

@adriangbot

adriangbot commented Mar 16, 2026

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and throttle_dfbench
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ throttle_dfbench ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.63 / 4.95 ±6.48 / 17.91 ms │ 1.70 / 5.06 ±6.53 / 18.11 ms │ no change │
│ QQuery 1 │ 15.22 / 15.90 ±0.40 / 16.47 ms │ 16.23 / 16.44 ±0.24 / 16.89 ms │ no change │
│ QQuery 2 │ 57.84 / 58.27 ±0.33 / 58.85 ms │ 56.66 / 57.18 ±0.47 / 58.04 ms │ no change │
│ QQuery 3 │ 49.67 / 51.49 ±1.35 / 53.74 ms │ 49.75 / 53.97 ±2.98 / 57.95 ms │ no change │
│ QQuery 4 │ 313.42 / 325.18 ±8.88 / 339.39 ms │ 339.91 / 353.17 ±16.29 / 385.13 ms │ 1.09x slower │
│ QQuery 5 │ 358.47 / 369.54 ±6.11 / 374.45 ms │ 362.95 / 373.52 ±10.26 / 388.10 ms │ no change │
│ QQuery 6 │ 5.96 / 6.69 ±0.65 / 7.78 ms │ 5.98 / 7.12 ±0.64 / 7.88 ms │ 1.06x slower │
│ QQuery 7 │ 18.23 / 18.44 ±0.15 / 18.66 ms │ 19.24 / 19.42 ±0.15 / 19.66 ms │ 1.05x slower │
│ QQuery 8 │ 453.85 / 464.72 ±8.01 / 474.40 ms │ 450.95 / 484.24 ±21.52 / 517.98 ms │ no change │
│ QQuery 9 │ 728.75 / 746.81 ±12.32 / 763.86 ms │ 695.14 / 724.18 ±19.19 / 747.65 ms │ no change │
│ QQuery 10 │ 94.12 / 97.53 ±3.77 / 104.63 ms │ 93.89 / 98.13 ±4.24 / 103.84 ms │ no change │
│ QQuery 11 │ 110.00 / 110.81 ±0.84 / 112.18 ms │ 109.15 / 111.47 ±1.39 / 112.83 ms │ no change │
│ QQuery 12 │ 352.22 / 368.48 ±10.80 / 380.62 ms │ 364.02 / 371.28 ±5.49 / 377.38 ms │ no change │
│ QQuery 13 │ 477.30 / 488.58 ±8.97 / 504.86 ms │ 480.07 / 496.80 ±10.88 / 511.05 ms │ no change │
│ QQuery 14 │ 357.62 / 383.20 ±12.85 / 391.98 ms │ 365.79 / 375.16 ±6.56 / 382.89 ms │ no change │
│ QQuery 15 │ 393.07 / 402.67 ±7.71 / 414.48 ms │ 407.34 / 428.25 ±15.66 / 452.83 ms │ 1.06x slower │
│ QQuery 16 │ 769.32 / 789.09 ±14.16 / 809.24 ms │ 776.60 / 802.10 ±23.57 / 842.90 ms │ no change │
│ QQuery 17 │ 752.99 / 781.27 ±18.19 / 803.42 ms │ 770.75 / 787.69 ±14.10 / 809.04 ms │ no change │
│ QQuery 18 │ 1456.67 / 1464.45 ±4.14 / 1467.99 ms │ 1512.37 / 1588.80 ±47.86 / 1644.18 ms │ 1.08x slower │
│ QQuery 19 │ 36.39 / 37.91 ±1.32 / 40.03 ms │ 37.36 / 40.27 ±4.42 / 49.02 ms │ 1.06x slower │
│ QQuery 20 │ 716.75 / 726.19 ±10.91 / 743.74 ms │ 717.58 / 732.75 ±13.09 / 753.71 ms │ no change │
│ QQuery 21 │ 761.21 / 772.91 ±6.31 / 779.99 ms │ 767.43 / 775.49 ±8.34 / 790.96 ms │ no change │
│ QQuery 22 │ 1135.63 / 1151.69 ±9.89 / 1160.31 ms │ 1139.56 / 1148.73 ±9.68 / 1166.02 ms │ no change │
│ QQuery 23 │ 3238.13 / 3256.59 ±18.20 / 3288.46 ms │ 3259.67 / 3297.92 ±24.99 / 3329.50 ms │ no change │
│ QQuery 24 │ 99.51 / 106.13 ±3.99 / 111.30 ms │ 100.03 / 108.54 ±4.33 / 112.23 ms │ no change │
│ QQuery 25 │ 140.17 / 141.48 ±0.87 / 142.65 ms │ 142.92 / 143.63 ±0.69 / 144.84 ms │ no change │
│ QQuery 26 │ 104.96 / 106.03 ±1.40 / 108.75 ms │ 104.99 / 108.79 ±3.01 / 112.84 ms │ no change │
│ QQuery 27 │ 856.11 / 863.36 ±5.55 / 870.90 ms │ 858.98 / 863.62 ±6.38 / 875.93 ms │ no change │
│ QQuery 28 │ 7802.00 / 7831.65 ±30.72 / 7872.67 ms │ 7793.29 / 7840.54 ±29.45 / 7876.49 ms │ no change │
│ QQuery 29 │ 56.88 / 60.63 ±4.43 / 69.18 ms │ 58.20 / 61.59 ±2.25 / 65.02 ms │ no change │
│ QQuery 30 │ 367.16 / 382.87 ±9.40 / 393.29 ms │ 384.18 / 394.10 ±5.62 / 400.53 ms │ no change │
│ QQuery 31 │ 378.78 / 387.67 ±8.37 / 400.04 ms │ 392.99 / 402.35 ±6.28 / 411.83 ms │ no change │
│ QQuery 32 │ 1113.76 / 1142.35 ±28.00 / 1176.91 ms │ 1331.14 / 1376.11 ±52.19 / 1472.65 ms │ 1.20x slower │
│ QQuery 33 │ 1531.25 / 1554.31 ±13.00 / 1570.95 ms │ 1666.96 / 1702.11 ±38.24 / 1770.79 ms │ 1.10x slower │
│ QQuery 34 │ 1556.93 / 1586.22 ±18.01 / 1613.26 ms │ 1577.89 / 1626.11 ±41.52 / 1675.95 ms │ no change │
│ QQuery 35 │ 449.16 / 467.10 ±11.90 / 480.57 ms │ 433.37 / 460.69 ±15.87 / 479.30 ms │ no change │
│ QQuery 36 │ 122.96 / 127.24 ±4.63 / 134.62 ms │ 128.18 / 129.50 ±1.97 / 133.36 ms │ no change │
│ QQuery 37 │ 52.82 / 53.54 ±1.04 / 55.56 ms │ 51.98 / 53.38 ±1.21 / 55.39 ms │ no change │
│ QQuery 38 │ 76.63 / 78.73 ±1.85 / 82.08 ms │ 78.03 / 80.08 ±1.40 / 81.71 ms │ no change │
│ QQuery 39 │ 219.33 / 237.58 ±13.02 / 253.55 ms │ 230.57 / 238.92 ±5.78 / 245.42 ms │ no change │
│ QQuery 40 │ 28.18 / 29.90 ±1.67 / 32.76 ms │ 25.71 / 27.60 ±1.51 / 30.27 ms │ +1.08x faster │
│ QQuery 41 │ 23.49 / 24.12 ±0.42 / 24.63 ms │ 22.14 / 22.97 ±0.67 / 23.99 ms │ no change │
│ QQuery 42 │ 21.95 / 22.90 ±0.49 / 23.29 ms │ 22.17 / 23.08 ±0.68 / 24.02 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 28097.18ms │
│ Total Time (throttle_dfbench) │ 28812.85ms │
│ Average Time (HEAD) │ 653.42ms │
│ Average Time (throttle_dfbench) │ 670.07ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 8 │
│ Queries with No Change │ 34 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

MetricValue
Wall time141.7s
Peak memory42.5 GiB
Avg memory33.8 GiB
CPU user1337.4s
CPU sys86.2s
Disk read0 B
Disk write3.8 GiB

clickbench_partitioned — branch

MetricValue
Wall time145.1s
Peak memory38.4 GiB
Avg memory29.8 GiB
CPU user1339.5s
CPU sys113.9s
Disk read0 B
Disk write760.0 KiB

@Dandandan

Dandandan commented Mar 16, 2026

Copy link
Copy Markdown
ContributorAuthor

run benchmark tpch

env: SIMULATE_LATENCY: true

@adriangbot

adriangbot commented Mar 16, 2026

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Linux bench-c4066093679-291-82bd7 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing throttle_dfbench (4bab068) to ab28234 (merge-base) diff using: tpch
Results will be posted here when complete

@adriangbot

adriangbot commented Mar 16, 2026

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and throttle_dfbench
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ throttle_dfbench ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 47.03 / 48.36 ±1.43 / 50.95 ms │ 46.85 / 47.40 ±0.74 / 48.86 ms │ no change │
│ QQuery 2 │ 25.06 / 25.40 ±0.21 / 25.73 ms │ 25.30 / 25.74 ±0.46 / 26.55 ms │ no change │
│ QQuery 3 │ 34.04 / 34.74 ±0.46 / 35.30 ms │ 33.90 / 34.26 ±0.24 / 34.49 ms │ no change │
│ QQuery 4 │ 22.19 / 22.85 ±0.72 / 24.24 ms │ 21.99 / 22.35 ±0.50 / 23.34 ms │ no change │
│ QQuery 5 │ 50.86 / 54.07 ±1.78 / 55.68 ms │ 51.65 / 52.70 ±1.22 / 55.04 ms │ no change │
│ QQuery 6 │ 17.87 / 18.00 ±0.14 / 18.24 ms │ 17.80 / 18.21 ±0.47 / 19.12 ms │ no change │
│ QQuery 7 │ 56.92 / 58.33 ±0.87 / 59.61 ms │ 58.02 / 59.59 ±1.48 / 61.61 ms │ no change │
│ QQuery 8 │ 51.12 / 52.01 ±0.55 / 52.64 ms │ 51.69 / 52.22 ±0.52 / 52.86 ms │ no change │
│ QQuery 9 │ 56.83 / 58.23 ±1.23 / 60.10 ms │ 56.69 / 58.55 ±1.20 / 60.39 ms │ no change │
│ QQuery 10 │ 74.60 / 76.22 ±2.25 / 80.67 ms │ 75.00 / 75.62 ±1.04 / 77.70 ms │ no change │
│ QQuery 11 │ 16.85 / 17.03 ±0.20 / 17.41 ms │ 17.46 / 17.75 ±0.33 / 18.38 ms │ no change │
│ QQuery 12 │ 29.02 / 29.59 ±0.57 / 30.45 ms │ 29.21 / 29.97 ±1.03 / 32.00 ms │ no change │
│ QQuery 13 │ 39.46 / 41.40 ±1.23 / 42.63 ms │ 39.70 / 40.35 ±0.57 / 41.36 ms │ no change │
│ QQuery 14 │ 29.87 / 30.29 ±0.28 / 30.75 ms │ 29.90 / 30.30 ±0.54 / 31.37 ms │ no change │
│ QQuery 15 │ 36.84 / 37.19 ±0.33 / 37.69 ms │ 36.78 / 37.17 ±0.25 / 37.47 ms │ no change │
│ QQuery 16 │ 17.70 / 18.54 ±0.73 / 19.89 ms │ 18.11 / 18.69 ±0.61 / 19.87 ms │ no change │
│ QQuery 17 │ 76.78 / 78.30 ±1.79 / 81.68 ms │ 76.47 / 77.73 ±1.50 / 80.66 ms │ no change │
│ QQuery 18 │ 82.96 / 84.78 ±1.88 / 88.24 ms │ 81.82 / 84.13 ±1.42 / 86.10 ms │ no change │
│ QQuery 19 │ 39.58 / 40.51 ±0.95 / 42.29 ms │ 39.22 / 39.97 ±0.48 / 40.73 ms │ no change │
│ QQuery 20 │ 43.63 / 44.46 ±0.87 / 45.61 ms │ 44.12 / 45.24 ±0.94 / 46.47 ms │ no change │
│ QQuery 21 │ 69.99 / 72.25 ±1.23 / 73.65 ms │ 71.14 / 72.84 ±1.54 / 75.10 ms │ no change │
│ QQuery 22 │ 20.05 / 20.56 ±0.45 / 21.37 ms │ 20.78 / 21.17 ±0.32 / 21.62 ms │ no change │
└───────────┴────────────────────────────────┴────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 963.10ms │
│ Total Time (throttle_dfbench) │ 961.94ms │
│ Average Time (HEAD) │ 43.78ms │
│ Average Time (throttle_dfbench) │ 43.72ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 22 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴──────────┘

Resource Usage

tpch — base (merge-base)

MetricValue
Wall time5.1s
Peak memory3.6 GiB
Avg memory3.1 GiB
CPU user34.6s
CPU sys3.0s
Disk read0 B
Disk write148.0 KiB

tpch — branch

MetricValue
Wall time5.1s
Peak memory3.6 GiB
Avg memory3.2 GiB
CPU user34.4s
CPU sys3.1s
Disk read0 B
Disk write56.0 KiB

@Dandandan
Dandandan added this pull request to the merge queueMar 16, 2026
Merged via the queue into apache:main with commit 8609288Mar 16, 2026
33 checks passed
@alambalamb mentioned this pull request Mar 17, 2026
Closed
5 tasks
alamb
alamb reviewed Mar 17, 2026
View reviewed changes
Comment threadbenchmarks/src/util/latency_object_store.rs

/// An ObjectStore wrapper that injects simulated latency on get and list calls.
#[derive(Debug)]
pub struct LatencyObjectStore<T: ObjectStore> {

@alambalambMar 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is pretty clever

de-bgunter pushed a commit to de-bgunter/datafusion that referenced this pull request Mar 24, 2026
@Dandandan@claude@de-bgunter
Add --simulate-latency / SIMULATE_LATENCY option to dfbench / ./bench…
5691b98
….sh (apache#20954)
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closesapache#123` indicates that this PR will close issue apache#123.
-->
- Closesapache#20957
## Rationale for this change
Measure IO / CPU pipelining (prefetching).
## What changes are included in this PR?
## Are these changes tested?
## Are there any user-facing changes?
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@alambalamb mentioned this pull request Apr 4, 2026
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alambalambalamb left review comments
@adriangbadriangbadriangb approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add --simulate-latency option to simulate object storage performance

4 participants

@Dandandan@adriangb@adriangbot@alamb