Uh oh!
There was an error while loading. Please reload this page.
[benchmarks] Split STSO by inline vs queue-hop steps, add distribution diffs vs main - #3213
Conversation
The sequential-steps benchmark's STSO metric mixed two unrelated
phenomena: gaps between steps running back-to-back in the same warm
process, and gaps across an invocation boundary (queue dispatch, client
reinit, event-log replay), which cost ~10x more. The old step-index
windows (1-20 / 101-120 / 1001-1020) sampled 19 gaps each and captured
neither cleanly: whether a boundary happened to land inside a window
moved that window's P99 by hundreds of percent, which is most of the
run-to-run variance the benchmark comment was reporting.
The workflow now tags each step with whether it was the first step body
executed in its process ('queue-hop') or a later one in the same warm
process ('inline') via a process-global, so the split is ground truth
rather than inferred from step index or trace timestamps. STSO is
reported as two rows over *every* gap in the run instead of three
sampled windows. No targets on the new rows — the old ones described the
index-bucketed grouping.
computeStats now keeps the full sorted sample array alongside the
percentiles, and the comment renders a histogram + cumulative-time diff
against `main` under the table, one per STSO kind. Percentiles alone
hide how many samples moved and by how much, which is exactly where the
variance lives. Inline rows use a fixed 50ms bin width (the adaptive
width is coarse enough to hide structure inside that cluster); queue-hop
rows keep the adaptive width. Negative gaps (clock skew between two step
bodies' clocks) get their own bucket rather than being counted with the
slow tail.
Raw samples are stripped from the comment's embedded data block — ~1000
per run would exceed GitHub's comment size limit within a couple of
history entries — so the histogram renders for the current run only,
while collapsed history keeps its tables. Until this lands on `main` no
baseline has raw samples, so the section renders this run's distribution
as a single series.
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>🦋 Changeset detectedLatest commit: 342f5a3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests💻 Local Development (1 failed)nextjs-webpack-stable (1 failed):
📦 Local Production (1 failed)nextjs-webpack-stable (1 failed):
E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
❌ 💻 Local Development
❌ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: 416691ms over 1016 samples No 1020 steps (queue-hop) Cumulative STSO time: 6302ms over 3 samples No 📜 Previous results (2)1bfc037Thu, 30 Jul 2026 19:20:17 GMT · run logs
95cf46aThu, 30 Jul 2026 18:51:40 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
karthikscale3
left a comment
There was a problem hiding this comment.
Reviewed the benchmark instrumentation and reporting changes. The current E2E failures are in unrelated webpack webhook/HMR tests and do not appear caused by this PR.
VaguelySerious
left a comment
There was a problem hiding this comment.
LGTM, but can we hide "STSO distribution" in a collapsible entry in the PR comment?
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
No backport to This is an enhancement to internal benchmarking tooling: it re-partitions the STSO metric into new inline/queue-hop categories, adds raw sample retention, and adds a new histogram/cumulative-time diff section to the benchmark PR comment. It adds new measurement capability and reporting surface rather than fixing a defect that affects To override, re-run the Backport to stable workflow manually via |
Why
The sequential-steps benchmark's STSO metric mixed two unrelated phenomena:
The old step-index windows (
1-20/101-120/1001-1020) sampled 19 gaps each and captured neither cleanly. A 1020-step run crosses ~3 invocation boundaries at non-deterministic step indices, so whether a boundary happened to land inside a window moved that window's P99 by hundreds of percent between two runs of the same commit against the same deployment. That was most of the run-to-run variance the benchmark comment has been reporting.Diagnosed on
shalabhc/benchmarks-variance, where separating the two categories showed inline STSO to be stable within ~3% across runs while the visible "regressions" were entirely boundary placement.What changed
Ground-truth tagging.
97_bench.tstags each step with whether it was the first step body executed in its process (queue-hop) or a later one in the same warm process (inline), via a process-global. Not inferred from step index or trace timestamps.Two STSO rows over every gap instead of three sampled 19-gap windows. No targets on the new rows — the old ones described the index-bucketed grouping.
Full sample retention.
computeStatskeeps the sorted sample array alongside the percentiles.Histogram + cumulative-time diff vs
main, one per STSO kind, rendered under the existing table (which is unchanged). Percentiles hide how many samples moved and by how much, which is exactly where the variance lives.No changes to
benchmarks.yml— this is still one run diffed against themainbaseline the workflow already downloads.Sample output
█ismain,┃marks where this run lands,░bridges the gap when this run has more samples in a bucket. Per-bucket counts and deltas ride on the bar line — no second table restating them.Notes for review
mainhas recorded raw samples yet, so the section renders as a single series (with a note) rather than diffing this run against itself. Deltas start on the next PR.<0 (skew)bucket. Step timestamps come from two different step bodies, so a gap can come out slightly negative under clock skew; counting those with the slowest samples inverts what the tail bucket means.Draft until the benchmark job runs on it and the rendered comment is confirmed.
🤖 Generated with Claude Code