Uh oh!
There was an error while loading. Please reload this page.
[benchmarks/ci] SO payload variants + restructured E2E Test Results comment - #3080
Conversation
… variant Addresses review feedback on #3077: - The SO writer now cycles deterministic variable-length text fragments ("The", " quick", ".\n", …; ~4.5 UTF-8 bytes avg) instead of repeating "aaaa", better approximating real token-stream traffic while staying byte-for-byte reproducible. - Adds a structured-delta payload mode ({ type: "text-delta", id, text }) resembling AI SDK events, run as a second SO scenario at the same 300 chunks / 100-per-sec pacing so payload shape is the only changed variable; the SO gap between the two scenarios isolates serialization cost. The two scenarios are labelled "stream overhead (text)" / "stream overhead (structured)"; the rename also keeps the changed text payload from diffing against the old fixed-payload baseline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 8181c25 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
ℹ️ Metric definitions & methodologyBest/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
- Move failed tests to the top as "Failed E2E Tests", hidden entirely when everything passed. Each category lists its failures inline under a heading; only categories with >=10 failures collapse into a <details>. - Group the rest under an "E2E Test Summary" section with two collapsibles: the overall summary table, and a flat "Details by Category" breakdown (no nested collapsibles). - Drop the redundant "Some E2E test jobs failed" append step in tests.yml (superseded by the failed-tests section) and its duplicate workflow-run link, keeping the single "View full workflow run" link from the script. - Add unit coverage for the aggregate renderer (.github/scripts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Backport to This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details. Once the underlying issue is fixed, re-run the Backport to stable workflow manually via |
Two independent PR-comment / benchmark polish changes, both following up on #3077.
1. SO benchmark payload variants
Addresses review feedback from @karthikscale3 on #3077.
"The"," quick", …".\n", ~4.5 UTF-8 bytes avg incl. punctuation/newline tokens) instead of repeating"aaaa"— closer to real token-stream traffic, still byte-for-byte reproducible.mode: 'text' | 'structured'arg onbenchSoWorkflow; structured mode wraps each fragment as{ type: "text-delta", id: "0", text }(AI SDK shape). Both scenarios stream the same 300 chunks @ 100/s over 3 s and reuse the same fragments, so payload shape is the only variable and the SO gap between them isolates serialization cost.Renders as two rows — SO / stream overhead (text) and SO / stream overhead (structured). The text scenario was renamed from
stream overhead, which also drops its stale fixed-payload baseline (both SO deltas start blank and re-baseline on the nextmainrun; no methodology bump, so other metrics keep their baselines).2. Restructured "E2E Test Results" PR comment
Reworks
.github/scripts/aggregate-e2e-results.js(aggregate mode) and.github/workflows/tests.yml:<details>.Testing
node --test ".github/scripts/**/*.test.js"— 29/29 pass, including new coverage for the aggregate renderer (few/many failures, all-pass, no-nested-details) and both SO rows.@workflow/coretypecheck passes;97_bench.tsrecompiles cleanly via SWC; both SO modes resolve through the workbench registry.🤖 Generated with Claude Code