Uh oh!
There was an error while loading. Please reload this page.
[next] Bundle the Vercel world into the Next.js server output - #3493
Conversation
`withWorkflow` put `@workflow/world-vercel` in `serverExternalPackages` alongside `@vercel/queue` and friends. The stated reason was to keep bundlers out of `@vercel/queue`'s keyring dependency tree, but that only ever applied to the queue side: no published `@vercel/queue` or `@vercel/oidc` version depends on `@vercel/cli-auth` or `@napi-rs/keyring`. Leaving the world external means its whole module graph is resolved from disk at runtime. On a cold start that lands in front of the first request, and it is not small: importing `workflow/runtime` from a server-startup hook resolved ~440 files, ~250 of them from the world's own subtree. Bundle the world and keep only the queue external. `@vercel/queue` has to stay external for a separate, still-current reason: it performs a fully dynamic `import(absolutePath)` that Turbopack rejects with "server relative imports are not implemented yet". Measured on a Next.js 15.5 app installed from npm (M4 Pro, warm fs), target world `vercel`: - `register()` in `instrumentation.ts`: 155ms -> 104ms (turbopack), 160ms -> 97ms (webpack) - flow route module evaluation: 118ms -> 103ms - API surface and `specVersion` of the resulting world are unchanged The workspace workbenches cannot show this difference: pnpm links the world out of `node_modules`, so Next.js bundles it either way. Only a real install reproduces the external path. Also flag the cold-start cost on the Postgres world's "Starting the World" section, which is the snippet that gets copied into apps that do not need it.
🦋 Changeset detectedLatest commit: 8d77de1 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 350992ms → this run 244557ms (Δ -106435ms, -30%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
🧪 E2E Test Results✅ All tests passed 🛠 Infra Events (absorbed by the harness)Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 🌐 Cross-language Conformance
✅ vercel-multi-region
|
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
| <Callout type="warn"> | ||
| This step is specific to worlds that run a background worker, such as Postgres | ||
| World. Do not copy it into an app deployed to Vercel: the Vercel world has no | ||
| `start()`, so the call is a no-op, and importing `workflow/runtime` from a |
There was a problem hiding this comment.
This seems untrue, we should have recently added a no-op world.start for Vercel too
There was a problem hiding this comment.
Checked before changing it: the no-op start() is on pgp/world-start-recovery (#2544), still open, so on main today createVercelWorld() returns no start at all and world.start?.() short-circuits.
Either way the wording was wrong to hang the warning on that, since it goes stale the moment #2544 lands. Rephrased to warn about the import instead, which is the part that costs anything:
Worlds whose queue delivers work over HTTP, including the Vercel world, have no worker to subscribe, so
start()does nothing there while the import still costs you: pulling inworkflow/runtimefrom a server-startup hook puts the whole runtime into the cold-start path before the first request is served.
Worth flagging on #2544 separately: it wires register() into the Next workbench un-gated, which is the pattern this warning is steering apps away from on Vercel.
pranaygp
commented
Aug 12, 2026
maybe we can fix this on the queue side too? and then bundle the entire thing in :) |
A no-op `start()` for the Vercel world is in flight (#2544), so phrasing the warning as "the Vercel world has no start()" would go stale the moment that lands. The cost being warned about is the import, not the call: pulling in `workflow/runtime` from a server-startup hook is what puts the runtime in front of the first request, whether or not there is a `start()` to call at the end of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
VaguelySerious
commented
Aug 13, 2026
Fixed on the queue side: vercel/vqs#189. The specifier is an absolute path that only exists once the process is running, so there is nothing for a bundler to resolve. Turbopack tries anyway, reads it as a request relative to the project root, and fails the build. It is now wrapped in a helper carrying Verified against the built
Both bundlers leave a live Not folding it into this PR, because dropping the external entry only works once that lands and ships to npm; until then it would break every build. Once it is released, the whole list can go, and that is worth measuring separately: the queue's tail is the bigger half of what is still resolved from disk at cold start. |
`require.resolve('@vercel/queue/package.json')` always throws: the published
package's `exports` map declares only `.`, so Node answers a manifest subpath
with ERR_PACKAGE_PATH_NOT_EXPORTED. The version gate therefore always failed
open and bundled the queue client regardless of the installed version. Read the
manifest from the nearest `node_modules` copy instead, and give the fixture the
same `exports` map the published package has so a resolver-based lookup cannot
pass against it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>Uh oh!
There was an error while loading. Please reload this page.
Vercel registered no GitHub deployment resource for four of the eighteen projects on af860ff, so the deployment-wait step in the docs smoke job and in the nextjs-webpack and nuxt Vercel Prod lanes had nothing to poll.
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
No backport to This is a cold-start performance optimization: it changes To override, re-run the Backport to stable workflow manually via |
withWorkflowadded@workflow/world-verceltoserverExternalPackages. This PR bundles the world and the queue client into the Next.js server output instead, and keeps only@vercel/oidc,@vercel/cli-authand@napi-rs/keyringexternal.Why
Everything left external is resolved from disk at runtime, one file at a time, and
register()ininstrumentation.tssits in front of the first request.The stated reason for externalizing the world was to keep bundlers out of
@vercel/queue's keyring dependency tree. That only ever applied to the queue side, and the queue client needed@vercel/queue0.5.0 first: earlier releases leave their dynamicimport()unannotated, 0.5.0 marks it withwebpackIgnore/turbopackIgnore.@vercel/oidcand the two CLI packages stay external. oidc 3.3+ reaches@vercel/cli-config, whosexdg-app-pathsdependency builds its config directory at module scope, and bundling that has failed a webpack build here withThe "path" argument must be of type string. It does bundle cleanly in some app layouts, so this is the conservative setting rather than a hard constraint. It leaves ~175 files on the runtime path (jose83,zod70, oidc itself 21, pluscli-exec/execa).Measurements
await import('workflow/runtime')insideregister(). Standalone Next 16.2.11 app installing this PR's tarballs versus main's (not a workspace link),WORKFLOW_TARGET_WORLD=vercel, default production build, Node 24.13, M4 Pro. Median of 12 alternating freshnext startprocesses per arm.Runtime files traced for the flow route in the npm app: 776 → 352. Server output grows 7.0 MB → 7.3 MB.
The last row is the caveat. Under webpack,
serverExternalPackagesonly takes effect for an entry that resolves from the app root, and a pnpm app has neither@workflow/world-vercelnor@vercel/queuethere, so main already bundles them,@vercel/oidcincluded. That is why the pnpm/webpack baseline is the fastest column in the table and why this PR cannot improve it. Turbopack externalizes by name, so pnpm apps do see the change there, and Next 16 defaults to turbopack.The resulting world has an identical API surface and
specVersion.Workbench coverage
nextjs-webpackcannot show the difference, for the pnpm reason above: builds before and after are equivalent.nextjs-turbopackdoes show it. Against main, the flow route's build trace drops from 216 to 179 runtime files, losing@vercel/queueand its tail (minimatch,brace-expansion,balanced-match,mixpart,picocolors).@workflow/world-vercelis bundled there in both revisions, being a workspace link.Docs
The Postgres world's "Starting the World" section is the snippet apps copy into
instrumentation.ts. It now carries a warning that the Vercel world has nostart()and that importingworkflow/runtimefrom a startup hook puts the whole runtime in front of the first request.