[core] Prepare replay payloads as event frames arrive

[NathanColosimo]

Summary

• prepare validated Node replay payloads as streamed events arrive
• lazily resolve the invocation encryption key and create the payload cache only after the run is known to use Node
• skip host-side compile, key lookup, replay cache, and payload preparation for QuickJS
• start run_started before speculative compile/cache work so setup I/O overlaps Node startup
• correct speculative workflow names from persisted run_created
• classify invalid VM configuration as run_failed in turbo, ordinary, and streamed setup paths
• count materialized replay events without per-event spans
• share one replay-run reconstruction path between run_started and atomic-hook preload
• leave primitive payload caching uncapped

Prewarm deliberately scans the event array on each replay and relies on the existing event-ID map to skip prepared payloads. This avoids positional scan state, reset hooks, and event-log replacement wrappers; only the cheap array walk repeats. Prepared plaintext and memoized primitive results remain invocation-scoped.

Stack

Depends on #3798.

Validation

• focused regression suites — 131 tests passed
• final runtime regression suites — 60 tests passed
• full @workflow/core suite — 2,295 tests passed
• full @workflow/world-vercel suite — 572 tests passed
• full @workflow/world suite — 165 tests passed
• pnpm --filter @workflow/core typecheck
• pnpm --filter @workflow/world-vercel typecheck
• Biome check on changed files
• git diff --check

[changeset-bot]

🦋 Changeset detected

Latest commit: 9f03030

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages

Name	Type
@workflow/core	Patch
@workflow/world	Patch
@workflow/world-vercel	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/world-local	Patch
@workflow/world-postgres	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-nextjs-workflow-webpack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-astro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-express-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-fastify-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-hono-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nestjs-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nitro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nuxt-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-python-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-sveltekit-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-tanstack-start-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-vite-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-docs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-swc-playground	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-tarballs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-web	Ready	Preview, v0	Aug 31, 2026 9:15pm

[github-actions]

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (8 failed)

python-node (8 failed):

• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW | 🔍 observability
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ | 🔍 observability
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z | 🔍 observability
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD | 🔍 observability
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95 | 🔍 observability
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D | 🔍 observability
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H | 🔍 observability
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3 | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

• deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M1CV028S4VSJGT08W5VNCKF3
• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

• abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-webpack)
• addTenWorkflow (nitro)
• cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
• hookWorkflow (tanstack-start)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

33 infra events

• cold-start-warmup · suite warmup (python) · at 21:14:09Z · abandoned wrun_41M1CTMMKH0GJHNMWTRT0RBSZQ · (+6 more)
• run-pickup-stall · sleepingWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB7
• run-pickup-stall · nullByteWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9860GSR1V009T2GZWY1
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9FH0GQ8P3X9DK42SSWT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB8
• run-pickup-stall · promiseAllWorkflow (python) · at 21:14:27Z · abandoned wrun_41M1CTR97V0GTFVK2A34P47G2P
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:59Z · abandoned wrun_41M1CTSAE60GTGN55WM85C4WRF
• run-pickup-stall · promiseAllWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FE0GZDVGGTYRREZFGE
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FC0GT13M6TSBCMD2CF
• run-pickup-stall · sleepingWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4GF0GMKRKA7ZPKGVQKA
• run-pickup-stall · nullByteWorkflow (python) · at 21:15:26Z · abandoned wrun_41M1CTT4TK0GZ59NA0E5DEECN9
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:16:07Z · abandoned wrun_41M1CTTA0V0GW8STRJ0NXH1MAK
• cold-start-warmup · suite warmup (tanstack-start) · at 21:16:29Z · abandoned wrun_01M1CTVQS8HB5PGN6S1W5JJ3GK
• cold-start-warmup · suite warmup (python) · at 21:17:08Z · abandoned wrun_01M1CTT3VDW1P4DARKPKNK56SV · (+7 more)
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZQS5VPNGWV6D9P0WTH
• run-pickup-stall · sleepingWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZXKFDZMW0BJ2PQSTKM
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZZEX0PC8YRTF8BCXJ9
• run-pickup-stall · promiseAllWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZR016BGCJPXEQCN0NT
• run-pickup-stall · nullByteWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXS01F6Q7CDA91SEW04AA
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:17:49Z · abandoned wrun_41M1CTW61K0GKDF638N6P0JFST
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:17:49Z · abandoned wrun_41M1CTX1XD0GZMJW4BJNHFQGRT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKSM4NA1F8PYJ58GY8E
• run-pickup-stall · nullByteWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKT6SQCSMMZVDCS78N1
• run-pickup-stall · sleepingWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKP72PTRG1PX6QAD913
• run-pickup-stall · promiseAllWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKRFTNJ0EFDTE3WRYPV
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKMGB6EP5KBB0VYFFTT
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:18:47Z · abandoned wrun_41M1CTYXY30GGKEAW3PNEB5333
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E77VC437FEHQ0P588M8
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7EHTPTCK35VG9GA5CF
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7JA10XMP3BF8G2B77F
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK3ASZB6STVRTJ0HZ44
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK43AY290VT6J6GVK68
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:20:24Z · abandoned wrun_01M1CV38TQG0ZDSCF4TG5BHPH3

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
❌ ▲ Vercel Production	3570	8	742	4320
✅ 💻 Local Development	3922	0	558	4480
✅ 📦 Local Production	3922	0	558	4480
✅ 🐘 Local Postgres	3922	0	558	4480
✅ 🪟 Windows	320	0	0	320
❌ 🌐 Cross-language Conformance	0	9	132	141
✅ vercel-http-transport	817	0	143	960
✅ vercel-multi-region	27	0	0	27
✅ vercel-ws-transport	553	0	87	640
Total	17053	17	2778	19848

Details by Category

❌ ▲ Vercel Production

App	Passed	Failed	Skipped
✅ astro-node	132	0	28
✅ astro-quickjs	132	0	28
✅ example-node	132	0	28
✅ example-quickjs	132	0	28
✅ express-node	132	0	28
✅ express-quickjs	132	0	28
✅ fastify-node	132	0	28
✅ fastify-quickjs	132	0	28
✅ hono-node	132	0	28
✅ hono-quickjs	132	0	28
✅ nest-node	132	0	28
✅ nest-quickjs	132	0	28
✅ nextjs-turbopack-node	157	0	3
✅ nextjs-turbopack-quickjs	157	0	3
✅ nextjs-webpack-node	157	0	3
✅ nextjs-webpack-quickjs	157	0	3
✅ nitro-node	132	0	28
✅ nitro-quickjs	132	0	28
✅ nuxt-node	132	0	28
✅ nuxt-quickjs	132	0	28
❌ python-node	0	8	152
✅ sveltekit-node	151	0	9
✅ sveltekit-quickjs	151	0	9
✅ tanstack-start-node	132	0	28
✅ tanstack-start-quickjs	132	0	28
✅ vite-node	132	0	28
✅ vite-quickjs	132	0	28

✅ 💻 Local Development

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 📦 Local Production

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🐘 Local Postgres

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	160	0	0
✅ nextjs-turbopack-quickjs	160	0	0

❌ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
❌ python	0	9	132

✅ vercel-http-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ hono	132	0	28
✅ nextjs-turbopack	157	0	3
✅ nitro	132	0	28
✅ vite	132	0	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ nextjs-turbopack	157	0	3
✅ vite	132	0	28

---

📋 View full workflow run

[github-actions]

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario	outcome	events	virt	replay	violations
✅ smoke-no-steps	completed	3	0ms	ok	0
✅ smoke-one-step	completed	6	0ms	ok	0
✅ hook-at-step-started	completed	12	0ms	ok	0
✅ hook-at-step-completed	completed	12	0ms	ok	0
✅ hook-at-hook-created	completed	12	0ms	ok	0
✅ deadline-hook-wins	completed	7	1.0h	ok	0
✅ deadline-expires	completed	7	1.0h	ok	0
✅ long-sleep	completed	11	30.0d	ok	0
✅ hook-never-arrives	stalled	3	0ms	skipped	0
✅ step-retries-twice	completed	10	2.0s	ok	0
✅ parallel-steps	completed	9	0ms	ok	0
✅ hook-on-execution-state	completed	12	0ms	ok	0
✅ peek-hook-before-branch	completed	12	0ms	ok	0
✅ peek-hook-after-branch	completed	12	0ms	ok	0
✅ peek-hook-at-registration	completed	12	0ms	ok	0
✅ race-hook-before-probe	completed	12	0ms	ok	0
✅ race-hook-after-probe	completed	12	0ms	ok	0
✅ race-duplicate-delivery	completed	13	0ms	ok	0
✅ attr-hook-before-step	completed	11	0ms	ok	0
✅ attr-hook-after-step	completed	11	0ms	ok	0
✅ attr-from-step-body	completed	13	0ms	ok	0
✅ fork-hook-after-timeout	completed	14	1.0m	ok	0
✅ fork-hook-before-timeout	completed	14	1.0m	ok	0
✅ count-hook-after-timeout	completed	17	1.0m	ok	0
✅ count-hook-before-timeout	completed	20	1.0m	ok	0
✅ stale-read-step-count-fork	completed	20	1.0m	ok	0
✅ stale-read-equal-step-counts	completed	14	1.0m	ok	0
✅ step-vs-step-fork	completed	12	0ms	ok	0
✅ step-vs-step-fork-fenced	completed	12	0ms	ok	0
✅ fence-catches-benign-direction	completed	12	5ms	ok	0
✅ in-flight-before-decision	completed	17	1.0m	ok	0
❌ in-flight-before-decision-counted	completed	17	1.0m	ok	0
✅ in-flight-after-decision	completed	19	2.0m	ok	0
✅ stale-read-step-count-fork-fenced	completed	20	1.0m	ok	0
✅ fork-hook-wins	completed	13	1.0m	ok	0
✅ fork-timeout-wins	completed	13	1.0m	ok	0
✅ unclaimed-payload-under-fork	completed	17	1.0m	ok	0
✅ claimed-payload-under-fork	completed	17	1.0m	ok	0
✅ writers-independent-step-bodies	completed	12	0ms	ok	0
✅ writers-scripted-tempo	completed	12	0ms	ok	0
✅ cancel-mid-step	cancelled	7	0ms	skipped	0

Full trace: world-sim.txt

[github-actions]

📊 Workflow Benchmarks

commit 9f03030 · Mon, 31 Aug 2026 21:38:22 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1386 (+657%) 🔻	1500 🔴 (+46%) 🔻	1543 🔴 (+43%) 🔻	1646 🔴 (+36%) 🔻	30
TTFS	stream	250 (+16%) 🔻	1454 🔴 (+35%) 🔻	1519 🔴 (+37%) 🔻	1897 🔴 (+26%) 🔻	30
TTFS	hook + stream	1709 (+41%) 🔻	1804 🔴 (+33%) 🔻	1877 🔴 (+34%) 🔻	1976 🔴 (+37%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	612 (-4.1%)	1255 (-27%) 💚	2513 (+39%) 🔻	6361 (+252%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2114 (+18%) 🔻	6996 (+132%) 🔻	11047 (+169%) 🔻	11298 (+165%) 🔻	10
STSO	1020 steps (inline)	108 (+4.9%)	166 (+21%) 🔻	185 (+17%) 🔻	267 (+5.5%)	1019
WO	1020 steps	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	1
CRTT	first chunk (pooled)	123 (+24%) 🔻	165 (-11%)	194 (-18%) 💚	284 (+8.8%)	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	151 (+12%)	195 (+8%)	270 (-5%)	495 (+14%)	146 (-20%)	10
size sweep (100/s, 160B-12KB)	135 (-7%)	180 (-12%)	354 (-19%)	632 (-34%)	128 (-39%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	150 (+13%)	151 (-14%)	179 (-35%)	454 (-51%)	170 (-64%)	3
replay eve-gpt-5.6-sol-2000t (1x)	159 (-3%)	155 (-20%)	199 (-33%)	304 (-81%)	199 (-83%)	2
replay eve-gpt-5.6-sol-2000t (2x)	147 (+4%)	206 (-29%)	274 (-37%)	418 (-47%)	229 (-57%)	3

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 136556ms → this run 162927ms (Δ +26371ms, +19%)

100-150 ms ███████████┃████████████ main 863 this 433 -430
150-200 ms ████░░░░░░░░░░┃ main 130 this 530 +400
200-250 ms ┃ main 15 this 41 +26
250-300 ms ┃ main 8 this 7 -1
300-350 ms ┃ main 2 this 4 +2
350-400 ms ┃ main 1 this 1 +0
400-450 ms ┃ main 0 this 1 +1
500-550 ms ┃ main 0 this 2 +2

📈 CRTT drill-down vs main (RTT distributions & profiles)

variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▂▁··· 152.3 (-3%) 141 (+1%) 270 (-5%) 495 (+14%) 3000
sweep ······▂█▂▁··· 150.4 (-10%) 135 (+2%) 354 (-19%) 632 (-34%) 3000
gw 1x ·····▁▂█▁▁··· 131.2 (-13%) 126 (-2%) 179 (-35%) 454 (-51%) 5295
eve 1x ······▃█▁···· 134.4 (-28%) 126 (-8%) 199 (-33%) 304 (-81%) 5186
eve 2x ·····▁▁█▃···· 171.3 (-18%) 161 (-4%) 274 (-37%) 418 (-47%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control █▆▇▆▅▁▃▁▆▁ 133–172ms
sweep █▇▃▁▅▅▂▂▂▁ 128–189ms
gw 1x ▅█▃▃▁▄▂▃▁▃ 122–150ms
eve 1x ▆▂▄▃▃▃█▆▃▁ 119–156ms
eve 2x ▂▁▃▂▁▅▅█▅▃ 142–220ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep ▇▆█▆▅▁▄ 148–152ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▁▆▅▆▃▄▅▄█▄ 35–57ms
sweep ▁▁▃▅█▄▅▃▁▄ 50–74ms
gw 1x ▄█▁▁▄▅▅▆▁▂ 38–47ms
eve 1x ▇▅▃▇▇▄█▃▃▁ 21–28ms
eve 2x ▇█▃▂█▅▅▁█▃ 24–31ms

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

[VaguelySerious]

AI review: blocking issues found

[github-actions]

Framework	Flow route	Step reg.	Framework output
hono	200.9 KiB (±0)	40.4 KiB (±0)	1.76 MiB (-45 B)
nextjs-turbopack	206.3 KiB (±0)	439 B (±0)	765.5 KiB (+338 B)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

9f03030 · run

[alangenfeld]

local agent review surfaced:

The new speculative key lookup always calls:

resolveRunEncryptionKey(world,runId)

This discards the WorkflowRun.deploymentId that the previous implementation supplied by calling getEncryptionKeyForRun(workflowRun).

That breaks encrypted Node replays outside a Vercel Function—such as local tooling or external runners—because world-vercel cannot resolve a key from a bare run ID without { deploymentId }; it returns undefined. The replay then attempts to hydrate encrypted input/events without the key. It can also select the wrong key when deployment routing matters.

The lookup should start from the streamed run_created/runInput deployment context, for example:

resolveRunEncryptionKey(world,runId,{deploymentId: workflow.deploymentId,})

or defer until the authoritative WorkflowRun is available and pass that object.

[github-actions]

No backport to stable for e9d5c56 (AI decision).

This is a replay-startup performance optimization: it prepares payloads as event frames stream in, lazily resolves the encryption key, skips Node-only work for QuickJS, and adds a new replayEventObserver option to the CreateEventParams public World interface. None of it fixes a user-visible defect on stable — the incidental hardening (setup failures classified as run_failed, unhandled-rejection guards, removal of the primitive memoization size cap) exists to support the new streaming path rather than to repair existing behavior, and it touches the hot replay path plus a World API surface, which is exactly the kind of churn a maintenance line should not take.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e9d5c56701821b090108a85b74bf8b0cbef8ea8e

[NathanColosimo]

Summary

• prepare validated Node replay payloads as streamed events arrive
• lazily resolve the invocation encryption key and create the payload cache only after the run is known to use Node
• skip host-side compile, key lookup, replay cache, and payload preparation for QuickJS
• start run_started before speculative compile/cache work so setup I/O overlaps Node startup
• correct speculative workflow names from persisted run_created
• classify invalid VM configuration as run_failed in turbo, ordinary, and streamed setup paths
• count materialized replay events without per-event spans
• share one replay-run reconstruction path between run_started and atomic-hook preload
• leave primitive payload caching uncapped

Prewarm deliberately scans the event array on each replay and relies on the existing event-ID map to skip prepared payloads. This avoids positional scan state, reset hooks, and event-log replacement wrappers; only the cheap array walk repeats. Prepared plaintext and memoized primitive results remain invocation-scoped.

Stack

Depends on #3798.

Validation

• focused regression suites — 131 tests passed
• final runtime regression suites — 60 tests passed
• full @workflow/core suite — 2,295 tests passed
• full @workflow/world-vercel suite — 572 tests passed
• full @workflow/world suite — 165 tests passed
• pnpm --filter @workflow/core typecheck
• pnpm --filter @workflow/world-vercel typecheck
• Biome check on changed files
• git diff --check

[changeset-bot]

🦋 Changeset detected

Latest commit: 9f03030

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages

Name	Type
@workflow/core	Patch
@workflow/world	Patch
@workflow/world-vercel	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/world-local	Patch
@workflow/world-postgres	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-nextjs-workflow-webpack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-astro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-express-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-fastify-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-hono-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nestjs-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nitro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nuxt-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-python-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-sveltekit-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-tanstack-start-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-vite-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-docs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-swc-playground	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-tarballs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-web	Ready	Preview, v0	Aug 31, 2026 9:15pm

[github-actions]

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (8 failed)

python-node (8 failed):

• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW | 🔍 observability
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ | 🔍 observability
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z | 🔍 observability
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD | 🔍 observability
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95 | 🔍 observability
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D | 🔍 observability
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H | 🔍 observability
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3 | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

• deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M1CV028S4VSJGT08W5VNCKF3
• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

• abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-webpack)
• addTenWorkflow (nitro)
• cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
• hookWorkflow (tanstack-start)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

33 infra events

• cold-start-warmup · suite warmup (python) · at 21:14:09Z · abandoned wrun_41M1CTMMKH0GJHNMWTRT0RBSZQ · (+6 more)
• run-pickup-stall · sleepingWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB7
• run-pickup-stall · nullByteWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9860GSR1V009T2GZWY1
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9FH0GQ8P3X9DK42SSWT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB8
• run-pickup-stall · promiseAllWorkflow (python) · at 21:14:27Z · abandoned wrun_41M1CTR97V0GTFVK2A34P47G2P
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:59Z · abandoned wrun_41M1CTSAE60GTGN55WM85C4WRF
• run-pickup-stall · promiseAllWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FE0GZDVGGTYRREZFGE
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FC0GT13M6TSBCMD2CF
• run-pickup-stall · sleepingWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4GF0GMKRKA7ZPKGVQKA
• run-pickup-stall · nullByteWorkflow (python) · at 21:15:26Z · abandoned wrun_41M1CTT4TK0GZ59NA0E5DEECN9
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:16:07Z · abandoned wrun_41M1CTTA0V0GW8STRJ0NXH1MAK
• cold-start-warmup · suite warmup (tanstack-start) · at 21:16:29Z · abandoned wrun_01M1CTVQS8HB5PGN6S1W5JJ3GK
• cold-start-warmup · suite warmup (python) · at 21:17:08Z · abandoned wrun_01M1CTT3VDW1P4DARKPKNK56SV · (+7 more)
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZQS5VPNGWV6D9P0WTH
• run-pickup-stall · sleepingWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZXKFDZMW0BJ2PQSTKM
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZZEX0PC8YRTF8BCXJ9
• run-pickup-stall · promiseAllWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZR016BGCJPXEQCN0NT
• run-pickup-stall · nullByteWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXS01F6Q7CDA91SEW04AA
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:17:49Z · abandoned wrun_41M1CTW61K0GKDF638N6P0JFST
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:17:49Z · abandoned wrun_41M1CTX1XD0GZMJW4BJNHFQGRT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKSM4NA1F8PYJ58GY8E
• run-pickup-stall · nullByteWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKT6SQCSMMZVDCS78N1
• run-pickup-stall · sleepingWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKP72PTRG1PX6QAD913
• run-pickup-stall · promiseAllWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKRFTNJ0EFDTE3WRYPV
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKMGB6EP5KBB0VYFFTT
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:18:47Z · abandoned wrun_41M1CTYXY30GGKEAW3PNEB5333
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E77VC437FEHQ0P588M8
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7EHTPTCK35VG9GA5CF
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7JA10XMP3BF8G2B77F
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK3ASZB6STVRTJ0HZ44
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK43AY290VT6J6GVK68
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:20:24Z · abandoned wrun_01M1CV38TQG0ZDSCF4TG5BHPH3

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
❌ ▲ Vercel Production	3570	8	742	4320
✅ 💻 Local Development	3922	0	558	4480
✅ 📦 Local Production	3922	0	558	4480
✅ 🐘 Local Postgres	3922	0	558	4480
✅ 🪟 Windows	320	0	0	320
❌ 🌐 Cross-language Conformance	0	9	132	141
✅ vercel-http-transport	817	0	143	960
✅ vercel-multi-region	27	0	0	27
✅ vercel-ws-transport	553	0	87	640
Total	17053	17	2778	19848

Details by Category

❌ ▲ Vercel Production

App	Passed	Failed	Skipped
✅ astro-node	132	0	28
✅ astro-quickjs	132	0	28
✅ example-node	132	0	28
✅ example-quickjs	132	0	28
✅ express-node	132	0	28
✅ express-quickjs	132	0	28
✅ fastify-node	132	0	28
✅ fastify-quickjs	132	0	28
✅ hono-node	132	0	28
✅ hono-quickjs	132	0	28
✅ nest-node	132	0	28
✅ nest-quickjs	132	0	28
✅ nextjs-turbopack-node	157	0	3
✅ nextjs-turbopack-quickjs	157	0	3
✅ nextjs-webpack-node	157	0	3
✅ nextjs-webpack-quickjs	157	0	3
✅ nitro-node	132	0	28
✅ nitro-quickjs	132	0	28
✅ nuxt-node	132	0	28
✅ nuxt-quickjs	132	0	28
❌ python-node	0	8	152
✅ sveltekit-node	151	0	9
✅ sveltekit-quickjs	151	0	9
✅ tanstack-start-node	132	0	28
✅ tanstack-start-quickjs	132	0	28
✅ vite-node	132	0	28
✅ vite-quickjs	132	0	28

✅ 💻 Local Development

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 📦 Local Production

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🐘 Local Postgres

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	160	0	0
✅ nextjs-turbopack-quickjs	160	0	0

❌ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
❌ python	0	9	132

✅ vercel-http-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ hono	132	0	28
✅ nextjs-turbopack	157	0	3
✅ nitro	132	0	28
✅ vite	132	0	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ nextjs-turbopack	157	0	3
✅ vite	132	0	28

---

📋 View full workflow run

[github-actions]

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario	outcome	events	virt	replay	violations
✅ smoke-no-steps	completed	3	0ms	ok	0
✅ smoke-one-step	completed	6	0ms	ok	0
✅ hook-at-step-started	completed	12	0ms	ok	0
✅ hook-at-step-completed	completed	12	0ms	ok	0
✅ hook-at-hook-created	completed	12	0ms	ok	0
✅ deadline-hook-wins	completed	7	1.0h	ok	0
✅ deadline-expires	completed	7	1.0h	ok	0
✅ long-sleep	completed	11	30.0d	ok	0
✅ hook-never-arrives	stalled	3	0ms	skipped	0
✅ step-retries-twice	completed	10	2.0s	ok	0
✅ parallel-steps	completed	9	0ms	ok	0
✅ hook-on-execution-state	completed	12	0ms	ok	0
✅ peek-hook-before-branch	completed	12	0ms	ok	0
✅ peek-hook-after-branch	completed	12	0ms	ok	0
✅ peek-hook-at-registration	completed	12	0ms	ok	0
✅ race-hook-before-probe	completed	12	0ms	ok	0
✅ race-hook-after-probe	completed	12	0ms	ok	0
✅ race-duplicate-delivery	completed	13	0ms	ok	0
✅ attr-hook-before-step	completed	11	0ms	ok	0
✅ attr-hook-after-step	completed	11	0ms	ok	0
✅ attr-from-step-body	completed	13	0ms	ok	0
✅ fork-hook-after-timeout	completed	14	1.0m	ok	0
✅ fork-hook-before-timeout	completed	14	1.0m	ok	0
✅ count-hook-after-timeout	completed	17	1.0m	ok	0
✅ count-hook-before-timeout	completed	20	1.0m	ok	0
✅ stale-read-step-count-fork	completed	20	1.0m	ok	0
✅ stale-read-equal-step-counts	completed	14	1.0m	ok	0
✅ step-vs-step-fork	completed	12	0ms	ok	0
✅ step-vs-step-fork-fenced	completed	12	0ms	ok	0
✅ fence-catches-benign-direction	completed	12	5ms	ok	0
✅ in-flight-before-decision	completed	17	1.0m	ok	0
❌ in-flight-before-decision-counted	completed	17	1.0m	ok	0
✅ in-flight-after-decision	completed	19	2.0m	ok	0
✅ stale-read-step-count-fork-fenced	completed	20	1.0m	ok	0
✅ fork-hook-wins	completed	13	1.0m	ok	0
✅ fork-timeout-wins	completed	13	1.0m	ok	0
✅ unclaimed-payload-under-fork	completed	17	1.0m	ok	0
✅ claimed-payload-under-fork	completed	17	1.0m	ok	0
✅ writers-independent-step-bodies	completed	12	0ms	ok	0
✅ writers-scripted-tempo	completed	12	0ms	ok	0
✅ cancel-mid-step	cancelled	7	0ms	skipped	0

Full trace: world-sim.txt

[github-actions]

📊 Workflow Benchmarks

commit 9f03030 · Mon, 31 Aug 2026 21:38:22 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1386 (+657%) 🔻	1500 🔴 (+46%) 🔻	1543 🔴 (+43%) 🔻	1646 🔴 (+36%) 🔻	30
TTFS	stream	250 (+16%) 🔻	1454 🔴 (+35%) 🔻	1519 🔴 (+37%) 🔻	1897 🔴 (+26%) 🔻	30
TTFS	hook + stream	1709 (+41%) 🔻	1804 🔴 (+33%) 🔻	1877 🔴 (+34%) 🔻	1976 🔴 (+37%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	612 (-4.1%)	1255 (-27%) 💚	2513 (+39%) 🔻	6361 (+252%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2114 (+18%) 🔻	6996 (+132%) 🔻	11047 (+169%) 🔻	11298 (+165%) 🔻	10
STSO	1020 steps (inline)	108 (+4.9%)	166 (+21%) 🔻	185 (+17%) 🔻	267 (+5.5%)	1019
WO	1020 steps	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	1
CRTT	first chunk (pooled)	123 (+24%) 🔻	165 (-11%)	194 (-18%) 💚	284 (+8.8%)	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	151 (+12%)	195 (+8%)	270 (-5%)	495 (+14%)	146 (-20%)	10
size sweep (100/s, 160B-12KB)	135 (-7%)	180 (-12%)	354 (-19%)	632 (-34%)	128 (-39%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	150 (+13%)	151 (-14%)	179 (-35%)	454 (-51%)	170 (-64%)	3
replay eve-gpt-5.6-sol-2000t (1x)	159 (-3%)	155 (-20%)	199 (-33%)	304 (-81%)	199 (-83%)	2
replay eve-gpt-5.6-sol-2000t (2x)	147 (+4%)	206 (-29%)	274 (-37%)	418 (-47%)	229 (-57%)	3

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 136556ms → this run 162927ms (Δ +26371ms, +19%)

100-150 ms ███████████┃████████████ main 863 this 433 -430
150-200 ms ████░░░░░░░░░░┃ main 130 this 530 +400
200-250 ms ┃ main 15 this 41 +26
250-300 ms ┃ main 8 this 7 -1
300-350 ms ┃ main 2 this 4 +2
350-400 ms ┃ main 1 this 1 +0
400-450 ms ┃ main 0 this 1 +1
500-550 ms ┃ main 0 this 2 +2

📈 CRTT drill-down vs main (RTT distributions & profiles)

variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▂▁··· 152.3 (-3%) 141 (+1%) 270 (-5%) 495 (+14%) 3000
sweep ······▂█▂▁··· 150.4 (-10%) 135 (+2%) 354 (-19%) 632 (-34%) 3000
gw 1x ·····▁▂█▁▁··· 131.2 (-13%) 126 (-2%) 179 (-35%) 454 (-51%) 5295
eve 1x ······▃█▁···· 134.4 (-28%) 126 (-8%) 199 (-33%) 304 (-81%) 5186
eve 2x ·····▁▁█▃···· 171.3 (-18%) 161 (-4%) 274 (-37%) 418 (-47%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control █▆▇▆▅▁▃▁▆▁ 133–172ms
sweep █▇▃▁▅▅▂▂▂▁ 128–189ms
gw 1x ▅█▃▃▁▄▂▃▁▃ 122–150ms
eve 1x ▆▂▄▃▃▃█▆▃▁ 119–156ms
eve 2x ▂▁▃▂▁▅▅█▅▃ 142–220ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep ▇▆█▆▅▁▄ 148–152ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▁▆▅▆▃▄▅▄█▄ 35–57ms
sweep ▁▁▃▅█▄▅▃▁▄ 50–74ms
gw 1x ▄█▁▁▄▅▅▆▁▂ 38–47ms
eve 1x ▇▅▃▇▇▄█▃▃▁ 21–28ms
eve 2x ▇█▃▂█▅▅▁█▃ 24–31ms

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

[VaguelySerious]

AI review: blocking issues found

[github-actions]

Framework	Flow route	Step reg.	Framework output
hono	200.9 KiB (±0)	40.4 KiB (±0)	1.76 MiB (-45 B)
nextjs-turbopack	206.3 KiB (±0)	439 B (±0)	765.5 KiB (+338 B)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

9f03030 · run

[alangenfeld]

local agent review surfaced:

The new speculative key lookup always calls:

resolveRunEncryptionKey(world,runId)

This discards the WorkflowRun.deploymentId that the previous implementation supplied by calling getEncryptionKeyForRun(workflowRun).

That breaks encrypted Node replays outside a Vercel Function—such as local tooling or external runners—because world-vercel cannot resolve a key from a bare run ID without { deploymentId }; it returns undefined. The replay then attempts to hydrate encrypted input/events without the key. It can also select the wrong key when deployment routing matters.

The lookup should start from the streamed run_created/runInput deployment context, for example:

resolveRunEncryptionKey(world,runId,{deploymentId: workflow.deploymentId,})

or defer until the authoritative WorkflowRun is available and pass that object.

[github-actions]

No backport to stable for e9d5c56 (AI decision).

This is a replay-startup performance optimization: it prepares payloads as event frames stream in, lazily resolves the encryption key, skips Node-only work for QuickJS, and adds a new replayEventObserver option to the CreateEventParams public World interface. None of it fixes a user-visible defect on stable — the incidental hardening (setup failures classified as run_failed, unhandled-rejection guards, removal of the primitive memoization size cap) exists to support the new streaming path rather than to repair existing behavior, and it touches the hot replay path plus a World API surface, which is exactly the kind of churn a maintenance line should not take.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e9d5c56701821b090108a85b74bf8b0cbef8ea8e

[NathanColosimo]

Summary

• prepare validated Node replay payloads as streamed events arrive
• lazily resolve the invocation encryption key and create the payload cache only after the run is known to use Node
• skip host-side compile, key lookup, replay cache, and payload preparation for QuickJS
• start run_started before speculative compile/cache work so setup I/O overlaps Node startup
• correct speculative workflow names from persisted run_created
• classify invalid VM configuration as run_failed in turbo, ordinary, and streamed setup paths
• count materialized replay events without per-event spans
• share one replay-run reconstruction path between run_started and atomic-hook preload
• leave primitive payload caching uncapped

Prewarm deliberately scans the event array on each replay and relies on the existing event-ID map to skip prepared payloads. This avoids positional scan state, reset hooks, and event-log replacement wrappers; only the cheap array walk repeats. Prepared plaintext and memoized primitive results remain invocation-scoped.

Stack

Depends on #3798.

Validation

• focused regression suites — 131 tests passed
• final runtime regression suites — 60 tests passed
• full @workflow/core suite — 2,295 tests passed
• full @workflow/world-vercel suite — 572 tests passed
• full @workflow/world suite — 165 tests passed
• pnpm --filter @workflow/core typecheck
• pnpm --filter @workflow/world-vercel typecheck
• Biome check on changed files
• git diff --check

[changeset-bot]

🦋 Changeset detected

Latest commit: 9f03030

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages

Name	Type
@workflow/core	Patch
@workflow/world	Patch
@workflow/world-vercel	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/world-local	Patch
@workflow/world-postgres	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-nextjs-workflow-webpack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-astro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-express-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-fastify-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-hono-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nestjs-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nitro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nuxt-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-python-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-sveltekit-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-tanstack-start-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-vite-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-docs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-swc-playground	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-tarballs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-web	Ready	Preview, v0	Aug 31, 2026 9:15pm

[github-actions]

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (8 failed)

python-node (8 failed):

• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW | 🔍 observability
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ | 🔍 observability
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z | 🔍 observability
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD | 🔍 observability
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95 | 🔍 observability
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D | 🔍 observability
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H | 🔍 observability
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3 | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

• deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M1CV028S4VSJGT08W5VNCKF3
• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

• abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-webpack)
• addTenWorkflow (nitro)
• cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
• hookWorkflow (tanstack-start)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

33 infra events

• cold-start-warmup · suite warmup (python) · at 21:14:09Z · abandoned wrun_41M1CTMMKH0GJHNMWTRT0RBSZQ · (+6 more)
• run-pickup-stall · sleepingWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB7
• run-pickup-stall · nullByteWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9860GSR1V009T2GZWY1
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9FH0GQ8P3X9DK42SSWT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB8
• run-pickup-stall · promiseAllWorkflow (python) · at 21:14:27Z · abandoned wrun_41M1CTR97V0GTFVK2A34P47G2P
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:59Z · abandoned wrun_41M1CTSAE60GTGN55WM85C4WRF
• run-pickup-stall · promiseAllWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FE0GZDVGGTYRREZFGE
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FC0GT13M6TSBCMD2CF
• run-pickup-stall · sleepingWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4GF0GMKRKA7ZPKGVQKA
• run-pickup-stall · nullByteWorkflow (python) · at 21:15:26Z · abandoned wrun_41M1CTT4TK0GZ59NA0E5DEECN9
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:16:07Z · abandoned wrun_41M1CTTA0V0GW8STRJ0NXH1MAK
• cold-start-warmup · suite warmup (tanstack-start) · at 21:16:29Z · abandoned wrun_01M1CTVQS8HB5PGN6S1W5JJ3GK
• cold-start-warmup · suite warmup (python) · at 21:17:08Z · abandoned wrun_01M1CTT3VDW1P4DARKPKNK56SV · (+7 more)
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZQS5VPNGWV6D9P0WTH
• run-pickup-stall · sleepingWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZXKFDZMW0BJ2PQSTKM
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZZEX0PC8YRTF8BCXJ9
• run-pickup-stall · promiseAllWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZR016BGCJPXEQCN0NT
• run-pickup-stall · nullByteWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXS01F6Q7CDA91SEW04AA
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:17:49Z · abandoned wrun_41M1CTW61K0GKDF638N6P0JFST
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:17:49Z · abandoned wrun_41M1CTX1XD0GZMJW4BJNHFQGRT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKSM4NA1F8PYJ58GY8E
• run-pickup-stall · nullByteWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKT6SQCSMMZVDCS78N1
• run-pickup-stall · sleepingWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKP72PTRG1PX6QAD913
• run-pickup-stall · promiseAllWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKRFTNJ0EFDTE3WRYPV
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKMGB6EP5KBB0VYFFTT
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:18:47Z · abandoned wrun_41M1CTYXY30GGKEAW3PNEB5333
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E77VC437FEHQ0P588M8
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7EHTPTCK35VG9GA5CF
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7JA10XMP3BF8G2B77F
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK3ASZB6STVRTJ0HZ44
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK43AY290VT6J6GVK68
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:20:24Z · abandoned wrun_01M1CV38TQG0ZDSCF4TG5BHPH3

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
❌ ▲ Vercel Production	3570	8	742	4320
✅ 💻 Local Development	3922	0	558	4480
✅ 📦 Local Production	3922	0	558	4480
✅ 🐘 Local Postgres	3922	0	558	4480
✅ 🪟 Windows	320	0	0	320
❌ 🌐 Cross-language Conformance	0	9	132	141
✅ vercel-http-transport	817	0	143	960
✅ vercel-multi-region	27	0	0	27
✅ vercel-ws-transport	553	0	87	640
Total	17053	17	2778	19848

Details by Category

❌ ▲ Vercel Production

App	Passed	Failed	Skipped
✅ astro-node	132	0	28
✅ astro-quickjs	132	0	28
✅ example-node	132	0	28
✅ example-quickjs	132	0	28
✅ express-node	132	0	28
✅ express-quickjs	132	0	28
✅ fastify-node	132	0	28
✅ fastify-quickjs	132	0	28
✅ hono-node	132	0	28
✅ hono-quickjs	132	0	28
✅ nest-node	132	0	28
✅ nest-quickjs	132	0	28
✅ nextjs-turbopack-node	157	0	3
✅ nextjs-turbopack-quickjs	157	0	3
✅ nextjs-webpack-node	157	0	3
✅ nextjs-webpack-quickjs	157	0	3
✅ nitro-node	132	0	28
✅ nitro-quickjs	132	0	28
✅ nuxt-node	132	0	28
✅ nuxt-quickjs	132	0	28
❌ python-node	0	8	152
✅ sveltekit-node	151	0	9
✅ sveltekit-quickjs	151	0	9
✅ tanstack-start-node	132	0	28
✅ tanstack-start-quickjs	132	0	28
✅ vite-node	132	0	28
✅ vite-quickjs	132	0	28

✅ 💻 Local Development

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 📦 Local Production

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🐘 Local Postgres

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	160	0	0
✅ nextjs-turbopack-quickjs	160	0	0

❌ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
❌ python	0	9	132

✅ vercel-http-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ hono	132	0	28
✅ nextjs-turbopack	157	0	3
✅ nitro	132	0	28
✅ vite	132	0	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ nextjs-turbopack	157	0	3
✅ vite	132	0	28

---

📋 View full workflow run

[github-actions]

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario	outcome	events	virt	replay	violations
✅ smoke-no-steps	completed	3	0ms	ok	0
✅ smoke-one-step	completed	6	0ms	ok	0
✅ hook-at-step-started	completed	12	0ms	ok	0
✅ hook-at-step-completed	completed	12	0ms	ok	0
✅ hook-at-hook-created	completed	12	0ms	ok	0
✅ deadline-hook-wins	completed	7	1.0h	ok	0
✅ deadline-expires	completed	7	1.0h	ok	0
✅ long-sleep	completed	11	30.0d	ok	0
✅ hook-never-arrives	stalled	3	0ms	skipped	0
✅ step-retries-twice	completed	10	2.0s	ok	0
✅ parallel-steps	completed	9	0ms	ok	0
✅ hook-on-execution-state	completed	12	0ms	ok	0
✅ peek-hook-before-branch	completed	12	0ms	ok	0
✅ peek-hook-after-branch	completed	12	0ms	ok	0
✅ peek-hook-at-registration	completed	12	0ms	ok	0
✅ race-hook-before-probe	completed	12	0ms	ok	0
✅ race-hook-after-probe	completed	12	0ms	ok	0
✅ race-duplicate-delivery	completed	13	0ms	ok	0
✅ attr-hook-before-step	completed	11	0ms	ok	0
✅ attr-hook-after-step	completed	11	0ms	ok	0
✅ attr-from-step-body	completed	13	0ms	ok	0
✅ fork-hook-after-timeout	completed	14	1.0m	ok	0
✅ fork-hook-before-timeout	completed	14	1.0m	ok	0
✅ count-hook-after-timeout	completed	17	1.0m	ok	0
✅ count-hook-before-timeout	completed	20	1.0m	ok	0
✅ stale-read-step-count-fork	completed	20	1.0m	ok	0
✅ stale-read-equal-step-counts	completed	14	1.0m	ok	0
✅ step-vs-step-fork	completed	12	0ms	ok	0
✅ step-vs-step-fork-fenced	completed	12	0ms	ok	0
✅ fence-catches-benign-direction	completed	12	5ms	ok	0
✅ in-flight-before-decision	completed	17	1.0m	ok	0
❌ in-flight-before-decision-counted	completed	17	1.0m	ok	0
✅ in-flight-after-decision	completed	19	2.0m	ok	0
✅ stale-read-step-count-fork-fenced	completed	20	1.0m	ok	0
✅ fork-hook-wins	completed	13	1.0m	ok	0
✅ fork-timeout-wins	completed	13	1.0m	ok	0
✅ unclaimed-payload-under-fork	completed	17	1.0m	ok	0
✅ claimed-payload-under-fork	completed	17	1.0m	ok	0
✅ writers-independent-step-bodies	completed	12	0ms	ok	0
✅ writers-scripted-tempo	completed	12	0ms	ok	0
✅ cancel-mid-step	cancelled	7	0ms	skipped	0

Full trace: world-sim.txt

[github-actions]

📊 Workflow Benchmarks

commit 9f03030 · Mon, 31 Aug 2026 21:38:22 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1386 (+657%) 🔻	1500 🔴 (+46%) 🔻	1543 🔴 (+43%) 🔻	1646 🔴 (+36%) 🔻	30
TTFS	stream	250 (+16%) 🔻	1454 🔴 (+35%) 🔻	1519 🔴 (+37%) 🔻	1897 🔴 (+26%) 🔻	30
TTFS	hook + stream	1709 (+41%) 🔻	1804 🔴 (+33%) 🔻	1877 🔴 (+34%) 🔻	1976 🔴 (+37%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	612 (-4.1%)	1255 (-27%) 💚	2513 (+39%) 🔻	6361 (+252%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2114 (+18%) 🔻	6996 (+132%) 🔻	11047 (+169%) 🔻	11298 (+165%) 🔻	10
STSO	1020 steps (inline)	108 (+4.9%)	166 (+21%) 🔻	185 (+17%) 🔻	267 (+5.5%)	1019
WO	1020 steps	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	1
CRTT	first chunk (pooled)	123 (+24%) 🔻	165 (-11%)	194 (-18%) 💚	284 (+8.8%)	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	151 (+12%)	195 (+8%)	270 (-5%)	495 (+14%)	146 (-20%)	10
size sweep (100/s, 160B-12KB)	135 (-7%)	180 (-12%)	354 (-19%)	632 (-34%)	128 (-39%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	150 (+13%)	151 (-14%)	179 (-35%)	454 (-51%)	170 (-64%)	3
replay eve-gpt-5.6-sol-2000t (1x)	159 (-3%)	155 (-20%)	199 (-33%)	304 (-81%)	199 (-83%)	2
replay eve-gpt-5.6-sol-2000t (2x)	147 (+4%)	206 (-29%)	274 (-37%)	418 (-47%)	229 (-57%)	3

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 136556ms → this run 162927ms (Δ +26371ms, +19%)

100-150 ms ███████████┃████████████ main 863 this 433 -430
150-200 ms ████░░░░░░░░░░┃ main 130 this 530 +400
200-250 ms ┃ main 15 this 41 +26
250-300 ms ┃ main 8 this 7 -1
300-350 ms ┃ main 2 this 4 +2
350-400 ms ┃ main 1 this 1 +0
400-450 ms ┃ main 0 this 1 +1
500-550 ms ┃ main 0 this 2 +2

📈 CRTT drill-down vs main (RTT distributions & profiles)

variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▂▁··· 152.3 (-3%) 141 (+1%) 270 (-5%) 495 (+14%) 3000
sweep ······▂█▂▁··· 150.4 (-10%) 135 (+2%) 354 (-19%) 632 (-34%) 3000
gw 1x ·····▁▂█▁▁··· 131.2 (-13%) 126 (-2%) 179 (-35%) 454 (-51%) 5295
eve 1x ······▃█▁···· 134.4 (-28%) 126 (-8%) 199 (-33%) 304 (-81%) 5186
eve 2x ·····▁▁█▃···· 171.3 (-18%) 161 (-4%) 274 (-37%) 418 (-47%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control █▆▇▆▅▁▃▁▆▁ 133–172ms
sweep █▇▃▁▅▅▂▂▂▁ 128–189ms
gw 1x ▅█▃▃▁▄▂▃▁▃ 122–150ms
eve 1x ▆▂▄▃▃▃█▆▃▁ 119–156ms
eve 2x ▂▁▃▂▁▅▅█▅▃ 142–220ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep ▇▆█▆▅▁▄ 148–152ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▁▆▅▆▃▄▅▄█▄ 35–57ms
sweep ▁▁▃▅█▄▅▃▁▄ 50–74ms
gw 1x ▄█▁▁▄▅▅▆▁▂ 38–47ms
eve 1x ▇▅▃▇▇▄█▃▃▁ 21–28ms
eve 2x ▇█▃▂█▅▅▁█▃ 24–31ms

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

[VaguelySerious]

AI review: blocking issues found

[github-actions]

Framework	Flow route	Step reg.	Framework output
hono	200.9 KiB (±0)	40.4 KiB (±0)	1.76 MiB (-45 B)
nextjs-turbopack	206.3 KiB (±0)	439 B (±0)	765.5 KiB (+338 B)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

9f03030 · run

[alangenfeld]

local agent review surfaced:

The new speculative key lookup always calls:

resolveRunEncryptionKey(world,runId)

This discards the WorkflowRun.deploymentId that the previous implementation supplied by calling getEncryptionKeyForRun(workflowRun).

That breaks encrypted Node replays outside a Vercel Function—such as local tooling or external runners—because world-vercel cannot resolve a key from a bare run ID without { deploymentId }; it returns undefined. The replay then attempts to hydrate encrypted input/events without the key. It can also select the wrong key when deployment routing matters.

The lookup should start from the streamed run_created/runInput deployment context, for example:

resolveRunEncryptionKey(world,runId,{deploymentId: workflow.deploymentId,})

or defer until the authoritative WorkflowRun is available and pass that object.

[github-actions]

No backport to stable for e9d5c56 (AI decision).

This is a replay-startup performance optimization: it prepares payloads as event frames stream in, lazily resolves the encryption key, skips Node-only work for QuickJS, and adds a new replayEventObserver option to the CreateEventParams public World interface. None of it fixes a user-visible defect on stable — the incidental hardening (setup failures classified as run_failed, unhandled-rejection guards, removal of the primitive memoization size cap) exists to support the new streaming path rather than to repair existing behavior, and it touches the hot replay path plus a World API surface, which is exactly the kind of churn a maintenance line should not take.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e9d5c56701821b090108a85b74bf8b0cbef8ea8e

[NathanColosimo]

Summary

• prepare validated Node replay payloads as streamed events arrive
• lazily resolve the invocation encryption key and create the payload cache only after the run is known to use Node
• skip host-side compile, key lookup, replay cache, and payload preparation for QuickJS
• start run_started before speculative compile/cache work so setup I/O overlaps Node startup
• correct speculative workflow names from persisted run_created
• classify invalid VM configuration as run_failed in turbo, ordinary, and streamed setup paths
• count materialized replay events without per-event spans
• share one replay-run reconstruction path between run_started and atomic-hook preload
• leave primitive payload caching uncapped

Prewarm deliberately scans the event array on each replay and relies on the existing event-ID map to skip prepared payloads. This avoids positional scan state, reset hooks, and event-log replacement wrappers; only the cheap array walk repeats. Prepared plaintext and memoized primitive results remain invocation-scoped.

Stack

Depends on #3798.

Validation

• focused regression suites — 131 tests passed
• final runtime regression suites — 60 tests passed
• full @workflow/core suite — 2,295 tests passed
• full @workflow/world-vercel suite — 572 tests passed
• full @workflow/world suite — 165 tests passed
• pnpm --filter @workflow/core typecheck
• pnpm --filter @workflow/world-vercel typecheck
• Biome check on changed files
• git diff --check

[changeset-bot]

🦋 Changeset detected

Latest commit: 9f03030

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages

Name	Type
@workflow/core	Patch
@workflow/world	Patch
@workflow/world-vercel	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/world-local	Patch
@workflow/world-postgres	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-nextjs-workflow-webpack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-astro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-express-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-fastify-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-hono-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nestjs-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nitro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nuxt-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-python-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-sveltekit-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-tanstack-start-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-vite-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-docs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-swc-playground	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-tarballs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-web	Ready	Preview, v0	Aug 31, 2026 9:15pm

[github-actions]

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (8 failed)

python-node (8 failed):

• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW | 🔍 observability
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ | 🔍 observability
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z | 🔍 observability
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD | 🔍 observability
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95 | 🔍 observability
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D | 🔍 observability
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H | 🔍 observability
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3 | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

• deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M1CV028S4VSJGT08W5VNCKF3
• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

• abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-webpack)
• addTenWorkflow (nitro)
• cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
• hookWorkflow (tanstack-start)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

33 infra events

• cold-start-warmup · suite warmup (python) · at 21:14:09Z · abandoned wrun_41M1CTMMKH0GJHNMWTRT0RBSZQ · (+6 more)
• run-pickup-stall · sleepingWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB7
• run-pickup-stall · nullByteWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9860GSR1V009T2GZWY1
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9FH0GQ8P3X9DK42SSWT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB8
• run-pickup-stall · promiseAllWorkflow (python) · at 21:14:27Z · abandoned wrun_41M1CTR97V0GTFVK2A34P47G2P
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:59Z · abandoned wrun_41M1CTSAE60GTGN55WM85C4WRF
• run-pickup-stall · promiseAllWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FE0GZDVGGTYRREZFGE
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FC0GT13M6TSBCMD2CF
• run-pickup-stall · sleepingWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4GF0GMKRKA7ZPKGVQKA
• run-pickup-stall · nullByteWorkflow (python) · at 21:15:26Z · abandoned wrun_41M1CTT4TK0GZ59NA0E5DEECN9
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:16:07Z · abandoned wrun_41M1CTTA0V0GW8STRJ0NXH1MAK
• cold-start-warmup · suite warmup (tanstack-start) · at 21:16:29Z · abandoned wrun_01M1CTVQS8HB5PGN6S1W5JJ3GK
• cold-start-warmup · suite warmup (python) · at 21:17:08Z · abandoned wrun_01M1CTT3VDW1P4DARKPKNK56SV · (+7 more)
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZQS5VPNGWV6D9P0WTH
• run-pickup-stall · sleepingWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZXKFDZMW0BJ2PQSTKM
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZZEX0PC8YRTF8BCXJ9
• run-pickup-stall · promiseAllWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZR016BGCJPXEQCN0NT
• run-pickup-stall · nullByteWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXS01F6Q7CDA91SEW04AA
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:17:49Z · abandoned wrun_41M1CTW61K0GKDF638N6P0JFST
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:17:49Z · abandoned wrun_41M1CTX1XD0GZMJW4BJNHFQGRT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKSM4NA1F8PYJ58GY8E
• run-pickup-stall · nullByteWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKT6SQCSMMZVDCS78N1
• run-pickup-stall · sleepingWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKP72PTRG1PX6QAD913
• run-pickup-stall · promiseAllWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKRFTNJ0EFDTE3WRYPV
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKMGB6EP5KBB0VYFFTT
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:18:47Z · abandoned wrun_41M1CTYXY30GGKEAW3PNEB5333
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E77VC437FEHQ0P588M8
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7EHTPTCK35VG9GA5CF
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7JA10XMP3BF8G2B77F
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK3ASZB6STVRTJ0HZ44
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK43AY290VT6J6GVK68
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:20:24Z · abandoned wrun_01M1CV38TQG0ZDSCF4TG5BHPH3

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
❌ ▲ Vercel Production	3570	8	742	4320
✅ 💻 Local Development	3922	0	558	4480
✅ 📦 Local Production	3922	0	558	4480
✅ 🐘 Local Postgres	3922	0	558	4480
✅ 🪟 Windows	320	0	0	320
❌ 🌐 Cross-language Conformance	0	9	132	141
✅ vercel-http-transport	817	0	143	960
✅ vercel-multi-region	27	0	0	27
✅ vercel-ws-transport	553	0	87	640
Total	17053	17	2778	19848

Details by Category

❌ ▲ Vercel Production

App	Passed	Failed	Skipped
✅ astro-node	132	0	28
✅ astro-quickjs	132	0	28
✅ example-node	132	0	28
✅ example-quickjs	132	0	28
✅ express-node	132	0	28
✅ express-quickjs	132	0	28
✅ fastify-node	132	0	28
✅ fastify-quickjs	132	0	28
✅ hono-node	132	0	28
✅ hono-quickjs	132	0	28
✅ nest-node	132	0	28
✅ nest-quickjs	132	0	28
✅ nextjs-turbopack-node	157	0	3
✅ nextjs-turbopack-quickjs	157	0	3
✅ nextjs-webpack-node	157	0	3
✅ nextjs-webpack-quickjs	157	0	3
✅ nitro-node	132	0	28
✅ nitro-quickjs	132	0	28
✅ nuxt-node	132	0	28
✅ nuxt-quickjs	132	0	28
❌ python-node	0	8	152
✅ sveltekit-node	151	0	9
✅ sveltekit-quickjs	151	0	9
✅ tanstack-start-node	132	0	28
✅ tanstack-start-quickjs	132	0	28
✅ vite-node	132	0	28
✅ vite-quickjs	132	0	28

✅ 💻 Local Development

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 📦 Local Production

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🐘 Local Postgres

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	160	0	0
✅ nextjs-turbopack-quickjs	160	0	0

❌ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
❌ python	0	9	132

✅ vercel-http-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ hono	132	0	28
✅ nextjs-turbopack	157	0	3
✅ nitro	132	0	28
✅ vite	132	0	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ nextjs-turbopack	157	0	3
✅ vite	132	0	28

---

📋 View full workflow run

[github-actions]

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario	outcome	events	virt	replay	violations
✅ smoke-no-steps	completed	3	0ms	ok	0
✅ smoke-one-step	completed	6	0ms	ok	0
✅ hook-at-step-started	completed	12	0ms	ok	0
✅ hook-at-step-completed	completed	12	0ms	ok	0
✅ hook-at-hook-created	completed	12	0ms	ok	0
✅ deadline-hook-wins	completed	7	1.0h	ok	0
✅ deadline-expires	completed	7	1.0h	ok	0
✅ long-sleep	completed	11	30.0d	ok	0
✅ hook-never-arrives	stalled	3	0ms	skipped	0
✅ step-retries-twice	completed	10	2.0s	ok	0
✅ parallel-steps	completed	9	0ms	ok	0
✅ hook-on-execution-state	completed	12	0ms	ok	0
✅ peek-hook-before-branch	completed	12	0ms	ok	0
✅ peek-hook-after-branch	completed	12	0ms	ok	0
✅ peek-hook-at-registration	completed	12	0ms	ok	0
✅ race-hook-before-probe	completed	12	0ms	ok	0
✅ race-hook-after-probe	completed	12	0ms	ok	0
✅ race-duplicate-delivery	completed	13	0ms	ok	0
✅ attr-hook-before-step	completed	11	0ms	ok	0
✅ attr-hook-after-step	completed	11	0ms	ok	0
✅ attr-from-step-body	completed	13	0ms	ok	0
✅ fork-hook-after-timeout	completed	14	1.0m	ok	0
✅ fork-hook-before-timeout	completed	14	1.0m	ok	0
✅ count-hook-after-timeout	completed	17	1.0m	ok	0
✅ count-hook-before-timeout	completed	20	1.0m	ok	0
✅ stale-read-step-count-fork	completed	20	1.0m	ok	0
✅ stale-read-equal-step-counts	completed	14	1.0m	ok	0
✅ step-vs-step-fork	completed	12	0ms	ok	0
✅ step-vs-step-fork-fenced	completed	12	0ms	ok	0
✅ fence-catches-benign-direction	completed	12	5ms	ok	0
✅ in-flight-before-decision	completed	17	1.0m	ok	0
❌ in-flight-before-decision-counted	completed	17	1.0m	ok	0
✅ in-flight-after-decision	completed	19	2.0m	ok	0
✅ stale-read-step-count-fork-fenced	completed	20	1.0m	ok	0
✅ fork-hook-wins	completed	13	1.0m	ok	0
✅ fork-timeout-wins	completed	13	1.0m	ok	0
✅ unclaimed-payload-under-fork	completed	17	1.0m	ok	0
✅ claimed-payload-under-fork	completed	17	1.0m	ok	0
✅ writers-independent-step-bodies	completed	12	0ms	ok	0
✅ writers-scripted-tempo	completed	12	0ms	ok	0
✅ cancel-mid-step	cancelled	7	0ms	skipped	0

Full trace: world-sim.txt

[github-actions]

📊 Workflow Benchmarks

commit 9f03030 · Mon, 31 Aug 2026 21:38:22 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1386 (+657%) 🔻	1500 🔴 (+46%) 🔻	1543 🔴 (+43%) 🔻	1646 🔴 (+36%) 🔻	30
TTFS	stream	250 (+16%) 🔻	1454 🔴 (+35%) 🔻	1519 🔴 (+37%) 🔻	1897 🔴 (+26%) 🔻	30
TTFS	hook + stream	1709 (+41%) 🔻	1804 🔴 (+33%) 🔻	1877 🔴 (+34%) 🔻	1976 🔴 (+37%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	612 (-4.1%)	1255 (-27%) 💚	2513 (+39%) 🔻	6361 (+252%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2114 (+18%) 🔻	6996 (+132%) 🔻	11047 (+169%) 🔻	11298 (+165%) 🔻	10
STSO	1020 steps (inline)	108 (+4.9%)	166 (+21%) 🔻	185 (+17%) 🔻	267 (+5.5%)	1019
WO	1020 steps	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	1
CRTT	first chunk (pooled)	123 (+24%) 🔻	165 (-11%)	194 (-18%) 💚	284 (+8.8%)	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	151 (+12%)	195 (+8%)	270 (-5%)	495 (+14%)	146 (-20%)	10
size sweep (100/s, 160B-12KB)	135 (-7%)	180 (-12%)	354 (-19%)	632 (-34%)	128 (-39%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	150 (+13%)	151 (-14%)	179 (-35%)	454 (-51%)	170 (-64%)	3
replay eve-gpt-5.6-sol-2000t (1x)	159 (-3%)	155 (-20%)	199 (-33%)	304 (-81%)	199 (-83%)	2
replay eve-gpt-5.6-sol-2000t (2x)	147 (+4%)	206 (-29%)	274 (-37%)	418 (-47%)	229 (-57%)	3

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 136556ms → this run 162927ms (Δ +26371ms, +19%)

100-150 ms ███████████┃████████████ main 863 this 433 -430
150-200 ms ████░░░░░░░░░░┃ main 130 this 530 +400
200-250 ms ┃ main 15 this 41 +26
250-300 ms ┃ main 8 this 7 -1
300-350 ms ┃ main 2 this 4 +2
350-400 ms ┃ main 1 this 1 +0
400-450 ms ┃ main 0 this 1 +1
500-550 ms ┃ main 0 this 2 +2

📈 CRTT drill-down vs main (RTT distributions & profiles)

variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▂▁··· 152.3 (-3%) 141 (+1%) 270 (-5%) 495 (+14%) 3000
sweep ······▂█▂▁··· 150.4 (-10%) 135 (+2%) 354 (-19%) 632 (-34%) 3000
gw 1x ·····▁▂█▁▁··· 131.2 (-13%) 126 (-2%) 179 (-35%) 454 (-51%) 5295
eve 1x ······▃█▁···· 134.4 (-28%) 126 (-8%) 199 (-33%) 304 (-81%) 5186
eve 2x ·····▁▁█▃···· 171.3 (-18%) 161 (-4%) 274 (-37%) 418 (-47%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control █▆▇▆▅▁▃▁▆▁ 133–172ms
sweep █▇▃▁▅▅▂▂▂▁ 128–189ms
gw 1x ▅█▃▃▁▄▂▃▁▃ 122–150ms
eve 1x ▆▂▄▃▃▃█▆▃▁ 119–156ms
eve 2x ▂▁▃▂▁▅▅█▅▃ 142–220ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep ▇▆█▆▅▁▄ 148–152ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▁▆▅▆▃▄▅▄█▄ 35–57ms
sweep ▁▁▃▅█▄▅▃▁▄ 50–74ms
gw 1x ▄█▁▁▄▅▅▆▁▂ 38–47ms
eve 1x ▇▅▃▇▇▄█▃▃▁ 21–28ms
eve 2x ▇█▃▂█▅▅▁█▃ 24–31ms

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

[VaguelySerious]

AI review: blocking issues found

[github-actions]

Framework	Flow route	Step reg.	Framework output
hono	200.9 KiB (±0)	40.4 KiB (±0)	1.76 MiB (-45 B)
nextjs-turbopack	206.3 KiB (±0)	439 B (±0)	765.5 KiB (+338 B)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

9f03030 · run

[alangenfeld]

local agent review surfaced:

The new speculative key lookup always calls:

resolveRunEncryptionKey(world,runId)

This discards the WorkflowRun.deploymentId that the previous implementation supplied by calling getEncryptionKeyForRun(workflowRun).

That breaks encrypted Node replays outside a Vercel Function—such as local tooling or external runners—because world-vercel cannot resolve a key from a bare run ID without { deploymentId }; it returns undefined. The replay then attempts to hydrate encrypted input/events without the key. It can also select the wrong key when deployment routing matters.

The lookup should start from the streamed run_created/runInput deployment context, for example:

resolveRunEncryptionKey(world,runId,{deploymentId: workflow.deploymentId,})

or defer until the authoritative WorkflowRun is available and pass that object.

[github-actions]

No backport to stable for e9d5c56 (AI decision).

This is a replay-startup performance optimization: it prepares payloads as event frames stream in, lazily resolves the encryption key, skips Node-only work for QuickJS, and adds a new replayEventObserver option to the CreateEventParams public World interface. None of it fixes a user-visible defect on stable — the incidental hardening (setup failures classified as run_failed, unhandled-rejection guards, removal of the primitive memoization size cap) exists to support the new streaming path rather than to repair existing behavior, and it touches the hot replay path plus a World API surface, which is exactly the kind of churn a maintenance line should not take.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e9d5c56701821b090108a85b74bf8b0cbef8ea8e

[NathanColosimo]

Summary

• prepare validated Node replay payloads as streamed events arrive
• lazily resolve the invocation encryption key and create the payload cache only after the run is known to use Node
• skip host-side compile, key lookup, replay cache, and payload preparation for QuickJS
• start run_started before speculative compile/cache work so setup I/O overlaps Node startup
• correct speculative workflow names from persisted run_created
• classify invalid VM configuration as run_failed in turbo, ordinary, and streamed setup paths
• count materialized replay events without per-event spans
• share one replay-run reconstruction path between run_started and atomic-hook preload
• leave primitive payload caching uncapped

Prewarm deliberately scans the event array on each replay and relies on the existing event-ID map to skip prepared payloads. This avoids positional scan state, reset hooks, and event-log replacement wrappers; only the cheap array walk repeats. Prepared plaintext and memoized primitive results remain invocation-scoped.

Stack

Depends on #3798.

Validation

• focused regression suites — 131 tests passed
• final runtime regression suites — 60 tests passed
• full @workflow/core suite — 2,295 tests passed
• full @workflow/world-vercel suite — 572 tests passed
• full @workflow/world suite — 165 tests passed
• pnpm --filter @workflow/core typecheck
• pnpm --filter @workflow/world-vercel typecheck
• Biome check on changed files
• git diff --check

[changeset-bot]

🦋 Changeset detected

Latest commit: 9f03030

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages

Name	Type
@workflow/core	Patch
@workflow/world	Patch
@workflow/world-vercel	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/world-local	Patch
@workflow/world-postgres	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-nextjs-workflow-webpack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-astro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-express-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-fastify-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-hono-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nestjs-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nitro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nuxt-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-python-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-sveltekit-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-tanstack-start-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-vite-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-docs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-swc-playground	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-tarballs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-web	Ready	Preview, v0	Aug 31, 2026 9:15pm

[github-actions]

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (8 failed)

python-node (8 failed):

• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW | 🔍 observability
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ | 🔍 observability
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z | 🔍 observability
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD | 🔍 observability
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95 | 🔍 observability
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D | 🔍 observability
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H | 🔍 observability
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3 | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

• deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M1CV028S4VSJGT08W5VNCKF3
• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

• abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-webpack)
• addTenWorkflow (nitro)
• cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
• hookWorkflow (tanstack-start)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

33 infra events

• cold-start-warmup · suite warmup (python) · at 21:14:09Z · abandoned wrun_41M1CTMMKH0GJHNMWTRT0RBSZQ · (+6 more)
• run-pickup-stall · sleepingWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB7
• run-pickup-stall · nullByteWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9860GSR1V009T2GZWY1
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9FH0GQ8P3X9DK42SSWT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB8
• run-pickup-stall · promiseAllWorkflow (python) · at 21:14:27Z · abandoned wrun_41M1CTR97V0GTFVK2A34P47G2P
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:59Z · abandoned wrun_41M1CTSAE60GTGN55WM85C4WRF
• run-pickup-stall · promiseAllWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FE0GZDVGGTYRREZFGE
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FC0GT13M6TSBCMD2CF
• run-pickup-stall · sleepingWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4GF0GMKRKA7ZPKGVQKA
• run-pickup-stall · nullByteWorkflow (python) · at 21:15:26Z · abandoned wrun_41M1CTT4TK0GZ59NA0E5DEECN9
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:16:07Z · abandoned wrun_41M1CTTA0V0GW8STRJ0NXH1MAK
• cold-start-warmup · suite warmup (tanstack-start) · at 21:16:29Z · abandoned wrun_01M1CTVQS8HB5PGN6S1W5JJ3GK
• cold-start-warmup · suite warmup (python) · at 21:17:08Z · abandoned wrun_01M1CTT3VDW1P4DARKPKNK56SV · (+7 more)
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZQS5VPNGWV6D9P0WTH
• run-pickup-stall · sleepingWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZXKFDZMW0BJ2PQSTKM
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZZEX0PC8YRTF8BCXJ9
• run-pickup-stall · promiseAllWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZR016BGCJPXEQCN0NT
• run-pickup-stall · nullByteWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXS01F6Q7CDA91SEW04AA
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:17:49Z · abandoned wrun_41M1CTW61K0GKDF638N6P0JFST
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:17:49Z · abandoned wrun_41M1CTX1XD0GZMJW4BJNHFQGRT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKSM4NA1F8PYJ58GY8E
• run-pickup-stall · nullByteWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKT6SQCSMMZVDCS78N1
• run-pickup-stall · sleepingWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKP72PTRG1PX6QAD913
• run-pickup-stall · promiseAllWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKRFTNJ0EFDTE3WRYPV
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKMGB6EP5KBB0VYFFTT
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:18:47Z · abandoned wrun_41M1CTYXY30GGKEAW3PNEB5333
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E77VC437FEHQ0P588M8
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7EHTPTCK35VG9GA5CF
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7JA10XMP3BF8G2B77F
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK3ASZB6STVRTJ0HZ44
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK43AY290VT6J6GVK68
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:20:24Z · abandoned wrun_01M1CV38TQG0ZDSCF4TG5BHPH3

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
❌ ▲ Vercel Production	3570	8	742	4320
✅ 💻 Local Development	3922	0	558	4480
✅ 📦 Local Production	3922	0	558	4480
✅ 🐘 Local Postgres	3922	0	558	4480
✅ 🪟 Windows	320	0	0	320
❌ 🌐 Cross-language Conformance	0	9	132	141
✅ vercel-http-transport	817	0	143	960
✅ vercel-multi-region	27	0	0	27
✅ vercel-ws-transport	553	0	87	640
Total	17053	17	2778	19848

Details by Category

❌ ▲ Vercel Production

App	Passed	Failed	Skipped
✅ astro-node	132	0	28
✅ astro-quickjs	132	0	28
✅ example-node	132	0	28
✅ example-quickjs	132	0	28
✅ express-node	132	0	28
✅ express-quickjs	132	0	28
✅ fastify-node	132	0	28
✅ fastify-quickjs	132	0	28
✅ hono-node	132	0	28
✅ hono-quickjs	132	0	28
✅ nest-node	132	0	28
✅ nest-quickjs	132	0	28
✅ nextjs-turbopack-node	157	0	3
✅ nextjs-turbopack-quickjs	157	0	3
✅ nextjs-webpack-node	157	0	3
✅ nextjs-webpack-quickjs	157	0	3
✅ nitro-node	132	0	28
✅ nitro-quickjs	132	0	28
✅ nuxt-node	132	0	28
✅ nuxt-quickjs	132	0	28
❌ python-node	0	8	152
✅ sveltekit-node	151	0	9
✅ sveltekit-quickjs	151	0	9
✅ tanstack-start-node	132	0	28
✅ tanstack-start-quickjs	132	0	28
✅ vite-node	132	0	28
✅ vite-quickjs	132	0	28

✅ 💻 Local Development

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 📦 Local Production

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🐘 Local Postgres

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	160	0	0
✅ nextjs-turbopack-quickjs	160	0	0

❌ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
❌ python	0	9	132

✅ vercel-http-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ hono	132	0	28
✅ nextjs-turbopack	157	0	3
✅ nitro	132	0	28
✅ vite	132	0	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ nextjs-turbopack	157	0	3
✅ vite	132	0	28

---

📋 View full workflow run

[github-actions]

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario	outcome	events	virt	replay	violations
✅ smoke-no-steps	completed	3	0ms	ok	0
✅ smoke-one-step	completed	6	0ms	ok	0
✅ hook-at-step-started	completed	12	0ms	ok	0
✅ hook-at-step-completed	completed	12	0ms	ok	0
✅ hook-at-hook-created	completed	12	0ms	ok	0
✅ deadline-hook-wins	completed	7	1.0h	ok	0
✅ deadline-expires	completed	7	1.0h	ok	0
✅ long-sleep	completed	11	30.0d	ok	0
✅ hook-never-arrives	stalled	3	0ms	skipped	0
✅ step-retries-twice	completed	10	2.0s	ok	0
✅ parallel-steps	completed	9	0ms	ok	0
✅ hook-on-execution-state	completed	12	0ms	ok	0
✅ peek-hook-before-branch	completed	12	0ms	ok	0
✅ peek-hook-after-branch	completed	12	0ms	ok	0
✅ peek-hook-at-registration	completed	12	0ms	ok	0
✅ race-hook-before-probe	completed	12	0ms	ok	0
✅ race-hook-after-probe	completed	12	0ms	ok	0
✅ race-duplicate-delivery	completed	13	0ms	ok	0
✅ attr-hook-before-step	completed	11	0ms	ok	0
✅ attr-hook-after-step	completed	11	0ms	ok	0
✅ attr-from-step-body	completed	13	0ms	ok	0
✅ fork-hook-after-timeout	completed	14	1.0m	ok	0
✅ fork-hook-before-timeout	completed	14	1.0m	ok	0
✅ count-hook-after-timeout	completed	17	1.0m	ok	0
✅ count-hook-before-timeout	completed	20	1.0m	ok	0
✅ stale-read-step-count-fork	completed	20	1.0m	ok	0
✅ stale-read-equal-step-counts	completed	14	1.0m	ok	0
✅ step-vs-step-fork	completed	12	0ms	ok	0
✅ step-vs-step-fork-fenced	completed	12	0ms	ok	0
✅ fence-catches-benign-direction	completed	12	5ms	ok	0
✅ in-flight-before-decision	completed	17	1.0m	ok	0
❌ in-flight-before-decision-counted	completed	17	1.0m	ok	0
✅ in-flight-after-decision	completed	19	2.0m	ok	0
✅ stale-read-step-count-fork-fenced	completed	20	1.0m	ok	0
✅ fork-hook-wins	completed	13	1.0m	ok	0
✅ fork-timeout-wins	completed	13	1.0m	ok	0
✅ unclaimed-payload-under-fork	completed	17	1.0m	ok	0
✅ claimed-payload-under-fork	completed	17	1.0m	ok	0
✅ writers-independent-step-bodies	completed	12	0ms	ok	0
✅ writers-scripted-tempo	completed	12	0ms	ok	0
✅ cancel-mid-step	cancelled	7	0ms	skipped	0

Full trace: world-sim.txt

[github-actions]

📊 Workflow Benchmarks

commit 9f03030 · Mon, 31 Aug 2026 21:38:22 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1386 (+657%) 🔻	1500 🔴 (+46%) 🔻	1543 🔴 (+43%) 🔻	1646 🔴 (+36%) 🔻	30
TTFS	stream	250 (+16%) 🔻	1454 🔴 (+35%) 🔻	1519 🔴 (+37%) 🔻	1897 🔴 (+26%) 🔻	30
TTFS	hook + stream	1709 (+41%) 🔻	1804 🔴 (+33%) 🔻	1877 🔴 (+34%) 🔻	1976 🔴 (+37%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	612 (-4.1%)	1255 (-27%) 💚	2513 (+39%) 🔻	6361 (+252%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2114 (+18%) 🔻	6996 (+132%) 🔻	11047 (+169%) 🔻	11298 (+165%) 🔻	10
STSO	1020 steps (inline)	108 (+4.9%)	166 (+21%) 🔻	185 (+17%) 🔻	267 (+5.5%)	1019
WO	1020 steps	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	1
CRTT	first chunk (pooled)	123 (+24%) 🔻	165 (-11%)	194 (-18%) 💚	284 (+8.8%)	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	151 (+12%)	195 (+8%)	270 (-5%)	495 (+14%)	146 (-20%)	10
size sweep (100/s, 160B-12KB)	135 (-7%)	180 (-12%)	354 (-19%)	632 (-34%)	128 (-39%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	150 (+13%)	151 (-14%)	179 (-35%)	454 (-51%)	170 (-64%)	3
replay eve-gpt-5.6-sol-2000t (1x)	159 (-3%)	155 (-20%)	199 (-33%)	304 (-81%)	199 (-83%)	2
replay eve-gpt-5.6-sol-2000t (2x)	147 (+4%)	206 (-29%)	274 (-37%)	418 (-47%)	229 (-57%)	3

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 136556ms → this run 162927ms (Δ +26371ms, +19%)

100-150 ms ███████████┃████████████ main 863 this 433 -430
150-200 ms ████░░░░░░░░░░┃ main 130 this 530 +400
200-250 ms ┃ main 15 this 41 +26
250-300 ms ┃ main 8 this 7 -1
300-350 ms ┃ main 2 this 4 +2
350-400 ms ┃ main 1 this 1 +0
400-450 ms ┃ main 0 this 1 +1
500-550 ms ┃ main 0 this 2 +2

📈 CRTT drill-down vs main (RTT distributions & profiles)

variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▂▁··· 152.3 (-3%) 141 (+1%) 270 (-5%) 495 (+14%) 3000
sweep ······▂█▂▁··· 150.4 (-10%) 135 (+2%) 354 (-19%) 632 (-34%) 3000
gw 1x ·····▁▂█▁▁··· 131.2 (-13%) 126 (-2%) 179 (-35%) 454 (-51%) 5295
eve 1x ······▃█▁···· 134.4 (-28%) 126 (-8%) 199 (-33%) 304 (-81%) 5186
eve 2x ·····▁▁█▃···· 171.3 (-18%) 161 (-4%) 274 (-37%) 418 (-47%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control █▆▇▆▅▁▃▁▆▁ 133–172ms
sweep █▇▃▁▅▅▂▂▂▁ 128–189ms
gw 1x ▅█▃▃▁▄▂▃▁▃ 122–150ms
eve 1x ▆▂▄▃▃▃█▆▃▁ 119–156ms
eve 2x ▂▁▃▂▁▅▅█▅▃ 142–220ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep ▇▆█▆▅▁▄ 148–152ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▁▆▅▆▃▄▅▄█▄ 35–57ms
sweep ▁▁▃▅█▄▅▃▁▄ 50–74ms
gw 1x ▄█▁▁▄▅▅▆▁▂ 38–47ms
eve 1x ▇▅▃▇▇▄█▃▃▁ 21–28ms
eve 2x ▇█▃▂█▅▅▁█▃ 24–31ms

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

[VaguelySerious]

AI review: blocking issues found

[github-actions]

Framework	Flow route	Step reg.	Framework output
hono	200.9 KiB (±0)	40.4 KiB (±0)	1.76 MiB (-45 B)
nextjs-turbopack	206.3 KiB (±0)	439 B (±0)	765.5 KiB (+338 B)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

9f03030 · run

[alangenfeld]

local agent review surfaced:

The new speculative key lookup always calls:

resolveRunEncryptionKey(world,runId)

This discards the WorkflowRun.deploymentId that the previous implementation supplied by calling getEncryptionKeyForRun(workflowRun).

That breaks encrypted Node replays outside a Vercel Function—such as local tooling or external runners—because world-vercel cannot resolve a key from a bare run ID without { deploymentId }; it returns undefined. The replay then attempts to hydrate encrypted input/events without the key. It can also select the wrong key when deployment routing matters.

The lookup should start from the streamed run_created/runInput deployment context, for example:

resolveRunEncryptionKey(world,runId,{deploymentId: workflow.deploymentId,})

or defer until the authoritative WorkflowRun is available and pass that object.

[github-actions]

No backport to stable for e9d5c56 (AI decision).

This is a replay-startup performance optimization: it prepares payloads as event frames stream in, lazily resolves the encryption key, skips Node-only work for QuickJS, and adds a new replayEventObserver option to the CreateEventParams public World interface. None of it fixes a user-visible defect on stable — the incidental hardening (setup failures classified as run_failed, unhandled-rejection guards, removal of the primitive memoization size cap) exists to support the new streaming path rather than to repair existing behavior, and it touches the hot replay path plus a World API surface, which is exactly the kind of churn a maintenance line should not take.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e9d5c56701821b090108a85b74bf8b0cbef8ea8e

[NathanColosimo]

Summary

• prepare validated Node replay payloads as streamed events arrive
• lazily resolve the invocation encryption key and create the payload cache only after the run is known to use Node
• skip host-side compile, key lookup, replay cache, and payload preparation for QuickJS
• start run_started before speculative compile/cache work so setup I/O overlaps Node startup
• correct speculative workflow names from persisted run_created
• classify invalid VM configuration as run_failed in turbo, ordinary, and streamed setup paths
• count materialized replay events without per-event spans
• share one replay-run reconstruction path between run_started and atomic-hook preload
• leave primitive payload caching uncapped

Prewarm deliberately scans the event array on each replay and relies on the existing event-ID map to skip prepared payloads. This avoids positional scan state, reset hooks, and event-log replacement wrappers; only the cheap array walk repeats. Prepared plaintext and memoized primitive results remain invocation-scoped.

Stack

Depends on #3798.

Validation

• focused regression suites — 131 tests passed
• final runtime regression suites — 60 tests passed
• full @workflow/core suite — 2,295 tests passed
• full @workflow/world-vercel suite — 572 tests passed
• full @workflow/world suite — 165 tests passed
• pnpm --filter @workflow/core typecheck
• pnpm --filter @workflow/world-vercel typecheck
• Biome check on changed files
• git diff --check

[changeset-bot]

🦋 Changeset detected

Latest commit: 9f03030

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages

Name	Type
@workflow/core	Patch
@workflow/world	Patch
@workflow/world-vercel	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/world-local	Patch
@workflow/world-postgres	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-nextjs-workflow-webpack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-astro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-express-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-fastify-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-hono-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nestjs-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nitro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nuxt-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-python-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-sveltekit-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-tanstack-start-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-vite-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-docs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-swc-playground	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-tarballs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-web	Ready	Preview, v0	Aug 31, 2026 9:15pm

[github-actions]

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (8 failed)

python-node (8 failed):

• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW | 🔍 observability
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ | 🔍 observability
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z | 🔍 observability
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD | 🔍 observability
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95 | 🔍 observability
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D | 🔍 observability
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H | 🔍 observability
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3 | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

• deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M1CV028S4VSJGT08W5VNCKF3
• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

• abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-webpack)
• addTenWorkflow (nitro)
• cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
• hookWorkflow (tanstack-start)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

33 infra events

• cold-start-warmup · suite warmup (python) · at 21:14:09Z · abandoned wrun_41M1CTMMKH0GJHNMWTRT0RBSZQ · (+6 more)
• run-pickup-stall · sleepingWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB7
• run-pickup-stall · nullByteWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9860GSR1V009T2GZWY1
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9FH0GQ8P3X9DK42SSWT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB8
• run-pickup-stall · promiseAllWorkflow (python) · at 21:14:27Z · abandoned wrun_41M1CTR97V0GTFVK2A34P47G2P
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:59Z · abandoned wrun_41M1CTSAE60GTGN55WM85C4WRF
• run-pickup-stall · promiseAllWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FE0GZDVGGTYRREZFGE
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FC0GT13M6TSBCMD2CF
• run-pickup-stall · sleepingWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4GF0GMKRKA7ZPKGVQKA
• run-pickup-stall · nullByteWorkflow (python) · at 21:15:26Z · abandoned wrun_41M1CTT4TK0GZ59NA0E5DEECN9
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:16:07Z · abandoned wrun_41M1CTTA0V0GW8STRJ0NXH1MAK
• cold-start-warmup · suite warmup (tanstack-start) · at 21:16:29Z · abandoned wrun_01M1CTVQS8HB5PGN6S1W5JJ3GK
• cold-start-warmup · suite warmup (python) · at 21:17:08Z · abandoned wrun_01M1CTT3VDW1P4DARKPKNK56SV · (+7 more)
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZQS5VPNGWV6D9P0WTH
• run-pickup-stall · sleepingWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZXKFDZMW0BJ2PQSTKM
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZZEX0PC8YRTF8BCXJ9
• run-pickup-stall · promiseAllWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZR016BGCJPXEQCN0NT
• run-pickup-stall · nullByteWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXS01F6Q7CDA91SEW04AA
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:17:49Z · abandoned wrun_41M1CTW61K0GKDF638N6P0JFST
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:17:49Z · abandoned wrun_41M1CTX1XD0GZMJW4BJNHFQGRT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKSM4NA1F8PYJ58GY8E
• run-pickup-stall · nullByteWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKT6SQCSMMZVDCS78N1
• run-pickup-stall · sleepingWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKP72PTRG1PX6QAD913
• run-pickup-stall · promiseAllWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKRFTNJ0EFDTE3WRYPV
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKMGB6EP5KBB0VYFFTT
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:18:47Z · abandoned wrun_41M1CTYXY30GGKEAW3PNEB5333
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E77VC437FEHQ0P588M8
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7EHTPTCK35VG9GA5CF
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7JA10XMP3BF8G2B77F
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK3ASZB6STVRTJ0HZ44
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK43AY290VT6J6GVK68
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:20:24Z · abandoned wrun_01M1CV38TQG0ZDSCF4TG5BHPH3

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
❌ ▲ Vercel Production	3570	8	742	4320
✅ 💻 Local Development	3922	0	558	4480
✅ 📦 Local Production	3922	0	558	4480
✅ 🐘 Local Postgres	3922	0	558	4480
✅ 🪟 Windows	320	0	0	320
❌ 🌐 Cross-language Conformance	0	9	132	141
✅ vercel-http-transport	817	0	143	960
✅ vercel-multi-region	27	0	0	27
✅ vercel-ws-transport	553	0	87	640
Total	17053	17	2778	19848

Details by Category

❌ ▲ Vercel Production

App	Passed	Failed	Skipped
✅ astro-node	132	0	28
✅ astro-quickjs	132	0	28
✅ example-node	132	0	28
✅ example-quickjs	132	0	28
✅ express-node	132	0	28
✅ express-quickjs	132	0	28
✅ fastify-node	132	0	28
✅ fastify-quickjs	132	0	28
✅ hono-node	132	0	28
✅ hono-quickjs	132	0	28
✅ nest-node	132	0	28
✅ nest-quickjs	132	0	28
✅ nextjs-turbopack-node	157	0	3
✅ nextjs-turbopack-quickjs	157	0	3
✅ nextjs-webpack-node	157	0	3
✅ nextjs-webpack-quickjs	157	0	3
✅ nitro-node	132	0	28
✅ nitro-quickjs	132	0	28
✅ nuxt-node	132	0	28
✅ nuxt-quickjs	132	0	28
❌ python-node	0	8	152
✅ sveltekit-node	151	0	9
✅ sveltekit-quickjs	151	0	9
✅ tanstack-start-node	132	0	28
✅ tanstack-start-quickjs	132	0	28
✅ vite-node	132	0	28
✅ vite-quickjs	132	0	28

✅ 💻 Local Development

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 📦 Local Production

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🐘 Local Postgres

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	160	0	0
✅ nextjs-turbopack-quickjs	160	0	0

❌ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
❌ python	0	9	132

✅ vercel-http-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ hono	132	0	28
✅ nextjs-turbopack	157	0	3
✅ nitro	132	0	28
✅ vite	132	0	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ nextjs-turbopack	157	0	3
✅ vite	132	0	28

---

📋 View full workflow run

[github-actions]

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario	outcome	events	virt	replay	violations
✅ smoke-no-steps	completed	3	0ms	ok	0
✅ smoke-one-step	completed	6	0ms	ok	0
✅ hook-at-step-started	completed	12	0ms	ok	0
✅ hook-at-step-completed	completed	12	0ms	ok	0
✅ hook-at-hook-created	completed	12	0ms	ok	0
✅ deadline-hook-wins	completed	7	1.0h	ok	0
✅ deadline-expires	completed	7	1.0h	ok	0
✅ long-sleep	completed	11	30.0d	ok	0
✅ hook-never-arrives	stalled	3	0ms	skipped	0
✅ step-retries-twice	completed	10	2.0s	ok	0
✅ parallel-steps	completed	9	0ms	ok	0
✅ hook-on-execution-state	completed	12	0ms	ok	0
✅ peek-hook-before-branch	completed	12	0ms	ok	0
✅ peek-hook-after-branch	completed	12	0ms	ok	0
✅ peek-hook-at-registration	completed	12	0ms	ok	0
✅ race-hook-before-probe	completed	12	0ms	ok	0
✅ race-hook-after-probe	completed	12	0ms	ok	0
✅ race-duplicate-delivery	completed	13	0ms	ok	0
✅ attr-hook-before-step	completed	11	0ms	ok	0
✅ attr-hook-after-step	completed	11	0ms	ok	0
✅ attr-from-step-body	completed	13	0ms	ok	0
✅ fork-hook-after-timeout	completed	14	1.0m	ok	0
✅ fork-hook-before-timeout	completed	14	1.0m	ok	0
✅ count-hook-after-timeout	completed	17	1.0m	ok	0
✅ count-hook-before-timeout	completed	20	1.0m	ok	0
✅ stale-read-step-count-fork	completed	20	1.0m	ok	0
✅ stale-read-equal-step-counts	completed	14	1.0m	ok	0
✅ step-vs-step-fork	completed	12	0ms	ok	0
✅ step-vs-step-fork-fenced	completed	12	0ms	ok	0
✅ fence-catches-benign-direction	completed	12	5ms	ok	0
✅ in-flight-before-decision	completed	17	1.0m	ok	0
❌ in-flight-before-decision-counted	completed	17	1.0m	ok	0
✅ in-flight-after-decision	completed	19	2.0m	ok	0
✅ stale-read-step-count-fork-fenced	completed	20	1.0m	ok	0
✅ fork-hook-wins	completed	13	1.0m	ok	0
✅ fork-timeout-wins	completed	13	1.0m	ok	0
✅ unclaimed-payload-under-fork	completed	17	1.0m	ok	0
✅ claimed-payload-under-fork	completed	17	1.0m	ok	0
✅ writers-independent-step-bodies	completed	12	0ms	ok	0
✅ writers-scripted-tempo	completed	12	0ms	ok	0
✅ cancel-mid-step	cancelled	7	0ms	skipped	0

Full trace: world-sim.txt

[github-actions]

📊 Workflow Benchmarks

commit 9f03030 · Mon, 31 Aug 2026 21:38:22 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1386 (+657%) 🔻	1500 🔴 (+46%) 🔻	1543 🔴 (+43%) 🔻	1646 🔴 (+36%) 🔻	30
TTFS	stream	250 (+16%) 🔻	1454 🔴 (+35%) 🔻	1519 🔴 (+37%) 🔻	1897 🔴 (+26%) 🔻	30
TTFS	hook + stream	1709 (+41%) 🔻	1804 🔴 (+33%) 🔻	1877 🔴 (+34%) 🔻	1976 🔴 (+37%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	612 (-4.1%)	1255 (-27%) 💚	2513 (+39%) 🔻	6361 (+252%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2114 (+18%) 🔻	6996 (+132%) 🔻	11047 (+169%) 🔻	11298 (+165%) 🔻	10
STSO	1020 steps (inline)	108 (+4.9%)	166 (+21%) 🔻	185 (+17%) 🔻	267 (+5.5%)	1019
WO	1020 steps	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	1
CRTT	first chunk (pooled)	123 (+24%) 🔻	165 (-11%)	194 (-18%) 💚	284 (+8.8%)	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	151 (+12%)	195 (+8%)	270 (-5%)	495 (+14%)	146 (-20%)	10
size sweep (100/s, 160B-12KB)	135 (-7%)	180 (-12%)	354 (-19%)	632 (-34%)	128 (-39%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	150 (+13%)	151 (-14%)	179 (-35%)	454 (-51%)	170 (-64%)	3
replay eve-gpt-5.6-sol-2000t (1x)	159 (-3%)	155 (-20%)	199 (-33%)	304 (-81%)	199 (-83%)	2
replay eve-gpt-5.6-sol-2000t (2x)	147 (+4%)	206 (-29%)	274 (-37%)	418 (-47%)	229 (-57%)	3

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 136556ms → this run 162927ms (Δ +26371ms, +19%)

100-150 ms ███████████┃████████████ main 863 this 433 -430
150-200 ms ████░░░░░░░░░░┃ main 130 this 530 +400
200-250 ms ┃ main 15 this 41 +26
250-300 ms ┃ main 8 this 7 -1
300-350 ms ┃ main 2 this 4 +2
350-400 ms ┃ main 1 this 1 +0
400-450 ms ┃ main 0 this 1 +1
500-550 ms ┃ main 0 this 2 +2

📈 CRTT drill-down vs main (RTT distributions & profiles)

variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▂▁··· 152.3 (-3%) 141 (+1%) 270 (-5%) 495 (+14%) 3000
sweep ······▂█▂▁··· 150.4 (-10%) 135 (+2%) 354 (-19%) 632 (-34%) 3000
gw 1x ·····▁▂█▁▁··· 131.2 (-13%) 126 (-2%) 179 (-35%) 454 (-51%) 5295
eve 1x ······▃█▁···· 134.4 (-28%) 126 (-8%) 199 (-33%) 304 (-81%) 5186
eve 2x ·····▁▁█▃···· 171.3 (-18%) 161 (-4%) 274 (-37%) 418 (-47%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control █▆▇▆▅▁▃▁▆▁ 133–172ms
sweep █▇▃▁▅▅▂▂▂▁ 128–189ms
gw 1x ▅█▃▃▁▄▂▃▁▃ 122–150ms
eve 1x ▆▂▄▃▃▃█▆▃▁ 119–156ms
eve 2x ▂▁▃▂▁▅▅█▅▃ 142–220ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep ▇▆█▆▅▁▄ 148–152ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▁▆▅▆▃▄▅▄█▄ 35–57ms
sweep ▁▁▃▅█▄▅▃▁▄ 50–74ms
gw 1x ▄█▁▁▄▅▅▆▁▂ 38–47ms
eve 1x ▇▅▃▇▇▄█▃▃▁ 21–28ms
eve 2x ▇█▃▂█▅▅▁█▃ 24–31ms

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

[VaguelySerious]

AI review: blocking issues found

[github-actions]

Framework	Flow route	Step reg.	Framework output
hono	200.9 KiB (±0)	40.4 KiB (±0)	1.76 MiB (-45 B)
nextjs-turbopack	206.3 KiB (±0)	439 B (±0)	765.5 KiB (+338 B)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

9f03030 · run

[alangenfeld]

local agent review surfaced:

The new speculative key lookup always calls:

resolveRunEncryptionKey(world,runId)

This discards the WorkflowRun.deploymentId that the previous implementation supplied by calling getEncryptionKeyForRun(workflowRun).

That breaks encrypted Node replays outside a Vercel Function—such as local tooling or external runners—because world-vercel cannot resolve a key from a bare run ID without { deploymentId }; it returns undefined. The replay then attempts to hydrate encrypted input/events without the key. It can also select the wrong key when deployment routing matters.

The lookup should start from the streamed run_created/runInput deployment context, for example:

resolveRunEncryptionKey(world,runId,{deploymentId: workflow.deploymentId,})

or defer until the authoritative WorkflowRun is available and pass that object.

[github-actions]

No backport to stable for e9d5c56 (AI decision).

This is a replay-startup performance optimization: it prepares payloads as event frames stream in, lazily resolves the encryption key, skips Node-only work for QuickJS, and adds a new replayEventObserver option to the CreateEventParams public World interface. None of it fixes a user-visible defect on stable — the incidental hardening (setup failures classified as run_failed, unhandled-rejection guards, removal of the primitive memoization size cap) exists to support the new streaming path rather than to repair existing behavior, and it touches the hot replay path plus a World API surface, which is exactly the kind of churn a maintenance line should not take.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e9d5c56701821b090108a85b74bf8b0cbef8ea8e

[NathanColosimo]

Summary

• prepare validated Node replay payloads as streamed events arrive
• lazily resolve the invocation encryption key and create the payload cache only after the run is known to use Node
• skip host-side compile, key lookup, replay cache, and payload preparation for QuickJS
• start run_started before speculative compile/cache work so setup I/O overlaps Node startup
• correct speculative workflow names from persisted run_created
• classify invalid VM configuration as run_failed in turbo, ordinary, and streamed setup paths
• count materialized replay events without per-event spans
• share one replay-run reconstruction path between run_started and atomic-hook preload
• leave primitive payload caching uncapped

Prewarm deliberately scans the event array on each replay and relies on the existing event-ID map to skip prepared payloads. This avoids positional scan state, reset hooks, and event-log replacement wrappers; only the cheap array walk repeats. Prepared plaintext and memoized primitive results remain invocation-scoped.

Stack

Depends on #3798.

Validation

• focused regression suites — 131 tests passed
• final runtime regression suites — 60 tests passed
• full @workflow/core suite — 2,295 tests passed
• full @workflow/world-vercel suite — 572 tests passed
• full @workflow/world suite — 165 tests passed
• pnpm --filter @workflow/core typecheck
• pnpm --filter @workflow/world-vercel typecheck
• Biome check on changed files
• git diff --check

[changeset-bot]

🦋 Changeset detected

Latest commit: 9f03030

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages

Name	Type
@workflow/core	Patch
@workflow/world	Patch
@workflow/world-vercel	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/world-local	Patch
@workflow/world-postgres	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-nextjs-workflow-webpack	Ready	Preview, v0	Aug 31, 2026 9:15pm
example-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-astro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-express-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-fastify-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-hono-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nestjs-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nitro-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-nuxt-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-python-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-sveltekit-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-tanstack-start-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workbench-vite-workflow	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-docs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-swc-playground	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-tarballs	Ready	Preview, v0	Aug 31, 2026 9:15pm
workflow-web	Ready	Preview, v0	Aug 31, 2026 9:15pm

[github-actions]

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (8 failed)

python-node (8 failed):

• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW | 🔍 observability
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ | 🔍 observability
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z | 🔍 observability
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD | 🔍 observability
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95 | 🔍 observability
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D | 🔍 observability
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H | 🔍 observability
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3 | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

• deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M1CV028S4VSJGT08W5VNCKF3
• promiseAllWorkflow | wrun_41M1CTQKXW0GMJ8VY1KMZ3H0PW
• sleepingWorkflow | wrun_41M1CTR9NE0GVXYZEBQZDNTKRJ
• parallelSleepWorkflow | wrun_41M1CTRBRV0GZ7EB1GBK2MZA2Z
• nullByteWorkflow | wrun_41M1CTRJ6K0GVDWDXCZ6A9AMAD
• cancelRun - cancelling a running workflow | wrun_41M1CTWNBB0GGY0H4VVF7B8H95
• cancelRun via CLI - cancelling a running workflow | wrun_41M1CTWWC60GZ7TBW2AZX2GJ5D
• sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1CTWYQG0GQ4Y2VFNJTC5Y9H
• resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M1CTXGQ50GTF7R74N5F0AYK3

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

• abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-webpack)
• addTenWorkflow (nitro)
• cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
• hookWorkflow (tanstack-start)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

33 infra events

• cold-start-warmup · suite warmup (python) · at 21:14:09Z · abandoned wrun_41M1CTMMKH0GJHNMWTRT0RBSZQ · (+6 more)
• run-pickup-stall · sleepingWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB7
• run-pickup-stall · nullByteWorkflow (python) · at 21:14:25Z · abandoned wrun_41M1CTR9860GSR1V009T2GZWY1
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9FH0GQ8P3X9DK42SSWT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:14:26Z · abandoned wrun_41M1CTR9820GPXQ4EMDACJ3GB8
• run-pickup-stall · promiseAllWorkflow (python) · at 21:14:27Z · abandoned wrun_41M1CTR97V0GTFVK2A34P47G2P
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:14:59Z · abandoned wrun_41M1CTSAE60GTGN55WM85C4WRF
• run-pickup-stall · promiseAllWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FE0GZDVGGTYRREZFGE
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4FC0GT13M6TSBCMD2CF
• run-pickup-stall · sleepingWorkflow (python) · at 21:15:25Z · abandoned wrun_41M1CTT4GF0GMKRKA7ZPKGVQKA
• run-pickup-stall · nullByteWorkflow (python) · at 21:15:26Z · abandoned wrun_41M1CTT4TK0GZ59NA0E5DEECN9
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:16:07Z · abandoned wrun_41M1CTTA0V0GW8STRJ0NXH1MAK
• cold-start-warmup · suite warmup (tanstack-start) · at 21:16:29Z · abandoned wrun_01M1CTVQS8HB5PGN6S1W5JJ3GK
• cold-start-warmup · suite warmup (python) · at 21:17:08Z · abandoned wrun_01M1CTT3VDW1P4DARKPKNK56SV · (+7 more)
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZQS5VPNGWV6D9P0WTH
• run-pickup-stall · sleepingWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZXKFDZMW0BJ2PQSTKM
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZZEX0PC8YRTF8BCXJ9
• run-pickup-stall · promiseAllWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXRZR016BGCJPXEQCN0NT
• run-pickup-stall · nullByteWorkflow (python) · at 21:17:24Z · abandoned wrun_01M1CTXS01F6Q7CDA91SEW04AA
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:17:49Z · abandoned wrun_41M1CTW61K0GKDF638N6P0JFST
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:17:49Z · abandoned wrun_41M1CTX1XD0GZMJW4BJNHFQGRT
• run-pickup-stall · parallelSleepWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKSM4NA1F8PYJ58GY8E
• run-pickup-stall · nullByteWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKT6SQCSMMZVDCS78N1
• run-pickup-stall · sleepingWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKP72PTRG1PX6QAD913
• run-pickup-stall · promiseAllWorkflow (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKRFTNJ0EFDTE3WRYPV
• run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:18:24Z · abandoned wrun_01M1CTZKKMGB6EP5KBB0VYFFTT
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:18:47Z · abandoned wrun_41M1CTYXY30GGKEAW3PNEB5333
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E77VC437FEHQ0P588M8
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7EHTPTCK35VG9GA5CF
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:19:24Z · abandoned wrun_01M1CV1E7JA10XMP3BF8G2B77F
• run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK3ASZB6STVRTJ0HZ44
• run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:19:54Z · abandoned wrun_01M1CV2BK43AY290VT6J6GVK68
• run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:20:24Z · abandoned wrun_01M1CV38TQG0ZDSCF4TG5BHPH3

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
❌ ▲ Vercel Production	3570	8	742	4320
✅ 💻 Local Development	3922	0	558	4480
✅ 📦 Local Production	3922	0	558	4480
✅ 🐘 Local Postgres	3922	0	558	4480
✅ 🪟 Windows	320	0	0	320
❌ 🌐 Cross-language Conformance	0	9	132	141
✅ vercel-http-transport	817	0	143	960
✅ vercel-multi-region	27	0	0	27
✅ vercel-ws-transport	553	0	87	640
Total	17053	17	2778	19848

Details by Category

❌ ▲ Vercel Production

App	Passed	Failed	Skipped
✅ astro-node	132	0	28
✅ astro-quickjs	132	0	28
✅ example-node	132	0	28
✅ example-quickjs	132	0	28
✅ express-node	132	0	28
✅ express-quickjs	132	0	28
✅ fastify-node	132	0	28
✅ fastify-quickjs	132	0	28
✅ hono-node	132	0	28
✅ hono-quickjs	132	0	28
✅ nest-node	132	0	28
✅ nest-quickjs	132	0	28
✅ nextjs-turbopack-node	157	0	3
✅ nextjs-turbopack-quickjs	157	0	3
✅ nextjs-webpack-node	157	0	3
✅ nextjs-webpack-quickjs	157	0	3
✅ nitro-node	132	0	28
✅ nitro-quickjs	132	0	28
✅ nuxt-node	132	0	28
✅ nuxt-quickjs	132	0	28
❌ python-node	0	8	152
✅ sveltekit-node	151	0	9
✅ sveltekit-quickjs	151	0	9
✅ tanstack-start-node	132	0	28
✅ tanstack-start-quickjs	132	0	28
✅ vite-node	132	0	28
✅ vite-quickjs	132	0	28

✅ 💻 Local Development

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 📦 Local Production

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🐘 Local Postgres

App	Passed	Failed	Skipped
✅ astro-stable-node	134	0	26
✅ astro-stable-quickjs	134	0	26
✅ express-stable-node	134	0	26
✅ express-stable-quickjs	134	0	26
✅ fastify-stable-node	134	0	26
✅ fastify-stable-quickjs	134	0	26
✅ hono-stable-node	134	0	26
✅ hono-stable-quickjs	134	0	26
✅ nest-stable-node	134	0	26
✅ nest-stable-quickjs	134	0	26
✅ nextjs-turbopack-canary-node	141	0	19
✅ nextjs-turbopack-canary-quickjs	141	0	19
✅ nextjs-turbopack-stable-node	160	0	0
✅ nextjs-turbopack-stable-quickjs	160	0	0
✅ nextjs-webpack-canary-node	141	0	19
✅ nextjs-webpack-canary-quickjs	141	0	19
✅ nextjs-webpack-stable-node	160	0	0
✅ nextjs-webpack-stable-quickjs	160	0	0
✅ nitro-stable-node	134	0	26
✅ nitro-stable-quickjs	134	0	26
✅ nuxt-stable-node	134	0	26
✅ nuxt-stable-quickjs	134	0	26
✅ sveltekit-stable-node	153	0	7
✅ sveltekit-stable-quickjs	153	0	7
✅ tanstack-start-node	134	0	26
✅ tanstack-start-quickjs	134	0	26
✅ vite-stable-node	134	0	26
✅ vite-stable-quickjs	134	0	26

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	160	0	0
✅ nextjs-turbopack-quickjs	160	0	0

❌ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
❌ python	0	9	132

✅ vercel-http-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ hono	132	0	28
✅ nextjs-turbopack	157	0	3
✅ nitro	132	0	28
✅ vite	132	0	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

✅ vercel-ws-transport

App	Passed	Failed	Skipped
✅ example	132	0	28
✅ express	132	0	28
✅ nextjs-turbopack	157	0	3
✅ vite	132	0	28

---

📋 View full workflow run

[github-actions]

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario	outcome	events	virt	replay	violations
✅ smoke-no-steps	completed	3	0ms	ok	0
✅ smoke-one-step	completed	6	0ms	ok	0
✅ hook-at-step-started	completed	12	0ms	ok	0
✅ hook-at-step-completed	completed	12	0ms	ok	0
✅ hook-at-hook-created	completed	12	0ms	ok	0
✅ deadline-hook-wins	completed	7	1.0h	ok	0
✅ deadline-expires	completed	7	1.0h	ok	0
✅ long-sleep	completed	11	30.0d	ok	0
✅ hook-never-arrives	stalled	3	0ms	skipped	0
✅ step-retries-twice	completed	10	2.0s	ok	0
✅ parallel-steps	completed	9	0ms	ok	0
✅ hook-on-execution-state	completed	12	0ms	ok	0
✅ peek-hook-before-branch	completed	12	0ms	ok	0
✅ peek-hook-after-branch	completed	12	0ms	ok	0
✅ peek-hook-at-registration	completed	12	0ms	ok	0
✅ race-hook-before-probe	completed	12	0ms	ok	0
✅ race-hook-after-probe	completed	12	0ms	ok	0
✅ race-duplicate-delivery	completed	13	0ms	ok	0
✅ attr-hook-before-step	completed	11	0ms	ok	0
✅ attr-hook-after-step	completed	11	0ms	ok	0
✅ attr-from-step-body	completed	13	0ms	ok	0
✅ fork-hook-after-timeout	completed	14	1.0m	ok	0
✅ fork-hook-before-timeout	completed	14	1.0m	ok	0
✅ count-hook-after-timeout	completed	17	1.0m	ok	0
✅ count-hook-before-timeout	completed	20	1.0m	ok	0
✅ stale-read-step-count-fork	completed	20	1.0m	ok	0
✅ stale-read-equal-step-counts	completed	14	1.0m	ok	0
✅ step-vs-step-fork	completed	12	0ms	ok	0
✅ step-vs-step-fork-fenced	completed	12	0ms	ok	0
✅ fence-catches-benign-direction	completed	12	5ms	ok	0
✅ in-flight-before-decision	completed	17	1.0m	ok	0
❌ in-flight-before-decision-counted	completed	17	1.0m	ok	0
✅ in-flight-after-decision	completed	19	2.0m	ok	0
✅ stale-read-step-count-fork-fenced	completed	20	1.0m	ok	0
✅ fork-hook-wins	completed	13	1.0m	ok	0
✅ fork-timeout-wins	completed	13	1.0m	ok	0
✅ unclaimed-payload-under-fork	completed	17	1.0m	ok	0
✅ claimed-payload-under-fork	completed	17	1.0m	ok	0
✅ writers-independent-step-bodies	completed	12	0ms	ok	0
✅ writers-scripted-tempo	completed	12	0ms	ok	0
✅ cancel-mid-step	cancelled	7	0ms	skipped	0

Full trace: world-sim.txt

[github-actions]

📊 Workflow Benchmarks

commit 9f03030 · Mon, 31 Aug 2026 21:38:22 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1386 (+657%) 🔻	1500 🔴 (+46%) 🔻	1543 🔴 (+43%) 🔻	1646 🔴 (+36%) 🔻	30
TTFS	stream	250 (+16%) 🔻	1454 🔴 (+35%) 🔻	1519 🔴 (+37%) 🔻	1897 🔴 (+26%) 🔻	30
TTFS	hook + stream	1709 (+41%) 🔻	1804 🔴 (+33%) 🔻	1877 🔴 (+34%) 🔻	1976 🔴 (+37%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	612 (-4.1%)	1255 (-27%) 💚	2513 (+39%) 🔻	6361 (+252%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	2114 (+18%) 🔻	6996 (+132%) 🔻	11047 (+169%) 🔻	11298 (+165%) 🔻	10
STSO	1020 steps (inline)	108 (+4.9%)	166 (+21%) 🔻	185 (+17%) 🔻	267 (+5.5%)	1019
WO	1020 steps	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	164301 (+19%) 🔻	1
CRTT	first chunk (pooled)	123 (+24%) 🔻	165 (-11%)	194 (-18%) 💚	284 (+8.8%)	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	151 (+12%)	195 (+8%)	270 (-5%)	495 (+14%)	146 (-20%)	10
size sweep (100/s, 160B-12KB)	135 (-7%)	180 (-12%)	354 (-19%)	632 (-34%)	128 (-39%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	150 (+13%)	151 (-14%)	179 (-35%)	454 (-51%)	170 (-64%)	3
replay eve-gpt-5.6-sol-2000t (1x)	159 (-3%)	155 (-20%)	199 (-33%)	304 (-81%)	199 (-83%)	2
replay eve-gpt-5.6-sol-2000t (2x)	147 (+4%)	206 (-29%)	274 (-37%)	418 (-47%)	229 (-57%)	3

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 136556ms → this run 162927ms (Δ +26371ms, +19%)

100-150 ms ███████████┃████████████ main 863 this 433 -430
150-200 ms ████░░░░░░░░░░┃ main 130 this 530 +400
200-250 ms ┃ main 15 this 41 +26
250-300 ms ┃ main 8 this 7 -1
300-350 ms ┃ main 2 this 4 +2
350-400 ms ┃ main 1 this 1 +0
400-450 ms ┃ main 0 this 1 +1
500-550 ms ┃ main 0 this 2 +2

📈 CRTT drill-down vs main (RTT distributions & profiles)

variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▂▁··· 152.3 (-3%) 141 (+1%) 270 (-5%) 495 (+14%) 3000
sweep ······▂█▂▁··· 150.4 (-10%) 135 (+2%) 354 (-19%) 632 (-34%) 3000
gw 1x ·····▁▂█▁▁··· 131.2 (-13%) 126 (-2%) 179 (-35%) 454 (-51%) 5295
eve 1x ······▃█▁···· 134.4 (-28%) 126 (-8%) 199 (-33%) 304 (-81%) 5186
eve 2x ·····▁▁█▃···· 171.3 (-18%) 161 (-4%) 274 (-37%) 418 (-47%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control █▆▇▆▅▁▃▁▆▁ 133–172ms
sweep █▇▃▁▅▅▂▂▂▁ 128–189ms
gw 1x ▅█▃▃▁▄▂▃▁▃ 122–150ms
eve 1x ▆▂▄▃▃▃█▆▃▁ 119–156ms
eve 2x ▂▁▃▂▁▅▅█▅▃ 142–220ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep ▇▆█▆▅▁▄ 148–152ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▁▆▅▆▃▄▅▄█▄ 35–57ms
sweep ▁▁▃▅█▄▅▃▁▄ 50–74ms
gw 1x ▄█▁▁▄▅▅▆▁▂ 38–47ms
eve 1x ▇▅▃▇▇▄█▃▃▁ 21–28ms
eve 2x ▇█▃▂█▅▅▁█▃ 24–31ms

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.}

_{The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

[VaguelySerious]

AI review: blocking issues found

[github-actions]

Framework	Flow route	Step reg.	Framework output
hono	200.9 KiB (±0)	40.4 KiB (±0)	1.76 MiB (-45 B)
nextjs-turbopack	206.3 KiB (±0)	439 B (±0)	765.5 KiB (+338 B)

About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

9f03030 · run

[alangenfeld]

local agent review surfaced:

The new speculative key lookup always calls:

resolveRunEncryptionKey(world,runId)

This discards the WorkflowRun.deploymentId that the previous implementation supplied by calling getEncryptionKeyForRun(workflowRun).

That breaks encrypted Node replays outside a Vercel Function—such as local tooling or external runners—because world-vercel cannot resolve a key from a bare run ID without { deploymentId }; it returns undefined. The replay then attempts to hydrate encrypted input/events without the key. It can also select the wrong key when deployment routing matters.

The lookup should start from the streamed run_created/runInput deployment context, for example:

resolveRunEncryptionKey(world,runId,{deploymentId: workflow.deploymentId,})

or defer until the authoritative WorkflowRun is available and pass that object.

[github-actions]

No backport to stable for e9d5c56 (AI decision).

This is a replay-startup performance optimization: it prepares payloads as event frames stream in, lazily resolves the encryption key, skips Node-only work for QuickJS, and adds a new replayEventObserver option to the CreateEventParams public World interface. None of it fixes a user-visible defect on stable — the incidental hardening (setup failures classified as run_failed, unhandled-rejection guards, removal of the primitive memoization size cap) exists to support the new streaming path rather than to repair existing behavior, and it touches the hot replay path plus a World API surface, which is exactly the kind of churn a maintenance line should not take.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e9d5c56701821b090108a85b74bf8b0cbef8ea8e