fix(world-local): queue - bound stalled delivery timeouts by AndrewBarba · Pull Request #3255 · vercel/workflow

AndrewBarba · 2026-07-31T14:50:38Z

Description

Two intermittent eve e2e failures timed out with no events while local-world queue deliveries remained in flight:

@workflow/world-local configured undici with headersTimeout: 0 and no body timeout. If a delivery handler accepted a connection and then stopped making progress, the transport never rejected, so the durable message could not be retried. The socket hang up errors seen during teardown were pending deliveries being terminated, not the initiating failure.

This intentionally brings world-local on main in line with the transport resilience policy recently added to world-vercel on stable in #3169: 30-second header and body stall limits, environment overrides, and 0 as an explicit unbounded opt-out. The retry implementation is different by design: world-vercel uses RetryAgent and event-specific POST retries, while world-local already owns a durable delivery loop, so transport timeouts redeliver the same queued message there.

This change bounds response-header and response-body waits at 30 seconds by default, keeps body consumption inside the retry boundary, and redelivers stalled durable messages. Both limits can be overridden with WORKFLOW_LOCAL_HEADERS_TIMEOUT_MS and WORKFLOW_LOCAL_BODY_TIMEOUT_MS; setting either to 0 preserves an explicit unbounded opt-out.

How did you test your changes?

pnpm --filter @workflow/world-local build
pnpm --filter @workflow/world-local typecheck
pnpm --filter @workflow/world-local test (12 files, 498 tests)
Added real HTTP socket tests proving header and body stalls are redelivered.
pnpm test:docs was attempted, but this fresh checkout cannot complete the required workspace build because Rust/wasm32-unknown-unknown is not installed; CI remains the full docs check.

PR Checklist - Required to merge

📦 Added a patch changeset for @workflow/world-local
🔒 Commit includes DCO sign-off
📝 Pinged @vercel/workflow for review

Signed-off-by: Andrew Barba <barba@hey.com>

vercel · 2026-07-31T14:50:41Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview	Jul 31, 2026 2:53pm
example-nextjs-workflow-webpack	Ready	Preview	Jul 31, 2026 2:53pm
example-workflow	Ready	Preview	Jul 31, 2026 2:53pm
workbench-astro-workflow	Ready	Preview	Jul 31, 2026 2:53pm
workbench-express-workflow	Ready	Preview	Jul 31, 2026 2:53pm
workbench-fastify-workflow	Ready	Preview	Jul 31, 2026 2:53pm
workbench-hono-workflow	Ready	Preview	Jul 31, 2026 2:53pm
workbench-nestjs-workflow	Ready	Preview	Jul 31, 2026 2:53pm
workbench-nitro-workflow	Ready	Preview	Jul 31, 2026 2:53pm
workbench-nuxt-workflow	Ready	Preview	Jul 31, 2026 2:53pm
workbench-sveltekit-workflow	Ready	Preview	Jul 31, 2026 2:53pm
workbench-tanstack-start-workflow	Ready	Preview	Jul 31, 2026 2:53pm
workbench-vite-workflow	Ready	Preview	Jul 31, 2026 2:53pm
workflow-docs	Ready	Preview, v0	Jul 31, 2026 2:53pm
workflow-swc-playground	Ready	Preview	Jul 31, 2026 2:53pm
workflow-tarballs	Ready	Preview	Jul 31, 2026 2:53pm
workflow-web	Ready	Preview	Jul 31, 2026 2:53pm

changeset-bot · 2026-07-31T14:50:42Z

🦋 Changeset detected

Latest commit: aecf3da

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 18 packages

Name	Type
@workflow/world-local	Patch
@workflow/cli	Patch
@workflow/core	Patch
@workflow/vitest	Patch
@workflow/web	Patch
@workflow/world-postgres	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/builders	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/web-shared	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

AndrewBarba · 2026-07-31T14:50:43Z

@vercel/workflow ready for review.

github-actions · 2026-07-31T14:51:00Z

📊 Workflow Benchmarks

commit aecf3da · Fri, 31 Jul 2026 15:11:37 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1239 (+23%) 🔻	1300 🔴 (+19%) 🔻	1323 🔴 (+20%) 🔻	1350 🔴 (-16%) 💚	30
TTFS	stream	251 (-32%) 💚	1280 🔴 (+21%) 🔻	1295 🔴 (+20%) 🔻	1316 🔴 (+18%) 🔻	30
TTFS	hook + stream	654 (+31%) 🔻	1583 🔴 (+13%)	1673 🔴 (+14%)	2050 🔴 (+37%) 🔻	30
STSO	1020 steps (inline)	164 (-11%)	447 (-9.0%)	503 (-9.9%)	692 (-8.5%)	1016
STSO	1020 steps (queue-hop)	2521 (+23%) 🔻	3373 (+21%) 🔻	3373 (+21%) 🔻	3373 (+21%) 🔻	3
WO	1020 steps	388322 (-10%)	388322 (-10%)	388322 (-10%)	388322 (-10%)	1
SL	stream latency	101 (+17%) 🔻	144 🔴 (-1.4%)	159 🔴 (-14%)	182 🔴 (-29%) 💚	30
SO	stream overhead (text)	100 (-26%) 💚	146 (-71%) 💚	169 (-82%) 💚	221 (-82%) 💚	30
SO	stream overhead (structured)	108 (-3.6%)	163 (-20%) 💚	183 (-30%) 💚	282 (-36%) 💚	30

📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 423669ms → this run 378663ms (Δ -45006ms, -11%)

 150-200 ms █░░░┃ main 10 this 40 +30
200-250 ms ████████░░░░░░┃ main 63 this 119 +56
250-300 ms ██████████████░░░░░┃ main 108 this 156 +48
300-350 ms █████████████┃███ main 136 this 110 -26
350-400 ms ██████████████████░┃ main 143 this 159 +16
400-450 ms ████████████████████░░░┃ main 161 this 190 +29
450-500 ms ████████████████┃███ main 161 this 131 -30
500-550 ms ███████┃██████ main 111 this 65 -46
550-600 ms ██┃█████ main 63 this 20 -43
600-650 ms ┃███ main 28 this 7 -21
650-700 ms ┃█ main 13 this 9 -4
700-750 ms ┃ main 6 this 3 -3
750-800 ms ┃ main 6 this 6 +0
800-850 ms ┃ main 2 this 0 -2
850-900 ms ┃ main 1 this 0 -1
900-950 ms ┃ main 2 this 0 -2
1100-1150 ms ┃ main 2 this 0 -2
1150-1200 ms ┃ main 0 this 1 +1

1020 steps (queue-hop)

Cumulative STSO time: main 7586ms → this run 8500ms (Δ +914ms, +12%)

2000-2500 ms ┃███████████ main 1 this 0 -1
2500-3000 ms ███████████████████████┃ main 2 this 2 +0
3000-3500 ms ░░░░░░░░░░░┃ main 0 this 1 +1

ℹ️ Metric definitions & methodology

_{The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: █ is main, ┃ marks where this run lands, ░ bridges the gap when this run has more samples in a bucket.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000}

_{All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.}

_{Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.}

github-actions · 2026-07-31T14:51:02Z

🧪 E2E Test Results

✅ All tests passed

E2E Test Summary

Summary

	Passed	Skipped	Total
✅ ▲ Vercel Production	1455	239	1694
✅ 💻 Local Development	1621	227	1848
✅ 📦 Local Production	1621	227	1848
✅ 🐘 Local Postgres	1621	227	1848
✅ 🪟 Windows	154	0	154
✅ 📋 Other	1020	212	1232
✅ vercel-multi-region	27	0	27
Total	7519	1132	8651

Details by Category

✅ ▲ Vercel Production

App	Passed	Skipped
✅ astro	126	28
✅ example	126	28
✅ express	126	28
✅ fastify	126	28
✅ hono	126	28
✅ nextjs-turbopack	151	3
✅ nextjs-webpack	151	3
✅ nitro	126	28
✅ nuxt	126	28
✅ sveltekit	145	9
✅ vite	126	28

✅ 💻 Local Development

App	Passed	Skipped
✅ astro-stable	128	26
✅ express-stable	128	26
✅ fastify-stable	128	26
✅ hono-stable	128	26
✅ nextjs-turbopack-canary	135	19
✅ nextjs-turbopack-stable	154	0
✅ nextjs-webpack-canary	135	19
✅ nextjs-webpack-stable	154	0
✅ nitro-stable	128	26
✅ nuxt-stable	128	26
✅ sveltekit-stable	147	7
✅ vite-stable	128	26

✅ 📦 Local Production

App	Passed	Skipped
✅ astro-stable	128	26
✅ express-stable	128	26
✅ fastify-stable	128	26
✅ hono-stable	128	26
✅ nextjs-turbopack-canary	135	19
✅ nextjs-turbopack-stable	154	0
✅ nextjs-webpack-canary	135	19
✅ nextjs-webpack-stable	154	0
✅ nitro-stable	128	26
✅ nuxt-stable	128	26
✅ sveltekit-stable	147	7
✅ vite-stable	128	26

✅ 🐘 Local Postgres

App	Passed	Skipped
✅ astro-stable	128	26
✅ express-stable	128	26
✅ fastify-stable	128	26
✅ hono-stable	128	26
✅ nextjs-turbopack-canary	135	19
✅ nextjs-turbopack-stable	154	0
✅ nextjs-webpack-canary	135	19
✅ nextjs-webpack-stable	154	0
✅ nitro-stable	128	26
✅ nuxt-stable	128	26
✅ sveltekit-stable	147	7
✅ vite-stable	128	26

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack	154	0	0

✅ 📋 Other

App	Passed	Skipped
✅ e2e-local-dev-nest-stable	128	26
✅ e2e-local-dev-tanstack-start-	128	26
✅ e2e-local-postgres-nest-stable	128	26
✅ e2e-local-postgres-tanstack-start-	128	26
✅ e2e-local-prod-nest-stable	128	26
✅ e2e-local-prod-tanstack-start-	128	26
✅ e2e-vercel-prod-nest	126	28
✅ e2e-vercel-prod-tanstack-start	126	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

📋 View full workflow run

github-actions · 2026-07-31T15:29:04Z

No backport to stable for 2677653 (AI decision).

The underlying hang (undici headersTimeout: 0 with no body timeout in packages/world-local/src/queue.ts) does exist on stable, but this fix depends on behavior that only exists on main: the transport-level retry/delivery-counter loop that redelivers the same durable message (the commit edits that try/catch, and stable's createQueue has no such catch at all — a transport throw there is not retried). Applying just the bounded timeouts to stable would convert stalled deliveries into dropped messages instead of redeliveries, and it also changes an existing default from unbounded to 30s on a maintenance line. If the hang is judged to affect stable users, backport the prerequisite transport-retry change together with this one via workflow_dispatch.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

2677653759aa34c5c9fe28950fe1a02cec294551

…ent-guard * origin/main: ci: stop deploying changeset-release/main, run its e2e against production (#3243) fix(world-local): bound stalled queue deliveries (#3255) Sort imports in runtime.ts and step-executor.ts (#3241)

fix(world-local): bound stalled queue deliveries
aecf3da
Signed-off-by: Andrew Barba <barba@hey.com>

AndrewBarba requested review from a team and ijjk as code owners July 31, 2026 14:50

vercelBottemporarily deployed to Preview – workflow-docs July 31, 2026 14:50 Inactive

vercelBotdeployed to Preview – workflow-web July 31, 2026 14:51 View deployment

vercelBotdeployed to Preview – workbench-hono-workflow July 31, 2026 14:51 View deployment

vercelBotdeployed to Preview – workflow-tarballs July 31, 2026 14:51 View deployment

vercelBotdeployed to Preview – workbench-astro-workflow July 31, 2026 14:51 View deployment

vercelBotdeployed to Preview – workbench-express-workflow July 31, 2026 14:51 View deployment

vercelBotdeployed to Preview – workbench-nestjs-workflow July 31, 2026 14:51 View deployment

vercelBotdeployed to Preview – workbench-fastify-workflow July 31, 2026 14:51 View deployment

vercelBotdeployed to Preview – workbench-vite-workflow July 31, 2026 14:51 View deployment

vercelBotdeployed to Preview – workbench-tanstack-start-workflow July 31, 2026 14:51 View deployment

vercelBotdeployed to Preview – workbench-nitro-workflow July 31, 2026 14:51 View deployment

vercelBotdeployed to Preview – workbench-sveltekit-workflow July 31, 2026 14:51 View deployment

vercelBotdeployed to Preview – example-nextjs-workflow-turbopack July 31, 2026 14:52 View deployment

vercelBotdeployed to Preview – example-nextjs-workflow-webpack July 31, 2026 14:52 View deployment

vercelBotdeployed to Preview – example-workflow July 31, 2026 14:52 View deployment

vercelBotdeployed to Preview – workbench-nuxt-workflow July 31, 2026 14:52 View deployment

vercelBotdeployed to Preview – workflow-swc-playground July 31, 2026 14:53 View deployment

karthikscale3 approved these changes Jul 31, 2026
View reviewed changes

karthikscale3 merged commit 2677653 into mainJul 31, 2026
107 checks passed

karthikscale3 deleted the barba/fix-world-local-queue-timeouts branch July 31, 2026 15:27

github-actionsBot mentioned this pull request Jul 31, 2026
Version Packages (beta) #3256
Merged

This was referenced Aug 1, 2026
Version Packages (beta) ben-vargas/workflow#104
Open
Version Packages (beta) Thegreatsura/workflow#1
Open
Version Packages (beta) Jokacar10/jkworkflow#391
Closed
Version Packages (beta) erickirt/workflow#340
Closed

vercel-gh-bot-2Bot mentioned this pull request Aug 10, 2026
Quarantined local Workflow runs starve all new turn deliveries (silent ~65min hang, 503 socket hang up) vercel/eve#961
Closed

Navigation Menu

Uh oh!

fix(world-local): queue - bound stalled delivery timeouts - #3255

fix(world-local): queue - bound stalled delivery timeouts#3255
karthikscale3 merged 1 commit into
mainfrom
barba/fix-world-local-queue-timeouts

AndrewBarba commented Jul 31, 2026•
edited
Loading

Uh oh!

vercelBot commented Jul 31, 2026•
edited
Loading

Uh oh!

changeset-botBot commented Jul 31, 2026

Uh oh!

AndrewBarba commented Jul 31, 2026

Uh oh!

github-actionsBot commented Jul 31, 2026•
edited
Loading

Uh oh!

github-actionsBot commented Jul 31, 2026•
edited
Loading

Uh oh!

Uh oh!

github-actionsBot commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

AndrewBarba commented Jul 31, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How did you test your changes?

PR Checklist - Required to merge

Uh oh!

vercelBot commented Jul 31, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-botBot commented Jul 31, 2026

🦋 Changeset detected

Uh oh!

AndrewBarba commented Jul 31, 2026

Uh oh!

github-actionsBot commented Jul 31, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Workflow Benchmarks

Uh oh!

github-actionsBot commented Jul 31, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 E2E Test Results

E2E Test Summary

Uh oh!

Uh oh!

github-actionsBot commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AndrewBarba commented Jul 31, 2026•
edited
Loading

vercelBot commented Jul 31, 2026•
edited
Loading

github-actionsBot commented Jul 31, 2026•
edited
Loading

github-actionsBot commented Jul 31, 2026•
edited
Loading