[core] Enforce maxRetries for steps that time out by VaguelySerious · Pull Request #3035 · vercel/workflow

VaguelySerious · 2026-07-21T19:14:43Z

Problem

A step that throws is bounded correctly by maxRetries. A step that times out (the platform hard-kills the function mid-run) is not — it retries unbounded.

Chain:

Every step_started increments attempt (world storage, "increment on every start").
A timeout kills the function while awaiting, so no step_failed/step_retrying is written — step.error stays null. The queue then redelivers the step and it runs again.
The pre-body max-retries guard in step-executor.ts is gated on && step.error, so it never fires for timeouts. The post-body guard only runs when the body actually throws — a timeout never reaches it.

Additionally, the optimistic-inline-start path synthesizes step.attempt = 1, so even the world-returned attempt can't be trusted inline.

Minimal repro (retries forever instead of stopping at the default 3):

asyncfunctiontimeoutWorkflow(){"use workflow";awaittimeoutStep();}asyncfunctiontimeoutStep(){"use step";awaitnewPromise((r)=>setTimeout(r,15*60*1000));// killed by function timeout}

Fix

Enforce the retry ceiling before the body runs, via a new authoritativeAttempt param on executeStep. Callers supply a count that reflects real attempts:

Inline (combined handler): the number of step_started events already in the event log for the step, plus one for the attempt about to run. The log is authoritative (the optimistic path synthesizes attempt = 1; concurrent double-starts are prevented by the atomic create-claim / single-flight).
Background (queue-dispatched): the queue delivery count (metadata.attempt), which increments on the visibility-timeout redelivery a timed-out step produces.

When the attempt exceeds maxRetries + 1, the step is failed (a catchable FatalError bubbled to the workflow) without starting another attempt. Thrown-error exhaustion is unchanged: it still terminates one attempt earlier via the post-body guard, with the thrown error as cause.

Validation

New unit tests (step-executor.test.ts): the ceiling fails a step without running the body — and without writing a new step_started — once the attempt exceeds maxRetries + 1, and permits the final allowed attempt (maxRetries + 1).
Full packages/core runtime suite green (336 tests).
A true end-to-end timeout can't be reproduced locally (world-local can't hard-kill a function invocation); the enforcement decision is unit-tested at executeStep, and both call sites are thin param passes.

Notes / open questions

For the pure-timeout exhaustion the FatalError has no cause (there is no recorded error to attach). Matches the shape of the existing max-retries error; happy to attach a best-effort prior-error cause if preferred.

🤖 Generated with Claude Code

A step that is hard-killed by the platform function timeout writes no step_failed/step_retrying, so `step.error` stays null and the error-based max-retries guards never fire. Each redelivery re-runs step_started (incrementing the attempt), so a timing-out step retried without bound instead of stopping at maxRetries. Enforce the retry ceiling BEFORE running the body, via a new `authoritativeAttempt` param on executeStep: - Inline (combined handler): count the step_started events already in the event log for the step (+1 for this attempt). The log is authoritative because the optimistic-start path synthesizes step.attempt = 1. - Background (queue-dispatched): the queue delivery count (metadata.attempt), which increments on the visibility-timeout redelivery a timed-out step produces. When the attempt exceeds maxRetries + 1 the step is failed without starting another attempt. Thrown-error exhaustion is unchanged — it still terminates via the post-body guard one attempt earlier, with the thrown error as cause. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

changeset-bot · 2026-07-21T19:14:48Z

🦋 Changeset detected

Latest commit: 75b39f3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages

Name	Type
@workflow/core	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/nuxt	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2026-07-21T19:14:49Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview, Comment	Jul 21, 2026 11:05pm
example-nextjs-workflow-webpack	Ready	Preview, Comment	Jul 21, 2026 11:05pm
example-workflow	Ready	Preview, Comment	Jul 21, 2026 11:05pm
workbench-astro-workflow	Ready	Preview, Comment	Jul 21, 2026 11:05pm
workbench-express-workflow	Ready	Preview, Comment	Jul 21, 2026 11:05pm
workbench-fastify-workflow	Ready	Preview, Comment	Jul 21, 2026 11:05pm
workbench-hono-workflow	Ready	Preview, Comment	Jul 21, 2026 11:05pm
workbench-nestjs-workflow	Ready	Preview, Comment	Jul 21, 2026 11:05pm
workbench-nitro-workflow	Ready	Preview, Comment	Jul 21, 2026 11:05pm
workbench-nuxt-workflow	Building	Preview, Comment	Jul 21, 2026 11:05pm
workbench-sveltekit-workflow	Ready	Preview, Comment	Jul 21, 2026 11:05pm
workbench-tanstack-start-workflow	Ready	Preview, Comment	Jul 21, 2026 11:05pm
workbench-vite-workflow	Ready	Preview, Comment	Jul 21, 2026 11:05pm
workflow-docs	Ready	Preview, Comment, Open in v0	Jul 21, 2026 11:05pm
workflow-swc-playground	Ready	Preview, Comment	Jul 21, 2026 11:05pm
workflow-tarballs	Ready	Preview, Comment	Jul 21, 2026 11:05pm
workflow-web	Ready	Preview, Comment	Jul 21, 2026 11:05pm

github-actions · 2026-07-21T19:15:01Z

🧪 E2E Test Results

✅ All tests passed

Summary

	Passed	Skipped	Total
✅ ▲ Vercel Production	1455	239	1694
✅ 💻 Local Development	1621	227	1848
✅ 📦 Local Production	1621	227	1848
✅ 🐘 Local Postgres	1621	227	1848
✅ 🪟 Windows	154	0	154
✅ 📋 Other	1020	212	1232
✅ vercel-multi-region	27	0	27
Total	7519	1132	8651

Details by Category

✅ ▲ Vercel Production

App	Passed	Skipped
✅ astro	126	28
✅ example	126	28
✅ express	126	28
✅ fastify	126	28
✅ hono	126	28
✅ nextjs-turbopack	151	3
✅ nextjs-webpack	151	3
✅ nitro	126	28
✅ nuxt	126	28
✅ sveltekit	145	9
✅ vite	126	28

✅ 💻 Local Development

App	Passed	Skipped
✅ astro-stable	128	26
✅ express-stable	128	26
✅ fastify-stable	128	26
✅ hono-stable	128	26
✅ nextjs-turbopack-canary	135	19
✅ nextjs-turbopack-stable	154	0
✅ nextjs-webpack-canary	135	19
✅ nextjs-webpack-stable	154	0
✅ nitro-stable	128	26
✅ nuxt-stable	128	26
✅ sveltekit-stable	147	7
✅ vite-stable	128	26

✅ 📦 Local Production

App	Passed	Skipped
✅ astro-stable	128	26
✅ express-stable	128	26
✅ fastify-stable	128	26
✅ hono-stable	128	26
✅ nextjs-turbopack-canary	135	19
✅ nextjs-turbopack-stable	154	0
✅ nextjs-webpack-canary	135	19
✅ nextjs-webpack-stable	154	0
✅ nitro-stable	128	26
✅ nuxt-stable	128	26
✅ sveltekit-stable	147	7
✅ vite-stable	128	26

✅ 🐘 Local Postgres

App	Passed	Skipped
✅ astro-stable	128	26
✅ express-stable	128	26
✅ fastify-stable	128	26
✅ hono-stable	128	26
✅ nextjs-turbopack-canary	135	19
✅ nextjs-turbopack-stable	154	0
✅ nextjs-webpack-canary	135	19
✅ nextjs-webpack-stable	154	0
✅ nitro-stable	128	26
✅ nuxt-stable	128	26
✅ sveltekit-stable	147	7
✅ vite-stable	128	26

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack	154	0	0

✅ 📋 Other

App	Passed	Skipped
✅ e2e-local-dev-nest-stable	128	26
✅ e2e-local-dev-tanstack-start-	128	26
✅ e2e-local-postgres-nest-stable	128	26
✅ e2e-local-postgres-tanstack-start-	128	26
✅ e2e-local-prod-nest-stable	128	26
✅ e2e-local-prod-tanstack-start-	128	26
✅ e2e-vercel-prod-nest	126	28
✅ e2e-vercel-prod-tanstack-start	126	28

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

📋 View full workflow run

github-actions · 2026-07-21T19:15:01Z

📊 Workflow Benchmarks

commit 75b39f3 · Tue, 21 Jul 2026 23:17:19 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	129 (+23%) 🔻	250 🔴 (-39%) 💚	353 🔴 (-37%) 💚	441 (-88%) 💚	30
TTFS	stream	108 (-28%) 💚	256 🔴 (-45%) 💚	333 🔴 (-51%) 💚	405 (-86%) 💚	30
TTFS	hook + stream	302 (+11%)	463 🔴 (-15%)	549 🔴 (-31%) 💚	888 🔴 (-76%) 💚	30
STSO	1020 steps (1-20)	225 (+25%) 🔻	364 🔴 (+11%)	429 🔴 (+3.6%)	512 🔴 (+23%) 🔻	19
STSO	1020 steps (101-120)	272 (+14%)	439 🔴 (+20%) 🔻	460 🔴 (+8.7%)	1343 🔴 (+198%) 🔻	19
STSO	1020 steps (1001-1020)	692 (+3.7%)	839 🔴 (-1.6%)	1024 🔴 (+9.8%)	1037 🔴 (+3.6%)	19
WO	1020 steps	566144 (+3.6%)	566144 (+3.6%)	566144 (+3.6%)	566144 (+3.6%)	1
SL	stream latency	81 (-9.0%)	124 🔴 (-3.1%)	147 🔴 (-6.4%)	282 🔴 (+26%) 🔻	30

📜 Previous results (3)

`f7f8745`

Tue, 21 Jul 2026 22:05:55 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	148 (+41%) 🔻	268 🔴 (-34%) 💚	379 🔴 (-32%) 💚	439 (-88%) 💚	30
TTFS	stream	146 (-2.0%)	203 🔴 (-56%) 💚	282 (-59%) 💚	429 (-85%) 💚	30
TTFS	hook + stream	327 (+20%) 🔻	486 🔴 (-10%)	571 🔴 (-29%) 💚	962 🔴 (-74%) 💚	30
STSO	1020 steps (1-20)	187 (+3.9%)	315 🔴 (-4.3%)	334 🔴 (-19%) 💚	352 🔴 (-15%) 💚	19
STSO	1020 steps (101-120)	216 (-9.6%)	338 🔴 (-7.7%)	432 🔴 (+2.1%)	670 🔴 (+49%) 🔻	19
STSO	1020 steps (1001-1020)	582 (-13%)	754 🔴 (-12%)	796 🔴 (-15%)	925 🔴 (-7.6%)	19
WO	1020 steps	519288 (-5.0%)	519288 (-5.0%)	519288 (-5.0%)	519288 (-5.0%)	1
SL	stream latency	81 (-9.0%)	116 🔴 (-9.4%)	134 🔴 (-15%)	187 🔴 (-17%) 💚	30

`67b5082`

Tue, 21 Jul 2026 20:47:19 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	141 (+27%) 🔻	272 🔴 (+22%) 🔻	337 🔴 (-17%) 💚	411 (-22%) 💚	30
TTFS	stream	129 (-5.1%)	341 🔴 (+18%) 🔻	408 🔴 (+13%)	482 (-40%) 💚	30
TTFS	hook + stream	319 (-9.1%)	500 🔴 (-1.6%)	659 🔴 (+14%)	852 🔴 (+38%) 🔻	30
STSO	1020 steps (1-20)	187 (+9.4%)	290 🔴 (+9.4%)	292 🔴 (-9.3%)	300 🔴 (-18%) 💚	19
STSO	1020 steps (101-120)	265 (+29%) 🔻	382 🔴 (+25%) 🔻	467 🔴 (+37%) 🔻	613 🔴 (+30%) 🔻	19
STSO	1020 steps (1001-1020)	631 (+7.7%)	788 🔴 (+10%)	896 🔴 (+15%) 🔻	1301 🔴 (+59%) 🔻	19
WO	1020 steps	554332 (+5.1%)	554332 (+5.1%)	554332 (+5.1%)	554332 (+5.1%)	1
SL	stream latency	96 (+26%) 🔻	187 🔴 (+80%) 🔻	264 🔴 (+59%) 🔻	365 🔴 (+51%) 🔻	30

`46b0233`

Tue, 21 Jul 2026 19:30:48 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	139 (+25%) 🔻	286 🔴 (+28%) 🔻	313 🔴 (-23%) 💚	2578 🔴 (+390%) 🔻	30
TTFS	stream	138 (+1.5%)	268 🔴 (-7.3%)	351 🔴 (-3.0%)	730 🔴 (-9.1%)	30
TTFS	hook + stream	269 (-23%) 💚	406 🔴 (-20%) 💚	442 🔴 (-23%) 💚	611 🔴 (-1.0%)	30
STSO	1020 steps (1-20)	187 (+9.4%)	287 🔴 (+8.3%)	316 🔴 (-1.9%)	374 🔴 (+2.7%)	19
STSO	1020 steps (101-120)	242 (+17%) 🔻	365 🔴 (+20%) 🔻	381 🔴 (+11%)	446 🔴 (-5.5%)	19
STSO	1020 steps (1001-1020)	549 (-6.3%)	667 🔴 (-6.6%)	739 🔴 (-4.9%)	788 🔴 (-3.9%)	19
WO	1020 steps	518699 (-1.6%)	518699 (-1.6%)	518699 (-1.6%)	518699 (-1.6%)	1
SL	stream latency	90 (+18%) 🔻	163 🔴 (+57%) 🔻	215 🔴 (+30%) 🔻	431 🔴 (+78%) 🔻	30

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt)}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120}

_{All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.}

_{Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.}

Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Peter Wielander <mittgfu@gmail.com>

karthikscale3

Re-reviewed after the background retry-count fix. The correctness blocker is resolved; remaining performance concern is non-blocking.

Deriving an inline step's attempt number by scanning the cumulative event log for step_started events ran for every inline execution, which is O(n²) across a long sequential workflow. A lazy inline step is brand-new by construction (it only enters the batch with no step_created yet), so it has zero prior starts and is always attempt 1 — no scan needed. Reserve the scan for owned-recovery re-runs (this message re-executing a step it crashed/timed out on), which are uncommon and few per batch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-21T23:32:24Z

No backport to stable for 9177ba8 (AI decision).

This fix targets the main-only step execution architecture: packages/core/src/runtime/step-executor.ts does not exist on stable (verified via git ls-tree), and the modified runtime.ts paths (optimistic inline starts, lazy steps, authoritativeAttempt plumbing) are part of the combined-handler refactor absent from stable. Moreover, the underlying bug appears not to exist on stable: its step-handler.ts already enforces step.attempt > maxRetries + 1 after step_started without the step.error gate that caused unbounded retries on main, so timed-out steps are already bounded there.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

9177ba83d3168866d13ff34ca3d651312d1d87d2

vercelBotdeployed to Preview – workbench-hono-workflow July 21, 2026 19:15 View deployment

vercelBotdeployed to Preview – workbench-fastify-workflow July 21, 2026 19:15 View deployment

vercelBotdeployed to Preview – workbench-astro-workflow July 21, 2026 19:15 View deployment

vercelBotdeployed to Preview – workbench-nestjs-workflow July 21, 2026 19:16 View deployment

vercelBotdeployed to Preview – workflow-web July 21, 2026 19:16 View deployment

vercelBotdeployed to Preview – workbench-nitro-workflow July 21, 2026 19:16 View deployment

vercelBotdeployed to Preview – workbench-express-workflow July 21, 2026 19:16 View deployment

vercelBotdeployed to Preview – workbench-tanstack-start-workflow July 21, 2026 19:16 View deployment

vercelBotdeployed to Preview – workbench-sveltekit-workflow July 21, 2026 19:16 View deployment

vercelBotdeployed to Preview – workflow-tarballs July 21, 2026 19:16 View deployment

vercelBotdeployed to Preview – workbench-vite-workflow July 21, 2026 19:16 View deployment

vercelBotdeployed to Preview – example-nextjs-workflow-turbopack July 21, 2026 19:16 View deployment

vercelBotdeployed to Preview – example-workflow July 21, 2026 19:16 View deployment

vercelBotdeployed to Preview – example-nextjs-workflow-webpack July 21, 2026 19:16 View deployment

vercelBotdeployed to Preview – workbench-nuxt-workflow July 21, 2026 19:17 View deployment

vercelBotdeployed to Preview – workflow-docs July 21, 2026 19:18 View deployment

vercelBotdeployed to Preview – workflow-swc-playground July 21, 2026 19:18 View deployment

vercelBot reviewed Jul 21, 2026
View reviewed changes

Comment threadpackages/core/src/runtime.ts Outdated