Uh oh!
There was an error while loading. Please reload this page.
Enable additional perf optimizations when correctness guarantees are met - #2970
Conversation
…imistic start under sequential replays Two relaxations of conservative gates in the inline replay loop, each tied to the mechanism that makes it safe: 1. The inline-delta fast path (skip one events.list per sequential step) no longer turns off for runs with an open hook when the precondition guard (WORKFLOW_PRECONDITION_GUARD=1) is enabled. A hook_received landing in the delta window is the same read-to-write race the fetch path already has, and with the guard on it is fenced: the marker bump 412s the stale replay's guarded creates, which retry over the reloaded log or exhaust into a fresh-replay re-invocation. Open waits keep the conservative gate (wait_completed does not bump the outside-event marker). 2. Turbo keeps forcing optimistic inline start after the run creates a hook or wait when WORKFLOW_SEQUENTIAL_REPLAYS=1: per-run maxConcurrency: 1 flow topics serialize the resume invocations hooks/waits introduce, so no concurrent orchestrator replay can race the optimistic create-claim, restoring the single-handler guarantee turbo relies on. Adds isSequentialReplaysEnabled() to the core runtime (mirroring the @workflow/builders / @workflow/world-vercel copies) and splits hasOpenHookOrWait into per-kind state so the two gates can differ. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 5e28160 The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📜 Previous results (2)a7f082aFri, 17 Jul 2026 17:50:37 GMT · run logs
9b1bd46Fri, 17 Jul 2026 01:06:18 GMT · run logs
Avg deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader) Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges 🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request. |
| !suspensionResult.waitTimeout && | ||
| !hasOpenHookOrWait(cachedEvents ?? []); | ||
| !openHookWaitState.openWait && | ||
| (isPreconditionGuardEnabled() || |
There was a problem hiding this comment.
[P1] Guard the lazy step claim before enabling deltas with open hooks
A hook_received can land after the terminal write that produced the delta, so the next loop replays a stale view. If that replay schedules a lazy inline step, handleSuspension defers step_created and both lazy step_started paths in executeStep omit stateUpdatedAt. workflow-server therefore skips the precondition check, the claim succeeds, and the step body can execute/commit before a fresh replay observes the hook—the loser-step/replay-divergence case this gate is meant to prevent. Please keep the open-hook gate, or pass the snapshot into lazy step_started and ensure the body waits for the guarded claim (including optimistic-start cases), with an interleaving test rather than only asserting sinceCursor.
There was a problem hiding this comment.
Good catch — the fence I described didn't actually cover the hot path: the lazy claim is a hot-path step's first durable write (its step_created is deferred), and it carried no snapshot. Fixed in a7f082a by taking your second option:
- The runtime now computes the guard snapshot from the loaded log and threads it into the lazy
step_startedclaim on both paths (optimistic and await-then-run) via a newstateUpdatedAtexecutor param. - A stale (412) rejection is intentionally not translated by the claim-error mapper — re-claiming in place would still commit the stale schedule — so it propagates: the batch is abandoned, any optimistic body result is discarded by the existing reconciliation (no events are ever written by the loser), and the run is re-invoked for a fresh replay that observes the new event.
- Added the interleaving test you asked for: open hook + two sequential steps, the second step's claim rejected as stale by the backend → the fenced step's body never runs, no events are written for it, no
run_failed, and the message redelivers (plus an assertion that the claim actually carried the snapshot).
Two notes on scope: (1) under optimistic start the body may still start before the claim settles — a fenced claim discards the result and writes nothing durable, so the side-effect exposure is exactly the documented optimistic-start idempotency contract, unchanged by this PR; (2) the very first batch of a run loads an empty log and has no snapshot to send — that's pre-existing guard semantics shared with every suspension create (best-effort by design, see latestEventStateUpdatedAt).
| !suspensionResult.waitTimeout && | ||
| !hasOpenHookOrWait(cachedEvents ?? []); | ||
| !openHookWaitState.openWait && | ||
| (isPreconditionGuardEnabled() || |
There was a problem hiding this comment.
[P1] Verify guard support rather than trusting the env flag
This branch treats WORKFLOW_PRECONDITION_GUARD=1 as proof that the World enforces stateUpdatedAt, but the docs say unsupported backends may ignore that field. On such a World, merely setting the env var opens the hook-delta path with no 412 fence at all. Please gate this on an explicit World/backend capability (or otherwise fail closed) instead of a process-local flag alone.
There was a problem hiding this comment.
Agreed — the env flag only makes the runtime send snapshots; it can't prove the backend enforces them. Fixed in a7f082a: the World interface now has an optional capabilities?: WorldCapabilities field, and the open-hook delta relaxation requires WORKFLOW_PRECONDITION_GUARD=1andworld.capabilities.preconditionGuard === true, failing closed to the conservative gate otherwise. @workflow/world-vercel declares the capability (workflow-server enforces the marker); world-local and world-postgres ignore stateUpdatedAt entirely and correctly declare nothing. Added a negative test: flag set + no capability → the delta is not requested.
| !suspensionResult.hasAttributeEvents && | ||
| !suspensionResult.hasAwaitedHookCreation && | ||
| !hasOpenHookOrWait(cachedEvents ?? []); | ||
| (isSequentialReplaysEnabled() || |
There was a problem hiding this comment.
[P1] Do not waive the hook/wait latch from a runtime-only flag
isSequentialReplaysEnabled() only checks the runtime env var; it cannot establish that the built flow trigger actually has maxConcurrency: 1. If runtime is set without matching build config, hook/wait resume messages may overlap, yet this branch forces optimistic start. Two handlers can then run the same step body before either atomic claim settles; the loser drops its result but external side effects already happened. Please require a verified serialization capability/config signal before taking this arm, rather than relying on the raw env flag plus documentation.
There was a problem hiding this comment.
Fixed in a7f082a: the sequential-replays waiver now also requires world.capabilities.maxConcurrency === true (declared by @workflow/world-vercel, whose queue implements maxConcurrency-limited consumers), failing closed to the conservative hook/wait latch otherwise — with a negative test (env set, no capability → await-then-run ordering preserved).
To be explicit about the residual: the capability confines the waiver to Worlds whose queue actually supports serialized consumption; it cannot by itself verify the build-time half of the contract (the flow trigger's maxConcurrency: 1 config), which the runtime has no way to introspect from inside a function invocation. That remains the documented set-it-at-build-AND-runtime requirement (called out in the Vercel World docs and the gate's code comment). If we later surface the built trigger config to the runtime (e.g. via the manifest), the gate can tighten further — happy to file a follow-up.
…ies; fence the lazy inline claim Address review on #2970: - Add `capabilities?: WorldCapabilities` to the World interface (`preconditionGuard`, `maxConcurrency`); the Vercel World declares both. The env flags alone cannot prove backend enforcement, so the inline-delta open-hook relaxation and turbo's sequential-replays waiver now also require the matching capability and fail closed on Worlds that don't declare it. - Thread the precondition-guard `stateUpdatedAt` snapshot into the lazy inline `step_started` claim (both the optimistic and await-then-run paths). The claim is a hot-path step's first durable write, so without it a stale replay could claim — and commit — a step scheduled off a view that misses an out-of-band event. A 412-rejected claim is not translated: the batch is abandoned (any optimistic body result is discarded) and the run is re-invoked for a fresh replay. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
karthikscale3
left a comment
There was a problem hiding this comment.
Re-reviewed at 5e28160. The outstanding concurrency concerns are addressed: guard-enforced hook-sensitive batches now await the fenced step_started claim before running user code, and the sequential-replays optimistic-start waiver has been removed until deployed maxConcurrency configuration can be verified. No blocking findings.
Uh oh!
There was an error while loading. Please reload this page.
No backport to This commit relaxes gates in To override, re-run the Backport to stable workflow manually via |
* origin/main: (21 commits) docs(agents): note lint/format/typecheck are advisory, not blocking (#2886) Retry transient connection timeouts (#3013) fix(world-vercel): append caller User-Agent products instead of discarding them (#2998) [ci] Enable NestJS e2e-vercel-prod and add to docs as "experimental" (#3011) [ci] Benchmark comment: Best column + best/p75/p99 deltas (drop Avg/P10) (#3005) docs: fall back to first child page for sidebar folders without an index (#3009) [nest] Fix NestJS Vercel build output (#2988) Avoid resolving run data for background steps (#2993) chore(docs): update @vercel/geistdocs to 1.14.0 (#3002) fix(docs): add version-switcher fallback redirects for pages missing in one version (#3003) ci: update opencode to 1.18.4 and switch backport AI model to claude-fable-5 (#3006) fix(core): batch stream writes via writeMulti (#2995) perf(core): cache port discovery in step invocations for self-hosted worlds (#2996) feat(web-shared): Alt+hover span measurement in the new trace viewer (#2985) fix(world-postgres): throw EntityConflictError on duplicate run_created (#2983) [ci] Run benchmarks in-deployment to avoid proxy overhead (#2967) Enable additional perf optimizations when correctness guarantees are met (#2970) perf(core): prepare replay payloads concurrently (#2980) Fix dotted tsconfig alias workflow discovery (#2963) Adjust helper position on trace viewer (#2968) ...
Summary
Two relaxations of conservative gates in the inline replay loop, each tied to the mechanism that makes it safe. Behavior is byte-identical unless the corresponding flag is enabled.
1. Inline-delta fast path stays active with open hooks (requires
WORKFLOW_PRECONDITION_GUARD=1on a World that declarescapabilities.preconditionGuard)The per-step event-log delta optimization (#2475) — consuming the delta returned by the step-terminal write instead of issuing one
events.listper sequential step — previously turned off for the rest of the run once any hook was open, because ahook_receivedlanding between the terminal write and the next replay would be absent from the delta.With the precondition guard enabled, that window is fenced rather than open-ended:
events.listreturns and before the suspension's writes, and is observed one iteration late. The delta path widens that window; it does not create a new class of it. Delta windows are also contiguous (each delta covers everything since the pre-write cursor), so an event missed by one window is always delivered by the next one.hook_receivedbumps the run's outside-event marker, so a replay acting on the stale view has its guarded suspension creates rejected with 412 and retried over the reloaded log — or, if reloads cannot surface the event, exhausted into a queue re-invocation whose fresh full replay observes it.Open waits keep the conservative gate:
wait_completeddoes not bump the outside-event marker, so nothing fences a replay from a delta that missed one. The gate now distinguishes the two kinds (openHookAndWaitState). Hooks created by the same suspension (err.hookCount) are also allowed under the guard — theirhook_createdlands before the step-terminal write and is therefore inside the delta.For hook-heavy sequential workflows this removes one world round-trip per step for the entire post-hook stretch of the run.
2.
Turbo keeps forced optimistic inline start under(retracted in review)WORKFLOW_SEQUENTIAL_REPLAYS=1Turbo's forced optimistic inline start (run the step body immediately instead of awaiting the
step_startedcreate-claim) previously latched off the moment the run created a hook or wait: those introduce resume invocations, ending the single-handler guarantee that makes running a body before the claim confirms safe.With sequential replays enabled, the resume invocations hooks and waits introduce are run-topic messages on a per-run
maxConcurrency: 1topic — the queue does not deliver them until the current delivery acks, so no concurrent orchestrator replay can race the optimistic create-claim, and the latch is waived.What stays concurrent is unchanged from clean turbo today:
step_startedcreate-claim still guarantees at most one winner writes events. This window exists in clean (hook-free) turbo onmaintoday.hook_received; they never execute steps.maxConcurrency: 1is a lease guarantee; an invocation past its visibility timeout can overlap its redelivery. Same at-least-once envelope as crash-redelivery.Note the pre-existing configuration caveat also applies here:
WORKFLOW_SEQUENTIAL_REPLAYSneeds the matching build-time flow-trigger config. Setting the env var at runtime only, on a setup whose flow trigger lacksmaxConcurrency: 1, does not serialize anything — the new turbo behavior assumes the documented full configuration.The attr-events check stays in both arms: attribute suspensions resolve through an in-process replay pass that must decide races before any step body runs, independent of queue serialization.
Changes
packages/core/src/runtime.ts— splithasOpenHookOrWaitintoopenHookAndWaitState(per-kind); rework therequestInlineDeltaandforceOptimisticStartgates as above, with the safety analysis in comments.packages/core/src/runtime/constants.ts— addisSequentialReplaysEnabled()(mirrors the@workflow/builders/@workflow/world-vercelcopies; core must not depend on either).configuration/runtime-tuning.mdx(guard + turbo entries),worlds/vercel.mdx(sequential-replays section). v4 docs untouched — this ships onmain(5.0 beta) only unless backported.Tests
runtime.test.ts— new: withWORKFLOW_SEQUENTIAL_REPLAYS=1, a wait-creating suspension keeps optimistic start (body observed while the gatedstep_startedcreate is still in flight — impossible on the await-then-run path); the existing turbo-exit test covers the flag-off behavior. New describe for the delta gate: with an open hook, the step-terminal write carriessinceCursoronly whenWORKFLOW_PRECONDITION_GUARD=1and the World declarescapabilities.preconditionGuard(negative tests cover flag-without-capability for both arms).constants.test.ts—isSequentialReplaysEnabledstrict-'1'semantics.cd packages/core && pnpm test: 69 files, 1489 passed (3 expected fail, pre-existing). Typecheck and Biome clean (all warnings pre-existing onmain).Review follow-up (a7f082a)
Addressed the review with three changes:
capabilities?: WorldCapabilitieson the World interface (@workflow/world), withpreconditionGuardandmaxConcurrencymembers. Both relaxations now require the matching capability in addition to the env flag and fail closed on Worlds that don't declare it.@workflow/world-verceldeclares both;world-local/world-postgres(which ignorestateUpdatedAtand have no queue-concurrency concept) declare nothing.step_startedclaim is now guard-fenced. It is a hot-path step's first durable write, so it now carries thestateUpdatedAtsnapshot (both optimistic and await-then-run paths). A stale (412) rejection is not retried in place: the batch is abandoned — any optimistic body result is discarded, nothing durable is written — and the run re-invokes for a fresh replay. Covered by an interleaving test (open hook + two steps, second claim rejected as stale).maxConcurrencycapability confines the sequential-replays waiver to queues that support serialized consumption; it cannot verify the build-time trigger config from inside a function invocation — the build-AND-runtime configuration contract stays documented on the Vercel World page.Review follow-up, round 2 (5e28160)
WORKFLOW_OPTIMISTIC_INLINE_START=1/ turbo's force — the claim (carrying the snapshot) is awaited before user code runs, so a 412-fenced step never executes its body. Combined open-hook + optimistic-start + stale-claim test added.maxConcurrency: 1(build-time env + per-integration trigger config;@workflow/nitrocurrently never emits it at all), so the conservative hook/wait latch stays.capabilities.maxConcurrencyremains declared as the queue-support half for a follow-up that pairs it with a build-verified serialization signal. This PR is now scoped to the guard-fenced inline-delta relaxation.Docs Preview
WORKFLOW_PRECONDITION_GUARDWORKFLOW_TURBOWORKFLOW_SEQUENTIAL_REPLAYS(Links require Vercel team access — the preview deployment is behind deployment protection.)
🤖 Generated with Claude Code