Uh oh!
There was an error while loading. Please reload this page.
[core] Fix turbo-mode step-body writes racing run_started - #2629
Merged
Conversation
In turbo mode the first step body runs optimistically — before the backgrounded `run_started` event is durable. The host-side `experimental_setAttributes` path wrote its `attr_set` event directly, so an attribute set early in that step could reach the World before the run existed and be rejected as run-not-found. The workflow-side path was already gated on the run-ready barrier; this extends the same gate to the step-body path by threading the barrier through the step context. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
🦋 Changeset detectedLatest commit: 341cec2 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
workflow with 1 step💻 Local Development
▲ Production (Vercel)
workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
workflow with 10 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
workflow with 25 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
workflow with 50 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
workflow with 10 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
workflow with 25 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
workflow with 50 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
Stream Benchmarks(includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
stream pipeline with 5 transform steps (1MB)💻 Local Development
▲ Production (Vercel)
10 parallel streams (1MB each)💻 Local Development
▲ Production (Vercel)
fan-out fan-in 10 streams (1MB each)💻 Local Development
▲ Production (Vercel)
SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
❌ Some benchmark jobs failed:
Check the workflow run for details. |
Contributor
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
|
In turbo mode the first step body runs optimistically — before the backgrounded `run_started` event is durable. `getWritable()` stream writes flushed directly to the server; the first chunk of a brand-new stream requires the run to exist (the World does a run lookup), so an early write could be rejected as run-not-found, failing the step or silently dropping the chunk. Thread the run-ready barrier (already on the step context) into `WorkflowServerWritableStream` and await it once before the first flush/close. No-op outside turbo and on the await path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Additional Suggestion:
In turbo optimistic start, a nested ReadableStream written into a step's getWritable() stream is piped to its own server stream via getExternalReducers without the run-ready barrier, so its first chunk can reach world.streams.write() before the backgrounded run_started is durable (run-not-found race).
…o a step's `getWritable()` stream is piped to its own server stream via `getExternalReducers` without the run-ready barrier, so its first chunk can reach `world.streams.write()` before the backgrounded `run_started` is durable (run-not-found race). This commit fixes the issue reported at packages/core/src/step/writable-stream.ts:81 ## The bug The PR closes a turbo-mode race where step-body world writes can reach the World before the backgrounded `run_started` is durable. It threaded a `runReadyBarrier` into: 1. `experimental_setAttributes` (`set-attributes.ts`), 2. the outer `WorkflowServerWritableStream` built by `getWritable()` (`step/writable-stream.ts` ~line 92), and 3. step return-value serialization via `getStepReducers` (`serialization.ts` ~line 1448), including its recursive call. **Gap:** `getWritable()` builds its serialize transform with `getExternalReducers(globalThis, ctx.ops, runId, ctx.encryptionKey, true)` — *not* `getStepReducers`. Inside `getExternalReducers`, the `ReadableStream` reducer (`serialization.ts` ~line 1236) constructed: ```ts const writable = new WorkflowServerWritableStream(runId, name); // no barrier ... ops.push(value.pipeTo(writable)); // or framed / recursive-serialize variants ``` This pipe runs **independently** of the gated outer `serverWritable`. The barrier on the outer writable only orders writes to `(runId, name)` — the nested stream writes to a *different* `(runId, strm_<id>)` via an ungated sink. ## Reachability / trigger - A turbo optimistic first-step body calls `getWritable()` (body runs before `run_started` is durable). - The user writes a value that contains a nested `ReadableStream` into that writable. - `getSerializeStream` invokes `getExternalReducers`' `ReadableStream` reducer synchronously on that chunk, which creates an ungated `WorkflowServerWritableStream(runId, name)` and pushes `value.pipeTo(writable)` to `ctx.ops`. - That background pipe flushes the nested stream's first chunk to `world.streams.write(runId, strm_id, chunk)` with no barrier → it can land before `run_started` is durable → run-not-found, exactly the race the PR is closing, one level deeper. This is the same code shape that *was* fixed in `getStepReducers` (commit dd61604); `getExternalReducers` was simply missed. ## The fix Mirror the `getStepReducers` change: - Added an optional trailing `runReadyBarrier?: Promise<unknown>` param to `getExternalReducers`. - Passed it into `new WorkflowServerWritableStream(runId, name, runReadyBarrier)`. - Threaded it through the recursive `getExternalReducers(...)` call used for non-byte streams. - Passed `ctx.runReadyBarrier` from `step/writable-stream.ts`'s `getWritable()`. The param is an optional trailing argument, so all other `getExternalReducers` callers (which pass `undefined`/nothing) are unaffected and the barrier is a no-op outside turbo and on the await path, consistent with the existing `WorkflowServerWritableStream` barrier semantics. Note: type-checking could not be executed in this environment because dependencies (`tsc`) are not installed, but the change is type-safe — the new optional param is `Promise<unknown> | undefined`, the constructor already accepts that signature, and `ctx.runReadyBarrier` is typed `Promise<unknown> | undefined`. Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com> Co-authored-by: VaguelySerious <mittgfu@gmail.com>
karthikscale3
approved these changes
Jun 26, 2026
karthikscale3
left a comment
Contributor
There was a problem hiding this comment.
Re-reviewed current head 341cec2. I checked the runReadyBarrier propagation through setAttributes, getWritable, returned ReadableStream serialization, and nested ReadableStream serialization, plus the focused stream/attribute tests. No confirmed P0/P1 code blockers remain from my review.
Uh oh!
There was an error while loading. Please reload this page.
Contributor
No backport to This fix targets a race in turbo-mode optimistic start, a To override, re-run the Backport to stable workflow manually via |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In turbo mode the first step's body runs optimistically — before the backgrounded
run_startedevent is durable. Direct step-body world writes had no ordering gate, so they could reach the server before the run existed. Two such paths:experimental_setAttributes(host-side) wrote itsattr_setevent directly.getWritable()stream writes flushed chunks directly. The first chunk of a brand-new stream requires the run to exist (the World does a run lookup on stream creation), so an early write could be rejected as run-not-found — failing the step, or silently dropping the chunk if the rejection lands after the step's op-flush timeout.The workflow-side
experimental_setAttributespath (a"use workflow"body) was already gated on the run-ready barrier in the suspension handler. Only the step-body paths were unguarded. (Step-initiated aborts write directly too, but they require a pre-existing hook, which disables turbo's forced optimistic start — so they can't race.)Fix
Thread the run-ready barrier through the step execution context and await it before the first direct step-body write:
step/context-storage.ts— addrunReadyBarrier?: Promise<unknown>toStepContext.runtime/step-executor.ts— populate it only on the optimistic-start path;undefinedon the await path (run already durable).set-attributes.ts—awaitthe barrier before theattr_setcreate.serialization.ts(WorkflowServerWritableStream) — accept the barrier; await it once before the first flush write and beforeclose(a close with an empty buffer can itself be the first write to a new stream).step/writable-stream.ts— passctx.runReadyBarrierinto the stream.In every case the rejection is swallowed for ordering only — a genuinely failed
run_startedthen surfaces the real error from the write itself, mirroring the workflow-sideensureRunReadybehavior. All gates are no-ops outside turbo and on the await path.Tests
set-attributes.test.ts— theattr_setwrite fires strictly after the barrier resolves; a rejecting barrier still writes.writable-stream.test.ts— the first server write/close waits for the barrier; later flushes aren't re-gated; a rejecting barrier still writes.Typecheck +
step-handler/set-attributes/serialization/writable-stream/flushable-streamsuites pass. (world-local has no run-existence check on stream writes, so the stream race is Vercel-only; the unit tests gate against a fake World that models the server contract.)🤖 Generated with Claude Code