Uh oh!
There was an error while loading. Please reload this page.
perf(core): parallel inline steps + optimistic lazy step start - #2516
Merged
Conversation
Build two latency wins on top of lazy inline step start (#2478), both client-side only (no world/backend changes): 1. Inline up to WORKFLOW_MAX_INLINE_STEPS (default 3) steps per suspension, in parallel. The suspension handler defers step_created for the first N uncreated steps (lazyInlineSteps) and the runtime runs them via Promise.all of lazy step_started events; steps beyond the cap keep their eager step_created and are queued as before. The inline-delta fast path stays single-step only. WORKFLOW_MAX_INLINE_STEPS=1 reproduces prior behavior. 2. Optimistic inline start (WORKFLOW_OPTIMISTIC_INLINE_START, default on): executeStep fires step_started without awaiting and runs the body against locally-synthesized attempt-1 state, reconciling the in-flight start before the terminal write. A lost create-claim (409) / gone / throttle discards the body result; a transient failure propagates for redelivery. Exactly-one terminal write is preserved; bodies may run more than once under contention so inline steps must be idempotent (disable with =0). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: ba626b6 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro workflow with 10 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) workflow with 25 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) workflow with 50 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 10 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 25 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) workflow with 50 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Stream Benchmarks(includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) stream pipeline with 5 transform steps (1MB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro 10 parallel streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express fan-out fan-in 10 streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
|
Contributor
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
|
Uh oh!
There was an error while loading. Please reload this page.
… input-less background step, but since a throttle rejects the lazy `step_started` before the step is created, the re-delivery sends a bare `step_started` that the world rejects with "Step not found", causing repeated redeliveries until MAX_QUEUE_DELIVERIES marks the step (and run) failed.
This commit fixes the issue reported at packages/core/src/runtime.ts:1484
## Bug
In `packages/core/src/runtime.ts`, the inline parallel-step batch aggregation handled a `throttled` result like this:
```ts
} else if (r.type === 'throttled') {
if (lazyInlineSteps.length === 1) {
soleThrottleTimeout = r.timeoutSeconds; // re-delivers orchestrator (safe)
} else {
toRetry.push({ step: s, delaySeconds: r.timeoutSeconds }); // re-queues as background step (broken)
}
}
```
`toRetry` entries are re-queued as workflow-queue messages carrying only `{ runId, stepId, stepName }` (no input). They are consumed by the background-step branch (~line 510), which calls `executeStep` **without** `lazyStepInput`.
### Why a throttled step has no input to recover
A `throttled` result is produced by `startErrorToResult` in `step-executor.ts` when the lazy `step_started` (the atomic create-claim) is rejected with a `ThrottleError`. That means the step was **never created** — there is no `step_created` event and no step entity.
When the background-step branch later runs `executeStep` without `lazyStepInput`, `executeStep` sends a bare `step_started` with `eventData: { stepName }` (no `input`). In `packages/world-local/src/storage/events-storage.ts`, the lazy-start detector requires both `stepName` **and** `input`:
```ts
const lazyStepStart = data.eventType === 'step_started' && ... &&
(data.eventData as { input?: unknown }).input !== undefined;
...
if (!validatedStep && !lazyStepStart) {
throw new WorkflowWorldError(`Step "${data.correlationId}" not found`);
}
```
So a bare `step_started` for a never-created step throws `WorkflowWorldError("Step not found")`. That error is not one of the translatable errors (Throttle/RunExpired/EntityConflict/TooEarly), so `executeStep` rethrows it, the queue message is redelivered, and it fails repeatedly until `MAX_QUEUE_DELIVERIES`, at which point the step is marked failed and the run can fail.
### Trigger
Two or more steps run inline in parallel (`lazyInlineSteps.length > 1`) and at least one of them is rate-limited on its lazy `step_started` create-claim (returns `throttled`). The single-step path was safe only because it returned `soleThrottleTimeout`, re-delivering the orchestrator so the step is re-attempted inline **with** its input on replay.
## Fix
Generalized the single-step `soleThrottleTimeout` contract to all batch sizes. A throttled step is no longer pushed into `toRetry`; instead the batch tracks the longest throttle backoff (`throttleTimeout = Math.max(...)`) and returns `{ timeoutSeconds: throttleTimeout }`, deferring redelivery of the orchestrator message. On replay:
- still-throttled steps are re-attempted **inline with their input** (their `step_created` is deferred anew),
- `retry` steps from the same batch are re-dispatched as background steps by the suspension handler (they already exist, so a bare `step_started` is valid; their `retryAfter` is honored),
- terminal steps (completed/failed/skipped/gone) are observed from their events and not re-run,
- pending background ops are flushed via `waitUntil` before the replay reads them.
Because the orchestrator replay drives all remaining work, the early `return` deliberately skips re-queuing `toRetry`, avoiding double-dispatch of those steps. `retry` results remain safe to re-queue as background steps because a retry implies `step_started` already succeeded and the step exists — only `throttled` (which fails on the create-claim itself) leaves the step uncreated.
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: VaguelySerious <mittgfu@gmail.com>Optimistic inline start can run a step body more than once when handlers race for the create-claim. For non-idempotent steps this is unsafe — notably, two concurrent runs of a step writing to the workflow stream (e.g. an AI agent streaming tokens) can interleave and corrupt the stream. Make the optimization opt-in (enable with WORKFLOW_OPTIMISTIC_INLINE_START=1); parallel inline execution (WORKFLOW_MAX_INLINE_STEPS) stays on by default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pranaygp
left a comment
Contributor
There was a problem hiding this comment.
Review — parallel inline steps + optimistic lazy step start
Nice, carefully-reasoned change. The core safety properties hold up under inspection:
- Default config is safe. With
WORKFLOW_OPTIMISTIC_INLINE_STARTnow off, the parallel inline path still awaits each lazystep_startedcreate-claim before running a body, so exactly-one body execution is preserved per step (the loser 409s →skipped). Only the opt-in optimistic path widens double-execution, and that's documented + kill-switchable. - Optimistic path is sound where it matters. Every reachable terminal write on the optimistic path is preceded by
reconcileOptimisticStart()(completed @ step-executor.ts:611, failed/retrying @ :685); theattempt > maxRetries+1early-fail path is unreachable for a synthesized attempt-1 step; andoptimisticStartSettledattaches its.then(ok, err)synchronously so a faststep_startedrejection can't surface as an unhandledRejection. The attempt-1 synthesis is correct because lazy candidates are always brand-new (!hasCreatedEvent). - Throttle fix is correct. Deferring the orchestrator (rather than re-queuing a never-created step as an input-less background step) is the right call, and I confirmed world-local rejects a bare
step_startedfor an uncreated step withStep "<id>" not found(events-storage.ts:782-784), exactly as the comment claims. Good regression test.
A few things below — one process item and three low/medium inline notes. None are blocking on their own; the inline #1 (retry + pendingOps) is the one most worth a second look.
⚠️ CI has not run against the head commit
The last green Tests run (27784908704, 19:43Z) was against 01290df — before the WORKFLOW_OPTIMISTIC_INLINE_START default-off flip (145a848, 20:41Z), which changed both constants.ts and constants.test.ts. The 583d3ca "re-trigger CI" commit has no Tests / Lint / Benchmark check-runs attached (only DCO/Socket/Vercel). So the default-off behavior and its updated tests are currently unverified by CI. I read both diffs and they're mutually consistent, but please get Tests green on the head SHA before merge — the empty commit didn't actually re-trigger them.
Notes
- Behavioral default change (non-blocking): parallel inline (default 3) now runs up to 3 step bodies concurrently in one invocation vs. 1 before — ~3× peak per-handler memory/CPU for a clean
Promise.allfan-out. Bounded and documented; just flagging the default shift.
Details inline.
| // continuation so waitUntil can flush them before the | ||
| // next replay reads them. This matches V1 behavior where | ||
| // each step ran in a separate function invocation. | ||
| if (anyPendingOps) { |
Contributor
There was a problem hiding this comment.
Double-dispatch when a batch has both a retry step and a completed step with hasPendingOps.
Walk the ordering for a batch like Promise.all([streamingStep(), flakyStep()]) where streamingStep completes with unflushed ops and flakyStep throws and is retried:
- L1533 queues the retry message for
flakyStep— delayed bydelaySeconds, noidempotencyKey. - This
anyPendingOpsblock queues a plain continuation and returns. That continuation replays immediately, re-encounters the still-retryingflakyStep, and the dispatch loop (~L1315) re-dispatches it — immediate, withidempotencyKey.
So flakyStep is dispatched twice via two un-deduped messages. For a RetryableError the world's retryAfter guard TooEarlys the premature one (just a wasted round-trip). But a plain-Error retry writes step_retryingwithout a retryAfter (step-executor.ts:882 only sets it for RetryableError), so there's no world-side early-start guard — the immediate re-dispatch starts the step and runs the body before the intended (≈1s) backoff, and can run concurrently with the delayed message for a slow body.
Exactly-one terminal write still holds and steps must tolerate re-execution, so this isn't a correctness violation — but it diverges from the single-step path (which queued the retry and returned, one execution honoring the delay) and adds redundant queue traffic + an early body run. Worth considering whether the anyPendingOps continuation should avoid re-driving steps already in toRetry (e.g. fold the two toRetry checks together and let the explicit retry messages be the sole driver when ops are pending).
| return { | ||
| timeoutSeconds: stepResult.timeoutSeconds, | ||
| }; | ||
| if (throttleTimeout !== undefined) { |
Contributor
There was a problem hiding this comment.
This early return (correct for the backpressure contract) sits before the anyPendingOps flush continuation below. So a batch containing both a throttled step and a completed step with hasPendingOps won't queue the explicit flush continuation — the stream flush then relies solely on waitUntil completing within this invocation before the post-backoff redelivery replays and reads the stream. In practice the throttle backoff (≥1s) exceeds the 500ms flush window so it's fine, but a one-line note here would save the next reader from re-deriving that this is safe.
| if (queueItem.correlationId === lazyInlineCorrelationId) { | ||
| lazyInlineStep = { | ||
| if (lazyInlineCorrelationIds.has(queueItem.correlationId)) { | ||
| lazyInlineSteps.push({ |
Contributor
There was a problem hiding this comment.
lazyInlineSteps is populated by .push() inside the parallel ops closures (awaited by Promise.all(ops) at ~L564), so its final order reflects dehydrateStepArguments completion order, not stepItems order. Correctness is unaffected — the runtime zips stepResults to this array by index via Promise.all, so index alignment holds regardless of order, and which steps are deferred is fixed deterministically by the .slice(0, getMaxInlineSteps()) selection above.
The only snag is the test defers up to getMaxInlineSteps() uncreated steps asserting result.lazyInlineSteps.map(s => s.correlationId)toEqual(['s1','s2','s3']), which assumes stepItems order. With empty args dehydration resolves in start order so it passes today, but it could flake if dehydration ever does real async work (encryption / large-arg stream uploads). Consider asserting on a set, or pushing in index order to make it robust.
Two review follow-ups: - suspension-handler: collect deferred lazy inline steps into a Map keyed by correlationId and rebuild lazyInlineSteps in deterministic lazyInlineCorrelationIds (ordered first-N slice) order after the concurrent dehydration ops settle, instead of push()-ing in ops-completion order. No behavior change (results are zipped by index in the runtime), but the batch order is now stable regardless of dehydration timing. - runtime: document why the throttle-defer early return is safe to skip the anyPendingOps flush continuation (throttle backoff >= 1s always exceeds the <= 500ms + waitUntil in-invocation flush window). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Signed-off-by: Peter Wielander <mittgfu@gmail.com>
pranaygp
approved these changes
Jun 18, 2026
Contributor
No backport to This commit builds directly on the To override, re-run the Backport to stable workflow manually via |
This was referenced Jun 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two latency wins on top of lazy inline step start (#2478), both inside the core runtime.
1. Inline up to N steps in parallel
Previously the owned-inline path ran exactly one step inline per suspension and queued the rest, so a
Promise.all([a(), b(), c()])fan-out paid a queue round-trip per extra branch before making progress.handleSuspensionnow defersstep_createdfor up toWORKFLOW_MAX_INLINE_STEPS(default 3) steps and returns them aslazyInlineSteps.Promise.all), each via its own lazystep_started; steps beyond the cap keep their eagerstep_createdand are queued exactly as before.retry/throttledsteps are re-queued individually; we only loop back to replay once every inline step reaches a terminal state. A lone throttled inline step still delays redelivery of the orchestrator message (single-step backpressure contract preserved).WORKFLOW_MAX_INLINE_STEPS=1reproduces prior behavior exactly (kill-switch).2. Optimistic inline start
executeStepalready holds the step input locally on the inline path, so it doesn't need thestep_startedround-trip to begin.WORKFLOW_OPTIMISTIC_INLINE_START(default on), it firesstep_startedwithout awaiting and runs the body against locally-synthesized attempt-1 state, reconciling the in-flight start just before the terminal write:skipped/gone/throttled/retry.created → started → completed).Safety
step_startedcreate-claim; losers returnskipped.!hasCreatedEvent) steps are lazy; retried steps take the normal await-then-run path with the real attempt counter.WORKFLOW_OPTIMISTIC_INLINE_START=0restores await-first.