Uh oh!
There was an error while loading. Please reload this page.
[world-vercel] Retry throttled (429) event writes in-process, honoring Retry-After - #3504
Merged
Conversation
Contributor
🦋 Changeset detectedLatest commit: f5f6c37 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ vercel-multi-region
|
Contributor
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 165564ms → this run 163269ms (Δ -2295ms, -1%) 📜 Previous results (1)943410dWed, 12 Aug 2026 20:32:51 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Contributor
Sim WorldSimulated world deterministic testing for races. Traces 🟠 Mint-ordered log — 6 fail of 41 total
Full trace: 🟢 Append-only log — 0 fail of 41 total
Full trace: |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an in-process retry policy for event-write throttling (HTTP 429 / ThrottleError) in @workflow/world-vercel, honoring server-provided Retry-After and bounding total wait time to avoid pinning an invocation. This aligns event-write behavior with the intended semantics for throttle handling and reduces reliance on queue redelivery timing.
Changes:
- Implement budgeted,
Retry-After-honoring in-process retries forThrottleErroracross all event types inwithEventPostRetry. - Refactor transient retry eligibility into a helper and keep throttle waits independent of the transient retry attempt counter.
- Add unit tests covering throttle retry behavior (honor
retryAfter, default wait, eligibility-excluded events, and budget exhaustion), plus a changeset for a patch release.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| packages/world-vercel/src/event-retry.ts | Adds a separate 429 throttle retry path with a cumulative wait budget and Retry-After handling, without altering transient retry classification. |
| packages/world-vercel/src/event-retry.test.ts | Adds targeted unit tests validating throttle retry timing, applicability, and budget boundaries. |
| .changeset/throttle-in-process-retry.md | Declares a patch release note describing the new in-process 429 retry behavior and its motivation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
VaguelySerious
approved these changes
Aug 12, 2026
VaguelySerious
left a comment
Member
There was a problem hiding this comment.
Looks like the right approach to me
TooTallNate
enabled auto-merge (squash)
August 12, 2026 20:56
Uh oh!
There was an error while loading. Please reload this page.
github-actionsBot
added a commit
that referenced
this pull request
Aug 13, 2026
…g Retry-After (#3504) Signed-off-by: Nathan Rajlich <n@n8.io>
Contributor
Backport PR opened against |
This was referenced Aug 13, 2026
pranaygp added a commit
that referenced
this pull request
Aug 14, 2026
…h per-event results
The client half of workflow-server#646's v4 batch endpoint, rebuilt from
scratch against the merged contract (this branch's previous fence-based
design is retired; old head tagged batch-client-v2-fence-design).
- @workflow/world: optional `events.createBatch(runId, events, params)`
— ordered events in, per-event outcomes out. Method presence IS the
capability declaration: worlds that don't implement it keep the
single-event path (world-local and world-postgres stamp specVersion 5
and gain nothing from batching a local write, so they deliberately
don't implement it). Each result reports what that event's own single
create would have returned: 200 + the materialized entity, or the
single-path status/code (409 conflict for an already-applied event),
so callers reuse single-path conflict handling per event.
- @workflow/world-vercel: POST /v4/runs/:runId/events/batch — the
events' single-POST frames back-to-back (byte-identical framing, no
batch-level meta, no fence fields), CBOR { results } decoded against
the SAME per-type schemas as the single POST, with a loud
SCHEMA_VALIDATION on a malformed response (length mismatch, invalid
item). Wired into createStorage.
- Retry: the whole batch POST is idempotent-on-retry regardless of the
event types it carries (every batchable event is guarded by its own
entity condition; a retried committed batch converges to per-event
409s), so a new batchIdempotent override joins the per-type
eligibility matrix — including #3504's in-process 429 Retry-After
handling.
Runtime integration (suspension fan-out fold, then the deferred
sequential transition) ships separately on top of this contract — see
the PR description for the staged plan, caps, and kill switch.
Tests: 7 wire tests (frame encoding + ordering + no fence fields,
per-event result mapping, malformed-response failures, typed 400s,
5xx in-process retry, empty-batch guard); world-vercel suite 508/508.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>pranaygp added a commit
that referenced
this pull request
Aug 15, 2026
…h per-event results (#3025) ## createBatch: the client half of the v4 batch event write (per-event results, no fence) > **Note:** this PR was rebuilt from scratch. The previous revision implemented the retired "v2 suspension fence" design (`expectedRunVersion` / `batchId` / `logicalCreatedAt`, a grammar-validated collect mode, world-postgres migration 0016, world-local claim machinery). The server redesigned its endpoint in place (vercel/workflow-server#646, merged and deployed) and this branch now targets that contract on top of current `main` (specVersion 6 slot identity). The old head is tagged `batch-client-v2-fence-design`; prior review threads reference deleted code. ### The server contract this targets `POST /api/v4/runs/:runId/events/batch` (workflow-server#646): an **ordered** list of v4 frames — byte-identical to single-event POST frames, **no batch-level meta** — committed in one DynamoDB transaction per attempt, answered with HTTP 200 + `{ results }`: one entry per frame, in request order. Each event reports what its own single POST would have returned: `200` + the materialized entity, or the single-path status/code (e.g. `409`/`conflict` for an event an earlier delivery already applied). A transport retry of a committed batch converges to all-409s with nothing written twice — idempotency comes from per-entity conditions, not batch bookkeeping. Slot-identity runs only (specVersion ≥ 6 — what `world-vercel` stamps on every new run since #3389). ### What this revision ships 1. **`@workflow/world` — the spec addition.** `Storage['events']` gains one optional method; **method presence is the capability declaration** (no capability flag, no stub required): ```ts createBatch?( runId: string, events: BatchEventRequest[], params?: CreateEventBatchParams ): Promise<EventBatchResult>; interface BatchEventRequest { event: CreateEventRequest; // same discriminated union as the single create occurredAt?: Date; // under slot identity: the source of the durable createdAt } type BatchEventItemResult = // one per submitted event, in request order | { status: 200; event: Event; run?: WorkflowRun; step?: Step; wait?: Wait } | { status: number; error: string; message: string }; interface EventBatchResult { results: BatchEventItemResult[] } ``` Contract: **ordered** (events land in the log in request order), **per-event outcomes** (each event reports what its own single `create` would have returned — success discriminated by `error === undefined`), **idempotent on retry** (per-entity conditions make a retried committed batch converge to per-event 409s). Worlds that don't implement it keep the single-event path. `world-local` and `world-postgres` deliberately do NOT implement it — batching a local/in-process write buys nothing (this deletes the old revision's riskiest surface: the hand-written postgres migration and the world-local claim machinery). 2. **`@workflow/world-vercel`** — the wire adapter: per-event frames concatenated in order (reusing the single-frame encoder; each frame carries its own `occurredAt`, which under slot identity is the source of the durable `createdAt` — this natively closes the replay-clock question the old `logicalCreatedAt` field existed for), CBOR `{ results }` decoded against the **same per-type zod schemas as the single POST**, loud `SCHEMA_VALIDATION` on any malformed response (wrong length, invalid item), and the standard typed error mapping for request-level failures. 3. **Retry policy** — a `batchIdempotent` override in the event-retry eligibility machinery: the whole batch POST retries transient transport failures/5xx (and waits out 429 `Retry-After` per #3504) regardless of the contained event types, because per-event entity conditions make the retry converge; the per-type non-retryability matrix guards *single* posts (where e.g. a retried bare `step_started` would increment `attempt`) and doesn't apply inside a batch. Tests: 7 wire tests — frame encoding/ordering + **no fence fields on the wire**, per-event result mapping (successes typed, failures passed through), malformed-response failures (length mismatch, invalid item body with index), typed request-level 400s, in-process 5xx retry, empty-batch guard — plus 9 suspension-handler tests for the runtime fold: ordering (steps then waits), per-event 409 tolerance, non-409 failure propagation, every gate exclusion (flag off / no `createBatch` / pre-slot run / hook writes), 32-cap chunking, and lazy-inline exclusion. Full `world-vercel` suite: 508 passed; full `@workflow/core` suite: 2126 passed. ### The runtime integration: batched suspension fan-out (ON by default) The suspension handler folds a **clean fan-out** — the suspension's eager `step_created` + `wait_created` writes — into `createBatch` calls of at most **32 events**, and uses the batch endpoint **exactly when two or more batchable eager events exist**: a lone eager event takes the ordinary single write (same round trip, and it keeps the slot-snapshot + bump-and-report the single path provides) (mirroring the server's transaction budgets: 2 items/event against the 100-item cap, 768 KB inline-byte budget; larger fan-outs commit in successive batches). The gate requires: World implements `createBatch` ∧ run on slot identity (specVersion ≥ 6) ∧ no attribute writes ∧ no hook writes ∧ no resilient step dispatch. **Everything outside the gate keeps the single-event path byte-for-byte**, and lazy-inline steps keep deferring their `step_created` to the lazy start exactly as before. Per-event semantics mirror the single path: a `409` is the same already-exists tolerance as `EntityConflictError` (the conflicted step is not marked owned); any other per-event failure fails the suspension write the way a single-path rejection would. Slot bumps (the batch endpoint has no bump-and-report) are tolerated and logged — the same accepted exposure as a dropped truncated skipped-slot report on the single path. **On by default**, with the `WORKFLOW_TURBO`-shaped kill switch as the operator escape hatch: **`WORKFLOW_BATCH_TRANSITIONS=0`** (or `false`) disables batching and restores the exact prior one-write-per-event path. Documented in the worlds configuration reference and the changelog entry. Burn-in watch: the `event_batch`-tagged slot-conflict metrics and DynamoDB throttle monitors on the server side. ### Docs - New v5 changelog entry **`changelog/batched-event-writes`** documenting the World spec addition (full `createBatch` signature + contract — the signature block is compile-checked against `@workflow/world` by the docs code-sample checker), the runtime fold, and the follow-up. - `configuration/worlds` gains the **`WORKFLOW_BATCH_TRANSITIONS`** reference entry: default on, `=0`/`false` as the documented escape hatch. ### Staged follow-up: the deferred sequential transition (the STSO win) Hold `step_completed(N)` across the replay turn and commit `[step_completed(N), step_created(N+1), step_started(N+1)]` as one batch at the next lazy start (the server folds the pair born-running). This needs the synthetic-completion replay machinery rebuilt against today's runtime (parallel inline batches, turbo's run-ready barrier, optimistic starts, slot bookkeeping) — it stays a separate PR so the SDK's most sensitive replay path gets its own focused review. Its acceptance criteria are already agreed: the runtime eligibility matrix as unit tests, and an e2e that asserts ≥1 POST to `/events/batch` and **zero** single-event POSTs for the batched transitions. ### Compatibility - Old servers: no `/batch` route → 404/405 → callers fall back to single-event posts (the runtime PRs will latch this per run). - Pre-slot runs: request-level 400 (`batch-requires-slot-identity`) → same fallback. - No `WORKFLOW_SERVER_URL_OVERRIDE` pin this time — the server endpoint is merged and deployed to production. Refs: vercel/workflow-server#646 (endpoint), vercel/workflow-server#780 (unbatchable-types design space), #3389 (slot identity), #3504 (429 retry). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
The backend can reject an event write with a 429 and a
Retry-Afterof a few seconds (e.g. under write contention). The SDK mapped these toThrottleErrorbut never retried them in-process: the error escaped to the queue handler, and queue redelivery scheduling can delay the retry far beyond the requestedRetry-After— observed in practice as aretryAfter: 14throttle followed by a ~5-minute stall before the next delivery attempt.The intended behavior for
ThrottleErroris in-process retry.Change
withEventPostRetrynow retries 429s in-process:EVENT_RETRY_ELIGIBILITY-excluded ones (step_started,step_retrying,hook_received): a genuine application 429 is a definitive no-write — the server rejected the request outright — so the duplicate-row / attempt-double-count hazards that matrix protects against don't apply. This is the same reasoningSTREAM_RETRY_OPTIONSalready uses to retry 429 on stream PUTs.retryAfter(default 1s when absent).THROTTLE_RETRY_BUDGET_MS), addressing the "no attempt tracking → could spin until the flow route dies" concern. Once the budget can't cover the nextretryAfter, theThrottleErrorsurfaces and queue redelivery takes over exactly as before.errorForResponsemapsx-vercel-mitigated: challengeto a transportWorkflowWorldError, notThrottleError, so the original "let 429s pass through" rationale (never hot-loop against the firewall) is preserved.Each in-process throttle retry logs a visible
console.warn(it stalls the invocation for whole seconds, which would otherwise read as unexplained latency).Notes
Retry-Aftersizing may be worth revisiting now that the client waits it out in-process.Testing
event-retry.test.tscover: honoringretryAfter, applying to eligibility-excluded types, the no-retryAfterdefault, budget exhaustion (cumulative and single-wait-exceeds-budget), and budget independence from the transient allowance.pnpm vitest run src/event-retry.test.ts src/events-retry.test.ts src/events-v4-ws.test.ts— 53 passed.