Uh oh!
There was an error while loading. Please reload this page.
[world-vercel] Idempotent retry policy for stream close (5xx retriable) - #3038
Conversation
Stream close is the one idempotent stream PUT: a duplicate close of a completed stream early-returns on the server, and the close-barrier protocol's durable `closing` fence is an if_not_exists stamp that a re-entered close resumes. The barrier protocol relies on close retrying 5xx: transient reconciliation failures — and unsafe close shapes awaiting in-flight backups — surface as retriable 503s with the stream left durably closing, expecting the writer to close again. Under the write dispatcher's no-5xx policy (correct for non-idempotent chunk appends), that 503 rejected writer.close() outright and left the stream fenced until run expiry. Close now uses its own shared RetryAgent (429 + 5xx + transient connection errors, Retry-After honored); chunk writes keep the narrowed no-5xx policy unchanged. Contract pinned by tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: e9e1dee The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
No backport to This fix builds on the stream-specific dispatcher infrastructure ( To override, re-run the Backport to stable workflow manually via |
* origin/main: (162 commits) Implement `max_events` per run limit (#2986) [core] Enforce maxRetries for steps that time out (#3035) [world-vercel] Idempotent retry policy for stream close (5xx retriable) (#3038) [world] Guard hook_received against a concurrent run termination (#2987) docs: fix stale/incorrect v5 API reference details (#3017) Default WORKFLOW_PRECONDITION_GUARD on (#2946) docs: replace migration guides with a Comparisons section (#2676) feat(core): add experimental Hook minimum retention (#2865) test: regression coverage for hook.resume() from isolated route bundles (o2flow beta.26 incident) (#3001) docs(agents): note lint/format/typecheck are advisory, not blocking (#2886) Retry transient connection timeouts (#3013) fix(world-vercel): append caller User-Agent products instead of discarding them (#2998) [ci] Enable NestJS e2e-vercel-prod and add to docs as "experimental" (#3011) [ci] Benchmark comment: Best column + best/p75/p99 deltas (drop Avg/P10) (#3005) docs: fall back to first child page for sidebar folders without an index (#3009) [nest] Fix NestJS Vercel build output (#2988) Avoid resolving run data for background steps (#2993) chore(docs): update @vercel/geistdocs to 1.14.0 (#3002) fix(docs): add version-switcher fallback redirects for pages missing in one version (#3003) ci: update opencode to 1.18.4 and switch backport AI model to claude-fable-5 (#3006) ... # Conflicts: # docs/content/docs/v4/deploying/meta.json # docs/content/docs/v5/deploying/meta.json # packages/core/src/runtime.ts # packages/world-local/src/index.ts # packages/world-postgres/src/index.ts # packages/world/src/events.ts # packages/world/src/interfaces.ts # packages/world/src/recovery.ts # workbench/nest/src/main.ts # workbench/sveltekit/src/hooks.server.ts
Backport of #2631, adapted to stable. The v3 makeRequest path, the v4 events client, the stream write/close path and the direct Vercel API calls each had their own request envelope; the 412 -> PreconditionFailed and firewall-challenge-429 -> retryable TRANSPORT mappings were duplicated between utils.ts and events-v4.ts. http-core.ts now owns that envelope: the OTEL CLIENT span and its attributes, W3C trace-context injection, the X-Request-Time cache bust, the per-request deadline (mapped to a typed retryable WorkflowWorldError instead of an opaque AbortError), DEBUG logging with a curl repro, and the status -> typed-error mapping. utils.ts, events-v4.ts, streamer.ts, encryption.ts and resolve-latest-deployment.ts delegate to it. Behavioral gains on stable: the v4 events path and stream writes now emit client spans and propagate traceparent, so the backend can parent its spans to the caller without relying on the app having undici auto-instrumentation. Adapted rather than cherry-picked: stable keeps #3169's bounded headers/body timeouts (documented as firing ahead of the outer REQUEST_TIMEOUT_MS backstop), its body-parse retry loop, and #3038's write-vs-close stream dispatcher split. Only injectTraceContextIntoHeaders is taken from #2363, not its WORKFLOW_TRACE_MODE machinery. Routing v4 through the global fetch makes the H2 interceptor's re-buffering gate live on this path (fetch streamifies request bodies), so the new http-client test drives createWorkflowRunEventV4 concurrently against a local H2 server: 16 writes on one session with the interceptor, 8 without it.
Stream close is the one idempotent stream PUT, and its retry policy now reflects that.
Why
Chunk appends deliberately never retry 5xx: a 5xx can mean the chunk was written but the response failed, and a retry would duplicate it (
STREAM_RETRY_OPTIONS, unchanged here). Close is different:writer.close()outright, turning a self-healing timing condition into a user-visible stream error.What changed
STREAM_CLOSE_RETRY_OPTIONSand a dedicated sharedRetryAgentfor the close PUT: retries 429 + 500/502/503/504 plus transient connection errors, honoringRetry-After.streamer.tsclose()switched to the new dispatcher. Chunk writes (write/writeMulti) keep the narrowed no-5xx policy unchanged.Notes for reviewers
world-verceltests green; the change is confined tohttp-client.ts/streamer.tsplus tests and a changeset.🤖 Generated with Claude Code