Uh oh!
There was an error while loading. Please reload this page.
Backport #2445: Redrive on transient workflow-server transport failures instead of failing the run - #2689
Merged
Merged
Conversation
…iling the run (#2445) * Redrive on transient workflow-server transport failures instead of failing the run A firewall in front of workflow-server shedding load with sustained 429/503 makes undici's shared RetryAgent exhaust its retries and throw UND_ERR_REQ_RETRY. That raw error was rethrown unwrapped, so it was classified as USER_ERROR and the replay terminal branch wrote run_failed — permanently failing a run on a transient blip (or, in an outage, falling back to the ~5min queue visibility-timeout redrive). - world-vercel: map exhausted-retry / socket / connect / DNS / timeout failures to a typed WorkflowWorldError (code TRANSPORT/TIMEOUT) by walking the fetch() cause chain. - core: add isRetryableWorldError (429 / 5xx / TRANSPORT / TIMEOUT) and rethrow such errors from the replay terminal branch so the queue redrives quickly (1s->60s backoff) instead of failing the run. Reuse it in start() and step_started handling. - world-vercel: surface the Vercel firewall x-vercel-mitigated (challenge/deny) header alongside x-vercel-id in error diagnostics and logs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: back off + cap on every retry path; fix mock; refine scope Revises the transport-error handling per PR review (VaguelySerious, karthikscale3). Blocking fix — step_started no longer self-enqueues a throttled defer for transient world errors. Returning `{ type: 'throttled', timeoutSeconds: 1 }` acked the delivery and enqueued a fresh message, resetting the delivery count so the path never backed off and never reached MAX_QUEUE_DELIVERIES — an unbounded flat-1s loop if step_started kept failing. It now throws, so the error flows through the replay loop's retryable-world-error rethrow and earns both the delivery-count backoff and the max-delivery cap. Throwing is safe on step_started (the body hasn't run; a write that landed dedupes to skipped). Also in this revision: - Backoff that lasts: raise the queue handler-error retry ceiling 60s -> 900s. VQS clamps each redelivery to its 900s SQS limit and adds its own post-32 exponential, so ramping our base toward 900s stretches survival from ~3.7h to most of the 24h message-visibility window. Corrected the stale MAX_QUEUE_DELIVERIES comment to match the real VQS schedule. - Stop amplifying firewall challenges: the undici RetryAgent no longer retries 429 in-process (a challenge is a 429 the client can't solve). 429s surface immediately as ThrottleError carrying x-vercel-mitigated / x-vercel-id, so the diagnostic header now reaches us for the challenge case too. - Track world faults as WORLD_CONTRACT_ERROR (not USER_ERROR) in classifyRunError so an outage isn't attributed to user code. - Fix queue.test.ts mock that `biome check --write` had rewritten from a newable `function` into an arrow (broke `new QueueClient`); pin with a biome-ignore. All Vercel-specific logic stays in @workflow/world-vercel; @workflow/core operates only on the generic WorkflowWorldError abstraction. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Update .changeset/transport-error-redrive.md Signed-off-by: Peter Wielander <mittgfu@gmail.com> * Trim changeset to a single sentence per review Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Route firewall-challenge 429s to the retryable transport path, not ThrottleError A 429 carrying `x-vercel-mitigated: challenge` is a firewall challenge our server-to-server client cannot solve, so it recurs for the life of the incident. Mapping it to `ThrottleError` meant the `step_started` write deferred it as `{ type: 'throttled' }`, which self-enqueues a FRESH queue message and resets the delivery count — so it never backed off past `retryAfter` and never reached `MAX_QUEUE_DELIVERIES`, hot-looping against an already-overloaded firewall (the exact amplification this PR set out to remove, and contrary to the "step_started can't loop unbounded" invariant, which only held for 5xx). Map a challenge to a retryable transport `WorkflowWorldError` (`code: 'TRANSPORT'`) in both the v3 `makeRequest` and v4 `throwForErrorResponse` (the hot event-write path) error mappings, via a shared `isFirewallChallenge429` helper. It then propagates through the V1/V2 step paths and the replay loop's retryable-world-error rethrow, earning the delivery-count backoff AND the delivery cap. A genuine application-level 429 (no `challenge` mitigation) stays a `ThrottleError` and keeps its `Retry-After`-paced defer. Also correct the survival-window comments: with the 900s ceiling, MAX_QUEUE_DELIVERIES=48 spans ~9-10h (~35,000s), not "the better part of 24h"; reaching 24h would need a higher delivery cap, not a higher per-hop ceiling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Signed-off-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
🦋 Changeset detectedLatest commit: 41e2ad0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated backport of #2445 to
stable(backport job run).AI recommendation: This is a self-contained bug fix to existing functionality — transient workflow-server transport failures were being misclassified as
USER_ERRORand permanently failing runs, and this redrives them via the queue instead. The modified files (classify-error.ts,runtime.ts,start.ts,queue.ts,http-client.ts,utils.ts,events-v4.ts) all exist onstable, and the change doesn't depend onmain-only APIs.Merge conflicts were resolved by AI (opencode with
anthropic/claude-opus-4.8). Please review the conflict resolution carefully before merging.