fix(world-vercel): add default request timeout to workflow-server HTTP calls - #1807

Merged
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout
May 4, 2026
Merged

fix(world-vercel): add default request timeout to workflow-server HTTP calls#1807
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout

Conversation

@karthikscale3

@karthikscale3karthikscale3 commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a default per-request timeout to makeRequest() in world-vercel so hung responses from workflow-server can't burn compute up to the function's maxDuration.

Background (original problem)

A production workflow (wrun_01KPDFGK4QBFZ7XERXN9NP7VY2) on a preview deployment showed:

  • run_started POST to workflow-server took 47s (server under load)
  • Replay timeout fired at 240s
  • The run_failed write was sent but the response never came back (External APIs showed ∅ "Timed out while waiting for a response")
  • The function continued running for 15 minutes (hit maxDuration) before SIGTERM — ~11 minutes of compute burned doing nothing

Original fix (reverted)

The first version of this PR added a 30s hard-exit deadline to the replay timeout handler in packages/core/src/runtime.ts. Per Nate's review, this was the wrong layer: it only protected one of 27 world.events.create() call sites in core, leaving the other 26 (and every other world.* method going through makeRequest()) exposed to the exact same hang.

New fix

Moved the mitigation down into the world-vercel transport layer, where all world.* calls funnel through makeRequest():

  • packages/world-vercel/src/utils.ts — attaches AbortSignal.timeout(60_000) to every makeRequest() fetch. A TimeoutError or AbortError from fetch is converted into a WorkflowWorldError (with the original error preserved as cause and elapsed ms in the message), so existing catch sites handle it uniformly. The span is tagged with ErrorType('TIMEOUT').
  • Reverted the runtime-level exit deadline and the REPLAY_TIMEOUT_EXIT_DEADLINE_MS constant.

Why 60s: comfortably above the observed 47s p99 in the incident, well under the 240s replay timeout so upstream retries still have room, and much shorter than the maxDuration SIGTERM horizon.

Impact

  • Covers allworld.* calls through world-vercel (events, runs, steps, queue, hooks, etc.), not just the replay timeout path.
  • Hangs now surface as typed WorkflowWorldErrors — existing catch sites get predictable retry/failure semantics instead of an infinite await.
  • Happy path is unchanged: AbortSignal.timeout() doesn't fire on normal requests and the unref'd timer doesn't hold the event loop open.

Test plan

  • pnpm typecheck on @workflow/core + @workflow/world-vercel
  • Added packages/world-vercel/src/utils.test.ts with two cases:
    • TimeoutError from fetch is wrapped into WorkflowWorldError with elapsed ms and preserved cause
    • Non-timeout errors (e.g. TypeError) propagate unchanged
  • Full world-vercel suite passes (81 tests)
  • Full core suite passes (591 tests)
  • Preview deployment: verify normal requests still succeed under the 60s budget
  • Preview deployment with simulated server hang: verify the call fails fast as WorkflowWorldError instead of running to maxDuration

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts
…imeout
Made-with: Cursor
# Conflicts:
#	packages/world-vercel/src/utils.test.ts
#	packages/world-vercel/src/utils.ts
- Compose per-request timeout with caller-provided AbortSignal via
AbortSignal.any() so a future caller passing options.signal doesn't
silently lose the hang protection (and vice versa).
- Move the floating eslint-disable-next-line for the undici dispatcher
cast back next to the actual `fetch(... as any)` call where the
suppression applies, instead of pointing at `const fetchStart`.
Both nits flagged by @VaguelySerious in the AI review on PR #1807.
Made-with: Cursor
Comment thread.changeset/fix-world-vercel-request-timeout.md Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
karthikscale3and others added 2 commits May 1, 2026 14:07
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — withdrawing my prior CHANGES_REQUESTED. The author took the suggestion from my earlier review: the process.exit workaround in runtime.ts is gone, replaced with a transport-level timeout in world-vercel/utils.ts:355 (AbortSignal.timeout(60_000) plumbed into the fetch call). This is the right layer — fixes the underlying issue at the single point where it lives, covers all 27+ world.events.create() call sites uniformly, produces a typed WorkflowWorldError that existing catch sites recognize.

Implementation looks solid:

  • Timeout primitive: AbortSignal.timeout(60_000) → DOMException with name: 'TimeoutError' → caught and wrapped as WorkflowWorldError with cause preserved. Standard pattern, clean.
  • Composition with options.signal: AbortSignal.any([options.signal, timeoutSignal]) for when callers eventually pass their own signals. Currently dead code per the comment, but wired correctly for future use.
  • Error mapping: error message includes ${method} ${endpoint} and ${elapsed}ms — useful for debugging, attaches url via the WorkflowWorldError constructor.
  • Span attributes: ErrorType('TIMEOUT') for OTEL, plus recordException. Consistent with sibling status-code branches.
  • Tests: utils.test.ts covers both the wrap-on-timeout path and the pass-through-on-non-timeout path. Mocks fetch with synthetic errors. All 94 world-vercel tests still pass.
  • Changeset scope: @workflow/world-vercel only, which is correct — no behavior change in core.

A few non-blocking concerns worth raising. None are gating; mostly forward-looking.

1. start() retry classification doesn't match timeouts as retryable

isRetryableStartError in start.ts:331 only matches WorkflowWorldError with status >= 500. The new timeout error has no status set, so it falls into the throw err branch at line 283.

Concrete consequence: when events.create(run_created) times out but the parallel queue dispatch already succeeded, the user sees start() throw "POST /runs/... timed out after 60000ms" while the workflow actually does run via the queue path. That's misleading — the right behavior is to mark this as "resilient start" and continue (runtime.ts will retry the run_created event later).

Suggested adjustment:

functionisRetryableStartError(err: unknown): boolean{if(ThrottleError.is(err))returntrue;if(WorkflowWorldError.is(err)){// 5xx server errors and timeouts (no status) are both transientif(err.status===undefined)returntrue;if(err.status>=500)returntrue;}returnfalse;}

This is technically a behavior change in core, so it'd need a separate @workflow/core patch in the changeset. Could be a follow-up if you want to keep this PR scoped to world-vercel.

2. Node 18 + AbortSignal.any

The repo's root engines.node is ^18.0.0 || ^20.0.0 || ^22.0.0 || ^24.0.0. AbortSignal.any() was added in Node 20.3 (May 2023) — not available on Node 18. The branch at utils.ts:359 only fires when options.signal is set, which the comment notes is currently never. So today this is a latent issue, not an active one. But anyone wiring up a caller-provided signal in the future will get a TypeError at runtime on Node 18.

Either drop Node 18 from engines (probably the right move overall — Node 18 is EOL April 2025 as of writing) or guard with a feature check. Could go in a separate PR.

3. Hardcoded 60s timeout

The chosen value barely covers the slowest legitimate case in your incident report (47s). A successful but slow-starting request at, say, 55s would now succeed but be on the edge; a hung request takes 60s to detect.

That's reasonable as a default, but I'd consider exposing it as a config knob (like the existing VERCEL_WORKFLOW_SERVER_URL env var pattern) for users with different latency profiles. Probably fine to wait for someone to ask.

Wrap-up

Good fix. The transport-level approach is correct and the concerns above are forward-looking polish, not gating.

@TooTallNate

Copy link
Copy Markdown
Member

Aside on my own concerns from the approval — I should retract one thing. I was briefly worried the 60s timeout might affect the long-lived stream GET endpoint, which can legitimately stay open for the full function duration. Confirmed it doesn't: looking at streamer.ts, the long-lived read at line 283 (getReadable/get) goes through a direct fetch() call, notmakeRequest. Same for the write/writeMulti/close paths — direct fetch.

Only the discrete request/response calls go through makeRequest:

  • events.create (and similar) — milliseconds normally
  • runs.get / runs.list — milliseconds normally
  • getChunks (paginated stream chunks) — bounded per-page
  • getInfo (stream metadata) — trivial

60s is appropriately generous for all of those. So the design is correct as-is — no concern about the streaming endpoint.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@karthikscale3@VaguelySerious@TooTallNate
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

fix(world-vercel): add default request timeout to workflow-server HTTP calls - #1807

Merged
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout
May 4, 2026
Merged

fix(world-vercel): add default request timeout to workflow-server HTTP calls#1807
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout

Conversation

@karthikscale3

@karthikscale3karthikscale3 commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a default per-request timeout to makeRequest() in world-vercel so hung responses from workflow-server can't burn compute up to the function's maxDuration.

Background (original problem)

A production workflow (wrun_01KPDFGK4QBFZ7XERXN9NP7VY2) on a preview deployment showed:

  • run_started POST to workflow-server took 47s (server under load)
  • Replay timeout fired at 240s
  • The run_failed write was sent but the response never came back (External APIs showed ∅ "Timed out while waiting for a response")
  • The function continued running for 15 minutes (hit maxDuration) before SIGTERM — ~11 minutes of compute burned doing nothing

Original fix (reverted)

The first version of this PR added a 30s hard-exit deadline to the replay timeout handler in packages/core/src/runtime.ts. Per Nate's review, this was the wrong layer: it only protected one of 27 world.events.create() call sites in core, leaving the other 26 (and every other world.* method going through makeRequest()) exposed to the exact same hang.

New fix

Moved the mitigation down into the world-vercel transport layer, where all world.* calls funnel through makeRequest():

  • packages/world-vercel/src/utils.ts — attaches AbortSignal.timeout(60_000) to every makeRequest() fetch. A TimeoutError or AbortError from fetch is converted into a WorkflowWorldError (with the original error preserved as cause and elapsed ms in the message), so existing catch sites handle it uniformly. The span is tagged with ErrorType('TIMEOUT').
  • Reverted the runtime-level exit deadline and the REPLAY_TIMEOUT_EXIT_DEADLINE_MS constant.

Why 60s: comfortably above the observed 47s p99 in the incident, well under the 240s replay timeout so upstream retries still have room, and much shorter than the maxDuration SIGTERM horizon.

Impact

  • Covers allworld.* calls through world-vercel (events, runs, steps, queue, hooks, etc.), not just the replay timeout path.
  • Hangs now surface as typed WorkflowWorldErrors — existing catch sites get predictable retry/failure semantics instead of an infinite await.
  • Happy path is unchanged: AbortSignal.timeout() doesn't fire on normal requests and the unref'd timer doesn't hold the event loop open.

Test plan

  • pnpm typecheck on @workflow/core + @workflow/world-vercel
  • Added packages/world-vercel/src/utils.test.ts with two cases:
    • TimeoutError from fetch is wrapped into WorkflowWorldError with elapsed ms and preserved cause
    • Non-timeout errors (e.g. TypeError) propagate unchanged
  • Full world-vercel suite passes (81 tests)
  • Full core suite passes (591 tests)
  • Preview deployment: verify normal requests still succeed under the 60s budget
  • Preview deployment with simulated server hang: verify the call fails fast as WorkflowWorldError instead of running to maxDuration

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts
…imeout
Made-with: Cursor
# Conflicts:
#	packages/world-vercel/src/utils.test.ts
#	packages/world-vercel/src/utils.ts
- Compose per-request timeout with caller-provided AbortSignal via
AbortSignal.any() so a future caller passing options.signal doesn't
silently lose the hang protection (and vice versa).
- Move the floating eslint-disable-next-line for the undici dispatcher
cast back next to the actual `fetch(... as any)` call where the
suppression applies, instead of pointing at `const fetchStart`.
Both nits flagged by @VaguelySerious in the AI review on PR #1807.
Made-with: Cursor
Comment thread.changeset/fix-world-vercel-request-timeout.md Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
karthikscale3and others added 2 commits May 1, 2026 14:07
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — withdrawing my prior CHANGES_REQUESTED. The author took the suggestion from my earlier review: the process.exit workaround in runtime.ts is gone, replaced with a transport-level timeout in world-vercel/utils.ts:355 (AbortSignal.timeout(60_000) plumbed into the fetch call). This is the right layer — fixes the underlying issue at the single point where it lives, covers all 27+ world.events.create() call sites uniformly, produces a typed WorkflowWorldError that existing catch sites recognize.

Implementation looks solid:

  • Timeout primitive: AbortSignal.timeout(60_000) → DOMException with name: 'TimeoutError' → caught and wrapped as WorkflowWorldError with cause preserved. Standard pattern, clean.
  • Composition with options.signal: AbortSignal.any([options.signal, timeoutSignal]) for when callers eventually pass their own signals. Currently dead code per the comment, but wired correctly for future use.
  • Error mapping: error message includes ${method} ${endpoint} and ${elapsed}ms — useful for debugging, attaches url via the WorkflowWorldError constructor.
  • Span attributes: ErrorType('TIMEOUT') for OTEL, plus recordException. Consistent with sibling status-code branches.
  • Tests: utils.test.ts covers both the wrap-on-timeout path and the pass-through-on-non-timeout path. Mocks fetch with synthetic errors. All 94 world-vercel tests still pass.
  • Changeset scope: @workflow/world-vercel only, which is correct — no behavior change in core.

A few non-blocking concerns worth raising. None are gating; mostly forward-looking.

1. start() retry classification doesn't match timeouts as retryable

isRetryableStartError in start.ts:331 only matches WorkflowWorldError with status >= 500. The new timeout error has no status set, so it falls into the throw err branch at line 283.

Concrete consequence: when events.create(run_created) times out but the parallel queue dispatch already succeeded, the user sees start() throw "POST /runs/... timed out after 60000ms" while the workflow actually does run via the queue path. That's misleading — the right behavior is to mark this as "resilient start" and continue (runtime.ts will retry the run_created event later).

Suggested adjustment:

functionisRetryableStartError(err: unknown): boolean{if(ThrottleError.is(err))returntrue;if(WorkflowWorldError.is(err)){// 5xx server errors and timeouts (no status) are both transientif(err.status===undefined)returntrue;if(err.status>=500)returntrue;}returnfalse;}

This is technically a behavior change in core, so it'd need a separate @workflow/core patch in the changeset. Could be a follow-up if you want to keep this PR scoped to world-vercel.

2. Node 18 + AbortSignal.any

The repo's root engines.node is ^18.0.0 || ^20.0.0 || ^22.0.0 || ^24.0.0. AbortSignal.any() was added in Node 20.3 (May 2023) — not available on Node 18. The branch at utils.ts:359 only fires when options.signal is set, which the comment notes is currently never. So today this is a latent issue, not an active one. But anyone wiring up a caller-provided signal in the future will get a TypeError at runtime on Node 18.

Either drop Node 18 from engines (probably the right move overall — Node 18 is EOL April 2025 as of writing) or guard with a feature check. Could go in a separate PR.

3. Hardcoded 60s timeout

The chosen value barely covers the slowest legitimate case in your incident report (47s). A successful but slow-starting request at, say, 55s would now succeed but be on the edge; a hung request takes 60s to detect.

That's reasonable as a default, but I'd consider exposing it as a config knob (like the existing VERCEL_WORKFLOW_SERVER_URL env var pattern) for users with different latency profiles. Probably fine to wait for someone to ask.

Wrap-up

Good fix. The transport-level approach is correct and the concerns above are forward-looking polish, not gating.

@TooTallNate

Copy link
Copy Markdown
Member

Aside on my own concerns from the approval — I should retract one thing. I was briefly worried the 60s timeout might affect the long-lived stream GET endpoint, which can legitimately stay open for the full function duration. Confirmed it doesn't: looking at streamer.ts, the long-lived read at line 283 (getReadable/get) goes through a direct fetch() call, notmakeRequest. Same for the write/writeMulti/close paths — direct fetch.

Only the discrete request/response calls go through makeRequest:

  • events.create (and similar) — milliseconds normally
  • runs.get / runs.list — milliseconds normally
  • getChunks (paginated stream chunks) — bounded per-page
  • getInfo (stream metadata) — trivial

60s is appropriately generous for all of those. So the design is correct as-is — no concern about the streaming endpoint.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@karthikscale3@VaguelySerious@TooTallNate
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(world-vercel): add default request timeout to workflow-server HTTP calls - #1807

Merged
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout
May 4, 2026
Merged

fix(world-vercel): add default request timeout to workflow-server HTTP calls#1807
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout

Conversation

@karthikscale3

@karthikscale3karthikscale3 commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a default per-request timeout to makeRequest() in world-vercel so hung responses from workflow-server can't burn compute up to the function's maxDuration.

Background (original problem)

A production workflow (wrun_01KPDFGK4QBFZ7XERXN9NP7VY2) on a preview deployment showed:

  • run_started POST to workflow-server took 47s (server under load)
  • Replay timeout fired at 240s
  • The run_failed write was sent but the response never came back (External APIs showed ∅ "Timed out while waiting for a response")
  • The function continued running for 15 minutes (hit maxDuration) before SIGTERM — ~11 minutes of compute burned doing nothing

Original fix (reverted)

The first version of this PR added a 30s hard-exit deadline to the replay timeout handler in packages/core/src/runtime.ts. Per Nate's review, this was the wrong layer: it only protected one of 27 world.events.create() call sites in core, leaving the other 26 (and every other world.* method going through makeRequest()) exposed to the exact same hang.

New fix

Moved the mitigation down into the world-vercel transport layer, where all world.* calls funnel through makeRequest():

  • packages/world-vercel/src/utils.ts — attaches AbortSignal.timeout(60_000) to every makeRequest() fetch. A TimeoutError or AbortError from fetch is converted into a WorkflowWorldError (with the original error preserved as cause and elapsed ms in the message), so existing catch sites handle it uniformly. The span is tagged with ErrorType('TIMEOUT').
  • Reverted the runtime-level exit deadline and the REPLAY_TIMEOUT_EXIT_DEADLINE_MS constant.

Why 60s: comfortably above the observed 47s p99 in the incident, well under the 240s replay timeout so upstream retries still have room, and much shorter than the maxDuration SIGTERM horizon.

Impact

  • Covers allworld.* calls through world-vercel (events, runs, steps, queue, hooks, etc.), not just the replay timeout path.
  • Hangs now surface as typed WorkflowWorldErrors — existing catch sites get predictable retry/failure semantics instead of an infinite await.
  • Happy path is unchanged: AbortSignal.timeout() doesn't fire on normal requests and the unref'd timer doesn't hold the event loop open.

Test plan

  • pnpm typecheck on @workflow/core + @workflow/world-vercel
  • Added packages/world-vercel/src/utils.test.ts with two cases:
    • TimeoutError from fetch is wrapped into WorkflowWorldError with elapsed ms and preserved cause
    • Non-timeout errors (e.g. TypeError) propagate unchanged
  • Full world-vercel suite passes (81 tests)
  • Full core suite passes (591 tests)
  • Preview deployment: verify normal requests still succeed under the 60s budget
  • Preview deployment with simulated server hang: verify the call fails fast as WorkflowWorldError instead of running to maxDuration

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts
…imeout
Made-with: Cursor
# Conflicts:
#	packages/world-vercel/src/utils.test.ts
#	packages/world-vercel/src/utils.ts
- Compose per-request timeout with caller-provided AbortSignal via
AbortSignal.any() so a future caller passing options.signal doesn't
silently lose the hang protection (and vice versa).
- Move the floating eslint-disable-next-line for the undici dispatcher
cast back next to the actual `fetch(... as any)` call where the
suppression applies, instead of pointing at `const fetchStart`.
Both nits flagged by @VaguelySerious in the AI review on PR #1807.
Made-with: Cursor
Comment thread.changeset/fix-world-vercel-request-timeout.md Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
karthikscale3and others added 2 commits May 1, 2026 14:07
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — withdrawing my prior CHANGES_REQUESTED. The author took the suggestion from my earlier review: the process.exit workaround in runtime.ts is gone, replaced with a transport-level timeout in world-vercel/utils.ts:355 (AbortSignal.timeout(60_000) plumbed into the fetch call). This is the right layer — fixes the underlying issue at the single point where it lives, covers all 27+ world.events.create() call sites uniformly, produces a typed WorkflowWorldError that existing catch sites recognize.

Implementation looks solid:

  • Timeout primitive: AbortSignal.timeout(60_000) → DOMException with name: 'TimeoutError' → caught and wrapped as WorkflowWorldError with cause preserved. Standard pattern, clean.
  • Composition with options.signal: AbortSignal.any([options.signal, timeoutSignal]) for when callers eventually pass their own signals. Currently dead code per the comment, but wired correctly for future use.
  • Error mapping: error message includes ${method} ${endpoint} and ${elapsed}ms — useful for debugging, attaches url via the WorkflowWorldError constructor.
  • Span attributes: ErrorType('TIMEOUT') for OTEL, plus recordException. Consistent with sibling status-code branches.
  • Tests: utils.test.ts covers both the wrap-on-timeout path and the pass-through-on-non-timeout path. Mocks fetch with synthetic errors. All 94 world-vercel tests still pass.
  • Changeset scope: @workflow/world-vercel only, which is correct — no behavior change in core.

A few non-blocking concerns worth raising. None are gating; mostly forward-looking.

1. start() retry classification doesn't match timeouts as retryable

isRetryableStartError in start.ts:331 only matches WorkflowWorldError with status >= 500. The new timeout error has no status set, so it falls into the throw err branch at line 283.

Concrete consequence: when events.create(run_created) times out but the parallel queue dispatch already succeeded, the user sees start() throw "POST /runs/... timed out after 60000ms" while the workflow actually does run via the queue path. That's misleading — the right behavior is to mark this as "resilient start" and continue (runtime.ts will retry the run_created event later).

Suggested adjustment:

functionisRetryableStartError(err: unknown): boolean{if(ThrottleError.is(err))returntrue;if(WorkflowWorldError.is(err)){// 5xx server errors and timeouts (no status) are both transientif(err.status===undefined)returntrue;if(err.status>=500)returntrue;}returnfalse;}

This is technically a behavior change in core, so it'd need a separate @workflow/core patch in the changeset. Could be a follow-up if you want to keep this PR scoped to world-vercel.

2. Node 18 + AbortSignal.any

The repo's root engines.node is ^18.0.0 || ^20.0.0 || ^22.0.0 || ^24.0.0. AbortSignal.any() was added in Node 20.3 (May 2023) — not available on Node 18. The branch at utils.ts:359 only fires when options.signal is set, which the comment notes is currently never. So today this is a latent issue, not an active one. But anyone wiring up a caller-provided signal in the future will get a TypeError at runtime on Node 18.

Either drop Node 18 from engines (probably the right move overall — Node 18 is EOL April 2025 as of writing) or guard with a feature check. Could go in a separate PR.

3. Hardcoded 60s timeout

The chosen value barely covers the slowest legitimate case in your incident report (47s). A successful but slow-starting request at, say, 55s would now succeed but be on the edge; a hung request takes 60s to detect.

That's reasonable as a default, but I'd consider exposing it as a config knob (like the existing VERCEL_WORKFLOW_SERVER_URL env var pattern) for users with different latency profiles. Probably fine to wait for someone to ask.

Wrap-up

Good fix. The transport-level approach is correct and the concerns above are forward-looking polish, not gating.

@TooTallNate

Copy link
Copy Markdown
Member

Aside on my own concerns from the approval — I should retract one thing. I was briefly worried the 60s timeout might affect the long-lived stream GET endpoint, which can legitimately stay open for the full function duration. Confirmed it doesn't: looking at streamer.ts, the long-lived read at line 283 (getReadable/get) goes through a direct fetch() call, notmakeRequest. Same for the write/writeMulti/close paths — direct fetch.

Only the discrete request/response calls go through makeRequest:

  • events.create (and similar) — milliseconds normally
  • runs.get / runs.list — milliseconds normally
  • getChunks (paginated stream chunks) — bounded per-page
  • getInfo (stream metadata) — trivial

60s is appropriately generous for all of those. So the design is correct as-is — no concern about the streaming endpoint.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@karthikscale3@VaguelySerious@TooTallNate
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(world-vercel): add default request timeout to workflow-server HTTP calls - #1807

Merged
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout
May 4, 2026
Merged

fix(world-vercel): add default request timeout to workflow-server HTTP calls#1807
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout

Conversation

@karthikscale3

@karthikscale3karthikscale3 commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a default per-request timeout to makeRequest() in world-vercel so hung responses from workflow-server can't burn compute up to the function's maxDuration.

Background (original problem)

A production workflow (wrun_01KPDFGK4QBFZ7XERXN9NP7VY2) on a preview deployment showed:

  • run_started POST to workflow-server took 47s (server under load)
  • Replay timeout fired at 240s
  • The run_failed write was sent but the response never came back (External APIs showed ∅ "Timed out while waiting for a response")
  • The function continued running for 15 minutes (hit maxDuration) before SIGTERM — ~11 minutes of compute burned doing nothing

Original fix (reverted)

The first version of this PR added a 30s hard-exit deadline to the replay timeout handler in packages/core/src/runtime.ts. Per Nate's review, this was the wrong layer: it only protected one of 27 world.events.create() call sites in core, leaving the other 26 (and every other world.* method going through makeRequest()) exposed to the exact same hang.

New fix

Moved the mitigation down into the world-vercel transport layer, where all world.* calls funnel through makeRequest():

  • packages/world-vercel/src/utils.ts — attaches AbortSignal.timeout(60_000) to every makeRequest() fetch. A TimeoutError or AbortError from fetch is converted into a WorkflowWorldError (with the original error preserved as cause and elapsed ms in the message), so existing catch sites handle it uniformly. The span is tagged with ErrorType('TIMEOUT').
  • Reverted the runtime-level exit deadline and the REPLAY_TIMEOUT_EXIT_DEADLINE_MS constant.

Why 60s: comfortably above the observed 47s p99 in the incident, well under the 240s replay timeout so upstream retries still have room, and much shorter than the maxDuration SIGTERM horizon.

Impact

  • Covers allworld.* calls through world-vercel (events, runs, steps, queue, hooks, etc.), not just the replay timeout path.
  • Hangs now surface as typed WorkflowWorldErrors — existing catch sites get predictable retry/failure semantics instead of an infinite await.
  • Happy path is unchanged: AbortSignal.timeout() doesn't fire on normal requests and the unref'd timer doesn't hold the event loop open.

Test plan

  • pnpm typecheck on @workflow/core + @workflow/world-vercel
  • Added packages/world-vercel/src/utils.test.ts with two cases:
    • TimeoutError from fetch is wrapped into WorkflowWorldError with elapsed ms and preserved cause
    • Non-timeout errors (e.g. TypeError) propagate unchanged
  • Full world-vercel suite passes (81 tests)
  • Full core suite passes (591 tests)
  • Preview deployment: verify normal requests still succeed under the 60s budget
  • Preview deployment with simulated server hang: verify the call fails fast as WorkflowWorldError instead of running to maxDuration

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts
…imeout
Made-with: Cursor
# Conflicts:
#	packages/world-vercel/src/utils.test.ts
#	packages/world-vercel/src/utils.ts
- Compose per-request timeout with caller-provided AbortSignal via
AbortSignal.any() so a future caller passing options.signal doesn't
silently lose the hang protection (and vice versa).
- Move the floating eslint-disable-next-line for the undici dispatcher
cast back next to the actual `fetch(... as any)` call where the
suppression applies, instead of pointing at `const fetchStart`.
Both nits flagged by @VaguelySerious in the AI review on PR #1807.
Made-with: Cursor
Comment thread.changeset/fix-world-vercel-request-timeout.md Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
karthikscale3and others added 2 commits May 1, 2026 14:07
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — withdrawing my prior CHANGES_REQUESTED. The author took the suggestion from my earlier review: the process.exit workaround in runtime.ts is gone, replaced with a transport-level timeout in world-vercel/utils.ts:355 (AbortSignal.timeout(60_000) plumbed into the fetch call). This is the right layer — fixes the underlying issue at the single point where it lives, covers all 27+ world.events.create() call sites uniformly, produces a typed WorkflowWorldError that existing catch sites recognize.

Implementation looks solid:

  • Timeout primitive: AbortSignal.timeout(60_000) → DOMException with name: 'TimeoutError' → caught and wrapped as WorkflowWorldError with cause preserved. Standard pattern, clean.
  • Composition with options.signal: AbortSignal.any([options.signal, timeoutSignal]) for when callers eventually pass their own signals. Currently dead code per the comment, but wired correctly for future use.
  • Error mapping: error message includes ${method} ${endpoint} and ${elapsed}ms — useful for debugging, attaches url via the WorkflowWorldError constructor.
  • Span attributes: ErrorType('TIMEOUT') for OTEL, plus recordException. Consistent with sibling status-code branches.
  • Tests: utils.test.ts covers both the wrap-on-timeout path and the pass-through-on-non-timeout path. Mocks fetch with synthetic errors. All 94 world-vercel tests still pass.
  • Changeset scope: @workflow/world-vercel only, which is correct — no behavior change in core.

A few non-blocking concerns worth raising. None are gating; mostly forward-looking.

1. start() retry classification doesn't match timeouts as retryable

isRetryableStartError in start.ts:331 only matches WorkflowWorldError with status >= 500. The new timeout error has no status set, so it falls into the throw err branch at line 283.

Concrete consequence: when events.create(run_created) times out but the parallel queue dispatch already succeeded, the user sees start() throw "POST /runs/... timed out after 60000ms" while the workflow actually does run via the queue path. That's misleading — the right behavior is to mark this as "resilient start" and continue (runtime.ts will retry the run_created event later).

Suggested adjustment:

functionisRetryableStartError(err: unknown): boolean{if(ThrottleError.is(err))returntrue;if(WorkflowWorldError.is(err)){// 5xx server errors and timeouts (no status) are both transientif(err.status===undefined)returntrue;if(err.status>=500)returntrue;}returnfalse;}

This is technically a behavior change in core, so it'd need a separate @workflow/core patch in the changeset. Could be a follow-up if you want to keep this PR scoped to world-vercel.

2. Node 18 + AbortSignal.any

The repo's root engines.node is ^18.0.0 || ^20.0.0 || ^22.0.0 || ^24.0.0. AbortSignal.any() was added in Node 20.3 (May 2023) — not available on Node 18. The branch at utils.ts:359 only fires when options.signal is set, which the comment notes is currently never. So today this is a latent issue, not an active one. But anyone wiring up a caller-provided signal in the future will get a TypeError at runtime on Node 18.

Either drop Node 18 from engines (probably the right move overall — Node 18 is EOL April 2025 as of writing) or guard with a feature check. Could go in a separate PR.

3. Hardcoded 60s timeout

The chosen value barely covers the slowest legitimate case in your incident report (47s). A successful but slow-starting request at, say, 55s would now succeed but be on the edge; a hung request takes 60s to detect.

That's reasonable as a default, but I'd consider exposing it as a config knob (like the existing VERCEL_WORKFLOW_SERVER_URL env var pattern) for users with different latency profiles. Probably fine to wait for someone to ask.

Wrap-up

Good fix. The transport-level approach is correct and the concerns above are forward-looking polish, not gating.

@TooTallNate

Copy link
Copy Markdown
Member

Aside on my own concerns from the approval — I should retract one thing. I was briefly worried the 60s timeout might affect the long-lived stream GET endpoint, which can legitimately stay open for the full function duration. Confirmed it doesn't: looking at streamer.ts, the long-lived read at line 283 (getReadable/get) goes through a direct fetch() call, notmakeRequest. Same for the write/writeMulti/close paths — direct fetch.

Only the discrete request/response calls go through makeRequest:

  • events.create (and similar) — milliseconds normally
  • runs.get / runs.list — milliseconds normally
  • getChunks (paginated stream chunks) — bounded per-page
  • getInfo (stream metadata) — trivial

60s is appropriately generous for all of those. So the design is correct as-is — no concern about the streaming endpoint.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@karthikscale3@VaguelySerious@TooTallNate
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

fix(world-vercel): add default request timeout to workflow-server HTTP calls - #1807

Merged
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout
May 4, 2026
Merged

fix(world-vercel): add default request timeout to workflow-server HTTP calls#1807
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout

Conversation

@karthikscale3

@karthikscale3karthikscale3 commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a default per-request timeout to makeRequest() in world-vercel so hung responses from workflow-server can't burn compute up to the function's maxDuration.

Background (original problem)

A production workflow (wrun_01KPDFGK4QBFZ7XERXN9NP7VY2) on a preview deployment showed:

  • run_started POST to workflow-server took 47s (server under load)
  • Replay timeout fired at 240s
  • The run_failed write was sent but the response never came back (External APIs showed ∅ "Timed out while waiting for a response")
  • The function continued running for 15 minutes (hit maxDuration) before SIGTERM — ~11 minutes of compute burned doing nothing

Original fix (reverted)

The first version of this PR added a 30s hard-exit deadline to the replay timeout handler in packages/core/src/runtime.ts. Per Nate's review, this was the wrong layer: it only protected one of 27 world.events.create() call sites in core, leaving the other 26 (and every other world.* method going through makeRequest()) exposed to the exact same hang.

New fix

Moved the mitigation down into the world-vercel transport layer, where all world.* calls funnel through makeRequest():

  • packages/world-vercel/src/utils.ts — attaches AbortSignal.timeout(60_000) to every makeRequest() fetch. A TimeoutError or AbortError from fetch is converted into a WorkflowWorldError (with the original error preserved as cause and elapsed ms in the message), so existing catch sites handle it uniformly. The span is tagged with ErrorType('TIMEOUT').
  • Reverted the runtime-level exit deadline and the REPLAY_TIMEOUT_EXIT_DEADLINE_MS constant.

Why 60s: comfortably above the observed 47s p99 in the incident, well under the 240s replay timeout so upstream retries still have room, and much shorter than the maxDuration SIGTERM horizon.

Impact

  • Covers allworld.* calls through world-vercel (events, runs, steps, queue, hooks, etc.), not just the replay timeout path.
  • Hangs now surface as typed WorkflowWorldErrors — existing catch sites get predictable retry/failure semantics instead of an infinite await.
  • Happy path is unchanged: AbortSignal.timeout() doesn't fire on normal requests and the unref'd timer doesn't hold the event loop open.

Test plan

  • pnpm typecheck on @workflow/core + @workflow/world-vercel
  • Added packages/world-vercel/src/utils.test.ts with two cases:
    • TimeoutError from fetch is wrapped into WorkflowWorldError with elapsed ms and preserved cause
    • Non-timeout errors (e.g. TypeError) propagate unchanged
  • Full world-vercel suite passes (81 tests)
  • Full core suite passes (591 tests)
  • Preview deployment: verify normal requests still succeed under the 60s budget
  • Preview deployment with simulated server hang: verify the call fails fast as WorkflowWorldError instead of running to maxDuration

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts
…imeout
Made-with: Cursor
# Conflicts:
#	packages/world-vercel/src/utils.test.ts
#	packages/world-vercel/src/utils.ts
- Compose per-request timeout with caller-provided AbortSignal via
AbortSignal.any() so a future caller passing options.signal doesn't
silently lose the hang protection (and vice versa).
- Move the floating eslint-disable-next-line for the undici dispatcher
cast back next to the actual `fetch(... as any)` call where the
suppression applies, instead of pointing at `const fetchStart`.
Both nits flagged by @VaguelySerious in the AI review on PR #1807.
Made-with: Cursor
Comment thread.changeset/fix-world-vercel-request-timeout.md Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
karthikscale3and others added 2 commits May 1, 2026 14:07
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — withdrawing my prior CHANGES_REQUESTED. The author took the suggestion from my earlier review: the process.exit workaround in runtime.ts is gone, replaced with a transport-level timeout in world-vercel/utils.ts:355 (AbortSignal.timeout(60_000) plumbed into the fetch call). This is the right layer — fixes the underlying issue at the single point where it lives, covers all 27+ world.events.create() call sites uniformly, produces a typed WorkflowWorldError that existing catch sites recognize.

Implementation looks solid:

  • Timeout primitive: AbortSignal.timeout(60_000) → DOMException with name: 'TimeoutError' → caught and wrapped as WorkflowWorldError with cause preserved. Standard pattern, clean.
  • Composition with options.signal: AbortSignal.any([options.signal, timeoutSignal]) for when callers eventually pass their own signals. Currently dead code per the comment, but wired correctly for future use.
  • Error mapping: error message includes ${method} ${endpoint} and ${elapsed}ms — useful for debugging, attaches url via the WorkflowWorldError constructor.
  • Span attributes: ErrorType('TIMEOUT') for OTEL, plus recordException. Consistent with sibling status-code branches.
  • Tests: utils.test.ts covers both the wrap-on-timeout path and the pass-through-on-non-timeout path. Mocks fetch with synthetic errors. All 94 world-vercel tests still pass.
  • Changeset scope: @workflow/world-vercel only, which is correct — no behavior change in core.

A few non-blocking concerns worth raising. None are gating; mostly forward-looking.

1. start() retry classification doesn't match timeouts as retryable

isRetryableStartError in start.ts:331 only matches WorkflowWorldError with status >= 500. The new timeout error has no status set, so it falls into the throw err branch at line 283.

Concrete consequence: when events.create(run_created) times out but the parallel queue dispatch already succeeded, the user sees start() throw "POST /runs/... timed out after 60000ms" while the workflow actually does run via the queue path. That's misleading — the right behavior is to mark this as "resilient start" and continue (runtime.ts will retry the run_created event later).

Suggested adjustment:

functionisRetryableStartError(err: unknown): boolean{if(ThrottleError.is(err))returntrue;if(WorkflowWorldError.is(err)){// 5xx server errors and timeouts (no status) are both transientif(err.status===undefined)returntrue;if(err.status>=500)returntrue;}returnfalse;}

This is technically a behavior change in core, so it'd need a separate @workflow/core patch in the changeset. Could be a follow-up if you want to keep this PR scoped to world-vercel.

2. Node 18 + AbortSignal.any

The repo's root engines.node is ^18.0.0 || ^20.0.0 || ^22.0.0 || ^24.0.0. AbortSignal.any() was added in Node 20.3 (May 2023) — not available on Node 18. The branch at utils.ts:359 only fires when options.signal is set, which the comment notes is currently never. So today this is a latent issue, not an active one. But anyone wiring up a caller-provided signal in the future will get a TypeError at runtime on Node 18.

Either drop Node 18 from engines (probably the right move overall — Node 18 is EOL April 2025 as of writing) or guard with a feature check. Could go in a separate PR.

3. Hardcoded 60s timeout

The chosen value barely covers the slowest legitimate case in your incident report (47s). A successful but slow-starting request at, say, 55s would now succeed but be on the edge; a hung request takes 60s to detect.

That's reasonable as a default, but I'd consider exposing it as a config knob (like the existing VERCEL_WORKFLOW_SERVER_URL env var pattern) for users with different latency profiles. Probably fine to wait for someone to ask.

Wrap-up

Good fix. The transport-level approach is correct and the concerns above are forward-looking polish, not gating.

@TooTallNate

Copy link
Copy Markdown
Member

Aside on my own concerns from the approval — I should retract one thing. I was briefly worried the 60s timeout might affect the long-lived stream GET endpoint, which can legitimately stay open for the full function duration. Confirmed it doesn't: looking at streamer.ts, the long-lived read at line 283 (getReadable/get) goes through a direct fetch() call, notmakeRequest. Same for the write/writeMulti/close paths — direct fetch.

Only the discrete request/response calls go through makeRequest:

  • events.create (and similar) — milliseconds normally
  • runs.get / runs.list — milliseconds normally
  • getChunks (paginated stream chunks) — bounded per-page
  • getInfo (stream metadata) — trivial

60s is appropriately generous for all of those. So the design is correct as-is — no concern about the streaming endpoint.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@karthikscale3@VaguelySerious@TooTallNate
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(world-vercel): add default request timeout to workflow-server HTTP calls - #1807

Merged
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout
May 4, 2026
Merged

fix(world-vercel): add default request timeout to workflow-server HTTP calls#1807
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout

Conversation

@karthikscale3

@karthikscale3karthikscale3 commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a default per-request timeout to makeRequest() in world-vercel so hung responses from workflow-server can't burn compute up to the function's maxDuration.

Background (original problem)

A production workflow (wrun_01KPDFGK4QBFZ7XERXN9NP7VY2) on a preview deployment showed:

  • run_started POST to workflow-server took 47s (server under load)
  • Replay timeout fired at 240s
  • The run_failed write was sent but the response never came back (External APIs showed ∅ "Timed out while waiting for a response")
  • The function continued running for 15 minutes (hit maxDuration) before SIGTERM — ~11 minutes of compute burned doing nothing

Original fix (reverted)

The first version of this PR added a 30s hard-exit deadline to the replay timeout handler in packages/core/src/runtime.ts. Per Nate's review, this was the wrong layer: it only protected one of 27 world.events.create() call sites in core, leaving the other 26 (and every other world.* method going through makeRequest()) exposed to the exact same hang.

New fix

Moved the mitigation down into the world-vercel transport layer, where all world.* calls funnel through makeRequest():

  • packages/world-vercel/src/utils.ts — attaches AbortSignal.timeout(60_000) to every makeRequest() fetch. A TimeoutError or AbortError from fetch is converted into a WorkflowWorldError (with the original error preserved as cause and elapsed ms in the message), so existing catch sites handle it uniformly. The span is tagged with ErrorType('TIMEOUT').
  • Reverted the runtime-level exit deadline and the REPLAY_TIMEOUT_EXIT_DEADLINE_MS constant.

Why 60s: comfortably above the observed 47s p99 in the incident, well under the 240s replay timeout so upstream retries still have room, and much shorter than the maxDuration SIGTERM horizon.

Impact

  • Covers allworld.* calls through world-vercel (events, runs, steps, queue, hooks, etc.), not just the replay timeout path.
  • Hangs now surface as typed WorkflowWorldErrors — existing catch sites get predictable retry/failure semantics instead of an infinite await.
  • Happy path is unchanged: AbortSignal.timeout() doesn't fire on normal requests and the unref'd timer doesn't hold the event loop open.

Test plan

  • pnpm typecheck on @workflow/core + @workflow/world-vercel
  • Added packages/world-vercel/src/utils.test.ts with two cases:
    • TimeoutError from fetch is wrapped into WorkflowWorldError with elapsed ms and preserved cause
    • Non-timeout errors (e.g. TypeError) propagate unchanged
  • Full world-vercel suite passes (81 tests)
  • Full core suite passes (591 tests)
  • Preview deployment: verify normal requests still succeed under the 60s budget
  • Preview deployment with simulated server hang: verify the call fails fast as WorkflowWorldError instead of running to maxDuration

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts
…imeout
Made-with: Cursor
# Conflicts:
#	packages/world-vercel/src/utils.test.ts
#	packages/world-vercel/src/utils.ts
- Compose per-request timeout with caller-provided AbortSignal via
AbortSignal.any() so a future caller passing options.signal doesn't
silently lose the hang protection (and vice versa).
- Move the floating eslint-disable-next-line for the undici dispatcher
cast back next to the actual `fetch(... as any)` call where the
suppression applies, instead of pointing at `const fetchStart`.
Both nits flagged by @VaguelySerious in the AI review on PR #1807.
Made-with: Cursor
Comment thread.changeset/fix-world-vercel-request-timeout.md Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
karthikscale3and others added 2 commits May 1, 2026 14:07
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — withdrawing my prior CHANGES_REQUESTED. The author took the suggestion from my earlier review: the process.exit workaround in runtime.ts is gone, replaced with a transport-level timeout in world-vercel/utils.ts:355 (AbortSignal.timeout(60_000) plumbed into the fetch call). This is the right layer — fixes the underlying issue at the single point where it lives, covers all 27+ world.events.create() call sites uniformly, produces a typed WorkflowWorldError that existing catch sites recognize.

Implementation looks solid:

  • Timeout primitive: AbortSignal.timeout(60_000) → DOMException with name: 'TimeoutError' → caught and wrapped as WorkflowWorldError with cause preserved. Standard pattern, clean.
  • Composition with options.signal: AbortSignal.any([options.signal, timeoutSignal]) for when callers eventually pass their own signals. Currently dead code per the comment, but wired correctly for future use.
  • Error mapping: error message includes ${method} ${endpoint} and ${elapsed}ms — useful for debugging, attaches url via the WorkflowWorldError constructor.
  • Span attributes: ErrorType('TIMEOUT') for OTEL, plus recordException. Consistent with sibling status-code branches.
  • Tests: utils.test.ts covers both the wrap-on-timeout path and the pass-through-on-non-timeout path. Mocks fetch with synthetic errors. All 94 world-vercel tests still pass.
  • Changeset scope: @workflow/world-vercel only, which is correct — no behavior change in core.

A few non-blocking concerns worth raising. None are gating; mostly forward-looking.

1. start() retry classification doesn't match timeouts as retryable

isRetryableStartError in start.ts:331 only matches WorkflowWorldError with status >= 500. The new timeout error has no status set, so it falls into the throw err branch at line 283.

Concrete consequence: when events.create(run_created) times out but the parallel queue dispatch already succeeded, the user sees start() throw "POST /runs/... timed out after 60000ms" while the workflow actually does run via the queue path. That's misleading — the right behavior is to mark this as "resilient start" and continue (runtime.ts will retry the run_created event later).

Suggested adjustment:

functionisRetryableStartError(err: unknown): boolean{if(ThrottleError.is(err))returntrue;if(WorkflowWorldError.is(err)){// 5xx server errors and timeouts (no status) are both transientif(err.status===undefined)returntrue;if(err.status>=500)returntrue;}returnfalse;}

This is technically a behavior change in core, so it'd need a separate @workflow/core patch in the changeset. Could be a follow-up if you want to keep this PR scoped to world-vercel.

2. Node 18 + AbortSignal.any

The repo's root engines.node is ^18.0.0 || ^20.0.0 || ^22.0.0 || ^24.0.0. AbortSignal.any() was added in Node 20.3 (May 2023) — not available on Node 18. The branch at utils.ts:359 only fires when options.signal is set, which the comment notes is currently never. So today this is a latent issue, not an active one. But anyone wiring up a caller-provided signal in the future will get a TypeError at runtime on Node 18.

Either drop Node 18 from engines (probably the right move overall — Node 18 is EOL April 2025 as of writing) or guard with a feature check. Could go in a separate PR.

3. Hardcoded 60s timeout

The chosen value barely covers the slowest legitimate case in your incident report (47s). A successful but slow-starting request at, say, 55s would now succeed but be on the edge; a hung request takes 60s to detect.

That's reasonable as a default, but I'd consider exposing it as a config knob (like the existing VERCEL_WORKFLOW_SERVER_URL env var pattern) for users with different latency profiles. Probably fine to wait for someone to ask.

Wrap-up

Good fix. The transport-level approach is correct and the concerns above are forward-looking polish, not gating.

@TooTallNate

Copy link
Copy Markdown
Member

Aside on my own concerns from the approval — I should retract one thing. I was briefly worried the 60s timeout might affect the long-lived stream GET endpoint, which can legitimately stay open for the full function duration. Confirmed it doesn't: looking at streamer.ts, the long-lived read at line 283 (getReadable/get) goes through a direct fetch() call, notmakeRequest. Same for the write/writeMulti/close paths — direct fetch.

Only the discrete request/response calls go through makeRequest:

  • events.create (and similar) — milliseconds normally
  • runs.get / runs.list — milliseconds normally
  • getChunks (paginated stream chunks) — bounded per-page
  • getInfo (stream metadata) — trivial

60s is appropriately generous for all of those. So the design is correct as-is — no concern about the streaming endpoint.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@karthikscale3@VaguelySerious@TooTallNate
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(world-vercel): add default request timeout to workflow-server HTTP calls - #1807

Merged
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout
May 4, 2026
Merged

fix(world-vercel): add default request timeout to workflow-server HTTP calls#1807
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout

Conversation

@karthikscale3

@karthikscale3karthikscale3 commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a default per-request timeout to makeRequest() in world-vercel so hung responses from workflow-server can't burn compute up to the function's maxDuration.

Background (original problem)

A production workflow (wrun_01KPDFGK4QBFZ7XERXN9NP7VY2) on a preview deployment showed:

  • run_started POST to workflow-server took 47s (server under load)
  • Replay timeout fired at 240s
  • The run_failed write was sent but the response never came back (External APIs showed ∅ "Timed out while waiting for a response")
  • The function continued running for 15 minutes (hit maxDuration) before SIGTERM — ~11 minutes of compute burned doing nothing

Original fix (reverted)

The first version of this PR added a 30s hard-exit deadline to the replay timeout handler in packages/core/src/runtime.ts. Per Nate's review, this was the wrong layer: it only protected one of 27 world.events.create() call sites in core, leaving the other 26 (and every other world.* method going through makeRequest()) exposed to the exact same hang.

New fix

Moved the mitigation down into the world-vercel transport layer, where all world.* calls funnel through makeRequest():

  • packages/world-vercel/src/utils.ts — attaches AbortSignal.timeout(60_000) to every makeRequest() fetch. A TimeoutError or AbortError from fetch is converted into a WorkflowWorldError (with the original error preserved as cause and elapsed ms in the message), so existing catch sites handle it uniformly. The span is tagged with ErrorType('TIMEOUT').
  • Reverted the runtime-level exit deadline and the REPLAY_TIMEOUT_EXIT_DEADLINE_MS constant.

Why 60s: comfortably above the observed 47s p99 in the incident, well under the 240s replay timeout so upstream retries still have room, and much shorter than the maxDuration SIGTERM horizon.

Impact

  • Covers allworld.* calls through world-vercel (events, runs, steps, queue, hooks, etc.), not just the replay timeout path.
  • Hangs now surface as typed WorkflowWorldErrors — existing catch sites get predictable retry/failure semantics instead of an infinite await.
  • Happy path is unchanged: AbortSignal.timeout() doesn't fire on normal requests and the unref'd timer doesn't hold the event loop open.

Test plan

  • pnpm typecheck on @workflow/core + @workflow/world-vercel
  • Added packages/world-vercel/src/utils.test.ts with two cases:
    • TimeoutError from fetch is wrapped into WorkflowWorldError with elapsed ms and preserved cause
    • Non-timeout errors (e.g. TypeError) propagate unchanged
  • Full world-vercel suite passes (81 tests)
  • Full core suite passes (591 tests)
  • Preview deployment: verify normal requests still succeed under the 60s budget
  • Preview deployment with simulated server hang: verify the call fails fast as WorkflowWorldError instead of running to maxDuration

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts
…imeout
Made-with: Cursor
# Conflicts:
#	packages/world-vercel/src/utils.test.ts
#	packages/world-vercel/src/utils.ts
- Compose per-request timeout with caller-provided AbortSignal via
AbortSignal.any() so a future caller passing options.signal doesn't
silently lose the hang protection (and vice versa).
- Move the floating eslint-disable-next-line for the undici dispatcher
cast back next to the actual `fetch(... as any)` call where the
suppression applies, instead of pointing at `const fetchStart`.
Both nits flagged by @VaguelySerious in the AI review on PR #1807.
Made-with: Cursor
Comment thread.changeset/fix-world-vercel-request-timeout.md Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
karthikscale3and others added 2 commits May 1, 2026 14:07
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — withdrawing my prior CHANGES_REQUESTED. The author took the suggestion from my earlier review: the process.exit workaround in runtime.ts is gone, replaced with a transport-level timeout in world-vercel/utils.ts:355 (AbortSignal.timeout(60_000) plumbed into the fetch call). This is the right layer — fixes the underlying issue at the single point where it lives, covers all 27+ world.events.create() call sites uniformly, produces a typed WorkflowWorldError that existing catch sites recognize.

Implementation looks solid:

  • Timeout primitive: AbortSignal.timeout(60_000) → DOMException with name: 'TimeoutError' → caught and wrapped as WorkflowWorldError with cause preserved. Standard pattern, clean.
  • Composition with options.signal: AbortSignal.any([options.signal, timeoutSignal]) for when callers eventually pass their own signals. Currently dead code per the comment, but wired correctly for future use.
  • Error mapping: error message includes ${method} ${endpoint} and ${elapsed}ms — useful for debugging, attaches url via the WorkflowWorldError constructor.
  • Span attributes: ErrorType('TIMEOUT') for OTEL, plus recordException. Consistent with sibling status-code branches.
  • Tests: utils.test.ts covers both the wrap-on-timeout path and the pass-through-on-non-timeout path. Mocks fetch with synthetic errors. All 94 world-vercel tests still pass.
  • Changeset scope: @workflow/world-vercel only, which is correct — no behavior change in core.

A few non-blocking concerns worth raising. None are gating; mostly forward-looking.

1. start() retry classification doesn't match timeouts as retryable

isRetryableStartError in start.ts:331 only matches WorkflowWorldError with status >= 500. The new timeout error has no status set, so it falls into the throw err branch at line 283.

Concrete consequence: when events.create(run_created) times out but the parallel queue dispatch already succeeded, the user sees start() throw "POST /runs/... timed out after 60000ms" while the workflow actually does run via the queue path. That's misleading — the right behavior is to mark this as "resilient start" and continue (runtime.ts will retry the run_created event later).

Suggested adjustment:

functionisRetryableStartError(err: unknown): boolean{if(ThrottleError.is(err))returntrue;if(WorkflowWorldError.is(err)){// 5xx server errors and timeouts (no status) are both transientif(err.status===undefined)returntrue;if(err.status>=500)returntrue;}returnfalse;}

This is technically a behavior change in core, so it'd need a separate @workflow/core patch in the changeset. Could be a follow-up if you want to keep this PR scoped to world-vercel.

2. Node 18 + AbortSignal.any

The repo's root engines.node is ^18.0.0 || ^20.0.0 || ^22.0.0 || ^24.0.0. AbortSignal.any() was added in Node 20.3 (May 2023) — not available on Node 18. The branch at utils.ts:359 only fires when options.signal is set, which the comment notes is currently never. So today this is a latent issue, not an active one. But anyone wiring up a caller-provided signal in the future will get a TypeError at runtime on Node 18.

Either drop Node 18 from engines (probably the right move overall — Node 18 is EOL April 2025 as of writing) or guard with a feature check. Could go in a separate PR.

3. Hardcoded 60s timeout

The chosen value barely covers the slowest legitimate case in your incident report (47s). A successful but slow-starting request at, say, 55s would now succeed but be on the edge; a hung request takes 60s to detect.

That's reasonable as a default, but I'd consider exposing it as a config knob (like the existing VERCEL_WORKFLOW_SERVER_URL env var pattern) for users with different latency profiles. Probably fine to wait for someone to ask.

Wrap-up

Good fix. The transport-level approach is correct and the concerns above are forward-looking polish, not gating.

@TooTallNate

Copy link
Copy Markdown
Member

Aside on my own concerns from the approval — I should retract one thing. I was briefly worried the 60s timeout might affect the long-lived stream GET endpoint, which can legitimately stay open for the full function duration. Confirmed it doesn't: looking at streamer.ts, the long-lived read at line 283 (getReadable/get) goes through a direct fetch() call, notmakeRequest. Same for the write/writeMulti/close paths — direct fetch.

Only the discrete request/response calls go through makeRequest:

  • events.create (and similar) — milliseconds normally
  • runs.get / runs.list — milliseconds normally
  • getChunks (paginated stream chunks) — bounded per-page
  • getInfo (stream metadata) — trivial

60s is appropriately generous for all of those. So the design is correct as-is — no concern about the streaming endpoint.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@karthikscale3@VaguelySerious@TooTallNate
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

fix(world-vercel): add default request timeout to workflow-server HTTP calls - #1807

Merged
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout
May 4, 2026
Merged

fix(world-vercel): add default request timeout to workflow-server HTTP calls#1807
karthikscale3 merged 99 commits into
mainfrom
karthik/fix-runtime-timeout

Conversation

@karthikscale3

@karthikscale3karthikscale3 commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a default per-request timeout to makeRequest() in world-vercel so hung responses from workflow-server can't burn compute up to the function's maxDuration.

Background (original problem)

A production workflow (wrun_01KPDFGK4QBFZ7XERXN9NP7VY2) on a preview deployment showed:

  • run_started POST to workflow-server took 47s (server under load)
  • Replay timeout fired at 240s
  • The run_failed write was sent but the response never came back (External APIs showed ∅ "Timed out while waiting for a response")
  • The function continued running for 15 minutes (hit maxDuration) before SIGTERM — ~11 minutes of compute burned doing nothing

Original fix (reverted)

The first version of this PR added a 30s hard-exit deadline to the replay timeout handler in packages/core/src/runtime.ts. Per Nate's review, this was the wrong layer: it only protected one of 27 world.events.create() call sites in core, leaving the other 26 (and every other world.* method going through makeRequest()) exposed to the exact same hang.

New fix

Moved the mitigation down into the world-vercel transport layer, where all world.* calls funnel through makeRequest():

  • packages/world-vercel/src/utils.ts — attaches AbortSignal.timeout(60_000) to every makeRequest() fetch. A TimeoutError or AbortError from fetch is converted into a WorkflowWorldError (with the original error preserved as cause and elapsed ms in the message), so existing catch sites handle it uniformly. The span is tagged with ErrorType('TIMEOUT').
  • Reverted the runtime-level exit deadline and the REPLAY_TIMEOUT_EXIT_DEADLINE_MS constant.

Why 60s: comfortably above the observed 47s p99 in the incident, well under the 240s replay timeout so upstream retries still have room, and much shorter than the maxDuration SIGTERM horizon.

Impact

  • Covers allworld.* calls through world-vercel (events, runs, steps, queue, hooks, etc.), not just the replay timeout path.
  • Hangs now surface as typed WorkflowWorldErrors — existing catch sites get predictable retry/failure semantics instead of an infinite await.
  • Happy path is unchanged: AbortSignal.timeout() doesn't fire on normal requests and the unref'd timer doesn't hold the event loop open.

Test plan

  • pnpm typecheck on @workflow/core + @workflow/world-vercel
  • Added packages/world-vercel/src/utils.test.ts with two cases:
    • TimeoutError from fetch is wrapped into WorkflowWorldError with elapsed ms and preserved cause
    • Non-timeout errors (e.g. TypeError) propagate unchanged
  • Full world-vercel suite passes (81 tests)
  • Full core suite passes (591 tests)
  • Preview deployment: verify normal requests still succeed under the 60s budget
  • Preview deployment with simulated server hang: verify the call fails fast as WorkflowWorldError instead of running to maxDuration

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
Comment threadpackages/world-vercel/src/utils.ts
…imeout
Made-with: Cursor
# Conflicts:
#	packages/world-vercel/src/utils.test.ts
#	packages/world-vercel/src/utils.ts
- Compose per-request timeout with caller-provided AbortSignal via
AbortSignal.any() so a future caller passing options.signal doesn't
silently lose the hang protection (and vice versa).
- Move the floating eslint-disable-next-line for the undici dispatcher
cast back next to the actual `fetch(... as any)` call where the
suppression applies, instead of pointing at `const fetchStart`.
Both nits flagged by @VaguelySerious in the AI review on PR #1807.
Made-with: Cursor
Comment thread.changeset/fix-world-vercel-request-timeout.md Outdated
Comment threadpackages/world-vercel/src/utils.ts Outdated
karthikscale3and others added 2 commits May 1, 2026 14:07
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — withdrawing my prior CHANGES_REQUESTED. The author took the suggestion from my earlier review: the process.exit workaround in runtime.ts is gone, replaced with a transport-level timeout in world-vercel/utils.ts:355 (AbortSignal.timeout(60_000) plumbed into the fetch call). This is the right layer — fixes the underlying issue at the single point where it lives, covers all 27+ world.events.create() call sites uniformly, produces a typed WorkflowWorldError that existing catch sites recognize.

Implementation looks solid:

  • Timeout primitive: AbortSignal.timeout(60_000) → DOMException with name: 'TimeoutError' → caught and wrapped as WorkflowWorldError with cause preserved. Standard pattern, clean.
  • Composition with options.signal: AbortSignal.any([options.signal, timeoutSignal]) for when callers eventually pass their own signals. Currently dead code per the comment, but wired correctly for future use.
  • Error mapping: error message includes ${method} ${endpoint} and ${elapsed}ms — useful for debugging, attaches url via the WorkflowWorldError constructor.
  • Span attributes: ErrorType('TIMEOUT') for OTEL, plus recordException. Consistent with sibling status-code branches.
  • Tests: utils.test.ts covers both the wrap-on-timeout path and the pass-through-on-non-timeout path. Mocks fetch with synthetic errors. All 94 world-vercel tests still pass.
  • Changeset scope: @workflow/world-vercel only, which is correct — no behavior change in core.

A few non-blocking concerns worth raising. None are gating; mostly forward-looking.

1. start() retry classification doesn't match timeouts as retryable

isRetryableStartError in start.ts:331 only matches WorkflowWorldError with status >= 500. The new timeout error has no status set, so it falls into the throw err branch at line 283.

Concrete consequence: when events.create(run_created) times out but the parallel queue dispatch already succeeded, the user sees start() throw "POST /runs/... timed out after 60000ms" while the workflow actually does run via the queue path. That's misleading — the right behavior is to mark this as "resilient start" and continue (runtime.ts will retry the run_created event later).

Suggested adjustment:

functionisRetryableStartError(err: unknown): boolean{if(ThrottleError.is(err))returntrue;if(WorkflowWorldError.is(err)){// 5xx server errors and timeouts (no status) are both transientif(err.status===undefined)returntrue;if(err.status>=500)returntrue;}returnfalse;}

This is technically a behavior change in core, so it'd need a separate @workflow/core patch in the changeset. Could be a follow-up if you want to keep this PR scoped to world-vercel.

2. Node 18 + AbortSignal.any

The repo's root engines.node is ^18.0.0 || ^20.0.0 || ^22.0.0 || ^24.0.0. AbortSignal.any() was added in Node 20.3 (May 2023) — not available on Node 18. The branch at utils.ts:359 only fires when options.signal is set, which the comment notes is currently never. So today this is a latent issue, not an active one. But anyone wiring up a caller-provided signal in the future will get a TypeError at runtime on Node 18.

Either drop Node 18 from engines (probably the right move overall — Node 18 is EOL April 2025 as of writing) or guard with a feature check. Could go in a separate PR.

3. Hardcoded 60s timeout

The chosen value barely covers the slowest legitimate case in your incident report (47s). A successful but slow-starting request at, say, 55s would now succeed but be on the edge; a hung request takes 60s to detect.

That's reasonable as a default, but I'd consider exposing it as a config knob (like the existing VERCEL_WORKFLOW_SERVER_URL env var pattern) for users with different latency profiles. Probably fine to wait for someone to ask.

Wrap-up

Good fix. The transport-level approach is correct and the concerns above are forward-looking polish, not gating.

@TooTallNate

Copy link
Copy Markdown
Member

Aside on my own concerns from the approval — I should retract one thing. I was briefly worried the 60s timeout might affect the long-lived stream GET endpoint, which can legitimately stay open for the full function duration. Confirmed it doesn't: looking at streamer.ts, the long-lived read at line 283 (getReadable/get) goes through a direct fetch() call, notmakeRequest. Same for the write/writeMulti/close paths — direct fetch.

Only the discrete request/response calls go through makeRequest:

  • events.create (and similar) — milliseconds normally
  • runs.get / runs.list — milliseconds normally
  • getChunks (paginated stream chunks) — bounded per-page
  • getInfo (stream metadata) — trivial

60s is appropriately generous for all of those. So the design is correct as-is — no concern about the streaming endpoint.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@karthikscale3@VaguelySerious@TooTallNate