Uh oh!
There was an error while loading. Please reload this page.
Lazy hook resumption: parallel event write + queue publish - #3230
Conversation
🦋 Changeset detectedLatest commit: 1d147c7 The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 437759ms → this run 445330ms (Δ +7571ms, +2%) 1020 steps (queue-hop) Cumulative STSO time: main 8262ms → this run 8007ms (Δ -255ms, -3%) 📜 Previous results (6)2f9a62fSat, 01 Aug 2026 01:21:07 GMT · run logs
59abb1cFri, 31 Jul 2026 23:54:52 GMT · run logs
77d6ba2Fri, 31 Jul 2026 23:11:54 GMT · run logs
68ef5aeFri, 31 Jul 2026 22:36:54 GMT · run logs
0251b95Fri, 31 Jul 2026 16:10:32 GMT · run logs
1f16ba9Thu, 30 Jul 2026 21:52:00 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests💻 Local Development (1 failed)astro-stable (1 failed):
E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
❌ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
30b9711 to
1f16ba9Compare…lish (rebased onto #1834 + #3145) Rebase of #3230 onto current main (2677653 + #1834 resilient resumeHook + #3145 event-count-gated replay restart). Reconstructed as a single commit since `git rebase -i` is unavailable in this environment. Reconciliation vs the pre-rebase branch: - Replaces #1834's version-prediction (`supportsQueueHookInput`, `QUEUE_HOOK_INPUT_MIN_VERSION`) with #3230's capability protocol (persisted `hookResumeInputVersion` + static `hookResumeDedup`). - One idempotency protocol: a single `resumeId` + SHA-256 payload digest per resume, sent to both the direct event write and the queue `hookInput`. - Two execution tiers: backend+consumer attest dedup -> parallel `Promise.allSettled(event write, queue publish)`; otherwise plain sequential (no hookInput/resumeId, event-write errors propagate). - Consumer re-ensures the `hook_received` event (keyed by resumeId/digest) after event loading, before replay; skips when already preloaded. - Preserves #3145: event-count guard, `preconditionReinvocations`, in-process replay restart, `insertEventByEventId`. - Removes #1834's resumeId-only test (never released); adds parallel + consumer-preload + world-local dedup/producer-consumer suites. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nreleased #1834 docs/changeset - hook.ts: dedup hook_received replay on top-level event.resumeId (the backend now hoists it to a first-class column), with the legacy nested eventData.resumeId retained as a deprecated parse-only fallback. - workflow.test.ts: cover dedup across both top-level and legacy nested forms. - resume-hook.ts: emit producer recovery telemetry when a transient event-write failure is swallowed on the parallel path. - resume-hook.consumer-preload.test.ts: add terminal-run (consume) and transient-conflict (rethrow/redeliver) re-ensure cases. - Consolidate the two overlapping changesets into resilient-resume-hook.md and delete the redundant lazy-hook-resumption.md. - Docs: return type back to Promise<Hook> (resume-hook.mdx), rewrite the resilience changelog to the final parallel/deduplicated design, and correct the WORKFLOW_DISABLE_LAZY_HOOK_RESUME resilience wording. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nsistency - changelog/index.mdx: update the changelog entry title. - hook.ts: update the dedup comment label to "Lazy-resume dedup". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The lazy-hook-resume work had been folded into #1834's pre-existing `resilient-resume-hook.md` changeset. Give this PR its own changeset and delete the superseded #1834 one, whose `resilientResume: true` flag promise no longer holds (resumeHook() returns plain Promise<Hook>). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restore #1834's own artifacts that #3230 had rewritten: - .changeset/resilient-resume-hook.md (restored verbatim) - docs/.../changelog/resilient-resume.mdx (restored verbatim) - docs/.../changelog/index.mdx (restored verbatim) #3230 keeps only its own changeset plus the two docs its code/config genuinely require: the resumeHook() Promise<Hook> return type (ResumedHook is removed from the code) and the new WORKFLOW_DISABLE_LAZY_HOOK_RESUME env var. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Preserve the resilient-resume contract from #1834 on the parallelized resumeHook() fast path instead of dropping it: - Restore the `ResumedHook` type (Hook + optional `resilientResume`) and its exports (`@workflow/core/runtime`, `workflow/api`); resumeHook/resumeHookImpl return `Promise<ResumedHook>`. - Set `resilientResume: true` on the swallow-recover branch (transient direct write failure + successful queue dispatch), absent on the happy/sequential paths. - Restore the producer OTEL convention `workflow.hook.resilient_resume` and the consumer `workflow.hook.resilient_resume_materialized`, wired where the consumer re-ensures the event. - Restore the consumer `occurredAt` derivation from the resume ULID so the materialized hook_received is dated to resume time, not queue-round-trip time. - Fix the #3230 changeset's contradictory "Still returns Promise<Hook>" line and update the resilient-resume changelog + resume-hook API reference to the shipped parallel/dedup behavior. - Port the #1834 failure-path coverage into resume-hook.parallel.test.ts (non-retryable event-write rethrow, both-fail prioritizes the queue error, resilientResume flag + payload delivery on the recovered path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TooTallNate
left a comment
There was a problem hiding this comment.
Reviewed the rebased head (59abb1cf2) end to end as the author of #1834/#674 — this is a genuinely better design than what it supersedes, and every contract we pinned in those review rounds is honored. Local verification: full build + typecheck clean, all 45 monorepo test tasks pass (including the new parallel/consumer-preload/world-local dedup suites), and the hook e2e tests pass against the nextjs-turbopack workbench dev server, where world-local's static capability + the same-deployment marker put the parallel path live in dev. (One unrelated env note: @workflow/utils get-port tests fail if something already holds :3000 — a leftover dev server on my machine, not this PR.)
Contracts from the predecessor rounds, verified honored:
- EntityConflictError rebase contract (pinned in #1834): the consumer re-ensure converges a matching claim as success server-side — canonical event returned, spliced into the preload via order-preserving
insertEventByEventId— and rethrows a real conflict so the queue redelivers instead of acking a delivery that may carry the only payload copy. Terminal rejections checked before the retryable classification, on both writers. Exactly the pinned behavior. - Replay dedup retained as defense-in-depth, now keyed off the hoisted top-level
event.resumeIdwith the nested form as fallback, and the workflow.test.ts dedup test upgraded to cover both forms (one inline note on the fallback's reachability below). occurredAtfrom the resumeId ULID retained on the re-ensure;ResumedHook/resilientResumepublic contract preserved (note: the PR description still says the flag is removed — it was reconciled back in d833e57; worth updating the description before merge).- The version-cutoff gate is gone, and good riddance: deleting
QUEUE_HOOK_INPUT_MIN_VERSIONretires the entire TODO(release) re-verification hazard. Two-sided explicit attestation (per-run consumer marker + response-only backend capability, freshness-guarded so a caller can't forge it through the public API) is structurally safer — even an early-published SDK keeps the parallel path off until the live server attests. The cross-deploymentstart()stamping the target's marker from the health-check probe also closes the executionContext-records-the-caller caveat I noted back in #1834. - The 409-mapping heads-up from the server-side review is addressed: the parallel path treats 409 as expected concurrency with a converging consumer, while the sequential path keeps the historical re-key to
HookNotFoundError— with the distinction documented atisHookGoneError. - The oversized-payload sequential fallback resolves my #1834 note about payload size on the queue message, and
WORKFLOW_DISABLE_LAZY_HOOK_RESUMEis documented in runtime-tuning per repo policy. The changelog rewrite is honest about the fallback trading resilience for safety.
Three non-blocking inline notes (dead legacy fallback risk, a world-local error-classification divergence, a repo-reference nit) plus one body-level note:
E2E fault-injection coverage: #1834's e2e test (stub hook_received write → 500, assert the payload still arrives) was removed and not replaced at the e2e layer. The world-local producer/consumer test covers the mechanics in-process and the regular hook e2e now exercises the parallel happy path, but nothing end-to-end exercises the recovery path (direct write fails → consumer re-ensures) through a real server boundary anymore. The old setWorld stubbing approach would be simpler now — no version simulation needed, since dev attests via the static capability + marker. Fine as a fast-follow.
Release-gate bookkeeping now satisfied on merge: the server-side dedup (deployed) + this PR are the two halves the #1834 gate was waiting for, so the "hold Version Packages" instruction dissolves once this lands.
| // deprecated — new writes populate the top-level field. | ||
| const resumeId = | ||
| event.resumeId ?? | ||
| (event.eventData as { resumeId?: unknown }).resumeId; |
There was a problem hiding this comment.
Non-blocking: the legacy nested fallback may be dead code on the paths it claims to serve. HookReceivedEventSchema.eventData no longer declares resumeId, and zod strips undeclared keys — so any event that round-trips through EventSchema.parse (world-local reads events from disk via readJSONWithFallback(..., EventSchema, ...); anything else that validates through the schema) loses the nested key before this consumer ever sees it. The unit test passes because it constructs Event objects in memory, bypassing parsing.
The blast radius is tiny — only events written by unreleased main builds (preview deployments) ever carried the nested form — but the comment overpromises. Either re-declare resumeId: z.string().optional() in the eventData schema (marked deprecated) so the fallback actually survives parsing, or drop the fallback and say the nested form is only honored for events that never re-parse. Both are fine; the current halfway state is the only wrong option.
There was a problem hiding this comment.
Fixed in 2f9a62f — dropped the fallback rather than re-declaring it in the schema. Dedup now keys solely off the top-level event.resumeId column; the consumer no longer reads eventData.resumeId. You're right that the nested form was dead on any parsed path (it was only ever written by unreleased preview builds and EventSchema strips it), so there was nothing real to preserve.
Repointed the replay dedup test (workflow.test.ts) accordingly: event-1 now duplicates event-0's top-levelresumeId (still must collapse), and the nested-form case is gone, so it no longer relies on constructing unparsed Event objects to exercise a path that can't survive parsing. Updated the PR description to drop the "retained as a deprecated fallback" line.
| claim.payloadDigest !== params.resumePayloadDigest | ||
| ) { | ||
| throw new EntityConflictError( | ||
| `hook_received resumeId "${params.resumeId}" already recorded with a different payload` |
There was a problem hiding this comment.
Non-blocking, cross-world divergence: the digest/hookId mismatch throws EntityConflictError here, but the producer's parallel path swallowsEntityConflictError as expected convergence and reports resilientResume: true — so on world-local, idempotency-key reuse (a caller bug) would be silently classified as a successful resume delivering the old payload, whereas world-vercel surfaces the server's 422 hook-resume-id-reuse as a thrown error. Unreachable through the public API today (resumeIds are minted internally by a monotonic ULID factory), so this is about world parity, not a live bug.
A distinct error type (or reusing whatever world-vercel maps the 422 to) would make the two worlds agree that key reuse is an error, not convergence — and would keep the consumer's re-ensure from retry-looping on it via redelivery, since a mismatch can never heal. Fine as a follow-up.
There was a problem hiding this comment.
Agreed, and deferring to a follow-up as you suggested. This is world parity, not a live bug: resumeIds are minted internally by a monotonic ULID factory, so idempotency-key reuse is unreachable through the public API today. The follow-up will give world-local a distinct error type for (runId, resumeId)-with-mismatched-digest reuse (mapping to whatever world-vercel surfaces the 422 hook-resume-id-reuse as), so both worlds agree that key reuse is an error rather than convergence — and so the producer's EntityConflictError swallow doesn't classify a genuine mismatch as a resilient resume, and the consumer's re-ensure doesn't retry-loop on an unhealable mismatch. Tracking separately so this PR stays scoped to latency.
| // (response-only, recomputed every by-token read). This lets a server | ||
| // rollback or kill switch drop new resumes to the sequential path | ||
| // immediately, without a redeploy of this adapter. See | ||
| // vercel/workflow-server#685. |
There was a problem hiding this comment.
Nit: this references a workflow-server PR from the public repo. There's precedent just above (#484 on line 39), so this may be accepted practice — but when shipping #674/#1834 we deliberately kept workflow-server references out of the public repo, since the link 404s for anyone outside the org. "See the server's (runId, resumeId) dedup constraint and its by-token attestation" carries the same information without the dead link. Your call; flagging for consistency either way.
There was a problem hiding this comment.
Fixed in 2f9a62f — dropped the vercel/workflow-server#685 link. The note keeps the same information ("the server enforces the (runId, resumeId) dedup constraint … attested per-lookup via Hook.resumeCapabilities.hookResumeDedupVersion") without the dead link, matching how we kept server refs out when shipping #674/#1834. Left the pre-existing #484 reference on line 39 alone to avoid unrelated churn.
…link - Drop the legacy nested `eventData.resumeId` fallback in the hook consumer. The nested form was only ever written by unreleased preview builds and is stripped by `EventSchema` parsing (the `hook_received` eventData schema does not declare it), so the fallback was dead code. Dedup now keys solely off the top-level `event.resumeId` column. Repoint the replay dedup test to the surviving top-level path (it previously exercised the nested form only by building unparsed Event objects in memory). - Remove the internal workflow-server PR reference from world-vercel's capability note (the link 404s outside the org); the note keeps the same information without the dead link. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
No backport to This is a latency optimization and new capability protocol, not a stability fix: it adds a parallel event-write/queue-publish fast path, a new To override, re-run the Backport to stable workflow manually via |
What
resumeHook()now writes thehook_receivedevent and dispatches the workflow queue message concurrently (Promise.allSettled) instead of sequentially, cutting a round trip off resume latency. A client-mintedresumeIdplus a payload digest let the direct write and the queue consumer's replay-time re-ensure converge on exactly onehook_receivedevent via a(runId, resumeId)dedup constraint.Builds on the unreleased #1834 rather than replacing it: the
ResumedHookcontract from that PR is preserved.resumeHook()resolves to aResumedHook(aHookplus an optionalresilientResume: trueflag), set only when the direct write failed transiently and the resume was recovered via the consumer's re-ensure. #1834 owns resilience; this PR owns latency (parallel dispatch + dedup).Gating — re-evaluated per resume
The parallel fast path activates only when both the target run's consumer and the live backend independently attest dedup support — the consumer via a per-run
hookResumeInputVersionexecution-context marker, the backend via a fresh, response-only capability on the by-token hook lookup. Otherwise — or for oversized payloads, legacy (v1Compat) runs, or withWORKFLOW_DISABLE_LAZY_HOOK_RESUME=1— it falls back to the original sequential path. Because both attestations are re-checked on every resume, every rollout/rollback direction degrades safely.Notes
event.resumeId(a first-class event column). The consumer materializes the event dated to resume time via the ULID embedded inresumeId(occurredAt), not the queue round-trip time.resumeIdconflict is swallowed because the consumer re-ensures the event (and the returned hook carriesresilientResume: true). Terminal-run rejections surface asHookNotFoundError.workflow.hook.resume_strategy+workflow.hook.resume_fallback_reasonon the resume span,workflow.hook.resilient_resume(producer) andworkflow.hook.resilient_resume_materialized(consumer) for the recovery path.Promise.allSettledofrun_created+ queue invoke, idempotent onrunId).