Uh oh!
There was an error while loading. Please reload this page.
perf(core): prepare replay payloads concurrently - #2980
Conversation
🦋 Changeset detectedLatest commit: 9259e59 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
📜 Previous results (3)cbb98ceFri, 17 Jul 2026 08:52:24 GMT · run logs
358ed35Fri, 17 Jul 2026 06:11:42 GMT · run logs
4605884Fri, 17 Jul 2026 04:32:33 GMT · run logs
Avg deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader) Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges 🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request. |
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
Uh oh!
There was an error while loading. Please reload this page.
Benchmark against workflow-server PR #632 is complete: Actions run 29554462485, generated comparison. The run used SDK feature commit End-to-end STSO versus main
Matched |
| Event-count range | Prior avg | This PR avg | Change |
|---|---|---|---|
| 0-70 | 101.6 ms | 100.8 ms | -0.8% |
| 300-365 | 141.2 ms | 102.4 ms | -27.5% |
| 3000-3065 | 570.2 ms | 388.6 ms | -31.9% |
Datadog: new full trace, prior full trace, new representative event 3059 span (406.5 ms), prior event 3059 span (559.4 ms).
All 1,020 prior workflow.run spans carried workflow.serialization.operation=deserialize; the new trace has zero such workflow.run spans because decrypt/decompress now starts in the host-side prewarm phase and prepared bytes are reused. Fresh VM-specific devalue revival still occurs in ordered delivery slots, which is why replay time still grows with depth.
The flat early window and increasing mid/late savings are the expected shape: fixed VM and durable transport costs remain, while repeated payload preparation grows with replay depth and is now removed. The benchmark-only URL override remains as the second commit and must be removed before merge; it intentionally trips No Test Overrides and three world-vercel HTTP mock tests that expect the default origin.
CI follow-up: the one initial Astro readableStreamWorkflow failure passed on a targeted rerun (job 87806063509, 126/126 tests). The remaining red checks are only the intentional URL-override checks described above.
VaguelySerious
left a comment
There was a problem hiding this comment.
Traced the full change (cache, serialization split, and all four consumer sites). Approving — no correctness regressions found; the only real cost is memory, which the PR describes honestly.
Correctness / regression assessment — low risk
- Buffer aliasing across VMs is safe. The same prepared
Uint8Arrayis now fed todeserializeon every replay. The read path —decodeFormatPrefix(subarrayviews only) →TextDecoder.decode→ devalueparse— never mutates the buffer, so sharing one decrypted copy across fresh VMs cannot corrupt a later replay. - The one mutating path (legacy
unflatten) is correctly excluded. Bothprewarm'sstart()andconsumePreparationbail on!(value instanceof Uint8Array), so legacy flattened arrays are never cached — matching prior per-consumer behavior. That branch is effectively dead in prod (SDK version pinned per run). - Determinism preserved. All
prepare*/getStepResultawaits happen inside the samectx.promiseQueueslot (step.ts,hook.ts,abort-controller.ts), so ordered delivery,pendingDeliveries, and barriers are unchanged. Fresh object graphs are still produced per VM; only immutable primitives are returned by reference (same 4 KiB bound as before). - Failure semantics preserved. Prewarm retains rejections (
Promise.allSettled); the ordered consumer evicts a rejected entry only after observing it (consumePreparation's.catch), so a later replay re-attempts — same as the old "errors never cached" rule. No unhandled-rejection window, sinceallSettledattaches handlers eagerly. maybeDecrypt→decryptswap is a no-op (maybeDecryptis literallyreturn decrypt(data, key)).- No dangling references. Only one
runWorkflowcaller (updated); no non-test imports of the removedstep-hydration-cacheremain.WorkflowOrchestratorContext.replayPayloadCacheis now required and TS-enforced; all internal construction + test harnesses updated.
Minor, non-blocking
- Telemetry frequency change.
recordCompression(_, 'deserialize')used to fire on every hydrate (O(N²) across replays); it now fires once per unique payload at prepare time, and fromprewarmit runs outside the workflow'spromiseQueue, sogetActiveSpan()may resolve to a different/absent span. Deserialize-compression metric volume and span attribution will shift. Observability-only. prewarmonly coversstep_completed/step_failed/hook_received. If abort-controller payloads arrive as a distinct event type, they fall to an on-demand cache miss — still correct, just a missed concurrency win.
Estimated memory overhead — the real tradeoff
Genuine, intentional increase (matches the PR's "Tradeoff" section):
- Old: retained only hydrated primitive results ≤4 KiB. Decompressed object plaintext was transient (decompressed one-at-a-time per hydrate, then GC'd). Steady-state extra ≈ 0.
- New:
preparedPayloadsholds the decrypted + decompressed serialized bytes of every workflow input +step_completed.result+step_failed.error+hook_received.payloadsimultaneously, for the whole invocation, on top of the compressed/encrypted copies already incachedEvents.
Peak overhead:
ΔRSS ≈ Σ (decompressed serialized size) over {input, all step results, all step errors, all hook payloads} in the replayed log
≈ (payload bytes already in cachedEvents) × avg_compression_ratio
Scales linearly with step count × payload size, and a single re-invocation of a long-running workflow replays the entire history — so early step results the workflow no longer references still stay resident.
Rough magnitudes:
- Typical small payloads: negligible (KBs).
- 1,000 steps × ~5 KB decompressed → ~5 MB.
- 10,000 steps × ~50 KB → ~500 MB — could threaten OOM in a memory-constrained function (128–1024 MB), and raises peak transient RSS during replay vs. the old one-at-a-time decompress.
Lifetime is sound: constructed once per invocation outside the replay loop, never crosses runs or queue deliveries, GC'd on return — bounded and self-freeing, just larger.
Bottom line: aliasing, determinism, and failure-retry invariants all hold. The cost is memory — peak RSS rises by roughly the full decompressed plaintext of all replayed payloads (a few MB typical; hundreds of MB for high-step-count or large-payload workflows). Worth sanity-checking real-world worst-case step counts before/after rollout, but the tradeoff is reasonable for the reported deep-replay latency win. 🤖 Reviewed with Claude Opus 4.8 (1M context)
Backport to To resolve manually, push a backport branch and open a PR against git fetch origin stable
git checkout -b backport/pr-2980-to-stable origin/stable
git cherry-pick -S 268fede627b3a83dbabcff9d35fd946132bf9a06 # -S signs the commit# Fix conflicts, then:
git add -A
git cherry-pick --continue
git push -u origin backport/pr-2980-to-stable
gh pr create --base stable --head backport/pr-2980-to-stable \
--title "Backport #2980: <original PR title>" \
--body "Manual backport of #2980 (cherry-pick 268fede627b3) to \`stable\`." |
Summary
ReplayPayloadCache, with one instance shared across inline replay VMsReplayPayloadPreparerboundary so a future Node synchronous decrypt implementation can replace Web Crypto without changing the cache or consumersmain, the cache coverage grows by about 30 lines rather than roughly 300Benchmark
Benchmarked this SDK change against the immutable preview for vercel/workflow-server#632 at server commit
063557ca5565d9dc3b182f602372c0222a7c9379. The temporaryWORKFLOW_SERVER_URL_OVERRIDEpin has been removed.End-to-end main -> combined SDK + server:
Marginal SDK gain with both branches on the same server preview: approximately 0%, 16.2%, and 22.3% for those windows. Matched
workflow.runspans improved from 101.6/141.2/570.2 ms to 100.8/102.4/388.6 ms.Detailed trace analysis: #2980 (comment)
Tradeoff
Prepared plaintext remains cached for the invocation lifetime so replay preparation stays once per payload. Peak cache memory is therefore the sum of decrypted and decompressed payload sizes for that invocation; the cache never crosses runs or queue deliveries.
Validation
pnpm testinpackages/core(69 files, 1,477 passing, 3 expected failures)pnpm buildinpackages/coregit diff --checkgpt-5.6-solxhigh and Claude Opus 4.8 xhigh: clean, zero findings