Uh oh!
There was an error while loading. Please reload this page.
perf(core): build-time V8 code cache for workflow bundles - #2523
Draft
VaguelySerious wants to merge 1 commit into
Draft
perf(core): build-time V8 code cache for workflow bundles#2523VaguelySerious wants to merge 1 commit into
VaguelySerious wants to merge 1 commit into
Conversation
On a cold serverless instance the first replay parses the entire workflow bundle before it can reach the first step. The in-process vm.Script cache (#2471) only helps subsequent replays in the same process; the cold first one still pays a full parse, which grows ~30ms/MB and dominates time-to-first-step for large bundles. Producer (builders/next): for bundles over a size threshold, compile the bundle at build time and emit Script.createCachedData() as base64 in the generated flow route, passed via workflowEntrypoint's cachedData option. Skipped for small bundles (parse is ~1ms; the blob would just bloat the artifact) and via WORKFLOW_DISABLE_BUNDLE_CODE_CACHE. Consumer (core): getCachedWorkflowScript compiles with the cachedData so V8 skips parsing on the first compile. The blob is filename-independent so one build-time cache serves every per-workflow filename; V8 validates it and falls back to a full parse on any version/source mismatch, so a stale blob is never a correctness risk. Measured fresh-process parse: 2MB 34->4.8ms, 8MB 147->21ms (~7x). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: fd621b1 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests▲ Vercel Production (2 failed)express (1 failed):
nextjs-turbopack (1 failed):
🐘 Local Postgres (1 failed)nextjs-turbopack-canary (1 failed):
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
❌ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
❌ Some E2E test jobs failed:
Check the workflow run for details. |
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro workflow with 10 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro workflow with 25 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro workflow with 50 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro workflow with 10 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro workflow with 25 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro workflow with 50 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro Stream Benchmarks(includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro stream pipeline with 5 transform steps (1MB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro 10 parallel streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro fan-out fan-in 10 streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
❌ Some benchmark jobs failed:
Check the workflow run for details. |
VaguelySerious
marked this pull request as draft
June 18, 2026 21:51
pranaygp
left a comment
Contributor
There was a problem hiding this comment.
Reviewed the producer (@workflow/builders / @workflow/next) and consumer (@workflow/core) sides in full. The design is sound: cachedData is filename-independent, V8 validates the blob and falls back to a full parse on any mismatch, so there's no correctness risk — only a missed optimization. Threading it through workflowEntrypoint → runWorkflow → getCachedWorkflowScript is clean and backward-compatible, and the tests cover the producer/consumer contract. Notes are inline; the two I'd act on before merging:
1. Build perf — the cache is generated twice per build. See the bundleFinal comment. In both the combined and Next-deferred paths, createWorkflowsBundle generates a code cache into a temp file that's immediately deleted, then the caller regenerates it. Large bundles pay createCachedData() twice.
2. The code-cache path is likely unexercised end-to-end. The 256KB threshold means the workbench apps almost certainly fall below it, so neither the E2E suites nor the green preview deployments actually emit or consume a cache. That leaves the integration unverified at runtime:
- The astro/sveltekit route post-processing regex
workflowEntrypoint\(workflowCode(?<options>[^)]*)\)now has to capture the new, { cachedData: __workflowCodeCachedData }option and rely on the separateconst __workflowCodeCachedData = ...line surviving the rewrite. By my reading it does (the.replaceonly rewrites thePOSTline, leaving the decl above it), but it's verified only by reasoning, not by an exercised route. - Each framework bundler (Next turbopack/webpack, vite, rollup) has to re-embed the multi-MB base64 literal intact for V8 to accept it at runtime.
I'd want at least one >256KB-bundle deploy exercised per route-generation path (combined, Next-deferred, astro/sveltekit regex) before relying on this — which also yields the real Vercel cold-start number the PR description asks for.
3. Artifact-size tradeoff (already flagged in the description). Worth confirming the doubled function artifact doesn't eat the parse win on a real cold start and stays under function size limits for the largest realistic bundles. A size cap on the emitted cache (skip if base64 > N) would be a cheap safety valve.
Unrelated to this PR: the failing E2E tests (abortFromStepWorkflow, distributedAbortController, recursive fibonacciWorkflow) are known flakes that also fail on main.
| const bundleFinal = async (interimBundle: string) => { | ||
| const workflowBundleCode = interimBundle; | ||
| const { cachedDataDecl, secondArg } = | ||
| createWorkflowEntrypointArgs(workflowBundleCode); |
Contributor
There was a problem hiding this comment.
createWorkflowEntrypointArgs(workflowBundleCode) runs unconditionally here — before the if (!bundleFinalOutput) check below — but every current caller of createWorkflowsBundle passes bundleFinalOutput: false (createCombinedBundle at L1353 and the Next deferred builder at builder-deferred.ts:619). In that mode this bundleFinal writes workflowFunctionCode to a .__wf_tmp.js file that the caller deletes immediately (L1370 / builder-deferred.ts:636) and then regenerates its own route + cache from the same bundle (L1379 / builder-deferred.ts:691).
So for a >256KB bundle, V8 new Script() + createCachedData() runs twice per build and one result is thrown away (~300-400ms of avoidable build cost on an 8MB bundle). Consider gating the cache generation on bundleFinalOutput — the throwaway intermediate route never needs a code cache.
| script = cachedData | ||
| ? new Script(code, { filename, cachedData }) | ||
| : new Script(code, { filename }); | ||
| if (cachedData && script.cachedDataRejected) { |
Contributor
There was a problem hiding this comment.
Observability gap: a rejected blob degrades silently, which is correct, but a silent cachedDataRejected is also a silent perf loss in prod — e.g. a Node minor bump on the runtime invalidates every build-time cache and nobody notices the cold-start regression. runWorkflow already opens a span with attributes; consider surfacing cache hit/reject as a span attribute there (or a counter) so this is detectable in production, not only via debug logs.
| let cachedDataB64 = ''; | ||
| if ( | ||
| process.env.WORKFLOW_DISABLE_BUNDLE_CODE_CACHE !== '1' && | ||
| workflowCode.length >= MIN_CODE_CACHE_BYTES |
Contributor
There was a problem hiding this comment.
Minor: MIN_CODE_CACHE_BYTES is a byte count, but it's compared against workflowCode.length, which is UTF-16 code units, not bytes. For ASCII-ish JS bundles these are ~equal so the heuristic holds, but the name implies bytes — either rename (e.g. MIN_CODE_CACHE_CHARS) or compare Buffer.byteLength(workflowCode).
karthikscale3
commented
Jun 18, 2026
Contributor
AI Review: No blocking correctness findings from me. The implementation is narrow and plausibly correctness-safe: build emits cached data for large embedded routes, runtime decodes it once, and V8 rejection falls back to normal parsing. I would leave one non-blocking ask: add an integration-style generated-route test for a large bundle, not just the helper/consumer tests, so the BaseBuilder/Next/Astro/SvelteKit route glue stays covered. Focused checks I ran:
The first core test run needed local package builds in the temp worktree; after that it passed. Node here is Main tradeoff: What we gain:
What we pay:
My take: the mechanism is sound, but default-on at 256 KB is the product decision. I would want real Vercel cold-start numbers that include function artifact load/module init, not just isolated |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On a cold serverless instance, a workflow's first replay must parse the entire workflow bundle before it can reach the first step. The in-process
vm.Scriptcache from #2471 only helps subsequent replays in the same process — the cold first replay still pays a full parse, which grows ~linearly with bundle size (~30ms/MB) and dominates time-to-first-step for large bundles.This PR ships a build-time V8 code cache (
Script.createCachedData()) so V8 can skip parsing on that first compile:@workflow/builders,@workflow/next): for bundles over a size threshold, compile the bundle at build time and emit the cache as base64 in the generated flow route, handed toworkflowEntrypointvia acachedDataoption.@workflow/core):getCachedWorkflowScriptcompiles withcachedData; V8 validates it and falls back to a full parse on any mismatch (cachedDataRejected), so a stale blob is never a correctness risk.Measured win (fresh-process parse, no in-process cache)
new ScriptcachedDataThis is genuinely different from #2471, which evaluated
createCachedData()and saw "no benefit" — but only at ≤821KB and in-process, where V8's internal compile cache already hides re-parse. At realistic sizes, fresh-process, the win is large.cachedDatais also filename-independent (verified), so one build-time blob is valid for every per-workflow filename the bundle is compiled under.Why this is parked
The delivery mechanism — embedding the cache as base64 in the route source — is not viable. The blob is roughly bundle-sized, so it ~doubles an already-large route. Measured on the 155-workflow example workbench:
esbuild (standalone / Build Output API) tolerates 16 MB, but in CI:
FATAL ERROR: JavaScript heap out of memory(the bundler choking on the giant string literal);TurbopackInternalError: failed to receive message / unexpected end of file(worker IPC crash passing the oversized route).No size threshold fixes this: the cache scales with the bundle, so it's largest exactly where the parse win matters most and where it can least be afforded. Embedding-in-source is the wrong transport.
Comparison with #2524
#2524 (precompile workflow
vm.Scriptat module init) targets the same cold-TTFB problem on a different axis, and ships safely:workflowFilenamesarray;workflowEntrypointprecompiles theScriptat module-init so the first replay is a cache hitThey are complementary, not duplicates. #2524 removes the parse from the replay window (and pays it once, regardless of replay count); this PR would make that parse itself cheap. The right composition is: land #2524, and if cold-start boot parse still matters afterward, reduce it with a code cache — but delivered file-based (a sibling
.cacheread at runtime via fs, surviving each framework's function file-tracing), never embedded in source.The reusable, harmless part of this PR is the runtime consumer (
getCachedWorkflowScript/runWorkflow/workflowEntrypointaccept optionalcachedData, withcachedDataRejectedfallback). The producer (embed) is the part to discard or rework.Recommendation
Prefer #2524 for the immediate cold-TTFB win. This PR should either be closed (superseded by #2524 for the "earlier" win; embed delivery unviable) or repurposed into a follow-up that delivers
cachedDatavia a file-based transport, stacked on top of #2524 — only if measurement after #2524 shows boot-time parse is still worth attacking.🤖 Generated with Claude Code