perf(core): memoize step return value hydration across inline replays - #2472

Merged
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration
Jun 22, 2026
Merged

perf(core): memoize step return value hydration across inline replays#2472
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration

Conversation

@pranaygp

@pranaygppranaygp commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

The inline replay loop (runtime.tsrunWorkflow, workflow.ts) re-executes the workflow body and re-consumes the full event log on every iteration. For each already-completed step, the step consumer (step.ts, step_completed path) re-ran hydrateStepReturnValue — AES-GCM decrypt + devalue-parse of the serialized result — on every replay, even though that exact result was already hydrated on every prior replay.

For a sequential workflow of N steps, replay K hydrates K results, so the aggregate cost across a single invocation is O(N²) decrypt+parse operations.

This PR adds a per-run memoization cache so a completed step's hydrated result is returned in O(1) on subsequent replays within the same invocation, making the aggregate cost O(N).

Before / after

  • Before: replay 1 hydrates 1 result, replay 2 hydrates 2, …, replay K hydrates K → Σ = O(N²) decrypt+parse over a sequential run.
  • After: each completed step's result is hydrated once and memoized for the rest of the invocation → O(N) total. Replay K hydrates only the one newly-completed step; the K−1 prior results are cache hits.

Cache scope & keying

  • Lifetime / scope: owned by the inline loop in runtime.ts (created once per run invocation, alongside cachedEvents), threaded into runWorkflow(..., stepHydrationCache?) and stored on WorkflowOrchestratorContext.stepHydrationCache. A fresh context is created each loop iteration, so the cache deliberately lives outside the per-iteration context to survive across iterations of the same run. It is never shared across unrelated runs or process-level invocations.
  • Keying: by the persisted step_completed event's eventId — a stable, world-assigned id. The same event carries the same immutable serialized bytes across every replay, so a hit is guaranteed to correspond to identical input.
  • Optional / backward compatible: the parameter and context field are optional. Callers/harnesses that omit them (and the many runWorkflow(...) unit tests) degrade to re-hydrating every replay — identical to previous behavior.

Memory characteristic

A cached entry holds the decrypted/devalue-parsed plaintext of a step result, retained for the rest of the invocation on top of the serialized bytes already held in cachedEvents — so for large primitive results it roughly doubles peak retained memory for those results during the run. This residual is:

  • Scoped to one invocation — the Map is created per run and GC'd when the invocation returns; nothing accumulates across runs or process-level invocations (a much weaker concern than a process-wide cache, where the dominant residency — the full event log in cachedEvents — already exists for the same lifetime).
  • Bounded by the primitive-returning completed-step count — at most one small entry per such step.
  • Byte-bounded. Most primitives (numbers, booleans, null/undefined, symbols, short ids/strings) are tiny. The only primitive that can be large is a string (or a pathologically long bigint), so a string/bigint result longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) is not memoized — it falls through to the existing per-replay re-hydrate path. Large payloads are cheap to re-hydrate relative to their footprint, so this caps the worst case at negligible cost. The cap only ever reduces what is cached, so deterministic replay is unaffected.

Ordering safety analysis

The cache lookup replaces only the await hydrateStepReturnValue(...) call inside the existing ctx.promiseQueue.then(async () => { ... }) slot. Everything else is byte-for-byte unchanged:

  • ctx.pendingDeliveries++ / -- accounting is untouched.
  • The hydrate (or cache hit) still happens inside the same serial promiseQueue slot, at the same log position, and still resolves via the same resolve(...).
  • The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the exact position in the ordered delivery chain a re-hydrate would have.

So delivery order, pendingDeliveries-gated suspensions, the pendingDeliveryBarriers / awaitEarlierDeliveries machinery, and Promise.race/Promise.all replay determinism are all unaffected. Hook, wait, and abort hydration paths are intentionally not cached (they're the ordering-sensitive paths and not the O(N²) hotspot).

Identity / immutability safety

hydrateStepReturnValue (devalue.parse) returns a fresh object graph on every call, and each replay iteration runs in a fresh workflow VM. Today the workflow therefore receives a brand-new value on every replay. If we cached and returned the same object reference across replays, workflow code that mutates a step result (const r = await step(); r.count++) would observe a previous replay's mutation on the next replay — a non-deterministic divergence. (structuredClone on each hit is both lossy — revivers reconstruct stream handles, step-function proxies, Request/Response, and AbortController/AbortSignal class instances — and still O(size).)

Decision: only primitives are memoized (string, number, boolean, bigint, symbol, null, undefined). Primitives are immutable and compared by value, so sharing the reference is provably indistinguishable from re-parsing. Any non-primitive result falls through to a full re-hydrate every replay, preserving current behavior exactly. Errors are never cached, so a rejected hydrate re-attempts on the next replay (no parked rejected promise). This trades away the optimization in the object-returning case to keep deterministic replay airtight — correctness over speed.

What I verified

  • Unit:step-hydration-cache.test.ts (14 tests: primitive detection, memoization, non-primitive eviction/fresh-object, falsy primitives, keying, error non-caching, no-cache passthrough, plus the size-bound — at-bound string is a hit, oversized string/bigint are not memoized and cache.size stays 0) and step-hydration-memoization.test.ts (3 tests through the real createUseStep consumer: hydrate-skipped-on-replay-2 via spy, event-log ordering preserved on cache hits, fresh object per replay for object results).
  • Full core suite:cd packages/core && pnpm test1253 passed / 56 files, including async-deserialization-ordering.test.ts, workflow.test.ts (79 tests), runtime.test.ts, hook-sleep-interaction, abort-consistency. No regressions.
  • Build / format / typecheck:pnpm build (full repo, 27/27), @workflow/core build + tsc --noEmit clean; Biome format applied; new files Biome-clean (the only lint errors were import-ordering, auto-fixed; remaining warnings are pre-existing noExcessiveCognitiveComplexity on functions I only edited).
  • E2E (local nextjs-turbopack dev server, the determinism-sensitive subset): promiseAllWorkflow, promiseRaceWorkflow, promiseAnyWorkflow, sleepWinsRaceWorkflow, stepWinsRaceWorkflow, promiseRaceStressTestWorkflow, hookWorkflow, webhookWorkflow, parallel-steps-then-webhook replay race, sleepingWorkflow, parallelSleepWorkflow, retry/error/catchability suite, fetchWorkflowall passed.

Risks / deferred

  • Only primitive step results are accelerated; object-returning steps still re-hydrate each replay (intentional, for determinism). A future safe extension could deep-freeze + share frozen object graphs, but that needs care around reviver-produced special objects and is out of scope here.
  • Large (>4 KiB) string/bigint results are intentionally not memoized to bound peak retained memory (see Memory characteristic); they re-hydrate each replay.
  • Hook/wait/abort hydration paths are uncached by design.

🤖 Generated with Claude Code

The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CopilotAI review requested due to automatic review settings June 17, 2026 01:47
@pranaygp
pranaygp requested a review from a team as a code ownerJune 17, 2026 01:47

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@changeset-bot

changeset-botBot commented Jun 17, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3ca022f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
workflowPatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production144112301672
✅ 💻 Local Development190902192128
✅ 📦 Local Production190902192128
❌ 🐘 Local Postgres189412332128
✅ 🪟 Windows15200152
✅ 📋 Other88501791064
Total8190210809272

❌ Failed Tests

▲ Vercel Production (1 failed)

nitro (1 failed):

  • startFromWorkflow - calling start() directly inside a workflow function with hook communication | wrun_01KVCQ35C5K211Z7CFGRHG4655 | 🔍 observability
🐘 Local Postgres (1 failed)

nextjs-turbopack-stable-lazy-discovery-enabled (1 failed):

  • wellKnownAgentWorkflow (.well-known/agent) | wrun_01KVCPM9CVA5BNHKTABQFTWTWF

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro125027
✅ example125027
✅ express125027
✅ fastify125027
✅ hono125027
✅ nextjs-turbopack14903
✅ nextjs-webpack14903
❌ nitro124127
✅ nuxt125027
✅ sveltekit14408
✅ vite125027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
❌ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable126026
✅ express-stable126026
✅ fastify-stable126026
✅ hono-stable126026
✅ nextjs-turbopack-canary132020
✅ nextjs-turbopack-stable-lazy-discovery-disabled15101
❌ nextjs-turbopack-stable-lazy-discovery-enabled15011
✅ nextjs-webpack-canary132020
✅ nextjs-webpack-stable-lazy-discovery-disabled15101
✅ nextjs-webpack-stable-lazy-discovery-enabled15101
✅ nitro-stable126026
✅ nuxt-stable126026
✅ sveltekit-stable14507
✅ vite-stable126026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15200
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable127025
✅ e2e-local-dev-tanstack-start-127025
✅ e2e-local-postgres-nest-stable126026
✅ e2e-local-postgres-tanstack-start-126026
✅ e2e-local-prod-nest-stable127025
✅ e2e-local-prod-tanstack-start-127025
✅ e2e-vercel-prod-tanstack-start125027

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: failure
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.042s (-2.5%)1.006s (~)0.963s101.00x
💻 LocalNitro0.045s (+10.1% 🔺)1.006s (~)0.962s101.06x
💻 LocalNext.js (Turbopack)0.062s (-5.5% 🟢)1.007s (~)0.944s101.47x
🐘 PostgresExpress0.067s (-8.0% 🟢)1.013s (~)0.946s101.58x
🐘 PostgresNext.js (Turbopack)0.070s (-0.9%)1.013s (~)0.943s101.65x
🐘 PostgresNitro0.073s (+15.6% 🔺)1.013s (~)0.940s101.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express0.271s (-28.1% 🟢)2.235s (-2.6%)1.964s101.00x
▲ VercelNitro0.319s (-14.9% 🟢)2.203s (-2.0%)1.883s101.18x
▲ VercelNext.js (Turbopack)0.351s (+20.6% 🔺)2.531s (+33.8% 🔺)2.180s101.29x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.094s (-0.7%)2.006s (~)0.912s101.00x
💻 LocalNitro1.095s (+0.9%)2.007s (~)0.912s101.00x
🐘 PostgresNitro1.108s (-1.1%)2.009s (~)0.901s101.01x
🐘 PostgresExpress1.110s (~)2.008s (~)0.898s101.02x
💻 LocalNext.js (Turbopack)1.132s (-1.4%)2.007s (~)0.875s101.03x
🐘 PostgresNext.js (Turbopack)1.141s (~)2.009s (~)0.868s101.04x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.723s (-11.0% 🟢)3.716s (+1.1%)1.993s101.00x
▲ VercelNext.js (Turbopack)1.761s (~)3.708s (~)1.947s101.02x
▲ VercelNitro1.788s (-22.8% 🟢)3.409s (-14.6% 🟢)1.621s101.04x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro10.527s (~)11.022s (~)0.495s31.00x
💻 LocalExpress10.566s (~)11.023s (~)0.457s31.00x
🐘 PostgresExpress10.585s (~)11.023s (~)0.438s31.01x
🐘 PostgresNitro10.598s (+0.6%)11.022s (~)0.424s31.01x
💻 LocalNext.js (Turbopack)10.812s (~)11.022s (~)0.210s31.03x
🐘 PostgresNext.js (Turbopack)10.829s (~)11.017s (~)0.189s31.03x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express13.691s (-24.2% 🟢)15.347s (-23.1% 🟢)1.656s21.00x
▲ VercelNitro13.728s (-27.6% 🟢)15.441s (-27.5% 🟢)1.713s21.00x
▲ VercelNext.js (Turbopack)14.688s (+2.6%)16.926s (+3.9%)2.238s21.07x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro13.690s (~)14.027s (~)0.337s51.00x
🐘 PostgresExpress13.756s (-0.7%)14.019s (~)0.263s51.00x
💻 LocalExpress13.798s (~)14.028s (~)0.231s51.01x
🐘 PostgresNitro13.832s (~)14.023s (~)0.191s51.01x
💻 LocalNext.js (Turbopack)14.383s (~)15.030s (~)0.647s41.05x
🐘 PostgresNext.js (Turbopack)14.396s (~)15.017s (~)0.621s41.05x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express23.028s (-4.9%)25.270s (-3.5%)2.242s31.00x
▲ VercelNext.js (Turbopack)24.609s (-30.5% 🟢)26.392s (-28.3% 🟢)1.783s31.07x
▲ VercelNitro24.684s (-20.8% 🟢)26.313s (-19.2% 🟢)1.629s31.07x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro12.373s (+2.5%)13.027s (+1.1%)0.654s71.00x
💻 LocalExpress12.444s (-0.8%)13.024s (~)0.580s71.01x
🐘 PostgresExpress12.512s (+0.6%)13.016s (~)0.503s71.01x
🐘 PostgresNitro12.712s (+1.6%)13.021s (~)0.310s71.03x
💻 LocalNext.js (Turbopack)13.662s (~)14.027s (~)0.365s71.10x
🐘 PostgresNext.js (Turbopack)13.897s (~)14.308s (+1.0%)0.411s71.12x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro27.276s (-29.9% 🟢)29.146s (-28.8% 🟢)1.870s41.00x
▲ VercelNext.js (Turbopack)27.460s (-22.0% 🟢)29.404s (-20.4% 🟢)1.944s41.01x
▲ VercelExpress28.389s (-19.0% 🟢)30.612s (-17.0% 🟢)2.223s31.04x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.204s (-0.8%)2.007s (~)0.803s151.00x
💻 LocalNitro1.222s (+4.5%)2.006s (~)0.784s151.01x
💻 LocalExpress1.231s (+5.6% 🔺)2.007s (~)0.776s151.02x
🐘 PostgresNitro1.251s (+4.4%)2.009s (~)0.757s151.04x
🐘 PostgresNext.js (Turbopack)1.256s (-2.3%)2.007s (~)0.751s151.04x
💻 LocalNext.js (Turbopack)1.405s (+8.4% 🔺)2.006s (~)0.602s151.17x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.571s (+15.9% 🔺)4.329s (+11.4% 🔺)1.757s81.00x
▲ VercelNitro3.123s (+7.3% 🔺)4.469s (+1.4%)1.346s71.21x
▲ VercelNext.js (Turbopack)3.608s (+35.3% 🔺)4.968s (+21.2% 🔺)1.360s71.40x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.309s (-8.1% 🟢)2.008s (-16.1% 🟢)0.700s151.00x
🐘 PostgresNitro1.332s (-2.4%)2.074s (-17.3% 🟢)0.742s151.02x
🐘 PostgresNext.js (Turbopack)1.452s (-11.3% 🟢)2.075s (-13.3% 🟢)0.623s151.11x
💻 LocalExpress1.987s (+22.2% 🔺)2.592s (+29.2% 🔺)0.604s121.52x
💻 LocalNitro2.051s (+32.2% 🔺)2.507s (+24.7% 🔺)0.456s121.57x
💻 LocalNext.js (Turbopack)2.359s (+23.5% 🔺)3.008s (+31.2% 🔺)0.649s101.80x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.443s (-13.1% 🟢)5.587s (-1.5%)2.143s61.00x
▲ VercelNitro3.655s (-9.7% 🟢)5.233s (-8.2% 🟢)1.577s61.06x
▲ VercelNext.js (Turbopack)4.245s (-5.5% 🟢)5.891s (-9.1% 🟢)1.646s61.23x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.468s (-16.7% 🟢)3.886s (~)2.417s81.00x
🐘 PostgresNitro1.470s (-7.8% 🟢)3.885s (-3.1%)2.415s81.00x
🐘 PostgresNext.js (Turbopack)2.642s (-16.3% 🟢)3.456s (-19.7% 🟢)0.814s91.80x
💻 LocalNitro4.340s (+26.8% 🔺)5.013s (+25.0% 🔺)0.673s62.96x
💻 LocalExpress5.253s (+20.2% 🔺)5.679s (+16.7% 🔺)0.425s63.58x
💻 LocalNext.js (Turbopack)6.408s (+11.0% 🔺)7.019s (+12.9% 🔺)0.611s54.36x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro4.491s (-26.7% 🟢)6.303s (-24.4% 🟢)1.812s51.00x
▲ VercelNext.js (Turbopack)4.524s (-19.6% 🟢)6.497s (-15.8% 🟢)1.973s51.01x
▲ VercelExpress4.646s (-15.6% 🟢)7.023s (-11.4% 🟢)2.378s51.03x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.202s (-1.3%)2.007s (~)0.805s151.00x
🐘 PostgresNitro1.214s (~)2.008s (~)0.794s151.01x
💻 LocalExpress1.228s (-22.1% 🟢)2.006s (~)0.778s151.02x
💻 LocalNitro1.268s (+5.6% 🔺)2.006s (~)0.738s151.06x
🐘 PostgresNext.js (Turbopack)1.271s (-0.8%)2.008s (~)0.737s151.06x
💻 LocalNext.js (Turbopack)1.449s (+3.6%)2.006s (~)0.557s151.21x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.668s (-25.2% 🟢)4.041s (-23.6% 🟢)1.373s81.00x
▲ VercelNext.js (Turbopack)2.876s (-22.7% 🟢)4.782s (-8.6% 🟢)1.907s71.08x
▲ VercelExpress3.339s (+43.7% 🔺)5.449s (+43.2% 🔺)2.110s61.25x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Nitro1.304s (-9.6% 🟢)2.075s (-13.3% 🟢)0.771s151.00x
🐘 PostgresExpress1.310s (-4.1%)2.007s (-13.3% 🟢)0.697s151.00x
🐘 PostgresNext.js (Turbopack)1.436s (-6.9% 🟢)2.076s (-6.6% 🟢)0.640s151.10x
💻 LocalNitro1.935s (+14.3% 🔺)2.293s (+14.2% 🔺)0.358s141.48x
💻 LocalExpress2.003s (+5.5% 🔺)2.393s (+11.3% 🔺)0.390s131.54x
💻 LocalNext.js (Turbopack)2.369s (+9.3% 🔺)3.009s (~)0.639s101.82x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.899s (+6.3% 🔺)4.683s (+2.5%)1.785s71.00x
▲ VercelNitro3.978s (+42.1% 🔺)5.734s (+37.1% 🔺)1.756s61.37x
▲ VercelNext.js (Turbopack)4.216s (-5.6% 🟢)5.944s (-3.4%)1.728s61.45x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.452s (-19.3% 🟢)4.014s (~)2.561s81.00x
🐘 PostgresNitro1.589s (-12.0% 🟢)3.678s (-14.4% 🟢)2.090s91.09x
🐘 PostgresNext.js (Turbopack)2.136s (-47.5% 🟢)3.454s (-22.3% 🟢)1.318s91.47x
💻 LocalExpress5.580s (+21.6% 🔺)6.014s (+20.0% 🔺)0.434s53.84x
💻 LocalNitro5.616s (+25.0% 🔺)6.017s (+20.1% 🔺)0.401s53.87x
💻 LocalNext.js (Turbopack)6.852s (+18.0% 🔺)7.416s (+15.6% 🔺)0.563s54.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.610s (-57.2% 🟢)5.562s (-45.7% 🟢)1.952s61.00x
▲ VercelNitro4.046s (-70.8% 🟢)5.724s (-63.4% 🟢)1.678s61.12x
▲ VercelNext.js (Turbopack)4.124s (+9.0% 🔺)6.272s (+19.9% 🔺)2.147s51.14x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.605s (+6.0% 🔺)1.041s (+3.5%)0.436s581.00x
💻 LocalExpress0.616s (-1.5%)1.005s (-1.7%)0.389s601.02x
💻 LocalNitro0.629s (+25.8% 🔺)1.039s (+3.1%)0.410s581.04x
🐘 PostgresNitro0.692s (+16.5% 🔺)1.078s (+3.6%)0.386s561.14x
🐘 PostgresNext.js (Turbopack)0.858s (+2.7%)1.041s (+1.7%)0.183s581.42x
💻 LocalNext.js (Turbopack)0.860s (-3.1%)1.005s (-3.3%)0.144s601.42x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.774s (-42.9% 🟢)5.457s (-34.5% 🟢)1.683s111.00x
▲ VercelNext.js (Turbopack)3.815s (-39.5% 🟢)5.390s (-33.8% 🟢)1.575s121.01x
▲ VercelExpress4.488s (-1.3%)6.377s (-1.6%)1.889s101.19x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.375s (+1.4%)2.052s (+1.2%)0.678s441.00x
🐘 PostgresNitro1.466s (+5.6% 🔺)2.030s (~)0.564s451.07x
💻 LocalNitro1.485s (+25.1% 🔺)2.006s (~)0.521s451.08x
💻 LocalExpress1.530s (+2.8%)2.007s (~)0.476s451.11x
🐘 PostgresNext.js (Turbopack)1.989s (+2.5%)2.308s (+11.2% 🔺)0.319s401.45x
💻 LocalNext.js (Turbopack)2.090s (-0.6%)2.944s (-2.1%)0.854s311.52x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express10.297s (-41.1% 🟢)12.369s (-35.5% 🟢)2.072s81.00x
▲ VercelNitro10.774s (-8.8% 🟢)12.396s (-12.5% 🟢)1.621s81.05x
▲ VercelNext.js (Turbopack)11.321s (-16.7% 🟢)13.637s (-11.0% 🟢)2.316s71.10x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express2.654s (-2.1%)3.058s (-1.7%)0.404s401.00x
🐘 PostgresNitro2.903s (+3.7%)3.280s (+3.6%)0.376s371.09x
💻 LocalExpress3.275s (+2.4%)4.010s (~)0.735s301.23x
💻 LocalNitro3.362s (+23.1% 🔺)4.010s (+24.4% 🔺)0.648s301.27x
🐘 PostgresNext.js (Turbopack)3.983s (+2.9%)4.253s (+4.3%)0.270s291.50x
💻 LocalNext.js (Turbopack)4.363s (~)5.010s (~)0.647s241.64x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express18.286s (-29.9% 🟢)20.532s (-26.7% 🟢)2.246s61.00x
▲ VercelNitro18.713s (-28.9% 🟢)20.382s (-27.5% 🟢)1.669s61.02x
▲ VercelNext.js (Turbopack)20.206s (-19.2% 🟢)22.420s (-16.3% 🟢)2.214s61.11x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.232s (~)1.006s (~)0.773s601.00x
🐘 PostgresNitro0.237s (+0.7%)1.006s (~)0.769s601.02x
🐘 PostgresNext.js (Turbopack)0.297s (+1.1%)1.023s (+1.7%)0.726s591.28x
💻 LocalExpress0.400s (-9.2% 🟢)1.005s (~)0.605s601.72x
💻 LocalNitro0.417s (+11.8% 🔺)1.004s (~)0.588s601.79x
💻 LocalNext.js (Turbopack)0.633s (+8.8% 🔺)1.004s (~)0.371s602.73x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.787s (+15.3% 🔺)3.598s (+25.4% 🔺)1.811s181.00x
▲ VercelNitro1.933s (+49.7% 🔺)3.615s (+25.8% 🔺)1.682s171.08x
▲ VercelNext.js (Turbopack)2.387s (+27.6% 🔺)4.344s (+11.8% 🔺)1.956s141.34x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.331s (-6.9% 🟢)1.006s (-3.3%)0.675s901.00x
🐘 PostgresNitro0.356s (+4.1%)1.006s (-1.1%)0.650s901.08x
🐘 PostgresNext.js (Turbopack)0.476s (-11.7% 🟢)1.103s (-1.3%)0.628s831.44x
💻 LocalNitro2.182s (+43.2% 🔺)2.737s (+27.2% 🔺)0.555s336.59x
💻 LocalExpress2.222s (+6.3% 🔺)2.738s (+3.1%)0.517s336.71x
💻 LocalNext.js (Turbopack)2.493s (+7.0% 🔺)3.344s (+8.7% 🔺)0.851s277.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.638s (-9.8% 🟢)4.553s (-5.1% 🟢)1.915s201.00x
▲ VercelNitro2.705s (-1.5%)4.386s (-8.3% 🟢)1.681s211.03x
▲ VercelNext.js (Turbopack)3.092s (+1.6%)4.802s (~)1.710s191.17x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.526s (-7.4% 🟢)1.059s (-13.0% 🟢)0.532s1141.00x
🐘 PostgresNitro0.570s (~)1.078s (-12.5% 🟢)0.508s1121.08x
🐘 PostgresNext.js (Turbopack)1.918s (-26.7% 🟢)2.763s (-20.4% 🟢)0.845s443.64x
💻 LocalNitro9.521s (+50.8% 🔺)10.445s (+54.9% 🔺)0.924s1218.09x
💻 LocalExpress10.198s (+22.8% 🔺)11.029s (+24.2% 🔺)0.831s1219.38x
💻 LocalNext.js (Turbopack)10.284s (-4.4%)11.663s (+0.8%)1.379s1119.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.245s (-38.5% 🟢)5.031s (-28.3% 🟢)1.786s241.00x
▲ VercelExpress3.483s (-36.2% 🟢)5.687s (-19.8% 🟢)2.204s221.07x
▲ VercelNext.js (Turbopack)4.699s (-2.9%)6.708s (+2.7%)2.008s181.45x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.165s (-1.1%)2.000s (~)0.001s (+9.1% 🔺)2.009s (~)0.844s101.00x
🐘 PostgresNitro1.181s (+1.4%)1.995s (~)0.001s (+30.0% 🔺)2.011s (~)0.831s101.01x
💻 LocalNitro1.181s (+5.7% 🔺)2.005s (~)0.010s (-62.9% 🟢)2.017s (-1.0%)0.836s101.01x
💻 LocalExpress1.185s (+3.0%)2.005s (~)0.012s (+19.8% 🔺)2.019s (~)0.835s101.02x
💻 LocalNext.js (Turbopack)1.216s (~)2.003s (~)0.013s (+2.4%)2.020s (~)0.803s101.04x
🐘 PostgresNext.js (Turbopack)1.232s (~)2.002s (~)0.001s (-15.4% 🟢)2.011s (~)0.779s101.06x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.206s (-8.2% 🟢)3.426s (-6.9% 🟢)0.602s (-18.8% 🟢)4.513s (-7.4% 🟢)2.307s101.00x
▲ VercelNext.js (Turbopack)2.304s (-8.6% 🟢)3.523s (-7.8% 🟢)0.772s (-4.3%)4.790s (-5.8% 🟢)2.486s101.04x
▲ VercelNitro2.331s (-11.7% 🟢)3.158s (-17.0% 🟢)1.253s (+57.9% 🔺)4.805s (-4.9%)2.474s101.06x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro1.572s (+9.3% 🔺)2.011s (~)0.012s (-32.0% 🟢)2.025s (~)0.453s301.00x
🐘 PostgresExpress1.577s (~)2.004s (~)0.005s (+5.7% 🔺)2.026s (~)0.449s301.00x
💻 LocalExpress1.583s (+1.0%)2.010s (~)0.014s (+4.9%)2.025s (~)0.442s301.01x
🐘 PostgresNitro1.602s (+1.5%)2.008s (~)0.005s (-5.6% 🟢)2.027s (~)0.425s301.02x
💻 LocalNext.js (Turbopack)1.740s (-0.5%)2.010s (~)0.013s (+1.3%)2.025s (~)0.285s301.11x
🐘 PostgresNext.js (Turbopack)1.892s (+5.9% 🔺)2.011s (~)0.005s (+5.8% 🔺)2.029s (~)0.137s301.20x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Next.js (Turbopack)6.842s (-19.5% 🟢)8.374s (-14.7% 🟢)0.220s (-16.3% 🟢)9.079s (-14.1% 🟢)2.236s71.00x
▲ VercelNitro6.905s (-3.6%)7.768s (-10.3% 🟢)0.455s (+91.9% 🔺)8.766s (-6.9% 🟢)1.861s71.01x
▲ VercelExpress8.068s (+26.4% 🔺)9.373s (+20.1% 🔺)0.307s (+71.5% 🔺)10.412s (+23.1% 🔺)2.345s61.18x

🔍 Observability: Next.js (Turbopack) | Nitro | Express

10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.789s (+3.3%)1.101s (+7.5% 🔺)0.000s (-100.0% 🟢)1.118s (+6.0% 🔺)0.329s541.00x
🐘 PostgresNitro0.795s (+2.9%)1.044s (~)0.000s (-3.5%)1.061s (-3.0%)0.265s571.01x
🐘 PostgresNext.js (Turbopack)0.986s (-3.3%)1.397s (-4.6%)0.000s (-100.0% 🟢)1.407s (-4.4%)0.421s431.25x
💻 LocalExpress1.533s (+12.5% 🔺)2.014s (~)0.000s (+16.7% 🔺)2.016s (~)0.483s301.94x
💻 LocalNitro1.568s (+45.8% 🔺)2.014s (+9.6% 🔺)0.000s (-56.8% 🟢)2.016s (+9.5% 🔺)0.448s301.99x
💻 LocalNext.js (Turbopack)1.916s (+26.7% 🔺)2.193s (+8.9% 🔺)0.000s (+7.1% 🔺)2.196s (+8.9% 🔺)0.280s282.43x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.961s (-21.2% 🟢)4.002s (-25.7% 🟢)0.000s (-21.4% 🟢)4.479s (-23.6% 🟢)1.518s141.00x
▲ VercelExpress3.140s (+3.6%)4.516s (-1.1%)0.000s (NaN%)5.069s (+0.6%)1.929s121.06x
▲ VercelNext.js (Turbopack)3.540s (+2.7%)5.016s (+5.2% 🔺)0.000s (-100.0% 🟢)5.553s (+5.5% 🔺)2.013s111.20x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.691s (+4.7%)2.302s (+11.5% 🔺)0.000s (NaN%)2.316s (+10.9% 🔺)0.625s261.00x
🐘 PostgresNitro1.779s (+14.7% 🔺)2.381s (+11.6% 🔺)0.000s (+11.5% 🔺)2.394s (+11.5% 🔺)0.615s261.05x
🐘 PostgresNext.js (Turbopack)2.251s (+6.6% 🔺)2.651s (+2.5%)0.000s (-100.0% 🟢)2.664s (+2.6%)0.413s231.33x
💻 LocalNitro4.321s (+95.8% 🔺)4.717s (+72.0% 🔺)0.001s (+12.8% 🔺)4.729s (+71.7% 🔺)0.409s132.56x
💻 LocalExpress4.772s (+62.9% 🔺)5.362s (+46.1% 🔺)0.001s (+98.3% 🔺)5.367s (+46.1% 🔺)0.595s122.82x
💻 LocalNext.js (Turbopack)5.632s (+94.2% 🔺)6.226s (+85.4% 🔺)0.000s (-60.0% 🟢)6.235s (+85.4% 🔺)0.602s103.33x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express4.310s (-24.3% 🟢)6.054s (-16.2% 🟢)0.000s (+Infinity% 🔺)6.546s (-14.9% 🟢)2.236s101.00x
▲ VercelNext.js (Turbopack)4.772s (-34.2% 🟢)6.201s (-26.1% 🟢)0.000s (-100.0% 🟢)6.679s (-24.5% 🟢)1.907s101.11x
▲ VercelNitro4.835s (-7.9% 🟢)5.805s (-10.1% 🟢)0.000s (+Infinity% 🔺)6.308s (-8.9% 🟢)1.473s101.12x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalNitro12/21
🐘 PostgresExpress19/21
▲ VercelExpress14/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres17/21
Next.js (Turbopack)🐘 Postgres14/21
Nitro🐘 Postgres13/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run

@pranaygp

Copy link
Copy Markdown
ContributorAuthor

CI failure triage — pre-existing Vercel-prod e2e flake (not a regression)

The two red checks (E2E Vercel Prod Tests (example), E2E Vercel Prod Tests (fastify), which roll up into E2E Required Check) are the shared Vercel-prod timing flake, not caused by this PR. Evidence:

Scope is wrong for a hydration regression. This PR only memoizes primitive step-result hydration. A determinism/stale-value bug there would surface across all worlds — yet every local suite is 100% green:

SuitePassedFailed
▲ Vercel Production14402
💻 Local Development19090
📦 Local Production19090
🐘 Local Postgres18950
🪟 Windows1520

Only 2 failures, only on Vercel Production.

The two failing tests are unrelated to result hydration, and are abort/hook timing races:

  1. exampleAbortController > abortFromStepWorkflow: step abort cancels an in-flight sibling step. The run completed successfully; the assertion failed only because the abort lost a race. From the run diagnostics (wrun_01KV9MZ1Y00N90XFCKFTSPX5N1):

    +2.2s step_completed (longStep) <- sibling finished on its own
    +3.0s hook_received <- abort signal arrived AFTER
    

    The sibling longStep self-completed (2.2s) before the abort hook arrived (3.0s) under Vercel-prod queue/network latency, so there was no in-flight step left to cancel. Step results hydrated fine; this is purely environmental latency.

  2. fastifystartFromWorkflow - calling start() directly inside a workflow function with hook communication. This exact test passed (4128ms) in the example job of this same commit (a56f5c90b) — a textbook cross-run flake.

The same test is red on plain main, without this change. On main run 27704378960 (commit 2acf13cc7):

  • E2E Vercel Prod Tests (tanstack-start)abortFromStepWorkflow: step abort cancels an in-flight sibling step (the identical test that failed here on example)
  • E2E Vercel Prod Tests (nextjs-turbopack)distributedAbortController - reconnect to existing controller

And on main run 27657696161 (cb181392b, the commit this branch is based on): E2E Vercel Prod Tests (fastify)hookWithSleepFinalStepWorkflow. The set of red workbenches rotates run-to-run — the signature of environment flakiness, not a code regression.

Local verification of this branch (rebuilt @workflow/core first): cd packages/core && pnpm test1249 passed / 56 files, 0 failures, including the determinism/ordering replay tests in workflow.test.ts and the new step-hydration-cache.test.ts (10) + step-hydration-memoization.test.ts (3). The memoization tests assert byte-identical delivery ordering on cache hits and that objects re-hydrate fresh each replay.

Re-running the e2e jobs should clear them. No code change is warranted.

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — the O(N²)→O(N) hydration memoization, done with the right safety bias

This is the most safety-sensitive of the four (it's the only one that caches a value across replays), and the design lands on the conservative side of every judgment call, which is exactly right for replay determinism.

The primitives-only decision is the crux and it's correct.hydrateStepReturnValue (devalue.parse) returns a fresh object graph each call, and each replay runs in a fresh VM, so today the workflow gets a brand-new value every replay. Caching and returning the same object reference would let const r = await step(); r.count++ observe a prior replay's mutation — silent divergence. The alternatives are both worse: structuredClone is lossy for reviver-produced specials (stream handles, step-fn proxies, Request/Response, AbortController/Signal) and still O(size). Restricting the cache to primitives (immutable, compared by value) makes "share the reference" provably indistinguishable from re-parsing, and non-primitives fall through to a full re-hydrate every replay — preserving current behavior exactly. Trading the object-case optimization for airtight determinism is the right call.

What I verified in the integration:

  • Surgical wrap: only the await hydrateStepReturnValue(...) call is replaced; pendingDeliveries++/--, the ctx.promiseQueue.then(...) slot, and resolve(...) are byte-for-byte unchanged. The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the identical position in the ordered delivery chain a re-hydrate would have — preserving the pendingDeliveries-gated suspension/barrier/Promise.race determinism.
  • Lifetime: cache is created once per invocation in runtime.ts (outside the per-iteration context), threaded through runWorkflow, never shared across runs. The optional param/context field degrades to re-hydrating-every-replay for harnesses that omit it.
  • Keying by step_completedeventId (stable, world-assigned, same immutable bytes every replay) is sound, and has() rather than get() !== undefined correctly treats a memoized undefined result as a hit.
  • Errors never cached — a rejected hydrate re-attempts next replay, no parked rejected promise.

Built @workflow/core; full suite green (1249) including the new step-hydration-cache.test.ts / step-hydration-memoization.test.ts, async-deserialization-ordering.test.ts, and workflow.test.ts.

One forward-looking note (non-blocking): the object case is left on the table by design. If it's ever revisited, the safe path is deep-freeze + share frozen graphs — but only after handling the reviver-produced specials, so I agree with deferring it. LGTM.

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

// Only memoize values that are safe to return by reference across replays.
// Non-primitives fall through and are re-hydrated fresh on every replay.
if (isMemoizablePrimitive(value)) {
cache.set(eventId, value);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

The per-run stepHydrationCache is never size-bounded or evicted: it grows one entry per primitive-returning completed step and lives for the entire invocation. The cost worth calling out (the PR doesn't) is the new residency — the decrypted/devalue-parsed plaintext of each cached primitive is now held for the whole invocation, on top of the serialized bytes already retained in cachedEvents. For a long sequential workflow whose steps return large strings, that roughly doubles peak retained memory for those results.

The dominant residency (the full event log in cachedEvents) already exists, so this isn't blocking, but a byte-size threshold would be a cheap safeguard — large primitives are exactly the cheap-to-re-hydrate case relative to their footprint, so letting them fall through to the existing re-hydrate path costs little and bounds the worst case. Ideally with a test asserting the bound.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — capped it, with a test. Pushed in a60baad.

What changed (step-hydration-cache.ts):

  • Added MAX_MEMOIZED_PRIMITIVE_LENGTH = 4096 and extended isMemoizablePrimitive so a string/bigint longer than 4 KiB is treated as non-memoizable. Those are the only primitive types that can carry a large payload — number/boolean/null/undefined/symbol are inherently small, so they're never length-checked. Oversized values now fall through to the existing per-replay re-hydrate path, exactly as you suggested: large primitives are cheap to re-hydrate relative to their footprint, so this caps the doubled-residency worst case at negligible cost.
  • Documented the memory characteristic on the cache module: per-invocation lifetime (fresh Map per run in runtime.ts, GC'd when the invocation returns), bounded by the number of primitive-returning completed steps, primitives-only, now byte-bounded.

Tests (step-hydration-cache.test.ts, +4):isMemoizablePrimitive true at the bound / false beyond it (string and bigint), and an end-to-end assertion that an oversized string re-hydrates on every replay and cache.size === 0 (the bound assertion you asked for); plus an at-bound string is a cache hit.

The cap only ever reduces what gets cached, so determinism is untouched — oversized values just take the already-correct re-hydrate path. Full core suite green (1253, incl. the ordering/determinism + memoization suites); biome + tsc clean.

On consistency with #2471 (the sibling scriptCache): noting the distinction since they're bounded for different reasons. #2471's cache is process-wide and monotonic across the whole process — in dev/watch it pins every historical bundle string (hundreds of MB over a session), which is a genuine regression vs. the prior keep-only-latest behavior, hence the Blocking bound there. This cache is per-invocation and freed wholesale when the run returns, so it can never accumulate across runs; the only real cost is the doubled residency for large primitives during one run, which the size cap here now bounds. Different scope, different severity, but both bounded now.

Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-hydration
* origin/main:
perf(core): lazy inline step start (save one world round-trip per step) (#2478)
perf(core): skip per-step events.list via inline event-log delta (#2475)
Version Packages (beta) (#2491)
[world-vercel] Honor hasMore flag from v4 list pagination endpoint (#2486)
Version Packages (beta) (#2451)
Fix Next workflow module specifier root (#2455)
[world-vercel] Send remoteRefBehavior=lazy on v4 metadata-only event listings (#2415)
[swc-plugin] Fix eager discovery for object property steps (#2484)
fix(web-shared): align attributes panel styling (#2483)
[web-shared] Auto-scroll trace viewer on J/K span navigation (#2366)
fix(web): render restarted step segment as solid gray, not running stripes (#2480)
fix(web-shared): use solid gray for queued trace segment (#2474)
Add trace viewer span markers for hooks and attributes (#2452)
test: support Vercel protection bypass secret in e2e headers (#2458)
fix(core): bump payload-compression cutoff to 5.0.0-beta.18 (#2470)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts:
#	packages/core/src/runtime.ts
@pranaygp
pranaygp enabled auto-merge (squash) June 18, 2026 06:24
@pranaygp
pranaygp disabled auto-merge June 22, 2026 20:30
@pranaygp
pranaygp merged commit 66ca0dc into mainJun 22, 2026
118 of 121 checks passed
@pranaygp
pranaygp deleted the pgp/perf-memoize-step-hydration branch June 22, 2026 20:30
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
github-actionsBot added a commit that referenced this pull request Jun 22, 2026
…#2472)
* perf(core): memoize step return value hydration across replays
The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): bound memoized step-hydration cache by primitive size
Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #2563. Merge conflicts were resolved by AI — please review carefully. (backport job run)

pranaygp added a commit that referenced this pull request Jun 22, 2026
…testing
* origin/main:
Version Packages (beta) (#2540)
perf(core): memoize step return value hydration across inline replays (#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (#2412)
Amend lazy discovery removal changeset (#2560)
[docs] Document minimum SDK version for using hook.getConflict (#2423)
Update default CODEOWNERS (#2556)
Optimize and fix the default eager build mode (#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (#2547)" (#2554)
[core] Turbo mode: fast-path the first invocation (#2526)
Remove lazy discovery from workflow/next (#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (#2547)
feat(docs): add eve and AI SDK to product switcher (#2543)
[vitest] Fix local imports failing to load in test step bundles (#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (#2324)
Version Packages (beta) (#2495)
otel(world-vercel): inject trace context on v4 event requests (#2533)
Bump undici to 7.28.0 (#2534)
Default source maps to dev-on / prod-off (#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (#2527)
perf(core): parallel inline steps + optimistic lazy step start (#2516)
pranaygp added a commit to marcopiraccini/workflow that referenced this pull request Jun 22, 2026
* origin/main: (120 commits)
Version Packages (beta) (vercel#2540)
perf(core): memoize step return value hydration across inline replays (vercel#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (vercel#2412)
Amend lazy discovery removal changeset (vercel#2560)
[docs] Document minimum SDK version for using hook.getConflict (vercel#2423)
Update default CODEOWNERS (vercel#2556)
Optimize and fix the default eager build mode (vercel#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)" (vercel#2554)
[core] Turbo mode: fast-path the first invocation (vercel#2526)
Remove lazy discovery from workflow/next (vercel#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)
feat(docs): add eve and AI SDK to product switcher (vercel#2543)
[vitest] Fix local imports failing to load in test step bundles (vercel#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (vercel#2324)
Version Packages (beta) (vercel#2495)
otel(world-vercel): inject trace context on v4 event requests (vercel#2533)
Bump undici to 7.28.0 (vercel#2534)
Default source maps to dev-on / prod-off (vercel#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (vercel#2527)
perf(core): parallel inline steps + optimistic lazy step start (vercel#2516)
...
VaguelySerious pushed a commit that referenced this pull request Jun 22, 2026
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pranaygp@TooTallNate@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

perf(core): memoize step return value hydration across inline replays - #2472

Merged
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration
Jun 22, 2026
Merged

perf(core): memoize step return value hydration across inline replays#2472
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration

Conversation

@pranaygp

@pranaygppranaygp commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

The inline replay loop (runtime.tsrunWorkflow, workflow.ts) re-executes the workflow body and re-consumes the full event log on every iteration. For each already-completed step, the step consumer (step.ts, step_completed path) re-ran hydrateStepReturnValue — AES-GCM decrypt + devalue-parse of the serialized result — on every replay, even though that exact result was already hydrated on every prior replay.

For a sequential workflow of N steps, replay K hydrates K results, so the aggregate cost across a single invocation is O(N²) decrypt+parse operations.

This PR adds a per-run memoization cache so a completed step's hydrated result is returned in O(1) on subsequent replays within the same invocation, making the aggregate cost O(N).

Before / after

  • Before: replay 1 hydrates 1 result, replay 2 hydrates 2, …, replay K hydrates K → Σ = O(N²) decrypt+parse over a sequential run.
  • After: each completed step's result is hydrated once and memoized for the rest of the invocation → O(N) total. Replay K hydrates only the one newly-completed step; the K−1 prior results are cache hits.

Cache scope & keying

  • Lifetime / scope: owned by the inline loop in runtime.ts (created once per run invocation, alongside cachedEvents), threaded into runWorkflow(..., stepHydrationCache?) and stored on WorkflowOrchestratorContext.stepHydrationCache. A fresh context is created each loop iteration, so the cache deliberately lives outside the per-iteration context to survive across iterations of the same run. It is never shared across unrelated runs or process-level invocations.
  • Keying: by the persisted step_completed event's eventId — a stable, world-assigned id. The same event carries the same immutable serialized bytes across every replay, so a hit is guaranteed to correspond to identical input.
  • Optional / backward compatible: the parameter and context field are optional. Callers/harnesses that omit them (and the many runWorkflow(...) unit tests) degrade to re-hydrating every replay — identical to previous behavior.

Memory characteristic

A cached entry holds the decrypted/devalue-parsed plaintext of a step result, retained for the rest of the invocation on top of the serialized bytes already held in cachedEvents — so for large primitive results it roughly doubles peak retained memory for those results during the run. This residual is:

  • Scoped to one invocation — the Map is created per run and GC'd when the invocation returns; nothing accumulates across runs or process-level invocations (a much weaker concern than a process-wide cache, where the dominant residency — the full event log in cachedEvents — already exists for the same lifetime).
  • Bounded by the primitive-returning completed-step count — at most one small entry per such step.
  • Byte-bounded. Most primitives (numbers, booleans, null/undefined, symbols, short ids/strings) are tiny. The only primitive that can be large is a string (or a pathologically long bigint), so a string/bigint result longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) is not memoized — it falls through to the existing per-replay re-hydrate path. Large payloads are cheap to re-hydrate relative to their footprint, so this caps the worst case at negligible cost. The cap only ever reduces what is cached, so deterministic replay is unaffected.

Ordering safety analysis

The cache lookup replaces only the await hydrateStepReturnValue(...) call inside the existing ctx.promiseQueue.then(async () => { ... }) slot. Everything else is byte-for-byte unchanged:

  • ctx.pendingDeliveries++ / -- accounting is untouched.
  • The hydrate (or cache hit) still happens inside the same serial promiseQueue slot, at the same log position, and still resolves via the same resolve(...).
  • The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the exact position in the ordered delivery chain a re-hydrate would have.

So delivery order, pendingDeliveries-gated suspensions, the pendingDeliveryBarriers / awaitEarlierDeliveries machinery, and Promise.race/Promise.all replay determinism are all unaffected. Hook, wait, and abort hydration paths are intentionally not cached (they're the ordering-sensitive paths and not the O(N²) hotspot).

Identity / immutability safety

hydrateStepReturnValue (devalue.parse) returns a fresh object graph on every call, and each replay iteration runs in a fresh workflow VM. Today the workflow therefore receives a brand-new value on every replay. If we cached and returned the same object reference across replays, workflow code that mutates a step result (const r = await step(); r.count++) would observe a previous replay's mutation on the next replay — a non-deterministic divergence. (structuredClone on each hit is both lossy — revivers reconstruct stream handles, step-function proxies, Request/Response, and AbortController/AbortSignal class instances — and still O(size).)

Decision: only primitives are memoized (string, number, boolean, bigint, symbol, null, undefined). Primitives are immutable and compared by value, so sharing the reference is provably indistinguishable from re-parsing. Any non-primitive result falls through to a full re-hydrate every replay, preserving current behavior exactly. Errors are never cached, so a rejected hydrate re-attempts on the next replay (no parked rejected promise). This trades away the optimization in the object-returning case to keep deterministic replay airtight — correctness over speed.

What I verified

  • Unit:step-hydration-cache.test.ts (14 tests: primitive detection, memoization, non-primitive eviction/fresh-object, falsy primitives, keying, error non-caching, no-cache passthrough, plus the size-bound — at-bound string is a hit, oversized string/bigint are not memoized and cache.size stays 0) and step-hydration-memoization.test.ts (3 tests through the real createUseStep consumer: hydrate-skipped-on-replay-2 via spy, event-log ordering preserved on cache hits, fresh object per replay for object results).
  • Full core suite:cd packages/core && pnpm test1253 passed / 56 files, including async-deserialization-ordering.test.ts, workflow.test.ts (79 tests), runtime.test.ts, hook-sleep-interaction, abort-consistency. No regressions.
  • Build / format / typecheck:pnpm build (full repo, 27/27), @workflow/core build + tsc --noEmit clean; Biome format applied; new files Biome-clean (the only lint errors were import-ordering, auto-fixed; remaining warnings are pre-existing noExcessiveCognitiveComplexity on functions I only edited).
  • E2E (local nextjs-turbopack dev server, the determinism-sensitive subset): promiseAllWorkflow, promiseRaceWorkflow, promiseAnyWorkflow, sleepWinsRaceWorkflow, stepWinsRaceWorkflow, promiseRaceStressTestWorkflow, hookWorkflow, webhookWorkflow, parallel-steps-then-webhook replay race, sleepingWorkflow, parallelSleepWorkflow, retry/error/catchability suite, fetchWorkflowall passed.

Risks / deferred

  • Only primitive step results are accelerated; object-returning steps still re-hydrate each replay (intentional, for determinism). A future safe extension could deep-freeze + share frozen object graphs, but that needs care around reviver-produced special objects and is out of scope here.
  • Large (>4 KiB) string/bigint results are intentionally not memoized to bound peak retained memory (see Memory characteristic); they re-hydrate each replay.
  • Hook/wait/abort hydration paths are uncached by design.

🤖 Generated with Claude Code

The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CopilotAI review requested due to automatic review settings June 17, 2026 01:47
@pranaygp
pranaygp requested a review from a team as a code ownerJune 17, 2026 01:47

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@changeset-bot

changeset-botBot commented Jun 17, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3ca022f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
workflowPatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production144112301672
✅ 💻 Local Development190902192128
✅ 📦 Local Production190902192128
❌ 🐘 Local Postgres189412332128
✅ 🪟 Windows15200152
✅ 📋 Other88501791064
Total8190210809272

❌ Failed Tests

▲ Vercel Production (1 failed)

nitro (1 failed):

  • startFromWorkflow - calling start() directly inside a workflow function with hook communication | wrun_01KVCQ35C5K211Z7CFGRHG4655 | 🔍 observability
🐘 Local Postgres (1 failed)

nextjs-turbopack-stable-lazy-discovery-enabled (1 failed):

  • wellKnownAgentWorkflow (.well-known/agent) | wrun_01KVCPM9CVA5BNHKTABQFTWTWF

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro125027
✅ example125027
✅ express125027
✅ fastify125027
✅ hono125027
✅ nextjs-turbopack14903
✅ nextjs-webpack14903
❌ nitro124127
✅ nuxt125027
✅ sveltekit14408
✅ vite125027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
❌ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable126026
✅ express-stable126026
✅ fastify-stable126026
✅ hono-stable126026
✅ nextjs-turbopack-canary132020
✅ nextjs-turbopack-stable-lazy-discovery-disabled15101
❌ nextjs-turbopack-stable-lazy-discovery-enabled15011
✅ nextjs-webpack-canary132020
✅ nextjs-webpack-stable-lazy-discovery-disabled15101
✅ nextjs-webpack-stable-lazy-discovery-enabled15101
✅ nitro-stable126026
✅ nuxt-stable126026
✅ sveltekit-stable14507
✅ vite-stable126026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15200
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable127025
✅ e2e-local-dev-tanstack-start-127025
✅ e2e-local-postgres-nest-stable126026
✅ e2e-local-postgres-tanstack-start-126026
✅ e2e-local-prod-nest-stable127025
✅ e2e-local-prod-tanstack-start-127025
✅ e2e-vercel-prod-tanstack-start125027

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: failure
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.042s (-2.5%)1.006s (~)0.963s101.00x
💻 LocalNitro0.045s (+10.1% 🔺)1.006s (~)0.962s101.06x
💻 LocalNext.js (Turbopack)0.062s (-5.5% 🟢)1.007s (~)0.944s101.47x
🐘 PostgresExpress0.067s (-8.0% 🟢)1.013s (~)0.946s101.58x
🐘 PostgresNext.js (Turbopack)0.070s (-0.9%)1.013s (~)0.943s101.65x
🐘 PostgresNitro0.073s (+15.6% 🔺)1.013s (~)0.940s101.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express0.271s (-28.1% 🟢)2.235s (-2.6%)1.964s101.00x
▲ VercelNitro0.319s (-14.9% 🟢)2.203s (-2.0%)1.883s101.18x
▲ VercelNext.js (Turbopack)0.351s (+20.6% 🔺)2.531s (+33.8% 🔺)2.180s101.29x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.094s (-0.7%)2.006s (~)0.912s101.00x
💻 LocalNitro1.095s (+0.9%)2.007s (~)0.912s101.00x
🐘 PostgresNitro1.108s (-1.1%)2.009s (~)0.901s101.01x
🐘 PostgresExpress1.110s (~)2.008s (~)0.898s101.02x
💻 LocalNext.js (Turbopack)1.132s (-1.4%)2.007s (~)0.875s101.03x
🐘 PostgresNext.js (Turbopack)1.141s (~)2.009s (~)0.868s101.04x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.723s (-11.0% 🟢)3.716s (+1.1%)1.993s101.00x
▲ VercelNext.js (Turbopack)1.761s (~)3.708s (~)1.947s101.02x
▲ VercelNitro1.788s (-22.8% 🟢)3.409s (-14.6% 🟢)1.621s101.04x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro10.527s (~)11.022s (~)0.495s31.00x
💻 LocalExpress10.566s (~)11.023s (~)0.457s31.00x
🐘 PostgresExpress10.585s (~)11.023s (~)0.438s31.01x
🐘 PostgresNitro10.598s (+0.6%)11.022s (~)0.424s31.01x
💻 LocalNext.js (Turbopack)10.812s (~)11.022s (~)0.210s31.03x
🐘 PostgresNext.js (Turbopack)10.829s (~)11.017s (~)0.189s31.03x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express13.691s (-24.2% 🟢)15.347s (-23.1% 🟢)1.656s21.00x
▲ VercelNitro13.728s (-27.6% 🟢)15.441s (-27.5% 🟢)1.713s21.00x
▲ VercelNext.js (Turbopack)14.688s (+2.6%)16.926s (+3.9%)2.238s21.07x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro13.690s (~)14.027s (~)0.337s51.00x
🐘 PostgresExpress13.756s (-0.7%)14.019s (~)0.263s51.00x
💻 LocalExpress13.798s (~)14.028s (~)0.231s51.01x
🐘 PostgresNitro13.832s (~)14.023s (~)0.191s51.01x
💻 LocalNext.js (Turbopack)14.383s (~)15.030s (~)0.647s41.05x
🐘 PostgresNext.js (Turbopack)14.396s (~)15.017s (~)0.621s41.05x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express23.028s (-4.9%)25.270s (-3.5%)2.242s31.00x
▲ VercelNext.js (Turbopack)24.609s (-30.5% 🟢)26.392s (-28.3% 🟢)1.783s31.07x
▲ VercelNitro24.684s (-20.8% 🟢)26.313s (-19.2% 🟢)1.629s31.07x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro12.373s (+2.5%)13.027s (+1.1%)0.654s71.00x
💻 LocalExpress12.444s (-0.8%)13.024s (~)0.580s71.01x
🐘 PostgresExpress12.512s (+0.6%)13.016s (~)0.503s71.01x
🐘 PostgresNitro12.712s (+1.6%)13.021s (~)0.310s71.03x
💻 LocalNext.js (Turbopack)13.662s (~)14.027s (~)0.365s71.10x
🐘 PostgresNext.js (Turbopack)13.897s (~)14.308s (+1.0%)0.411s71.12x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro27.276s (-29.9% 🟢)29.146s (-28.8% 🟢)1.870s41.00x
▲ VercelNext.js (Turbopack)27.460s (-22.0% 🟢)29.404s (-20.4% 🟢)1.944s41.01x
▲ VercelExpress28.389s (-19.0% 🟢)30.612s (-17.0% 🟢)2.223s31.04x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.204s (-0.8%)2.007s (~)0.803s151.00x
💻 LocalNitro1.222s (+4.5%)2.006s (~)0.784s151.01x
💻 LocalExpress1.231s (+5.6% 🔺)2.007s (~)0.776s151.02x
🐘 PostgresNitro1.251s (+4.4%)2.009s (~)0.757s151.04x
🐘 PostgresNext.js (Turbopack)1.256s (-2.3%)2.007s (~)0.751s151.04x
💻 LocalNext.js (Turbopack)1.405s (+8.4% 🔺)2.006s (~)0.602s151.17x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.571s (+15.9% 🔺)4.329s (+11.4% 🔺)1.757s81.00x
▲ VercelNitro3.123s (+7.3% 🔺)4.469s (+1.4%)1.346s71.21x
▲ VercelNext.js (Turbopack)3.608s (+35.3% 🔺)4.968s (+21.2% 🔺)1.360s71.40x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.309s (-8.1% 🟢)2.008s (-16.1% 🟢)0.700s151.00x
🐘 PostgresNitro1.332s (-2.4%)2.074s (-17.3% 🟢)0.742s151.02x
🐘 PostgresNext.js (Turbopack)1.452s (-11.3% 🟢)2.075s (-13.3% 🟢)0.623s151.11x
💻 LocalExpress1.987s (+22.2% 🔺)2.592s (+29.2% 🔺)0.604s121.52x
💻 LocalNitro2.051s (+32.2% 🔺)2.507s (+24.7% 🔺)0.456s121.57x
💻 LocalNext.js (Turbopack)2.359s (+23.5% 🔺)3.008s (+31.2% 🔺)0.649s101.80x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.443s (-13.1% 🟢)5.587s (-1.5%)2.143s61.00x
▲ VercelNitro3.655s (-9.7% 🟢)5.233s (-8.2% 🟢)1.577s61.06x
▲ VercelNext.js (Turbopack)4.245s (-5.5% 🟢)5.891s (-9.1% 🟢)1.646s61.23x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.468s (-16.7% 🟢)3.886s (~)2.417s81.00x
🐘 PostgresNitro1.470s (-7.8% 🟢)3.885s (-3.1%)2.415s81.00x
🐘 PostgresNext.js (Turbopack)2.642s (-16.3% 🟢)3.456s (-19.7% 🟢)0.814s91.80x
💻 LocalNitro4.340s (+26.8% 🔺)5.013s (+25.0% 🔺)0.673s62.96x
💻 LocalExpress5.253s (+20.2% 🔺)5.679s (+16.7% 🔺)0.425s63.58x
💻 LocalNext.js (Turbopack)6.408s (+11.0% 🔺)7.019s (+12.9% 🔺)0.611s54.36x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro4.491s (-26.7% 🟢)6.303s (-24.4% 🟢)1.812s51.00x
▲ VercelNext.js (Turbopack)4.524s (-19.6% 🟢)6.497s (-15.8% 🟢)1.973s51.01x
▲ VercelExpress4.646s (-15.6% 🟢)7.023s (-11.4% 🟢)2.378s51.03x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.202s (-1.3%)2.007s (~)0.805s151.00x
🐘 PostgresNitro1.214s (~)2.008s (~)0.794s151.01x
💻 LocalExpress1.228s (-22.1% 🟢)2.006s (~)0.778s151.02x
💻 LocalNitro1.268s (+5.6% 🔺)2.006s (~)0.738s151.06x
🐘 PostgresNext.js (Turbopack)1.271s (-0.8%)2.008s (~)0.737s151.06x
💻 LocalNext.js (Turbopack)1.449s (+3.6%)2.006s (~)0.557s151.21x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.668s (-25.2% 🟢)4.041s (-23.6% 🟢)1.373s81.00x
▲ VercelNext.js (Turbopack)2.876s (-22.7% 🟢)4.782s (-8.6% 🟢)1.907s71.08x
▲ VercelExpress3.339s (+43.7% 🔺)5.449s (+43.2% 🔺)2.110s61.25x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Nitro1.304s (-9.6% 🟢)2.075s (-13.3% 🟢)0.771s151.00x
🐘 PostgresExpress1.310s (-4.1%)2.007s (-13.3% 🟢)0.697s151.00x
🐘 PostgresNext.js (Turbopack)1.436s (-6.9% 🟢)2.076s (-6.6% 🟢)0.640s151.10x
💻 LocalNitro1.935s (+14.3% 🔺)2.293s (+14.2% 🔺)0.358s141.48x
💻 LocalExpress2.003s (+5.5% 🔺)2.393s (+11.3% 🔺)0.390s131.54x
💻 LocalNext.js (Turbopack)2.369s (+9.3% 🔺)3.009s (~)0.639s101.82x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.899s (+6.3% 🔺)4.683s (+2.5%)1.785s71.00x
▲ VercelNitro3.978s (+42.1% 🔺)5.734s (+37.1% 🔺)1.756s61.37x
▲ VercelNext.js (Turbopack)4.216s (-5.6% 🟢)5.944s (-3.4%)1.728s61.45x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.452s (-19.3% 🟢)4.014s (~)2.561s81.00x
🐘 PostgresNitro1.589s (-12.0% 🟢)3.678s (-14.4% 🟢)2.090s91.09x
🐘 PostgresNext.js (Turbopack)2.136s (-47.5% 🟢)3.454s (-22.3% 🟢)1.318s91.47x
💻 LocalExpress5.580s (+21.6% 🔺)6.014s (+20.0% 🔺)0.434s53.84x
💻 LocalNitro5.616s (+25.0% 🔺)6.017s (+20.1% 🔺)0.401s53.87x
💻 LocalNext.js (Turbopack)6.852s (+18.0% 🔺)7.416s (+15.6% 🔺)0.563s54.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.610s (-57.2% 🟢)5.562s (-45.7% 🟢)1.952s61.00x
▲ VercelNitro4.046s (-70.8% 🟢)5.724s (-63.4% 🟢)1.678s61.12x
▲ VercelNext.js (Turbopack)4.124s (+9.0% 🔺)6.272s (+19.9% 🔺)2.147s51.14x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.605s (+6.0% 🔺)1.041s (+3.5%)0.436s581.00x
💻 LocalExpress0.616s (-1.5%)1.005s (-1.7%)0.389s601.02x
💻 LocalNitro0.629s (+25.8% 🔺)1.039s (+3.1%)0.410s581.04x
🐘 PostgresNitro0.692s (+16.5% 🔺)1.078s (+3.6%)0.386s561.14x
🐘 PostgresNext.js (Turbopack)0.858s (+2.7%)1.041s (+1.7%)0.183s581.42x
💻 LocalNext.js (Turbopack)0.860s (-3.1%)1.005s (-3.3%)0.144s601.42x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.774s (-42.9% 🟢)5.457s (-34.5% 🟢)1.683s111.00x
▲ VercelNext.js (Turbopack)3.815s (-39.5% 🟢)5.390s (-33.8% 🟢)1.575s121.01x
▲ VercelExpress4.488s (-1.3%)6.377s (-1.6%)1.889s101.19x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.375s (+1.4%)2.052s (+1.2%)0.678s441.00x
🐘 PostgresNitro1.466s (+5.6% 🔺)2.030s (~)0.564s451.07x
💻 LocalNitro1.485s (+25.1% 🔺)2.006s (~)0.521s451.08x
💻 LocalExpress1.530s (+2.8%)2.007s (~)0.476s451.11x
🐘 PostgresNext.js (Turbopack)1.989s (+2.5%)2.308s (+11.2% 🔺)0.319s401.45x
💻 LocalNext.js (Turbopack)2.090s (-0.6%)2.944s (-2.1%)0.854s311.52x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express10.297s (-41.1% 🟢)12.369s (-35.5% 🟢)2.072s81.00x
▲ VercelNitro10.774s (-8.8% 🟢)12.396s (-12.5% 🟢)1.621s81.05x
▲ VercelNext.js (Turbopack)11.321s (-16.7% 🟢)13.637s (-11.0% 🟢)2.316s71.10x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express2.654s (-2.1%)3.058s (-1.7%)0.404s401.00x
🐘 PostgresNitro2.903s (+3.7%)3.280s (+3.6%)0.376s371.09x
💻 LocalExpress3.275s (+2.4%)4.010s (~)0.735s301.23x
💻 LocalNitro3.362s (+23.1% 🔺)4.010s (+24.4% 🔺)0.648s301.27x
🐘 PostgresNext.js (Turbopack)3.983s (+2.9%)4.253s (+4.3%)0.270s291.50x
💻 LocalNext.js (Turbopack)4.363s (~)5.010s (~)0.647s241.64x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express18.286s (-29.9% 🟢)20.532s (-26.7% 🟢)2.246s61.00x
▲ VercelNitro18.713s (-28.9% 🟢)20.382s (-27.5% 🟢)1.669s61.02x
▲ VercelNext.js (Turbopack)20.206s (-19.2% 🟢)22.420s (-16.3% 🟢)2.214s61.11x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.232s (~)1.006s (~)0.773s601.00x
🐘 PostgresNitro0.237s (+0.7%)1.006s (~)0.769s601.02x
🐘 PostgresNext.js (Turbopack)0.297s (+1.1%)1.023s (+1.7%)0.726s591.28x
💻 LocalExpress0.400s (-9.2% 🟢)1.005s (~)0.605s601.72x
💻 LocalNitro0.417s (+11.8% 🔺)1.004s (~)0.588s601.79x
💻 LocalNext.js (Turbopack)0.633s (+8.8% 🔺)1.004s (~)0.371s602.73x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.787s (+15.3% 🔺)3.598s (+25.4% 🔺)1.811s181.00x
▲ VercelNitro1.933s (+49.7% 🔺)3.615s (+25.8% 🔺)1.682s171.08x
▲ VercelNext.js (Turbopack)2.387s (+27.6% 🔺)4.344s (+11.8% 🔺)1.956s141.34x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.331s (-6.9% 🟢)1.006s (-3.3%)0.675s901.00x
🐘 PostgresNitro0.356s (+4.1%)1.006s (-1.1%)0.650s901.08x
🐘 PostgresNext.js (Turbopack)0.476s (-11.7% 🟢)1.103s (-1.3%)0.628s831.44x
💻 LocalNitro2.182s (+43.2% 🔺)2.737s (+27.2% 🔺)0.555s336.59x
💻 LocalExpress2.222s (+6.3% 🔺)2.738s (+3.1%)0.517s336.71x
💻 LocalNext.js (Turbopack)2.493s (+7.0% 🔺)3.344s (+8.7% 🔺)0.851s277.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.638s (-9.8% 🟢)4.553s (-5.1% 🟢)1.915s201.00x
▲ VercelNitro2.705s (-1.5%)4.386s (-8.3% 🟢)1.681s211.03x
▲ VercelNext.js (Turbopack)3.092s (+1.6%)4.802s (~)1.710s191.17x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.526s (-7.4% 🟢)1.059s (-13.0% 🟢)0.532s1141.00x
🐘 PostgresNitro0.570s (~)1.078s (-12.5% 🟢)0.508s1121.08x
🐘 PostgresNext.js (Turbopack)1.918s (-26.7% 🟢)2.763s (-20.4% 🟢)0.845s443.64x
💻 LocalNitro9.521s (+50.8% 🔺)10.445s (+54.9% 🔺)0.924s1218.09x
💻 LocalExpress10.198s (+22.8% 🔺)11.029s (+24.2% 🔺)0.831s1219.38x
💻 LocalNext.js (Turbopack)10.284s (-4.4%)11.663s (+0.8%)1.379s1119.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.245s (-38.5% 🟢)5.031s (-28.3% 🟢)1.786s241.00x
▲ VercelExpress3.483s (-36.2% 🟢)5.687s (-19.8% 🟢)2.204s221.07x
▲ VercelNext.js (Turbopack)4.699s (-2.9%)6.708s (+2.7%)2.008s181.45x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.165s (-1.1%)2.000s (~)0.001s (+9.1% 🔺)2.009s (~)0.844s101.00x
🐘 PostgresNitro1.181s (+1.4%)1.995s (~)0.001s (+30.0% 🔺)2.011s (~)0.831s101.01x
💻 LocalNitro1.181s (+5.7% 🔺)2.005s (~)0.010s (-62.9% 🟢)2.017s (-1.0%)0.836s101.01x
💻 LocalExpress1.185s (+3.0%)2.005s (~)0.012s (+19.8% 🔺)2.019s (~)0.835s101.02x
💻 LocalNext.js (Turbopack)1.216s (~)2.003s (~)0.013s (+2.4%)2.020s (~)0.803s101.04x
🐘 PostgresNext.js (Turbopack)1.232s (~)2.002s (~)0.001s (-15.4% 🟢)2.011s (~)0.779s101.06x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.206s (-8.2% 🟢)3.426s (-6.9% 🟢)0.602s (-18.8% 🟢)4.513s (-7.4% 🟢)2.307s101.00x
▲ VercelNext.js (Turbopack)2.304s (-8.6% 🟢)3.523s (-7.8% 🟢)0.772s (-4.3%)4.790s (-5.8% 🟢)2.486s101.04x
▲ VercelNitro2.331s (-11.7% 🟢)3.158s (-17.0% 🟢)1.253s (+57.9% 🔺)4.805s (-4.9%)2.474s101.06x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro1.572s (+9.3% 🔺)2.011s (~)0.012s (-32.0% 🟢)2.025s (~)0.453s301.00x
🐘 PostgresExpress1.577s (~)2.004s (~)0.005s (+5.7% 🔺)2.026s (~)0.449s301.00x
💻 LocalExpress1.583s (+1.0%)2.010s (~)0.014s (+4.9%)2.025s (~)0.442s301.01x
🐘 PostgresNitro1.602s (+1.5%)2.008s (~)0.005s (-5.6% 🟢)2.027s (~)0.425s301.02x
💻 LocalNext.js (Turbopack)1.740s (-0.5%)2.010s (~)0.013s (+1.3%)2.025s (~)0.285s301.11x
🐘 PostgresNext.js (Turbopack)1.892s (+5.9% 🔺)2.011s (~)0.005s (+5.8% 🔺)2.029s (~)0.137s301.20x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Next.js (Turbopack)6.842s (-19.5% 🟢)8.374s (-14.7% 🟢)0.220s (-16.3% 🟢)9.079s (-14.1% 🟢)2.236s71.00x
▲ VercelNitro6.905s (-3.6%)7.768s (-10.3% 🟢)0.455s (+91.9% 🔺)8.766s (-6.9% 🟢)1.861s71.01x
▲ VercelExpress8.068s (+26.4% 🔺)9.373s (+20.1% 🔺)0.307s (+71.5% 🔺)10.412s (+23.1% 🔺)2.345s61.18x

🔍 Observability: Next.js (Turbopack) | Nitro | Express

10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.789s (+3.3%)1.101s (+7.5% 🔺)0.000s (-100.0% 🟢)1.118s (+6.0% 🔺)0.329s541.00x
🐘 PostgresNitro0.795s (+2.9%)1.044s (~)0.000s (-3.5%)1.061s (-3.0%)0.265s571.01x
🐘 PostgresNext.js (Turbopack)0.986s (-3.3%)1.397s (-4.6%)0.000s (-100.0% 🟢)1.407s (-4.4%)0.421s431.25x
💻 LocalExpress1.533s (+12.5% 🔺)2.014s (~)0.000s (+16.7% 🔺)2.016s (~)0.483s301.94x
💻 LocalNitro1.568s (+45.8% 🔺)2.014s (+9.6% 🔺)0.000s (-56.8% 🟢)2.016s (+9.5% 🔺)0.448s301.99x
💻 LocalNext.js (Turbopack)1.916s (+26.7% 🔺)2.193s (+8.9% 🔺)0.000s (+7.1% 🔺)2.196s (+8.9% 🔺)0.280s282.43x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.961s (-21.2% 🟢)4.002s (-25.7% 🟢)0.000s (-21.4% 🟢)4.479s (-23.6% 🟢)1.518s141.00x
▲ VercelExpress3.140s (+3.6%)4.516s (-1.1%)0.000s (NaN%)5.069s (+0.6%)1.929s121.06x
▲ VercelNext.js (Turbopack)3.540s (+2.7%)5.016s (+5.2% 🔺)0.000s (-100.0% 🟢)5.553s (+5.5% 🔺)2.013s111.20x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.691s (+4.7%)2.302s (+11.5% 🔺)0.000s (NaN%)2.316s (+10.9% 🔺)0.625s261.00x
🐘 PostgresNitro1.779s (+14.7% 🔺)2.381s (+11.6% 🔺)0.000s (+11.5% 🔺)2.394s (+11.5% 🔺)0.615s261.05x
🐘 PostgresNext.js (Turbopack)2.251s (+6.6% 🔺)2.651s (+2.5%)0.000s (-100.0% 🟢)2.664s (+2.6%)0.413s231.33x
💻 LocalNitro4.321s (+95.8% 🔺)4.717s (+72.0% 🔺)0.001s (+12.8% 🔺)4.729s (+71.7% 🔺)0.409s132.56x
💻 LocalExpress4.772s (+62.9% 🔺)5.362s (+46.1% 🔺)0.001s (+98.3% 🔺)5.367s (+46.1% 🔺)0.595s122.82x
💻 LocalNext.js (Turbopack)5.632s (+94.2% 🔺)6.226s (+85.4% 🔺)0.000s (-60.0% 🟢)6.235s (+85.4% 🔺)0.602s103.33x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express4.310s (-24.3% 🟢)6.054s (-16.2% 🟢)0.000s (+Infinity% 🔺)6.546s (-14.9% 🟢)2.236s101.00x
▲ VercelNext.js (Turbopack)4.772s (-34.2% 🟢)6.201s (-26.1% 🟢)0.000s (-100.0% 🟢)6.679s (-24.5% 🟢)1.907s101.11x
▲ VercelNitro4.835s (-7.9% 🟢)5.805s (-10.1% 🟢)0.000s (+Infinity% 🔺)6.308s (-8.9% 🟢)1.473s101.12x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalNitro12/21
🐘 PostgresExpress19/21
▲ VercelExpress14/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres17/21
Next.js (Turbopack)🐘 Postgres14/21
Nitro🐘 Postgres13/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run

@pranaygp

Copy link
Copy Markdown
ContributorAuthor

CI failure triage — pre-existing Vercel-prod e2e flake (not a regression)

The two red checks (E2E Vercel Prod Tests (example), E2E Vercel Prod Tests (fastify), which roll up into E2E Required Check) are the shared Vercel-prod timing flake, not caused by this PR. Evidence:

Scope is wrong for a hydration regression. This PR only memoizes primitive step-result hydration. A determinism/stale-value bug there would surface across all worlds — yet every local suite is 100% green:

SuitePassedFailed
▲ Vercel Production14402
💻 Local Development19090
📦 Local Production19090
🐘 Local Postgres18950
🪟 Windows1520

Only 2 failures, only on Vercel Production.

The two failing tests are unrelated to result hydration, and are abort/hook timing races:

  1. exampleAbortController > abortFromStepWorkflow: step abort cancels an in-flight sibling step. The run completed successfully; the assertion failed only because the abort lost a race. From the run diagnostics (wrun_01KV9MZ1Y00N90XFCKFTSPX5N1):

    +2.2s step_completed (longStep) <- sibling finished on its own
    +3.0s hook_received <- abort signal arrived AFTER
    

    The sibling longStep self-completed (2.2s) before the abort hook arrived (3.0s) under Vercel-prod queue/network latency, so there was no in-flight step left to cancel. Step results hydrated fine; this is purely environmental latency.

  2. fastifystartFromWorkflow - calling start() directly inside a workflow function with hook communication. This exact test passed (4128ms) in the example job of this same commit (a56f5c90b) — a textbook cross-run flake.

The same test is red on plain main, without this change. On main run 27704378960 (commit 2acf13cc7):

  • E2E Vercel Prod Tests (tanstack-start)abortFromStepWorkflow: step abort cancels an in-flight sibling step (the identical test that failed here on example)
  • E2E Vercel Prod Tests (nextjs-turbopack)distributedAbortController - reconnect to existing controller

And on main run 27657696161 (cb181392b, the commit this branch is based on): E2E Vercel Prod Tests (fastify)hookWithSleepFinalStepWorkflow. The set of red workbenches rotates run-to-run — the signature of environment flakiness, not a code regression.

Local verification of this branch (rebuilt @workflow/core first): cd packages/core && pnpm test1249 passed / 56 files, 0 failures, including the determinism/ordering replay tests in workflow.test.ts and the new step-hydration-cache.test.ts (10) + step-hydration-memoization.test.ts (3). The memoization tests assert byte-identical delivery ordering on cache hits and that objects re-hydrate fresh each replay.

Re-running the e2e jobs should clear them. No code change is warranted.

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — the O(N²)→O(N) hydration memoization, done with the right safety bias

This is the most safety-sensitive of the four (it's the only one that caches a value across replays), and the design lands on the conservative side of every judgment call, which is exactly right for replay determinism.

The primitives-only decision is the crux and it's correct.hydrateStepReturnValue (devalue.parse) returns a fresh object graph each call, and each replay runs in a fresh VM, so today the workflow gets a brand-new value every replay. Caching and returning the same object reference would let const r = await step(); r.count++ observe a prior replay's mutation — silent divergence. The alternatives are both worse: structuredClone is lossy for reviver-produced specials (stream handles, step-fn proxies, Request/Response, AbortController/Signal) and still O(size). Restricting the cache to primitives (immutable, compared by value) makes "share the reference" provably indistinguishable from re-parsing, and non-primitives fall through to a full re-hydrate every replay — preserving current behavior exactly. Trading the object-case optimization for airtight determinism is the right call.

What I verified in the integration:

  • Surgical wrap: only the await hydrateStepReturnValue(...) call is replaced; pendingDeliveries++/--, the ctx.promiseQueue.then(...) slot, and resolve(...) are byte-for-byte unchanged. The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the identical position in the ordered delivery chain a re-hydrate would have — preserving the pendingDeliveries-gated suspension/barrier/Promise.race determinism.
  • Lifetime: cache is created once per invocation in runtime.ts (outside the per-iteration context), threaded through runWorkflow, never shared across runs. The optional param/context field degrades to re-hydrating-every-replay for harnesses that omit it.
  • Keying by step_completedeventId (stable, world-assigned, same immutable bytes every replay) is sound, and has() rather than get() !== undefined correctly treats a memoized undefined result as a hit.
  • Errors never cached — a rejected hydrate re-attempts next replay, no parked rejected promise.

Built @workflow/core; full suite green (1249) including the new step-hydration-cache.test.ts / step-hydration-memoization.test.ts, async-deserialization-ordering.test.ts, and workflow.test.ts.

One forward-looking note (non-blocking): the object case is left on the table by design. If it's ever revisited, the safe path is deep-freeze + share frozen graphs — but only after handling the reviver-produced specials, so I agree with deferring it. LGTM.

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

// Only memoize values that are safe to return by reference across replays.
// Non-primitives fall through and are re-hydrated fresh on every replay.
if (isMemoizablePrimitive(value)) {
cache.set(eventId, value);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

The per-run stepHydrationCache is never size-bounded or evicted: it grows one entry per primitive-returning completed step and lives for the entire invocation. The cost worth calling out (the PR doesn't) is the new residency — the decrypted/devalue-parsed plaintext of each cached primitive is now held for the whole invocation, on top of the serialized bytes already retained in cachedEvents. For a long sequential workflow whose steps return large strings, that roughly doubles peak retained memory for those results.

The dominant residency (the full event log in cachedEvents) already exists, so this isn't blocking, but a byte-size threshold would be a cheap safeguard — large primitives are exactly the cheap-to-re-hydrate case relative to their footprint, so letting them fall through to the existing re-hydrate path costs little and bounds the worst case. Ideally with a test asserting the bound.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — capped it, with a test. Pushed in a60baad.

What changed (step-hydration-cache.ts):

  • Added MAX_MEMOIZED_PRIMITIVE_LENGTH = 4096 and extended isMemoizablePrimitive so a string/bigint longer than 4 KiB is treated as non-memoizable. Those are the only primitive types that can carry a large payload — number/boolean/null/undefined/symbol are inherently small, so they're never length-checked. Oversized values now fall through to the existing per-replay re-hydrate path, exactly as you suggested: large primitives are cheap to re-hydrate relative to their footprint, so this caps the doubled-residency worst case at negligible cost.
  • Documented the memory characteristic on the cache module: per-invocation lifetime (fresh Map per run in runtime.ts, GC'd when the invocation returns), bounded by the number of primitive-returning completed steps, primitives-only, now byte-bounded.

Tests (step-hydration-cache.test.ts, +4):isMemoizablePrimitive true at the bound / false beyond it (string and bigint), and an end-to-end assertion that an oversized string re-hydrates on every replay and cache.size === 0 (the bound assertion you asked for); plus an at-bound string is a cache hit.

The cap only ever reduces what gets cached, so determinism is untouched — oversized values just take the already-correct re-hydrate path. Full core suite green (1253, incl. the ordering/determinism + memoization suites); biome + tsc clean.

On consistency with #2471 (the sibling scriptCache): noting the distinction since they're bounded for different reasons. #2471's cache is process-wide and monotonic across the whole process — in dev/watch it pins every historical bundle string (hundreds of MB over a session), which is a genuine regression vs. the prior keep-only-latest behavior, hence the Blocking bound there. This cache is per-invocation and freed wholesale when the run returns, so it can never accumulate across runs; the only real cost is the doubled residency for large primitives during one run, which the size cap here now bounds. Different scope, different severity, but both bounded now.

Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-hydration
* origin/main:
perf(core): lazy inline step start (save one world round-trip per step) (#2478)
perf(core): skip per-step events.list via inline event-log delta (#2475)
Version Packages (beta) (#2491)
[world-vercel] Honor hasMore flag from v4 list pagination endpoint (#2486)
Version Packages (beta) (#2451)
Fix Next workflow module specifier root (#2455)
[world-vercel] Send remoteRefBehavior=lazy on v4 metadata-only event listings (#2415)
[swc-plugin] Fix eager discovery for object property steps (#2484)
fix(web-shared): align attributes panel styling (#2483)
[web-shared] Auto-scroll trace viewer on J/K span navigation (#2366)
fix(web): render restarted step segment as solid gray, not running stripes (#2480)
fix(web-shared): use solid gray for queued trace segment (#2474)
Add trace viewer span markers for hooks and attributes (#2452)
test: support Vercel protection bypass secret in e2e headers (#2458)
fix(core): bump payload-compression cutoff to 5.0.0-beta.18 (#2470)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts:
#	packages/core/src/runtime.ts
@pranaygp
pranaygp enabled auto-merge (squash) June 18, 2026 06:24
@pranaygp
pranaygp disabled auto-merge June 22, 2026 20:30
@pranaygp
pranaygp merged commit 66ca0dc into mainJun 22, 2026
118 of 121 checks passed
@pranaygp
pranaygp deleted the pgp/perf-memoize-step-hydration branch June 22, 2026 20:30
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
github-actionsBot added a commit that referenced this pull request Jun 22, 2026
…#2472)
* perf(core): memoize step return value hydration across replays
The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): bound memoized step-hydration cache by primitive size
Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #2563. Merge conflicts were resolved by AI — please review carefully. (backport job run)

pranaygp added a commit that referenced this pull request Jun 22, 2026
…testing
* origin/main:
Version Packages (beta) (#2540)
perf(core): memoize step return value hydration across inline replays (#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (#2412)
Amend lazy discovery removal changeset (#2560)
[docs] Document minimum SDK version for using hook.getConflict (#2423)
Update default CODEOWNERS (#2556)
Optimize and fix the default eager build mode (#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (#2547)" (#2554)
[core] Turbo mode: fast-path the first invocation (#2526)
Remove lazy discovery from workflow/next (#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (#2547)
feat(docs): add eve and AI SDK to product switcher (#2543)
[vitest] Fix local imports failing to load in test step bundles (#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (#2324)
Version Packages (beta) (#2495)
otel(world-vercel): inject trace context on v4 event requests (#2533)
Bump undici to 7.28.0 (#2534)
Default source maps to dev-on / prod-off (#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (#2527)
perf(core): parallel inline steps + optimistic lazy step start (#2516)
pranaygp added a commit to marcopiraccini/workflow that referenced this pull request Jun 22, 2026
* origin/main: (120 commits)
Version Packages (beta) (vercel#2540)
perf(core): memoize step return value hydration across inline replays (vercel#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (vercel#2412)
Amend lazy discovery removal changeset (vercel#2560)
[docs] Document minimum SDK version for using hook.getConflict (vercel#2423)
Update default CODEOWNERS (vercel#2556)
Optimize and fix the default eager build mode (vercel#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)" (vercel#2554)
[core] Turbo mode: fast-path the first invocation (vercel#2526)
Remove lazy discovery from workflow/next (vercel#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)
feat(docs): add eve and AI SDK to product switcher (vercel#2543)
[vitest] Fix local imports failing to load in test step bundles (vercel#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (vercel#2324)
Version Packages (beta) (vercel#2495)
otel(world-vercel): inject trace context on v4 event requests (vercel#2533)
Bump undici to 7.28.0 (vercel#2534)
Default source maps to dev-on / prod-off (vercel#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (vercel#2527)
perf(core): parallel inline steps + optimistic lazy step start (vercel#2516)
...
VaguelySerious pushed a commit that referenced this pull request Jun 22, 2026
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pranaygp@TooTallNate@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

perf(core): memoize step return value hydration across inline replays - #2472

Merged
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration
Jun 22, 2026
Merged

perf(core): memoize step return value hydration across inline replays#2472
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration

Conversation

@pranaygp

@pranaygppranaygp commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

The inline replay loop (runtime.tsrunWorkflow, workflow.ts) re-executes the workflow body and re-consumes the full event log on every iteration. For each already-completed step, the step consumer (step.ts, step_completed path) re-ran hydrateStepReturnValue — AES-GCM decrypt + devalue-parse of the serialized result — on every replay, even though that exact result was already hydrated on every prior replay.

For a sequential workflow of N steps, replay K hydrates K results, so the aggregate cost across a single invocation is O(N²) decrypt+parse operations.

This PR adds a per-run memoization cache so a completed step's hydrated result is returned in O(1) on subsequent replays within the same invocation, making the aggregate cost O(N).

Before / after

  • Before: replay 1 hydrates 1 result, replay 2 hydrates 2, …, replay K hydrates K → Σ = O(N²) decrypt+parse over a sequential run.
  • After: each completed step's result is hydrated once and memoized for the rest of the invocation → O(N) total. Replay K hydrates only the one newly-completed step; the K−1 prior results are cache hits.

Cache scope & keying

  • Lifetime / scope: owned by the inline loop in runtime.ts (created once per run invocation, alongside cachedEvents), threaded into runWorkflow(..., stepHydrationCache?) and stored on WorkflowOrchestratorContext.stepHydrationCache. A fresh context is created each loop iteration, so the cache deliberately lives outside the per-iteration context to survive across iterations of the same run. It is never shared across unrelated runs or process-level invocations.
  • Keying: by the persisted step_completed event's eventId — a stable, world-assigned id. The same event carries the same immutable serialized bytes across every replay, so a hit is guaranteed to correspond to identical input.
  • Optional / backward compatible: the parameter and context field are optional. Callers/harnesses that omit them (and the many runWorkflow(...) unit tests) degrade to re-hydrating every replay — identical to previous behavior.

Memory characteristic

A cached entry holds the decrypted/devalue-parsed plaintext of a step result, retained for the rest of the invocation on top of the serialized bytes already held in cachedEvents — so for large primitive results it roughly doubles peak retained memory for those results during the run. This residual is:

  • Scoped to one invocation — the Map is created per run and GC'd when the invocation returns; nothing accumulates across runs or process-level invocations (a much weaker concern than a process-wide cache, where the dominant residency — the full event log in cachedEvents — already exists for the same lifetime).
  • Bounded by the primitive-returning completed-step count — at most one small entry per such step.
  • Byte-bounded. Most primitives (numbers, booleans, null/undefined, symbols, short ids/strings) are tiny. The only primitive that can be large is a string (or a pathologically long bigint), so a string/bigint result longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) is not memoized — it falls through to the existing per-replay re-hydrate path. Large payloads are cheap to re-hydrate relative to their footprint, so this caps the worst case at negligible cost. The cap only ever reduces what is cached, so deterministic replay is unaffected.

Ordering safety analysis

The cache lookup replaces only the await hydrateStepReturnValue(...) call inside the existing ctx.promiseQueue.then(async () => { ... }) slot. Everything else is byte-for-byte unchanged:

  • ctx.pendingDeliveries++ / -- accounting is untouched.
  • The hydrate (or cache hit) still happens inside the same serial promiseQueue slot, at the same log position, and still resolves via the same resolve(...).
  • The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the exact position in the ordered delivery chain a re-hydrate would have.

So delivery order, pendingDeliveries-gated suspensions, the pendingDeliveryBarriers / awaitEarlierDeliveries machinery, and Promise.race/Promise.all replay determinism are all unaffected. Hook, wait, and abort hydration paths are intentionally not cached (they're the ordering-sensitive paths and not the O(N²) hotspot).

Identity / immutability safety

hydrateStepReturnValue (devalue.parse) returns a fresh object graph on every call, and each replay iteration runs in a fresh workflow VM. Today the workflow therefore receives a brand-new value on every replay. If we cached and returned the same object reference across replays, workflow code that mutates a step result (const r = await step(); r.count++) would observe a previous replay's mutation on the next replay — a non-deterministic divergence. (structuredClone on each hit is both lossy — revivers reconstruct stream handles, step-function proxies, Request/Response, and AbortController/AbortSignal class instances — and still O(size).)

Decision: only primitives are memoized (string, number, boolean, bigint, symbol, null, undefined). Primitives are immutable and compared by value, so sharing the reference is provably indistinguishable from re-parsing. Any non-primitive result falls through to a full re-hydrate every replay, preserving current behavior exactly. Errors are never cached, so a rejected hydrate re-attempts on the next replay (no parked rejected promise). This trades away the optimization in the object-returning case to keep deterministic replay airtight — correctness over speed.

What I verified

  • Unit:step-hydration-cache.test.ts (14 tests: primitive detection, memoization, non-primitive eviction/fresh-object, falsy primitives, keying, error non-caching, no-cache passthrough, plus the size-bound — at-bound string is a hit, oversized string/bigint are not memoized and cache.size stays 0) and step-hydration-memoization.test.ts (3 tests through the real createUseStep consumer: hydrate-skipped-on-replay-2 via spy, event-log ordering preserved on cache hits, fresh object per replay for object results).
  • Full core suite:cd packages/core && pnpm test1253 passed / 56 files, including async-deserialization-ordering.test.ts, workflow.test.ts (79 tests), runtime.test.ts, hook-sleep-interaction, abort-consistency. No regressions.
  • Build / format / typecheck:pnpm build (full repo, 27/27), @workflow/core build + tsc --noEmit clean; Biome format applied; new files Biome-clean (the only lint errors were import-ordering, auto-fixed; remaining warnings are pre-existing noExcessiveCognitiveComplexity on functions I only edited).
  • E2E (local nextjs-turbopack dev server, the determinism-sensitive subset): promiseAllWorkflow, promiseRaceWorkflow, promiseAnyWorkflow, sleepWinsRaceWorkflow, stepWinsRaceWorkflow, promiseRaceStressTestWorkflow, hookWorkflow, webhookWorkflow, parallel-steps-then-webhook replay race, sleepingWorkflow, parallelSleepWorkflow, retry/error/catchability suite, fetchWorkflowall passed.

Risks / deferred

  • Only primitive step results are accelerated; object-returning steps still re-hydrate each replay (intentional, for determinism). A future safe extension could deep-freeze + share frozen object graphs, but that needs care around reviver-produced special objects and is out of scope here.
  • Large (>4 KiB) string/bigint results are intentionally not memoized to bound peak retained memory (see Memory characteristic); they re-hydrate each replay.
  • Hook/wait/abort hydration paths are uncached by design.

🤖 Generated with Claude Code

The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CopilotAI review requested due to automatic review settings June 17, 2026 01:47
@pranaygp
pranaygp requested a review from a team as a code ownerJune 17, 2026 01:47

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@changeset-bot

changeset-botBot commented Jun 17, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3ca022f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
workflowPatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production144112301672
✅ 💻 Local Development190902192128
✅ 📦 Local Production190902192128
❌ 🐘 Local Postgres189412332128
✅ 🪟 Windows15200152
✅ 📋 Other88501791064
Total8190210809272

❌ Failed Tests

▲ Vercel Production (1 failed)

nitro (1 failed):

  • startFromWorkflow - calling start() directly inside a workflow function with hook communication | wrun_01KVCQ35C5K211Z7CFGRHG4655 | 🔍 observability
🐘 Local Postgres (1 failed)

nextjs-turbopack-stable-lazy-discovery-enabled (1 failed):

  • wellKnownAgentWorkflow (.well-known/agent) | wrun_01KVCPM9CVA5BNHKTABQFTWTWF

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro125027
✅ example125027
✅ express125027
✅ fastify125027
✅ hono125027
✅ nextjs-turbopack14903
✅ nextjs-webpack14903
❌ nitro124127
✅ nuxt125027
✅ sveltekit14408
✅ vite125027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
❌ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable126026
✅ express-stable126026
✅ fastify-stable126026
✅ hono-stable126026
✅ nextjs-turbopack-canary132020
✅ nextjs-turbopack-stable-lazy-discovery-disabled15101
❌ nextjs-turbopack-stable-lazy-discovery-enabled15011
✅ nextjs-webpack-canary132020
✅ nextjs-webpack-stable-lazy-discovery-disabled15101
✅ nextjs-webpack-stable-lazy-discovery-enabled15101
✅ nitro-stable126026
✅ nuxt-stable126026
✅ sveltekit-stable14507
✅ vite-stable126026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15200
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable127025
✅ e2e-local-dev-tanstack-start-127025
✅ e2e-local-postgres-nest-stable126026
✅ e2e-local-postgres-tanstack-start-126026
✅ e2e-local-prod-nest-stable127025
✅ e2e-local-prod-tanstack-start-127025
✅ e2e-vercel-prod-tanstack-start125027

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: failure
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.042s (-2.5%)1.006s (~)0.963s101.00x
💻 LocalNitro0.045s (+10.1% 🔺)1.006s (~)0.962s101.06x
💻 LocalNext.js (Turbopack)0.062s (-5.5% 🟢)1.007s (~)0.944s101.47x
🐘 PostgresExpress0.067s (-8.0% 🟢)1.013s (~)0.946s101.58x
🐘 PostgresNext.js (Turbopack)0.070s (-0.9%)1.013s (~)0.943s101.65x
🐘 PostgresNitro0.073s (+15.6% 🔺)1.013s (~)0.940s101.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express0.271s (-28.1% 🟢)2.235s (-2.6%)1.964s101.00x
▲ VercelNitro0.319s (-14.9% 🟢)2.203s (-2.0%)1.883s101.18x
▲ VercelNext.js (Turbopack)0.351s (+20.6% 🔺)2.531s (+33.8% 🔺)2.180s101.29x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.094s (-0.7%)2.006s (~)0.912s101.00x
💻 LocalNitro1.095s (+0.9%)2.007s (~)0.912s101.00x
🐘 PostgresNitro1.108s (-1.1%)2.009s (~)0.901s101.01x
🐘 PostgresExpress1.110s (~)2.008s (~)0.898s101.02x
💻 LocalNext.js (Turbopack)1.132s (-1.4%)2.007s (~)0.875s101.03x
🐘 PostgresNext.js (Turbopack)1.141s (~)2.009s (~)0.868s101.04x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.723s (-11.0% 🟢)3.716s (+1.1%)1.993s101.00x
▲ VercelNext.js (Turbopack)1.761s (~)3.708s (~)1.947s101.02x
▲ VercelNitro1.788s (-22.8% 🟢)3.409s (-14.6% 🟢)1.621s101.04x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro10.527s (~)11.022s (~)0.495s31.00x
💻 LocalExpress10.566s (~)11.023s (~)0.457s31.00x
🐘 PostgresExpress10.585s (~)11.023s (~)0.438s31.01x
🐘 PostgresNitro10.598s (+0.6%)11.022s (~)0.424s31.01x
💻 LocalNext.js (Turbopack)10.812s (~)11.022s (~)0.210s31.03x
🐘 PostgresNext.js (Turbopack)10.829s (~)11.017s (~)0.189s31.03x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express13.691s (-24.2% 🟢)15.347s (-23.1% 🟢)1.656s21.00x
▲ VercelNitro13.728s (-27.6% 🟢)15.441s (-27.5% 🟢)1.713s21.00x
▲ VercelNext.js (Turbopack)14.688s (+2.6%)16.926s (+3.9%)2.238s21.07x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro13.690s (~)14.027s (~)0.337s51.00x
🐘 PostgresExpress13.756s (-0.7%)14.019s (~)0.263s51.00x
💻 LocalExpress13.798s (~)14.028s (~)0.231s51.01x
🐘 PostgresNitro13.832s (~)14.023s (~)0.191s51.01x
💻 LocalNext.js (Turbopack)14.383s (~)15.030s (~)0.647s41.05x
🐘 PostgresNext.js (Turbopack)14.396s (~)15.017s (~)0.621s41.05x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express23.028s (-4.9%)25.270s (-3.5%)2.242s31.00x
▲ VercelNext.js (Turbopack)24.609s (-30.5% 🟢)26.392s (-28.3% 🟢)1.783s31.07x
▲ VercelNitro24.684s (-20.8% 🟢)26.313s (-19.2% 🟢)1.629s31.07x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro12.373s (+2.5%)13.027s (+1.1%)0.654s71.00x
💻 LocalExpress12.444s (-0.8%)13.024s (~)0.580s71.01x
🐘 PostgresExpress12.512s (+0.6%)13.016s (~)0.503s71.01x
🐘 PostgresNitro12.712s (+1.6%)13.021s (~)0.310s71.03x
💻 LocalNext.js (Turbopack)13.662s (~)14.027s (~)0.365s71.10x
🐘 PostgresNext.js (Turbopack)13.897s (~)14.308s (+1.0%)0.411s71.12x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro27.276s (-29.9% 🟢)29.146s (-28.8% 🟢)1.870s41.00x
▲ VercelNext.js (Turbopack)27.460s (-22.0% 🟢)29.404s (-20.4% 🟢)1.944s41.01x
▲ VercelExpress28.389s (-19.0% 🟢)30.612s (-17.0% 🟢)2.223s31.04x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.204s (-0.8%)2.007s (~)0.803s151.00x
💻 LocalNitro1.222s (+4.5%)2.006s (~)0.784s151.01x
💻 LocalExpress1.231s (+5.6% 🔺)2.007s (~)0.776s151.02x
🐘 PostgresNitro1.251s (+4.4%)2.009s (~)0.757s151.04x
🐘 PostgresNext.js (Turbopack)1.256s (-2.3%)2.007s (~)0.751s151.04x
💻 LocalNext.js (Turbopack)1.405s (+8.4% 🔺)2.006s (~)0.602s151.17x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.571s (+15.9% 🔺)4.329s (+11.4% 🔺)1.757s81.00x
▲ VercelNitro3.123s (+7.3% 🔺)4.469s (+1.4%)1.346s71.21x
▲ VercelNext.js (Turbopack)3.608s (+35.3% 🔺)4.968s (+21.2% 🔺)1.360s71.40x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.309s (-8.1% 🟢)2.008s (-16.1% 🟢)0.700s151.00x
🐘 PostgresNitro1.332s (-2.4%)2.074s (-17.3% 🟢)0.742s151.02x
🐘 PostgresNext.js (Turbopack)1.452s (-11.3% 🟢)2.075s (-13.3% 🟢)0.623s151.11x
💻 LocalExpress1.987s (+22.2% 🔺)2.592s (+29.2% 🔺)0.604s121.52x
💻 LocalNitro2.051s (+32.2% 🔺)2.507s (+24.7% 🔺)0.456s121.57x
💻 LocalNext.js (Turbopack)2.359s (+23.5% 🔺)3.008s (+31.2% 🔺)0.649s101.80x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.443s (-13.1% 🟢)5.587s (-1.5%)2.143s61.00x
▲ VercelNitro3.655s (-9.7% 🟢)5.233s (-8.2% 🟢)1.577s61.06x
▲ VercelNext.js (Turbopack)4.245s (-5.5% 🟢)5.891s (-9.1% 🟢)1.646s61.23x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.468s (-16.7% 🟢)3.886s (~)2.417s81.00x
🐘 PostgresNitro1.470s (-7.8% 🟢)3.885s (-3.1%)2.415s81.00x
🐘 PostgresNext.js (Turbopack)2.642s (-16.3% 🟢)3.456s (-19.7% 🟢)0.814s91.80x
💻 LocalNitro4.340s (+26.8% 🔺)5.013s (+25.0% 🔺)0.673s62.96x
💻 LocalExpress5.253s (+20.2% 🔺)5.679s (+16.7% 🔺)0.425s63.58x
💻 LocalNext.js (Turbopack)6.408s (+11.0% 🔺)7.019s (+12.9% 🔺)0.611s54.36x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro4.491s (-26.7% 🟢)6.303s (-24.4% 🟢)1.812s51.00x
▲ VercelNext.js (Turbopack)4.524s (-19.6% 🟢)6.497s (-15.8% 🟢)1.973s51.01x
▲ VercelExpress4.646s (-15.6% 🟢)7.023s (-11.4% 🟢)2.378s51.03x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.202s (-1.3%)2.007s (~)0.805s151.00x
🐘 PostgresNitro1.214s (~)2.008s (~)0.794s151.01x
💻 LocalExpress1.228s (-22.1% 🟢)2.006s (~)0.778s151.02x
💻 LocalNitro1.268s (+5.6% 🔺)2.006s (~)0.738s151.06x
🐘 PostgresNext.js (Turbopack)1.271s (-0.8%)2.008s (~)0.737s151.06x
💻 LocalNext.js (Turbopack)1.449s (+3.6%)2.006s (~)0.557s151.21x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.668s (-25.2% 🟢)4.041s (-23.6% 🟢)1.373s81.00x
▲ VercelNext.js (Turbopack)2.876s (-22.7% 🟢)4.782s (-8.6% 🟢)1.907s71.08x
▲ VercelExpress3.339s (+43.7% 🔺)5.449s (+43.2% 🔺)2.110s61.25x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Nitro1.304s (-9.6% 🟢)2.075s (-13.3% 🟢)0.771s151.00x
🐘 PostgresExpress1.310s (-4.1%)2.007s (-13.3% 🟢)0.697s151.00x
🐘 PostgresNext.js (Turbopack)1.436s (-6.9% 🟢)2.076s (-6.6% 🟢)0.640s151.10x
💻 LocalNitro1.935s (+14.3% 🔺)2.293s (+14.2% 🔺)0.358s141.48x
💻 LocalExpress2.003s (+5.5% 🔺)2.393s (+11.3% 🔺)0.390s131.54x
💻 LocalNext.js (Turbopack)2.369s (+9.3% 🔺)3.009s (~)0.639s101.82x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.899s (+6.3% 🔺)4.683s (+2.5%)1.785s71.00x
▲ VercelNitro3.978s (+42.1% 🔺)5.734s (+37.1% 🔺)1.756s61.37x
▲ VercelNext.js (Turbopack)4.216s (-5.6% 🟢)5.944s (-3.4%)1.728s61.45x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.452s (-19.3% 🟢)4.014s (~)2.561s81.00x
🐘 PostgresNitro1.589s (-12.0% 🟢)3.678s (-14.4% 🟢)2.090s91.09x
🐘 PostgresNext.js (Turbopack)2.136s (-47.5% 🟢)3.454s (-22.3% 🟢)1.318s91.47x
💻 LocalExpress5.580s (+21.6% 🔺)6.014s (+20.0% 🔺)0.434s53.84x
💻 LocalNitro5.616s (+25.0% 🔺)6.017s (+20.1% 🔺)0.401s53.87x
💻 LocalNext.js (Turbopack)6.852s (+18.0% 🔺)7.416s (+15.6% 🔺)0.563s54.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.610s (-57.2% 🟢)5.562s (-45.7% 🟢)1.952s61.00x
▲ VercelNitro4.046s (-70.8% 🟢)5.724s (-63.4% 🟢)1.678s61.12x
▲ VercelNext.js (Turbopack)4.124s (+9.0% 🔺)6.272s (+19.9% 🔺)2.147s51.14x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.605s (+6.0% 🔺)1.041s (+3.5%)0.436s581.00x
💻 LocalExpress0.616s (-1.5%)1.005s (-1.7%)0.389s601.02x
💻 LocalNitro0.629s (+25.8% 🔺)1.039s (+3.1%)0.410s581.04x
🐘 PostgresNitro0.692s (+16.5% 🔺)1.078s (+3.6%)0.386s561.14x
🐘 PostgresNext.js (Turbopack)0.858s (+2.7%)1.041s (+1.7%)0.183s581.42x
💻 LocalNext.js (Turbopack)0.860s (-3.1%)1.005s (-3.3%)0.144s601.42x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.774s (-42.9% 🟢)5.457s (-34.5% 🟢)1.683s111.00x
▲ VercelNext.js (Turbopack)3.815s (-39.5% 🟢)5.390s (-33.8% 🟢)1.575s121.01x
▲ VercelExpress4.488s (-1.3%)6.377s (-1.6%)1.889s101.19x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.375s (+1.4%)2.052s (+1.2%)0.678s441.00x
🐘 PostgresNitro1.466s (+5.6% 🔺)2.030s (~)0.564s451.07x
💻 LocalNitro1.485s (+25.1% 🔺)2.006s (~)0.521s451.08x
💻 LocalExpress1.530s (+2.8%)2.007s (~)0.476s451.11x
🐘 PostgresNext.js (Turbopack)1.989s (+2.5%)2.308s (+11.2% 🔺)0.319s401.45x
💻 LocalNext.js (Turbopack)2.090s (-0.6%)2.944s (-2.1%)0.854s311.52x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express10.297s (-41.1% 🟢)12.369s (-35.5% 🟢)2.072s81.00x
▲ VercelNitro10.774s (-8.8% 🟢)12.396s (-12.5% 🟢)1.621s81.05x
▲ VercelNext.js (Turbopack)11.321s (-16.7% 🟢)13.637s (-11.0% 🟢)2.316s71.10x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express2.654s (-2.1%)3.058s (-1.7%)0.404s401.00x
🐘 PostgresNitro2.903s (+3.7%)3.280s (+3.6%)0.376s371.09x
💻 LocalExpress3.275s (+2.4%)4.010s (~)0.735s301.23x
💻 LocalNitro3.362s (+23.1% 🔺)4.010s (+24.4% 🔺)0.648s301.27x
🐘 PostgresNext.js (Turbopack)3.983s (+2.9%)4.253s (+4.3%)0.270s291.50x
💻 LocalNext.js (Turbopack)4.363s (~)5.010s (~)0.647s241.64x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express18.286s (-29.9% 🟢)20.532s (-26.7% 🟢)2.246s61.00x
▲ VercelNitro18.713s (-28.9% 🟢)20.382s (-27.5% 🟢)1.669s61.02x
▲ VercelNext.js (Turbopack)20.206s (-19.2% 🟢)22.420s (-16.3% 🟢)2.214s61.11x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.232s (~)1.006s (~)0.773s601.00x
🐘 PostgresNitro0.237s (+0.7%)1.006s (~)0.769s601.02x
🐘 PostgresNext.js (Turbopack)0.297s (+1.1%)1.023s (+1.7%)0.726s591.28x
💻 LocalExpress0.400s (-9.2% 🟢)1.005s (~)0.605s601.72x
💻 LocalNitro0.417s (+11.8% 🔺)1.004s (~)0.588s601.79x
💻 LocalNext.js (Turbopack)0.633s (+8.8% 🔺)1.004s (~)0.371s602.73x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.787s (+15.3% 🔺)3.598s (+25.4% 🔺)1.811s181.00x
▲ VercelNitro1.933s (+49.7% 🔺)3.615s (+25.8% 🔺)1.682s171.08x
▲ VercelNext.js (Turbopack)2.387s (+27.6% 🔺)4.344s (+11.8% 🔺)1.956s141.34x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.331s (-6.9% 🟢)1.006s (-3.3%)0.675s901.00x
🐘 PostgresNitro0.356s (+4.1%)1.006s (-1.1%)0.650s901.08x
🐘 PostgresNext.js (Turbopack)0.476s (-11.7% 🟢)1.103s (-1.3%)0.628s831.44x
💻 LocalNitro2.182s (+43.2% 🔺)2.737s (+27.2% 🔺)0.555s336.59x
💻 LocalExpress2.222s (+6.3% 🔺)2.738s (+3.1%)0.517s336.71x
💻 LocalNext.js (Turbopack)2.493s (+7.0% 🔺)3.344s (+8.7% 🔺)0.851s277.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.638s (-9.8% 🟢)4.553s (-5.1% 🟢)1.915s201.00x
▲ VercelNitro2.705s (-1.5%)4.386s (-8.3% 🟢)1.681s211.03x
▲ VercelNext.js (Turbopack)3.092s (+1.6%)4.802s (~)1.710s191.17x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.526s (-7.4% 🟢)1.059s (-13.0% 🟢)0.532s1141.00x
🐘 PostgresNitro0.570s (~)1.078s (-12.5% 🟢)0.508s1121.08x
🐘 PostgresNext.js (Turbopack)1.918s (-26.7% 🟢)2.763s (-20.4% 🟢)0.845s443.64x
💻 LocalNitro9.521s (+50.8% 🔺)10.445s (+54.9% 🔺)0.924s1218.09x
💻 LocalExpress10.198s (+22.8% 🔺)11.029s (+24.2% 🔺)0.831s1219.38x
💻 LocalNext.js (Turbopack)10.284s (-4.4%)11.663s (+0.8%)1.379s1119.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.245s (-38.5% 🟢)5.031s (-28.3% 🟢)1.786s241.00x
▲ VercelExpress3.483s (-36.2% 🟢)5.687s (-19.8% 🟢)2.204s221.07x
▲ VercelNext.js (Turbopack)4.699s (-2.9%)6.708s (+2.7%)2.008s181.45x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.165s (-1.1%)2.000s (~)0.001s (+9.1% 🔺)2.009s (~)0.844s101.00x
🐘 PostgresNitro1.181s (+1.4%)1.995s (~)0.001s (+30.0% 🔺)2.011s (~)0.831s101.01x
💻 LocalNitro1.181s (+5.7% 🔺)2.005s (~)0.010s (-62.9% 🟢)2.017s (-1.0%)0.836s101.01x
💻 LocalExpress1.185s (+3.0%)2.005s (~)0.012s (+19.8% 🔺)2.019s (~)0.835s101.02x
💻 LocalNext.js (Turbopack)1.216s (~)2.003s (~)0.013s (+2.4%)2.020s (~)0.803s101.04x
🐘 PostgresNext.js (Turbopack)1.232s (~)2.002s (~)0.001s (-15.4% 🟢)2.011s (~)0.779s101.06x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.206s (-8.2% 🟢)3.426s (-6.9% 🟢)0.602s (-18.8% 🟢)4.513s (-7.4% 🟢)2.307s101.00x
▲ VercelNext.js (Turbopack)2.304s (-8.6% 🟢)3.523s (-7.8% 🟢)0.772s (-4.3%)4.790s (-5.8% 🟢)2.486s101.04x
▲ VercelNitro2.331s (-11.7% 🟢)3.158s (-17.0% 🟢)1.253s (+57.9% 🔺)4.805s (-4.9%)2.474s101.06x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro1.572s (+9.3% 🔺)2.011s (~)0.012s (-32.0% 🟢)2.025s (~)0.453s301.00x
🐘 PostgresExpress1.577s (~)2.004s (~)0.005s (+5.7% 🔺)2.026s (~)0.449s301.00x
💻 LocalExpress1.583s (+1.0%)2.010s (~)0.014s (+4.9%)2.025s (~)0.442s301.01x
🐘 PostgresNitro1.602s (+1.5%)2.008s (~)0.005s (-5.6% 🟢)2.027s (~)0.425s301.02x
💻 LocalNext.js (Turbopack)1.740s (-0.5%)2.010s (~)0.013s (+1.3%)2.025s (~)0.285s301.11x
🐘 PostgresNext.js (Turbopack)1.892s (+5.9% 🔺)2.011s (~)0.005s (+5.8% 🔺)2.029s (~)0.137s301.20x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Next.js (Turbopack)6.842s (-19.5% 🟢)8.374s (-14.7% 🟢)0.220s (-16.3% 🟢)9.079s (-14.1% 🟢)2.236s71.00x
▲ VercelNitro6.905s (-3.6%)7.768s (-10.3% 🟢)0.455s (+91.9% 🔺)8.766s (-6.9% 🟢)1.861s71.01x
▲ VercelExpress8.068s (+26.4% 🔺)9.373s (+20.1% 🔺)0.307s (+71.5% 🔺)10.412s (+23.1% 🔺)2.345s61.18x

🔍 Observability: Next.js (Turbopack) | Nitro | Express

10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.789s (+3.3%)1.101s (+7.5% 🔺)0.000s (-100.0% 🟢)1.118s (+6.0% 🔺)0.329s541.00x
🐘 PostgresNitro0.795s (+2.9%)1.044s (~)0.000s (-3.5%)1.061s (-3.0%)0.265s571.01x
🐘 PostgresNext.js (Turbopack)0.986s (-3.3%)1.397s (-4.6%)0.000s (-100.0% 🟢)1.407s (-4.4%)0.421s431.25x
💻 LocalExpress1.533s (+12.5% 🔺)2.014s (~)0.000s (+16.7% 🔺)2.016s (~)0.483s301.94x
💻 LocalNitro1.568s (+45.8% 🔺)2.014s (+9.6% 🔺)0.000s (-56.8% 🟢)2.016s (+9.5% 🔺)0.448s301.99x
💻 LocalNext.js (Turbopack)1.916s (+26.7% 🔺)2.193s (+8.9% 🔺)0.000s (+7.1% 🔺)2.196s (+8.9% 🔺)0.280s282.43x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.961s (-21.2% 🟢)4.002s (-25.7% 🟢)0.000s (-21.4% 🟢)4.479s (-23.6% 🟢)1.518s141.00x
▲ VercelExpress3.140s (+3.6%)4.516s (-1.1%)0.000s (NaN%)5.069s (+0.6%)1.929s121.06x
▲ VercelNext.js (Turbopack)3.540s (+2.7%)5.016s (+5.2% 🔺)0.000s (-100.0% 🟢)5.553s (+5.5% 🔺)2.013s111.20x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.691s (+4.7%)2.302s (+11.5% 🔺)0.000s (NaN%)2.316s (+10.9% 🔺)0.625s261.00x
🐘 PostgresNitro1.779s (+14.7% 🔺)2.381s (+11.6% 🔺)0.000s (+11.5% 🔺)2.394s (+11.5% 🔺)0.615s261.05x
🐘 PostgresNext.js (Turbopack)2.251s (+6.6% 🔺)2.651s (+2.5%)0.000s (-100.0% 🟢)2.664s (+2.6%)0.413s231.33x
💻 LocalNitro4.321s (+95.8% 🔺)4.717s (+72.0% 🔺)0.001s (+12.8% 🔺)4.729s (+71.7% 🔺)0.409s132.56x
💻 LocalExpress4.772s (+62.9% 🔺)5.362s (+46.1% 🔺)0.001s (+98.3% 🔺)5.367s (+46.1% 🔺)0.595s122.82x
💻 LocalNext.js (Turbopack)5.632s (+94.2% 🔺)6.226s (+85.4% 🔺)0.000s (-60.0% 🟢)6.235s (+85.4% 🔺)0.602s103.33x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express4.310s (-24.3% 🟢)6.054s (-16.2% 🟢)0.000s (+Infinity% 🔺)6.546s (-14.9% 🟢)2.236s101.00x
▲ VercelNext.js (Turbopack)4.772s (-34.2% 🟢)6.201s (-26.1% 🟢)0.000s (-100.0% 🟢)6.679s (-24.5% 🟢)1.907s101.11x
▲ VercelNitro4.835s (-7.9% 🟢)5.805s (-10.1% 🟢)0.000s (+Infinity% 🔺)6.308s (-8.9% 🟢)1.473s101.12x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalNitro12/21
🐘 PostgresExpress19/21
▲ VercelExpress14/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres17/21
Next.js (Turbopack)🐘 Postgres14/21
Nitro🐘 Postgres13/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run

@pranaygp

Copy link
Copy Markdown
ContributorAuthor

CI failure triage — pre-existing Vercel-prod e2e flake (not a regression)

The two red checks (E2E Vercel Prod Tests (example), E2E Vercel Prod Tests (fastify), which roll up into E2E Required Check) are the shared Vercel-prod timing flake, not caused by this PR. Evidence:

Scope is wrong for a hydration regression. This PR only memoizes primitive step-result hydration. A determinism/stale-value bug there would surface across all worlds — yet every local suite is 100% green:

SuitePassedFailed
▲ Vercel Production14402
💻 Local Development19090
📦 Local Production19090
🐘 Local Postgres18950
🪟 Windows1520

Only 2 failures, only on Vercel Production.

The two failing tests are unrelated to result hydration, and are abort/hook timing races:

  1. exampleAbortController > abortFromStepWorkflow: step abort cancels an in-flight sibling step. The run completed successfully; the assertion failed only because the abort lost a race. From the run diagnostics (wrun_01KV9MZ1Y00N90XFCKFTSPX5N1):

    +2.2s step_completed (longStep) <- sibling finished on its own
    +3.0s hook_received <- abort signal arrived AFTER
    

    The sibling longStep self-completed (2.2s) before the abort hook arrived (3.0s) under Vercel-prod queue/network latency, so there was no in-flight step left to cancel. Step results hydrated fine; this is purely environmental latency.

  2. fastifystartFromWorkflow - calling start() directly inside a workflow function with hook communication. This exact test passed (4128ms) in the example job of this same commit (a56f5c90b) — a textbook cross-run flake.

The same test is red on plain main, without this change. On main run 27704378960 (commit 2acf13cc7):

  • E2E Vercel Prod Tests (tanstack-start)abortFromStepWorkflow: step abort cancels an in-flight sibling step (the identical test that failed here on example)
  • E2E Vercel Prod Tests (nextjs-turbopack)distributedAbortController - reconnect to existing controller

And on main run 27657696161 (cb181392b, the commit this branch is based on): E2E Vercel Prod Tests (fastify)hookWithSleepFinalStepWorkflow. The set of red workbenches rotates run-to-run — the signature of environment flakiness, not a code regression.

Local verification of this branch (rebuilt @workflow/core first): cd packages/core && pnpm test1249 passed / 56 files, 0 failures, including the determinism/ordering replay tests in workflow.test.ts and the new step-hydration-cache.test.ts (10) + step-hydration-memoization.test.ts (3). The memoization tests assert byte-identical delivery ordering on cache hits and that objects re-hydrate fresh each replay.

Re-running the e2e jobs should clear them. No code change is warranted.

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — the O(N²)→O(N) hydration memoization, done with the right safety bias

This is the most safety-sensitive of the four (it's the only one that caches a value across replays), and the design lands on the conservative side of every judgment call, which is exactly right for replay determinism.

The primitives-only decision is the crux and it's correct.hydrateStepReturnValue (devalue.parse) returns a fresh object graph each call, and each replay runs in a fresh VM, so today the workflow gets a brand-new value every replay. Caching and returning the same object reference would let const r = await step(); r.count++ observe a prior replay's mutation — silent divergence. The alternatives are both worse: structuredClone is lossy for reviver-produced specials (stream handles, step-fn proxies, Request/Response, AbortController/Signal) and still O(size). Restricting the cache to primitives (immutable, compared by value) makes "share the reference" provably indistinguishable from re-parsing, and non-primitives fall through to a full re-hydrate every replay — preserving current behavior exactly. Trading the object-case optimization for airtight determinism is the right call.

What I verified in the integration:

  • Surgical wrap: only the await hydrateStepReturnValue(...) call is replaced; pendingDeliveries++/--, the ctx.promiseQueue.then(...) slot, and resolve(...) are byte-for-byte unchanged. The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the identical position in the ordered delivery chain a re-hydrate would have — preserving the pendingDeliveries-gated suspension/barrier/Promise.race determinism.
  • Lifetime: cache is created once per invocation in runtime.ts (outside the per-iteration context), threaded through runWorkflow, never shared across runs. The optional param/context field degrades to re-hydrating-every-replay for harnesses that omit it.
  • Keying by step_completedeventId (stable, world-assigned, same immutable bytes every replay) is sound, and has() rather than get() !== undefined correctly treats a memoized undefined result as a hit.
  • Errors never cached — a rejected hydrate re-attempts next replay, no parked rejected promise.

Built @workflow/core; full suite green (1249) including the new step-hydration-cache.test.ts / step-hydration-memoization.test.ts, async-deserialization-ordering.test.ts, and workflow.test.ts.

One forward-looking note (non-blocking): the object case is left on the table by design. If it's ever revisited, the safe path is deep-freeze + share frozen graphs — but only after handling the reviver-produced specials, so I agree with deferring it. LGTM.

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

// Only memoize values that are safe to return by reference across replays.
// Non-primitives fall through and are re-hydrated fresh on every replay.
if (isMemoizablePrimitive(value)) {
cache.set(eventId, value);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

The per-run stepHydrationCache is never size-bounded or evicted: it grows one entry per primitive-returning completed step and lives for the entire invocation. The cost worth calling out (the PR doesn't) is the new residency — the decrypted/devalue-parsed plaintext of each cached primitive is now held for the whole invocation, on top of the serialized bytes already retained in cachedEvents. For a long sequential workflow whose steps return large strings, that roughly doubles peak retained memory for those results.

The dominant residency (the full event log in cachedEvents) already exists, so this isn't blocking, but a byte-size threshold would be a cheap safeguard — large primitives are exactly the cheap-to-re-hydrate case relative to their footprint, so letting them fall through to the existing re-hydrate path costs little and bounds the worst case. Ideally with a test asserting the bound.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — capped it, with a test. Pushed in a60baad.

What changed (step-hydration-cache.ts):

  • Added MAX_MEMOIZED_PRIMITIVE_LENGTH = 4096 and extended isMemoizablePrimitive so a string/bigint longer than 4 KiB is treated as non-memoizable. Those are the only primitive types that can carry a large payload — number/boolean/null/undefined/symbol are inherently small, so they're never length-checked. Oversized values now fall through to the existing per-replay re-hydrate path, exactly as you suggested: large primitives are cheap to re-hydrate relative to their footprint, so this caps the doubled-residency worst case at negligible cost.
  • Documented the memory characteristic on the cache module: per-invocation lifetime (fresh Map per run in runtime.ts, GC'd when the invocation returns), bounded by the number of primitive-returning completed steps, primitives-only, now byte-bounded.

Tests (step-hydration-cache.test.ts, +4):isMemoizablePrimitive true at the bound / false beyond it (string and bigint), and an end-to-end assertion that an oversized string re-hydrates on every replay and cache.size === 0 (the bound assertion you asked for); plus an at-bound string is a cache hit.

The cap only ever reduces what gets cached, so determinism is untouched — oversized values just take the already-correct re-hydrate path. Full core suite green (1253, incl. the ordering/determinism + memoization suites); biome + tsc clean.

On consistency with #2471 (the sibling scriptCache): noting the distinction since they're bounded for different reasons. #2471's cache is process-wide and monotonic across the whole process — in dev/watch it pins every historical bundle string (hundreds of MB over a session), which is a genuine regression vs. the prior keep-only-latest behavior, hence the Blocking bound there. This cache is per-invocation and freed wholesale when the run returns, so it can never accumulate across runs; the only real cost is the doubled residency for large primitives during one run, which the size cap here now bounds. Different scope, different severity, but both bounded now.

Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-hydration
* origin/main:
perf(core): lazy inline step start (save one world round-trip per step) (#2478)
perf(core): skip per-step events.list via inline event-log delta (#2475)
Version Packages (beta) (#2491)
[world-vercel] Honor hasMore flag from v4 list pagination endpoint (#2486)
Version Packages (beta) (#2451)
Fix Next workflow module specifier root (#2455)
[world-vercel] Send remoteRefBehavior=lazy on v4 metadata-only event listings (#2415)
[swc-plugin] Fix eager discovery for object property steps (#2484)
fix(web-shared): align attributes panel styling (#2483)
[web-shared] Auto-scroll trace viewer on J/K span navigation (#2366)
fix(web): render restarted step segment as solid gray, not running stripes (#2480)
fix(web-shared): use solid gray for queued trace segment (#2474)
Add trace viewer span markers for hooks and attributes (#2452)
test: support Vercel protection bypass secret in e2e headers (#2458)
fix(core): bump payload-compression cutoff to 5.0.0-beta.18 (#2470)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts:
#	packages/core/src/runtime.ts
@pranaygp
pranaygp enabled auto-merge (squash) June 18, 2026 06:24
@pranaygp
pranaygp disabled auto-merge June 22, 2026 20:30
@pranaygp
pranaygp merged commit 66ca0dc into mainJun 22, 2026
118 of 121 checks passed
@pranaygp
pranaygp deleted the pgp/perf-memoize-step-hydration branch June 22, 2026 20:30
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
github-actionsBot added a commit that referenced this pull request Jun 22, 2026
…#2472)
* perf(core): memoize step return value hydration across replays
The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): bound memoized step-hydration cache by primitive size
Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #2563. Merge conflicts were resolved by AI — please review carefully. (backport job run)

pranaygp added a commit that referenced this pull request Jun 22, 2026
…testing
* origin/main:
Version Packages (beta) (#2540)
perf(core): memoize step return value hydration across inline replays (#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (#2412)
Amend lazy discovery removal changeset (#2560)
[docs] Document minimum SDK version for using hook.getConflict (#2423)
Update default CODEOWNERS (#2556)
Optimize and fix the default eager build mode (#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (#2547)" (#2554)
[core] Turbo mode: fast-path the first invocation (#2526)
Remove lazy discovery from workflow/next (#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (#2547)
feat(docs): add eve and AI SDK to product switcher (#2543)
[vitest] Fix local imports failing to load in test step bundles (#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (#2324)
Version Packages (beta) (#2495)
otel(world-vercel): inject trace context on v4 event requests (#2533)
Bump undici to 7.28.0 (#2534)
Default source maps to dev-on / prod-off (#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (#2527)
perf(core): parallel inline steps + optimistic lazy step start (#2516)
pranaygp added a commit to marcopiraccini/workflow that referenced this pull request Jun 22, 2026
* origin/main: (120 commits)
Version Packages (beta) (vercel#2540)
perf(core): memoize step return value hydration across inline replays (vercel#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (vercel#2412)
Amend lazy discovery removal changeset (vercel#2560)
[docs] Document minimum SDK version for using hook.getConflict (vercel#2423)
Update default CODEOWNERS (vercel#2556)
Optimize and fix the default eager build mode (vercel#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)" (vercel#2554)
[core] Turbo mode: fast-path the first invocation (vercel#2526)
Remove lazy discovery from workflow/next (vercel#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)
feat(docs): add eve and AI SDK to product switcher (vercel#2543)
[vitest] Fix local imports failing to load in test step bundles (vercel#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (vercel#2324)
Version Packages (beta) (vercel#2495)
otel(world-vercel): inject trace context on v4 event requests (vercel#2533)
Bump undici to 7.28.0 (vercel#2534)
Default source maps to dev-on / prod-off (vercel#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (vercel#2527)
perf(core): parallel inline steps + optimistic lazy step start (vercel#2516)
...
VaguelySerious pushed a commit that referenced this pull request Jun 22, 2026
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pranaygp@TooTallNate@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

perf(core): memoize step return value hydration across inline replays - #2472

Merged
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration
Jun 22, 2026
Merged

perf(core): memoize step return value hydration across inline replays#2472
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration

Conversation

@pranaygp

@pranaygppranaygp commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

The inline replay loop (runtime.tsrunWorkflow, workflow.ts) re-executes the workflow body and re-consumes the full event log on every iteration. For each already-completed step, the step consumer (step.ts, step_completed path) re-ran hydrateStepReturnValue — AES-GCM decrypt + devalue-parse of the serialized result — on every replay, even though that exact result was already hydrated on every prior replay.

For a sequential workflow of N steps, replay K hydrates K results, so the aggregate cost across a single invocation is O(N²) decrypt+parse operations.

This PR adds a per-run memoization cache so a completed step's hydrated result is returned in O(1) on subsequent replays within the same invocation, making the aggregate cost O(N).

Before / after

  • Before: replay 1 hydrates 1 result, replay 2 hydrates 2, …, replay K hydrates K → Σ = O(N²) decrypt+parse over a sequential run.
  • After: each completed step's result is hydrated once and memoized for the rest of the invocation → O(N) total. Replay K hydrates only the one newly-completed step; the K−1 prior results are cache hits.

Cache scope & keying

  • Lifetime / scope: owned by the inline loop in runtime.ts (created once per run invocation, alongside cachedEvents), threaded into runWorkflow(..., stepHydrationCache?) and stored on WorkflowOrchestratorContext.stepHydrationCache. A fresh context is created each loop iteration, so the cache deliberately lives outside the per-iteration context to survive across iterations of the same run. It is never shared across unrelated runs or process-level invocations.
  • Keying: by the persisted step_completed event's eventId — a stable, world-assigned id. The same event carries the same immutable serialized bytes across every replay, so a hit is guaranteed to correspond to identical input.
  • Optional / backward compatible: the parameter and context field are optional. Callers/harnesses that omit them (and the many runWorkflow(...) unit tests) degrade to re-hydrating every replay — identical to previous behavior.

Memory characteristic

A cached entry holds the decrypted/devalue-parsed plaintext of a step result, retained for the rest of the invocation on top of the serialized bytes already held in cachedEvents — so for large primitive results it roughly doubles peak retained memory for those results during the run. This residual is:

  • Scoped to one invocation — the Map is created per run and GC'd when the invocation returns; nothing accumulates across runs or process-level invocations (a much weaker concern than a process-wide cache, where the dominant residency — the full event log in cachedEvents — already exists for the same lifetime).
  • Bounded by the primitive-returning completed-step count — at most one small entry per such step.
  • Byte-bounded. Most primitives (numbers, booleans, null/undefined, symbols, short ids/strings) are tiny. The only primitive that can be large is a string (or a pathologically long bigint), so a string/bigint result longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) is not memoized — it falls through to the existing per-replay re-hydrate path. Large payloads are cheap to re-hydrate relative to their footprint, so this caps the worst case at negligible cost. The cap only ever reduces what is cached, so deterministic replay is unaffected.

Ordering safety analysis

The cache lookup replaces only the await hydrateStepReturnValue(...) call inside the existing ctx.promiseQueue.then(async () => { ... }) slot. Everything else is byte-for-byte unchanged:

  • ctx.pendingDeliveries++ / -- accounting is untouched.
  • The hydrate (or cache hit) still happens inside the same serial promiseQueue slot, at the same log position, and still resolves via the same resolve(...).
  • The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the exact position in the ordered delivery chain a re-hydrate would have.

So delivery order, pendingDeliveries-gated suspensions, the pendingDeliveryBarriers / awaitEarlierDeliveries machinery, and Promise.race/Promise.all replay determinism are all unaffected. Hook, wait, and abort hydration paths are intentionally not cached (they're the ordering-sensitive paths and not the O(N²) hotspot).

Identity / immutability safety

hydrateStepReturnValue (devalue.parse) returns a fresh object graph on every call, and each replay iteration runs in a fresh workflow VM. Today the workflow therefore receives a brand-new value on every replay. If we cached and returned the same object reference across replays, workflow code that mutates a step result (const r = await step(); r.count++) would observe a previous replay's mutation on the next replay — a non-deterministic divergence. (structuredClone on each hit is both lossy — revivers reconstruct stream handles, step-function proxies, Request/Response, and AbortController/AbortSignal class instances — and still O(size).)

Decision: only primitives are memoized (string, number, boolean, bigint, symbol, null, undefined). Primitives are immutable and compared by value, so sharing the reference is provably indistinguishable from re-parsing. Any non-primitive result falls through to a full re-hydrate every replay, preserving current behavior exactly. Errors are never cached, so a rejected hydrate re-attempts on the next replay (no parked rejected promise). This trades away the optimization in the object-returning case to keep deterministic replay airtight — correctness over speed.

What I verified

  • Unit:step-hydration-cache.test.ts (14 tests: primitive detection, memoization, non-primitive eviction/fresh-object, falsy primitives, keying, error non-caching, no-cache passthrough, plus the size-bound — at-bound string is a hit, oversized string/bigint are not memoized and cache.size stays 0) and step-hydration-memoization.test.ts (3 tests through the real createUseStep consumer: hydrate-skipped-on-replay-2 via spy, event-log ordering preserved on cache hits, fresh object per replay for object results).
  • Full core suite:cd packages/core && pnpm test1253 passed / 56 files, including async-deserialization-ordering.test.ts, workflow.test.ts (79 tests), runtime.test.ts, hook-sleep-interaction, abort-consistency. No regressions.
  • Build / format / typecheck:pnpm build (full repo, 27/27), @workflow/core build + tsc --noEmit clean; Biome format applied; new files Biome-clean (the only lint errors were import-ordering, auto-fixed; remaining warnings are pre-existing noExcessiveCognitiveComplexity on functions I only edited).
  • E2E (local nextjs-turbopack dev server, the determinism-sensitive subset): promiseAllWorkflow, promiseRaceWorkflow, promiseAnyWorkflow, sleepWinsRaceWorkflow, stepWinsRaceWorkflow, promiseRaceStressTestWorkflow, hookWorkflow, webhookWorkflow, parallel-steps-then-webhook replay race, sleepingWorkflow, parallelSleepWorkflow, retry/error/catchability suite, fetchWorkflowall passed.

Risks / deferred

  • Only primitive step results are accelerated; object-returning steps still re-hydrate each replay (intentional, for determinism). A future safe extension could deep-freeze + share frozen object graphs, but that needs care around reviver-produced special objects and is out of scope here.
  • Large (>4 KiB) string/bigint results are intentionally not memoized to bound peak retained memory (see Memory characteristic); they re-hydrate each replay.
  • Hook/wait/abort hydration paths are uncached by design.

🤖 Generated with Claude Code

The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CopilotAI review requested due to automatic review settings June 17, 2026 01:47
@pranaygp
pranaygp requested a review from a team as a code ownerJune 17, 2026 01:47

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@changeset-bot

changeset-botBot commented Jun 17, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3ca022f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
workflowPatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production144112301672
✅ 💻 Local Development190902192128
✅ 📦 Local Production190902192128
❌ 🐘 Local Postgres189412332128
✅ 🪟 Windows15200152
✅ 📋 Other88501791064
Total8190210809272

❌ Failed Tests

▲ Vercel Production (1 failed)

nitro (1 failed):

  • startFromWorkflow - calling start() directly inside a workflow function with hook communication | wrun_01KVCQ35C5K211Z7CFGRHG4655 | 🔍 observability
🐘 Local Postgres (1 failed)

nextjs-turbopack-stable-lazy-discovery-enabled (1 failed):

  • wellKnownAgentWorkflow (.well-known/agent) | wrun_01KVCPM9CVA5BNHKTABQFTWTWF

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro125027
✅ example125027
✅ express125027
✅ fastify125027
✅ hono125027
✅ nextjs-turbopack14903
✅ nextjs-webpack14903
❌ nitro124127
✅ nuxt125027
✅ sveltekit14408
✅ vite125027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
❌ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable126026
✅ express-stable126026
✅ fastify-stable126026
✅ hono-stable126026
✅ nextjs-turbopack-canary132020
✅ nextjs-turbopack-stable-lazy-discovery-disabled15101
❌ nextjs-turbopack-stable-lazy-discovery-enabled15011
✅ nextjs-webpack-canary132020
✅ nextjs-webpack-stable-lazy-discovery-disabled15101
✅ nextjs-webpack-stable-lazy-discovery-enabled15101
✅ nitro-stable126026
✅ nuxt-stable126026
✅ sveltekit-stable14507
✅ vite-stable126026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15200
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable127025
✅ e2e-local-dev-tanstack-start-127025
✅ e2e-local-postgres-nest-stable126026
✅ e2e-local-postgres-tanstack-start-126026
✅ e2e-local-prod-nest-stable127025
✅ e2e-local-prod-tanstack-start-127025
✅ e2e-vercel-prod-tanstack-start125027

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: failure
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.042s (-2.5%)1.006s (~)0.963s101.00x
💻 LocalNitro0.045s (+10.1% 🔺)1.006s (~)0.962s101.06x
💻 LocalNext.js (Turbopack)0.062s (-5.5% 🟢)1.007s (~)0.944s101.47x
🐘 PostgresExpress0.067s (-8.0% 🟢)1.013s (~)0.946s101.58x
🐘 PostgresNext.js (Turbopack)0.070s (-0.9%)1.013s (~)0.943s101.65x
🐘 PostgresNitro0.073s (+15.6% 🔺)1.013s (~)0.940s101.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express0.271s (-28.1% 🟢)2.235s (-2.6%)1.964s101.00x
▲ VercelNitro0.319s (-14.9% 🟢)2.203s (-2.0%)1.883s101.18x
▲ VercelNext.js (Turbopack)0.351s (+20.6% 🔺)2.531s (+33.8% 🔺)2.180s101.29x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.094s (-0.7%)2.006s (~)0.912s101.00x
💻 LocalNitro1.095s (+0.9%)2.007s (~)0.912s101.00x
🐘 PostgresNitro1.108s (-1.1%)2.009s (~)0.901s101.01x
🐘 PostgresExpress1.110s (~)2.008s (~)0.898s101.02x
💻 LocalNext.js (Turbopack)1.132s (-1.4%)2.007s (~)0.875s101.03x
🐘 PostgresNext.js (Turbopack)1.141s (~)2.009s (~)0.868s101.04x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.723s (-11.0% 🟢)3.716s (+1.1%)1.993s101.00x
▲ VercelNext.js (Turbopack)1.761s (~)3.708s (~)1.947s101.02x
▲ VercelNitro1.788s (-22.8% 🟢)3.409s (-14.6% 🟢)1.621s101.04x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro10.527s (~)11.022s (~)0.495s31.00x
💻 LocalExpress10.566s (~)11.023s (~)0.457s31.00x
🐘 PostgresExpress10.585s (~)11.023s (~)0.438s31.01x
🐘 PostgresNitro10.598s (+0.6%)11.022s (~)0.424s31.01x
💻 LocalNext.js (Turbopack)10.812s (~)11.022s (~)0.210s31.03x
🐘 PostgresNext.js (Turbopack)10.829s (~)11.017s (~)0.189s31.03x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express13.691s (-24.2% 🟢)15.347s (-23.1% 🟢)1.656s21.00x
▲ VercelNitro13.728s (-27.6% 🟢)15.441s (-27.5% 🟢)1.713s21.00x
▲ VercelNext.js (Turbopack)14.688s (+2.6%)16.926s (+3.9%)2.238s21.07x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro13.690s (~)14.027s (~)0.337s51.00x
🐘 PostgresExpress13.756s (-0.7%)14.019s (~)0.263s51.00x
💻 LocalExpress13.798s (~)14.028s (~)0.231s51.01x
🐘 PostgresNitro13.832s (~)14.023s (~)0.191s51.01x
💻 LocalNext.js (Turbopack)14.383s (~)15.030s (~)0.647s41.05x
🐘 PostgresNext.js (Turbopack)14.396s (~)15.017s (~)0.621s41.05x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express23.028s (-4.9%)25.270s (-3.5%)2.242s31.00x
▲ VercelNext.js (Turbopack)24.609s (-30.5% 🟢)26.392s (-28.3% 🟢)1.783s31.07x
▲ VercelNitro24.684s (-20.8% 🟢)26.313s (-19.2% 🟢)1.629s31.07x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro12.373s (+2.5%)13.027s (+1.1%)0.654s71.00x
💻 LocalExpress12.444s (-0.8%)13.024s (~)0.580s71.01x
🐘 PostgresExpress12.512s (+0.6%)13.016s (~)0.503s71.01x
🐘 PostgresNitro12.712s (+1.6%)13.021s (~)0.310s71.03x
💻 LocalNext.js (Turbopack)13.662s (~)14.027s (~)0.365s71.10x
🐘 PostgresNext.js (Turbopack)13.897s (~)14.308s (+1.0%)0.411s71.12x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro27.276s (-29.9% 🟢)29.146s (-28.8% 🟢)1.870s41.00x
▲ VercelNext.js (Turbopack)27.460s (-22.0% 🟢)29.404s (-20.4% 🟢)1.944s41.01x
▲ VercelExpress28.389s (-19.0% 🟢)30.612s (-17.0% 🟢)2.223s31.04x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.204s (-0.8%)2.007s (~)0.803s151.00x
💻 LocalNitro1.222s (+4.5%)2.006s (~)0.784s151.01x
💻 LocalExpress1.231s (+5.6% 🔺)2.007s (~)0.776s151.02x
🐘 PostgresNitro1.251s (+4.4%)2.009s (~)0.757s151.04x
🐘 PostgresNext.js (Turbopack)1.256s (-2.3%)2.007s (~)0.751s151.04x
💻 LocalNext.js (Turbopack)1.405s (+8.4% 🔺)2.006s (~)0.602s151.17x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.571s (+15.9% 🔺)4.329s (+11.4% 🔺)1.757s81.00x
▲ VercelNitro3.123s (+7.3% 🔺)4.469s (+1.4%)1.346s71.21x
▲ VercelNext.js (Turbopack)3.608s (+35.3% 🔺)4.968s (+21.2% 🔺)1.360s71.40x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.309s (-8.1% 🟢)2.008s (-16.1% 🟢)0.700s151.00x
🐘 PostgresNitro1.332s (-2.4%)2.074s (-17.3% 🟢)0.742s151.02x
🐘 PostgresNext.js (Turbopack)1.452s (-11.3% 🟢)2.075s (-13.3% 🟢)0.623s151.11x
💻 LocalExpress1.987s (+22.2% 🔺)2.592s (+29.2% 🔺)0.604s121.52x
💻 LocalNitro2.051s (+32.2% 🔺)2.507s (+24.7% 🔺)0.456s121.57x
💻 LocalNext.js (Turbopack)2.359s (+23.5% 🔺)3.008s (+31.2% 🔺)0.649s101.80x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.443s (-13.1% 🟢)5.587s (-1.5%)2.143s61.00x
▲ VercelNitro3.655s (-9.7% 🟢)5.233s (-8.2% 🟢)1.577s61.06x
▲ VercelNext.js (Turbopack)4.245s (-5.5% 🟢)5.891s (-9.1% 🟢)1.646s61.23x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.468s (-16.7% 🟢)3.886s (~)2.417s81.00x
🐘 PostgresNitro1.470s (-7.8% 🟢)3.885s (-3.1%)2.415s81.00x
🐘 PostgresNext.js (Turbopack)2.642s (-16.3% 🟢)3.456s (-19.7% 🟢)0.814s91.80x
💻 LocalNitro4.340s (+26.8% 🔺)5.013s (+25.0% 🔺)0.673s62.96x
💻 LocalExpress5.253s (+20.2% 🔺)5.679s (+16.7% 🔺)0.425s63.58x
💻 LocalNext.js (Turbopack)6.408s (+11.0% 🔺)7.019s (+12.9% 🔺)0.611s54.36x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro4.491s (-26.7% 🟢)6.303s (-24.4% 🟢)1.812s51.00x
▲ VercelNext.js (Turbopack)4.524s (-19.6% 🟢)6.497s (-15.8% 🟢)1.973s51.01x
▲ VercelExpress4.646s (-15.6% 🟢)7.023s (-11.4% 🟢)2.378s51.03x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.202s (-1.3%)2.007s (~)0.805s151.00x
🐘 PostgresNitro1.214s (~)2.008s (~)0.794s151.01x
💻 LocalExpress1.228s (-22.1% 🟢)2.006s (~)0.778s151.02x
💻 LocalNitro1.268s (+5.6% 🔺)2.006s (~)0.738s151.06x
🐘 PostgresNext.js (Turbopack)1.271s (-0.8%)2.008s (~)0.737s151.06x
💻 LocalNext.js (Turbopack)1.449s (+3.6%)2.006s (~)0.557s151.21x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.668s (-25.2% 🟢)4.041s (-23.6% 🟢)1.373s81.00x
▲ VercelNext.js (Turbopack)2.876s (-22.7% 🟢)4.782s (-8.6% 🟢)1.907s71.08x
▲ VercelExpress3.339s (+43.7% 🔺)5.449s (+43.2% 🔺)2.110s61.25x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Nitro1.304s (-9.6% 🟢)2.075s (-13.3% 🟢)0.771s151.00x
🐘 PostgresExpress1.310s (-4.1%)2.007s (-13.3% 🟢)0.697s151.00x
🐘 PostgresNext.js (Turbopack)1.436s (-6.9% 🟢)2.076s (-6.6% 🟢)0.640s151.10x
💻 LocalNitro1.935s (+14.3% 🔺)2.293s (+14.2% 🔺)0.358s141.48x
💻 LocalExpress2.003s (+5.5% 🔺)2.393s (+11.3% 🔺)0.390s131.54x
💻 LocalNext.js (Turbopack)2.369s (+9.3% 🔺)3.009s (~)0.639s101.82x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.899s (+6.3% 🔺)4.683s (+2.5%)1.785s71.00x
▲ VercelNitro3.978s (+42.1% 🔺)5.734s (+37.1% 🔺)1.756s61.37x
▲ VercelNext.js (Turbopack)4.216s (-5.6% 🟢)5.944s (-3.4%)1.728s61.45x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.452s (-19.3% 🟢)4.014s (~)2.561s81.00x
🐘 PostgresNitro1.589s (-12.0% 🟢)3.678s (-14.4% 🟢)2.090s91.09x
🐘 PostgresNext.js (Turbopack)2.136s (-47.5% 🟢)3.454s (-22.3% 🟢)1.318s91.47x
💻 LocalExpress5.580s (+21.6% 🔺)6.014s (+20.0% 🔺)0.434s53.84x
💻 LocalNitro5.616s (+25.0% 🔺)6.017s (+20.1% 🔺)0.401s53.87x
💻 LocalNext.js (Turbopack)6.852s (+18.0% 🔺)7.416s (+15.6% 🔺)0.563s54.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.610s (-57.2% 🟢)5.562s (-45.7% 🟢)1.952s61.00x
▲ VercelNitro4.046s (-70.8% 🟢)5.724s (-63.4% 🟢)1.678s61.12x
▲ VercelNext.js (Turbopack)4.124s (+9.0% 🔺)6.272s (+19.9% 🔺)2.147s51.14x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.605s (+6.0% 🔺)1.041s (+3.5%)0.436s581.00x
💻 LocalExpress0.616s (-1.5%)1.005s (-1.7%)0.389s601.02x
💻 LocalNitro0.629s (+25.8% 🔺)1.039s (+3.1%)0.410s581.04x
🐘 PostgresNitro0.692s (+16.5% 🔺)1.078s (+3.6%)0.386s561.14x
🐘 PostgresNext.js (Turbopack)0.858s (+2.7%)1.041s (+1.7%)0.183s581.42x
💻 LocalNext.js (Turbopack)0.860s (-3.1%)1.005s (-3.3%)0.144s601.42x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.774s (-42.9% 🟢)5.457s (-34.5% 🟢)1.683s111.00x
▲ VercelNext.js (Turbopack)3.815s (-39.5% 🟢)5.390s (-33.8% 🟢)1.575s121.01x
▲ VercelExpress4.488s (-1.3%)6.377s (-1.6%)1.889s101.19x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.375s (+1.4%)2.052s (+1.2%)0.678s441.00x
🐘 PostgresNitro1.466s (+5.6% 🔺)2.030s (~)0.564s451.07x
💻 LocalNitro1.485s (+25.1% 🔺)2.006s (~)0.521s451.08x
💻 LocalExpress1.530s (+2.8%)2.007s (~)0.476s451.11x
🐘 PostgresNext.js (Turbopack)1.989s (+2.5%)2.308s (+11.2% 🔺)0.319s401.45x
💻 LocalNext.js (Turbopack)2.090s (-0.6%)2.944s (-2.1%)0.854s311.52x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express10.297s (-41.1% 🟢)12.369s (-35.5% 🟢)2.072s81.00x
▲ VercelNitro10.774s (-8.8% 🟢)12.396s (-12.5% 🟢)1.621s81.05x
▲ VercelNext.js (Turbopack)11.321s (-16.7% 🟢)13.637s (-11.0% 🟢)2.316s71.10x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express2.654s (-2.1%)3.058s (-1.7%)0.404s401.00x
🐘 PostgresNitro2.903s (+3.7%)3.280s (+3.6%)0.376s371.09x
💻 LocalExpress3.275s (+2.4%)4.010s (~)0.735s301.23x
💻 LocalNitro3.362s (+23.1% 🔺)4.010s (+24.4% 🔺)0.648s301.27x
🐘 PostgresNext.js (Turbopack)3.983s (+2.9%)4.253s (+4.3%)0.270s291.50x
💻 LocalNext.js (Turbopack)4.363s (~)5.010s (~)0.647s241.64x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express18.286s (-29.9% 🟢)20.532s (-26.7% 🟢)2.246s61.00x
▲ VercelNitro18.713s (-28.9% 🟢)20.382s (-27.5% 🟢)1.669s61.02x
▲ VercelNext.js (Turbopack)20.206s (-19.2% 🟢)22.420s (-16.3% 🟢)2.214s61.11x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.232s (~)1.006s (~)0.773s601.00x
🐘 PostgresNitro0.237s (+0.7%)1.006s (~)0.769s601.02x
🐘 PostgresNext.js (Turbopack)0.297s (+1.1%)1.023s (+1.7%)0.726s591.28x
💻 LocalExpress0.400s (-9.2% 🟢)1.005s (~)0.605s601.72x
💻 LocalNitro0.417s (+11.8% 🔺)1.004s (~)0.588s601.79x
💻 LocalNext.js (Turbopack)0.633s (+8.8% 🔺)1.004s (~)0.371s602.73x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.787s (+15.3% 🔺)3.598s (+25.4% 🔺)1.811s181.00x
▲ VercelNitro1.933s (+49.7% 🔺)3.615s (+25.8% 🔺)1.682s171.08x
▲ VercelNext.js (Turbopack)2.387s (+27.6% 🔺)4.344s (+11.8% 🔺)1.956s141.34x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.331s (-6.9% 🟢)1.006s (-3.3%)0.675s901.00x
🐘 PostgresNitro0.356s (+4.1%)1.006s (-1.1%)0.650s901.08x
🐘 PostgresNext.js (Turbopack)0.476s (-11.7% 🟢)1.103s (-1.3%)0.628s831.44x
💻 LocalNitro2.182s (+43.2% 🔺)2.737s (+27.2% 🔺)0.555s336.59x
💻 LocalExpress2.222s (+6.3% 🔺)2.738s (+3.1%)0.517s336.71x
💻 LocalNext.js (Turbopack)2.493s (+7.0% 🔺)3.344s (+8.7% 🔺)0.851s277.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.638s (-9.8% 🟢)4.553s (-5.1% 🟢)1.915s201.00x
▲ VercelNitro2.705s (-1.5%)4.386s (-8.3% 🟢)1.681s211.03x
▲ VercelNext.js (Turbopack)3.092s (+1.6%)4.802s (~)1.710s191.17x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.526s (-7.4% 🟢)1.059s (-13.0% 🟢)0.532s1141.00x
🐘 PostgresNitro0.570s (~)1.078s (-12.5% 🟢)0.508s1121.08x
🐘 PostgresNext.js (Turbopack)1.918s (-26.7% 🟢)2.763s (-20.4% 🟢)0.845s443.64x
💻 LocalNitro9.521s (+50.8% 🔺)10.445s (+54.9% 🔺)0.924s1218.09x
💻 LocalExpress10.198s (+22.8% 🔺)11.029s (+24.2% 🔺)0.831s1219.38x
💻 LocalNext.js (Turbopack)10.284s (-4.4%)11.663s (+0.8%)1.379s1119.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.245s (-38.5% 🟢)5.031s (-28.3% 🟢)1.786s241.00x
▲ VercelExpress3.483s (-36.2% 🟢)5.687s (-19.8% 🟢)2.204s221.07x
▲ VercelNext.js (Turbopack)4.699s (-2.9%)6.708s (+2.7%)2.008s181.45x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.165s (-1.1%)2.000s (~)0.001s (+9.1% 🔺)2.009s (~)0.844s101.00x
🐘 PostgresNitro1.181s (+1.4%)1.995s (~)0.001s (+30.0% 🔺)2.011s (~)0.831s101.01x
💻 LocalNitro1.181s (+5.7% 🔺)2.005s (~)0.010s (-62.9% 🟢)2.017s (-1.0%)0.836s101.01x
💻 LocalExpress1.185s (+3.0%)2.005s (~)0.012s (+19.8% 🔺)2.019s (~)0.835s101.02x
💻 LocalNext.js (Turbopack)1.216s (~)2.003s (~)0.013s (+2.4%)2.020s (~)0.803s101.04x
🐘 PostgresNext.js (Turbopack)1.232s (~)2.002s (~)0.001s (-15.4% 🟢)2.011s (~)0.779s101.06x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.206s (-8.2% 🟢)3.426s (-6.9% 🟢)0.602s (-18.8% 🟢)4.513s (-7.4% 🟢)2.307s101.00x
▲ VercelNext.js (Turbopack)2.304s (-8.6% 🟢)3.523s (-7.8% 🟢)0.772s (-4.3%)4.790s (-5.8% 🟢)2.486s101.04x
▲ VercelNitro2.331s (-11.7% 🟢)3.158s (-17.0% 🟢)1.253s (+57.9% 🔺)4.805s (-4.9%)2.474s101.06x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro1.572s (+9.3% 🔺)2.011s (~)0.012s (-32.0% 🟢)2.025s (~)0.453s301.00x
🐘 PostgresExpress1.577s (~)2.004s (~)0.005s (+5.7% 🔺)2.026s (~)0.449s301.00x
💻 LocalExpress1.583s (+1.0%)2.010s (~)0.014s (+4.9%)2.025s (~)0.442s301.01x
🐘 PostgresNitro1.602s (+1.5%)2.008s (~)0.005s (-5.6% 🟢)2.027s (~)0.425s301.02x
💻 LocalNext.js (Turbopack)1.740s (-0.5%)2.010s (~)0.013s (+1.3%)2.025s (~)0.285s301.11x
🐘 PostgresNext.js (Turbopack)1.892s (+5.9% 🔺)2.011s (~)0.005s (+5.8% 🔺)2.029s (~)0.137s301.20x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Next.js (Turbopack)6.842s (-19.5% 🟢)8.374s (-14.7% 🟢)0.220s (-16.3% 🟢)9.079s (-14.1% 🟢)2.236s71.00x
▲ VercelNitro6.905s (-3.6%)7.768s (-10.3% 🟢)0.455s (+91.9% 🔺)8.766s (-6.9% 🟢)1.861s71.01x
▲ VercelExpress8.068s (+26.4% 🔺)9.373s (+20.1% 🔺)0.307s (+71.5% 🔺)10.412s (+23.1% 🔺)2.345s61.18x

🔍 Observability: Next.js (Turbopack) | Nitro | Express

10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.789s (+3.3%)1.101s (+7.5% 🔺)0.000s (-100.0% 🟢)1.118s (+6.0% 🔺)0.329s541.00x
🐘 PostgresNitro0.795s (+2.9%)1.044s (~)0.000s (-3.5%)1.061s (-3.0%)0.265s571.01x
🐘 PostgresNext.js (Turbopack)0.986s (-3.3%)1.397s (-4.6%)0.000s (-100.0% 🟢)1.407s (-4.4%)0.421s431.25x
💻 LocalExpress1.533s (+12.5% 🔺)2.014s (~)0.000s (+16.7% 🔺)2.016s (~)0.483s301.94x
💻 LocalNitro1.568s (+45.8% 🔺)2.014s (+9.6% 🔺)0.000s (-56.8% 🟢)2.016s (+9.5% 🔺)0.448s301.99x
💻 LocalNext.js (Turbopack)1.916s (+26.7% 🔺)2.193s (+8.9% 🔺)0.000s (+7.1% 🔺)2.196s (+8.9% 🔺)0.280s282.43x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.961s (-21.2% 🟢)4.002s (-25.7% 🟢)0.000s (-21.4% 🟢)4.479s (-23.6% 🟢)1.518s141.00x
▲ VercelExpress3.140s (+3.6%)4.516s (-1.1%)0.000s (NaN%)5.069s (+0.6%)1.929s121.06x
▲ VercelNext.js (Turbopack)3.540s (+2.7%)5.016s (+5.2% 🔺)0.000s (-100.0% 🟢)5.553s (+5.5% 🔺)2.013s111.20x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.691s (+4.7%)2.302s (+11.5% 🔺)0.000s (NaN%)2.316s (+10.9% 🔺)0.625s261.00x
🐘 PostgresNitro1.779s (+14.7% 🔺)2.381s (+11.6% 🔺)0.000s (+11.5% 🔺)2.394s (+11.5% 🔺)0.615s261.05x
🐘 PostgresNext.js (Turbopack)2.251s (+6.6% 🔺)2.651s (+2.5%)0.000s (-100.0% 🟢)2.664s (+2.6%)0.413s231.33x
💻 LocalNitro4.321s (+95.8% 🔺)4.717s (+72.0% 🔺)0.001s (+12.8% 🔺)4.729s (+71.7% 🔺)0.409s132.56x
💻 LocalExpress4.772s (+62.9% 🔺)5.362s (+46.1% 🔺)0.001s (+98.3% 🔺)5.367s (+46.1% 🔺)0.595s122.82x
💻 LocalNext.js (Turbopack)5.632s (+94.2% 🔺)6.226s (+85.4% 🔺)0.000s (-60.0% 🟢)6.235s (+85.4% 🔺)0.602s103.33x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express4.310s (-24.3% 🟢)6.054s (-16.2% 🟢)0.000s (+Infinity% 🔺)6.546s (-14.9% 🟢)2.236s101.00x
▲ VercelNext.js (Turbopack)4.772s (-34.2% 🟢)6.201s (-26.1% 🟢)0.000s (-100.0% 🟢)6.679s (-24.5% 🟢)1.907s101.11x
▲ VercelNitro4.835s (-7.9% 🟢)5.805s (-10.1% 🟢)0.000s (+Infinity% 🔺)6.308s (-8.9% 🟢)1.473s101.12x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalNitro12/21
🐘 PostgresExpress19/21
▲ VercelExpress14/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres17/21
Next.js (Turbopack)🐘 Postgres14/21
Nitro🐘 Postgres13/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run

@pranaygp

Copy link
Copy Markdown
ContributorAuthor

CI failure triage — pre-existing Vercel-prod e2e flake (not a regression)

The two red checks (E2E Vercel Prod Tests (example), E2E Vercel Prod Tests (fastify), which roll up into E2E Required Check) are the shared Vercel-prod timing flake, not caused by this PR. Evidence:

Scope is wrong for a hydration regression. This PR only memoizes primitive step-result hydration. A determinism/stale-value bug there would surface across all worlds — yet every local suite is 100% green:

SuitePassedFailed
▲ Vercel Production14402
💻 Local Development19090
📦 Local Production19090
🐘 Local Postgres18950
🪟 Windows1520

Only 2 failures, only on Vercel Production.

The two failing tests are unrelated to result hydration, and are abort/hook timing races:

  1. exampleAbortController > abortFromStepWorkflow: step abort cancels an in-flight sibling step. The run completed successfully; the assertion failed only because the abort lost a race. From the run diagnostics (wrun_01KV9MZ1Y00N90XFCKFTSPX5N1):

    +2.2s step_completed (longStep) <- sibling finished on its own
    +3.0s hook_received <- abort signal arrived AFTER
    

    The sibling longStep self-completed (2.2s) before the abort hook arrived (3.0s) under Vercel-prod queue/network latency, so there was no in-flight step left to cancel. Step results hydrated fine; this is purely environmental latency.

  2. fastifystartFromWorkflow - calling start() directly inside a workflow function with hook communication. This exact test passed (4128ms) in the example job of this same commit (a56f5c90b) — a textbook cross-run flake.

The same test is red on plain main, without this change. On main run 27704378960 (commit 2acf13cc7):

  • E2E Vercel Prod Tests (tanstack-start)abortFromStepWorkflow: step abort cancels an in-flight sibling step (the identical test that failed here on example)
  • E2E Vercel Prod Tests (nextjs-turbopack)distributedAbortController - reconnect to existing controller

And on main run 27657696161 (cb181392b, the commit this branch is based on): E2E Vercel Prod Tests (fastify)hookWithSleepFinalStepWorkflow. The set of red workbenches rotates run-to-run — the signature of environment flakiness, not a code regression.

Local verification of this branch (rebuilt @workflow/core first): cd packages/core && pnpm test1249 passed / 56 files, 0 failures, including the determinism/ordering replay tests in workflow.test.ts and the new step-hydration-cache.test.ts (10) + step-hydration-memoization.test.ts (3). The memoization tests assert byte-identical delivery ordering on cache hits and that objects re-hydrate fresh each replay.

Re-running the e2e jobs should clear them. No code change is warranted.

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — the O(N²)→O(N) hydration memoization, done with the right safety bias

This is the most safety-sensitive of the four (it's the only one that caches a value across replays), and the design lands on the conservative side of every judgment call, which is exactly right for replay determinism.

The primitives-only decision is the crux and it's correct.hydrateStepReturnValue (devalue.parse) returns a fresh object graph each call, and each replay runs in a fresh VM, so today the workflow gets a brand-new value every replay. Caching and returning the same object reference would let const r = await step(); r.count++ observe a prior replay's mutation — silent divergence. The alternatives are both worse: structuredClone is lossy for reviver-produced specials (stream handles, step-fn proxies, Request/Response, AbortController/Signal) and still O(size). Restricting the cache to primitives (immutable, compared by value) makes "share the reference" provably indistinguishable from re-parsing, and non-primitives fall through to a full re-hydrate every replay — preserving current behavior exactly. Trading the object-case optimization for airtight determinism is the right call.

What I verified in the integration:

  • Surgical wrap: only the await hydrateStepReturnValue(...) call is replaced; pendingDeliveries++/--, the ctx.promiseQueue.then(...) slot, and resolve(...) are byte-for-byte unchanged. The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the identical position in the ordered delivery chain a re-hydrate would have — preserving the pendingDeliveries-gated suspension/barrier/Promise.race determinism.
  • Lifetime: cache is created once per invocation in runtime.ts (outside the per-iteration context), threaded through runWorkflow, never shared across runs. The optional param/context field degrades to re-hydrating-every-replay for harnesses that omit it.
  • Keying by step_completedeventId (stable, world-assigned, same immutable bytes every replay) is sound, and has() rather than get() !== undefined correctly treats a memoized undefined result as a hit.
  • Errors never cached — a rejected hydrate re-attempts next replay, no parked rejected promise.

Built @workflow/core; full suite green (1249) including the new step-hydration-cache.test.ts / step-hydration-memoization.test.ts, async-deserialization-ordering.test.ts, and workflow.test.ts.

One forward-looking note (non-blocking): the object case is left on the table by design. If it's ever revisited, the safe path is deep-freeze + share frozen graphs — but only after handling the reviver-produced specials, so I agree with deferring it. LGTM.

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

// Only memoize values that are safe to return by reference across replays.
// Non-primitives fall through and are re-hydrated fresh on every replay.
if (isMemoizablePrimitive(value)) {
cache.set(eventId, value);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

The per-run stepHydrationCache is never size-bounded or evicted: it grows one entry per primitive-returning completed step and lives for the entire invocation. The cost worth calling out (the PR doesn't) is the new residency — the decrypted/devalue-parsed plaintext of each cached primitive is now held for the whole invocation, on top of the serialized bytes already retained in cachedEvents. For a long sequential workflow whose steps return large strings, that roughly doubles peak retained memory for those results.

The dominant residency (the full event log in cachedEvents) already exists, so this isn't blocking, but a byte-size threshold would be a cheap safeguard — large primitives are exactly the cheap-to-re-hydrate case relative to their footprint, so letting them fall through to the existing re-hydrate path costs little and bounds the worst case. Ideally with a test asserting the bound.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — capped it, with a test. Pushed in a60baad.

What changed (step-hydration-cache.ts):

  • Added MAX_MEMOIZED_PRIMITIVE_LENGTH = 4096 and extended isMemoizablePrimitive so a string/bigint longer than 4 KiB is treated as non-memoizable. Those are the only primitive types that can carry a large payload — number/boolean/null/undefined/symbol are inherently small, so they're never length-checked. Oversized values now fall through to the existing per-replay re-hydrate path, exactly as you suggested: large primitives are cheap to re-hydrate relative to their footprint, so this caps the doubled-residency worst case at negligible cost.
  • Documented the memory characteristic on the cache module: per-invocation lifetime (fresh Map per run in runtime.ts, GC'd when the invocation returns), bounded by the number of primitive-returning completed steps, primitives-only, now byte-bounded.

Tests (step-hydration-cache.test.ts, +4):isMemoizablePrimitive true at the bound / false beyond it (string and bigint), and an end-to-end assertion that an oversized string re-hydrates on every replay and cache.size === 0 (the bound assertion you asked for); plus an at-bound string is a cache hit.

The cap only ever reduces what gets cached, so determinism is untouched — oversized values just take the already-correct re-hydrate path. Full core suite green (1253, incl. the ordering/determinism + memoization suites); biome + tsc clean.

On consistency with #2471 (the sibling scriptCache): noting the distinction since they're bounded for different reasons. #2471's cache is process-wide and monotonic across the whole process — in dev/watch it pins every historical bundle string (hundreds of MB over a session), which is a genuine regression vs. the prior keep-only-latest behavior, hence the Blocking bound there. This cache is per-invocation and freed wholesale when the run returns, so it can never accumulate across runs; the only real cost is the doubled residency for large primitives during one run, which the size cap here now bounds. Different scope, different severity, but both bounded now.

Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-hydration
* origin/main:
perf(core): lazy inline step start (save one world round-trip per step) (#2478)
perf(core): skip per-step events.list via inline event-log delta (#2475)
Version Packages (beta) (#2491)
[world-vercel] Honor hasMore flag from v4 list pagination endpoint (#2486)
Version Packages (beta) (#2451)
Fix Next workflow module specifier root (#2455)
[world-vercel] Send remoteRefBehavior=lazy on v4 metadata-only event listings (#2415)
[swc-plugin] Fix eager discovery for object property steps (#2484)
fix(web-shared): align attributes panel styling (#2483)
[web-shared] Auto-scroll trace viewer on J/K span navigation (#2366)
fix(web): render restarted step segment as solid gray, not running stripes (#2480)
fix(web-shared): use solid gray for queued trace segment (#2474)
Add trace viewer span markers for hooks and attributes (#2452)
test: support Vercel protection bypass secret in e2e headers (#2458)
fix(core): bump payload-compression cutoff to 5.0.0-beta.18 (#2470)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts:
#	packages/core/src/runtime.ts
@pranaygp
pranaygp enabled auto-merge (squash) June 18, 2026 06:24
@pranaygp
pranaygp disabled auto-merge June 22, 2026 20:30
@pranaygp
pranaygp merged commit 66ca0dc into mainJun 22, 2026
118 of 121 checks passed
@pranaygp
pranaygp deleted the pgp/perf-memoize-step-hydration branch June 22, 2026 20:30
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
github-actionsBot added a commit that referenced this pull request Jun 22, 2026
…#2472)
* perf(core): memoize step return value hydration across replays
The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): bound memoized step-hydration cache by primitive size
Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #2563. Merge conflicts were resolved by AI — please review carefully. (backport job run)

pranaygp added a commit that referenced this pull request Jun 22, 2026
…testing
* origin/main:
Version Packages (beta) (#2540)
perf(core): memoize step return value hydration across inline replays (#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (#2412)
Amend lazy discovery removal changeset (#2560)
[docs] Document minimum SDK version for using hook.getConflict (#2423)
Update default CODEOWNERS (#2556)
Optimize and fix the default eager build mode (#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (#2547)" (#2554)
[core] Turbo mode: fast-path the first invocation (#2526)
Remove lazy discovery from workflow/next (#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (#2547)
feat(docs): add eve and AI SDK to product switcher (#2543)
[vitest] Fix local imports failing to load in test step bundles (#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (#2324)
Version Packages (beta) (#2495)
otel(world-vercel): inject trace context on v4 event requests (#2533)
Bump undici to 7.28.0 (#2534)
Default source maps to dev-on / prod-off (#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (#2527)
perf(core): parallel inline steps + optimistic lazy step start (#2516)
pranaygp added a commit to marcopiraccini/workflow that referenced this pull request Jun 22, 2026
* origin/main: (120 commits)
Version Packages (beta) (vercel#2540)
perf(core): memoize step return value hydration across inline replays (vercel#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (vercel#2412)
Amend lazy discovery removal changeset (vercel#2560)
[docs] Document minimum SDK version for using hook.getConflict (vercel#2423)
Update default CODEOWNERS (vercel#2556)
Optimize and fix the default eager build mode (vercel#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)" (vercel#2554)
[core] Turbo mode: fast-path the first invocation (vercel#2526)
Remove lazy discovery from workflow/next (vercel#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)
feat(docs): add eve and AI SDK to product switcher (vercel#2543)
[vitest] Fix local imports failing to load in test step bundles (vercel#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (vercel#2324)
Version Packages (beta) (vercel#2495)
otel(world-vercel): inject trace context on v4 event requests (vercel#2533)
Bump undici to 7.28.0 (vercel#2534)
Default source maps to dev-on / prod-off (vercel#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (vercel#2527)
perf(core): parallel inline steps + optimistic lazy step start (vercel#2516)
...
VaguelySerious pushed a commit that referenced this pull request Jun 22, 2026
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pranaygp@TooTallNate@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

perf(core): memoize step return value hydration across inline replays - #2472

Merged
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration
Jun 22, 2026
Merged

perf(core): memoize step return value hydration across inline replays#2472
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration

Conversation

@pranaygp

@pranaygppranaygp commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

The inline replay loop (runtime.tsrunWorkflow, workflow.ts) re-executes the workflow body and re-consumes the full event log on every iteration. For each already-completed step, the step consumer (step.ts, step_completed path) re-ran hydrateStepReturnValue — AES-GCM decrypt + devalue-parse of the serialized result — on every replay, even though that exact result was already hydrated on every prior replay.

For a sequential workflow of N steps, replay K hydrates K results, so the aggregate cost across a single invocation is O(N²) decrypt+parse operations.

This PR adds a per-run memoization cache so a completed step's hydrated result is returned in O(1) on subsequent replays within the same invocation, making the aggregate cost O(N).

Before / after

  • Before: replay 1 hydrates 1 result, replay 2 hydrates 2, …, replay K hydrates K → Σ = O(N²) decrypt+parse over a sequential run.
  • After: each completed step's result is hydrated once and memoized for the rest of the invocation → O(N) total. Replay K hydrates only the one newly-completed step; the K−1 prior results are cache hits.

Cache scope & keying

  • Lifetime / scope: owned by the inline loop in runtime.ts (created once per run invocation, alongside cachedEvents), threaded into runWorkflow(..., stepHydrationCache?) and stored on WorkflowOrchestratorContext.stepHydrationCache. A fresh context is created each loop iteration, so the cache deliberately lives outside the per-iteration context to survive across iterations of the same run. It is never shared across unrelated runs or process-level invocations.
  • Keying: by the persisted step_completed event's eventId — a stable, world-assigned id. The same event carries the same immutable serialized bytes across every replay, so a hit is guaranteed to correspond to identical input.
  • Optional / backward compatible: the parameter and context field are optional. Callers/harnesses that omit them (and the many runWorkflow(...) unit tests) degrade to re-hydrating every replay — identical to previous behavior.

Memory characteristic

A cached entry holds the decrypted/devalue-parsed plaintext of a step result, retained for the rest of the invocation on top of the serialized bytes already held in cachedEvents — so for large primitive results it roughly doubles peak retained memory for those results during the run. This residual is:

  • Scoped to one invocation — the Map is created per run and GC'd when the invocation returns; nothing accumulates across runs or process-level invocations (a much weaker concern than a process-wide cache, where the dominant residency — the full event log in cachedEvents — already exists for the same lifetime).
  • Bounded by the primitive-returning completed-step count — at most one small entry per such step.
  • Byte-bounded. Most primitives (numbers, booleans, null/undefined, symbols, short ids/strings) are tiny. The only primitive that can be large is a string (or a pathologically long bigint), so a string/bigint result longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) is not memoized — it falls through to the existing per-replay re-hydrate path. Large payloads are cheap to re-hydrate relative to their footprint, so this caps the worst case at negligible cost. The cap only ever reduces what is cached, so deterministic replay is unaffected.

Ordering safety analysis

The cache lookup replaces only the await hydrateStepReturnValue(...) call inside the existing ctx.promiseQueue.then(async () => { ... }) slot. Everything else is byte-for-byte unchanged:

  • ctx.pendingDeliveries++ / -- accounting is untouched.
  • The hydrate (or cache hit) still happens inside the same serial promiseQueue slot, at the same log position, and still resolves via the same resolve(...).
  • The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the exact position in the ordered delivery chain a re-hydrate would have.

So delivery order, pendingDeliveries-gated suspensions, the pendingDeliveryBarriers / awaitEarlierDeliveries machinery, and Promise.race/Promise.all replay determinism are all unaffected. Hook, wait, and abort hydration paths are intentionally not cached (they're the ordering-sensitive paths and not the O(N²) hotspot).

Identity / immutability safety

hydrateStepReturnValue (devalue.parse) returns a fresh object graph on every call, and each replay iteration runs in a fresh workflow VM. Today the workflow therefore receives a brand-new value on every replay. If we cached and returned the same object reference across replays, workflow code that mutates a step result (const r = await step(); r.count++) would observe a previous replay's mutation on the next replay — a non-deterministic divergence. (structuredClone on each hit is both lossy — revivers reconstruct stream handles, step-function proxies, Request/Response, and AbortController/AbortSignal class instances — and still O(size).)

Decision: only primitives are memoized (string, number, boolean, bigint, symbol, null, undefined). Primitives are immutable and compared by value, so sharing the reference is provably indistinguishable from re-parsing. Any non-primitive result falls through to a full re-hydrate every replay, preserving current behavior exactly. Errors are never cached, so a rejected hydrate re-attempts on the next replay (no parked rejected promise). This trades away the optimization in the object-returning case to keep deterministic replay airtight — correctness over speed.

What I verified

  • Unit:step-hydration-cache.test.ts (14 tests: primitive detection, memoization, non-primitive eviction/fresh-object, falsy primitives, keying, error non-caching, no-cache passthrough, plus the size-bound — at-bound string is a hit, oversized string/bigint are not memoized and cache.size stays 0) and step-hydration-memoization.test.ts (3 tests through the real createUseStep consumer: hydrate-skipped-on-replay-2 via spy, event-log ordering preserved on cache hits, fresh object per replay for object results).
  • Full core suite:cd packages/core && pnpm test1253 passed / 56 files, including async-deserialization-ordering.test.ts, workflow.test.ts (79 tests), runtime.test.ts, hook-sleep-interaction, abort-consistency. No regressions.
  • Build / format / typecheck:pnpm build (full repo, 27/27), @workflow/core build + tsc --noEmit clean; Biome format applied; new files Biome-clean (the only lint errors were import-ordering, auto-fixed; remaining warnings are pre-existing noExcessiveCognitiveComplexity on functions I only edited).
  • E2E (local nextjs-turbopack dev server, the determinism-sensitive subset): promiseAllWorkflow, promiseRaceWorkflow, promiseAnyWorkflow, sleepWinsRaceWorkflow, stepWinsRaceWorkflow, promiseRaceStressTestWorkflow, hookWorkflow, webhookWorkflow, parallel-steps-then-webhook replay race, sleepingWorkflow, parallelSleepWorkflow, retry/error/catchability suite, fetchWorkflowall passed.

Risks / deferred

  • Only primitive step results are accelerated; object-returning steps still re-hydrate each replay (intentional, for determinism). A future safe extension could deep-freeze + share frozen object graphs, but that needs care around reviver-produced special objects and is out of scope here.
  • Large (>4 KiB) string/bigint results are intentionally not memoized to bound peak retained memory (see Memory characteristic); they re-hydrate each replay.
  • Hook/wait/abort hydration paths are uncached by design.

🤖 Generated with Claude Code

The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CopilotAI review requested due to automatic review settings June 17, 2026 01:47
@pranaygp
pranaygp requested a review from a team as a code ownerJune 17, 2026 01:47

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@changeset-bot

changeset-botBot commented Jun 17, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3ca022f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
workflowPatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production144112301672
✅ 💻 Local Development190902192128
✅ 📦 Local Production190902192128
❌ 🐘 Local Postgres189412332128
✅ 🪟 Windows15200152
✅ 📋 Other88501791064
Total8190210809272

❌ Failed Tests

▲ Vercel Production (1 failed)

nitro (1 failed):

  • startFromWorkflow - calling start() directly inside a workflow function with hook communication | wrun_01KVCQ35C5K211Z7CFGRHG4655 | 🔍 observability
🐘 Local Postgres (1 failed)

nextjs-turbopack-stable-lazy-discovery-enabled (1 failed):

  • wellKnownAgentWorkflow (.well-known/agent) | wrun_01KVCPM9CVA5BNHKTABQFTWTWF

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro125027
✅ example125027
✅ express125027
✅ fastify125027
✅ hono125027
✅ nextjs-turbopack14903
✅ nextjs-webpack14903
❌ nitro124127
✅ nuxt125027
✅ sveltekit14408
✅ vite125027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
❌ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable126026
✅ express-stable126026
✅ fastify-stable126026
✅ hono-stable126026
✅ nextjs-turbopack-canary132020
✅ nextjs-turbopack-stable-lazy-discovery-disabled15101
❌ nextjs-turbopack-stable-lazy-discovery-enabled15011
✅ nextjs-webpack-canary132020
✅ nextjs-webpack-stable-lazy-discovery-disabled15101
✅ nextjs-webpack-stable-lazy-discovery-enabled15101
✅ nitro-stable126026
✅ nuxt-stable126026
✅ sveltekit-stable14507
✅ vite-stable126026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15200
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable127025
✅ e2e-local-dev-tanstack-start-127025
✅ e2e-local-postgres-nest-stable126026
✅ e2e-local-postgres-tanstack-start-126026
✅ e2e-local-prod-nest-stable127025
✅ e2e-local-prod-tanstack-start-127025
✅ e2e-vercel-prod-tanstack-start125027

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: failure
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.042s (-2.5%)1.006s (~)0.963s101.00x
💻 LocalNitro0.045s (+10.1% 🔺)1.006s (~)0.962s101.06x
💻 LocalNext.js (Turbopack)0.062s (-5.5% 🟢)1.007s (~)0.944s101.47x
🐘 PostgresExpress0.067s (-8.0% 🟢)1.013s (~)0.946s101.58x
🐘 PostgresNext.js (Turbopack)0.070s (-0.9%)1.013s (~)0.943s101.65x
🐘 PostgresNitro0.073s (+15.6% 🔺)1.013s (~)0.940s101.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express0.271s (-28.1% 🟢)2.235s (-2.6%)1.964s101.00x
▲ VercelNitro0.319s (-14.9% 🟢)2.203s (-2.0%)1.883s101.18x
▲ VercelNext.js (Turbopack)0.351s (+20.6% 🔺)2.531s (+33.8% 🔺)2.180s101.29x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.094s (-0.7%)2.006s (~)0.912s101.00x
💻 LocalNitro1.095s (+0.9%)2.007s (~)0.912s101.00x
🐘 PostgresNitro1.108s (-1.1%)2.009s (~)0.901s101.01x
🐘 PostgresExpress1.110s (~)2.008s (~)0.898s101.02x
💻 LocalNext.js (Turbopack)1.132s (-1.4%)2.007s (~)0.875s101.03x
🐘 PostgresNext.js (Turbopack)1.141s (~)2.009s (~)0.868s101.04x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.723s (-11.0% 🟢)3.716s (+1.1%)1.993s101.00x
▲ VercelNext.js (Turbopack)1.761s (~)3.708s (~)1.947s101.02x
▲ VercelNitro1.788s (-22.8% 🟢)3.409s (-14.6% 🟢)1.621s101.04x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro10.527s (~)11.022s (~)0.495s31.00x
💻 LocalExpress10.566s (~)11.023s (~)0.457s31.00x
🐘 PostgresExpress10.585s (~)11.023s (~)0.438s31.01x
🐘 PostgresNitro10.598s (+0.6%)11.022s (~)0.424s31.01x
💻 LocalNext.js (Turbopack)10.812s (~)11.022s (~)0.210s31.03x
🐘 PostgresNext.js (Turbopack)10.829s (~)11.017s (~)0.189s31.03x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express13.691s (-24.2% 🟢)15.347s (-23.1% 🟢)1.656s21.00x
▲ VercelNitro13.728s (-27.6% 🟢)15.441s (-27.5% 🟢)1.713s21.00x
▲ VercelNext.js (Turbopack)14.688s (+2.6%)16.926s (+3.9%)2.238s21.07x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro13.690s (~)14.027s (~)0.337s51.00x
🐘 PostgresExpress13.756s (-0.7%)14.019s (~)0.263s51.00x
💻 LocalExpress13.798s (~)14.028s (~)0.231s51.01x
🐘 PostgresNitro13.832s (~)14.023s (~)0.191s51.01x
💻 LocalNext.js (Turbopack)14.383s (~)15.030s (~)0.647s41.05x
🐘 PostgresNext.js (Turbopack)14.396s (~)15.017s (~)0.621s41.05x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express23.028s (-4.9%)25.270s (-3.5%)2.242s31.00x
▲ VercelNext.js (Turbopack)24.609s (-30.5% 🟢)26.392s (-28.3% 🟢)1.783s31.07x
▲ VercelNitro24.684s (-20.8% 🟢)26.313s (-19.2% 🟢)1.629s31.07x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro12.373s (+2.5%)13.027s (+1.1%)0.654s71.00x
💻 LocalExpress12.444s (-0.8%)13.024s (~)0.580s71.01x
🐘 PostgresExpress12.512s (+0.6%)13.016s (~)0.503s71.01x
🐘 PostgresNitro12.712s (+1.6%)13.021s (~)0.310s71.03x
💻 LocalNext.js (Turbopack)13.662s (~)14.027s (~)0.365s71.10x
🐘 PostgresNext.js (Turbopack)13.897s (~)14.308s (+1.0%)0.411s71.12x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro27.276s (-29.9% 🟢)29.146s (-28.8% 🟢)1.870s41.00x
▲ VercelNext.js (Turbopack)27.460s (-22.0% 🟢)29.404s (-20.4% 🟢)1.944s41.01x
▲ VercelExpress28.389s (-19.0% 🟢)30.612s (-17.0% 🟢)2.223s31.04x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.204s (-0.8%)2.007s (~)0.803s151.00x
💻 LocalNitro1.222s (+4.5%)2.006s (~)0.784s151.01x
💻 LocalExpress1.231s (+5.6% 🔺)2.007s (~)0.776s151.02x
🐘 PostgresNitro1.251s (+4.4%)2.009s (~)0.757s151.04x
🐘 PostgresNext.js (Turbopack)1.256s (-2.3%)2.007s (~)0.751s151.04x
💻 LocalNext.js (Turbopack)1.405s (+8.4% 🔺)2.006s (~)0.602s151.17x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.571s (+15.9% 🔺)4.329s (+11.4% 🔺)1.757s81.00x
▲ VercelNitro3.123s (+7.3% 🔺)4.469s (+1.4%)1.346s71.21x
▲ VercelNext.js (Turbopack)3.608s (+35.3% 🔺)4.968s (+21.2% 🔺)1.360s71.40x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.309s (-8.1% 🟢)2.008s (-16.1% 🟢)0.700s151.00x
🐘 PostgresNitro1.332s (-2.4%)2.074s (-17.3% 🟢)0.742s151.02x
🐘 PostgresNext.js (Turbopack)1.452s (-11.3% 🟢)2.075s (-13.3% 🟢)0.623s151.11x
💻 LocalExpress1.987s (+22.2% 🔺)2.592s (+29.2% 🔺)0.604s121.52x
💻 LocalNitro2.051s (+32.2% 🔺)2.507s (+24.7% 🔺)0.456s121.57x
💻 LocalNext.js (Turbopack)2.359s (+23.5% 🔺)3.008s (+31.2% 🔺)0.649s101.80x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.443s (-13.1% 🟢)5.587s (-1.5%)2.143s61.00x
▲ VercelNitro3.655s (-9.7% 🟢)5.233s (-8.2% 🟢)1.577s61.06x
▲ VercelNext.js (Turbopack)4.245s (-5.5% 🟢)5.891s (-9.1% 🟢)1.646s61.23x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.468s (-16.7% 🟢)3.886s (~)2.417s81.00x
🐘 PostgresNitro1.470s (-7.8% 🟢)3.885s (-3.1%)2.415s81.00x
🐘 PostgresNext.js (Turbopack)2.642s (-16.3% 🟢)3.456s (-19.7% 🟢)0.814s91.80x
💻 LocalNitro4.340s (+26.8% 🔺)5.013s (+25.0% 🔺)0.673s62.96x
💻 LocalExpress5.253s (+20.2% 🔺)5.679s (+16.7% 🔺)0.425s63.58x
💻 LocalNext.js (Turbopack)6.408s (+11.0% 🔺)7.019s (+12.9% 🔺)0.611s54.36x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro4.491s (-26.7% 🟢)6.303s (-24.4% 🟢)1.812s51.00x
▲ VercelNext.js (Turbopack)4.524s (-19.6% 🟢)6.497s (-15.8% 🟢)1.973s51.01x
▲ VercelExpress4.646s (-15.6% 🟢)7.023s (-11.4% 🟢)2.378s51.03x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.202s (-1.3%)2.007s (~)0.805s151.00x
🐘 PostgresNitro1.214s (~)2.008s (~)0.794s151.01x
💻 LocalExpress1.228s (-22.1% 🟢)2.006s (~)0.778s151.02x
💻 LocalNitro1.268s (+5.6% 🔺)2.006s (~)0.738s151.06x
🐘 PostgresNext.js (Turbopack)1.271s (-0.8%)2.008s (~)0.737s151.06x
💻 LocalNext.js (Turbopack)1.449s (+3.6%)2.006s (~)0.557s151.21x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.668s (-25.2% 🟢)4.041s (-23.6% 🟢)1.373s81.00x
▲ VercelNext.js (Turbopack)2.876s (-22.7% 🟢)4.782s (-8.6% 🟢)1.907s71.08x
▲ VercelExpress3.339s (+43.7% 🔺)5.449s (+43.2% 🔺)2.110s61.25x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Nitro1.304s (-9.6% 🟢)2.075s (-13.3% 🟢)0.771s151.00x
🐘 PostgresExpress1.310s (-4.1%)2.007s (-13.3% 🟢)0.697s151.00x
🐘 PostgresNext.js (Turbopack)1.436s (-6.9% 🟢)2.076s (-6.6% 🟢)0.640s151.10x
💻 LocalNitro1.935s (+14.3% 🔺)2.293s (+14.2% 🔺)0.358s141.48x
💻 LocalExpress2.003s (+5.5% 🔺)2.393s (+11.3% 🔺)0.390s131.54x
💻 LocalNext.js (Turbopack)2.369s (+9.3% 🔺)3.009s (~)0.639s101.82x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.899s (+6.3% 🔺)4.683s (+2.5%)1.785s71.00x
▲ VercelNitro3.978s (+42.1% 🔺)5.734s (+37.1% 🔺)1.756s61.37x
▲ VercelNext.js (Turbopack)4.216s (-5.6% 🟢)5.944s (-3.4%)1.728s61.45x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.452s (-19.3% 🟢)4.014s (~)2.561s81.00x
🐘 PostgresNitro1.589s (-12.0% 🟢)3.678s (-14.4% 🟢)2.090s91.09x
🐘 PostgresNext.js (Turbopack)2.136s (-47.5% 🟢)3.454s (-22.3% 🟢)1.318s91.47x
💻 LocalExpress5.580s (+21.6% 🔺)6.014s (+20.0% 🔺)0.434s53.84x
💻 LocalNitro5.616s (+25.0% 🔺)6.017s (+20.1% 🔺)0.401s53.87x
💻 LocalNext.js (Turbopack)6.852s (+18.0% 🔺)7.416s (+15.6% 🔺)0.563s54.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.610s (-57.2% 🟢)5.562s (-45.7% 🟢)1.952s61.00x
▲ VercelNitro4.046s (-70.8% 🟢)5.724s (-63.4% 🟢)1.678s61.12x
▲ VercelNext.js (Turbopack)4.124s (+9.0% 🔺)6.272s (+19.9% 🔺)2.147s51.14x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.605s (+6.0% 🔺)1.041s (+3.5%)0.436s581.00x
💻 LocalExpress0.616s (-1.5%)1.005s (-1.7%)0.389s601.02x
💻 LocalNitro0.629s (+25.8% 🔺)1.039s (+3.1%)0.410s581.04x
🐘 PostgresNitro0.692s (+16.5% 🔺)1.078s (+3.6%)0.386s561.14x
🐘 PostgresNext.js (Turbopack)0.858s (+2.7%)1.041s (+1.7%)0.183s581.42x
💻 LocalNext.js (Turbopack)0.860s (-3.1%)1.005s (-3.3%)0.144s601.42x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.774s (-42.9% 🟢)5.457s (-34.5% 🟢)1.683s111.00x
▲ VercelNext.js (Turbopack)3.815s (-39.5% 🟢)5.390s (-33.8% 🟢)1.575s121.01x
▲ VercelExpress4.488s (-1.3%)6.377s (-1.6%)1.889s101.19x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.375s (+1.4%)2.052s (+1.2%)0.678s441.00x
🐘 PostgresNitro1.466s (+5.6% 🔺)2.030s (~)0.564s451.07x
💻 LocalNitro1.485s (+25.1% 🔺)2.006s (~)0.521s451.08x
💻 LocalExpress1.530s (+2.8%)2.007s (~)0.476s451.11x
🐘 PostgresNext.js (Turbopack)1.989s (+2.5%)2.308s (+11.2% 🔺)0.319s401.45x
💻 LocalNext.js (Turbopack)2.090s (-0.6%)2.944s (-2.1%)0.854s311.52x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express10.297s (-41.1% 🟢)12.369s (-35.5% 🟢)2.072s81.00x
▲ VercelNitro10.774s (-8.8% 🟢)12.396s (-12.5% 🟢)1.621s81.05x
▲ VercelNext.js (Turbopack)11.321s (-16.7% 🟢)13.637s (-11.0% 🟢)2.316s71.10x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express2.654s (-2.1%)3.058s (-1.7%)0.404s401.00x
🐘 PostgresNitro2.903s (+3.7%)3.280s (+3.6%)0.376s371.09x
💻 LocalExpress3.275s (+2.4%)4.010s (~)0.735s301.23x
💻 LocalNitro3.362s (+23.1% 🔺)4.010s (+24.4% 🔺)0.648s301.27x
🐘 PostgresNext.js (Turbopack)3.983s (+2.9%)4.253s (+4.3%)0.270s291.50x
💻 LocalNext.js (Turbopack)4.363s (~)5.010s (~)0.647s241.64x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express18.286s (-29.9% 🟢)20.532s (-26.7% 🟢)2.246s61.00x
▲ VercelNitro18.713s (-28.9% 🟢)20.382s (-27.5% 🟢)1.669s61.02x
▲ VercelNext.js (Turbopack)20.206s (-19.2% 🟢)22.420s (-16.3% 🟢)2.214s61.11x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.232s (~)1.006s (~)0.773s601.00x
🐘 PostgresNitro0.237s (+0.7%)1.006s (~)0.769s601.02x
🐘 PostgresNext.js (Turbopack)0.297s (+1.1%)1.023s (+1.7%)0.726s591.28x
💻 LocalExpress0.400s (-9.2% 🟢)1.005s (~)0.605s601.72x
💻 LocalNitro0.417s (+11.8% 🔺)1.004s (~)0.588s601.79x
💻 LocalNext.js (Turbopack)0.633s (+8.8% 🔺)1.004s (~)0.371s602.73x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.787s (+15.3% 🔺)3.598s (+25.4% 🔺)1.811s181.00x
▲ VercelNitro1.933s (+49.7% 🔺)3.615s (+25.8% 🔺)1.682s171.08x
▲ VercelNext.js (Turbopack)2.387s (+27.6% 🔺)4.344s (+11.8% 🔺)1.956s141.34x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.331s (-6.9% 🟢)1.006s (-3.3%)0.675s901.00x
🐘 PostgresNitro0.356s (+4.1%)1.006s (-1.1%)0.650s901.08x
🐘 PostgresNext.js (Turbopack)0.476s (-11.7% 🟢)1.103s (-1.3%)0.628s831.44x
💻 LocalNitro2.182s (+43.2% 🔺)2.737s (+27.2% 🔺)0.555s336.59x
💻 LocalExpress2.222s (+6.3% 🔺)2.738s (+3.1%)0.517s336.71x
💻 LocalNext.js (Turbopack)2.493s (+7.0% 🔺)3.344s (+8.7% 🔺)0.851s277.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.638s (-9.8% 🟢)4.553s (-5.1% 🟢)1.915s201.00x
▲ VercelNitro2.705s (-1.5%)4.386s (-8.3% 🟢)1.681s211.03x
▲ VercelNext.js (Turbopack)3.092s (+1.6%)4.802s (~)1.710s191.17x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.526s (-7.4% 🟢)1.059s (-13.0% 🟢)0.532s1141.00x
🐘 PostgresNitro0.570s (~)1.078s (-12.5% 🟢)0.508s1121.08x
🐘 PostgresNext.js (Turbopack)1.918s (-26.7% 🟢)2.763s (-20.4% 🟢)0.845s443.64x
💻 LocalNitro9.521s (+50.8% 🔺)10.445s (+54.9% 🔺)0.924s1218.09x
💻 LocalExpress10.198s (+22.8% 🔺)11.029s (+24.2% 🔺)0.831s1219.38x
💻 LocalNext.js (Turbopack)10.284s (-4.4%)11.663s (+0.8%)1.379s1119.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.245s (-38.5% 🟢)5.031s (-28.3% 🟢)1.786s241.00x
▲ VercelExpress3.483s (-36.2% 🟢)5.687s (-19.8% 🟢)2.204s221.07x
▲ VercelNext.js (Turbopack)4.699s (-2.9%)6.708s (+2.7%)2.008s181.45x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.165s (-1.1%)2.000s (~)0.001s (+9.1% 🔺)2.009s (~)0.844s101.00x
🐘 PostgresNitro1.181s (+1.4%)1.995s (~)0.001s (+30.0% 🔺)2.011s (~)0.831s101.01x
💻 LocalNitro1.181s (+5.7% 🔺)2.005s (~)0.010s (-62.9% 🟢)2.017s (-1.0%)0.836s101.01x
💻 LocalExpress1.185s (+3.0%)2.005s (~)0.012s (+19.8% 🔺)2.019s (~)0.835s101.02x
💻 LocalNext.js (Turbopack)1.216s (~)2.003s (~)0.013s (+2.4%)2.020s (~)0.803s101.04x
🐘 PostgresNext.js (Turbopack)1.232s (~)2.002s (~)0.001s (-15.4% 🟢)2.011s (~)0.779s101.06x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.206s (-8.2% 🟢)3.426s (-6.9% 🟢)0.602s (-18.8% 🟢)4.513s (-7.4% 🟢)2.307s101.00x
▲ VercelNext.js (Turbopack)2.304s (-8.6% 🟢)3.523s (-7.8% 🟢)0.772s (-4.3%)4.790s (-5.8% 🟢)2.486s101.04x
▲ VercelNitro2.331s (-11.7% 🟢)3.158s (-17.0% 🟢)1.253s (+57.9% 🔺)4.805s (-4.9%)2.474s101.06x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro1.572s (+9.3% 🔺)2.011s (~)0.012s (-32.0% 🟢)2.025s (~)0.453s301.00x
🐘 PostgresExpress1.577s (~)2.004s (~)0.005s (+5.7% 🔺)2.026s (~)0.449s301.00x
💻 LocalExpress1.583s (+1.0%)2.010s (~)0.014s (+4.9%)2.025s (~)0.442s301.01x
🐘 PostgresNitro1.602s (+1.5%)2.008s (~)0.005s (-5.6% 🟢)2.027s (~)0.425s301.02x
💻 LocalNext.js (Turbopack)1.740s (-0.5%)2.010s (~)0.013s (+1.3%)2.025s (~)0.285s301.11x
🐘 PostgresNext.js (Turbopack)1.892s (+5.9% 🔺)2.011s (~)0.005s (+5.8% 🔺)2.029s (~)0.137s301.20x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Next.js (Turbopack)6.842s (-19.5% 🟢)8.374s (-14.7% 🟢)0.220s (-16.3% 🟢)9.079s (-14.1% 🟢)2.236s71.00x
▲ VercelNitro6.905s (-3.6%)7.768s (-10.3% 🟢)0.455s (+91.9% 🔺)8.766s (-6.9% 🟢)1.861s71.01x
▲ VercelExpress8.068s (+26.4% 🔺)9.373s (+20.1% 🔺)0.307s (+71.5% 🔺)10.412s (+23.1% 🔺)2.345s61.18x

🔍 Observability: Next.js (Turbopack) | Nitro | Express

10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.789s (+3.3%)1.101s (+7.5% 🔺)0.000s (-100.0% 🟢)1.118s (+6.0% 🔺)0.329s541.00x
🐘 PostgresNitro0.795s (+2.9%)1.044s (~)0.000s (-3.5%)1.061s (-3.0%)0.265s571.01x
🐘 PostgresNext.js (Turbopack)0.986s (-3.3%)1.397s (-4.6%)0.000s (-100.0% 🟢)1.407s (-4.4%)0.421s431.25x
💻 LocalExpress1.533s (+12.5% 🔺)2.014s (~)0.000s (+16.7% 🔺)2.016s (~)0.483s301.94x
💻 LocalNitro1.568s (+45.8% 🔺)2.014s (+9.6% 🔺)0.000s (-56.8% 🟢)2.016s (+9.5% 🔺)0.448s301.99x
💻 LocalNext.js (Turbopack)1.916s (+26.7% 🔺)2.193s (+8.9% 🔺)0.000s (+7.1% 🔺)2.196s (+8.9% 🔺)0.280s282.43x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.961s (-21.2% 🟢)4.002s (-25.7% 🟢)0.000s (-21.4% 🟢)4.479s (-23.6% 🟢)1.518s141.00x
▲ VercelExpress3.140s (+3.6%)4.516s (-1.1%)0.000s (NaN%)5.069s (+0.6%)1.929s121.06x
▲ VercelNext.js (Turbopack)3.540s (+2.7%)5.016s (+5.2% 🔺)0.000s (-100.0% 🟢)5.553s (+5.5% 🔺)2.013s111.20x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.691s (+4.7%)2.302s (+11.5% 🔺)0.000s (NaN%)2.316s (+10.9% 🔺)0.625s261.00x
🐘 PostgresNitro1.779s (+14.7% 🔺)2.381s (+11.6% 🔺)0.000s (+11.5% 🔺)2.394s (+11.5% 🔺)0.615s261.05x
🐘 PostgresNext.js (Turbopack)2.251s (+6.6% 🔺)2.651s (+2.5%)0.000s (-100.0% 🟢)2.664s (+2.6%)0.413s231.33x
💻 LocalNitro4.321s (+95.8% 🔺)4.717s (+72.0% 🔺)0.001s (+12.8% 🔺)4.729s (+71.7% 🔺)0.409s132.56x
💻 LocalExpress4.772s (+62.9% 🔺)5.362s (+46.1% 🔺)0.001s (+98.3% 🔺)5.367s (+46.1% 🔺)0.595s122.82x
💻 LocalNext.js (Turbopack)5.632s (+94.2% 🔺)6.226s (+85.4% 🔺)0.000s (-60.0% 🟢)6.235s (+85.4% 🔺)0.602s103.33x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express4.310s (-24.3% 🟢)6.054s (-16.2% 🟢)0.000s (+Infinity% 🔺)6.546s (-14.9% 🟢)2.236s101.00x
▲ VercelNext.js (Turbopack)4.772s (-34.2% 🟢)6.201s (-26.1% 🟢)0.000s (-100.0% 🟢)6.679s (-24.5% 🟢)1.907s101.11x
▲ VercelNitro4.835s (-7.9% 🟢)5.805s (-10.1% 🟢)0.000s (+Infinity% 🔺)6.308s (-8.9% 🟢)1.473s101.12x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalNitro12/21
🐘 PostgresExpress19/21
▲ VercelExpress14/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres17/21
Next.js (Turbopack)🐘 Postgres14/21
Nitro🐘 Postgres13/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run

@pranaygp

Copy link
Copy Markdown
ContributorAuthor

CI failure triage — pre-existing Vercel-prod e2e flake (not a regression)

The two red checks (E2E Vercel Prod Tests (example), E2E Vercel Prod Tests (fastify), which roll up into E2E Required Check) are the shared Vercel-prod timing flake, not caused by this PR. Evidence:

Scope is wrong for a hydration regression. This PR only memoizes primitive step-result hydration. A determinism/stale-value bug there would surface across all worlds — yet every local suite is 100% green:

SuitePassedFailed
▲ Vercel Production14402
💻 Local Development19090
📦 Local Production19090
🐘 Local Postgres18950
🪟 Windows1520

Only 2 failures, only on Vercel Production.

The two failing tests are unrelated to result hydration, and are abort/hook timing races:

  1. exampleAbortController > abortFromStepWorkflow: step abort cancels an in-flight sibling step. The run completed successfully; the assertion failed only because the abort lost a race. From the run diagnostics (wrun_01KV9MZ1Y00N90XFCKFTSPX5N1):

    +2.2s step_completed (longStep) <- sibling finished on its own
    +3.0s hook_received <- abort signal arrived AFTER
    

    The sibling longStep self-completed (2.2s) before the abort hook arrived (3.0s) under Vercel-prod queue/network latency, so there was no in-flight step left to cancel. Step results hydrated fine; this is purely environmental latency.

  2. fastifystartFromWorkflow - calling start() directly inside a workflow function with hook communication. This exact test passed (4128ms) in the example job of this same commit (a56f5c90b) — a textbook cross-run flake.

The same test is red on plain main, without this change. On main run 27704378960 (commit 2acf13cc7):

  • E2E Vercel Prod Tests (tanstack-start)abortFromStepWorkflow: step abort cancels an in-flight sibling step (the identical test that failed here on example)
  • E2E Vercel Prod Tests (nextjs-turbopack)distributedAbortController - reconnect to existing controller

And on main run 27657696161 (cb181392b, the commit this branch is based on): E2E Vercel Prod Tests (fastify)hookWithSleepFinalStepWorkflow. The set of red workbenches rotates run-to-run — the signature of environment flakiness, not a code regression.

Local verification of this branch (rebuilt @workflow/core first): cd packages/core && pnpm test1249 passed / 56 files, 0 failures, including the determinism/ordering replay tests in workflow.test.ts and the new step-hydration-cache.test.ts (10) + step-hydration-memoization.test.ts (3). The memoization tests assert byte-identical delivery ordering on cache hits and that objects re-hydrate fresh each replay.

Re-running the e2e jobs should clear them. No code change is warranted.

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — the O(N²)→O(N) hydration memoization, done with the right safety bias

This is the most safety-sensitive of the four (it's the only one that caches a value across replays), and the design lands on the conservative side of every judgment call, which is exactly right for replay determinism.

The primitives-only decision is the crux and it's correct.hydrateStepReturnValue (devalue.parse) returns a fresh object graph each call, and each replay runs in a fresh VM, so today the workflow gets a brand-new value every replay. Caching and returning the same object reference would let const r = await step(); r.count++ observe a prior replay's mutation — silent divergence. The alternatives are both worse: structuredClone is lossy for reviver-produced specials (stream handles, step-fn proxies, Request/Response, AbortController/Signal) and still O(size). Restricting the cache to primitives (immutable, compared by value) makes "share the reference" provably indistinguishable from re-parsing, and non-primitives fall through to a full re-hydrate every replay — preserving current behavior exactly. Trading the object-case optimization for airtight determinism is the right call.

What I verified in the integration:

  • Surgical wrap: only the await hydrateStepReturnValue(...) call is replaced; pendingDeliveries++/--, the ctx.promiseQueue.then(...) slot, and resolve(...) are byte-for-byte unchanged. The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the identical position in the ordered delivery chain a re-hydrate would have — preserving the pendingDeliveries-gated suspension/barrier/Promise.race determinism.
  • Lifetime: cache is created once per invocation in runtime.ts (outside the per-iteration context), threaded through runWorkflow, never shared across runs. The optional param/context field degrades to re-hydrating-every-replay for harnesses that omit it.
  • Keying by step_completedeventId (stable, world-assigned, same immutable bytes every replay) is sound, and has() rather than get() !== undefined correctly treats a memoized undefined result as a hit.
  • Errors never cached — a rejected hydrate re-attempts next replay, no parked rejected promise.

Built @workflow/core; full suite green (1249) including the new step-hydration-cache.test.ts / step-hydration-memoization.test.ts, async-deserialization-ordering.test.ts, and workflow.test.ts.

One forward-looking note (non-blocking): the object case is left on the table by design. If it's ever revisited, the safe path is deep-freeze + share frozen graphs — but only after handling the reviver-produced specials, so I agree with deferring it. LGTM.

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

// Only memoize values that are safe to return by reference across replays.
// Non-primitives fall through and are re-hydrated fresh on every replay.
if (isMemoizablePrimitive(value)) {
cache.set(eventId, value);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

The per-run stepHydrationCache is never size-bounded or evicted: it grows one entry per primitive-returning completed step and lives for the entire invocation. The cost worth calling out (the PR doesn't) is the new residency — the decrypted/devalue-parsed plaintext of each cached primitive is now held for the whole invocation, on top of the serialized bytes already retained in cachedEvents. For a long sequential workflow whose steps return large strings, that roughly doubles peak retained memory for those results.

The dominant residency (the full event log in cachedEvents) already exists, so this isn't blocking, but a byte-size threshold would be a cheap safeguard — large primitives are exactly the cheap-to-re-hydrate case relative to their footprint, so letting them fall through to the existing re-hydrate path costs little and bounds the worst case. Ideally with a test asserting the bound.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — capped it, with a test. Pushed in a60baad.

What changed (step-hydration-cache.ts):

  • Added MAX_MEMOIZED_PRIMITIVE_LENGTH = 4096 and extended isMemoizablePrimitive so a string/bigint longer than 4 KiB is treated as non-memoizable. Those are the only primitive types that can carry a large payload — number/boolean/null/undefined/symbol are inherently small, so they're never length-checked. Oversized values now fall through to the existing per-replay re-hydrate path, exactly as you suggested: large primitives are cheap to re-hydrate relative to their footprint, so this caps the doubled-residency worst case at negligible cost.
  • Documented the memory characteristic on the cache module: per-invocation lifetime (fresh Map per run in runtime.ts, GC'd when the invocation returns), bounded by the number of primitive-returning completed steps, primitives-only, now byte-bounded.

Tests (step-hydration-cache.test.ts, +4):isMemoizablePrimitive true at the bound / false beyond it (string and bigint), and an end-to-end assertion that an oversized string re-hydrates on every replay and cache.size === 0 (the bound assertion you asked for); plus an at-bound string is a cache hit.

The cap only ever reduces what gets cached, so determinism is untouched — oversized values just take the already-correct re-hydrate path. Full core suite green (1253, incl. the ordering/determinism + memoization suites); biome + tsc clean.

On consistency with #2471 (the sibling scriptCache): noting the distinction since they're bounded for different reasons. #2471's cache is process-wide and monotonic across the whole process — in dev/watch it pins every historical bundle string (hundreds of MB over a session), which is a genuine regression vs. the prior keep-only-latest behavior, hence the Blocking bound there. This cache is per-invocation and freed wholesale when the run returns, so it can never accumulate across runs; the only real cost is the doubled residency for large primitives during one run, which the size cap here now bounds. Different scope, different severity, but both bounded now.

Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-hydration
* origin/main:
perf(core): lazy inline step start (save one world round-trip per step) (#2478)
perf(core): skip per-step events.list via inline event-log delta (#2475)
Version Packages (beta) (#2491)
[world-vercel] Honor hasMore flag from v4 list pagination endpoint (#2486)
Version Packages (beta) (#2451)
Fix Next workflow module specifier root (#2455)
[world-vercel] Send remoteRefBehavior=lazy on v4 metadata-only event listings (#2415)
[swc-plugin] Fix eager discovery for object property steps (#2484)
fix(web-shared): align attributes panel styling (#2483)
[web-shared] Auto-scroll trace viewer on J/K span navigation (#2366)
fix(web): render restarted step segment as solid gray, not running stripes (#2480)
fix(web-shared): use solid gray for queued trace segment (#2474)
Add trace viewer span markers for hooks and attributes (#2452)
test: support Vercel protection bypass secret in e2e headers (#2458)
fix(core): bump payload-compression cutoff to 5.0.0-beta.18 (#2470)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts:
#	packages/core/src/runtime.ts
@pranaygp
pranaygp enabled auto-merge (squash) June 18, 2026 06:24
@pranaygp
pranaygp disabled auto-merge June 22, 2026 20:30
@pranaygp
pranaygp merged commit 66ca0dc into mainJun 22, 2026
118 of 121 checks passed
@pranaygp
pranaygp deleted the pgp/perf-memoize-step-hydration branch June 22, 2026 20:30
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
github-actionsBot added a commit that referenced this pull request Jun 22, 2026
…#2472)
* perf(core): memoize step return value hydration across replays
The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): bound memoized step-hydration cache by primitive size
Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #2563. Merge conflicts were resolved by AI — please review carefully. (backport job run)

pranaygp added a commit that referenced this pull request Jun 22, 2026
…testing
* origin/main:
Version Packages (beta) (#2540)
perf(core): memoize step return value hydration across inline replays (#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (#2412)
Amend lazy discovery removal changeset (#2560)
[docs] Document minimum SDK version for using hook.getConflict (#2423)
Update default CODEOWNERS (#2556)
Optimize and fix the default eager build mode (#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (#2547)" (#2554)
[core] Turbo mode: fast-path the first invocation (#2526)
Remove lazy discovery from workflow/next (#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (#2547)
feat(docs): add eve and AI SDK to product switcher (#2543)
[vitest] Fix local imports failing to load in test step bundles (#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (#2324)
Version Packages (beta) (#2495)
otel(world-vercel): inject trace context on v4 event requests (#2533)
Bump undici to 7.28.0 (#2534)
Default source maps to dev-on / prod-off (#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (#2527)
perf(core): parallel inline steps + optimistic lazy step start (#2516)
pranaygp added a commit to marcopiraccini/workflow that referenced this pull request Jun 22, 2026
* origin/main: (120 commits)
Version Packages (beta) (vercel#2540)
perf(core): memoize step return value hydration across inline replays (vercel#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (vercel#2412)
Amend lazy discovery removal changeset (vercel#2560)
[docs] Document minimum SDK version for using hook.getConflict (vercel#2423)
Update default CODEOWNERS (vercel#2556)
Optimize and fix the default eager build mode (vercel#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)" (vercel#2554)
[core] Turbo mode: fast-path the first invocation (vercel#2526)
Remove lazy discovery from workflow/next (vercel#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)
feat(docs): add eve and AI SDK to product switcher (vercel#2543)
[vitest] Fix local imports failing to load in test step bundles (vercel#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (vercel#2324)
Version Packages (beta) (vercel#2495)
otel(world-vercel): inject trace context on v4 event requests (vercel#2533)
Bump undici to 7.28.0 (vercel#2534)
Default source maps to dev-on / prod-off (vercel#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (vercel#2527)
perf(core): parallel inline steps + optimistic lazy step start (vercel#2516)
...
VaguelySerious pushed a commit that referenced this pull request Jun 22, 2026
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pranaygp@TooTallNate@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

perf(core): memoize step return value hydration across inline replays - #2472

Merged
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration
Jun 22, 2026
Merged

perf(core): memoize step return value hydration across inline replays#2472
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration

Conversation

@pranaygp

@pranaygppranaygp commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

The inline replay loop (runtime.tsrunWorkflow, workflow.ts) re-executes the workflow body and re-consumes the full event log on every iteration. For each already-completed step, the step consumer (step.ts, step_completed path) re-ran hydrateStepReturnValue — AES-GCM decrypt + devalue-parse of the serialized result — on every replay, even though that exact result was already hydrated on every prior replay.

For a sequential workflow of N steps, replay K hydrates K results, so the aggregate cost across a single invocation is O(N²) decrypt+parse operations.

This PR adds a per-run memoization cache so a completed step's hydrated result is returned in O(1) on subsequent replays within the same invocation, making the aggregate cost O(N).

Before / after

  • Before: replay 1 hydrates 1 result, replay 2 hydrates 2, …, replay K hydrates K → Σ = O(N²) decrypt+parse over a sequential run.
  • After: each completed step's result is hydrated once and memoized for the rest of the invocation → O(N) total. Replay K hydrates only the one newly-completed step; the K−1 prior results are cache hits.

Cache scope & keying

  • Lifetime / scope: owned by the inline loop in runtime.ts (created once per run invocation, alongside cachedEvents), threaded into runWorkflow(..., stepHydrationCache?) and stored on WorkflowOrchestratorContext.stepHydrationCache. A fresh context is created each loop iteration, so the cache deliberately lives outside the per-iteration context to survive across iterations of the same run. It is never shared across unrelated runs or process-level invocations.
  • Keying: by the persisted step_completed event's eventId — a stable, world-assigned id. The same event carries the same immutable serialized bytes across every replay, so a hit is guaranteed to correspond to identical input.
  • Optional / backward compatible: the parameter and context field are optional. Callers/harnesses that omit them (and the many runWorkflow(...) unit tests) degrade to re-hydrating every replay — identical to previous behavior.

Memory characteristic

A cached entry holds the decrypted/devalue-parsed plaintext of a step result, retained for the rest of the invocation on top of the serialized bytes already held in cachedEvents — so for large primitive results it roughly doubles peak retained memory for those results during the run. This residual is:

  • Scoped to one invocation — the Map is created per run and GC'd when the invocation returns; nothing accumulates across runs or process-level invocations (a much weaker concern than a process-wide cache, where the dominant residency — the full event log in cachedEvents — already exists for the same lifetime).
  • Bounded by the primitive-returning completed-step count — at most one small entry per such step.
  • Byte-bounded. Most primitives (numbers, booleans, null/undefined, symbols, short ids/strings) are tiny. The only primitive that can be large is a string (or a pathologically long bigint), so a string/bigint result longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) is not memoized — it falls through to the existing per-replay re-hydrate path. Large payloads are cheap to re-hydrate relative to their footprint, so this caps the worst case at negligible cost. The cap only ever reduces what is cached, so deterministic replay is unaffected.

Ordering safety analysis

The cache lookup replaces only the await hydrateStepReturnValue(...) call inside the existing ctx.promiseQueue.then(async () => { ... }) slot. Everything else is byte-for-byte unchanged:

  • ctx.pendingDeliveries++ / -- accounting is untouched.
  • The hydrate (or cache hit) still happens inside the same serial promiseQueue slot, at the same log position, and still resolves via the same resolve(...).
  • The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the exact position in the ordered delivery chain a re-hydrate would have.

So delivery order, pendingDeliveries-gated suspensions, the pendingDeliveryBarriers / awaitEarlierDeliveries machinery, and Promise.race/Promise.all replay determinism are all unaffected. Hook, wait, and abort hydration paths are intentionally not cached (they're the ordering-sensitive paths and not the O(N²) hotspot).

Identity / immutability safety

hydrateStepReturnValue (devalue.parse) returns a fresh object graph on every call, and each replay iteration runs in a fresh workflow VM. Today the workflow therefore receives a brand-new value on every replay. If we cached and returned the same object reference across replays, workflow code that mutates a step result (const r = await step(); r.count++) would observe a previous replay's mutation on the next replay — a non-deterministic divergence. (structuredClone on each hit is both lossy — revivers reconstruct stream handles, step-function proxies, Request/Response, and AbortController/AbortSignal class instances — and still O(size).)

Decision: only primitives are memoized (string, number, boolean, bigint, symbol, null, undefined). Primitives are immutable and compared by value, so sharing the reference is provably indistinguishable from re-parsing. Any non-primitive result falls through to a full re-hydrate every replay, preserving current behavior exactly. Errors are never cached, so a rejected hydrate re-attempts on the next replay (no parked rejected promise). This trades away the optimization in the object-returning case to keep deterministic replay airtight — correctness over speed.

What I verified

  • Unit:step-hydration-cache.test.ts (14 tests: primitive detection, memoization, non-primitive eviction/fresh-object, falsy primitives, keying, error non-caching, no-cache passthrough, plus the size-bound — at-bound string is a hit, oversized string/bigint are not memoized and cache.size stays 0) and step-hydration-memoization.test.ts (3 tests through the real createUseStep consumer: hydrate-skipped-on-replay-2 via spy, event-log ordering preserved on cache hits, fresh object per replay for object results).
  • Full core suite:cd packages/core && pnpm test1253 passed / 56 files, including async-deserialization-ordering.test.ts, workflow.test.ts (79 tests), runtime.test.ts, hook-sleep-interaction, abort-consistency. No regressions.
  • Build / format / typecheck:pnpm build (full repo, 27/27), @workflow/core build + tsc --noEmit clean; Biome format applied; new files Biome-clean (the only lint errors were import-ordering, auto-fixed; remaining warnings are pre-existing noExcessiveCognitiveComplexity on functions I only edited).
  • E2E (local nextjs-turbopack dev server, the determinism-sensitive subset): promiseAllWorkflow, promiseRaceWorkflow, promiseAnyWorkflow, sleepWinsRaceWorkflow, stepWinsRaceWorkflow, promiseRaceStressTestWorkflow, hookWorkflow, webhookWorkflow, parallel-steps-then-webhook replay race, sleepingWorkflow, parallelSleepWorkflow, retry/error/catchability suite, fetchWorkflowall passed.

Risks / deferred

  • Only primitive step results are accelerated; object-returning steps still re-hydrate each replay (intentional, for determinism). A future safe extension could deep-freeze + share frozen object graphs, but that needs care around reviver-produced special objects and is out of scope here.
  • Large (>4 KiB) string/bigint results are intentionally not memoized to bound peak retained memory (see Memory characteristic); they re-hydrate each replay.
  • Hook/wait/abort hydration paths are uncached by design.

🤖 Generated with Claude Code

The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CopilotAI review requested due to automatic review settings June 17, 2026 01:47
@pranaygp
pranaygp requested a review from a team as a code ownerJune 17, 2026 01:47

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@changeset-bot

changeset-botBot commented Jun 17, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3ca022f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
workflowPatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production144112301672
✅ 💻 Local Development190902192128
✅ 📦 Local Production190902192128
❌ 🐘 Local Postgres189412332128
✅ 🪟 Windows15200152
✅ 📋 Other88501791064
Total8190210809272

❌ Failed Tests

▲ Vercel Production (1 failed)

nitro (1 failed):

  • startFromWorkflow - calling start() directly inside a workflow function with hook communication | wrun_01KVCQ35C5K211Z7CFGRHG4655 | 🔍 observability
🐘 Local Postgres (1 failed)

nextjs-turbopack-stable-lazy-discovery-enabled (1 failed):

  • wellKnownAgentWorkflow (.well-known/agent) | wrun_01KVCPM9CVA5BNHKTABQFTWTWF

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro125027
✅ example125027
✅ express125027
✅ fastify125027
✅ hono125027
✅ nextjs-turbopack14903
✅ nextjs-webpack14903
❌ nitro124127
✅ nuxt125027
✅ sveltekit14408
✅ vite125027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
❌ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable126026
✅ express-stable126026
✅ fastify-stable126026
✅ hono-stable126026
✅ nextjs-turbopack-canary132020
✅ nextjs-turbopack-stable-lazy-discovery-disabled15101
❌ nextjs-turbopack-stable-lazy-discovery-enabled15011
✅ nextjs-webpack-canary132020
✅ nextjs-webpack-stable-lazy-discovery-disabled15101
✅ nextjs-webpack-stable-lazy-discovery-enabled15101
✅ nitro-stable126026
✅ nuxt-stable126026
✅ sveltekit-stable14507
✅ vite-stable126026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15200
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable127025
✅ e2e-local-dev-tanstack-start-127025
✅ e2e-local-postgres-nest-stable126026
✅ e2e-local-postgres-tanstack-start-126026
✅ e2e-local-prod-nest-stable127025
✅ e2e-local-prod-tanstack-start-127025
✅ e2e-vercel-prod-tanstack-start125027

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: failure
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.042s (-2.5%)1.006s (~)0.963s101.00x
💻 LocalNitro0.045s (+10.1% 🔺)1.006s (~)0.962s101.06x
💻 LocalNext.js (Turbopack)0.062s (-5.5% 🟢)1.007s (~)0.944s101.47x
🐘 PostgresExpress0.067s (-8.0% 🟢)1.013s (~)0.946s101.58x
🐘 PostgresNext.js (Turbopack)0.070s (-0.9%)1.013s (~)0.943s101.65x
🐘 PostgresNitro0.073s (+15.6% 🔺)1.013s (~)0.940s101.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express0.271s (-28.1% 🟢)2.235s (-2.6%)1.964s101.00x
▲ VercelNitro0.319s (-14.9% 🟢)2.203s (-2.0%)1.883s101.18x
▲ VercelNext.js (Turbopack)0.351s (+20.6% 🔺)2.531s (+33.8% 🔺)2.180s101.29x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.094s (-0.7%)2.006s (~)0.912s101.00x
💻 LocalNitro1.095s (+0.9%)2.007s (~)0.912s101.00x
🐘 PostgresNitro1.108s (-1.1%)2.009s (~)0.901s101.01x
🐘 PostgresExpress1.110s (~)2.008s (~)0.898s101.02x
💻 LocalNext.js (Turbopack)1.132s (-1.4%)2.007s (~)0.875s101.03x
🐘 PostgresNext.js (Turbopack)1.141s (~)2.009s (~)0.868s101.04x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.723s (-11.0% 🟢)3.716s (+1.1%)1.993s101.00x
▲ VercelNext.js (Turbopack)1.761s (~)3.708s (~)1.947s101.02x
▲ VercelNitro1.788s (-22.8% 🟢)3.409s (-14.6% 🟢)1.621s101.04x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro10.527s (~)11.022s (~)0.495s31.00x
💻 LocalExpress10.566s (~)11.023s (~)0.457s31.00x
🐘 PostgresExpress10.585s (~)11.023s (~)0.438s31.01x
🐘 PostgresNitro10.598s (+0.6%)11.022s (~)0.424s31.01x
💻 LocalNext.js (Turbopack)10.812s (~)11.022s (~)0.210s31.03x
🐘 PostgresNext.js (Turbopack)10.829s (~)11.017s (~)0.189s31.03x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express13.691s (-24.2% 🟢)15.347s (-23.1% 🟢)1.656s21.00x
▲ VercelNitro13.728s (-27.6% 🟢)15.441s (-27.5% 🟢)1.713s21.00x
▲ VercelNext.js (Turbopack)14.688s (+2.6%)16.926s (+3.9%)2.238s21.07x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro13.690s (~)14.027s (~)0.337s51.00x
🐘 PostgresExpress13.756s (-0.7%)14.019s (~)0.263s51.00x
💻 LocalExpress13.798s (~)14.028s (~)0.231s51.01x
🐘 PostgresNitro13.832s (~)14.023s (~)0.191s51.01x
💻 LocalNext.js (Turbopack)14.383s (~)15.030s (~)0.647s41.05x
🐘 PostgresNext.js (Turbopack)14.396s (~)15.017s (~)0.621s41.05x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express23.028s (-4.9%)25.270s (-3.5%)2.242s31.00x
▲ VercelNext.js (Turbopack)24.609s (-30.5% 🟢)26.392s (-28.3% 🟢)1.783s31.07x
▲ VercelNitro24.684s (-20.8% 🟢)26.313s (-19.2% 🟢)1.629s31.07x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro12.373s (+2.5%)13.027s (+1.1%)0.654s71.00x
💻 LocalExpress12.444s (-0.8%)13.024s (~)0.580s71.01x
🐘 PostgresExpress12.512s (+0.6%)13.016s (~)0.503s71.01x
🐘 PostgresNitro12.712s (+1.6%)13.021s (~)0.310s71.03x
💻 LocalNext.js (Turbopack)13.662s (~)14.027s (~)0.365s71.10x
🐘 PostgresNext.js (Turbopack)13.897s (~)14.308s (+1.0%)0.411s71.12x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro27.276s (-29.9% 🟢)29.146s (-28.8% 🟢)1.870s41.00x
▲ VercelNext.js (Turbopack)27.460s (-22.0% 🟢)29.404s (-20.4% 🟢)1.944s41.01x
▲ VercelExpress28.389s (-19.0% 🟢)30.612s (-17.0% 🟢)2.223s31.04x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.204s (-0.8%)2.007s (~)0.803s151.00x
💻 LocalNitro1.222s (+4.5%)2.006s (~)0.784s151.01x
💻 LocalExpress1.231s (+5.6% 🔺)2.007s (~)0.776s151.02x
🐘 PostgresNitro1.251s (+4.4%)2.009s (~)0.757s151.04x
🐘 PostgresNext.js (Turbopack)1.256s (-2.3%)2.007s (~)0.751s151.04x
💻 LocalNext.js (Turbopack)1.405s (+8.4% 🔺)2.006s (~)0.602s151.17x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.571s (+15.9% 🔺)4.329s (+11.4% 🔺)1.757s81.00x
▲ VercelNitro3.123s (+7.3% 🔺)4.469s (+1.4%)1.346s71.21x
▲ VercelNext.js (Turbopack)3.608s (+35.3% 🔺)4.968s (+21.2% 🔺)1.360s71.40x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.309s (-8.1% 🟢)2.008s (-16.1% 🟢)0.700s151.00x
🐘 PostgresNitro1.332s (-2.4%)2.074s (-17.3% 🟢)0.742s151.02x
🐘 PostgresNext.js (Turbopack)1.452s (-11.3% 🟢)2.075s (-13.3% 🟢)0.623s151.11x
💻 LocalExpress1.987s (+22.2% 🔺)2.592s (+29.2% 🔺)0.604s121.52x
💻 LocalNitro2.051s (+32.2% 🔺)2.507s (+24.7% 🔺)0.456s121.57x
💻 LocalNext.js (Turbopack)2.359s (+23.5% 🔺)3.008s (+31.2% 🔺)0.649s101.80x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.443s (-13.1% 🟢)5.587s (-1.5%)2.143s61.00x
▲ VercelNitro3.655s (-9.7% 🟢)5.233s (-8.2% 🟢)1.577s61.06x
▲ VercelNext.js (Turbopack)4.245s (-5.5% 🟢)5.891s (-9.1% 🟢)1.646s61.23x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.468s (-16.7% 🟢)3.886s (~)2.417s81.00x
🐘 PostgresNitro1.470s (-7.8% 🟢)3.885s (-3.1%)2.415s81.00x
🐘 PostgresNext.js (Turbopack)2.642s (-16.3% 🟢)3.456s (-19.7% 🟢)0.814s91.80x
💻 LocalNitro4.340s (+26.8% 🔺)5.013s (+25.0% 🔺)0.673s62.96x
💻 LocalExpress5.253s (+20.2% 🔺)5.679s (+16.7% 🔺)0.425s63.58x
💻 LocalNext.js (Turbopack)6.408s (+11.0% 🔺)7.019s (+12.9% 🔺)0.611s54.36x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro4.491s (-26.7% 🟢)6.303s (-24.4% 🟢)1.812s51.00x
▲ VercelNext.js (Turbopack)4.524s (-19.6% 🟢)6.497s (-15.8% 🟢)1.973s51.01x
▲ VercelExpress4.646s (-15.6% 🟢)7.023s (-11.4% 🟢)2.378s51.03x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.202s (-1.3%)2.007s (~)0.805s151.00x
🐘 PostgresNitro1.214s (~)2.008s (~)0.794s151.01x
💻 LocalExpress1.228s (-22.1% 🟢)2.006s (~)0.778s151.02x
💻 LocalNitro1.268s (+5.6% 🔺)2.006s (~)0.738s151.06x
🐘 PostgresNext.js (Turbopack)1.271s (-0.8%)2.008s (~)0.737s151.06x
💻 LocalNext.js (Turbopack)1.449s (+3.6%)2.006s (~)0.557s151.21x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.668s (-25.2% 🟢)4.041s (-23.6% 🟢)1.373s81.00x
▲ VercelNext.js (Turbopack)2.876s (-22.7% 🟢)4.782s (-8.6% 🟢)1.907s71.08x
▲ VercelExpress3.339s (+43.7% 🔺)5.449s (+43.2% 🔺)2.110s61.25x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Nitro1.304s (-9.6% 🟢)2.075s (-13.3% 🟢)0.771s151.00x
🐘 PostgresExpress1.310s (-4.1%)2.007s (-13.3% 🟢)0.697s151.00x
🐘 PostgresNext.js (Turbopack)1.436s (-6.9% 🟢)2.076s (-6.6% 🟢)0.640s151.10x
💻 LocalNitro1.935s (+14.3% 🔺)2.293s (+14.2% 🔺)0.358s141.48x
💻 LocalExpress2.003s (+5.5% 🔺)2.393s (+11.3% 🔺)0.390s131.54x
💻 LocalNext.js (Turbopack)2.369s (+9.3% 🔺)3.009s (~)0.639s101.82x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.899s (+6.3% 🔺)4.683s (+2.5%)1.785s71.00x
▲ VercelNitro3.978s (+42.1% 🔺)5.734s (+37.1% 🔺)1.756s61.37x
▲ VercelNext.js (Turbopack)4.216s (-5.6% 🟢)5.944s (-3.4%)1.728s61.45x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.452s (-19.3% 🟢)4.014s (~)2.561s81.00x
🐘 PostgresNitro1.589s (-12.0% 🟢)3.678s (-14.4% 🟢)2.090s91.09x
🐘 PostgresNext.js (Turbopack)2.136s (-47.5% 🟢)3.454s (-22.3% 🟢)1.318s91.47x
💻 LocalExpress5.580s (+21.6% 🔺)6.014s (+20.0% 🔺)0.434s53.84x
💻 LocalNitro5.616s (+25.0% 🔺)6.017s (+20.1% 🔺)0.401s53.87x
💻 LocalNext.js (Turbopack)6.852s (+18.0% 🔺)7.416s (+15.6% 🔺)0.563s54.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.610s (-57.2% 🟢)5.562s (-45.7% 🟢)1.952s61.00x
▲ VercelNitro4.046s (-70.8% 🟢)5.724s (-63.4% 🟢)1.678s61.12x
▲ VercelNext.js (Turbopack)4.124s (+9.0% 🔺)6.272s (+19.9% 🔺)2.147s51.14x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.605s (+6.0% 🔺)1.041s (+3.5%)0.436s581.00x
💻 LocalExpress0.616s (-1.5%)1.005s (-1.7%)0.389s601.02x
💻 LocalNitro0.629s (+25.8% 🔺)1.039s (+3.1%)0.410s581.04x
🐘 PostgresNitro0.692s (+16.5% 🔺)1.078s (+3.6%)0.386s561.14x
🐘 PostgresNext.js (Turbopack)0.858s (+2.7%)1.041s (+1.7%)0.183s581.42x
💻 LocalNext.js (Turbopack)0.860s (-3.1%)1.005s (-3.3%)0.144s601.42x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.774s (-42.9% 🟢)5.457s (-34.5% 🟢)1.683s111.00x
▲ VercelNext.js (Turbopack)3.815s (-39.5% 🟢)5.390s (-33.8% 🟢)1.575s121.01x
▲ VercelExpress4.488s (-1.3%)6.377s (-1.6%)1.889s101.19x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.375s (+1.4%)2.052s (+1.2%)0.678s441.00x
🐘 PostgresNitro1.466s (+5.6% 🔺)2.030s (~)0.564s451.07x
💻 LocalNitro1.485s (+25.1% 🔺)2.006s (~)0.521s451.08x
💻 LocalExpress1.530s (+2.8%)2.007s (~)0.476s451.11x
🐘 PostgresNext.js (Turbopack)1.989s (+2.5%)2.308s (+11.2% 🔺)0.319s401.45x
💻 LocalNext.js (Turbopack)2.090s (-0.6%)2.944s (-2.1%)0.854s311.52x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express10.297s (-41.1% 🟢)12.369s (-35.5% 🟢)2.072s81.00x
▲ VercelNitro10.774s (-8.8% 🟢)12.396s (-12.5% 🟢)1.621s81.05x
▲ VercelNext.js (Turbopack)11.321s (-16.7% 🟢)13.637s (-11.0% 🟢)2.316s71.10x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express2.654s (-2.1%)3.058s (-1.7%)0.404s401.00x
🐘 PostgresNitro2.903s (+3.7%)3.280s (+3.6%)0.376s371.09x
💻 LocalExpress3.275s (+2.4%)4.010s (~)0.735s301.23x
💻 LocalNitro3.362s (+23.1% 🔺)4.010s (+24.4% 🔺)0.648s301.27x
🐘 PostgresNext.js (Turbopack)3.983s (+2.9%)4.253s (+4.3%)0.270s291.50x
💻 LocalNext.js (Turbopack)4.363s (~)5.010s (~)0.647s241.64x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express18.286s (-29.9% 🟢)20.532s (-26.7% 🟢)2.246s61.00x
▲ VercelNitro18.713s (-28.9% 🟢)20.382s (-27.5% 🟢)1.669s61.02x
▲ VercelNext.js (Turbopack)20.206s (-19.2% 🟢)22.420s (-16.3% 🟢)2.214s61.11x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.232s (~)1.006s (~)0.773s601.00x
🐘 PostgresNitro0.237s (+0.7%)1.006s (~)0.769s601.02x
🐘 PostgresNext.js (Turbopack)0.297s (+1.1%)1.023s (+1.7%)0.726s591.28x
💻 LocalExpress0.400s (-9.2% 🟢)1.005s (~)0.605s601.72x
💻 LocalNitro0.417s (+11.8% 🔺)1.004s (~)0.588s601.79x
💻 LocalNext.js (Turbopack)0.633s (+8.8% 🔺)1.004s (~)0.371s602.73x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.787s (+15.3% 🔺)3.598s (+25.4% 🔺)1.811s181.00x
▲ VercelNitro1.933s (+49.7% 🔺)3.615s (+25.8% 🔺)1.682s171.08x
▲ VercelNext.js (Turbopack)2.387s (+27.6% 🔺)4.344s (+11.8% 🔺)1.956s141.34x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.331s (-6.9% 🟢)1.006s (-3.3%)0.675s901.00x
🐘 PostgresNitro0.356s (+4.1%)1.006s (-1.1%)0.650s901.08x
🐘 PostgresNext.js (Turbopack)0.476s (-11.7% 🟢)1.103s (-1.3%)0.628s831.44x
💻 LocalNitro2.182s (+43.2% 🔺)2.737s (+27.2% 🔺)0.555s336.59x
💻 LocalExpress2.222s (+6.3% 🔺)2.738s (+3.1%)0.517s336.71x
💻 LocalNext.js (Turbopack)2.493s (+7.0% 🔺)3.344s (+8.7% 🔺)0.851s277.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.638s (-9.8% 🟢)4.553s (-5.1% 🟢)1.915s201.00x
▲ VercelNitro2.705s (-1.5%)4.386s (-8.3% 🟢)1.681s211.03x
▲ VercelNext.js (Turbopack)3.092s (+1.6%)4.802s (~)1.710s191.17x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.526s (-7.4% 🟢)1.059s (-13.0% 🟢)0.532s1141.00x
🐘 PostgresNitro0.570s (~)1.078s (-12.5% 🟢)0.508s1121.08x
🐘 PostgresNext.js (Turbopack)1.918s (-26.7% 🟢)2.763s (-20.4% 🟢)0.845s443.64x
💻 LocalNitro9.521s (+50.8% 🔺)10.445s (+54.9% 🔺)0.924s1218.09x
💻 LocalExpress10.198s (+22.8% 🔺)11.029s (+24.2% 🔺)0.831s1219.38x
💻 LocalNext.js (Turbopack)10.284s (-4.4%)11.663s (+0.8%)1.379s1119.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.245s (-38.5% 🟢)5.031s (-28.3% 🟢)1.786s241.00x
▲ VercelExpress3.483s (-36.2% 🟢)5.687s (-19.8% 🟢)2.204s221.07x
▲ VercelNext.js (Turbopack)4.699s (-2.9%)6.708s (+2.7%)2.008s181.45x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.165s (-1.1%)2.000s (~)0.001s (+9.1% 🔺)2.009s (~)0.844s101.00x
🐘 PostgresNitro1.181s (+1.4%)1.995s (~)0.001s (+30.0% 🔺)2.011s (~)0.831s101.01x
💻 LocalNitro1.181s (+5.7% 🔺)2.005s (~)0.010s (-62.9% 🟢)2.017s (-1.0%)0.836s101.01x
💻 LocalExpress1.185s (+3.0%)2.005s (~)0.012s (+19.8% 🔺)2.019s (~)0.835s101.02x
💻 LocalNext.js (Turbopack)1.216s (~)2.003s (~)0.013s (+2.4%)2.020s (~)0.803s101.04x
🐘 PostgresNext.js (Turbopack)1.232s (~)2.002s (~)0.001s (-15.4% 🟢)2.011s (~)0.779s101.06x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.206s (-8.2% 🟢)3.426s (-6.9% 🟢)0.602s (-18.8% 🟢)4.513s (-7.4% 🟢)2.307s101.00x
▲ VercelNext.js (Turbopack)2.304s (-8.6% 🟢)3.523s (-7.8% 🟢)0.772s (-4.3%)4.790s (-5.8% 🟢)2.486s101.04x
▲ VercelNitro2.331s (-11.7% 🟢)3.158s (-17.0% 🟢)1.253s (+57.9% 🔺)4.805s (-4.9%)2.474s101.06x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro1.572s (+9.3% 🔺)2.011s (~)0.012s (-32.0% 🟢)2.025s (~)0.453s301.00x
🐘 PostgresExpress1.577s (~)2.004s (~)0.005s (+5.7% 🔺)2.026s (~)0.449s301.00x
💻 LocalExpress1.583s (+1.0%)2.010s (~)0.014s (+4.9%)2.025s (~)0.442s301.01x
🐘 PostgresNitro1.602s (+1.5%)2.008s (~)0.005s (-5.6% 🟢)2.027s (~)0.425s301.02x
💻 LocalNext.js (Turbopack)1.740s (-0.5%)2.010s (~)0.013s (+1.3%)2.025s (~)0.285s301.11x
🐘 PostgresNext.js (Turbopack)1.892s (+5.9% 🔺)2.011s (~)0.005s (+5.8% 🔺)2.029s (~)0.137s301.20x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Next.js (Turbopack)6.842s (-19.5% 🟢)8.374s (-14.7% 🟢)0.220s (-16.3% 🟢)9.079s (-14.1% 🟢)2.236s71.00x
▲ VercelNitro6.905s (-3.6%)7.768s (-10.3% 🟢)0.455s (+91.9% 🔺)8.766s (-6.9% 🟢)1.861s71.01x
▲ VercelExpress8.068s (+26.4% 🔺)9.373s (+20.1% 🔺)0.307s (+71.5% 🔺)10.412s (+23.1% 🔺)2.345s61.18x

🔍 Observability: Next.js (Turbopack) | Nitro | Express

10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.789s (+3.3%)1.101s (+7.5% 🔺)0.000s (-100.0% 🟢)1.118s (+6.0% 🔺)0.329s541.00x
🐘 PostgresNitro0.795s (+2.9%)1.044s (~)0.000s (-3.5%)1.061s (-3.0%)0.265s571.01x
🐘 PostgresNext.js (Turbopack)0.986s (-3.3%)1.397s (-4.6%)0.000s (-100.0% 🟢)1.407s (-4.4%)0.421s431.25x
💻 LocalExpress1.533s (+12.5% 🔺)2.014s (~)0.000s (+16.7% 🔺)2.016s (~)0.483s301.94x
💻 LocalNitro1.568s (+45.8% 🔺)2.014s (+9.6% 🔺)0.000s (-56.8% 🟢)2.016s (+9.5% 🔺)0.448s301.99x
💻 LocalNext.js (Turbopack)1.916s (+26.7% 🔺)2.193s (+8.9% 🔺)0.000s (+7.1% 🔺)2.196s (+8.9% 🔺)0.280s282.43x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.961s (-21.2% 🟢)4.002s (-25.7% 🟢)0.000s (-21.4% 🟢)4.479s (-23.6% 🟢)1.518s141.00x
▲ VercelExpress3.140s (+3.6%)4.516s (-1.1%)0.000s (NaN%)5.069s (+0.6%)1.929s121.06x
▲ VercelNext.js (Turbopack)3.540s (+2.7%)5.016s (+5.2% 🔺)0.000s (-100.0% 🟢)5.553s (+5.5% 🔺)2.013s111.20x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.691s (+4.7%)2.302s (+11.5% 🔺)0.000s (NaN%)2.316s (+10.9% 🔺)0.625s261.00x
🐘 PostgresNitro1.779s (+14.7% 🔺)2.381s (+11.6% 🔺)0.000s (+11.5% 🔺)2.394s (+11.5% 🔺)0.615s261.05x
🐘 PostgresNext.js (Turbopack)2.251s (+6.6% 🔺)2.651s (+2.5%)0.000s (-100.0% 🟢)2.664s (+2.6%)0.413s231.33x
💻 LocalNitro4.321s (+95.8% 🔺)4.717s (+72.0% 🔺)0.001s (+12.8% 🔺)4.729s (+71.7% 🔺)0.409s132.56x
💻 LocalExpress4.772s (+62.9% 🔺)5.362s (+46.1% 🔺)0.001s (+98.3% 🔺)5.367s (+46.1% 🔺)0.595s122.82x
💻 LocalNext.js (Turbopack)5.632s (+94.2% 🔺)6.226s (+85.4% 🔺)0.000s (-60.0% 🟢)6.235s (+85.4% 🔺)0.602s103.33x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express4.310s (-24.3% 🟢)6.054s (-16.2% 🟢)0.000s (+Infinity% 🔺)6.546s (-14.9% 🟢)2.236s101.00x
▲ VercelNext.js (Turbopack)4.772s (-34.2% 🟢)6.201s (-26.1% 🟢)0.000s (-100.0% 🟢)6.679s (-24.5% 🟢)1.907s101.11x
▲ VercelNitro4.835s (-7.9% 🟢)5.805s (-10.1% 🟢)0.000s (+Infinity% 🔺)6.308s (-8.9% 🟢)1.473s101.12x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalNitro12/21
🐘 PostgresExpress19/21
▲ VercelExpress14/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres17/21
Next.js (Turbopack)🐘 Postgres14/21
Nitro🐘 Postgres13/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run

@pranaygp

Copy link
Copy Markdown
ContributorAuthor

CI failure triage — pre-existing Vercel-prod e2e flake (not a regression)

The two red checks (E2E Vercel Prod Tests (example), E2E Vercel Prod Tests (fastify), which roll up into E2E Required Check) are the shared Vercel-prod timing flake, not caused by this PR. Evidence:

Scope is wrong for a hydration regression. This PR only memoizes primitive step-result hydration. A determinism/stale-value bug there would surface across all worlds — yet every local suite is 100% green:

SuitePassedFailed
▲ Vercel Production14402
💻 Local Development19090
📦 Local Production19090
🐘 Local Postgres18950
🪟 Windows1520

Only 2 failures, only on Vercel Production.

The two failing tests are unrelated to result hydration, and are abort/hook timing races:

  1. exampleAbortController > abortFromStepWorkflow: step abort cancels an in-flight sibling step. The run completed successfully; the assertion failed only because the abort lost a race. From the run diagnostics (wrun_01KV9MZ1Y00N90XFCKFTSPX5N1):

    +2.2s step_completed (longStep) <- sibling finished on its own
    +3.0s hook_received <- abort signal arrived AFTER
    

    The sibling longStep self-completed (2.2s) before the abort hook arrived (3.0s) under Vercel-prod queue/network latency, so there was no in-flight step left to cancel. Step results hydrated fine; this is purely environmental latency.

  2. fastifystartFromWorkflow - calling start() directly inside a workflow function with hook communication. This exact test passed (4128ms) in the example job of this same commit (a56f5c90b) — a textbook cross-run flake.

The same test is red on plain main, without this change. On main run 27704378960 (commit 2acf13cc7):

  • E2E Vercel Prod Tests (tanstack-start)abortFromStepWorkflow: step abort cancels an in-flight sibling step (the identical test that failed here on example)
  • E2E Vercel Prod Tests (nextjs-turbopack)distributedAbortController - reconnect to existing controller

And on main run 27657696161 (cb181392b, the commit this branch is based on): E2E Vercel Prod Tests (fastify)hookWithSleepFinalStepWorkflow. The set of red workbenches rotates run-to-run — the signature of environment flakiness, not a code regression.

Local verification of this branch (rebuilt @workflow/core first): cd packages/core && pnpm test1249 passed / 56 files, 0 failures, including the determinism/ordering replay tests in workflow.test.ts and the new step-hydration-cache.test.ts (10) + step-hydration-memoization.test.ts (3). The memoization tests assert byte-identical delivery ordering on cache hits and that objects re-hydrate fresh each replay.

Re-running the e2e jobs should clear them. No code change is warranted.

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — the O(N²)→O(N) hydration memoization, done with the right safety bias

This is the most safety-sensitive of the four (it's the only one that caches a value across replays), and the design lands on the conservative side of every judgment call, which is exactly right for replay determinism.

The primitives-only decision is the crux and it's correct.hydrateStepReturnValue (devalue.parse) returns a fresh object graph each call, and each replay runs in a fresh VM, so today the workflow gets a brand-new value every replay. Caching and returning the same object reference would let const r = await step(); r.count++ observe a prior replay's mutation — silent divergence. The alternatives are both worse: structuredClone is lossy for reviver-produced specials (stream handles, step-fn proxies, Request/Response, AbortController/Signal) and still O(size). Restricting the cache to primitives (immutable, compared by value) makes "share the reference" provably indistinguishable from re-parsing, and non-primitives fall through to a full re-hydrate every replay — preserving current behavior exactly. Trading the object-case optimization for airtight determinism is the right call.

What I verified in the integration:

  • Surgical wrap: only the await hydrateStepReturnValue(...) call is replaced; pendingDeliveries++/--, the ctx.promiseQueue.then(...) slot, and resolve(...) are byte-for-byte unchanged. The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the identical position in the ordered delivery chain a re-hydrate would have — preserving the pendingDeliveries-gated suspension/barrier/Promise.race determinism.
  • Lifetime: cache is created once per invocation in runtime.ts (outside the per-iteration context), threaded through runWorkflow, never shared across runs. The optional param/context field degrades to re-hydrating-every-replay for harnesses that omit it.
  • Keying by step_completedeventId (stable, world-assigned, same immutable bytes every replay) is sound, and has() rather than get() !== undefined correctly treats a memoized undefined result as a hit.
  • Errors never cached — a rejected hydrate re-attempts next replay, no parked rejected promise.

Built @workflow/core; full suite green (1249) including the new step-hydration-cache.test.ts / step-hydration-memoization.test.ts, async-deserialization-ordering.test.ts, and workflow.test.ts.

One forward-looking note (non-blocking): the object case is left on the table by design. If it's ever revisited, the safe path is deep-freeze + share frozen graphs — but only after handling the reviver-produced specials, so I agree with deferring it. LGTM.

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

// Only memoize values that are safe to return by reference across replays.
// Non-primitives fall through and are re-hydrated fresh on every replay.
if (isMemoizablePrimitive(value)) {
cache.set(eventId, value);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

The per-run stepHydrationCache is never size-bounded or evicted: it grows one entry per primitive-returning completed step and lives for the entire invocation. The cost worth calling out (the PR doesn't) is the new residency — the decrypted/devalue-parsed plaintext of each cached primitive is now held for the whole invocation, on top of the serialized bytes already retained in cachedEvents. For a long sequential workflow whose steps return large strings, that roughly doubles peak retained memory for those results.

The dominant residency (the full event log in cachedEvents) already exists, so this isn't blocking, but a byte-size threshold would be a cheap safeguard — large primitives are exactly the cheap-to-re-hydrate case relative to their footprint, so letting them fall through to the existing re-hydrate path costs little and bounds the worst case. Ideally with a test asserting the bound.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — capped it, with a test. Pushed in a60baad.

What changed (step-hydration-cache.ts):

  • Added MAX_MEMOIZED_PRIMITIVE_LENGTH = 4096 and extended isMemoizablePrimitive so a string/bigint longer than 4 KiB is treated as non-memoizable. Those are the only primitive types that can carry a large payload — number/boolean/null/undefined/symbol are inherently small, so they're never length-checked. Oversized values now fall through to the existing per-replay re-hydrate path, exactly as you suggested: large primitives are cheap to re-hydrate relative to their footprint, so this caps the doubled-residency worst case at negligible cost.
  • Documented the memory characteristic on the cache module: per-invocation lifetime (fresh Map per run in runtime.ts, GC'd when the invocation returns), bounded by the number of primitive-returning completed steps, primitives-only, now byte-bounded.

Tests (step-hydration-cache.test.ts, +4):isMemoizablePrimitive true at the bound / false beyond it (string and bigint), and an end-to-end assertion that an oversized string re-hydrates on every replay and cache.size === 0 (the bound assertion you asked for); plus an at-bound string is a cache hit.

The cap only ever reduces what gets cached, so determinism is untouched — oversized values just take the already-correct re-hydrate path. Full core suite green (1253, incl. the ordering/determinism + memoization suites); biome + tsc clean.

On consistency with #2471 (the sibling scriptCache): noting the distinction since they're bounded for different reasons. #2471's cache is process-wide and monotonic across the whole process — in dev/watch it pins every historical bundle string (hundreds of MB over a session), which is a genuine regression vs. the prior keep-only-latest behavior, hence the Blocking bound there. This cache is per-invocation and freed wholesale when the run returns, so it can never accumulate across runs; the only real cost is the doubled residency for large primitives during one run, which the size cap here now bounds. Different scope, different severity, but both bounded now.

Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-hydration
* origin/main:
perf(core): lazy inline step start (save one world round-trip per step) (#2478)
perf(core): skip per-step events.list via inline event-log delta (#2475)
Version Packages (beta) (#2491)
[world-vercel] Honor hasMore flag from v4 list pagination endpoint (#2486)
Version Packages (beta) (#2451)
Fix Next workflow module specifier root (#2455)
[world-vercel] Send remoteRefBehavior=lazy on v4 metadata-only event listings (#2415)
[swc-plugin] Fix eager discovery for object property steps (#2484)
fix(web-shared): align attributes panel styling (#2483)
[web-shared] Auto-scroll trace viewer on J/K span navigation (#2366)
fix(web): render restarted step segment as solid gray, not running stripes (#2480)
fix(web-shared): use solid gray for queued trace segment (#2474)
Add trace viewer span markers for hooks and attributes (#2452)
test: support Vercel protection bypass secret in e2e headers (#2458)
fix(core): bump payload-compression cutoff to 5.0.0-beta.18 (#2470)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts:
#	packages/core/src/runtime.ts
@pranaygp
pranaygp enabled auto-merge (squash) June 18, 2026 06:24
@pranaygp
pranaygp disabled auto-merge June 22, 2026 20:30
@pranaygp
pranaygp merged commit 66ca0dc into mainJun 22, 2026
118 of 121 checks passed
@pranaygp
pranaygp deleted the pgp/perf-memoize-step-hydration branch June 22, 2026 20:30
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
github-actionsBot added a commit that referenced this pull request Jun 22, 2026
…#2472)
* perf(core): memoize step return value hydration across replays
The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): bound memoized step-hydration cache by primitive size
Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #2563. Merge conflicts were resolved by AI — please review carefully. (backport job run)

pranaygp added a commit that referenced this pull request Jun 22, 2026
…testing
* origin/main:
Version Packages (beta) (#2540)
perf(core): memoize step return value hydration across inline replays (#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (#2412)
Amend lazy discovery removal changeset (#2560)
[docs] Document minimum SDK version for using hook.getConflict (#2423)
Update default CODEOWNERS (#2556)
Optimize and fix the default eager build mode (#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (#2547)" (#2554)
[core] Turbo mode: fast-path the first invocation (#2526)
Remove lazy discovery from workflow/next (#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (#2547)
feat(docs): add eve and AI SDK to product switcher (#2543)
[vitest] Fix local imports failing to load in test step bundles (#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (#2324)
Version Packages (beta) (#2495)
otel(world-vercel): inject trace context on v4 event requests (#2533)
Bump undici to 7.28.0 (#2534)
Default source maps to dev-on / prod-off (#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (#2527)
perf(core): parallel inline steps + optimistic lazy step start (#2516)
pranaygp added a commit to marcopiraccini/workflow that referenced this pull request Jun 22, 2026
* origin/main: (120 commits)
Version Packages (beta) (vercel#2540)
perf(core): memoize step return value hydration across inline replays (vercel#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (vercel#2412)
Amend lazy discovery removal changeset (vercel#2560)
[docs] Document minimum SDK version for using hook.getConflict (vercel#2423)
Update default CODEOWNERS (vercel#2556)
Optimize and fix the default eager build mode (vercel#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)" (vercel#2554)
[core] Turbo mode: fast-path the first invocation (vercel#2526)
Remove lazy discovery from workflow/next (vercel#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)
feat(docs): add eve and AI SDK to product switcher (vercel#2543)
[vitest] Fix local imports failing to load in test step bundles (vercel#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (vercel#2324)
Version Packages (beta) (vercel#2495)
otel(world-vercel): inject trace context on v4 event requests (vercel#2533)
Bump undici to 7.28.0 (vercel#2534)
Default source maps to dev-on / prod-off (vercel#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (vercel#2527)
perf(core): parallel inline steps + optimistic lazy step start (vercel#2516)
...
VaguelySerious pushed a commit that referenced this pull request Jun 22, 2026
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pranaygp@TooTallNate@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

perf(core): memoize step return value hydration across inline replays - #2472

Merged
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration
Jun 22, 2026
Merged

perf(core): memoize step return value hydration across inline replays#2472
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration

Conversation

@pranaygp

@pranaygppranaygp commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

The inline replay loop (runtime.tsrunWorkflow, workflow.ts) re-executes the workflow body and re-consumes the full event log on every iteration. For each already-completed step, the step consumer (step.ts, step_completed path) re-ran hydrateStepReturnValue — AES-GCM decrypt + devalue-parse of the serialized result — on every replay, even though that exact result was already hydrated on every prior replay.

For a sequential workflow of N steps, replay K hydrates K results, so the aggregate cost across a single invocation is O(N²) decrypt+parse operations.

This PR adds a per-run memoization cache so a completed step's hydrated result is returned in O(1) on subsequent replays within the same invocation, making the aggregate cost O(N).

Before / after

  • Before: replay 1 hydrates 1 result, replay 2 hydrates 2, …, replay K hydrates K → Σ = O(N²) decrypt+parse over a sequential run.
  • After: each completed step's result is hydrated once and memoized for the rest of the invocation → O(N) total. Replay K hydrates only the one newly-completed step; the K−1 prior results are cache hits.

Cache scope & keying

  • Lifetime / scope: owned by the inline loop in runtime.ts (created once per run invocation, alongside cachedEvents), threaded into runWorkflow(..., stepHydrationCache?) and stored on WorkflowOrchestratorContext.stepHydrationCache. A fresh context is created each loop iteration, so the cache deliberately lives outside the per-iteration context to survive across iterations of the same run. It is never shared across unrelated runs or process-level invocations.
  • Keying: by the persisted step_completed event's eventId — a stable, world-assigned id. The same event carries the same immutable serialized bytes across every replay, so a hit is guaranteed to correspond to identical input.
  • Optional / backward compatible: the parameter and context field are optional. Callers/harnesses that omit them (and the many runWorkflow(...) unit tests) degrade to re-hydrating every replay — identical to previous behavior.

Memory characteristic

A cached entry holds the decrypted/devalue-parsed plaintext of a step result, retained for the rest of the invocation on top of the serialized bytes already held in cachedEvents — so for large primitive results it roughly doubles peak retained memory for those results during the run. This residual is:

  • Scoped to one invocation — the Map is created per run and GC'd when the invocation returns; nothing accumulates across runs or process-level invocations (a much weaker concern than a process-wide cache, where the dominant residency — the full event log in cachedEvents — already exists for the same lifetime).
  • Bounded by the primitive-returning completed-step count — at most one small entry per such step.
  • Byte-bounded. Most primitives (numbers, booleans, null/undefined, symbols, short ids/strings) are tiny. The only primitive that can be large is a string (or a pathologically long bigint), so a string/bigint result longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) is not memoized — it falls through to the existing per-replay re-hydrate path. Large payloads are cheap to re-hydrate relative to their footprint, so this caps the worst case at negligible cost. The cap only ever reduces what is cached, so deterministic replay is unaffected.

Ordering safety analysis

The cache lookup replaces only the await hydrateStepReturnValue(...) call inside the existing ctx.promiseQueue.then(async () => { ... }) slot. Everything else is byte-for-byte unchanged:

  • ctx.pendingDeliveries++ / -- accounting is untouched.
  • The hydrate (or cache hit) still happens inside the same serial promiseQueue slot, at the same log position, and still resolves via the same resolve(...).
  • The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the exact position in the ordered delivery chain a re-hydrate would have.

So delivery order, pendingDeliveries-gated suspensions, the pendingDeliveryBarriers / awaitEarlierDeliveries machinery, and Promise.race/Promise.all replay determinism are all unaffected. Hook, wait, and abort hydration paths are intentionally not cached (they're the ordering-sensitive paths and not the O(N²) hotspot).

Identity / immutability safety

hydrateStepReturnValue (devalue.parse) returns a fresh object graph on every call, and each replay iteration runs in a fresh workflow VM. Today the workflow therefore receives a brand-new value on every replay. If we cached and returned the same object reference across replays, workflow code that mutates a step result (const r = await step(); r.count++) would observe a previous replay's mutation on the next replay — a non-deterministic divergence. (structuredClone on each hit is both lossy — revivers reconstruct stream handles, step-function proxies, Request/Response, and AbortController/AbortSignal class instances — and still O(size).)

Decision: only primitives are memoized (string, number, boolean, bigint, symbol, null, undefined). Primitives are immutable and compared by value, so sharing the reference is provably indistinguishable from re-parsing. Any non-primitive result falls through to a full re-hydrate every replay, preserving current behavior exactly. Errors are never cached, so a rejected hydrate re-attempts on the next replay (no parked rejected promise). This trades away the optimization in the object-returning case to keep deterministic replay airtight — correctness over speed.

What I verified

  • Unit:step-hydration-cache.test.ts (14 tests: primitive detection, memoization, non-primitive eviction/fresh-object, falsy primitives, keying, error non-caching, no-cache passthrough, plus the size-bound — at-bound string is a hit, oversized string/bigint are not memoized and cache.size stays 0) and step-hydration-memoization.test.ts (3 tests through the real createUseStep consumer: hydrate-skipped-on-replay-2 via spy, event-log ordering preserved on cache hits, fresh object per replay for object results).
  • Full core suite:cd packages/core && pnpm test1253 passed / 56 files, including async-deserialization-ordering.test.ts, workflow.test.ts (79 tests), runtime.test.ts, hook-sleep-interaction, abort-consistency. No regressions.
  • Build / format / typecheck:pnpm build (full repo, 27/27), @workflow/core build + tsc --noEmit clean; Biome format applied; new files Biome-clean (the only lint errors were import-ordering, auto-fixed; remaining warnings are pre-existing noExcessiveCognitiveComplexity on functions I only edited).
  • E2E (local nextjs-turbopack dev server, the determinism-sensitive subset): promiseAllWorkflow, promiseRaceWorkflow, promiseAnyWorkflow, sleepWinsRaceWorkflow, stepWinsRaceWorkflow, promiseRaceStressTestWorkflow, hookWorkflow, webhookWorkflow, parallel-steps-then-webhook replay race, sleepingWorkflow, parallelSleepWorkflow, retry/error/catchability suite, fetchWorkflowall passed.

Risks / deferred

  • Only primitive step results are accelerated; object-returning steps still re-hydrate each replay (intentional, for determinism). A future safe extension could deep-freeze + share frozen object graphs, but that needs care around reviver-produced special objects and is out of scope here.
  • Large (>4 KiB) string/bigint results are intentionally not memoized to bound peak retained memory (see Memory characteristic); they re-hydrate each replay.
  • Hook/wait/abort hydration paths are uncached by design.

🤖 Generated with Claude Code

The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CopilotAI review requested due to automatic review settings June 17, 2026 01:47
@pranaygp
pranaygp requested a review from a team as a code ownerJune 17, 2026 01:47

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@changeset-bot

changeset-botBot commented Jun 17, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3ca022f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
workflowPatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production144112301672
✅ 💻 Local Development190902192128
✅ 📦 Local Production190902192128
❌ 🐘 Local Postgres189412332128
✅ 🪟 Windows15200152
✅ 📋 Other88501791064
Total8190210809272

❌ Failed Tests

▲ Vercel Production (1 failed)

nitro (1 failed):

  • startFromWorkflow - calling start() directly inside a workflow function with hook communication | wrun_01KVCQ35C5K211Z7CFGRHG4655 | 🔍 observability
🐘 Local Postgres (1 failed)

nextjs-turbopack-stable-lazy-discovery-enabled (1 failed):

  • wellKnownAgentWorkflow (.well-known/agent) | wrun_01KVCPM9CVA5BNHKTABQFTWTWF

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro125027
✅ example125027
✅ express125027
✅ fastify125027
✅ hono125027
✅ nextjs-turbopack14903
✅ nextjs-webpack14903
❌ nitro124127
✅ nuxt125027
✅ sveltekit14408
✅ vite125027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
❌ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable126026
✅ express-stable126026
✅ fastify-stable126026
✅ hono-stable126026
✅ nextjs-turbopack-canary132020
✅ nextjs-turbopack-stable-lazy-discovery-disabled15101
❌ nextjs-turbopack-stable-lazy-discovery-enabled15011
✅ nextjs-webpack-canary132020
✅ nextjs-webpack-stable-lazy-discovery-disabled15101
✅ nextjs-webpack-stable-lazy-discovery-enabled15101
✅ nitro-stable126026
✅ nuxt-stable126026
✅ sveltekit-stable14507
✅ vite-stable126026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15200
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable127025
✅ e2e-local-dev-tanstack-start-127025
✅ e2e-local-postgres-nest-stable126026
✅ e2e-local-postgres-tanstack-start-126026
✅ e2e-local-prod-nest-stable127025
✅ e2e-local-prod-tanstack-start-127025
✅ e2e-vercel-prod-tanstack-start125027

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: failure
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.042s (-2.5%)1.006s (~)0.963s101.00x
💻 LocalNitro0.045s (+10.1% 🔺)1.006s (~)0.962s101.06x
💻 LocalNext.js (Turbopack)0.062s (-5.5% 🟢)1.007s (~)0.944s101.47x
🐘 PostgresExpress0.067s (-8.0% 🟢)1.013s (~)0.946s101.58x
🐘 PostgresNext.js (Turbopack)0.070s (-0.9%)1.013s (~)0.943s101.65x
🐘 PostgresNitro0.073s (+15.6% 🔺)1.013s (~)0.940s101.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express0.271s (-28.1% 🟢)2.235s (-2.6%)1.964s101.00x
▲ VercelNitro0.319s (-14.9% 🟢)2.203s (-2.0%)1.883s101.18x
▲ VercelNext.js (Turbopack)0.351s (+20.6% 🔺)2.531s (+33.8% 🔺)2.180s101.29x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.094s (-0.7%)2.006s (~)0.912s101.00x
💻 LocalNitro1.095s (+0.9%)2.007s (~)0.912s101.00x
🐘 PostgresNitro1.108s (-1.1%)2.009s (~)0.901s101.01x
🐘 PostgresExpress1.110s (~)2.008s (~)0.898s101.02x
💻 LocalNext.js (Turbopack)1.132s (-1.4%)2.007s (~)0.875s101.03x
🐘 PostgresNext.js (Turbopack)1.141s (~)2.009s (~)0.868s101.04x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.723s (-11.0% 🟢)3.716s (+1.1%)1.993s101.00x
▲ VercelNext.js (Turbopack)1.761s (~)3.708s (~)1.947s101.02x
▲ VercelNitro1.788s (-22.8% 🟢)3.409s (-14.6% 🟢)1.621s101.04x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro10.527s (~)11.022s (~)0.495s31.00x
💻 LocalExpress10.566s (~)11.023s (~)0.457s31.00x
🐘 PostgresExpress10.585s (~)11.023s (~)0.438s31.01x
🐘 PostgresNitro10.598s (+0.6%)11.022s (~)0.424s31.01x
💻 LocalNext.js (Turbopack)10.812s (~)11.022s (~)0.210s31.03x
🐘 PostgresNext.js (Turbopack)10.829s (~)11.017s (~)0.189s31.03x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express13.691s (-24.2% 🟢)15.347s (-23.1% 🟢)1.656s21.00x
▲ VercelNitro13.728s (-27.6% 🟢)15.441s (-27.5% 🟢)1.713s21.00x
▲ VercelNext.js (Turbopack)14.688s (+2.6%)16.926s (+3.9%)2.238s21.07x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro13.690s (~)14.027s (~)0.337s51.00x
🐘 PostgresExpress13.756s (-0.7%)14.019s (~)0.263s51.00x
💻 LocalExpress13.798s (~)14.028s (~)0.231s51.01x
🐘 PostgresNitro13.832s (~)14.023s (~)0.191s51.01x
💻 LocalNext.js (Turbopack)14.383s (~)15.030s (~)0.647s41.05x
🐘 PostgresNext.js (Turbopack)14.396s (~)15.017s (~)0.621s41.05x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express23.028s (-4.9%)25.270s (-3.5%)2.242s31.00x
▲ VercelNext.js (Turbopack)24.609s (-30.5% 🟢)26.392s (-28.3% 🟢)1.783s31.07x
▲ VercelNitro24.684s (-20.8% 🟢)26.313s (-19.2% 🟢)1.629s31.07x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro12.373s (+2.5%)13.027s (+1.1%)0.654s71.00x
💻 LocalExpress12.444s (-0.8%)13.024s (~)0.580s71.01x
🐘 PostgresExpress12.512s (+0.6%)13.016s (~)0.503s71.01x
🐘 PostgresNitro12.712s (+1.6%)13.021s (~)0.310s71.03x
💻 LocalNext.js (Turbopack)13.662s (~)14.027s (~)0.365s71.10x
🐘 PostgresNext.js (Turbopack)13.897s (~)14.308s (+1.0%)0.411s71.12x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro27.276s (-29.9% 🟢)29.146s (-28.8% 🟢)1.870s41.00x
▲ VercelNext.js (Turbopack)27.460s (-22.0% 🟢)29.404s (-20.4% 🟢)1.944s41.01x
▲ VercelExpress28.389s (-19.0% 🟢)30.612s (-17.0% 🟢)2.223s31.04x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.204s (-0.8%)2.007s (~)0.803s151.00x
💻 LocalNitro1.222s (+4.5%)2.006s (~)0.784s151.01x
💻 LocalExpress1.231s (+5.6% 🔺)2.007s (~)0.776s151.02x
🐘 PostgresNitro1.251s (+4.4%)2.009s (~)0.757s151.04x
🐘 PostgresNext.js (Turbopack)1.256s (-2.3%)2.007s (~)0.751s151.04x
💻 LocalNext.js (Turbopack)1.405s (+8.4% 🔺)2.006s (~)0.602s151.17x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.571s (+15.9% 🔺)4.329s (+11.4% 🔺)1.757s81.00x
▲ VercelNitro3.123s (+7.3% 🔺)4.469s (+1.4%)1.346s71.21x
▲ VercelNext.js (Turbopack)3.608s (+35.3% 🔺)4.968s (+21.2% 🔺)1.360s71.40x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.309s (-8.1% 🟢)2.008s (-16.1% 🟢)0.700s151.00x
🐘 PostgresNitro1.332s (-2.4%)2.074s (-17.3% 🟢)0.742s151.02x
🐘 PostgresNext.js (Turbopack)1.452s (-11.3% 🟢)2.075s (-13.3% 🟢)0.623s151.11x
💻 LocalExpress1.987s (+22.2% 🔺)2.592s (+29.2% 🔺)0.604s121.52x
💻 LocalNitro2.051s (+32.2% 🔺)2.507s (+24.7% 🔺)0.456s121.57x
💻 LocalNext.js (Turbopack)2.359s (+23.5% 🔺)3.008s (+31.2% 🔺)0.649s101.80x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.443s (-13.1% 🟢)5.587s (-1.5%)2.143s61.00x
▲ VercelNitro3.655s (-9.7% 🟢)5.233s (-8.2% 🟢)1.577s61.06x
▲ VercelNext.js (Turbopack)4.245s (-5.5% 🟢)5.891s (-9.1% 🟢)1.646s61.23x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.468s (-16.7% 🟢)3.886s (~)2.417s81.00x
🐘 PostgresNitro1.470s (-7.8% 🟢)3.885s (-3.1%)2.415s81.00x
🐘 PostgresNext.js (Turbopack)2.642s (-16.3% 🟢)3.456s (-19.7% 🟢)0.814s91.80x
💻 LocalNitro4.340s (+26.8% 🔺)5.013s (+25.0% 🔺)0.673s62.96x
💻 LocalExpress5.253s (+20.2% 🔺)5.679s (+16.7% 🔺)0.425s63.58x
💻 LocalNext.js (Turbopack)6.408s (+11.0% 🔺)7.019s (+12.9% 🔺)0.611s54.36x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro4.491s (-26.7% 🟢)6.303s (-24.4% 🟢)1.812s51.00x
▲ VercelNext.js (Turbopack)4.524s (-19.6% 🟢)6.497s (-15.8% 🟢)1.973s51.01x
▲ VercelExpress4.646s (-15.6% 🟢)7.023s (-11.4% 🟢)2.378s51.03x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.202s (-1.3%)2.007s (~)0.805s151.00x
🐘 PostgresNitro1.214s (~)2.008s (~)0.794s151.01x
💻 LocalExpress1.228s (-22.1% 🟢)2.006s (~)0.778s151.02x
💻 LocalNitro1.268s (+5.6% 🔺)2.006s (~)0.738s151.06x
🐘 PostgresNext.js (Turbopack)1.271s (-0.8%)2.008s (~)0.737s151.06x
💻 LocalNext.js (Turbopack)1.449s (+3.6%)2.006s (~)0.557s151.21x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.668s (-25.2% 🟢)4.041s (-23.6% 🟢)1.373s81.00x
▲ VercelNext.js (Turbopack)2.876s (-22.7% 🟢)4.782s (-8.6% 🟢)1.907s71.08x
▲ VercelExpress3.339s (+43.7% 🔺)5.449s (+43.2% 🔺)2.110s61.25x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Nitro1.304s (-9.6% 🟢)2.075s (-13.3% 🟢)0.771s151.00x
🐘 PostgresExpress1.310s (-4.1%)2.007s (-13.3% 🟢)0.697s151.00x
🐘 PostgresNext.js (Turbopack)1.436s (-6.9% 🟢)2.076s (-6.6% 🟢)0.640s151.10x
💻 LocalNitro1.935s (+14.3% 🔺)2.293s (+14.2% 🔺)0.358s141.48x
💻 LocalExpress2.003s (+5.5% 🔺)2.393s (+11.3% 🔺)0.390s131.54x
💻 LocalNext.js (Turbopack)2.369s (+9.3% 🔺)3.009s (~)0.639s101.82x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.899s (+6.3% 🔺)4.683s (+2.5%)1.785s71.00x
▲ VercelNitro3.978s (+42.1% 🔺)5.734s (+37.1% 🔺)1.756s61.37x
▲ VercelNext.js (Turbopack)4.216s (-5.6% 🟢)5.944s (-3.4%)1.728s61.45x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.452s (-19.3% 🟢)4.014s (~)2.561s81.00x
🐘 PostgresNitro1.589s (-12.0% 🟢)3.678s (-14.4% 🟢)2.090s91.09x
🐘 PostgresNext.js (Turbopack)2.136s (-47.5% 🟢)3.454s (-22.3% 🟢)1.318s91.47x
💻 LocalExpress5.580s (+21.6% 🔺)6.014s (+20.0% 🔺)0.434s53.84x
💻 LocalNitro5.616s (+25.0% 🔺)6.017s (+20.1% 🔺)0.401s53.87x
💻 LocalNext.js (Turbopack)6.852s (+18.0% 🔺)7.416s (+15.6% 🔺)0.563s54.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.610s (-57.2% 🟢)5.562s (-45.7% 🟢)1.952s61.00x
▲ VercelNitro4.046s (-70.8% 🟢)5.724s (-63.4% 🟢)1.678s61.12x
▲ VercelNext.js (Turbopack)4.124s (+9.0% 🔺)6.272s (+19.9% 🔺)2.147s51.14x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.605s (+6.0% 🔺)1.041s (+3.5%)0.436s581.00x
💻 LocalExpress0.616s (-1.5%)1.005s (-1.7%)0.389s601.02x
💻 LocalNitro0.629s (+25.8% 🔺)1.039s (+3.1%)0.410s581.04x
🐘 PostgresNitro0.692s (+16.5% 🔺)1.078s (+3.6%)0.386s561.14x
🐘 PostgresNext.js (Turbopack)0.858s (+2.7%)1.041s (+1.7%)0.183s581.42x
💻 LocalNext.js (Turbopack)0.860s (-3.1%)1.005s (-3.3%)0.144s601.42x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.774s (-42.9% 🟢)5.457s (-34.5% 🟢)1.683s111.00x
▲ VercelNext.js (Turbopack)3.815s (-39.5% 🟢)5.390s (-33.8% 🟢)1.575s121.01x
▲ VercelExpress4.488s (-1.3%)6.377s (-1.6%)1.889s101.19x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.375s (+1.4%)2.052s (+1.2%)0.678s441.00x
🐘 PostgresNitro1.466s (+5.6% 🔺)2.030s (~)0.564s451.07x
💻 LocalNitro1.485s (+25.1% 🔺)2.006s (~)0.521s451.08x
💻 LocalExpress1.530s (+2.8%)2.007s (~)0.476s451.11x
🐘 PostgresNext.js (Turbopack)1.989s (+2.5%)2.308s (+11.2% 🔺)0.319s401.45x
💻 LocalNext.js (Turbopack)2.090s (-0.6%)2.944s (-2.1%)0.854s311.52x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express10.297s (-41.1% 🟢)12.369s (-35.5% 🟢)2.072s81.00x
▲ VercelNitro10.774s (-8.8% 🟢)12.396s (-12.5% 🟢)1.621s81.05x
▲ VercelNext.js (Turbopack)11.321s (-16.7% 🟢)13.637s (-11.0% 🟢)2.316s71.10x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express2.654s (-2.1%)3.058s (-1.7%)0.404s401.00x
🐘 PostgresNitro2.903s (+3.7%)3.280s (+3.6%)0.376s371.09x
💻 LocalExpress3.275s (+2.4%)4.010s (~)0.735s301.23x
💻 LocalNitro3.362s (+23.1% 🔺)4.010s (+24.4% 🔺)0.648s301.27x
🐘 PostgresNext.js (Turbopack)3.983s (+2.9%)4.253s (+4.3%)0.270s291.50x
💻 LocalNext.js (Turbopack)4.363s (~)5.010s (~)0.647s241.64x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express18.286s (-29.9% 🟢)20.532s (-26.7% 🟢)2.246s61.00x
▲ VercelNitro18.713s (-28.9% 🟢)20.382s (-27.5% 🟢)1.669s61.02x
▲ VercelNext.js (Turbopack)20.206s (-19.2% 🟢)22.420s (-16.3% 🟢)2.214s61.11x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.232s (~)1.006s (~)0.773s601.00x
🐘 PostgresNitro0.237s (+0.7%)1.006s (~)0.769s601.02x
🐘 PostgresNext.js (Turbopack)0.297s (+1.1%)1.023s (+1.7%)0.726s591.28x
💻 LocalExpress0.400s (-9.2% 🟢)1.005s (~)0.605s601.72x
💻 LocalNitro0.417s (+11.8% 🔺)1.004s (~)0.588s601.79x
💻 LocalNext.js (Turbopack)0.633s (+8.8% 🔺)1.004s (~)0.371s602.73x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.787s (+15.3% 🔺)3.598s (+25.4% 🔺)1.811s181.00x
▲ VercelNitro1.933s (+49.7% 🔺)3.615s (+25.8% 🔺)1.682s171.08x
▲ VercelNext.js (Turbopack)2.387s (+27.6% 🔺)4.344s (+11.8% 🔺)1.956s141.34x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.331s (-6.9% 🟢)1.006s (-3.3%)0.675s901.00x
🐘 PostgresNitro0.356s (+4.1%)1.006s (-1.1%)0.650s901.08x
🐘 PostgresNext.js (Turbopack)0.476s (-11.7% 🟢)1.103s (-1.3%)0.628s831.44x
💻 LocalNitro2.182s (+43.2% 🔺)2.737s (+27.2% 🔺)0.555s336.59x
💻 LocalExpress2.222s (+6.3% 🔺)2.738s (+3.1%)0.517s336.71x
💻 LocalNext.js (Turbopack)2.493s (+7.0% 🔺)3.344s (+8.7% 🔺)0.851s277.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.638s (-9.8% 🟢)4.553s (-5.1% 🟢)1.915s201.00x
▲ VercelNitro2.705s (-1.5%)4.386s (-8.3% 🟢)1.681s211.03x
▲ VercelNext.js (Turbopack)3.092s (+1.6%)4.802s (~)1.710s191.17x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.526s (-7.4% 🟢)1.059s (-13.0% 🟢)0.532s1141.00x
🐘 PostgresNitro0.570s (~)1.078s (-12.5% 🟢)0.508s1121.08x
🐘 PostgresNext.js (Turbopack)1.918s (-26.7% 🟢)2.763s (-20.4% 🟢)0.845s443.64x
💻 LocalNitro9.521s (+50.8% 🔺)10.445s (+54.9% 🔺)0.924s1218.09x
💻 LocalExpress10.198s (+22.8% 🔺)11.029s (+24.2% 🔺)0.831s1219.38x
💻 LocalNext.js (Turbopack)10.284s (-4.4%)11.663s (+0.8%)1.379s1119.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.245s (-38.5% 🟢)5.031s (-28.3% 🟢)1.786s241.00x
▲ VercelExpress3.483s (-36.2% 🟢)5.687s (-19.8% 🟢)2.204s221.07x
▲ VercelNext.js (Turbopack)4.699s (-2.9%)6.708s (+2.7%)2.008s181.45x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.165s (-1.1%)2.000s (~)0.001s (+9.1% 🔺)2.009s (~)0.844s101.00x
🐘 PostgresNitro1.181s (+1.4%)1.995s (~)0.001s (+30.0% 🔺)2.011s (~)0.831s101.01x
💻 LocalNitro1.181s (+5.7% 🔺)2.005s (~)0.010s (-62.9% 🟢)2.017s (-1.0%)0.836s101.01x
💻 LocalExpress1.185s (+3.0%)2.005s (~)0.012s (+19.8% 🔺)2.019s (~)0.835s101.02x
💻 LocalNext.js (Turbopack)1.216s (~)2.003s (~)0.013s (+2.4%)2.020s (~)0.803s101.04x
🐘 PostgresNext.js (Turbopack)1.232s (~)2.002s (~)0.001s (-15.4% 🟢)2.011s (~)0.779s101.06x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.206s (-8.2% 🟢)3.426s (-6.9% 🟢)0.602s (-18.8% 🟢)4.513s (-7.4% 🟢)2.307s101.00x
▲ VercelNext.js (Turbopack)2.304s (-8.6% 🟢)3.523s (-7.8% 🟢)0.772s (-4.3%)4.790s (-5.8% 🟢)2.486s101.04x
▲ VercelNitro2.331s (-11.7% 🟢)3.158s (-17.0% 🟢)1.253s (+57.9% 🔺)4.805s (-4.9%)2.474s101.06x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro1.572s (+9.3% 🔺)2.011s (~)0.012s (-32.0% 🟢)2.025s (~)0.453s301.00x
🐘 PostgresExpress1.577s (~)2.004s (~)0.005s (+5.7% 🔺)2.026s (~)0.449s301.00x
💻 LocalExpress1.583s (+1.0%)2.010s (~)0.014s (+4.9%)2.025s (~)0.442s301.01x
🐘 PostgresNitro1.602s (+1.5%)2.008s (~)0.005s (-5.6% 🟢)2.027s (~)0.425s301.02x
💻 LocalNext.js (Turbopack)1.740s (-0.5%)2.010s (~)0.013s (+1.3%)2.025s (~)0.285s301.11x
🐘 PostgresNext.js (Turbopack)1.892s (+5.9% 🔺)2.011s (~)0.005s (+5.8% 🔺)2.029s (~)0.137s301.20x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Next.js (Turbopack)6.842s (-19.5% 🟢)8.374s (-14.7% 🟢)0.220s (-16.3% 🟢)9.079s (-14.1% 🟢)2.236s71.00x
▲ VercelNitro6.905s (-3.6%)7.768s (-10.3% 🟢)0.455s (+91.9% 🔺)8.766s (-6.9% 🟢)1.861s71.01x
▲ VercelExpress8.068s (+26.4% 🔺)9.373s (+20.1% 🔺)0.307s (+71.5% 🔺)10.412s (+23.1% 🔺)2.345s61.18x

🔍 Observability: Next.js (Turbopack) | Nitro | Express

10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.789s (+3.3%)1.101s (+7.5% 🔺)0.000s (-100.0% 🟢)1.118s (+6.0% 🔺)0.329s541.00x
🐘 PostgresNitro0.795s (+2.9%)1.044s (~)0.000s (-3.5%)1.061s (-3.0%)0.265s571.01x
🐘 PostgresNext.js (Turbopack)0.986s (-3.3%)1.397s (-4.6%)0.000s (-100.0% 🟢)1.407s (-4.4%)0.421s431.25x
💻 LocalExpress1.533s (+12.5% 🔺)2.014s (~)0.000s (+16.7% 🔺)2.016s (~)0.483s301.94x
💻 LocalNitro1.568s (+45.8% 🔺)2.014s (+9.6% 🔺)0.000s (-56.8% 🟢)2.016s (+9.5% 🔺)0.448s301.99x
💻 LocalNext.js (Turbopack)1.916s (+26.7% 🔺)2.193s (+8.9% 🔺)0.000s (+7.1% 🔺)2.196s (+8.9% 🔺)0.280s282.43x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.961s (-21.2% 🟢)4.002s (-25.7% 🟢)0.000s (-21.4% 🟢)4.479s (-23.6% 🟢)1.518s141.00x
▲ VercelExpress3.140s (+3.6%)4.516s (-1.1%)0.000s (NaN%)5.069s (+0.6%)1.929s121.06x
▲ VercelNext.js (Turbopack)3.540s (+2.7%)5.016s (+5.2% 🔺)0.000s (-100.0% 🟢)5.553s (+5.5% 🔺)2.013s111.20x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.691s (+4.7%)2.302s (+11.5% 🔺)0.000s (NaN%)2.316s (+10.9% 🔺)0.625s261.00x
🐘 PostgresNitro1.779s (+14.7% 🔺)2.381s (+11.6% 🔺)0.000s (+11.5% 🔺)2.394s (+11.5% 🔺)0.615s261.05x
🐘 PostgresNext.js (Turbopack)2.251s (+6.6% 🔺)2.651s (+2.5%)0.000s (-100.0% 🟢)2.664s (+2.6%)0.413s231.33x
💻 LocalNitro4.321s (+95.8% 🔺)4.717s (+72.0% 🔺)0.001s (+12.8% 🔺)4.729s (+71.7% 🔺)0.409s132.56x
💻 LocalExpress4.772s (+62.9% 🔺)5.362s (+46.1% 🔺)0.001s (+98.3% 🔺)5.367s (+46.1% 🔺)0.595s122.82x
💻 LocalNext.js (Turbopack)5.632s (+94.2% 🔺)6.226s (+85.4% 🔺)0.000s (-60.0% 🟢)6.235s (+85.4% 🔺)0.602s103.33x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express4.310s (-24.3% 🟢)6.054s (-16.2% 🟢)0.000s (+Infinity% 🔺)6.546s (-14.9% 🟢)2.236s101.00x
▲ VercelNext.js (Turbopack)4.772s (-34.2% 🟢)6.201s (-26.1% 🟢)0.000s (-100.0% 🟢)6.679s (-24.5% 🟢)1.907s101.11x
▲ VercelNitro4.835s (-7.9% 🟢)5.805s (-10.1% 🟢)0.000s (+Infinity% 🔺)6.308s (-8.9% 🟢)1.473s101.12x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalNitro12/21
🐘 PostgresExpress19/21
▲ VercelExpress14/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres17/21
Next.js (Turbopack)🐘 Postgres14/21
Nitro🐘 Postgres13/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run

@pranaygp

Copy link
Copy Markdown
ContributorAuthor

CI failure triage — pre-existing Vercel-prod e2e flake (not a regression)

The two red checks (E2E Vercel Prod Tests (example), E2E Vercel Prod Tests (fastify), which roll up into E2E Required Check) are the shared Vercel-prod timing flake, not caused by this PR. Evidence:

Scope is wrong for a hydration regression. This PR only memoizes primitive step-result hydration. A determinism/stale-value bug there would surface across all worlds — yet every local suite is 100% green:

SuitePassedFailed
▲ Vercel Production14402
💻 Local Development19090
📦 Local Production19090
🐘 Local Postgres18950
🪟 Windows1520

Only 2 failures, only on Vercel Production.

The two failing tests are unrelated to result hydration, and are abort/hook timing races:

  1. exampleAbortController > abortFromStepWorkflow: step abort cancels an in-flight sibling step. The run completed successfully; the assertion failed only because the abort lost a race. From the run diagnostics (wrun_01KV9MZ1Y00N90XFCKFTSPX5N1):

    +2.2s step_completed (longStep) <- sibling finished on its own
    +3.0s hook_received <- abort signal arrived AFTER
    

    The sibling longStep self-completed (2.2s) before the abort hook arrived (3.0s) under Vercel-prod queue/network latency, so there was no in-flight step left to cancel. Step results hydrated fine; this is purely environmental latency.

  2. fastifystartFromWorkflow - calling start() directly inside a workflow function with hook communication. This exact test passed (4128ms) in the example job of this same commit (a56f5c90b) — a textbook cross-run flake.

The same test is red on plain main, without this change. On main run 27704378960 (commit 2acf13cc7):

  • E2E Vercel Prod Tests (tanstack-start)abortFromStepWorkflow: step abort cancels an in-flight sibling step (the identical test that failed here on example)
  • E2E Vercel Prod Tests (nextjs-turbopack)distributedAbortController - reconnect to existing controller

And on main run 27657696161 (cb181392b, the commit this branch is based on): E2E Vercel Prod Tests (fastify)hookWithSleepFinalStepWorkflow. The set of red workbenches rotates run-to-run — the signature of environment flakiness, not a code regression.

Local verification of this branch (rebuilt @workflow/core first): cd packages/core && pnpm test1249 passed / 56 files, 0 failures, including the determinism/ordering replay tests in workflow.test.ts and the new step-hydration-cache.test.ts (10) + step-hydration-memoization.test.ts (3). The memoization tests assert byte-identical delivery ordering on cache hits and that objects re-hydrate fresh each replay.

Re-running the e2e jobs should clear them. No code change is warranted.

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — the O(N²)→O(N) hydration memoization, done with the right safety bias

This is the most safety-sensitive of the four (it's the only one that caches a value across replays), and the design lands on the conservative side of every judgment call, which is exactly right for replay determinism.

The primitives-only decision is the crux and it's correct.hydrateStepReturnValue (devalue.parse) returns a fresh object graph each call, and each replay runs in a fresh VM, so today the workflow gets a brand-new value every replay. Caching and returning the same object reference would let const r = await step(); r.count++ observe a prior replay's mutation — silent divergence. The alternatives are both worse: structuredClone is lossy for reviver-produced specials (stream handles, step-fn proxies, Request/Response, AbortController/Signal) and still O(size). Restricting the cache to primitives (immutable, compared by value) makes "share the reference" provably indistinguishable from re-parsing, and non-primitives fall through to a full re-hydrate every replay — preserving current behavior exactly. Trading the object-case optimization for airtight determinism is the right call.

What I verified in the integration:

  • Surgical wrap: only the await hydrateStepReturnValue(...) call is replaced; pendingDeliveries++/--, the ctx.promiseQueue.then(...) slot, and resolve(...) are byte-for-byte unchanged. The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the identical position in the ordered delivery chain a re-hydrate would have — preserving the pendingDeliveries-gated suspension/barrier/Promise.race determinism.
  • Lifetime: cache is created once per invocation in runtime.ts (outside the per-iteration context), threaded through runWorkflow, never shared across runs. The optional param/context field degrades to re-hydrating-every-replay for harnesses that omit it.
  • Keying by step_completedeventId (stable, world-assigned, same immutable bytes every replay) is sound, and has() rather than get() !== undefined correctly treats a memoized undefined result as a hit.
  • Errors never cached — a rejected hydrate re-attempts next replay, no parked rejected promise.

Built @workflow/core; full suite green (1249) including the new step-hydration-cache.test.ts / step-hydration-memoization.test.ts, async-deserialization-ordering.test.ts, and workflow.test.ts.

One forward-looking note (non-blocking): the object case is left on the table by design. If it's ever revisited, the safe path is deep-freeze + share frozen graphs — but only after handling the reviver-produced specials, so I agree with deferring it. LGTM.

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

// Only memoize values that are safe to return by reference across replays.
// Non-primitives fall through and are re-hydrated fresh on every replay.
if (isMemoizablePrimitive(value)) {
cache.set(eventId, value);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

The per-run stepHydrationCache is never size-bounded or evicted: it grows one entry per primitive-returning completed step and lives for the entire invocation. The cost worth calling out (the PR doesn't) is the new residency — the decrypted/devalue-parsed plaintext of each cached primitive is now held for the whole invocation, on top of the serialized bytes already retained in cachedEvents. For a long sequential workflow whose steps return large strings, that roughly doubles peak retained memory for those results.

The dominant residency (the full event log in cachedEvents) already exists, so this isn't blocking, but a byte-size threshold would be a cheap safeguard — large primitives are exactly the cheap-to-re-hydrate case relative to their footprint, so letting them fall through to the existing re-hydrate path costs little and bounds the worst case. Ideally with a test asserting the bound.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — capped it, with a test. Pushed in a60baad.

What changed (step-hydration-cache.ts):

  • Added MAX_MEMOIZED_PRIMITIVE_LENGTH = 4096 and extended isMemoizablePrimitive so a string/bigint longer than 4 KiB is treated as non-memoizable. Those are the only primitive types that can carry a large payload — number/boolean/null/undefined/symbol are inherently small, so they're never length-checked. Oversized values now fall through to the existing per-replay re-hydrate path, exactly as you suggested: large primitives are cheap to re-hydrate relative to their footprint, so this caps the doubled-residency worst case at negligible cost.
  • Documented the memory characteristic on the cache module: per-invocation lifetime (fresh Map per run in runtime.ts, GC'd when the invocation returns), bounded by the number of primitive-returning completed steps, primitives-only, now byte-bounded.

Tests (step-hydration-cache.test.ts, +4):isMemoizablePrimitive true at the bound / false beyond it (string and bigint), and an end-to-end assertion that an oversized string re-hydrates on every replay and cache.size === 0 (the bound assertion you asked for); plus an at-bound string is a cache hit.

The cap only ever reduces what gets cached, so determinism is untouched — oversized values just take the already-correct re-hydrate path. Full core suite green (1253, incl. the ordering/determinism + memoization suites); biome + tsc clean.

On consistency with #2471 (the sibling scriptCache): noting the distinction since they're bounded for different reasons. #2471's cache is process-wide and monotonic across the whole process — in dev/watch it pins every historical bundle string (hundreds of MB over a session), which is a genuine regression vs. the prior keep-only-latest behavior, hence the Blocking bound there. This cache is per-invocation and freed wholesale when the run returns, so it can never accumulate across runs; the only real cost is the doubled residency for large primitives during one run, which the size cap here now bounds. Different scope, different severity, but both bounded now.

Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-hydration
* origin/main:
perf(core): lazy inline step start (save one world round-trip per step) (#2478)
perf(core): skip per-step events.list via inline event-log delta (#2475)
Version Packages (beta) (#2491)
[world-vercel] Honor hasMore flag from v4 list pagination endpoint (#2486)
Version Packages (beta) (#2451)
Fix Next workflow module specifier root (#2455)
[world-vercel] Send remoteRefBehavior=lazy on v4 metadata-only event listings (#2415)
[swc-plugin] Fix eager discovery for object property steps (#2484)
fix(web-shared): align attributes panel styling (#2483)
[web-shared] Auto-scroll trace viewer on J/K span navigation (#2366)
fix(web): render restarted step segment as solid gray, not running stripes (#2480)
fix(web-shared): use solid gray for queued trace segment (#2474)
Add trace viewer span markers for hooks and attributes (#2452)
test: support Vercel protection bypass secret in e2e headers (#2458)
fix(core): bump payload-compression cutoff to 5.0.0-beta.18 (#2470)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts:
#	packages/core/src/runtime.ts
@pranaygp
pranaygp enabled auto-merge (squash) June 18, 2026 06:24
@pranaygp
pranaygp disabled auto-merge June 22, 2026 20:30
@pranaygp
pranaygp merged commit 66ca0dc into mainJun 22, 2026
118 of 121 checks passed
@pranaygp
pranaygp deleted the pgp/perf-memoize-step-hydration branch June 22, 2026 20:30
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
github-actionsBot added a commit that referenced this pull request Jun 22, 2026
…#2472)
* perf(core): memoize step return value hydration across replays
The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): bound memoized step-hydration cache by primitive size
Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #2563. Merge conflicts were resolved by AI — please review carefully. (backport job run)

pranaygp added a commit that referenced this pull request Jun 22, 2026
…testing
* origin/main:
Version Packages (beta) (#2540)
perf(core): memoize step return value hydration across inline replays (#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (#2412)
Amend lazy discovery removal changeset (#2560)
[docs] Document minimum SDK version for using hook.getConflict (#2423)
Update default CODEOWNERS (#2556)
Optimize and fix the default eager build mode (#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (#2547)" (#2554)
[core] Turbo mode: fast-path the first invocation (#2526)
Remove lazy discovery from workflow/next (#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (#2547)
feat(docs): add eve and AI SDK to product switcher (#2543)
[vitest] Fix local imports failing to load in test step bundles (#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (#2324)
Version Packages (beta) (#2495)
otel(world-vercel): inject trace context on v4 event requests (#2533)
Bump undici to 7.28.0 (#2534)
Default source maps to dev-on / prod-off (#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (#2527)
perf(core): parallel inline steps + optimistic lazy step start (#2516)
pranaygp added a commit to marcopiraccini/workflow that referenced this pull request Jun 22, 2026
* origin/main: (120 commits)
Version Packages (beta) (vercel#2540)
perf(core): memoize step return value hydration across inline replays (vercel#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (vercel#2412)
Amend lazy discovery removal changeset (vercel#2560)
[docs] Document minimum SDK version for using hook.getConflict (vercel#2423)
Update default CODEOWNERS (vercel#2556)
Optimize and fix the default eager build mode (vercel#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)" (vercel#2554)
[core] Turbo mode: fast-path the first invocation (vercel#2526)
Remove lazy discovery from workflow/next (vercel#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)
feat(docs): add eve and AI SDK to product switcher (vercel#2543)
[vitest] Fix local imports failing to load in test step bundles (vercel#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (vercel#2324)
Version Packages (beta) (vercel#2495)
otel(world-vercel): inject trace context on v4 event requests (vercel#2533)
Bump undici to 7.28.0 (vercel#2534)
Default source maps to dev-on / prod-off (vercel#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (vercel#2527)
perf(core): parallel inline steps + optimistic lazy step start (vercel#2516)
...
VaguelySerious pushed a commit that referenced this pull request Jun 22, 2026
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pranaygp@TooTallNate@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

perf(core): memoize step return value hydration across inline replays - #2472

Merged
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration
Jun 22, 2026
Merged

perf(core): memoize step return value hydration across inline replays#2472
pranaygp merged 3 commits into
mainfrom
pgp/perf-memoize-step-hydration

Conversation

@pranaygp

@pranaygppranaygp commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

The inline replay loop (runtime.tsrunWorkflow, workflow.ts) re-executes the workflow body and re-consumes the full event log on every iteration. For each already-completed step, the step consumer (step.ts, step_completed path) re-ran hydrateStepReturnValue — AES-GCM decrypt + devalue-parse of the serialized result — on every replay, even though that exact result was already hydrated on every prior replay.

For a sequential workflow of N steps, replay K hydrates K results, so the aggregate cost across a single invocation is O(N²) decrypt+parse operations.

This PR adds a per-run memoization cache so a completed step's hydrated result is returned in O(1) on subsequent replays within the same invocation, making the aggregate cost O(N).

Before / after

  • Before: replay 1 hydrates 1 result, replay 2 hydrates 2, …, replay K hydrates K → Σ = O(N²) decrypt+parse over a sequential run.
  • After: each completed step's result is hydrated once and memoized for the rest of the invocation → O(N) total. Replay K hydrates only the one newly-completed step; the K−1 prior results are cache hits.

Cache scope & keying

  • Lifetime / scope: owned by the inline loop in runtime.ts (created once per run invocation, alongside cachedEvents), threaded into runWorkflow(..., stepHydrationCache?) and stored on WorkflowOrchestratorContext.stepHydrationCache. A fresh context is created each loop iteration, so the cache deliberately lives outside the per-iteration context to survive across iterations of the same run. It is never shared across unrelated runs or process-level invocations.
  • Keying: by the persisted step_completed event's eventId — a stable, world-assigned id. The same event carries the same immutable serialized bytes across every replay, so a hit is guaranteed to correspond to identical input.
  • Optional / backward compatible: the parameter and context field are optional. Callers/harnesses that omit them (and the many runWorkflow(...) unit tests) degrade to re-hydrating every replay — identical to previous behavior.

Memory characteristic

A cached entry holds the decrypted/devalue-parsed plaintext of a step result, retained for the rest of the invocation on top of the serialized bytes already held in cachedEvents — so for large primitive results it roughly doubles peak retained memory for those results during the run. This residual is:

  • Scoped to one invocation — the Map is created per run and GC'd when the invocation returns; nothing accumulates across runs or process-level invocations (a much weaker concern than a process-wide cache, where the dominant residency — the full event log in cachedEvents — already exists for the same lifetime).
  • Bounded by the primitive-returning completed-step count — at most one small entry per such step.
  • Byte-bounded. Most primitives (numbers, booleans, null/undefined, symbols, short ids/strings) are tiny. The only primitive that can be large is a string (or a pathologically long bigint), so a string/bigint result longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) is not memoized — it falls through to the existing per-replay re-hydrate path. Large payloads are cheap to re-hydrate relative to their footprint, so this caps the worst case at negligible cost. The cap only ever reduces what is cached, so deterministic replay is unaffected.

Ordering safety analysis

The cache lookup replaces only the await hydrateStepReturnValue(...) call inside the existing ctx.promiseQueue.then(async () => { ... }) slot. Everything else is byte-for-byte unchanged:

  • ctx.pendingDeliveries++ / -- accounting is untouched.
  • The hydrate (or cache hit) still happens inside the same serial promiseQueue slot, at the same log position, and still resolves via the same resolve(...).
  • The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the exact position in the ordered delivery chain a re-hydrate would have.

So delivery order, pendingDeliveries-gated suspensions, the pendingDeliveryBarriers / awaitEarlierDeliveries machinery, and Promise.race/Promise.all replay determinism are all unaffected. Hook, wait, and abort hydration paths are intentionally not cached (they're the ordering-sensitive paths and not the O(N²) hotspot).

Identity / immutability safety

hydrateStepReturnValue (devalue.parse) returns a fresh object graph on every call, and each replay iteration runs in a fresh workflow VM. Today the workflow therefore receives a brand-new value on every replay. If we cached and returned the same object reference across replays, workflow code that mutates a step result (const r = await step(); r.count++) would observe a previous replay's mutation on the next replay — a non-deterministic divergence. (structuredClone on each hit is both lossy — revivers reconstruct stream handles, step-function proxies, Request/Response, and AbortController/AbortSignal class instances — and still O(size).)

Decision: only primitives are memoized (string, number, boolean, bigint, symbol, null, undefined). Primitives are immutable and compared by value, so sharing the reference is provably indistinguishable from re-parsing. Any non-primitive result falls through to a full re-hydrate every replay, preserving current behavior exactly. Errors are never cached, so a rejected hydrate re-attempts on the next replay (no parked rejected promise). This trades away the optimization in the object-returning case to keep deterministic replay airtight — correctness over speed.

What I verified

  • Unit:step-hydration-cache.test.ts (14 tests: primitive detection, memoization, non-primitive eviction/fresh-object, falsy primitives, keying, error non-caching, no-cache passthrough, plus the size-bound — at-bound string is a hit, oversized string/bigint are not memoized and cache.size stays 0) and step-hydration-memoization.test.ts (3 tests through the real createUseStep consumer: hydrate-skipped-on-replay-2 via spy, event-log ordering preserved on cache hits, fresh object per replay for object results).
  • Full core suite:cd packages/core && pnpm test1253 passed / 56 files, including async-deserialization-ordering.test.ts, workflow.test.ts (79 tests), runtime.test.ts, hook-sleep-interaction, abort-consistency. No regressions.
  • Build / format / typecheck:pnpm build (full repo, 27/27), @workflow/core build + tsc --noEmit clean; Biome format applied; new files Biome-clean (the only lint errors were import-ordering, auto-fixed; remaining warnings are pre-existing noExcessiveCognitiveComplexity on functions I only edited).
  • E2E (local nextjs-turbopack dev server, the determinism-sensitive subset): promiseAllWorkflow, promiseRaceWorkflow, promiseAnyWorkflow, sleepWinsRaceWorkflow, stepWinsRaceWorkflow, promiseRaceStressTestWorkflow, hookWorkflow, webhookWorkflow, parallel-steps-then-webhook replay race, sleepingWorkflow, parallelSleepWorkflow, retry/error/catchability suite, fetchWorkflowall passed.

Risks / deferred

  • Only primitive step results are accelerated; object-returning steps still re-hydrate each replay (intentional, for determinism). A future safe extension could deep-freeze + share frozen object graphs, but that needs care around reviver-produced special objects and is out of scope here.
  • Large (>4 KiB) string/bigint results are intentionally not memoized to bound peak retained memory (see Memory characteristic); they re-hydrate each replay.
  • Hook/wait/abort hydration paths are uncached by design.

🤖 Generated with Claude Code

The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CopilotAI review requested due to automatic review settings June 17, 2026 01:47
@pranaygp
pranaygp requested a review from a team as a code ownerJune 17, 2026 01:47

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@changeset-bot

changeset-botBot commented Jun 17, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3ca022f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
workflowPatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production144112301672
✅ 💻 Local Development190902192128
✅ 📦 Local Production190902192128
❌ 🐘 Local Postgres189412332128
✅ 🪟 Windows15200152
✅ 📋 Other88501791064
Total8190210809272

❌ Failed Tests

▲ Vercel Production (1 failed)

nitro (1 failed):

  • startFromWorkflow - calling start() directly inside a workflow function with hook communication | wrun_01KVCQ35C5K211Z7CFGRHG4655 | 🔍 observability
🐘 Local Postgres (1 failed)

nextjs-turbopack-stable-lazy-discovery-enabled (1 failed):

  • wellKnownAgentWorkflow (.well-known/agent) | wrun_01KVCPM9CVA5BNHKTABQFTWTWF

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro125027
✅ example125027
✅ express125027
✅ fastify125027
✅ hono125027
✅ nextjs-turbopack14903
✅ nextjs-webpack14903
❌ nitro124127
✅ nuxt125027
✅ sveltekit14408
✅ vite125027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable127025
✅ express-stable127025
✅ fastify-stable127025
✅ hono-stable127025
✅ nextjs-turbopack-canary133019
✅ nextjs-turbopack-stable-lazy-discovery-disabled15200
✅ nextjs-turbopack-stable-lazy-discovery-enabled15200
✅ nextjs-webpack-canary133019
✅ nextjs-webpack-stable-lazy-discovery-disabled15200
✅ nextjs-webpack-stable-lazy-discovery-enabled15200
✅ nitro-stable127025
✅ nuxt-stable127025
✅ sveltekit-stable14606
✅ vite-stable127025
❌ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable126026
✅ express-stable126026
✅ fastify-stable126026
✅ hono-stable126026
✅ nextjs-turbopack-canary132020
✅ nextjs-turbopack-stable-lazy-discovery-disabled15101
❌ nextjs-turbopack-stable-lazy-discovery-enabled15011
✅ nextjs-webpack-canary132020
✅ nextjs-webpack-stable-lazy-discovery-disabled15101
✅ nextjs-webpack-stable-lazy-discovery-enabled15101
✅ nitro-stable126026
✅ nuxt-stable126026
✅ sveltekit-stable14507
✅ vite-stable126026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15200
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable127025
✅ e2e-local-dev-tanstack-start-127025
✅ e2e-local-postgres-nest-stable126026
✅ e2e-local-postgres-tanstack-start-126026
✅ e2e-local-prod-nest-stable127025
✅ e2e-local-prod-tanstack-start-127025
✅ e2e-vercel-prod-tanstack-start125027

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: failure
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.042s (-2.5%)1.006s (~)0.963s101.00x
💻 LocalNitro0.045s (+10.1% 🔺)1.006s (~)0.962s101.06x
💻 LocalNext.js (Turbopack)0.062s (-5.5% 🟢)1.007s (~)0.944s101.47x
🐘 PostgresExpress0.067s (-8.0% 🟢)1.013s (~)0.946s101.58x
🐘 PostgresNext.js (Turbopack)0.070s (-0.9%)1.013s (~)0.943s101.65x
🐘 PostgresNitro0.073s (+15.6% 🔺)1.013s (~)0.940s101.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express0.271s (-28.1% 🟢)2.235s (-2.6%)1.964s101.00x
▲ VercelNitro0.319s (-14.9% 🟢)2.203s (-2.0%)1.883s101.18x
▲ VercelNext.js (Turbopack)0.351s (+20.6% 🔺)2.531s (+33.8% 🔺)2.180s101.29x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.094s (-0.7%)2.006s (~)0.912s101.00x
💻 LocalNitro1.095s (+0.9%)2.007s (~)0.912s101.00x
🐘 PostgresNitro1.108s (-1.1%)2.009s (~)0.901s101.01x
🐘 PostgresExpress1.110s (~)2.008s (~)0.898s101.02x
💻 LocalNext.js (Turbopack)1.132s (-1.4%)2.007s (~)0.875s101.03x
🐘 PostgresNext.js (Turbopack)1.141s (~)2.009s (~)0.868s101.04x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.723s (-11.0% 🟢)3.716s (+1.1%)1.993s101.00x
▲ VercelNext.js (Turbopack)1.761s (~)3.708s (~)1.947s101.02x
▲ VercelNitro1.788s (-22.8% 🟢)3.409s (-14.6% 🟢)1.621s101.04x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro10.527s (~)11.022s (~)0.495s31.00x
💻 LocalExpress10.566s (~)11.023s (~)0.457s31.00x
🐘 PostgresExpress10.585s (~)11.023s (~)0.438s31.01x
🐘 PostgresNitro10.598s (+0.6%)11.022s (~)0.424s31.01x
💻 LocalNext.js (Turbopack)10.812s (~)11.022s (~)0.210s31.03x
🐘 PostgresNext.js (Turbopack)10.829s (~)11.017s (~)0.189s31.03x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express13.691s (-24.2% 🟢)15.347s (-23.1% 🟢)1.656s21.00x
▲ VercelNitro13.728s (-27.6% 🟢)15.441s (-27.5% 🟢)1.713s21.00x
▲ VercelNext.js (Turbopack)14.688s (+2.6%)16.926s (+3.9%)2.238s21.07x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro13.690s (~)14.027s (~)0.337s51.00x
🐘 PostgresExpress13.756s (-0.7%)14.019s (~)0.263s51.00x
💻 LocalExpress13.798s (~)14.028s (~)0.231s51.01x
🐘 PostgresNitro13.832s (~)14.023s (~)0.191s51.01x
💻 LocalNext.js (Turbopack)14.383s (~)15.030s (~)0.647s41.05x
🐘 PostgresNext.js (Turbopack)14.396s (~)15.017s (~)0.621s41.05x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express23.028s (-4.9%)25.270s (-3.5%)2.242s31.00x
▲ VercelNext.js (Turbopack)24.609s (-30.5% 🟢)26.392s (-28.3% 🟢)1.783s31.07x
▲ VercelNitro24.684s (-20.8% 🟢)26.313s (-19.2% 🟢)1.629s31.07x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro12.373s (+2.5%)13.027s (+1.1%)0.654s71.00x
💻 LocalExpress12.444s (-0.8%)13.024s (~)0.580s71.01x
🐘 PostgresExpress12.512s (+0.6%)13.016s (~)0.503s71.01x
🐘 PostgresNitro12.712s (+1.6%)13.021s (~)0.310s71.03x
💻 LocalNext.js (Turbopack)13.662s (~)14.027s (~)0.365s71.10x
🐘 PostgresNext.js (Turbopack)13.897s (~)14.308s (+1.0%)0.411s71.12x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro27.276s (-29.9% 🟢)29.146s (-28.8% 🟢)1.870s41.00x
▲ VercelNext.js (Turbopack)27.460s (-22.0% 🟢)29.404s (-20.4% 🟢)1.944s41.01x
▲ VercelExpress28.389s (-19.0% 🟢)30.612s (-17.0% 🟢)2.223s31.04x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.204s (-0.8%)2.007s (~)0.803s151.00x
💻 LocalNitro1.222s (+4.5%)2.006s (~)0.784s151.01x
💻 LocalExpress1.231s (+5.6% 🔺)2.007s (~)0.776s151.02x
🐘 PostgresNitro1.251s (+4.4%)2.009s (~)0.757s151.04x
🐘 PostgresNext.js (Turbopack)1.256s (-2.3%)2.007s (~)0.751s151.04x
💻 LocalNext.js (Turbopack)1.405s (+8.4% 🔺)2.006s (~)0.602s151.17x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.571s (+15.9% 🔺)4.329s (+11.4% 🔺)1.757s81.00x
▲ VercelNitro3.123s (+7.3% 🔺)4.469s (+1.4%)1.346s71.21x
▲ VercelNext.js (Turbopack)3.608s (+35.3% 🔺)4.968s (+21.2% 🔺)1.360s71.40x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.309s (-8.1% 🟢)2.008s (-16.1% 🟢)0.700s151.00x
🐘 PostgresNitro1.332s (-2.4%)2.074s (-17.3% 🟢)0.742s151.02x
🐘 PostgresNext.js (Turbopack)1.452s (-11.3% 🟢)2.075s (-13.3% 🟢)0.623s151.11x
💻 LocalExpress1.987s (+22.2% 🔺)2.592s (+29.2% 🔺)0.604s121.52x
💻 LocalNitro2.051s (+32.2% 🔺)2.507s (+24.7% 🔺)0.456s121.57x
💻 LocalNext.js (Turbopack)2.359s (+23.5% 🔺)3.008s (+31.2% 🔺)0.649s101.80x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.443s (-13.1% 🟢)5.587s (-1.5%)2.143s61.00x
▲ VercelNitro3.655s (-9.7% 🟢)5.233s (-8.2% 🟢)1.577s61.06x
▲ VercelNext.js (Turbopack)4.245s (-5.5% 🟢)5.891s (-9.1% 🟢)1.646s61.23x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.468s (-16.7% 🟢)3.886s (~)2.417s81.00x
🐘 PostgresNitro1.470s (-7.8% 🟢)3.885s (-3.1%)2.415s81.00x
🐘 PostgresNext.js (Turbopack)2.642s (-16.3% 🟢)3.456s (-19.7% 🟢)0.814s91.80x
💻 LocalNitro4.340s (+26.8% 🔺)5.013s (+25.0% 🔺)0.673s62.96x
💻 LocalExpress5.253s (+20.2% 🔺)5.679s (+16.7% 🔺)0.425s63.58x
💻 LocalNext.js (Turbopack)6.408s (+11.0% 🔺)7.019s (+12.9% 🔺)0.611s54.36x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro4.491s (-26.7% 🟢)6.303s (-24.4% 🟢)1.812s51.00x
▲ VercelNext.js (Turbopack)4.524s (-19.6% 🟢)6.497s (-15.8% 🟢)1.973s51.01x
▲ VercelExpress4.646s (-15.6% 🟢)7.023s (-11.4% 🟢)2.378s51.03x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.202s (-1.3%)2.007s (~)0.805s151.00x
🐘 PostgresNitro1.214s (~)2.008s (~)0.794s151.01x
💻 LocalExpress1.228s (-22.1% 🟢)2.006s (~)0.778s151.02x
💻 LocalNitro1.268s (+5.6% 🔺)2.006s (~)0.738s151.06x
🐘 PostgresNext.js (Turbopack)1.271s (-0.8%)2.008s (~)0.737s151.06x
💻 LocalNext.js (Turbopack)1.449s (+3.6%)2.006s (~)0.557s151.21x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.668s (-25.2% 🟢)4.041s (-23.6% 🟢)1.373s81.00x
▲ VercelNext.js (Turbopack)2.876s (-22.7% 🟢)4.782s (-8.6% 🟢)1.907s71.08x
▲ VercelExpress3.339s (+43.7% 🔺)5.449s (+43.2% 🔺)2.110s61.25x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Nitro1.304s (-9.6% 🟢)2.075s (-13.3% 🟢)0.771s151.00x
🐘 PostgresExpress1.310s (-4.1%)2.007s (-13.3% 🟢)0.697s151.00x
🐘 PostgresNext.js (Turbopack)1.436s (-6.9% 🟢)2.076s (-6.6% 🟢)0.640s151.10x
💻 LocalNitro1.935s (+14.3% 🔺)2.293s (+14.2% 🔺)0.358s141.48x
💻 LocalExpress2.003s (+5.5% 🔺)2.393s (+11.3% 🔺)0.390s131.54x
💻 LocalNext.js (Turbopack)2.369s (+9.3% 🔺)3.009s (~)0.639s101.82x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.899s (+6.3% 🔺)4.683s (+2.5%)1.785s71.00x
▲ VercelNitro3.978s (+42.1% 🔺)5.734s (+37.1% 🔺)1.756s61.37x
▲ VercelNext.js (Turbopack)4.216s (-5.6% 🟢)5.944s (-3.4%)1.728s61.45x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.452s (-19.3% 🟢)4.014s (~)2.561s81.00x
🐘 PostgresNitro1.589s (-12.0% 🟢)3.678s (-14.4% 🟢)2.090s91.09x
🐘 PostgresNext.js (Turbopack)2.136s (-47.5% 🟢)3.454s (-22.3% 🟢)1.318s91.47x
💻 LocalExpress5.580s (+21.6% 🔺)6.014s (+20.0% 🔺)0.434s53.84x
💻 LocalNitro5.616s (+25.0% 🔺)6.017s (+20.1% 🔺)0.401s53.87x
💻 LocalNext.js (Turbopack)6.852s (+18.0% 🔺)7.416s (+15.6% 🔺)0.563s54.72x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express3.610s (-57.2% 🟢)5.562s (-45.7% 🟢)1.952s61.00x
▲ VercelNitro4.046s (-70.8% 🟢)5.724s (-63.4% 🟢)1.678s61.12x
▲ VercelNext.js (Turbopack)4.124s (+9.0% 🔺)6.272s (+19.9% 🔺)2.147s51.14x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.605s (+6.0% 🔺)1.041s (+3.5%)0.436s581.00x
💻 LocalExpress0.616s (-1.5%)1.005s (-1.7%)0.389s601.02x
💻 LocalNitro0.629s (+25.8% 🔺)1.039s (+3.1%)0.410s581.04x
🐘 PostgresNitro0.692s (+16.5% 🔺)1.078s (+3.6%)0.386s561.14x
🐘 PostgresNext.js (Turbopack)0.858s (+2.7%)1.041s (+1.7%)0.183s581.42x
💻 LocalNext.js (Turbopack)0.860s (-3.1%)1.005s (-3.3%)0.144s601.42x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.774s (-42.9% 🟢)5.457s (-34.5% 🟢)1.683s111.00x
▲ VercelNext.js (Turbopack)3.815s (-39.5% 🟢)5.390s (-33.8% 🟢)1.575s121.01x
▲ VercelExpress4.488s (-1.3%)6.377s (-1.6%)1.889s101.19x

🔍 Observability: Nitro | Next.js (Turbopack) | Express

workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.375s (+1.4%)2.052s (+1.2%)0.678s441.00x
🐘 PostgresNitro1.466s (+5.6% 🔺)2.030s (~)0.564s451.07x
💻 LocalNitro1.485s (+25.1% 🔺)2.006s (~)0.521s451.08x
💻 LocalExpress1.530s (+2.8%)2.007s (~)0.476s451.11x
🐘 PostgresNext.js (Turbopack)1.989s (+2.5%)2.308s (+11.2% 🔺)0.319s401.45x
💻 LocalNext.js (Turbopack)2.090s (-0.6%)2.944s (-2.1%)0.854s311.52x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express10.297s (-41.1% 🟢)12.369s (-35.5% 🟢)2.072s81.00x
▲ VercelNitro10.774s (-8.8% 🟢)12.396s (-12.5% 🟢)1.621s81.05x
▲ VercelNext.js (Turbopack)11.321s (-16.7% 🟢)13.637s (-11.0% 🟢)2.316s71.10x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express2.654s (-2.1%)3.058s (-1.7%)0.404s401.00x
🐘 PostgresNitro2.903s (+3.7%)3.280s (+3.6%)0.376s371.09x
💻 LocalExpress3.275s (+2.4%)4.010s (~)0.735s301.23x
💻 LocalNitro3.362s (+23.1% 🔺)4.010s (+24.4% 🔺)0.648s301.27x
🐘 PostgresNext.js (Turbopack)3.983s (+2.9%)4.253s (+4.3%)0.270s291.50x
💻 LocalNext.js (Turbopack)4.363s (~)5.010s (~)0.647s241.64x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express18.286s (-29.9% 🟢)20.532s (-26.7% 🟢)2.246s61.00x
▲ VercelNitro18.713s (-28.9% 🟢)20.382s (-27.5% 🟢)1.669s61.02x
▲ VercelNext.js (Turbopack)20.206s (-19.2% 🟢)22.420s (-16.3% 🟢)2.214s61.11x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.232s (~)1.006s (~)0.773s601.00x
🐘 PostgresNitro0.237s (+0.7%)1.006s (~)0.769s601.02x
🐘 PostgresNext.js (Turbopack)0.297s (+1.1%)1.023s (+1.7%)0.726s591.28x
💻 LocalExpress0.400s (-9.2% 🟢)1.005s (~)0.605s601.72x
💻 LocalNitro0.417s (+11.8% 🔺)1.004s (~)0.588s601.79x
💻 LocalNext.js (Turbopack)0.633s (+8.8% 🔺)1.004s (~)0.371s602.73x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express1.787s (+15.3% 🔺)3.598s (+25.4% 🔺)1.811s181.00x
▲ VercelNitro1.933s (+49.7% 🔺)3.615s (+25.8% 🔺)1.682s171.08x
▲ VercelNext.js (Turbopack)2.387s (+27.6% 🔺)4.344s (+11.8% 🔺)1.956s141.34x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.331s (-6.9% 🟢)1.006s (-3.3%)0.675s901.00x
🐘 PostgresNitro0.356s (+4.1%)1.006s (-1.1%)0.650s901.08x
🐘 PostgresNext.js (Turbopack)0.476s (-11.7% 🟢)1.103s (-1.3%)0.628s831.44x
💻 LocalNitro2.182s (+43.2% 🔺)2.737s (+27.2% 🔺)0.555s336.59x
💻 LocalExpress2.222s (+6.3% 🔺)2.738s (+3.1%)0.517s336.71x
💻 LocalNext.js (Turbopack)2.493s (+7.0% 🔺)3.344s (+8.7% 🔺)0.851s277.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.638s (-9.8% 🟢)4.553s (-5.1% 🟢)1.915s201.00x
▲ VercelNitro2.705s (-1.5%)4.386s (-8.3% 🟢)1.681s211.03x
▲ VercelNext.js (Turbopack)3.092s (+1.6%)4.802s (~)1.710s191.17x

🔍 Observability: Express | Nitro | Next.js (Turbopack)

workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.526s (-7.4% 🟢)1.059s (-13.0% 🟢)0.532s1141.00x
🐘 PostgresNitro0.570s (~)1.078s (-12.5% 🟢)0.508s1121.08x
🐘 PostgresNext.js (Turbopack)1.918s (-26.7% 🟢)2.763s (-20.4% 🟢)0.845s443.64x
💻 LocalNitro9.521s (+50.8% 🔺)10.445s (+54.9% 🔺)0.924s1218.09x
💻 LocalExpress10.198s (+22.8% 🔺)11.029s (+24.2% 🔺)0.831s1219.38x
💻 LocalNext.js (Turbopack)10.284s (-4.4%)11.663s (+0.8%)1.379s1119.54x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro3.245s (-38.5% 🟢)5.031s (-28.3% 🟢)1.786s241.00x
▲ VercelExpress3.483s (-36.2% 🟢)5.687s (-19.8% 🟢)2.204s221.07x
▲ VercelNext.js (Turbopack)4.699s (-2.9%)6.708s (+2.7%)2.008s181.45x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.165s (-1.1%)2.000s (~)0.001s (+9.1% 🔺)2.009s (~)0.844s101.00x
🐘 PostgresNitro1.181s (+1.4%)1.995s (~)0.001s (+30.0% 🔺)2.011s (~)0.831s101.01x
💻 LocalNitro1.181s (+5.7% 🔺)2.005s (~)0.010s (-62.9% 🟢)2.017s (-1.0%)0.836s101.01x
💻 LocalExpress1.185s (+3.0%)2.005s (~)0.012s (+19.8% 🔺)2.019s (~)0.835s101.02x
💻 LocalNext.js (Turbopack)1.216s (~)2.003s (~)0.013s (+2.4%)2.020s (~)0.803s101.04x
🐘 PostgresNext.js (Turbopack)1.232s (~)2.002s (~)0.001s (-15.4% 🟢)2.011s (~)0.779s101.06x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express2.206s (-8.2% 🟢)3.426s (-6.9% 🟢)0.602s (-18.8% 🟢)4.513s (-7.4% 🟢)2.307s101.00x
▲ VercelNext.js (Turbopack)2.304s (-8.6% 🟢)3.523s (-7.8% 🟢)0.772s (-4.3%)4.790s (-5.8% 🟢)2.486s101.04x
▲ VercelNitro2.331s (-11.7% 🟢)3.158s (-17.0% 🟢)1.253s (+57.9% 🔺)4.805s (-4.9%)2.474s101.06x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Nitro1.572s (+9.3% 🔺)2.011s (~)0.012s (-32.0% 🟢)2.025s (~)0.453s301.00x
🐘 PostgresExpress1.577s (~)2.004s (~)0.005s (+5.7% 🔺)2.026s (~)0.449s301.00x
💻 LocalExpress1.583s (+1.0%)2.010s (~)0.014s (+4.9%)2.025s (~)0.442s301.01x
🐘 PostgresNitro1.602s (+1.5%)2.008s (~)0.005s (-5.6% 🟢)2.027s (~)0.425s301.02x
💻 LocalNext.js (Turbopack)1.740s (-0.5%)2.010s (~)0.013s (+1.3%)2.025s (~)0.285s301.11x
🐘 PostgresNext.js (Turbopack)1.892s (+5.9% 🔺)2.011s (~)0.005s (+5.8% 🔺)2.029s (~)0.137s301.20x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Next.js (Turbopack)6.842s (-19.5% 🟢)8.374s (-14.7% 🟢)0.220s (-16.3% 🟢)9.079s (-14.1% 🟢)2.236s71.00x
▲ VercelNitro6.905s (-3.6%)7.768s (-10.3% 🟢)0.455s (+91.9% 🔺)8.766s (-6.9% 🟢)1.861s71.01x
▲ VercelExpress8.068s (+26.4% 🔺)9.373s (+20.1% 🔺)0.307s (+71.5% 🔺)10.412s (+23.1% 🔺)2.345s61.18x

🔍 Observability: Next.js (Turbopack) | Nitro | Express

10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.789s (+3.3%)1.101s (+7.5% 🔺)0.000s (-100.0% 🟢)1.118s (+6.0% 🔺)0.329s541.00x
🐘 PostgresNitro0.795s (+2.9%)1.044s (~)0.000s (-3.5%)1.061s (-3.0%)0.265s571.01x
🐘 PostgresNext.js (Turbopack)0.986s (-3.3%)1.397s (-4.6%)0.000s (-100.0% 🟢)1.407s (-4.4%)0.421s431.25x
💻 LocalExpress1.533s (+12.5% 🔺)2.014s (~)0.000s (+16.7% 🔺)2.016s (~)0.483s301.94x
💻 LocalNitro1.568s (+45.8% 🔺)2.014s (+9.6% 🔺)0.000s (-56.8% 🟢)2.016s (+9.5% 🔺)0.448s301.99x
💻 LocalNext.js (Turbopack)1.916s (+26.7% 🔺)2.193s (+8.9% 🔺)0.000s (+7.1% 🔺)2.196s (+8.9% 🔺)0.280s282.43x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Nitro2.961s (-21.2% 🟢)4.002s (-25.7% 🟢)0.000s (-21.4% 🟢)4.479s (-23.6% 🟢)1.518s141.00x
▲ VercelExpress3.140s (+3.6%)4.516s (-1.1%)0.000s (NaN%)5.069s (+0.6%)1.929s121.06x
▲ VercelNext.js (Turbopack)3.540s (+2.7%)5.016s (+5.2% 🔺)0.000s (-100.0% 🟢)5.553s (+5.5% 🔺)2.013s111.20x

🔍 Observability: Nitro | Express | Next.js (Turbopack)

fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.691s (+4.7%)2.302s (+11.5% 🔺)0.000s (NaN%)2.316s (+10.9% 🔺)0.625s261.00x
🐘 PostgresNitro1.779s (+14.7% 🔺)2.381s (+11.6% 🔺)0.000s (+11.5% 🔺)2.394s (+11.5% 🔺)0.615s261.05x
🐘 PostgresNext.js (Turbopack)2.251s (+6.6% 🔺)2.651s (+2.5%)0.000s (-100.0% 🟢)2.664s (+2.6%)0.413s231.33x
💻 LocalNitro4.321s (+95.8% 🔺)4.717s (+72.0% 🔺)0.001s (+12.8% 🔺)4.729s (+71.7% 🔺)0.409s132.56x
💻 LocalExpress4.772s (+62.9% 🔺)5.362s (+46.1% 🔺)0.001s (+98.3% 🔺)5.367s (+46.1% 🔺)0.595s122.82x
💻 LocalNext.js (Turbopack)5.632s (+94.2% 🔺)6.226s (+85.4% 🔺)0.000s (-60.0% 🟢)6.235s (+85.4% 🔺)0.602s103.33x

▲ Production (Vercel)

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
▲ Vercel🥇 Express4.310s (-24.3% 🟢)6.054s (-16.2% 🟢)0.000s (+Infinity% 🔺)6.546s (-14.9% 🟢)2.236s101.00x
▲ VercelNext.js (Turbopack)4.772s (-34.2% 🟢)6.201s (-26.1% 🟢)0.000s (-100.0% 🟢)6.679s (-24.5% 🟢)1.907s101.11x
▲ VercelNitro4.835s (-7.9% 🟢)5.805s (-10.1% 🟢)0.000s (+Infinity% 🔺)6.308s (-8.9% 🟢)1.473s101.12x

🔍 Observability: Express | Next.js (Turbopack) | Nitro

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalNitro12/21
🐘 PostgresExpress19/21
▲ VercelExpress14/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres17/21
Next.js (Turbopack)🐘 Postgres14/21
Nitro🐘 Postgres13/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run

@pranaygp

Copy link
Copy Markdown
ContributorAuthor

CI failure triage — pre-existing Vercel-prod e2e flake (not a regression)

The two red checks (E2E Vercel Prod Tests (example), E2E Vercel Prod Tests (fastify), which roll up into E2E Required Check) are the shared Vercel-prod timing flake, not caused by this PR. Evidence:

Scope is wrong for a hydration regression. This PR only memoizes primitive step-result hydration. A determinism/stale-value bug there would surface across all worlds — yet every local suite is 100% green:

SuitePassedFailed
▲ Vercel Production14402
💻 Local Development19090
📦 Local Production19090
🐘 Local Postgres18950
🪟 Windows1520

Only 2 failures, only on Vercel Production.

The two failing tests are unrelated to result hydration, and are abort/hook timing races:

  1. exampleAbortController > abortFromStepWorkflow: step abort cancels an in-flight sibling step. The run completed successfully; the assertion failed only because the abort lost a race. From the run diagnostics (wrun_01KV9MZ1Y00N90XFCKFTSPX5N1):

    +2.2s step_completed (longStep) <- sibling finished on its own
    +3.0s hook_received <- abort signal arrived AFTER
    

    The sibling longStep self-completed (2.2s) before the abort hook arrived (3.0s) under Vercel-prod queue/network latency, so there was no in-flight step left to cancel. Step results hydrated fine; this is purely environmental latency.

  2. fastifystartFromWorkflow - calling start() directly inside a workflow function with hook communication. This exact test passed (4128ms) in the example job of this same commit (a56f5c90b) — a textbook cross-run flake.

The same test is red on plain main, without this change. On main run 27704378960 (commit 2acf13cc7):

  • E2E Vercel Prod Tests (tanstack-start)abortFromStepWorkflow: step abort cancels an in-flight sibling step (the identical test that failed here on example)
  • E2E Vercel Prod Tests (nextjs-turbopack)distributedAbortController - reconnect to existing controller

And on main run 27657696161 (cb181392b, the commit this branch is based on): E2E Vercel Prod Tests (fastify)hookWithSleepFinalStepWorkflow. The set of red workbenches rotates run-to-run — the signature of environment flakiness, not a code regression.

Local verification of this branch (rebuilt @workflow/core first): cd packages/core && pnpm test1249 passed / 56 files, 0 failures, including the determinism/ordering replay tests in workflow.test.ts and the new step-hydration-cache.test.ts (10) + step-hydration-memoization.test.ts (3). The memoization tests assert byte-identical delivery ordering on cache hits and that objects re-hydrate fresh each replay.

Re-running the e2e jobs should clear them. No code change is warranted.

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — the O(N²)→O(N) hydration memoization, done with the right safety bias

This is the most safety-sensitive of the four (it's the only one that caches a value across replays), and the design lands on the conservative side of every judgment call, which is exactly right for replay determinism.

The primitives-only decision is the crux and it's correct.hydrateStepReturnValue (devalue.parse) returns a fresh object graph each call, and each replay runs in a fresh VM, so today the workflow gets a brand-new value every replay. Caching and returning the same object reference would let const r = await step(); r.count++ observe a prior replay's mutation — silent divergence. The alternatives are both worse: structuredClone is lossy for reviver-produced specials (stream handles, step-fn proxies, Request/Response, AbortController/Signal) and still O(size). Restricting the cache to primitives (immutable, compared by value) makes "share the reference" provably indistinguishable from re-parsing, and non-primitives fall through to a full re-hydrate every replay — preserving current behavior exactly. Trading the object-case optimization for airtight determinism is the right call.

What I verified in the integration:

  • Surgical wrap: only the await hydrateStepReturnValue(...) call is replaced; pendingDeliveries++/--, the ctx.promiseQueue.then(...) slot, and resolve(...) are byte-for-byte unchanged. The lookup helper always returns a Promise and awaits even on the miss path, so a cache hit occupies the identical position in the ordered delivery chain a re-hydrate would have — preserving the pendingDeliveries-gated suspension/barrier/Promise.race determinism.
  • Lifetime: cache is created once per invocation in runtime.ts (outside the per-iteration context), threaded through runWorkflow, never shared across runs. The optional param/context field degrades to re-hydrating-every-replay for harnesses that omit it.
  • Keying by step_completedeventId (stable, world-assigned, same immutable bytes every replay) is sound, and has() rather than get() !== undefined correctly treats a memoized undefined result as a hit.
  • Errors never cached — a rejected hydrate re-attempts next replay, no parked rejected promise.

Built @workflow/core; full suite green (1249) including the new step-hydration-cache.test.ts / step-hydration-memoization.test.ts, async-deserialization-ordering.test.ts, and workflow.test.ts.

One forward-looking note (non-blocking): the object case is left on the table by design. If it's ever revisited, the safe path is deep-freeze + share frozen graphs — but only after handling the reviver-produced specials, so I agree with deferring it. LGTM.

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

// Only memoize values that are safe to return by reference across replays.
// Non-primitives fall through and are re-hydrated fresh on every replay.
if (isMemoizablePrimitive(value)) {
cache.set(eventId, value);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

The per-run stepHydrationCache is never size-bounded or evicted: it grows one entry per primitive-returning completed step and lives for the entire invocation. The cost worth calling out (the PR doesn't) is the new residency — the decrypted/devalue-parsed plaintext of each cached primitive is now held for the whole invocation, on top of the serialized bytes already retained in cachedEvents. For a long sequential workflow whose steps return large strings, that roughly doubles peak retained memory for those results.

The dominant residency (the full event log in cachedEvents) already exists, so this isn't blocking, but a byte-size threshold would be a cheap safeguard — large primitives are exactly the cheap-to-re-hydrate case relative to their footprint, so letting them fall through to the existing re-hydrate path costs little and bounds the worst case. Ideally with a test asserting the bound.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — capped it, with a test. Pushed in a60baad.

What changed (step-hydration-cache.ts):

  • Added MAX_MEMOIZED_PRIMITIVE_LENGTH = 4096 and extended isMemoizablePrimitive so a string/bigint longer than 4 KiB is treated as non-memoizable. Those are the only primitive types that can carry a large payload — number/boolean/null/undefined/symbol are inherently small, so they're never length-checked. Oversized values now fall through to the existing per-replay re-hydrate path, exactly as you suggested: large primitives are cheap to re-hydrate relative to their footprint, so this caps the doubled-residency worst case at negligible cost.
  • Documented the memory characteristic on the cache module: per-invocation lifetime (fresh Map per run in runtime.ts, GC'd when the invocation returns), bounded by the number of primitive-returning completed steps, primitives-only, now byte-bounded.

Tests (step-hydration-cache.test.ts, +4):isMemoizablePrimitive true at the bound / false beyond it (string and bigint), and an end-to-end assertion that an oversized string re-hydrates on every replay and cache.size === 0 (the bound assertion you asked for); plus an at-bound string is a cache hit.

The cap only ever reduces what gets cached, so determinism is untouched — oversized values just take the already-correct re-hydrate path. Full core suite green (1253, incl. the ordering/determinism + memoization suites); biome + tsc clean.

On consistency with #2471 (the sibling scriptCache): noting the distinction since they're bounded for different reasons. #2471's cache is process-wide and monotonic across the whole process — in dev/watch it pins every historical bundle string (hundreds of MB over a session), which is a genuine regression vs. the prior keep-only-latest behavior, hence the Blocking bound there. This cache is per-invocation and freed wholesale when the run returns, so it can never accumulate across runs; the only real cost is the doubled residency for large primitives during one run, which the size cap here now bounds. Different scope, different severity, but both bounded now.

Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-hydration
* origin/main:
perf(core): lazy inline step start (save one world round-trip per step) (#2478)
perf(core): skip per-step events.list via inline event-log delta (#2475)
Version Packages (beta) (#2491)
[world-vercel] Honor hasMore flag from v4 list pagination endpoint (#2486)
Version Packages (beta) (#2451)
Fix Next workflow module specifier root (#2455)
[world-vercel] Send remoteRefBehavior=lazy on v4 metadata-only event listings (#2415)
[swc-plugin] Fix eager discovery for object property steps (#2484)
fix(web-shared): align attributes panel styling (#2483)
[web-shared] Auto-scroll trace viewer on J/K span navigation (#2366)
fix(web): render restarted step segment as solid gray, not running stripes (#2480)
fix(web-shared): use solid gray for queued trace segment (#2474)
Add trace viewer span markers for hooks and attributes (#2452)
test: support Vercel protection bypass secret in e2e headers (#2458)
fix(core): bump payload-compression cutoff to 5.0.0-beta.18 (#2470)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts:
#	packages/core/src/runtime.ts
@pranaygp
pranaygp enabled auto-merge (squash) June 18, 2026 06:24
@pranaygp
pranaygp disabled auto-merge June 22, 2026 20:30
@pranaygp
pranaygp merged commit 66ca0dc into mainJun 22, 2026
118 of 121 checks passed
@pranaygp
pranaygp deleted the pgp/perf-memoize-step-hydration branch June 22, 2026 20:30
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
github-actionsBot added a commit that referenced this pull request Jun 22, 2026
…#2472)
* perf(core): memoize step return value hydration across replays
The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): bound memoized step-hydration cache by primitive size
Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #2563. Merge conflicts were resolved by AI — please review carefully. (backport job run)

pranaygp added a commit that referenced this pull request Jun 22, 2026
…testing
* origin/main:
Version Packages (beta) (#2540)
perf(core): memoize step return value hydration across inline replays (#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (#2412)
Amend lazy discovery removal changeset (#2560)
[docs] Document minimum SDK version for using hook.getConflict (#2423)
Update default CODEOWNERS (#2556)
Optimize and fix the default eager build mode (#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (#2547)" (#2554)
[core] Turbo mode: fast-path the first invocation (#2526)
Remove lazy discovery from workflow/next (#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (#2547)
feat(docs): add eve and AI SDK to product switcher (#2543)
[vitest] Fix local imports failing to load in test step bundles (#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (#2324)
Version Packages (beta) (#2495)
otel(world-vercel): inject trace context on v4 event requests (#2533)
Bump undici to 7.28.0 (#2534)
Default source maps to dev-on / prod-off (#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (#2527)
perf(core): parallel inline steps + optimistic lazy step start (#2516)
pranaygp added a commit to marcopiraccini/workflow that referenced this pull request Jun 22, 2026
* origin/main: (120 commits)
Version Packages (beta) (vercel#2540)
perf(core): memoize step return value hydration across inline replays (vercel#2472)
[core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (vercel#2412)
Amend lazy discovery removal changeset (vercel#2560)
[docs] Document minimum SDK version for using hook.getConflict (vercel#2423)
Update default CODEOWNERS (vercel#2556)
Optimize and fix the default eager build mode (vercel#2546)
Revert "fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)" (vercel#2554)
[core] Turbo mode: fast-path the first invocation (vercel#2526)
Remove lazy discovery from workflow/next (vercel#2545)
fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)
feat(docs): add eve and AI SDK to product switcher (vercel#2543)
[vitest] Fix local imports failing to load in test step bundles (vercel#2351)
[builders] Fix unicode-escape crash in workflow graph extraction (vercel#2324)
Version Packages (beta) (vercel#2495)
otel(world-vercel): inject trace context on v4 event requests (vercel#2533)
Bump undici to 7.28.0 (vercel#2534)
Default source maps to dev-on / prod-off (vercel#2529)
otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (vercel#2527)
perf(core): parallel inline steps + optimistic lazy step start (vercel#2516)
...
VaguelySerious pushed a commit that referenced this pull request Jun 22, 2026
@github-actionsgithub-actionsBot mentioned this pull request Jun 22, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pranaygp@TooTallNate@VaguelySerious