Instrument workflow replay scaling - #2969

Closed
NathanColosimo wants to merge 9 commits into
mainfrom
codex/instrument-workflow-replay
Closed

Instrument workflow replay scaling#2969
NathanColosimo wants to merge 9 commits into
mainfrom
codex/instrument-workflow-replay

Conversation

@NathanColosimo

@NathanColosimoNathanColosimo commented Jul 16, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • split workflow.run into VM context, bundle evaluation, argument hydration, execution, serialization, and drain spans
  • aggregate replay event-consumer, subscription-wait, step hydration, and cache diagnostics without per-step spans
  • split step-result hydration into decrypt, decompress, compression telemetry, and deserialize timings
  • add an isolated 20/120/1020-step replay benchmark and route PR benchmark CI through the protected workflow-server preview

Production finding

The production VM itself is not intrinsically hundreds of milliseconds slower than local replay. The original comparison mixed a tiny synthetic local bundle with a 1.322 MB production bundle, encrypted object results, OpenTelemetry, and DEBUG=workflow:*.

Disabling debug logging on the same preview reduced late workflow.run from 604.6 ms to 247.6 ms. The synchronous event-consumer scan fell from 165.3 ms to 11.9 ms because each enabled per-event debug call emitted console output and an active-span event.

The final instrumented, debug-off, non-turbo run was CI 29548488910, workflow run wrun_41KXPWKTB70GJFGJB4Q00XBH2W.

Mean (ms)K1-K19K101-K119K1001-K1019
STSO232.0355.3503.5
prior step_completed durable write70.978.571.1
event refresh0.04102.8106.4
workflow.run72.990.8242.0
next step_started durable write83.079.279.3

The refresh cliff is the event pagination issue being handled separately in workflow-server PR #632. It is not the continuous replay slope.

Replay breakdown

Mean (ms)K1-K19K101-K119K1001-K1019
VM context1.000.961.08
evaluate 1.322 MB bundle49.1744.9851.45
workflow execute/replay16.8242.48187.21
synchronous event scan0.221.4411.54
step-result hydration14.3136.31144.70
AES-GCM decrypt13.5730.71101.85
decompress0.070.633.68
compression telemetry0.231.5514.51
deserialize0.352.6517.73
events / callback invocations32 / 75332 / 7753032 / 7075

The subscription-wait metric is 18.8 / 39.3 / 160.8 ms, but it overlaps hydration and VM continuation and must not be added to the rows above.

Every benchmark result is an object, so all 10 / 110 / 1010 historical step values are non-memoizable by the primitive-only hydration cache. Each replay serially performs one WebCrypto AES-GCM decrypt and one fresh deserialize per historical result. Across 1020 serial steps that is approximately sum(1..1020) = 520,710 decrypt/parse operations in one invocation.

The exact local reproduction used the generated 1.322 MB bundle, encrypted object results, Node 22 Linux, and a 1-vCPU/2-GB container. At K1020 it measured 183.7 ms total workflow.run, including 50.6 ms bundle evaluation, 124.5 ms execution, 86.0 ms hydration, and 9.1 ms synchronous event scan. The preview measured 242.0 / 51.5 / 187.2 / 144.7 / 11.5 ms respectively. The remaining local-cloud gap is therefore hydration/WebCrypto throughput and scheduling, not node:vm context or bundle evaluation.

Evidence

  • original debug-on benchmark: CI 29544762077, traces d6ec13404e2537410dac512a521da423 and fec6da2c5114eb50e1eb9a9c31a23902
  • debug-off A/B benchmark: CI 29547654943
  • hydration split benchmark: CI 29548488910, traces 0d3358d44cc286fec90623114cce3d92 and 80879ec66b238c5727fb3f981d164ea3

Verification

  • targeted replay telemetry and serialization tests: 4 passed
  • pnpm --filter @workflow/core typecheck
  • pnpm --filter @workflow/core build

@changeset-bot

changeset-botBot commented Jul 16, 2026•
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 21397aa

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actionsBot commented Jul 16, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

✅ All tests passed

Summary

PassedFailedSkippedTotal
✅ ▲ Vercel Production145302301683
✅ 💻 Local Development161702191836
✅ 📦 Local Production161702191836
✅ 🐘 Local Postgres161702191836
✅ 🪟 Windows15300153
✅ 📋 Other89401771071
✅ vercel-multi-region270027
Total7378010648442

Details by Category

✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro126027
✅ example126027
✅ express126027
✅ fastify126027
✅ hono126027
✅ nextjs-turbopack15003
✅ nextjs-webpack15003
✅ nitro126027
✅ nuxt126027
✅ sveltekit14508
✅ vite126027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15300
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128025
✅ e2e-local-dev-tanstack-start-128025
✅ e2e-local-postgres-nest-stable128025
✅ e2e-local-postgres-tanstack-start-128025
✅ e2e-local-prod-nest-stable128025
✅ e2e-local-prod-tanstack-start-128025
✅ e2e-vercel-prod-tanstack-start126027
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@github-actions

github-actionsBot commented Jul 16, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

⏳ Benchmarks are running for 21397aa... (run logs)

Results below are from a previous run.

commit 81a3155 · Fri, 17 Jul 2026 02:04:18 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream843 (-22%)834 🔴1036 🔴1887 🔴30
TTFShook + stream776 (-46%)801 🔴906 🔴1044 🔴30
STSO1020 steps (1-20)232 (-23%)239 🔴318 🔴482 🔴19
STSO1020 steps (101-120)355 (-18%)392 🔴462 🔴476 🔴19
STSO1020 steps (1001-1020)504 (-41%)515 🔴591 🔴808 🔴19
WOstream843 (-22%)8341036188730
WOhook + stream776 (-46%)801906104430
SLstream1718 (-63%)1788 🔴1877 🔴5819 🔴30
SLhook + stream1615 (-67%)1670 🔴1722 🔴1874 🔴30
📜 Previous results (3)

0ed0294

Fri, 17 Jul 2026 01:44:11 GMT · run logs

vercel / nextjs-turbopack

MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream842 (-23%)744 🔴798 🔴4537 🔴30
TTFShook + stream773 (-46%)833 🔴850 🔴1904 🔴30
STSO1020 steps (1-20)259 (-14%)259 🔴414 🔴591 🔴19
STSO1020 steps (101-120)438 (+1.2%)496 🔴614 🔴642 🔴19
STSO1020 steps (1001-1020)633 (-26%)550 🔴577 🔴2949 🔴19
WOstream842 (-23%)744798453730
WOhook + stream773 (-46%)833850190430
SLstream1588 (-66%)1625 🔴1743 🔴2798 🔴30
SLhook + stream1589 (-67%)1560 🔴1686 🔴5515 🔴30

b0ded7f

Fri, 17 Jul 2026 00:47:52 GMT · run logs

vercel / nextjs-turbopack

MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1499 (+38%)1890 🔴1925 🔴2052 🔴30
TTFShook + stream1468 (+2.7%)1988 🔴2083 🔴2225 🔴30
STSO1020 steps (1-20)301 (±0%)326 🔴351 🔴510 🔴19
STSO1020 steps (101-120)441 (+1.8%)496 🔴549 🔴590 🔴19
STSO1020 steps (1001-1020)887 (+3.5%)926 🔴1030 🔴1052 🔴19
WOstream1499 (+38%)18901925205230
WOhook + stream1468 (+2.7%)19882083222530
SLstream4955 (+5.8%)5589 🔴5679 🔴5856 🔴30
SLhook + stream5072 (+4.5%)5641 🔴5750 🔴5857 🔴30

5f7515a

Fri, 17 Jul 2026 00:18:52 GMT · run logs

vercel / nextjs-turbopack

MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1191 (+9.6%)1662 🔴1710 🔴1775 🔴30
TTFShook + stream1630 (+14%)1878 🔴1970 🔴2928 🔴30
STSO1020 steps (1-20)283 (-6.1%)331 🔴398 🔴418 🔴19
STSO1020 steps (101-120)433 (±0%)473 🔴564 🔴605 🔴19
STSO1020 steps (1001-1020)943 (+10%)1007 🔴1105 🔴1318 🔴19
WOstream1191 (+9.6%)16621710177530
WOhook + stream1630 (+14%)18781970292830
SLstream4605 (-1.6%)5569 🔴5617 🔴5920 🔴30
SLhook + stream4779 (-1.5%)5063 🔴5814 🔴6828 🔴30

Avg deltas compare against the most recent benchmark run on main at the time of this run.

Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader)

Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges

🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120

TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request.

@vercel

vercelBot commented Jul 16, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-nextjs-workflow-webpackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-astro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-express-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-fastify-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-hono-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nitro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nuxt-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-sveltekit-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-tanstack-start-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-vite-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-swc-playgroundReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-tarballsReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-webReadyReadyPreview, CommentJul 17, 2026 2:12am
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedJul 17, 2026 2:12am

@vercel
vercelBottemporarily deployed to Preview – workflow-docs July 17, 2026 01:31 Inactive
@NathanColosimo

NathanColosimo commented Jul 17, 2026

Copy link
Copy Markdown
ContributorAuthor

The benchmark triggered by the final preview-config restore commit was intentionally canceled because it would no longer run with the debug-off/non-turbo experimental controls. The latest valid diagnostic result is commit 81a3155, run 29548488910; the PR body now contains its full trace breakdown. The final branch has restored the normal preview config.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@NathanColosimo
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Instrument workflow replay scaling - #2969
Closed
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Closed
Instrument workflow replay scaling#2969
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • split workflow.run into VM context, bundle evaluation, argument hydration, execution, serialization, and drain spans
  • aggregate replay event-consumer, subscription-wait, step hydration, and cache diagnostics without per-step spans
  • split step-result hydration into decrypt, decompress, compression telemetry, and deserialize timings
  • add an isolated 20/120/1020-step replay benchmark and route PR benchmark CI through the protected workflow-server preview
Production finding
The production VM itself is not intrinsically hundreds of milliseconds slower than local replay. The original comparison mixed a tiny synthetic local bundle with a 1.322 MB production bundle, encrypted object results, OpenTelemetry, and DEBUG=workflow:*.
Disabling debug logging on the same preview reduced late workflow.run from 604.6 ms to 247.6 ms. The synchronous event-consumer scan fell from 165.3 ms to 11.9 ms because each enabled per-event debug call emitted console output and an active-span event.
The final instrumented, debug-off, non-turbo run was CI 29548488910, workflow run wrun_41KXPWKTB70GJFGJB4Q00XBH2W.
Mean (ms)K1-K19K101-K119K1001-K1019
STSO232.0355.3503.5
prior step_completed durable write70.978.571.1
event refresh0.04102.8106.4
workflow.run72.990.8242.0
next step_started durable write83.079.279.3
The refresh cliff is the event pagination issue being handled separately in workflow-server PR #632. It is not the continuous replay slope.
Replay breakdown
Mean (ms)K1-K19K101-K119K1001-K1019
VM context1.000.961.08
evaluate 1.322 MB bundle49.1744.9851.45
workflow execute/replay16.8242.48187.21
synchronous event scan0.221.4411.54
step-result hydration14.3136.31144.70
AES-GCM decrypt13.5730.71101.85
decompress0.070.633.68
compression telemetry0.231.5514.51
deserialize0.352.6517.73
events / callback invocations32 / 75332 / 7753032 / 7075
The subscription-wait metric is 18.8 / 39.3 / 160.8 ms, but it overlaps hydration and VM continuation and must not be added to the rows above.
Every benchmark result is an object, so all 10 / 110 / 1010 historical step values are non-memoizable by the primitive-only hydration cache. Each replay serially performs one WebCrypto AES-GCM decrypt and one fresh deserialize per historical result. Across 1020 serial steps that is approximately sum(1..1020) = 520,710 decrypt/parse operations in one invocation.
The exact local reproduction used the generated 1.322 MB bundle, encrypted object results, Node 22 Linux, and a 1-vCPU/2-GB container. At K1020 it measured 183.7 ms total workflow.run, including 50.6 ms bundle evaluation, 124.5 ms execution, 86.0 ms hydration, and 9.1 ms synchronous event scan. The preview measured 242.0 / 51.5 / 187.2 / 144.7 / 11.5 ms respectively. The remaining local-cloud gap is therefore hydration/WebCrypto throughput and scheduling, not node:vm context or bundle evaluation.
Evidence
  • original debug-on benchmark: CI 29544762077, traces d6ec13404e2537410dac512a521da423 and fec6da2c5114eb50e1eb9a9c31a23902
  • debug-off A/B benchmark: CI 29547654943
  • hydration split benchmark: CI 29548488910, traces 0d3358d44cc286fec90623114cce3d92 and 80879ec66b238c5727fb3f981d164ea3
Verification
  • targeted replay telemetry and serialization tests: 4 passed
  • pnpm --filter @workflow/core typecheck
  • pnpm --filter @workflow/core build
@changeset-bot
changeset-botBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 21397aa
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145302301683
✅ 💻 Local Development161702191836
✅ 📦 Local Production161702191836
✅ 🐘 Local Postgres161702191836
✅ 🪟 Windows15300153
✅ 📋 Other89401771071
✅ vercel-multi-region270027
Total7378010648442
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro126027
✅ example126027
✅ express126027
✅ fastify126027
✅ hono126027
✅ nextjs-turbopack15003
✅ nextjs-webpack15003
✅ nitro126027
✅ nuxt126027
✅ sveltekit14508
✅ vite126027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15300
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128025
✅ e2e-local-dev-tanstack-start-128025
✅ e2e-local-postgres-nest-stable128025
✅ e2e-local-postgres-tanstack-start-128025
✅ e2e-local-prod-nest-stable128025
✅ e2e-local-prod-tanstack-start-128025
✅ e2e-vercel-prod-tanstack-start126027
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
⏳ Benchmarks are running for 21397aa... (run logs)
Results below are from a previous run.
commit 81a3155 · Fri, 17 Jul 2026 02:04:18 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream843 (-22%)834 🔴1036 🔴1887 🔴30
TTFShook + stream776 (-46%)801 🔴906 🔴1044 🔴30
STSO1020 steps (1-20)232 (-23%)239 🔴318 🔴482 🔴19
STSO1020 steps (101-120)355 (-18%)392 🔴462 🔴476 🔴19
STSO1020 steps (1001-1020)504 (-41%)515 🔴591 🔴808 🔴19
WOstream843 (-22%)8341036188730
WOhook + stream776 (-46%)801906104430
SLstream1718 (-63%)1788 🔴1877 🔴5819 🔴30
SLhook + stream1615 (-67%)1670 🔴1722 🔴1874 🔴30
📜 Previous results (3)
0ed0294
Fri, 17 Jul 2026 01:44:11 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream842 (-23%)744 🔴798 🔴4537 🔴30
TTFShook + stream773 (-46%)833 🔴850 🔴1904 🔴30
STSO1020 steps (1-20)259 (-14%)259 🔴414 🔴591 🔴19
STSO1020 steps (101-120)438 (+1.2%)496 🔴614 🔴642 🔴19
STSO1020 steps (1001-1020)633 (-26%)550 🔴577 🔴2949 🔴19
WOstream842 (-23%)744798453730
WOhook + stream773 (-46%)833850190430
SLstream1588 (-66%)1625 🔴1743 🔴2798 🔴30
SLhook + stream1589 (-67%)1560 🔴1686 🔴5515 🔴30
b0ded7f
Fri, 17 Jul 2026 00:47:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1499 (+38%)1890 🔴1925 🔴2052 🔴30
TTFShook + stream1468 (+2.7%)1988 🔴2083 🔴2225 🔴30
STSO1020 steps (1-20)301 (±0%)326 🔴351 🔴510 🔴19
STSO1020 steps (101-120)441 (+1.8%)496 🔴549 🔴590 🔴19
STSO1020 steps (1001-1020)887 (+3.5%)926 🔴1030 🔴1052 🔴19
WOstream1499 (+38%)18901925205230
WOhook + stream1468 (+2.7%)19882083222530
SLstream4955 (+5.8%)5589 🔴5679 🔴5856 🔴30
SLhook + stream5072 (+4.5%)5641 🔴5750 🔴5857 🔴30
5f7515a
Fri, 17 Jul 2026 00:18:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1191 (+9.6%)1662 🔴1710 🔴1775 🔴30
TTFShook + stream1630 (+14%)1878 🔴1970 🔴2928 🔴30
STSO1020 steps (1-20)283 (-6.1%)331 🔴398 🔴418 🔴19
STSO1020 steps (101-120)433 (±0%)473 🔴564 🔴605 🔴19
STSO1020 steps (1001-1020)943 (+10%)1007 🔴1105 🔴1318 🔴19
WOstream1191 (+9.6%)16621710177530
WOhook + stream1630 (+14%)18781970292830
SLstream4605 (-1.6%)5569 🔴5617 🔴5920 🔴30
SLhook + stream4779 (-1.5%)5063 🔴5814 🔴6828 🔴30
Avg deltas compare against the most recent benchmark run on main at the time of this run.
Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader)
Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges
🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120
TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request.
@vercel
vercelBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-nextjs-workflow-webpackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-astro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-express-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-fastify-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-hono-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nitro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nuxt-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-sveltekit-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-tanstack-start-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-vite-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-swc-playgroundReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-tarballsReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-webReadyReadyPreview, CommentJul 17, 2026 2:12am
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedJul 17, 2026 2:12am
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
July 17, 2026 01:31
Inactive

@NathanColosimo
NathanColosimo
commented
Jul 17, 2026

Copy link


Copy Markdown

ContributorAuthor
The benchmark triggered by the final preview-config restore commit was intentionally canceled because it would no longer run with the debug-off/non-turbo experimental controls. The latest valid diagnostic result is commit 81a3155, run 29548488910; the PR body now contains its full trace breakdown. The final branch has restored the normal preview config.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@NathanColosimo

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Instrument workflow replay scaling - #2969
Closed
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Closed
Instrument workflow replay scaling#2969
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • split workflow.run into VM context, bundle evaluation, argument hydration, execution, serialization, and drain spans
  • aggregate replay event-consumer, subscription-wait, step hydration, and cache diagnostics without per-step spans
  • split step-result hydration into decrypt, decompress, compression telemetry, and deserialize timings
  • add an isolated 20/120/1020-step replay benchmark and route PR benchmark CI through the protected workflow-server preview
Production finding
The production VM itself is not intrinsically hundreds of milliseconds slower than local replay. The original comparison mixed a tiny synthetic local bundle with a 1.322 MB production bundle, encrypted object results, OpenTelemetry, and DEBUG=workflow:*.
Disabling debug logging on the same preview reduced late workflow.run from 604.6 ms to 247.6 ms. The synchronous event-consumer scan fell from 165.3 ms to 11.9 ms because each enabled per-event debug call emitted console output and an active-span event.
The final instrumented, debug-off, non-turbo run was CI 29548488910, workflow run wrun_41KXPWKTB70GJFGJB4Q00XBH2W.
Mean (ms)K1-K19K101-K119K1001-K1019
STSO232.0355.3503.5
prior step_completed durable write70.978.571.1
event refresh0.04102.8106.4
workflow.run72.990.8242.0
next step_started durable write83.079.279.3
The refresh cliff is the event pagination issue being handled separately in workflow-server PR #632. It is not the continuous replay slope.
Replay breakdown
Mean (ms)K1-K19K101-K119K1001-K1019
VM context1.000.961.08
evaluate 1.322 MB bundle49.1744.9851.45
workflow execute/replay16.8242.48187.21
synchronous event scan0.221.4411.54
step-result hydration14.3136.31144.70
AES-GCM decrypt13.5730.71101.85
decompress0.070.633.68
compression telemetry0.231.5514.51
deserialize0.352.6517.73
events / callback invocations32 / 75332 / 7753032 / 7075
The subscription-wait metric is 18.8 / 39.3 / 160.8 ms, but it overlaps hydration and VM continuation and must not be added to the rows above.
Every benchmark result is an object, so all 10 / 110 / 1010 historical step values are non-memoizable by the primitive-only hydration cache. Each replay serially performs one WebCrypto AES-GCM decrypt and one fresh deserialize per historical result. Across 1020 serial steps that is approximately sum(1..1020) = 520,710 decrypt/parse operations in one invocation.
The exact local reproduction used the generated 1.322 MB bundle, encrypted object results, Node 22 Linux, and a 1-vCPU/2-GB container. At K1020 it measured 183.7 ms total workflow.run, including 50.6 ms bundle evaluation, 124.5 ms execution, 86.0 ms hydration, and 9.1 ms synchronous event scan. The preview measured 242.0 / 51.5 / 187.2 / 144.7 / 11.5 ms respectively. The remaining local-cloud gap is therefore hydration/WebCrypto throughput and scheduling, not node:vm context or bundle evaluation.
Evidence
  • original debug-on benchmark: CI 29544762077, traces d6ec13404e2537410dac512a521da423 and fec6da2c5114eb50e1eb9a9c31a23902
  • debug-off A/B benchmark: CI 29547654943
  • hydration split benchmark: CI 29548488910, traces 0d3358d44cc286fec90623114cce3d92 and 80879ec66b238c5727fb3f981d164ea3
Verification
  • targeted replay telemetry and serialization tests: 4 passed
  • pnpm --filter @workflow/core typecheck
  • pnpm --filter @workflow/core build
@changeset-bot
changeset-botBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 21397aa
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145302301683
✅ 💻 Local Development161702191836
✅ 📦 Local Production161702191836
✅ 🐘 Local Postgres161702191836
✅ 🪟 Windows15300153
✅ 📋 Other89401771071
✅ vercel-multi-region270027
Total7378010648442
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro126027
✅ example126027
✅ express126027
✅ fastify126027
✅ hono126027
✅ nextjs-turbopack15003
✅ nextjs-webpack15003
✅ nitro126027
✅ nuxt126027
✅ sveltekit14508
✅ vite126027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15300
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128025
✅ e2e-local-dev-tanstack-start-128025
✅ e2e-local-postgres-nest-stable128025
✅ e2e-local-postgres-tanstack-start-128025
✅ e2e-local-prod-nest-stable128025
✅ e2e-local-prod-tanstack-start-128025
✅ e2e-vercel-prod-tanstack-start126027
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
⏳ Benchmarks are running for 21397aa... (run logs)
Results below are from a previous run.
commit 81a3155 · Fri, 17 Jul 2026 02:04:18 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream843 (-22%)834 🔴1036 🔴1887 🔴30
TTFShook + stream776 (-46%)801 🔴906 🔴1044 🔴30
STSO1020 steps (1-20)232 (-23%)239 🔴318 🔴482 🔴19
STSO1020 steps (101-120)355 (-18%)392 🔴462 🔴476 🔴19
STSO1020 steps (1001-1020)504 (-41%)515 🔴591 🔴808 🔴19
WOstream843 (-22%)8341036188730
WOhook + stream776 (-46%)801906104430
SLstream1718 (-63%)1788 🔴1877 🔴5819 🔴30
SLhook + stream1615 (-67%)1670 🔴1722 🔴1874 🔴30
📜 Previous results (3)
0ed0294
Fri, 17 Jul 2026 01:44:11 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream842 (-23%)744 🔴798 🔴4537 🔴30
TTFShook + stream773 (-46%)833 🔴850 🔴1904 🔴30
STSO1020 steps (1-20)259 (-14%)259 🔴414 🔴591 🔴19
STSO1020 steps (101-120)438 (+1.2%)496 🔴614 🔴642 🔴19
STSO1020 steps (1001-1020)633 (-26%)550 🔴577 🔴2949 🔴19
WOstream842 (-23%)744798453730
WOhook + stream773 (-46%)833850190430
SLstream1588 (-66%)1625 🔴1743 🔴2798 🔴30
SLhook + stream1589 (-67%)1560 🔴1686 🔴5515 🔴30
b0ded7f
Fri, 17 Jul 2026 00:47:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1499 (+38%)1890 🔴1925 🔴2052 🔴30
TTFShook + stream1468 (+2.7%)1988 🔴2083 🔴2225 🔴30
STSO1020 steps (1-20)301 (±0%)326 🔴351 🔴510 🔴19
STSO1020 steps (101-120)441 (+1.8%)496 🔴549 🔴590 🔴19
STSO1020 steps (1001-1020)887 (+3.5%)926 🔴1030 🔴1052 🔴19
WOstream1499 (+38%)18901925205230
WOhook + stream1468 (+2.7%)19882083222530
SLstream4955 (+5.8%)5589 🔴5679 🔴5856 🔴30
SLhook + stream5072 (+4.5%)5641 🔴5750 🔴5857 🔴30
5f7515a
Fri, 17 Jul 2026 00:18:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1191 (+9.6%)1662 🔴1710 🔴1775 🔴30
TTFShook + stream1630 (+14%)1878 🔴1970 🔴2928 🔴30
STSO1020 steps (1-20)283 (-6.1%)331 🔴398 🔴418 🔴19
STSO1020 steps (101-120)433 (±0%)473 🔴564 🔴605 🔴19
STSO1020 steps (1001-1020)943 (+10%)1007 🔴1105 🔴1318 🔴19
WOstream1191 (+9.6%)16621710177530
WOhook + stream1630 (+14%)18781970292830
SLstream4605 (-1.6%)5569 🔴5617 🔴5920 🔴30
SLhook + stream4779 (-1.5%)5063 🔴5814 🔴6828 🔴30
Avg deltas compare against the most recent benchmark run on main at the time of this run.
Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader)
Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges
🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120
TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request.
@vercel
vercelBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-nextjs-workflow-webpackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-astro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-express-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-fastify-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-hono-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nitro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nuxt-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-sveltekit-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-tanstack-start-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-vite-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-swc-playgroundReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-tarballsReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-webReadyReadyPreview, CommentJul 17, 2026 2:12am
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedJul 17, 2026 2:12am
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
July 17, 2026 01:31
Inactive

@NathanColosimo
NathanColosimo
commented
Jul 17, 2026

Copy link


Copy Markdown

ContributorAuthor
The benchmark triggered by the final preview-config restore commit was intentionally canceled because it would no longer run with the debug-off/non-turbo experimental controls. The latest valid diagnostic result is commit 81a3155, run 29548488910; the PR body now contains its full trace breakdown. The final branch has restored the normal preview config.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@NathanColosimo

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Instrument workflow replay scaling - #2969
Closed
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Closed
Instrument workflow replay scaling#2969
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • split workflow.run into VM context, bundle evaluation, argument hydration, execution, serialization, and drain spans
  • aggregate replay event-consumer, subscription-wait, step hydration, and cache diagnostics without per-step spans
  • split step-result hydration into decrypt, decompress, compression telemetry, and deserialize timings
  • add an isolated 20/120/1020-step replay benchmark and route PR benchmark CI through the protected workflow-server preview
Production finding
The production VM itself is not intrinsically hundreds of milliseconds slower than local replay. The original comparison mixed a tiny synthetic local bundle with a 1.322 MB production bundle, encrypted object results, OpenTelemetry, and DEBUG=workflow:*.
Disabling debug logging on the same preview reduced late workflow.run from 604.6 ms to 247.6 ms. The synchronous event-consumer scan fell from 165.3 ms to 11.9 ms because each enabled per-event debug call emitted console output and an active-span event.
The final instrumented, debug-off, non-turbo run was CI 29548488910, workflow run wrun_41KXPWKTB70GJFGJB4Q00XBH2W.
Mean (ms)K1-K19K101-K119K1001-K1019
STSO232.0355.3503.5
prior step_completed durable write70.978.571.1
event refresh0.04102.8106.4
workflow.run72.990.8242.0
next step_started durable write83.079.279.3
The refresh cliff is the event pagination issue being handled separately in workflow-server PR #632. It is not the continuous replay slope.
Replay breakdown
Mean (ms)K1-K19K101-K119K1001-K1019
VM context1.000.961.08
evaluate 1.322 MB bundle49.1744.9851.45
workflow execute/replay16.8242.48187.21
synchronous event scan0.221.4411.54
step-result hydration14.3136.31144.70
AES-GCM decrypt13.5730.71101.85
decompress0.070.633.68
compression telemetry0.231.5514.51
deserialize0.352.6517.73
events / callback invocations32 / 75332 / 7753032 / 7075
The subscription-wait metric is 18.8 / 39.3 / 160.8 ms, but it overlaps hydration and VM continuation and must not be added to the rows above.
Every benchmark result is an object, so all 10 / 110 / 1010 historical step values are non-memoizable by the primitive-only hydration cache. Each replay serially performs one WebCrypto AES-GCM decrypt and one fresh deserialize per historical result. Across 1020 serial steps that is approximately sum(1..1020) = 520,710 decrypt/parse operations in one invocation.
The exact local reproduction used the generated 1.322 MB bundle, encrypted object results, Node 22 Linux, and a 1-vCPU/2-GB container. At K1020 it measured 183.7 ms total workflow.run, including 50.6 ms bundle evaluation, 124.5 ms execution, 86.0 ms hydration, and 9.1 ms synchronous event scan. The preview measured 242.0 / 51.5 / 187.2 / 144.7 / 11.5 ms respectively. The remaining local-cloud gap is therefore hydration/WebCrypto throughput and scheduling, not node:vm context or bundle evaluation.
Evidence
  • original debug-on benchmark: CI 29544762077, traces d6ec13404e2537410dac512a521da423 and fec6da2c5114eb50e1eb9a9c31a23902
  • debug-off A/B benchmark: CI 29547654943
  • hydration split benchmark: CI 29548488910, traces 0d3358d44cc286fec90623114cce3d92 and 80879ec66b238c5727fb3f981d164ea3
Verification
  • targeted replay telemetry and serialization tests: 4 passed
  • pnpm --filter @workflow/core typecheck
  • pnpm --filter @workflow/core build
@changeset-bot
changeset-botBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 21397aa
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145302301683
✅ 💻 Local Development161702191836
✅ 📦 Local Production161702191836
✅ 🐘 Local Postgres161702191836
✅ 🪟 Windows15300153
✅ 📋 Other89401771071
✅ vercel-multi-region270027
Total7378010648442
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro126027
✅ example126027
✅ express126027
✅ fastify126027
✅ hono126027
✅ nextjs-turbopack15003
✅ nextjs-webpack15003
✅ nitro126027
✅ nuxt126027
✅ sveltekit14508
✅ vite126027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15300
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128025
✅ e2e-local-dev-tanstack-start-128025
✅ e2e-local-postgres-nest-stable128025
✅ e2e-local-postgres-tanstack-start-128025
✅ e2e-local-prod-nest-stable128025
✅ e2e-local-prod-tanstack-start-128025
✅ e2e-vercel-prod-tanstack-start126027
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
⏳ Benchmarks are running for 21397aa... (run logs)
Results below are from a previous run.
commit 81a3155 · Fri, 17 Jul 2026 02:04:18 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream843 (-22%)834 🔴1036 🔴1887 🔴30
TTFShook + stream776 (-46%)801 🔴906 🔴1044 🔴30
STSO1020 steps (1-20)232 (-23%)239 🔴318 🔴482 🔴19
STSO1020 steps (101-120)355 (-18%)392 🔴462 🔴476 🔴19
STSO1020 steps (1001-1020)504 (-41%)515 🔴591 🔴808 🔴19
WOstream843 (-22%)8341036188730
WOhook + stream776 (-46%)801906104430
SLstream1718 (-63%)1788 🔴1877 🔴5819 🔴30
SLhook + stream1615 (-67%)1670 🔴1722 🔴1874 🔴30
📜 Previous results (3)
0ed0294
Fri, 17 Jul 2026 01:44:11 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream842 (-23%)744 🔴798 🔴4537 🔴30
TTFShook + stream773 (-46%)833 🔴850 🔴1904 🔴30
STSO1020 steps (1-20)259 (-14%)259 🔴414 🔴591 🔴19
STSO1020 steps (101-120)438 (+1.2%)496 🔴614 🔴642 🔴19
STSO1020 steps (1001-1020)633 (-26%)550 🔴577 🔴2949 🔴19
WOstream842 (-23%)744798453730
WOhook + stream773 (-46%)833850190430
SLstream1588 (-66%)1625 🔴1743 🔴2798 🔴30
SLhook + stream1589 (-67%)1560 🔴1686 🔴5515 🔴30
b0ded7f
Fri, 17 Jul 2026 00:47:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1499 (+38%)1890 🔴1925 🔴2052 🔴30
TTFShook + stream1468 (+2.7%)1988 🔴2083 🔴2225 🔴30
STSO1020 steps (1-20)301 (±0%)326 🔴351 🔴510 🔴19
STSO1020 steps (101-120)441 (+1.8%)496 🔴549 🔴590 🔴19
STSO1020 steps (1001-1020)887 (+3.5%)926 🔴1030 🔴1052 🔴19
WOstream1499 (+38%)18901925205230
WOhook + stream1468 (+2.7%)19882083222530
SLstream4955 (+5.8%)5589 🔴5679 🔴5856 🔴30
SLhook + stream5072 (+4.5%)5641 🔴5750 🔴5857 🔴30
5f7515a
Fri, 17 Jul 2026 00:18:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1191 (+9.6%)1662 🔴1710 🔴1775 🔴30
TTFShook + stream1630 (+14%)1878 🔴1970 🔴2928 🔴30
STSO1020 steps (1-20)283 (-6.1%)331 🔴398 🔴418 🔴19
STSO1020 steps (101-120)433 (±0%)473 🔴564 🔴605 🔴19
STSO1020 steps (1001-1020)943 (+10%)1007 🔴1105 🔴1318 🔴19
WOstream1191 (+9.6%)16621710177530
WOhook + stream1630 (+14%)18781970292830
SLstream4605 (-1.6%)5569 🔴5617 🔴5920 🔴30
SLhook + stream4779 (-1.5%)5063 🔴5814 🔴6828 🔴30
Avg deltas compare against the most recent benchmark run on main at the time of this run.
Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader)
Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges
🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120
TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request.
@vercel
vercelBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-nextjs-workflow-webpackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-astro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-express-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-fastify-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-hono-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nitro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nuxt-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-sveltekit-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-tanstack-start-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-vite-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-swc-playgroundReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-tarballsReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-webReadyReadyPreview, CommentJul 17, 2026 2:12am
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedJul 17, 2026 2:12am
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
July 17, 2026 01:31
Inactive

@NathanColosimo
NathanColosimo
commented
Jul 17, 2026

Copy link


Copy Markdown

ContributorAuthor
The benchmark triggered by the final preview-config restore commit was intentionally canceled because it would no longer run with the debug-off/non-turbo experimental controls. The latest valid diagnostic result is commit 81a3155, run 29548488910; the PR body now contains its full trace breakdown. The final branch has restored the normal preview config.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@NathanColosimo

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Instrument workflow replay scaling - #2969
Closed
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Closed
Instrument workflow replay scaling#2969
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • split workflow.run into VM context, bundle evaluation, argument hydration, execution, serialization, and drain spans
  • aggregate replay event-consumer, subscription-wait, step hydration, and cache diagnostics without per-step spans
  • split step-result hydration into decrypt, decompress, compression telemetry, and deserialize timings
  • add an isolated 20/120/1020-step replay benchmark and route PR benchmark CI through the protected workflow-server preview
Production finding
The production VM itself is not intrinsically hundreds of milliseconds slower than local replay. The original comparison mixed a tiny synthetic local bundle with a 1.322 MB production bundle, encrypted object results, OpenTelemetry, and DEBUG=workflow:*.
Disabling debug logging on the same preview reduced late workflow.run from 604.6 ms to 247.6 ms. The synchronous event-consumer scan fell from 165.3 ms to 11.9 ms because each enabled per-event debug call emitted console output and an active-span event.
The final instrumented, debug-off, non-turbo run was CI 29548488910, workflow run wrun_41KXPWKTB70GJFGJB4Q00XBH2W.
Mean (ms)K1-K19K101-K119K1001-K1019
STSO232.0355.3503.5
prior step_completed durable write70.978.571.1
event refresh0.04102.8106.4
workflow.run72.990.8242.0
next step_started durable write83.079.279.3
The refresh cliff is the event pagination issue being handled separately in workflow-server PR #632. It is not the continuous replay slope.
Replay breakdown
Mean (ms)K1-K19K101-K119K1001-K1019
VM context1.000.961.08
evaluate 1.322 MB bundle49.1744.9851.45
workflow execute/replay16.8242.48187.21
synchronous event scan0.221.4411.54
step-result hydration14.3136.31144.70
AES-GCM decrypt13.5730.71101.85
decompress0.070.633.68
compression telemetry0.231.5514.51
deserialize0.352.6517.73
events / callback invocations32 / 75332 / 7753032 / 7075
The subscription-wait metric is 18.8 / 39.3 / 160.8 ms, but it overlaps hydration and VM continuation and must not be added to the rows above.
Every benchmark result is an object, so all 10 / 110 / 1010 historical step values are non-memoizable by the primitive-only hydration cache. Each replay serially performs one WebCrypto AES-GCM decrypt and one fresh deserialize per historical result. Across 1020 serial steps that is approximately sum(1..1020) = 520,710 decrypt/parse operations in one invocation.
The exact local reproduction used the generated 1.322 MB bundle, encrypted object results, Node 22 Linux, and a 1-vCPU/2-GB container. At K1020 it measured 183.7 ms total workflow.run, including 50.6 ms bundle evaluation, 124.5 ms execution, 86.0 ms hydration, and 9.1 ms synchronous event scan. The preview measured 242.0 / 51.5 / 187.2 / 144.7 / 11.5 ms respectively. The remaining local-cloud gap is therefore hydration/WebCrypto throughput and scheduling, not node:vm context or bundle evaluation.
Evidence
  • original debug-on benchmark: CI 29544762077, traces d6ec13404e2537410dac512a521da423 and fec6da2c5114eb50e1eb9a9c31a23902
  • debug-off A/B benchmark: CI 29547654943
  • hydration split benchmark: CI 29548488910, traces 0d3358d44cc286fec90623114cce3d92 and 80879ec66b238c5727fb3f981d164ea3
Verification
  • targeted replay telemetry and serialization tests: 4 passed
  • pnpm --filter @workflow/core typecheck
  • pnpm --filter @workflow/core build
@changeset-bot
changeset-botBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 21397aa
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145302301683
✅ 💻 Local Development161702191836
✅ 📦 Local Production161702191836
✅ 🐘 Local Postgres161702191836
✅ 🪟 Windows15300153
✅ 📋 Other89401771071
✅ vercel-multi-region270027
Total7378010648442
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro126027
✅ example126027
✅ express126027
✅ fastify126027
✅ hono126027
✅ nextjs-turbopack15003
✅ nextjs-webpack15003
✅ nitro126027
✅ nuxt126027
✅ sveltekit14508
✅ vite126027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15300
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128025
✅ e2e-local-dev-tanstack-start-128025
✅ e2e-local-postgres-nest-stable128025
✅ e2e-local-postgres-tanstack-start-128025
✅ e2e-local-prod-nest-stable128025
✅ e2e-local-prod-tanstack-start-128025
✅ e2e-vercel-prod-tanstack-start126027
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
⏳ Benchmarks are running for 21397aa... (run logs)
Results below are from a previous run.
commit 81a3155 · Fri, 17 Jul 2026 02:04:18 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream843 (-22%)834 🔴1036 🔴1887 🔴30
TTFShook + stream776 (-46%)801 🔴906 🔴1044 🔴30
STSO1020 steps (1-20)232 (-23%)239 🔴318 🔴482 🔴19
STSO1020 steps (101-120)355 (-18%)392 🔴462 🔴476 🔴19
STSO1020 steps (1001-1020)504 (-41%)515 🔴591 🔴808 🔴19
WOstream843 (-22%)8341036188730
WOhook + stream776 (-46%)801906104430
SLstream1718 (-63%)1788 🔴1877 🔴5819 🔴30
SLhook + stream1615 (-67%)1670 🔴1722 🔴1874 🔴30
📜 Previous results (3)
0ed0294
Fri, 17 Jul 2026 01:44:11 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream842 (-23%)744 🔴798 🔴4537 🔴30
TTFShook + stream773 (-46%)833 🔴850 🔴1904 🔴30
STSO1020 steps (1-20)259 (-14%)259 🔴414 🔴591 🔴19
STSO1020 steps (101-120)438 (+1.2%)496 🔴614 🔴642 🔴19
STSO1020 steps (1001-1020)633 (-26%)550 🔴577 🔴2949 🔴19
WOstream842 (-23%)744798453730
WOhook + stream773 (-46%)833850190430
SLstream1588 (-66%)1625 🔴1743 🔴2798 🔴30
SLhook + stream1589 (-67%)1560 🔴1686 🔴5515 🔴30
b0ded7f
Fri, 17 Jul 2026 00:47:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1499 (+38%)1890 🔴1925 🔴2052 🔴30
TTFShook + stream1468 (+2.7%)1988 🔴2083 🔴2225 🔴30
STSO1020 steps (1-20)301 (±0%)326 🔴351 🔴510 🔴19
STSO1020 steps (101-120)441 (+1.8%)496 🔴549 🔴590 🔴19
STSO1020 steps (1001-1020)887 (+3.5%)926 🔴1030 🔴1052 🔴19
WOstream1499 (+38%)18901925205230
WOhook + stream1468 (+2.7%)19882083222530
SLstream4955 (+5.8%)5589 🔴5679 🔴5856 🔴30
SLhook + stream5072 (+4.5%)5641 🔴5750 🔴5857 🔴30
5f7515a
Fri, 17 Jul 2026 00:18:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1191 (+9.6%)1662 🔴1710 🔴1775 🔴30
TTFShook + stream1630 (+14%)1878 🔴1970 🔴2928 🔴30
STSO1020 steps (1-20)283 (-6.1%)331 🔴398 🔴418 🔴19
STSO1020 steps (101-120)433 (±0%)473 🔴564 🔴605 🔴19
STSO1020 steps (1001-1020)943 (+10%)1007 🔴1105 🔴1318 🔴19
WOstream1191 (+9.6%)16621710177530
WOhook + stream1630 (+14%)18781970292830
SLstream4605 (-1.6%)5569 🔴5617 🔴5920 🔴30
SLhook + stream4779 (-1.5%)5063 🔴5814 🔴6828 🔴30
Avg deltas compare against the most recent benchmark run on main at the time of this run.
Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader)
Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges
🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120
TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request.
@vercel
vercelBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-nextjs-workflow-webpackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-astro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-express-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-fastify-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-hono-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nitro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nuxt-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-sveltekit-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-tanstack-start-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-vite-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-swc-playgroundReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-tarballsReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-webReadyReadyPreview, CommentJul 17, 2026 2:12am
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedJul 17, 2026 2:12am
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
July 17, 2026 01:31
Inactive

@NathanColosimo
NathanColosimo
commented
Jul 17, 2026

Copy link


Copy Markdown

ContributorAuthor
The benchmark triggered by the final preview-config restore commit was intentionally canceled because it would no longer run with the debug-off/non-turbo experimental controls. The latest valid diagnostic result is commit 81a3155, run 29548488910; the PR body now contains its full trace breakdown. The final branch has restored the normal preview config.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@NathanColosimo

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Instrument workflow replay scaling - #2969
Closed
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Closed
Instrument workflow replay scaling#2969
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • split workflow.run into VM context, bundle evaluation, argument hydration, execution, serialization, and drain spans
  • aggregate replay event-consumer, subscription-wait, step hydration, and cache diagnostics without per-step spans
  • split step-result hydration into decrypt, decompress, compression telemetry, and deserialize timings
  • add an isolated 20/120/1020-step replay benchmark and route PR benchmark CI through the protected workflow-server preview
Production finding
The production VM itself is not intrinsically hundreds of milliseconds slower than local replay. The original comparison mixed a tiny synthetic local bundle with a 1.322 MB production bundle, encrypted object results, OpenTelemetry, and DEBUG=workflow:*.
Disabling debug logging on the same preview reduced late workflow.run from 604.6 ms to 247.6 ms. The synchronous event-consumer scan fell from 165.3 ms to 11.9 ms because each enabled per-event debug call emitted console output and an active-span event.
The final instrumented, debug-off, non-turbo run was CI 29548488910, workflow run wrun_41KXPWKTB70GJFGJB4Q00XBH2W.
Mean (ms)K1-K19K101-K119K1001-K1019
STSO232.0355.3503.5
prior step_completed durable write70.978.571.1
event refresh0.04102.8106.4
workflow.run72.990.8242.0
next step_started durable write83.079.279.3
The refresh cliff is the event pagination issue being handled separately in workflow-server PR #632. It is not the continuous replay slope.
Replay breakdown
Mean (ms)K1-K19K101-K119K1001-K1019
VM context1.000.961.08
evaluate 1.322 MB bundle49.1744.9851.45
workflow execute/replay16.8242.48187.21
synchronous event scan0.221.4411.54
step-result hydration14.3136.31144.70
AES-GCM decrypt13.5730.71101.85
decompress0.070.633.68
compression telemetry0.231.5514.51
deserialize0.352.6517.73
events / callback invocations32 / 75332 / 7753032 / 7075
The subscription-wait metric is 18.8 / 39.3 / 160.8 ms, but it overlaps hydration and VM continuation and must not be added to the rows above.
Every benchmark result is an object, so all 10 / 110 / 1010 historical step values are non-memoizable by the primitive-only hydration cache. Each replay serially performs one WebCrypto AES-GCM decrypt and one fresh deserialize per historical result. Across 1020 serial steps that is approximately sum(1..1020) = 520,710 decrypt/parse operations in one invocation.
The exact local reproduction used the generated 1.322 MB bundle, encrypted object results, Node 22 Linux, and a 1-vCPU/2-GB container. At K1020 it measured 183.7 ms total workflow.run, including 50.6 ms bundle evaluation, 124.5 ms execution, 86.0 ms hydration, and 9.1 ms synchronous event scan. The preview measured 242.0 / 51.5 / 187.2 / 144.7 / 11.5 ms respectively. The remaining local-cloud gap is therefore hydration/WebCrypto throughput and scheduling, not node:vm context or bundle evaluation.
Evidence
  • original debug-on benchmark: CI 29544762077, traces d6ec13404e2537410dac512a521da423 and fec6da2c5114eb50e1eb9a9c31a23902
  • debug-off A/B benchmark: CI 29547654943
  • hydration split benchmark: CI 29548488910, traces 0d3358d44cc286fec90623114cce3d92 and 80879ec66b238c5727fb3f981d164ea3
Verification
  • targeted replay telemetry and serialization tests: 4 passed
  • pnpm --filter @workflow/core typecheck
  • pnpm --filter @workflow/core build
@changeset-bot
changeset-botBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 21397aa
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145302301683
✅ 💻 Local Development161702191836
✅ 📦 Local Production161702191836
✅ 🐘 Local Postgres161702191836
✅ 🪟 Windows15300153
✅ 📋 Other89401771071
✅ vercel-multi-region270027
Total7378010648442
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro126027
✅ example126027
✅ express126027
✅ fastify126027
✅ hono126027
✅ nextjs-turbopack15003
✅ nextjs-webpack15003
✅ nitro126027
✅ nuxt126027
✅ sveltekit14508
✅ vite126027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15300
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128025
✅ e2e-local-dev-tanstack-start-128025
✅ e2e-local-postgres-nest-stable128025
✅ e2e-local-postgres-tanstack-start-128025
✅ e2e-local-prod-nest-stable128025
✅ e2e-local-prod-tanstack-start-128025
✅ e2e-vercel-prod-tanstack-start126027
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
⏳ Benchmarks are running for 21397aa... (run logs)
Results below are from a previous run.
commit 81a3155 · Fri, 17 Jul 2026 02:04:18 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream843 (-22%)834 🔴1036 🔴1887 🔴30
TTFShook + stream776 (-46%)801 🔴906 🔴1044 🔴30
STSO1020 steps (1-20)232 (-23%)239 🔴318 🔴482 🔴19
STSO1020 steps (101-120)355 (-18%)392 🔴462 🔴476 🔴19
STSO1020 steps (1001-1020)504 (-41%)515 🔴591 🔴808 🔴19
WOstream843 (-22%)8341036188730
WOhook + stream776 (-46%)801906104430
SLstream1718 (-63%)1788 🔴1877 🔴5819 🔴30
SLhook + stream1615 (-67%)1670 🔴1722 🔴1874 🔴30
📜 Previous results (3)
0ed0294
Fri, 17 Jul 2026 01:44:11 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream842 (-23%)744 🔴798 🔴4537 🔴30
TTFShook + stream773 (-46%)833 🔴850 🔴1904 🔴30
STSO1020 steps (1-20)259 (-14%)259 🔴414 🔴591 🔴19
STSO1020 steps (101-120)438 (+1.2%)496 🔴614 🔴642 🔴19
STSO1020 steps (1001-1020)633 (-26%)550 🔴577 🔴2949 🔴19
WOstream842 (-23%)744798453730
WOhook + stream773 (-46%)833850190430
SLstream1588 (-66%)1625 🔴1743 🔴2798 🔴30
SLhook + stream1589 (-67%)1560 🔴1686 🔴5515 🔴30
b0ded7f
Fri, 17 Jul 2026 00:47:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1499 (+38%)1890 🔴1925 🔴2052 🔴30
TTFShook + stream1468 (+2.7%)1988 🔴2083 🔴2225 🔴30
STSO1020 steps (1-20)301 (±0%)326 🔴351 🔴510 🔴19
STSO1020 steps (101-120)441 (+1.8%)496 🔴549 🔴590 🔴19
STSO1020 steps (1001-1020)887 (+3.5%)926 🔴1030 🔴1052 🔴19
WOstream1499 (+38%)18901925205230
WOhook + stream1468 (+2.7%)19882083222530
SLstream4955 (+5.8%)5589 🔴5679 🔴5856 🔴30
SLhook + stream5072 (+4.5%)5641 🔴5750 🔴5857 🔴30
5f7515a
Fri, 17 Jul 2026 00:18:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1191 (+9.6%)1662 🔴1710 🔴1775 🔴30
TTFShook + stream1630 (+14%)1878 🔴1970 🔴2928 🔴30
STSO1020 steps (1-20)283 (-6.1%)331 🔴398 🔴418 🔴19
STSO1020 steps (101-120)433 (±0%)473 🔴564 🔴605 🔴19
STSO1020 steps (1001-1020)943 (+10%)1007 🔴1105 🔴1318 🔴19
WOstream1191 (+9.6%)16621710177530
WOhook + stream1630 (+14%)18781970292830
SLstream4605 (-1.6%)5569 🔴5617 🔴5920 🔴30
SLhook + stream4779 (-1.5%)5063 🔴5814 🔴6828 🔴30
Avg deltas compare against the most recent benchmark run on main at the time of this run.
Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader)
Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges
🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120
TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request.
@vercel
vercelBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-nextjs-workflow-webpackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-astro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-express-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-fastify-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-hono-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nitro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nuxt-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-sveltekit-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-tanstack-start-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-vite-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-swc-playgroundReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-tarballsReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-webReadyReadyPreview, CommentJul 17, 2026 2:12am
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedJul 17, 2026 2:12am
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
July 17, 2026 01:31
Inactive

@NathanColosimo
NathanColosimo
commented
Jul 17, 2026

Copy link


Copy Markdown

ContributorAuthor
The benchmark triggered by the final preview-config restore commit was intentionally canceled because it would no longer run with the debug-off/non-turbo experimental controls. The latest valid diagnostic result is commit 81a3155, run 29548488910; the PR body now contains its full trace breakdown. The final branch has restored the normal preview config.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@NathanColosimo

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Instrument workflow replay scaling - #2969
Closed
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Closed
Instrument workflow replay scaling#2969
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • split workflow.run into VM context, bundle evaluation, argument hydration, execution, serialization, and drain spans
  • aggregate replay event-consumer, subscription-wait, step hydration, and cache diagnostics without per-step spans
  • split step-result hydration into decrypt, decompress, compression telemetry, and deserialize timings
  • add an isolated 20/120/1020-step replay benchmark and route PR benchmark CI through the protected workflow-server preview
Production finding
The production VM itself is not intrinsically hundreds of milliseconds slower than local replay. The original comparison mixed a tiny synthetic local bundle with a 1.322 MB production bundle, encrypted object results, OpenTelemetry, and DEBUG=workflow:*.
Disabling debug logging on the same preview reduced late workflow.run from 604.6 ms to 247.6 ms. The synchronous event-consumer scan fell from 165.3 ms to 11.9 ms because each enabled per-event debug call emitted console output and an active-span event.
The final instrumented, debug-off, non-turbo run was CI 29548488910, workflow run wrun_41KXPWKTB70GJFGJB4Q00XBH2W.
Mean (ms)K1-K19K101-K119K1001-K1019
STSO232.0355.3503.5
prior step_completed durable write70.978.571.1
event refresh0.04102.8106.4
workflow.run72.990.8242.0
next step_started durable write83.079.279.3
The refresh cliff is the event pagination issue being handled separately in workflow-server PR #632. It is not the continuous replay slope.
Replay breakdown
Mean (ms)K1-K19K101-K119K1001-K1019
VM context1.000.961.08
evaluate 1.322 MB bundle49.1744.9851.45
workflow execute/replay16.8242.48187.21
synchronous event scan0.221.4411.54
step-result hydration14.3136.31144.70
AES-GCM decrypt13.5730.71101.85
decompress0.070.633.68
compression telemetry0.231.5514.51
deserialize0.352.6517.73
events / callback invocations32 / 75332 / 7753032 / 7075
The subscription-wait metric is 18.8 / 39.3 / 160.8 ms, but it overlaps hydration and VM continuation and must not be added to the rows above.
Every benchmark result is an object, so all 10 / 110 / 1010 historical step values are non-memoizable by the primitive-only hydration cache. Each replay serially performs one WebCrypto AES-GCM decrypt and one fresh deserialize per historical result. Across 1020 serial steps that is approximately sum(1..1020) = 520,710 decrypt/parse operations in one invocation.
The exact local reproduction used the generated 1.322 MB bundle, encrypted object results, Node 22 Linux, and a 1-vCPU/2-GB container. At K1020 it measured 183.7 ms total workflow.run, including 50.6 ms bundle evaluation, 124.5 ms execution, 86.0 ms hydration, and 9.1 ms synchronous event scan. The preview measured 242.0 / 51.5 / 187.2 / 144.7 / 11.5 ms respectively. The remaining local-cloud gap is therefore hydration/WebCrypto throughput and scheduling, not node:vm context or bundle evaluation.
Evidence
  • original debug-on benchmark: CI 29544762077, traces d6ec13404e2537410dac512a521da423 and fec6da2c5114eb50e1eb9a9c31a23902
  • debug-off A/B benchmark: CI 29547654943
  • hydration split benchmark: CI 29548488910, traces 0d3358d44cc286fec90623114cce3d92 and 80879ec66b238c5727fb3f981d164ea3
Verification
  • targeted replay telemetry and serialization tests: 4 passed
  • pnpm --filter @workflow/core typecheck
  • pnpm --filter @workflow/core build
@changeset-bot
changeset-botBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 21397aa
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145302301683
✅ 💻 Local Development161702191836
✅ 📦 Local Production161702191836
✅ 🐘 Local Postgres161702191836
✅ 🪟 Windows15300153
✅ 📋 Other89401771071
✅ vercel-multi-region270027
Total7378010648442
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro126027
✅ example126027
✅ express126027
✅ fastify126027
✅ hono126027
✅ nextjs-turbopack15003
✅ nextjs-webpack15003
✅ nitro126027
✅ nuxt126027
✅ sveltekit14508
✅ vite126027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15300
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128025
✅ e2e-local-dev-tanstack-start-128025
✅ e2e-local-postgres-nest-stable128025
✅ e2e-local-postgres-tanstack-start-128025
✅ e2e-local-prod-nest-stable128025
✅ e2e-local-prod-tanstack-start-128025
✅ e2e-vercel-prod-tanstack-start126027
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
⏳ Benchmarks are running for 21397aa... (run logs)
Results below are from a previous run.
commit 81a3155 · Fri, 17 Jul 2026 02:04:18 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream843 (-22%)834 🔴1036 🔴1887 🔴30
TTFShook + stream776 (-46%)801 🔴906 🔴1044 🔴30
STSO1020 steps (1-20)232 (-23%)239 🔴318 🔴482 🔴19
STSO1020 steps (101-120)355 (-18%)392 🔴462 🔴476 🔴19
STSO1020 steps (1001-1020)504 (-41%)515 🔴591 🔴808 🔴19
WOstream843 (-22%)8341036188730
WOhook + stream776 (-46%)801906104430
SLstream1718 (-63%)1788 🔴1877 🔴5819 🔴30
SLhook + stream1615 (-67%)1670 🔴1722 🔴1874 🔴30
📜 Previous results (3)
0ed0294
Fri, 17 Jul 2026 01:44:11 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream842 (-23%)744 🔴798 🔴4537 🔴30
TTFShook + stream773 (-46%)833 🔴850 🔴1904 🔴30
STSO1020 steps (1-20)259 (-14%)259 🔴414 🔴591 🔴19
STSO1020 steps (101-120)438 (+1.2%)496 🔴614 🔴642 🔴19
STSO1020 steps (1001-1020)633 (-26%)550 🔴577 🔴2949 🔴19
WOstream842 (-23%)744798453730
WOhook + stream773 (-46%)833850190430
SLstream1588 (-66%)1625 🔴1743 🔴2798 🔴30
SLhook + stream1589 (-67%)1560 🔴1686 🔴5515 🔴30
b0ded7f
Fri, 17 Jul 2026 00:47:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1499 (+38%)1890 🔴1925 🔴2052 🔴30
TTFShook + stream1468 (+2.7%)1988 🔴2083 🔴2225 🔴30
STSO1020 steps (1-20)301 (±0%)326 🔴351 🔴510 🔴19
STSO1020 steps (101-120)441 (+1.8%)496 🔴549 🔴590 🔴19
STSO1020 steps (1001-1020)887 (+3.5%)926 🔴1030 🔴1052 🔴19
WOstream1499 (+38%)18901925205230
WOhook + stream1468 (+2.7%)19882083222530
SLstream4955 (+5.8%)5589 🔴5679 🔴5856 🔴30
SLhook + stream5072 (+4.5%)5641 🔴5750 🔴5857 🔴30
5f7515a
Fri, 17 Jul 2026 00:18:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1191 (+9.6%)1662 🔴1710 🔴1775 🔴30
TTFShook + stream1630 (+14%)1878 🔴1970 🔴2928 🔴30
STSO1020 steps (1-20)283 (-6.1%)331 🔴398 🔴418 🔴19
STSO1020 steps (101-120)433 (±0%)473 🔴564 🔴605 🔴19
STSO1020 steps (1001-1020)943 (+10%)1007 🔴1105 🔴1318 🔴19
WOstream1191 (+9.6%)16621710177530
WOhook + stream1630 (+14%)18781970292830
SLstream4605 (-1.6%)5569 🔴5617 🔴5920 🔴30
SLhook + stream4779 (-1.5%)5063 🔴5814 🔴6828 🔴30
Avg deltas compare against the most recent benchmark run on main at the time of this run.
Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader)
Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges
🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120
TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request.
@vercel
vercelBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-nextjs-workflow-webpackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-astro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-express-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-fastify-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-hono-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nitro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nuxt-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-sveltekit-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-tanstack-start-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-vite-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-swc-playgroundReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-tarballsReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-webReadyReadyPreview, CommentJul 17, 2026 2:12am
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedJul 17, 2026 2:12am
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
July 17, 2026 01:31
Inactive

@NathanColosimo
NathanColosimo
commented
Jul 17, 2026

Copy link


Copy Markdown

ContributorAuthor
The benchmark triggered by the final preview-config restore commit was intentionally canceled because it would no longer run with the debug-off/non-turbo experimental controls. The latest valid diagnostic result is commit 81a3155, run 29548488910; the PR body now contains its full trace breakdown. The final branch has restored the normal preview config.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@NathanColosimo

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Instrument workflow replay scaling - #2969
Closed
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Closed
Instrument workflow replay scaling#2969
NathanColosimo wants to merge 9 commits into
mainfrom 
codex/instrument-workflow-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • split workflow.run into VM context, bundle evaluation, argument hydration, execution, serialization, and drain spans
  • aggregate replay event-consumer, subscription-wait, step hydration, and cache diagnostics without per-step spans
  • split step-result hydration into decrypt, decompress, compression telemetry, and deserialize timings
  • add an isolated 20/120/1020-step replay benchmark and route PR benchmark CI through the protected workflow-server preview
Production finding
The production VM itself is not intrinsically hundreds of milliseconds slower than local replay. The original comparison mixed a tiny synthetic local bundle with a 1.322 MB production bundle, encrypted object results, OpenTelemetry, and DEBUG=workflow:*.
Disabling debug logging on the same preview reduced late workflow.run from 604.6 ms to 247.6 ms. The synchronous event-consumer scan fell from 165.3 ms to 11.9 ms because each enabled per-event debug call emitted console output and an active-span event.
The final instrumented, debug-off, non-turbo run was CI 29548488910, workflow run wrun_41KXPWKTB70GJFGJB4Q00XBH2W.
Mean (ms)K1-K19K101-K119K1001-K1019
STSO232.0355.3503.5
prior step_completed durable write70.978.571.1
event refresh0.04102.8106.4
workflow.run72.990.8242.0
next step_started durable write83.079.279.3
The refresh cliff is the event pagination issue being handled separately in workflow-server PR #632. It is not the continuous replay slope.
Replay breakdown
Mean (ms)K1-K19K101-K119K1001-K1019
VM context1.000.961.08
evaluate 1.322 MB bundle49.1744.9851.45
workflow execute/replay16.8242.48187.21
synchronous event scan0.221.4411.54
step-result hydration14.3136.31144.70
AES-GCM decrypt13.5730.71101.85
decompress0.070.633.68
compression telemetry0.231.5514.51
deserialize0.352.6517.73
events / callback invocations32 / 75332 / 7753032 / 7075
The subscription-wait metric is 18.8 / 39.3 / 160.8 ms, but it overlaps hydration and VM continuation and must not be added to the rows above.
Every benchmark result is an object, so all 10 / 110 / 1010 historical step values are non-memoizable by the primitive-only hydration cache. Each replay serially performs one WebCrypto AES-GCM decrypt and one fresh deserialize per historical result. Across 1020 serial steps that is approximately sum(1..1020) = 520,710 decrypt/parse operations in one invocation.
The exact local reproduction used the generated 1.322 MB bundle, encrypted object results, Node 22 Linux, and a 1-vCPU/2-GB container. At K1020 it measured 183.7 ms total workflow.run, including 50.6 ms bundle evaluation, 124.5 ms execution, 86.0 ms hydration, and 9.1 ms synchronous event scan. The preview measured 242.0 / 51.5 / 187.2 / 144.7 / 11.5 ms respectively. The remaining local-cloud gap is therefore hydration/WebCrypto throughput and scheduling, not node:vm context or bundle evaluation.
Evidence
  • original debug-on benchmark: CI 29544762077, traces d6ec13404e2537410dac512a521da423 and fec6da2c5114eb50e1eb9a9c31a23902
  • debug-off A/B benchmark: CI 29547654943
  • hydration split benchmark: CI 29548488910, traces 0d3358d44cc286fec90623114cce3d92 and 80879ec66b238c5727fb3f981d164ea3
Verification
  • targeted replay telemetry and serialization tests: 4 passed
  • pnpm --filter @workflow/core typecheck
  • pnpm --filter @workflow/core build
@changeset-bot
changeset-botBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 21397aa
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145302301683
✅ 💻 Local Development161702191836
✅ 📦 Local Production161702191836
✅ 🐘 Local Postgres161702191836
✅ 🪟 Windows15300153
✅ 📋 Other89401771071
✅ vercel-multi-region270027
Total7378010648442
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro126027
✅ example126027
✅ express126027
✅ fastify126027
✅ hono126027
✅ nextjs-turbopack15003
✅ nextjs-webpack15003
✅ nitro126027
✅ nuxt126027
✅ sveltekit14508
✅ vite126027
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15300
✅ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128025
✅ e2e-local-dev-tanstack-start-128025
✅ e2e-local-postgres-nest-stable128025
✅ e2e-local-postgres-tanstack-start-128025
✅ e2e-local-prod-nest-stable128025
✅ e2e-local-prod-tanstack-start-128025
✅ e2e-vercel-prod-tanstack-start126027
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
⏳ Benchmarks are running for 21397aa... (run logs)
Results below are from a previous run.
commit 81a3155 · Fri, 17 Jul 2026 02:04:18 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream843 (-22%)834 🔴1036 🔴1887 🔴30
TTFShook + stream776 (-46%)801 🔴906 🔴1044 🔴30
STSO1020 steps (1-20)232 (-23%)239 🔴318 🔴482 🔴19
STSO1020 steps (101-120)355 (-18%)392 🔴462 🔴476 🔴19
STSO1020 steps (1001-1020)504 (-41%)515 🔴591 🔴808 🔴19
WOstream843 (-22%)8341036188730
WOhook + stream776 (-46%)801906104430
SLstream1718 (-63%)1788 🔴1877 🔴5819 🔴30
SLhook + stream1615 (-67%)1670 🔴1722 🔴1874 🔴30
📜 Previous results (3)
0ed0294
Fri, 17 Jul 2026 01:44:11 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream842 (-23%)744 🔴798 🔴4537 🔴30
TTFShook + stream773 (-46%)833 🔴850 🔴1904 🔴30
STSO1020 steps (1-20)259 (-14%)259 🔴414 🔴591 🔴19
STSO1020 steps (101-120)438 (+1.2%)496 🔴614 🔴642 🔴19
STSO1020 steps (1001-1020)633 (-26%)550 🔴577 🔴2949 🔴19
WOstream842 (-23%)744798453730
WOhook + stream773 (-46%)833850190430
SLstream1588 (-66%)1625 🔴1743 🔴2798 🔴30
SLhook + stream1589 (-67%)1560 🔴1686 🔴5515 🔴30
b0ded7f
Fri, 17 Jul 2026 00:47:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1499 (+38%)1890 🔴1925 🔴2052 🔴30
TTFShook + stream1468 (+2.7%)1988 🔴2083 🔴2225 🔴30
STSO1020 steps (1-20)301 (±0%)326 🔴351 🔴510 🔴19
STSO1020 steps (101-120)441 (+1.8%)496 🔴549 🔴590 🔴19
STSO1020 steps (1001-1020)887 (+3.5%)926 🔴1030 🔴1052 🔴19
WOstream1499 (+38%)18901925205230
WOhook + stream1468 (+2.7%)19882083222530
SLstream4955 (+5.8%)5589 🔴5679 🔴5856 🔴30
SLhook + stream5072 (+4.5%)5641 🔴5750 🔴5857 🔴30
5f7515a
Fri, 17 Jul 2026 00:18:52 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioAvg (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstream1191 (+9.6%)1662 🔴1710 🔴1775 🔴30
TTFShook + stream1630 (+14%)1878 🔴1970 🔴2928 🔴30
STSO1020 steps (1-20)283 (-6.1%)331 🔴398 🔴418 🔴19
STSO1020 steps (101-120)433 (±0%)473 🔴564 🔴605 🔴19
STSO1020 steps (1001-1020)943 (+10%)1007 🔴1105 🔴1318 🔴19
WOstream1191 (+9.6%)16621710177530
WOhook + stream1630 (+14%)18781970292830
SLstream4605 (-1.6%)5569 🔴5617 🔴5920 🔴30
SLhook + stream4779 (-1.5%)5063 🔴5814 🔴6828 🔴30
Avg deltas compare against the most recent benchmark run on main at the time of this run.
Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader)
Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges
🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120
TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request.
@vercel
vercelBot
commented
Jul 16, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-nextjs-workflow-webpackReadyReadyPreview, CommentJul 17, 2026 2:12am
example-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-astro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-express-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-fastify-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-hono-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nitro-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-nuxt-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-sveltekit-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-tanstack-start-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workbench-vite-workflowReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-swc-playgroundReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-tarballsReadyReadyPreview, CommentJul 17, 2026 2:12am
workflow-webReadyReadyPreview, CommentJul 17, 2026 2:12am
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedJul 17, 2026 2:12am
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
July 17, 2026 01:31
Inactive

@NathanColosimo
NathanColosimo
commented
Jul 17, 2026

Copy link


Copy Markdown

ContributorAuthor
The benchmark triggered by the final preview-config restore commit was intentionally canceled because it would no longer run with the debug-off/non-turbo experimental controls. The latest valid diagnostic result is commit 81a3155, run 29548488910; the PR body now contains its full trace breakdown. The final branch has restored the normal preview config.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@NathanColosimo