Skip to content

fix(next): converge workflow HMR rebuilds - #3333

Open
NathanColosimo wants to merge 18 commits into
mainfrom
codex/fix-next-hmr-build-race
Open

fix(next): converge workflow HMR rebuilds#3333
NathanColosimo wants to merge 18 commits into
mainfrom
codex/fix-next-hmr-build-race

Conversation

@NathanColosimo

@NathanColosimoNathanColosimo commented Aug 4, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • snapshot known workflow sources before every full development rebuild
  • serialize and debounce dirty paths, with one follow-up full rebuild when a filesystem event overlaps a build
  • read each dirty path once and use its final filesystem state to choose full discovery, a hot rebuild, or a snapshot-only update
  • keep byte-identical relevant notifications as conservative invalidations
  • expose queue-idle diagnostics so HMR tests distinguish isolated edits from edits made during convergence

Testing

  • pnpm --filter @workflow/next build
  • pnpm --filter @workflow/next test -- --run src/watch-rebuild.test.ts
  • focused stable Next.js Turbopack development HMR scenario

Stack

  1. This PR: rebuild convergence and unit coverage
  2. fix(next): watch before initial build #3470: watch before the initial build
  3. test(next): cover HMR rebuild convergence #3471: end-to-end HMR coverage

Stack created with GitHub Stacks CLI • Give Feedback 💬

@NathanColosimo
NathanColosimo requested a review from a team as a code ownerAugust 4, 2026 20:41
@changeset-bot

changeset-botBot commented Aug 4, 2026•
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 8b0bfea

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/nextPatch
@workflow/buildersPatch
workflowPatch
@workflow/astroPatch
@workflow/cliPatch
@workflow/nestPatch
@workflow/nitroPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/vitestPatch
@workflow/world-testingPatch
@workflow/nuxtPatch
@workflow/corePatch
@workflow/web-sharedPatch
@workflow/webPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Aug 4, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 28, 2026 9:31pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 28, 2026 9:31pm
example-workflowReadyReadyPreview, v0Aug 28, 2026 9:31pm
workbench-astro-workflowReadyReadyPreview, v0Aug 28, 2026 9:31pm
workbench-express-workflowReadyReadyPreview, v0Aug 28, 2026 9:31pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 28, 2026 9:31pm
workbench-hono-workflowReadyReadyPreview, v0Aug 28, 2026 9:31pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 28, 2026 9:31pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 28, 2026 9:31pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 28, 2026 9:31pm
workbench-python-workflowReadyReadyPreview, v0Aug 28, 2026 9:31pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 28, 2026 9:31pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 28, 2026 9:31pm
workbench-vite-workflowReadyReadyPreview, v0Aug 28, 2026 9:31pm
workflow-docsReadyReadyPreview, v0Aug 28, 2026 9:31pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 28, 2026 9:31pm
workflow-tarballsReadyReadyPreview, v0Aug 28, 2026 9:31pm
workflow-webReadyReadyPreview, v0Aug 28, 2026 9:31pm

@github-actions

github-actionsBot commented Aug 4, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (8 failed)

python-node (8 failed):

  • promiseAllWorkflow | wrun_41M154EHDD0GYPM54A0ZBVCT5P | 🔍 observability
  • sleepingWorkflow | wrun_41M154F8JZ0GN0Q47R4YGW3KP6 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M154FADA0GG40J2F380CMPPT | 🔍 observability
  • nullByteWorkflow | wrun_41M154FH680GRKBNMB0XP0ZRW8 | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M154MHPV0GT8JBXVMP122BS2 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154MP160GYFK15MB67KZS2K | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154MY910GJTVVWKXJNSKH9A | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154NHBQ0GRQ6FWG4RC6SSRF | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M154RRR7YPAATKTTG62FMHAW
  • promiseAllWorkflow | wrun_41M154EHDD0GYPM54A0ZBVCT5P
  • sleepingWorkflow | wrun_41M154F8JZ0GN0Q47R4YGW3KP6
  • parallelSleepWorkflow | wrun_41M154FADA0GG40J2F380CMPPT
  • nullByteWorkflow | wrun_41M154FH680GRKBNMB0XP0ZRW8
  • cancelRun - cancelling a running workflow | wrun_41M154MHPV0GT8JBXVMP122BS2
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154MP160GYFK15MB67KZS2K
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154MY910GJTVVWKXJNSKH9A
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154NHBQ0GRQ6FWG4RC6SSRF

vercel-multi-region (1 failed)

nextjs-turbopack (1 failed):

  • multi-region (world-vercel) hooks on non-iad1 runs hook created by a sfo1 run resolves by token and resumes the run

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

  • addTenWorkflow (nextjs-turbopack)
  • addTenWorkflow (tanstack-start)
  • cancelRun via CLI - cancelling a running workflow (nextjs-webpack)
  • hookWithSleepFinalStepWorkflow - step only on final payload (astro)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

32 infra events
  • cold-start-warmup · suite warmup (python) · at 21:31:18Z · abandoned wrun_41M154DWWM0GXDCJ114KWY8M44 · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 21:31:35Z · abandoned wrun_41M154HHVC0GPHCX7EPS34HFM8
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:31:35Z · abandoned wrun_41M154HHVC0GPHCX7EPS34HFM9
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:31:36Z · abandoned wrun_41M154HHV50GKZ8MBFHE2XH2KH
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:31:36Z · abandoned wrun_41M154HJ9E0GMB1EAD49N078CT
  • run-pickup-stall · nullByteWorkflow (python) · at 21:31:36Z · abandoned wrun_41M154HHVF0GZQ8SDFYC9PRMDH
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:32:09Z · abandoned wrun_41M154JKKN0GN73W6VT46NPQE3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:32:36Z · abandoned wrun_41M154KDDC0GVAEWH9GWWDG0MW
  • run-pickup-stall · sleepingWorkflow (python) · at 21:32:36Z · abandoned wrun_41M154KDJS0GR27K22T9FH6WTM
  • run-pickup-stall · nullByteWorkflow (python) · at 21:32:36Z · abandoned wrun_41M154KE130GKPQBWB0KDJBFR2
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:32:37Z · abandoned wrun_41M154KDZ40GVA6TN3FJJQ3P13
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:33:26Z · abandoned wrun_41M154KKRS0GWCT332GM3JWJFV
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:33:41Z · abandoned wrun_01M154N2TFMAQE1N80EPMWK2YB
  • cold-start-warmup · suite warmup (python) · at 21:34:00Z · abandoned wrun_01M154JTA88AXC8HZFVTV57874 · (+7 more)
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:34:15Z · abandoned wrun_01M154PFF0B5DWETMPV7GYV5QT
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:34:15Z · abandoned wrun_01M154PFF0B5DWETMPV7GYV5QV
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:34:15Z · abandoned wrun_01M154PFF71E0W2Q5KS01G08AM
  • run-pickup-stall · sleepingWorkflow (python) · at 21:34:15Z · abandoned wrun_01M154PFF5QRRRXJJE1TQMDR05
  • run-pickup-stall · nullByteWorkflow (python) · at 21:34:15Z · abandoned wrun_01M154PFF957T98R6XVSERQAND
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:35:09Z · abandoned wrun_41M154NG030GXKECRC3Y0ZT59H
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:35:10Z · abandoned wrun_41M154PBNB0GTF77M78J7FVEN2
  • run-pickup-stall · sleepingWorkflow (python) · at 21:35:15Z · abandoned wrun_01M154RA2VRG5S9JSXTDFME97F
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:35:15Z · abandoned wrun_01M154RA2XJ0THCACBQQ6JPZ0P
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:35:15Z · abandoned wrun_01M154RA2R5NDG2N1VE9NXTB2Q
  • run-pickup-stall · nullByteWorkflow (python) · at 21:35:15Z · abandoned wrun_01M154RA34C07WPGRS0GJBR81J
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:35:15Z · abandoned wrun_01M154RA2TA6BM8KXEEEYWRF6K
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:36:15Z · abandoned wrun_01M154T4PF42JPYZNFQ94N1K42
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:36:15Z · abandoned wrun_01M154T4PJCBVZZ3KMQDNXDHW0
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:36:15Z · abandoned wrun_01M154T4PSW3HY9R4N721PCCXG
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:36:45Z · abandoned wrun_01M154V224ZK96VW7AX1SE0SAC
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:36:45Z · abandoned wrun_01M154V222F7CBMHFPQ7Y85E02
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:37:15Z · abandoned wrun_01M154VZ9WTCXK1FRM5S0MXM2E

E2E Test Summary

Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
❌ vercel-multi-region261027
✅ vercel-ws-transport553087640
Total1705218277819848
Details by Category

❌ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000

❌ 🌐 Cross-language Conformance

AppPassedFailedSkipped
❌ python09132

✅ vercel-http-transport

AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028

❌ vercel-multi-region

AppPassedFailedSkipped
❌ nextjs-turbopack2610

✅ vercel-ws-transport

AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028

📋 View full workflow run

@github-actions

github-actionsBot commented Aug 4, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 8b0bfea · Fri, 28 Aug 2026 21:52:22 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep184 (-73%) 💚1260 🔴 (+11%)1272 🔴 (+11%)1390 🔴 (+21%) 🔻30
TTFSstream196 (-14%)1266 🔴 (+17%) 🔻1341 🔴 (+22%) 🔻1486 🔴 (+27%) 🔻30
TTFShook + stream1299 (+165%) 🔻1484 🔴 (+10%)1559 🔴 (+13%)1779 🔴 (+22%) 🔻30
Fan-out TTFSPromise.all(100 steps)583 (+3.6%)1987 (+16%) 🔻2052 (+13%)2094 (+6.6%)10
Fan-out TTLSPromise.all(100 steps)1783 (-19%) 💚3592 (-3.2%)3772 (±0%)4914 (-38%) 💚10
STSO1020 steps (inline)118 (+4.4%)146 (+4.3%)167 (+3.7%)358 (+82%) 🔻1019
WO1020 steps146437 (+4.0%)146437 (+4.0%)146437 (+4.0%)146437 (+4.0%)1
CRTTfirst chunk (pooled)98 (-15%)125 (-47%) 💚147 (-71%) 💚158 (-76%) 💚28

Streams

ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)120 (-17%)142 (-72%)180 (-74%)268 (-73%)124 (-73%)10
size sweep (100/s, 160B-12KB)116 (-35%)144 (-38%)238 (-58%)1257 (+40%)157 (-33%)10
replay gateway-gpt-5.4-nano-2000t (1x)112 (-43%)139 (-41%)184 (-58%)384 (-61%)227 (-69%)3
replay eve-gpt-5.6-sol-2000t (1x)130 (-66%)136 (-76%)181 (-90%)455 (-89%)383 (-82%)2
replay eve-gpt-5.6-sol-2000t (2x)115 (-27%)198 (-73%)272 (-74%)653 (-51%)416 (-27%)3
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 139837ms → this run 146230ms (Δ +6393ms, +5%)

100-150 ms █████████████████████┃██ main 857 this 799 -58
150-200 ms ████┃ main 152 this 185 +33
200-250 ms ┃ main 8 this 16 +8
250-300 ms ┃ main 0 this 3 +3
300-350 ms ┃ main 2 this 5 +3
350-400 ms ┃ main 0 this 7 +7
400-450 ms ┃ main 0 this 1 +1
500-550 ms ┃ main 0 this 1 +1
550-600 ms ┃ main 0 this 1 +1
600-650 ms ┃ main 0 this 1 +1
📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▄█▁···· 120.3 (-57%) 114 (-41%) 180 (-74%) 268 (-73%) 3000
sweep ······▄█▁▁▁·· 148.3 (-29%) 115 (-33%) 238 (-58%) 1257 (+40%) 3000
gw 1x ·····▁▅█▁···· 121.3 (-41%) 111 (-32%) 184 (-58%) 384 (-61%) 5295
eve 1x ·····▁▆█▂▁··· 121.6 (-71%) 108 (-41%) 181 (-90%) 455 (-89%) 5186
eve 2x ·····▁▂█▃▁··· 168.4 (-56%) 141 (-43%) 272 (-74%) 653 (-51%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control █▄▅▆▂▃▄▄▄▁ 113–130ms
sweep ▂▁▁▁▁▃▆█▅▃ 119–220ms
gw 1x ▃▆▄█▅▅▆▆▁▁ 111–132ms
eve 1x ▄▁▂▇▄▇█▅▄▅ 106–135ms
eve 2x ▁▂█▃▁▂▃█▂▃ 131–245ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep ▁▂▅█▆▆▃ 146–151ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▁▄█▁▆▆▇▇▆▄ 29–40ms
sweep ▁▃▃▂▄▃█▅▃▃ 37–86ms
gw 1x ▄▆▄▄█▅▇▇▁▅ 28–41ms
eve 1x ▃▁▃▇███▄▄█ 20–27ms
eve 2x ▄▄█▆▅▄▆▁▃▆ 22–32ms
ℹ️ Metric definitions & methodology

Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.

The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

This was referenced Aug 11, 2026
Open
Open
@github-actions

github-actionsBot commented Aug 11, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0

Full trace: world-sim.txt

@pranaygppranaygp mentioned this pull request Aug 17, 2026
Open
@pranaygp

pranaygp commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Ran this branch (ad4d8ea) against the failing CI scenario locally (staged tarball workbench, webpack stable node, full dev.test.ts): 1 of 2 runs still failed the body-only changes test with expected 2 to be 1 at the exact-{full: 1} fullCases assertions. Server log for the failing run shows full rediscovery → build (2.1s) → full rediscovery for a single fs.writeFile — a duplicate watcher event for the trigger write either lands mid-build (scheduleBuildOverlap schedules an unconditional follow-up full) or after it (byte-identical → conservative full). Same signature as the current main failures (expected 2/3/5 to be 1 across webpack/turbopack lanes).

The convergence design here is right — I stacked #3592 on top of this branch to close that residual: snapshots carry mtime alongside the content hash so a duplicate (same hash AND mtime) is a no-op while a same-content rewrite (newer mtime) keeps your conservative full; full decisions carry the classifier's reads so newly created files get seeded baselines; and with those two, the unconditional overlap-full becomes removable (overlapping events re-classify safely post-build). With the stack: 5/5 consecutive green full-suite runs on webpack stable node + 2/2 on turbopack, and 44/44 unit tests. Happy to fold it into this PR instead if you prefer — whichever lands these lanes faster.

🤖 Generated with Claude Code

@pranaygp

pranaygp commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Heads-up on c3d1f18 (directive file changes → full rediscovery): the body-only e2e (should follow Next flow-route HMR rebuild rules for body-only changes) has a case that edits hmr-fuzz-workflow.ts (a 'use workflow' file) and asserts exactly { hot: 1 } on stable lanes — under the new rule that edit classifies as full, so hot stays 0 and the case times out unless dev.test.ts is updated to expect the rediscovery. The queued run's Local Dev lanes should show it.

Also rebased #3592 on top of c3d1f18: the duplicate check (same hash AND mtime) now runs beforerequiresFullRediscovery, since a repeated notification for an already-consumed write is a no-op even for directive files — without that ordering, every duplicate watcher event on a workflow file becomes another full rediscovery, which is the exact-count flake again. (One toEqual→toMatchObject tweak to the new unit test since full decisions carry seed snapshots in my stack.)

🤖 Generated with Claude Code

@pranaygp

pranaygp commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

The bf2d678 turbopack stable Local Dev failures are the artifact-stability invariant, not the log counts this time: the annotation shows expected 1787091280593.9485 to be 1787091269521.0464 — that's the body-only test's expect(snapshot.stepMtimeMs).toBe(previousSnapshot.stepMtimeMs) for kind: 'workflow' cases. With directive files now taking a full rediscovery, the rediscovery rewrites __step_registrations.js, so the step-artifact mtime moves on a workflow-body edit. Options: preserve the invariant by skipping generated writes whose content didn't change (#3454's direction), or relax the mtime assertion for the cases that now rediscover.

Also worth flagging before this lands: the new exact { full: 1 } expectations inherit the duplicate-event exposure — a second watcher notification for the same write (they arrive up to ~1.2s apart in CI) classifies byte-identical → another full → expected 2 to be 1. That's the residual #3592 closes (hash+mtime duplicate → no-op), so it may be worth folding in or landing right behind this.

🤖 Generated with Claude Code

@pranaygppranaygp mentioned this pull request Aug 19, 2026
Open
@github-actions

github-actionsBot commented Aug 26, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.4 KiB (±0)1.76 MiB (±0)
nextjs-turbopack206.3 KiB (±0)439 B (±0)763.9 KiB (±0)
About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

8b0bfea · run

This was referenced Aug 26, 2026
Merged
Closed
@NathanColosimo
fix(next): converge workflow HMR rebuilds
9f8c803
@NathanColosimo
fix(next): rediscover directive file changes
f80c04e
@NathanColosimo
test(next): await HMR scheduler idle
7a0424f
@NathanColosimo
test(next): classify directive edits as full
f9336a2
@NathanColosimo
refactor(next): simplify HMR convergence
26ccbf1
@NathanColosimo
style(next): order HMR test imports
0052815
@NathanColosimo
fix(next): close remaining HMR consistency gaps
9505c6b
@NathanColosimo
fix(builders): preserve discovery helper API
8294a78
@NathanColosimo
fix(builders): preserve legacy import graph
c599903
@NathanColosimo
fix(builders): preserve discovery result compatibility
10d21d0
@NathanColosimo
fix(next): rebuild non-source workflow inputs
4147163
@NathanColosimo
fix(next): keep HMR rebuilds minimal
703cf38
@NathanColosimo
fix(next): bound full rebuild convergence
ce5a4a2
@NathanColosimo
perf(next): precompute affected workflow files
ac08470
@NathanColosimo
fix(next): honor selected source directory
694b773
@NathanColosimo
fix(next): match active entrypoint conventions
4d207be
@NathanColosimo
fix(next): classify Rspack entrypoints
2150e2b
@NathanColosimo
Merge remote-tracking branch 'origin/main' into codex/fix-next-hmr-bu…
8b0bfea
…ild-race
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@pranaygp@VaguelySerious