Skip to content

[world-vercel] Classify incomplete replay streams as transport failures - #3546

Merged
NathanColosimo merged 12 commits into
mainfrom
codex/replay-stream-recovery
Aug 31, 2026
Merged

[world-vercel] Classify incomplete replay streams as transport failures#3546
NathanColosimo merged 12 commits into
mainfrom
codex/replay-stream-recovery

Conversation

@NathanColosimo

@NathanColosimoNathanColosimo commented Aug 14, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • require the replay sentinel before treating a framed response as complete
  • preserve validated prefix events and resume from the last event cursor until the sentinel arrives
  • recover truncated run_started, atomic-hook preload, and unbounded events.list responses through the same direct GET loop
  • allow an initial GET plus two retries per GET chain; a partial setup POST can therefore issue at most three suffix GETs
  • validate cursor progress on every response before accepting its events
  • keep explicitly paginated requests one-page operations
  • retry only transport truncation, never schema or observer failures

Once a valid cursor exists, recovery keeps the accepted prefix and fetches only the missing suffix. The implementation is one direct do … while loop with only the current cursor and retry count as recovery state.

Stack

  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Trace fresh workflow replay phases #3797 — replay phase tracing
  3. [core] Overlap workflow compilation with replay loading #3798 — overlap Node bundle compilation
  4. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation

Validation

  • focused v4 event suite — 46 tests passed
  • full @workflow/world-vercel suite passed at the stack tip — 572 tests
  • pnpm --filter @workflow/world-vercel typecheck
  • git diff --check

@changeset-bot

changeset-botBot commented Aug 14, 2026•
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: e9d0c22

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Aug 14, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-astro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-express-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-hono-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-python-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-vite-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-docsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-tarballsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-webReadyReadyPreview, v0Aug 28, 2026 9:29pm

@github-actions

github-actionsBot commented Aug 14, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (8 failed)

python-node (8 failed):

  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593 | 🔍 observability
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC | 🔍 observability
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A | 🔍 observability
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0 | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M154KVRDKTZR0KJW5AE7APMN
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0

vercel-multi-region (1 failed)

nextjs-turbopack (1 failed):

  • multi-region (world-vercel) hooks on non-iad1 runs hook created by a sfo1 run resolves by token and resumes the run

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

14 flaky tests
  • addTenWorkflow (tanstack-start) — flaked in 2 jobs
  • addTenWorkflow (fastify)
  • addTenWorkflow (vite)
  • cancelRun via CLI - cancelling a running workflow (astro)
  • cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
  • cancelRun via CLI - cancelling a running workflow (nuxt)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (nitro)
  • hookWorkflow (express)
  • hookWorkflow (hono)
  • hookWorkflow (nest)
  • hookWorkflow (vite)
  • promiseAllWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (nest)
  • stepWinsRaceWorkflow (hono)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:28:37Z · abandoned wrun_41M1548R2Z0GG885SKHCNYMCDZ · (+5 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYM0GNGD2N01DRJP6TR
  • run-pickup-stall · sleepingWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYT0GP5T99M2JK2RNB3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYV0GXMSR1D3YJTJ610
  • run-pickup-stall · nullByteWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYZ0GG00VR3DZ0YNP0N
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:28:59Z · abandoned wrun_41M154CRS10GXV4254Y9GX493H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:29:34Z · abandoned wrun_41M154DT4Z0GM156G1JA91CMSJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFRY0GJEJ1Z0B94BVCQG
  • run-pickup-stall · nullByteWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFNB0GSXG82EEB0RQXDT
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EFTV0GQY9CA3YHZXKRHH
  • run-pickup-stall · sleepingWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EGAF0GJHMNKFRPRF44GY
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:30:22Z · abandoned wrun_01M154F0B7JB75RPZKHZ034NS7
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:30:45Z · abandoned wrun_41M154EXFW0GMM7P72ETDYBW6S
  • cold-start-warmup · suite warmup (python) · at 21:31:19Z · abandoned wrun_01M154DXAQ2WMDDXP5VZ5ZEACR · (+7 more)
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:31:31Z · abandoned wrun_41M154GCP60GWVR1RK1CJTWBQG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:31:34Z · abandoned wrun_01M154HJF8CHCNH305GH60575D
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJF9KX1602SFN1R769MG
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFG65FSMNG5KS7T8RYK
  • run-pickup-stall · sleepingWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFE6AFZ9RJVFSH4C697
  • run-pickup-stall · nullByteWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFJA17HEXHTG5HVYDWG
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:32:27Z · abandoned wrun_41M154HNCC0GH28PPPFDZAAJKZ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD39BMYM0PFJQ6F7CZXJ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:32:34Z · abandoned wrun_01M154KD31F5ND2T1R36J63R0S
  • run-pickup-stall · sleepingWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD37515KM9F5AXCPH0D9
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD339R9SBBJWEXZ9EHV1
  • run-pickup-stall · nullByteWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD3CQSQ19C8YMZ3RYDA9
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:14Z · abandoned wrun_41M154K4JR0GK30D8JPG4B2594
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PM9FAFWKYWTDJ2K72C
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:34Z · abandoned wrun_01M154N7Q0PB9SRVATNJNVRQSK
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PX7F4R77XDCK1DMH1P
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P534814GPCJK014PHNT6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P532HAGPS0W06MY4MPSQ
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:34:19Z · abandoned wrun_01M154PKKN2JBV7KWZ5JPHB62B
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:34:34Z · abandoned wrun_01M154Q2A3KDYJRE3BW5MC7R37

E2E Test Summary

Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
❌ vercel-multi-region261027
✅ vercel-ws-transport553087640
Total1705218277819848
Details by Category

❌ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000

❌ 🌐 Cross-language Conformance

AppPassedFailedSkipped
❌ python09132

✅ vercel-http-transport

AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028

❌ vercel-multi-region

AppPassedFailedSkipped
❌ nextjs-turbopack2610

✅ vercel-ws-transport

AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028

📋 View full workflow run

@github-actions

github-actionsBot commented Aug 14, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit e9d0c22 · Fri, 28 Aug 2026 21:51:49 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1275 (+88%) 🔻1368 🔴 (+21%) 🔻1402 🔴 (+23%) 🔻1658 🔴 (+44%) 🔻30
TTFSstream271 (+18%) 🔻1322 🔴 (+23%) 🔻1336 🔴 (+21%) 🔻1454 🔴 (+24%) 🔻30
TTFShook + stream1184 (+142%) 🔻1548 🔴 (+15%)1630 🔴 (+18%) 🔻2027 🔴 (+39%) 🔻30
Fan-out TTFSPromise.all(100 steps)613 (+8.9%)2049 (+20%) 🔻2051 (+13%)5964 (+204%) 🔻10
Fan-out TTLSPromise.all(100 steps)2045 (-6.9%)4006 (+8.0%)4238 (+13%)12428 (+58%) 🔻10
STSO1020 steps (inline)110 (-2.7%)142 (+1.4%)161 (±0%)218 (+11%)1019
WO1020 steps141217 (±0%)141217 (±0%)141217 (±0%)141217 (±0%)1
CRTTfirst chunk (pooled)97 (-16%) 💚142 (-39%) 💚156 (-69%) 💚380 (-43%) 💚28

Streams

ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)127 (-11%)144 (-71%)199 (-72%)343 (-65%)120 (-74%)10
size sweep (100/s, 160B-12KB)118 (-34%)158 (-32%)232 (-59%)430 (-52%)149 (-37%)10
replay gateway-gpt-5.4-nano-2000t (1x)121 (-38%)142 (-39%)179 (-60%)438 (-55%)245 (-66%)3
replay eve-gpt-5.6-sol-2000t (1x)117 (-70%)132 (-77%)164 (-91%)335 (-92%)245 (-89%)2
replay eve-gpt-5.6-sol-2000t (2x)153 (-3%)173 (-76%)232 (-78%)430 (-67%)289 (-49%)3
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 139837ms → this run 140733ms (Δ +896ms, +1%)

100-150 ms ███████████████████████┃ main 857 this 858 +1
150-200 ms ███┃ main 152 this 148 -4
200-250 ms ┃ main 8 this 7 -1
250-300 ms ┃ main 0 this 2 +2
300-350 ms ┃ main 2 this 2 +0
350-400 ms ┃ main 0 this 1 +1
450-500 ms ┃ main 0 this 1 +1
📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▄█▁···· 123.8 (-56%) 119 (-38%) 199 (-72%) 343 (-65%) 3000
sweep ······▃█▁···· 133.2 (-36%) 121 (-30%) 232 (-59%) 430 (-52%) 3000
gw 1x ·····▁▆█▁▁··· 116.4 (-43%) 108 (-34%) 179 (-60%) 438 (-55%) 5295
eve 1x ·····▁▆█▁···· 115.6 (-73%) 106 (-42%) 164 (-91%) 335 (-92%) 5186
eve 2x ·····▁▂█▂▁··· 149.1 (-61%) 136 (-45%) 232 (-78%) 430 (-67%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control ▆▆▄▆█▄▄▂▃▁ 114–135ms
sweep █▄▅▂▇▅▄▇▁▁ 117–151ms
gw 1x ▅▄▃▁▃▁▆▃▆█ 106–131ms
eve 1x ▅▄▇▃▂▃█▅▃▁ 95–140ms
eve 2x ▃▁▄▂▁▁▄█▃▂ 127–209ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep ▁▆▅▂▆█▆ 132–134ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▁▅▃▅▃█▄▃▇▄ 32–41ms
sweep ▁▄▆▅█▆██▅▃ 37–63ms
gw 1x ▅▆▄▁▆▁▆▆█▆ 29–38ms
eve 1x ▇▅▆▃▄▄█▂▅▁ 19–25ms
eve 2x █▄▂▁▆▂▅▂▁▄ 20–30ms
ℹ️ Metric definitions & methodology

Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.

The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@NathanColosimoNathanColosimo changed the title Resume partial replay event streams[world-vercel] Resume partial replay streams after validated eventsAug 14, 2026
This was referenced Aug 14, 2026
Merged
Merged
Draft
@NathanColosimo
NathanColosimoforce-pushed the codex/replay-stream-recovery branch from e9e7488 to dbdc9a2CompareAugust 14, 2026 03:56
@github-actions

github-actionsBot commented Aug 14, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0

Full trace: world-sim.txt

VaguelySerious
VaguelySerious reviewed Aug 21, 2026
View reviewed changes

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, will let agent approve if no blockers found

VaguelySerious
VaguelySerious reviewed Aug 21, 2026
View reviewed changes

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found

Comment threadpackages/world-vercel/src/events-v4.ts Outdated
Comment threadpackages/world-vercel/src/events-v4.ts Outdated
Comment threadpackages/world-vercel/src/storage.ts Outdated
vercel[bot]
vercelBot reviewed Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts Outdated
@github-actions

github-actionsBot commented Aug 25, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.4 KiB (±0)1.76 MiB (+827 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)764.3 KiB (+412 B)
About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

e9d0c22 · run

@NathanColosimoNathanColosimo mentioned this pull request Aug 25, 2026
Merged
vercel[bot]
vercelBot reviewed Aug 26, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts Outdated
vercel[bot]
vercelBot reviewed Aug 27, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts Outdated
NathanColosimoand others added 12 commits August 28, 2026 14:24
@NathanColosimo
Classify replay stream failures by ownership
f7a0c30
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@NathanColosimo
fix(world-vercel): resume truncated replay streams
2653939
@NathanColosimo
fix(world-vercel): require replay continuation cursor
cf646d9
@NathanColosimo
refactor(world-vercel): simplify replay recovery
bd05dca
@NathanColosimo
fix(world-vercel): append replay pages safely
b429134
@NathanColosimo
fix(world-vercel): bound partial stream continuations
32cd767
@NathanColosimo
fix(world-vercel): bound replay continuations
edeaf78
@NathanColosimo
fix(world-vercel): retain recovered replay cursors
7b8d0ff
@NathanColosimo
fix(world-vercel): validate each recovery cursor
c3aae5c
@NathanColosimo
test(world-vercel): include list response cursors
83e1ac7
@NathanColosimo
refactor(world-vercel): simplify replay recovery
b3095a5
@NathanColosimo@vercel
Update packages/world-vercel/src/events-v4.ts
e9d0c22
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@github-actionsgithub-actionsBot mentioned this pull request Aug 31, 2026
Open
@github-actions

github-actionsBot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

No backport to stable for aaeb8e4 (AI decision).

This change is built on the streamed replay-log machinery that only exists on main: origin/stable's packages/world-vercel/src/events-v4.ts has no createWorkflowRunStartedEventV4, no createHookReceivedPreloadEventV4, no consumeEventFrameStream/EventStreamEndSchema cursor+hasMore result, and no prior truncated-stream resume path, so the resume-and-retry loop it introduces has nothing to attach to there. On stable a truncated frame stream already fails loudly rather than silently dropping events, so what remains is resiliency work (multi-GET recovery, retry bounds, cursor-progress guard) plus new typed WorkflowWorldError classification of existing throws — enhancements, not a defect fix for the maintenance line. If the typed-error classification in decodeCreateEventResponse/events.ts is wanted on stable, it should be split out and forced through separately.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

aaeb8e44633c1974f087ee7ce38666dbc8ad8a4d

This was referenced Aug 31, 2026
Open
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@alangenfeldalangenfeldalangenfeld approved these changes
@VaguelySeriousVaguelySeriousAwaiting requested review from VaguelySerious

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@alangenfeld@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
[world-vercel] Classify incomplete replay streams as transport failures by NathanColosimo · Pull Request #3546 · vercel/workflow · GitHub
Skip to content
[world-vercel] Classify incomplete replay streams as transport failures - #3546
Merged
NathanColosimo merged 12 commits into
mainfrom 
codex/replay-stream-recovery
Aug 31, 2026
Merged
[world-vercel] Classify incomplete replay streams as transport failures#3546
NathanColosimo merged 12 commits into
mainfrom 
codex/replay-stream-recovery
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • require the replay sentinel before treating a framed response as complete
  • preserve validated prefix events and resume from the last event cursor until the sentinel arrives
  • recover truncated run_started, atomic-hook preload, and unbounded events.list responses through the same direct GET loop
  • allow an initial GET plus two retries per GET chain; a partial setup POST can therefore issue at most three suffix GETs
  • validate cursor progress on every response before accepting its events
  • keep explicitly paginated requests one-page operations
  • retry only transport truncation, never schema or observer failures
Once a valid cursor exists, recovery keeps the accepted prefix and fetches only the missing suffix. The implementation is one direct do … while loop with only the current cursor and retry count as recovery state.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Trace fresh workflow replay phases #3797 — replay phase tracing
  3. [core] Overlap workflow compilation with replay loading #3798 — overlap Node bundle compilation
  4. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
Validation
  • focused v4 event suite — 46 tests passed
  • full @workflow/world-vercel suite passed at the stack tip — 572 tests
  • pnpm --filter @workflow/world-vercel typecheck
  • git diff --check
@changeset-bot
changeset-botBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e9d0c22
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-astro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-express-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-hono-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-python-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-vite-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-docsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-tarballsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-webReadyReadyPreview, v0Aug 28, 2026 9:29pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593 | 🔍 observability
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC | 🔍 observability
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A | 🔍 observability
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M154KVRDKTZR0KJW5AE7APMN
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0
vercel-multi-region (1 failed)
nextjs-turbopack (1 failed):
  • multi-region (world-vercel) hooks on non-iad1 runs hook created by a sfo1 run resolves by token and resumes the run
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
14 flaky tests
  • addTenWorkflow (tanstack-start) — flaked in 2 jobs
  • addTenWorkflow (fastify)
  • addTenWorkflow (vite)
  • cancelRun via CLI - cancelling a running workflow (astro)
  • cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
  • cancelRun via CLI - cancelling a running workflow (nuxt)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (nitro)
  • hookWorkflow (express)
  • hookWorkflow (hono)
  • hookWorkflow (nest)
  • hookWorkflow (vite)
  • promiseAllWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (nest)
  • stepWinsRaceWorkflow (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:28:37Z · abandoned wrun_41M1548R2Z0GG885SKHCNYMCDZ · (+5 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYM0GNGD2N01DRJP6TR
  • run-pickup-stall · sleepingWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYT0GP5T99M2JK2RNB3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYV0GXMSR1D3YJTJ610
  • run-pickup-stall · nullByteWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYZ0GG00VR3DZ0YNP0N
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:28:59Z · abandoned wrun_41M154CRS10GXV4254Y9GX493H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:29:34Z · abandoned wrun_41M154DT4Z0GM156G1JA91CMSJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFRY0GJEJ1Z0B94BVCQG
  • run-pickup-stall · nullByteWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFNB0GSXG82EEB0RQXDT
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EFTV0GQY9CA3YHZXKRHH
  • run-pickup-stall · sleepingWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EGAF0GJHMNKFRPRF44GY
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:30:22Z · abandoned wrun_01M154F0B7JB75RPZKHZ034NS7
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:30:45Z · abandoned wrun_41M154EXFW0GMM7P72ETDYBW6S
  • cold-start-warmup · suite warmup (python) · at 21:31:19Z · abandoned wrun_01M154DXAQ2WMDDXP5VZ5ZEACR · (+7 more)
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:31:31Z · abandoned wrun_41M154GCP60GWVR1RK1CJTWBQG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:31:34Z · abandoned wrun_01M154HJF8CHCNH305GH60575D
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJF9KX1602SFN1R769MG
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFG65FSMNG5KS7T8RYK
  • run-pickup-stall · sleepingWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFE6AFZ9RJVFSH4C697
  • run-pickup-stall · nullByteWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFJA17HEXHTG5HVYDWG
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:32:27Z · abandoned wrun_41M154HNCC0GH28PPPFDZAAJKZ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD39BMYM0PFJQ6F7CZXJ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:32:34Z · abandoned wrun_01M154KD31F5ND2T1R36J63R0S
  • run-pickup-stall · sleepingWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD37515KM9F5AXCPH0D9
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD339R9SBBJWEXZ9EHV1
  • run-pickup-stall · nullByteWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD3CQSQ19C8YMZ3RYDA9
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:14Z · abandoned wrun_41M154K4JR0GK30D8JPG4B2594
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PM9FAFWKYWTDJ2K72C
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:34Z · abandoned wrun_01M154N7Q0PB9SRVATNJNVRQSK
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PX7F4R77XDCK1DMH1P
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P534814GPCJK014PHNT6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P532HAGPS0W06MY4MPSQ
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:34:19Z · abandoned wrun_01M154PKKN2JBV7KWZ5JPHB62B
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:34:34Z · abandoned wrun_01M154Q2A3KDYJRE3BW5MC7R37
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
❌ vercel-multi-region261027
✅ vercel-ws-transport553087640
Total1705218277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
❌ vercel-multi-region
AppPassedFailedSkipped
❌ nextjs-turbopack2610
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e9d0c22 · Fri, 28 Aug 2026 21:51:49 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1275 (+88%) 🔻1368 🔴 (+21%) 🔻1402 🔴 (+23%) 🔻1658 🔴 (+44%) 🔻30
TTFSstream271 (+18%) 🔻1322 🔴 (+23%) 🔻1336 🔴 (+21%) 🔻1454 🔴 (+24%) 🔻30
TTFShook + stream1184 (+142%) 🔻1548 🔴 (+15%)1630 🔴 (+18%) 🔻2027 🔴 (+39%) 🔻30
Fan-out TTFSPromise.all(100 steps)613 (+8.9%)2049 (+20%) 🔻2051 (+13%)5964 (+204%) 🔻10
Fan-out TTLSPromise.all(100 steps)2045 (-6.9%)4006 (+8.0%)4238 (+13%)12428 (+58%) 🔻10
STSO1020 steps (inline)110 (-2.7%)142 (+1.4%)161 (±0%)218 (+11%)1019
WO1020 steps141217 (±0%)141217 (±0%)141217 (±0%)141217 (±0%)1
CRTTfirst chunk (pooled)97 (-16%) 💚142 (-39%) 💚156 (-69%) 💚380 (-43%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)127 (-11%)144 (-71%)199 (-72%)343 (-65%)120 (-74%)10
size sweep (100/s, 160B-12KB)118 (-34%)158 (-32%)232 (-59%)430 (-52%)149 (-37%)10
replay gateway-gpt-5.4-nano-2000t (1x)121 (-38%)142 (-39%)179 (-60%)438 (-55%)245 (-66%)3
replay eve-gpt-5.6-sol-2000t (1x)117 (-70%)132 (-77%)164 (-91%)335 (-92%)245 (-89%)2
replay eve-gpt-5.6-sol-2000t (2x)153 (-3%)173 (-76%)232 (-78%)430 (-67%)289 (-49%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 139837ms → this run 140733ms (Δ +896ms, +1%)
100-150 ms ███████████████████████┃ main 857 this 858 +1
150-200 ms ███┃ main 152 this 148 -4
200-250 ms ┃ main 8 this 7 -1
250-300 ms ┃ main 0 this 2 +2
300-350 ms ┃ main 2 this 2 +0
350-400 ms ┃ main 0 this 1 +1
450-500 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▄█▁···· 123.8 (-56%) 119 (-38%) 199 (-72%) 343 (-65%) 3000
sweep ······▃█▁···· 133.2 (-36%) 121 (-30%) 232 (-59%) 430 (-52%) 3000
gw 1x ·····▁▆█▁▁··· 116.4 (-43%) 108 (-34%) 179 (-60%) 438 (-55%) 5295
eve 1x ·····▁▆█▁···· 115.6 (-73%) 106 (-42%) 164 (-91%) 335 (-92%) 5186
eve 2x ·····▁▂█▂▁··· 149.1 (-61%) 136 (-45%) 232 (-78%) 430 (-67%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▆▆▄▆█▄▄▂▃▁ 114–135ms
sweep █▄▅▂▇▅▄▇▁▁ 117–151ms
gw 1x ▅▄▃▁▃▁▆▃▆█ 106–131ms
eve 1x ▅▄▇▃▂▃█▅▃▁ 95–140ms
eve 2x ▃▁▄▂▁▁▄█▃▂ 127–209ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▆▅▂▆█▆ 132–134ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▁▅▃▅▃█▄▃▇▄ 32–41ms
sweep ▁▄▆▅█▆██▅▃ 37–63ms
gw 1x ▅▆▄▁▆▁▆▆█▆ 29–38ms
eve 1x ▇▅▆▃▄▄█▂▅▁ 19–25ms
eve 2x █▄▂▁▆▂▅▂▁▄ 20–30ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@NathanColosimoNathanColosimo
changed the title
Resume partial replay event streams[world-vercel] Resume partial replay streams after validated eventsAug 14, 2026

This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

@NathanColosimo
NathanColosimoforce-pushed
the
codex/replay-stream-recovery
branch
from
e9e7488 to
dbdc9a2CompareAugust 14, 2026 03:56
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, will let agent approve if no blockers found
VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

Comment threadpackages/world-vercel/src/storage.ts
Outdated

vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actions
github-actionsBot
commented
Aug 25, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.4 KiB (±0)1.76 MiB (+827 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)764.3 KiB (+412 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
e9d0c22 · run
@NathanColosimoNathanColosimo
mentioned this pull request
Aug 25, 2026
 Merged

vercel[bot]
vercelBot
reviewed
Aug 26, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

vercel[bot]
vercelBot
reviewed
Aug 27, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

NathanColosimoand others
added 12 commits
August 28, 2026 14:24
@NathanColosimo
Classify replay stream failures by ownership
f7a0c30
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@NathanColosimo
fix(world-vercel): resume truncated replay streams
2653939
@NathanColosimo
fix(world-vercel): require replay continuation cursor
cf646d9
@NathanColosimo
refactor(world-vercel): simplify replay recovery
bd05dca
@NathanColosimo
fix(world-vercel): append replay pages safely
b429134
@NathanColosimo
fix(world-vercel): bound partial stream continuations
32cd767
@NathanColosimo
fix(world-vercel): bound replay continuations
edeaf78
@NathanColosimo
fix(world-vercel): retain recovered replay cursors
7b8d0ff
@NathanColosimo
fix(world-vercel): validate each recovery cursor
c3aae5c
@NathanColosimo
test(world-vercel): include list response cursors
83e1ac7
@NathanColosimo
refactor(world-vercel): simplify replay recovery
b3095a5
@NathanColosimo@vercel
Update packages/world-vercel/src/events-v4.ts
e9d0c22
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Aug 31, 2026
 Open

@github-actions
github-actionsBot
commented
Aug 31, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for aaeb8e4 (AI decision).
This change is built on the streamed replay-log machinery that only exists on main: origin/stable's packages/world-vercel/src/events-v4.ts has no createWorkflowRunStartedEventV4, no createHookReceivedPreloadEventV4, no consumeEventFrameStream/EventStreamEndSchema cursor+hasMore result, and no prior truncated-stream resume path, so the resume-and-retry loop it introduces has nothing to attach to there. On stable a truncated frame stream already fails loudly rather than silently dropping events, so what remains is resiliency work (multi-GET recovery, retry bounds, cursor-progress guard) plus new typed WorkflowWorldError classification of existing throws — enhancements, not a defect fix for the maintenance line. If the typed-error classification in decodeCreateEventResponse/events.ts is wanted on stable, it should be split out and forced through separately.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
aaeb8e44633c1974f087ee7ce38666dbc8ad8a4d


This was referenced Aug 31, 2026
 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@alangenfeldalangenfeldalangenfeld approved these changes
@VaguelySeriousVaguelySeriousAwaiting requested review from VaguelySerious

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@alangenfeld@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
[world-vercel] Classify incomplete replay streams as transport failures by NathanColosimo · Pull Request #3546 · vercel/workflow · GitHub
Skip to content
[world-vercel] Classify incomplete replay streams as transport failures - #3546
Merged
NathanColosimo merged 12 commits into
mainfrom 
codex/replay-stream-recovery
Aug 31, 2026
Merged
[world-vercel] Classify incomplete replay streams as transport failures#3546
NathanColosimo merged 12 commits into
mainfrom 
codex/replay-stream-recovery
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • require the replay sentinel before treating a framed response as complete
  • preserve validated prefix events and resume from the last event cursor until the sentinel arrives
  • recover truncated run_started, atomic-hook preload, and unbounded events.list responses through the same direct GET loop
  • allow an initial GET plus two retries per GET chain; a partial setup POST can therefore issue at most three suffix GETs
  • validate cursor progress on every response before accepting its events
  • keep explicitly paginated requests one-page operations
  • retry only transport truncation, never schema or observer failures
Once a valid cursor exists, recovery keeps the accepted prefix and fetches only the missing suffix. The implementation is one direct do … while loop with only the current cursor and retry count as recovery state.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Trace fresh workflow replay phases #3797 — replay phase tracing
  3. [core] Overlap workflow compilation with replay loading #3798 — overlap Node bundle compilation
  4. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
Validation
  • focused v4 event suite — 46 tests passed
  • full @workflow/world-vercel suite passed at the stack tip — 572 tests
  • pnpm --filter @workflow/world-vercel typecheck
  • git diff --check
@changeset-bot
changeset-botBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e9d0c22
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-astro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-express-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-hono-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-python-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-vite-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-docsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-tarballsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-webReadyReadyPreview, v0Aug 28, 2026 9:29pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593 | 🔍 observability
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC | 🔍 observability
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A | 🔍 observability
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M154KVRDKTZR0KJW5AE7APMN
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0
vercel-multi-region (1 failed)
nextjs-turbopack (1 failed):
  • multi-region (world-vercel) hooks on non-iad1 runs hook created by a sfo1 run resolves by token and resumes the run
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
14 flaky tests
  • addTenWorkflow (tanstack-start) — flaked in 2 jobs
  • addTenWorkflow (fastify)
  • addTenWorkflow (vite)
  • cancelRun via CLI - cancelling a running workflow (astro)
  • cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
  • cancelRun via CLI - cancelling a running workflow (nuxt)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (nitro)
  • hookWorkflow (express)
  • hookWorkflow (hono)
  • hookWorkflow (nest)
  • hookWorkflow (vite)
  • promiseAllWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (nest)
  • stepWinsRaceWorkflow (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:28:37Z · abandoned wrun_41M1548R2Z0GG885SKHCNYMCDZ · (+5 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYM0GNGD2N01DRJP6TR
  • run-pickup-stall · sleepingWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYT0GP5T99M2JK2RNB3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYV0GXMSR1D3YJTJ610
  • run-pickup-stall · nullByteWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYZ0GG00VR3DZ0YNP0N
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:28:59Z · abandoned wrun_41M154CRS10GXV4254Y9GX493H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:29:34Z · abandoned wrun_41M154DT4Z0GM156G1JA91CMSJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFRY0GJEJ1Z0B94BVCQG
  • run-pickup-stall · nullByteWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFNB0GSXG82EEB0RQXDT
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EFTV0GQY9CA3YHZXKRHH
  • run-pickup-stall · sleepingWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EGAF0GJHMNKFRPRF44GY
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:30:22Z · abandoned wrun_01M154F0B7JB75RPZKHZ034NS7
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:30:45Z · abandoned wrun_41M154EXFW0GMM7P72ETDYBW6S
  • cold-start-warmup · suite warmup (python) · at 21:31:19Z · abandoned wrun_01M154DXAQ2WMDDXP5VZ5ZEACR · (+7 more)
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:31:31Z · abandoned wrun_41M154GCP60GWVR1RK1CJTWBQG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:31:34Z · abandoned wrun_01M154HJF8CHCNH305GH60575D
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJF9KX1602SFN1R769MG
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFG65FSMNG5KS7T8RYK
  • run-pickup-stall · sleepingWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFE6AFZ9RJVFSH4C697
  • run-pickup-stall · nullByteWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFJA17HEXHTG5HVYDWG
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:32:27Z · abandoned wrun_41M154HNCC0GH28PPPFDZAAJKZ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD39BMYM0PFJQ6F7CZXJ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:32:34Z · abandoned wrun_01M154KD31F5ND2T1R36J63R0S
  • run-pickup-stall · sleepingWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD37515KM9F5AXCPH0D9
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD339R9SBBJWEXZ9EHV1
  • run-pickup-stall · nullByteWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD3CQSQ19C8YMZ3RYDA9
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:14Z · abandoned wrun_41M154K4JR0GK30D8JPG4B2594
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PM9FAFWKYWTDJ2K72C
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:34Z · abandoned wrun_01M154N7Q0PB9SRVATNJNVRQSK
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PX7F4R77XDCK1DMH1P
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P534814GPCJK014PHNT6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P532HAGPS0W06MY4MPSQ
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:34:19Z · abandoned wrun_01M154PKKN2JBV7KWZ5JPHB62B
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:34:34Z · abandoned wrun_01M154Q2A3KDYJRE3BW5MC7R37
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
❌ vercel-multi-region261027
✅ vercel-ws-transport553087640
Total1705218277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
❌ vercel-multi-region
AppPassedFailedSkipped
❌ nextjs-turbopack2610
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e9d0c22 · Fri, 28 Aug 2026 21:51:49 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1275 (+88%) 🔻1368 🔴 (+21%) 🔻1402 🔴 (+23%) 🔻1658 🔴 (+44%) 🔻30
TTFSstream271 (+18%) 🔻1322 🔴 (+23%) 🔻1336 🔴 (+21%) 🔻1454 🔴 (+24%) 🔻30
TTFShook + stream1184 (+142%) 🔻1548 🔴 (+15%)1630 🔴 (+18%) 🔻2027 🔴 (+39%) 🔻30
Fan-out TTFSPromise.all(100 steps)613 (+8.9%)2049 (+20%) 🔻2051 (+13%)5964 (+204%) 🔻10
Fan-out TTLSPromise.all(100 steps)2045 (-6.9%)4006 (+8.0%)4238 (+13%)12428 (+58%) 🔻10
STSO1020 steps (inline)110 (-2.7%)142 (+1.4%)161 (±0%)218 (+11%)1019
WO1020 steps141217 (±0%)141217 (±0%)141217 (±0%)141217 (±0%)1
CRTTfirst chunk (pooled)97 (-16%) 💚142 (-39%) 💚156 (-69%) 💚380 (-43%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)127 (-11%)144 (-71%)199 (-72%)343 (-65%)120 (-74%)10
size sweep (100/s, 160B-12KB)118 (-34%)158 (-32%)232 (-59%)430 (-52%)149 (-37%)10
replay gateway-gpt-5.4-nano-2000t (1x)121 (-38%)142 (-39%)179 (-60%)438 (-55%)245 (-66%)3
replay eve-gpt-5.6-sol-2000t (1x)117 (-70%)132 (-77%)164 (-91%)335 (-92%)245 (-89%)2
replay eve-gpt-5.6-sol-2000t (2x)153 (-3%)173 (-76%)232 (-78%)430 (-67%)289 (-49%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 139837ms → this run 140733ms (Δ +896ms, +1%)
100-150 ms ███████████████████████┃ main 857 this 858 +1
150-200 ms ███┃ main 152 this 148 -4
200-250 ms ┃ main 8 this 7 -1
250-300 ms ┃ main 0 this 2 +2
300-350 ms ┃ main 2 this 2 +0
350-400 ms ┃ main 0 this 1 +1
450-500 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▄█▁···· 123.8 (-56%) 119 (-38%) 199 (-72%) 343 (-65%) 3000
sweep ······▃█▁···· 133.2 (-36%) 121 (-30%) 232 (-59%) 430 (-52%) 3000
gw 1x ·····▁▆█▁▁··· 116.4 (-43%) 108 (-34%) 179 (-60%) 438 (-55%) 5295
eve 1x ·····▁▆█▁···· 115.6 (-73%) 106 (-42%) 164 (-91%) 335 (-92%) 5186
eve 2x ·····▁▂█▂▁··· 149.1 (-61%) 136 (-45%) 232 (-78%) 430 (-67%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▆▆▄▆█▄▄▂▃▁ 114–135ms
sweep █▄▅▂▇▅▄▇▁▁ 117–151ms
gw 1x ▅▄▃▁▃▁▆▃▆█ 106–131ms
eve 1x ▅▄▇▃▂▃█▅▃▁ 95–140ms
eve 2x ▃▁▄▂▁▁▄█▃▂ 127–209ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▆▅▂▆█▆ 132–134ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▁▅▃▅▃█▄▃▇▄ 32–41ms
sweep ▁▄▆▅█▆██▅▃ 37–63ms
gw 1x ▅▆▄▁▆▁▆▆█▆ 29–38ms
eve 1x ▇▅▆▃▄▄█▂▅▁ 19–25ms
eve 2x █▄▂▁▆▂▅▂▁▄ 20–30ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@NathanColosimoNathanColosimo
changed the title
Resume partial replay event streams[world-vercel] Resume partial replay streams after validated eventsAug 14, 2026

This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

@NathanColosimo
NathanColosimoforce-pushed
the
codex/replay-stream-recovery
branch
from
e9e7488 to
dbdc9a2CompareAugust 14, 2026 03:56
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, will let agent approve if no blockers found
VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

Comment threadpackages/world-vercel/src/storage.ts
Outdated

vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actions
github-actionsBot
commented
Aug 25, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.4 KiB (±0)1.76 MiB (+827 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)764.3 KiB (+412 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
e9d0c22 · run
@NathanColosimoNathanColosimo
mentioned this pull request
Aug 25, 2026
 Merged

vercel[bot]
vercelBot
reviewed
Aug 26, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

vercel[bot]
vercelBot
reviewed
Aug 27, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

NathanColosimoand others
added 12 commits
August 28, 2026 14:24
@NathanColosimo
Classify replay stream failures by ownership
f7a0c30
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@NathanColosimo
fix(world-vercel): resume truncated replay streams
2653939
@NathanColosimo
fix(world-vercel): require replay continuation cursor
cf646d9
@NathanColosimo
refactor(world-vercel): simplify replay recovery
bd05dca
@NathanColosimo
fix(world-vercel): append replay pages safely
b429134
@NathanColosimo
fix(world-vercel): bound partial stream continuations
32cd767
@NathanColosimo
fix(world-vercel): bound replay continuations
edeaf78
@NathanColosimo
fix(world-vercel): retain recovered replay cursors
7b8d0ff
@NathanColosimo
fix(world-vercel): validate each recovery cursor
c3aae5c
@NathanColosimo
test(world-vercel): include list response cursors
83e1ac7
@NathanColosimo
refactor(world-vercel): simplify replay recovery
b3095a5
@NathanColosimo@vercel
Update packages/world-vercel/src/events-v4.ts
e9d0c22
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Aug 31, 2026
 Open

@github-actions
github-actionsBot
commented
Aug 31, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for aaeb8e4 (AI decision).
This change is built on the streamed replay-log machinery that only exists on main: origin/stable's packages/world-vercel/src/events-v4.ts has no createWorkflowRunStartedEventV4, no createHookReceivedPreloadEventV4, no consumeEventFrameStream/EventStreamEndSchema cursor+hasMore result, and no prior truncated-stream resume path, so the resume-and-retry loop it introduces has nothing to attach to there. On stable a truncated frame stream already fails loudly rather than silently dropping events, so what remains is resiliency work (multi-GET recovery, retry bounds, cursor-progress guard) plus new typed WorkflowWorldError classification of existing throws — enhancements, not a defect fix for the maintenance line. If the typed-error classification in decodeCreateEventResponse/events.ts is wanted on stable, it should be split out and forced through separately.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
aaeb8e44633c1974f087ee7ce38666dbc8ad8a4d


This was referenced Aug 31, 2026
 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@alangenfeldalangenfeldalangenfeld approved these changes
@VaguelySeriousVaguelySeriousAwaiting requested review from VaguelySerious

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@alangenfeld@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
[world-vercel] Classify incomplete replay streams as transport failures by NathanColosimo · Pull Request #3546 · vercel/workflow · GitHub
Skip to content
[world-vercel] Classify incomplete replay streams as transport failures - #3546
Merged
NathanColosimo merged 12 commits into
mainfrom 
codex/replay-stream-recovery
Aug 31, 2026
Merged
[world-vercel] Classify incomplete replay streams as transport failures#3546
NathanColosimo merged 12 commits into
mainfrom 
codex/replay-stream-recovery
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • require the replay sentinel before treating a framed response as complete
  • preserve validated prefix events and resume from the last event cursor until the sentinel arrives
  • recover truncated run_started, atomic-hook preload, and unbounded events.list responses through the same direct GET loop
  • allow an initial GET plus two retries per GET chain; a partial setup POST can therefore issue at most three suffix GETs
  • validate cursor progress on every response before accepting its events
  • keep explicitly paginated requests one-page operations
  • retry only transport truncation, never schema or observer failures
Once a valid cursor exists, recovery keeps the accepted prefix and fetches only the missing suffix. The implementation is one direct do … while loop with only the current cursor and retry count as recovery state.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Trace fresh workflow replay phases #3797 — replay phase tracing
  3. [core] Overlap workflow compilation with replay loading #3798 — overlap Node bundle compilation
  4. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
Validation
  • focused v4 event suite — 46 tests passed
  • full @workflow/world-vercel suite passed at the stack tip — 572 tests
  • pnpm --filter @workflow/world-vercel typecheck
  • git diff --check
@changeset-bot
changeset-botBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e9d0c22
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-astro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-express-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-hono-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-python-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-vite-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-docsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-tarballsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-webReadyReadyPreview, v0Aug 28, 2026 9:29pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593 | 🔍 observability
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC | 🔍 observability
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A | 🔍 observability
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M154KVRDKTZR0KJW5AE7APMN
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0
vercel-multi-region (1 failed)
nextjs-turbopack (1 failed):
  • multi-region (world-vercel) hooks on non-iad1 runs hook created by a sfo1 run resolves by token and resumes the run
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
14 flaky tests
  • addTenWorkflow (tanstack-start) — flaked in 2 jobs
  • addTenWorkflow (fastify)
  • addTenWorkflow (vite)
  • cancelRun via CLI - cancelling a running workflow (astro)
  • cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
  • cancelRun via CLI - cancelling a running workflow (nuxt)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (nitro)
  • hookWorkflow (express)
  • hookWorkflow (hono)
  • hookWorkflow (nest)
  • hookWorkflow (vite)
  • promiseAllWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (nest)
  • stepWinsRaceWorkflow (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:28:37Z · abandoned wrun_41M1548R2Z0GG885SKHCNYMCDZ · (+5 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYM0GNGD2N01DRJP6TR
  • run-pickup-stall · sleepingWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYT0GP5T99M2JK2RNB3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYV0GXMSR1D3YJTJ610
  • run-pickup-stall · nullByteWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYZ0GG00VR3DZ0YNP0N
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:28:59Z · abandoned wrun_41M154CRS10GXV4254Y9GX493H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:29:34Z · abandoned wrun_41M154DT4Z0GM156G1JA91CMSJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFRY0GJEJ1Z0B94BVCQG
  • run-pickup-stall · nullByteWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFNB0GSXG82EEB0RQXDT
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EFTV0GQY9CA3YHZXKRHH
  • run-pickup-stall · sleepingWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EGAF0GJHMNKFRPRF44GY
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:30:22Z · abandoned wrun_01M154F0B7JB75RPZKHZ034NS7
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:30:45Z · abandoned wrun_41M154EXFW0GMM7P72ETDYBW6S
  • cold-start-warmup · suite warmup (python) · at 21:31:19Z · abandoned wrun_01M154DXAQ2WMDDXP5VZ5ZEACR · (+7 more)
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:31:31Z · abandoned wrun_41M154GCP60GWVR1RK1CJTWBQG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:31:34Z · abandoned wrun_01M154HJF8CHCNH305GH60575D
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJF9KX1602SFN1R769MG
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFG65FSMNG5KS7T8RYK
  • run-pickup-stall · sleepingWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFE6AFZ9RJVFSH4C697
  • run-pickup-stall · nullByteWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFJA17HEXHTG5HVYDWG
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:32:27Z · abandoned wrun_41M154HNCC0GH28PPPFDZAAJKZ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD39BMYM0PFJQ6F7CZXJ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:32:34Z · abandoned wrun_01M154KD31F5ND2T1R36J63R0S
  • run-pickup-stall · sleepingWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD37515KM9F5AXCPH0D9
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD339R9SBBJWEXZ9EHV1
  • run-pickup-stall · nullByteWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD3CQSQ19C8YMZ3RYDA9
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:14Z · abandoned wrun_41M154K4JR0GK30D8JPG4B2594
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PM9FAFWKYWTDJ2K72C
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:34Z · abandoned wrun_01M154N7Q0PB9SRVATNJNVRQSK
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PX7F4R77XDCK1DMH1P
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P534814GPCJK014PHNT6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P532HAGPS0W06MY4MPSQ
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:34:19Z · abandoned wrun_01M154PKKN2JBV7KWZ5JPHB62B
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:34:34Z · abandoned wrun_01M154Q2A3KDYJRE3BW5MC7R37
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
❌ vercel-multi-region261027
✅ vercel-ws-transport553087640
Total1705218277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
❌ vercel-multi-region
AppPassedFailedSkipped
❌ nextjs-turbopack2610
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e9d0c22 · Fri, 28 Aug 2026 21:51:49 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1275 (+88%) 🔻1368 🔴 (+21%) 🔻1402 🔴 (+23%) 🔻1658 🔴 (+44%) 🔻30
TTFSstream271 (+18%) 🔻1322 🔴 (+23%) 🔻1336 🔴 (+21%) 🔻1454 🔴 (+24%) 🔻30
TTFShook + stream1184 (+142%) 🔻1548 🔴 (+15%)1630 🔴 (+18%) 🔻2027 🔴 (+39%) 🔻30
Fan-out TTFSPromise.all(100 steps)613 (+8.9%)2049 (+20%) 🔻2051 (+13%)5964 (+204%) 🔻10
Fan-out TTLSPromise.all(100 steps)2045 (-6.9%)4006 (+8.0%)4238 (+13%)12428 (+58%) 🔻10
STSO1020 steps (inline)110 (-2.7%)142 (+1.4%)161 (±0%)218 (+11%)1019
WO1020 steps141217 (±0%)141217 (±0%)141217 (±0%)141217 (±0%)1
CRTTfirst chunk (pooled)97 (-16%) 💚142 (-39%) 💚156 (-69%) 💚380 (-43%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)127 (-11%)144 (-71%)199 (-72%)343 (-65%)120 (-74%)10
size sweep (100/s, 160B-12KB)118 (-34%)158 (-32%)232 (-59%)430 (-52%)149 (-37%)10
replay gateway-gpt-5.4-nano-2000t (1x)121 (-38%)142 (-39%)179 (-60%)438 (-55%)245 (-66%)3
replay eve-gpt-5.6-sol-2000t (1x)117 (-70%)132 (-77%)164 (-91%)335 (-92%)245 (-89%)2
replay eve-gpt-5.6-sol-2000t (2x)153 (-3%)173 (-76%)232 (-78%)430 (-67%)289 (-49%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 139837ms → this run 140733ms (Δ +896ms, +1%)
100-150 ms ███████████████████████┃ main 857 this 858 +1
150-200 ms ███┃ main 152 this 148 -4
200-250 ms ┃ main 8 this 7 -1
250-300 ms ┃ main 0 this 2 +2
300-350 ms ┃ main 2 this 2 +0
350-400 ms ┃ main 0 this 1 +1
450-500 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▄█▁···· 123.8 (-56%) 119 (-38%) 199 (-72%) 343 (-65%) 3000
sweep ······▃█▁···· 133.2 (-36%) 121 (-30%) 232 (-59%) 430 (-52%) 3000
gw 1x ·····▁▆█▁▁··· 116.4 (-43%) 108 (-34%) 179 (-60%) 438 (-55%) 5295
eve 1x ·····▁▆█▁···· 115.6 (-73%) 106 (-42%) 164 (-91%) 335 (-92%) 5186
eve 2x ·····▁▂█▂▁··· 149.1 (-61%) 136 (-45%) 232 (-78%) 430 (-67%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▆▆▄▆█▄▄▂▃▁ 114–135ms
sweep █▄▅▂▇▅▄▇▁▁ 117–151ms
gw 1x ▅▄▃▁▃▁▆▃▆█ 106–131ms
eve 1x ▅▄▇▃▂▃█▅▃▁ 95–140ms
eve 2x ▃▁▄▂▁▁▄█▃▂ 127–209ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▆▅▂▆█▆ 132–134ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▁▅▃▅▃█▄▃▇▄ 32–41ms
sweep ▁▄▆▅█▆██▅▃ 37–63ms
gw 1x ▅▆▄▁▆▁▆▆█▆ 29–38ms
eve 1x ▇▅▆▃▄▄█▂▅▁ 19–25ms
eve 2x █▄▂▁▆▂▅▂▁▄ 20–30ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@NathanColosimoNathanColosimo
changed the title
Resume partial replay event streams[world-vercel] Resume partial replay streams after validated eventsAug 14, 2026

This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

@NathanColosimo
NathanColosimoforce-pushed
the
codex/replay-stream-recovery
branch
from
e9e7488 to
dbdc9a2CompareAugust 14, 2026 03:56
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, will let agent approve if no blockers found
VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

Comment threadpackages/world-vercel/src/storage.ts
Outdated

vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actions
github-actionsBot
commented
Aug 25, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.4 KiB (±0)1.76 MiB (+827 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)764.3 KiB (+412 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
e9d0c22 · run
@NathanColosimoNathanColosimo
mentioned this pull request
Aug 25, 2026
 Merged

vercel[bot]
vercelBot
reviewed
Aug 26, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

vercel[bot]
vercelBot
reviewed
Aug 27, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

NathanColosimoand others
added 12 commits
August 28, 2026 14:24
@NathanColosimo
Classify replay stream failures by ownership
f7a0c30
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@NathanColosimo
fix(world-vercel): resume truncated replay streams
2653939
@NathanColosimo
fix(world-vercel): require replay continuation cursor
cf646d9
@NathanColosimo
refactor(world-vercel): simplify replay recovery
bd05dca
@NathanColosimo
fix(world-vercel): append replay pages safely
b429134
@NathanColosimo
fix(world-vercel): bound partial stream continuations
32cd767
@NathanColosimo
fix(world-vercel): bound replay continuations
edeaf78
@NathanColosimo
fix(world-vercel): retain recovered replay cursors
7b8d0ff
@NathanColosimo
fix(world-vercel): validate each recovery cursor
c3aae5c
@NathanColosimo
test(world-vercel): include list response cursors
83e1ac7
@NathanColosimo
refactor(world-vercel): simplify replay recovery
b3095a5
@NathanColosimo@vercel
Update packages/world-vercel/src/events-v4.ts
e9d0c22
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Aug 31, 2026
 Open

@github-actions
github-actionsBot
commented
Aug 31, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for aaeb8e4 (AI decision).
This change is built on the streamed replay-log machinery that only exists on main: origin/stable's packages/world-vercel/src/events-v4.ts has no createWorkflowRunStartedEventV4, no createHookReceivedPreloadEventV4, no consumeEventFrameStream/EventStreamEndSchema cursor+hasMore result, and no prior truncated-stream resume path, so the resume-and-retry loop it introduces has nothing to attach to there. On stable a truncated frame stream already fails loudly rather than silently dropping events, so what remains is resiliency work (multi-GET recovery, retry bounds, cursor-progress guard) plus new typed WorkflowWorldError classification of existing throws — enhancements, not a defect fix for the maintenance line. If the typed-error classification in decodeCreateEventResponse/events.ts is wanted on stable, it should be split out and forced through separately.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
aaeb8e44633c1974f087ee7ce38666dbc8ad8a4d


This was referenced Aug 31, 2026
 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@alangenfeldalangenfeldalangenfeld approved these changes
@VaguelySeriousVaguelySeriousAwaiting requested review from VaguelySerious

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@alangenfeld@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
[world-vercel] Classify incomplete replay streams as transport failures by NathanColosimo · Pull Request #3546 · vercel/workflow · GitHub
Skip to content
[world-vercel] Classify incomplete replay streams as transport failures - #3546
Merged
NathanColosimo merged 12 commits into
mainfrom 
codex/replay-stream-recovery
Aug 31, 2026
Merged
[world-vercel] Classify incomplete replay streams as transport failures#3546
NathanColosimo merged 12 commits into
mainfrom 
codex/replay-stream-recovery
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • require the replay sentinel before treating a framed response as complete
  • preserve validated prefix events and resume from the last event cursor until the sentinel arrives
  • recover truncated run_started, atomic-hook preload, and unbounded events.list responses through the same direct GET loop
  • allow an initial GET plus two retries per GET chain; a partial setup POST can therefore issue at most three suffix GETs
  • validate cursor progress on every response before accepting its events
  • keep explicitly paginated requests one-page operations
  • retry only transport truncation, never schema or observer failures
Once a valid cursor exists, recovery keeps the accepted prefix and fetches only the missing suffix. The implementation is one direct do … while loop with only the current cursor and retry count as recovery state.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Trace fresh workflow replay phases #3797 — replay phase tracing
  3. [core] Overlap workflow compilation with replay loading #3798 — overlap Node bundle compilation
  4. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
Validation
  • focused v4 event suite — 46 tests passed
  • full @workflow/world-vercel suite passed at the stack tip — 572 tests
  • pnpm --filter @workflow/world-vercel typecheck
  • git diff --check
@changeset-bot
changeset-botBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e9d0c22
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-astro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-express-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-hono-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-python-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-vite-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-docsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-tarballsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-webReadyReadyPreview, v0Aug 28, 2026 9:29pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593 | 🔍 observability
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC | 🔍 observability
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A | 🔍 observability
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M154KVRDKTZR0KJW5AE7APMN
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0
vercel-multi-region (1 failed)
nextjs-turbopack (1 failed):
  • multi-region (world-vercel) hooks on non-iad1 runs hook created by a sfo1 run resolves by token and resumes the run
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
14 flaky tests
  • addTenWorkflow (tanstack-start) — flaked in 2 jobs
  • addTenWorkflow (fastify)
  • addTenWorkflow (vite)
  • cancelRun via CLI - cancelling a running workflow (astro)
  • cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
  • cancelRun via CLI - cancelling a running workflow (nuxt)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (nitro)
  • hookWorkflow (express)
  • hookWorkflow (hono)
  • hookWorkflow (nest)
  • hookWorkflow (vite)
  • promiseAllWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (nest)
  • stepWinsRaceWorkflow (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:28:37Z · abandoned wrun_41M1548R2Z0GG885SKHCNYMCDZ · (+5 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYM0GNGD2N01DRJP6TR
  • run-pickup-stall · sleepingWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYT0GP5T99M2JK2RNB3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYV0GXMSR1D3YJTJ610
  • run-pickup-stall · nullByteWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYZ0GG00VR3DZ0YNP0N
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:28:59Z · abandoned wrun_41M154CRS10GXV4254Y9GX493H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:29:34Z · abandoned wrun_41M154DT4Z0GM156G1JA91CMSJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFRY0GJEJ1Z0B94BVCQG
  • run-pickup-stall · nullByteWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFNB0GSXG82EEB0RQXDT
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EFTV0GQY9CA3YHZXKRHH
  • run-pickup-stall · sleepingWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EGAF0GJHMNKFRPRF44GY
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:30:22Z · abandoned wrun_01M154F0B7JB75RPZKHZ034NS7
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:30:45Z · abandoned wrun_41M154EXFW0GMM7P72ETDYBW6S
  • cold-start-warmup · suite warmup (python) · at 21:31:19Z · abandoned wrun_01M154DXAQ2WMDDXP5VZ5ZEACR · (+7 more)
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:31:31Z · abandoned wrun_41M154GCP60GWVR1RK1CJTWBQG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:31:34Z · abandoned wrun_01M154HJF8CHCNH305GH60575D
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJF9KX1602SFN1R769MG
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFG65FSMNG5KS7T8RYK
  • run-pickup-stall · sleepingWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFE6AFZ9RJVFSH4C697
  • run-pickup-stall · nullByteWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFJA17HEXHTG5HVYDWG
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:32:27Z · abandoned wrun_41M154HNCC0GH28PPPFDZAAJKZ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD39BMYM0PFJQ6F7CZXJ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:32:34Z · abandoned wrun_01M154KD31F5ND2T1R36J63R0S
  • run-pickup-stall · sleepingWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD37515KM9F5AXCPH0D9
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD339R9SBBJWEXZ9EHV1
  • run-pickup-stall · nullByteWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD3CQSQ19C8YMZ3RYDA9
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:14Z · abandoned wrun_41M154K4JR0GK30D8JPG4B2594
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PM9FAFWKYWTDJ2K72C
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:34Z · abandoned wrun_01M154N7Q0PB9SRVATNJNVRQSK
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PX7F4R77XDCK1DMH1P
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P534814GPCJK014PHNT6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P532HAGPS0W06MY4MPSQ
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:34:19Z · abandoned wrun_01M154PKKN2JBV7KWZ5JPHB62B
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:34:34Z · abandoned wrun_01M154Q2A3KDYJRE3BW5MC7R37
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
❌ vercel-multi-region261027
✅ vercel-ws-transport553087640
Total1705218277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
❌ vercel-multi-region
AppPassedFailedSkipped
❌ nextjs-turbopack2610
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e9d0c22 · Fri, 28 Aug 2026 21:51:49 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1275 (+88%) 🔻1368 🔴 (+21%) 🔻1402 🔴 (+23%) 🔻1658 🔴 (+44%) 🔻30
TTFSstream271 (+18%) 🔻1322 🔴 (+23%) 🔻1336 🔴 (+21%) 🔻1454 🔴 (+24%) 🔻30
TTFShook + stream1184 (+142%) 🔻1548 🔴 (+15%)1630 🔴 (+18%) 🔻2027 🔴 (+39%) 🔻30
Fan-out TTFSPromise.all(100 steps)613 (+8.9%)2049 (+20%) 🔻2051 (+13%)5964 (+204%) 🔻10
Fan-out TTLSPromise.all(100 steps)2045 (-6.9%)4006 (+8.0%)4238 (+13%)12428 (+58%) 🔻10
STSO1020 steps (inline)110 (-2.7%)142 (+1.4%)161 (±0%)218 (+11%)1019
WO1020 steps141217 (±0%)141217 (±0%)141217 (±0%)141217 (±0%)1
CRTTfirst chunk (pooled)97 (-16%) 💚142 (-39%) 💚156 (-69%) 💚380 (-43%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)127 (-11%)144 (-71%)199 (-72%)343 (-65%)120 (-74%)10
size sweep (100/s, 160B-12KB)118 (-34%)158 (-32%)232 (-59%)430 (-52%)149 (-37%)10
replay gateway-gpt-5.4-nano-2000t (1x)121 (-38%)142 (-39%)179 (-60%)438 (-55%)245 (-66%)3
replay eve-gpt-5.6-sol-2000t (1x)117 (-70%)132 (-77%)164 (-91%)335 (-92%)245 (-89%)2
replay eve-gpt-5.6-sol-2000t (2x)153 (-3%)173 (-76%)232 (-78%)430 (-67%)289 (-49%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 139837ms → this run 140733ms (Δ +896ms, +1%)
100-150 ms ███████████████████████┃ main 857 this 858 +1
150-200 ms ███┃ main 152 this 148 -4
200-250 ms ┃ main 8 this 7 -1
250-300 ms ┃ main 0 this 2 +2
300-350 ms ┃ main 2 this 2 +0
350-400 ms ┃ main 0 this 1 +1
450-500 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▄█▁···· 123.8 (-56%) 119 (-38%) 199 (-72%) 343 (-65%) 3000
sweep ······▃█▁···· 133.2 (-36%) 121 (-30%) 232 (-59%) 430 (-52%) 3000
gw 1x ·····▁▆█▁▁··· 116.4 (-43%) 108 (-34%) 179 (-60%) 438 (-55%) 5295
eve 1x ·····▁▆█▁···· 115.6 (-73%) 106 (-42%) 164 (-91%) 335 (-92%) 5186
eve 2x ·····▁▂█▂▁··· 149.1 (-61%) 136 (-45%) 232 (-78%) 430 (-67%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▆▆▄▆█▄▄▂▃▁ 114–135ms
sweep █▄▅▂▇▅▄▇▁▁ 117–151ms
gw 1x ▅▄▃▁▃▁▆▃▆█ 106–131ms
eve 1x ▅▄▇▃▂▃█▅▃▁ 95–140ms
eve 2x ▃▁▄▂▁▁▄█▃▂ 127–209ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▆▅▂▆█▆ 132–134ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▁▅▃▅▃█▄▃▇▄ 32–41ms
sweep ▁▄▆▅█▆██▅▃ 37–63ms
gw 1x ▅▆▄▁▆▁▆▆█▆ 29–38ms
eve 1x ▇▅▆▃▄▄█▂▅▁ 19–25ms
eve 2x █▄▂▁▆▂▅▂▁▄ 20–30ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@NathanColosimoNathanColosimo
changed the title
Resume partial replay event streams[world-vercel] Resume partial replay streams after validated eventsAug 14, 2026

This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

@NathanColosimo
NathanColosimoforce-pushed
the
codex/replay-stream-recovery
branch
from
e9e7488 to
dbdc9a2CompareAugust 14, 2026 03:56
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, will let agent approve if no blockers found
VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

Comment threadpackages/world-vercel/src/storage.ts
Outdated

vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actions
github-actionsBot
commented
Aug 25, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.4 KiB (±0)1.76 MiB (+827 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)764.3 KiB (+412 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
e9d0c22 · run
@NathanColosimoNathanColosimo
mentioned this pull request
Aug 25, 2026
 Merged

vercel[bot]
vercelBot
reviewed
Aug 26, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

vercel[bot]
vercelBot
reviewed
Aug 27, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

NathanColosimoand others
added 12 commits
August 28, 2026 14:24
@NathanColosimo
Classify replay stream failures by ownership
f7a0c30
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@NathanColosimo
fix(world-vercel): resume truncated replay streams
2653939
@NathanColosimo
fix(world-vercel): require replay continuation cursor
cf646d9
@NathanColosimo
refactor(world-vercel): simplify replay recovery
bd05dca
@NathanColosimo
fix(world-vercel): append replay pages safely
b429134
@NathanColosimo
fix(world-vercel): bound partial stream continuations
32cd767
@NathanColosimo
fix(world-vercel): bound replay continuations
edeaf78
@NathanColosimo
fix(world-vercel): retain recovered replay cursors
7b8d0ff
@NathanColosimo
fix(world-vercel): validate each recovery cursor
c3aae5c
@NathanColosimo
test(world-vercel): include list response cursors
83e1ac7
@NathanColosimo
refactor(world-vercel): simplify replay recovery
b3095a5
@NathanColosimo@vercel
Update packages/world-vercel/src/events-v4.ts
e9d0c22
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Aug 31, 2026
 Open

@github-actions
github-actionsBot
commented
Aug 31, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for aaeb8e4 (AI decision).
This change is built on the streamed replay-log machinery that only exists on main: origin/stable's packages/world-vercel/src/events-v4.ts has no createWorkflowRunStartedEventV4, no createHookReceivedPreloadEventV4, no consumeEventFrameStream/EventStreamEndSchema cursor+hasMore result, and no prior truncated-stream resume path, so the resume-and-retry loop it introduces has nothing to attach to there. On stable a truncated frame stream already fails loudly rather than silently dropping events, so what remains is resiliency work (multi-GET recovery, retry bounds, cursor-progress guard) plus new typed WorkflowWorldError classification of existing throws — enhancements, not a defect fix for the maintenance line. If the typed-error classification in decodeCreateEventResponse/events.ts is wanted on stable, it should be split out and forced through separately.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
aaeb8e44633c1974f087ee7ce38666dbc8ad8a4d


This was referenced Aug 31, 2026
 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@alangenfeldalangenfeldalangenfeld approved these changes
@VaguelySeriousVaguelySeriousAwaiting requested review from VaguelySerious

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@alangenfeld@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
[world-vercel] Classify incomplete replay streams as transport failures by NathanColosimo · Pull Request #3546 · vercel/workflow · GitHub
Skip to content
[world-vercel] Classify incomplete replay streams as transport failures - #3546
Merged
NathanColosimo merged 12 commits into
mainfrom 
codex/replay-stream-recovery
Aug 31, 2026
Merged
[world-vercel] Classify incomplete replay streams as transport failures#3546
NathanColosimo merged 12 commits into
mainfrom 
codex/replay-stream-recovery
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • require the replay sentinel before treating a framed response as complete
  • preserve validated prefix events and resume from the last event cursor until the sentinel arrives
  • recover truncated run_started, atomic-hook preload, and unbounded events.list responses through the same direct GET loop
  • allow an initial GET plus two retries per GET chain; a partial setup POST can therefore issue at most three suffix GETs
  • validate cursor progress on every response before accepting its events
  • keep explicitly paginated requests one-page operations
  • retry only transport truncation, never schema or observer failures
Once a valid cursor exists, recovery keeps the accepted prefix and fetches only the missing suffix. The implementation is one direct do … while loop with only the current cursor and retry count as recovery state.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Trace fresh workflow replay phases #3797 — replay phase tracing
  3. [core] Overlap workflow compilation with replay loading #3798 — overlap Node bundle compilation
  4. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
Validation
  • focused v4 event suite — 46 tests passed
  • full @workflow/world-vercel suite passed at the stack tip — 572 tests
  • pnpm --filter @workflow/world-vercel typecheck
  • git diff --check
@changeset-bot
changeset-botBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e9d0c22
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-astro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-express-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-hono-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-python-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-vite-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-docsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-tarballsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-webReadyReadyPreview, v0Aug 28, 2026 9:29pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593 | 🔍 observability
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC | 🔍 observability
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A | 🔍 observability
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M154KVRDKTZR0KJW5AE7APMN
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0
vercel-multi-region (1 failed)
nextjs-turbopack (1 failed):
  • multi-region (world-vercel) hooks on non-iad1 runs hook created by a sfo1 run resolves by token and resumes the run
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
14 flaky tests
  • addTenWorkflow (tanstack-start) — flaked in 2 jobs
  • addTenWorkflow (fastify)
  • addTenWorkflow (vite)
  • cancelRun via CLI - cancelling a running workflow (astro)
  • cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
  • cancelRun via CLI - cancelling a running workflow (nuxt)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (nitro)
  • hookWorkflow (express)
  • hookWorkflow (hono)
  • hookWorkflow (nest)
  • hookWorkflow (vite)
  • promiseAllWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (nest)
  • stepWinsRaceWorkflow (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:28:37Z · abandoned wrun_41M1548R2Z0GG885SKHCNYMCDZ · (+5 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYM0GNGD2N01DRJP6TR
  • run-pickup-stall · sleepingWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYT0GP5T99M2JK2RNB3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYV0GXMSR1D3YJTJ610
  • run-pickup-stall · nullByteWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYZ0GG00VR3DZ0YNP0N
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:28:59Z · abandoned wrun_41M154CRS10GXV4254Y9GX493H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:29:34Z · abandoned wrun_41M154DT4Z0GM156G1JA91CMSJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFRY0GJEJ1Z0B94BVCQG
  • run-pickup-stall · nullByteWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFNB0GSXG82EEB0RQXDT
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EFTV0GQY9CA3YHZXKRHH
  • run-pickup-stall · sleepingWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EGAF0GJHMNKFRPRF44GY
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:30:22Z · abandoned wrun_01M154F0B7JB75RPZKHZ034NS7
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:30:45Z · abandoned wrun_41M154EXFW0GMM7P72ETDYBW6S
  • cold-start-warmup · suite warmup (python) · at 21:31:19Z · abandoned wrun_01M154DXAQ2WMDDXP5VZ5ZEACR · (+7 more)
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:31:31Z · abandoned wrun_41M154GCP60GWVR1RK1CJTWBQG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:31:34Z · abandoned wrun_01M154HJF8CHCNH305GH60575D
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJF9KX1602SFN1R769MG
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFG65FSMNG5KS7T8RYK
  • run-pickup-stall · sleepingWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFE6AFZ9RJVFSH4C697
  • run-pickup-stall · nullByteWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFJA17HEXHTG5HVYDWG
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:32:27Z · abandoned wrun_41M154HNCC0GH28PPPFDZAAJKZ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD39BMYM0PFJQ6F7CZXJ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:32:34Z · abandoned wrun_01M154KD31F5ND2T1R36J63R0S
  • run-pickup-stall · sleepingWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD37515KM9F5AXCPH0D9
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD339R9SBBJWEXZ9EHV1
  • run-pickup-stall · nullByteWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD3CQSQ19C8YMZ3RYDA9
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:14Z · abandoned wrun_41M154K4JR0GK30D8JPG4B2594
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PM9FAFWKYWTDJ2K72C
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:34Z · abandoned wrun_01M154N7Q0PB9SRVATNJNVRQSK
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PX7F4R77XDCK1DMH1P
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P534814GPCJK014PHNT6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P532HAGPS0W06MY4MPSQ
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:34:19Z · abandoned wrun_01M154PKKN2JBV7KWZ5JPHB62B
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:34:34Z · abandoned wrun_01M154Q2A3KDYJRE3BW5MC7R37
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
❌ vercel-multi-region261027
✅ vercel-ws-transport553087640
Total1705218277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
❌ vercel-multi-region
AppPassedFailedSkipped
❌ nextjs-turbopack2610
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e9d0c22 · Fri, 28 Aug 2026 21:51:49 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1275 (+88%) 🔻1368 🔴 (+21%) 🔻1402 🔴 (+23%) 🔻1658 🔴 (+44%) 🔻30
TTFSstream271 (+18%) 🔻1322 🔴 (+23%) 🔻1336 🔴 (+21%) 🔻1454 🔴 (+24%) 🔻30
TTFShook + stream1184 (+142%) 🔻1548 🔴 (+15%)1630 🔴 (+18%) 🔻2027 🔴 (+39%) 🔻30
Fan-out TTFSPromise.all(100 steps)613 (+8.9%)2049 (+20%) 🔻2051 (+13%)5964 (+204%) 🔻10
Fan-out TTLSPromise.all(100 steps)2045 (-6.9%)4006 (+8.0%)4238 (+13%)12428 (+58%) 🔻10
STSO1020 steps (inline)110 (-2.7%)142 (+1.4%)161 (±0%)218 (+11%)1019
WO1020 steps141217 (±0%)141217 (±0%)141217 (±0%)141217 (±0%)1
CRTTfirst chunk (pooled)97 (-16%) 💚142 (-39%) 💚156 (-69%) 💚380 (-43%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)127 (-11%)144 (-71%)199 (-72%)343 (-65%)120 (-74%)10
size sweep (100/s, 160B-12KB)118 (-34%)158 (-32%)232 (-59%)430 (-52%)149 (-37%)10
replay gateway-gpt-5.4-nano-2000t (1x)121 (-38%)142 (-39%)179 (-60%)438 (-55%)245 (-66%)3
replay eve-gpt-5.6-sol-2000t (1x)117 (-70%)132 (-77%)164 (-91%)335 (-92%)245 (-89%)2
replay eve-gpt-5.6-sol-2000t (2x)153 (-3%)173 (-76%)232 (-78%)430 (-67%)289 (-49%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 139837ms → this run 140733ms (Δ +896ms, +1%)
100-150 ms ███████████████████████┃ main 857 this 858 +1
150-200 ms ███┃ main 152 this 148 -4
200-250 ms ┃ main 8 this 7 -1
250-300 ms ┃ main 0 this 2 +2
300-350 ms ┃ main 2 this 2 +0
350-400 ms ┃ main 0 this 1 +1
450-500 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▄█▁···· 123.8 (-56%) 119 (-38%) 199 (-72%) 343 (-65%) 3000
sweep ······▃█▁···· 133.2 (-36%) 121 (-30%) 232 (-59%) 430 (-52%) 3000
gw 1x ·····▁▆█▁▁··· 116.4 (-43%) 108 (-34%) 179 (-60%) 438 (-55%) 5295
eve 1x ·····▁▆█▁···· 115.6 (-73%) 106 (-42%) 164 (-91%) 335 (-92%) 5186
eve 2x ·····▁▂█▂▁··· 149.1 (-61%) 136 (-45%) 232 (-78%) 430 (-67%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▆▆▄▆█▄▄▂▃▁ 114–135ms
sweep █▄▅▂▇▅▄▇▁▁ 117–151ms
gw 1x ▅▄▃▁▃▁▆▃▆█ 106–131ms
eve 1x ▅▄▇▃▂▃█▅▃▁ 95–140ms
eve 2x ▃▁▄▂▁▁▄█▃▂ 127–209ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▆▅▂▆█▆ 132–134ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▁▅▃▅▃█▄▃▇▄ 32–41ms
sweep ▁▄▆▅█▆██▅▃ 37–63ms
gw 1x ▅▆▄▁▆▁▆▆█▆ 29–38ms
eve 1x ▇▅▆▃▄▄█▂▅▁ 19–25ms
eve 2x █▄▂▁▆▂▅▂▁▄ 20–30ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@NathanColosimoNathanColosimo
changed the title
Resume partial replay event streams[world-vercel] Resume partial replay streams after validated eventsAug 14, 2026

This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

@NathanColosimo
NathanColosimoforce-pushed
the
codex/replay-stream-recovery
branch
from
e9e7488 to
dbdc9a2CompareAugust 14, 2026 03:56
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, will let agent approve if no blockers found
VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

Comment threadpackages/world-vercel/src/storage.ts
Outdated

vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actions
github-actionsBot
commented
Aug 25, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.4 KiB (±0)1.76 MiB (+827 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)764.3 KiB (+412 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
e9d0c22 · run
@NathanColosimoNathanColosimo
mentioned this pull request
Aug 25, 2026
 Merged

vercel[bot]
vercelBot
reviewed
Aug 26, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

vercel[bot]
vercelBot
reviewed
Aug 27, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

NathanColosimoand others
added 12 commits
August 28, 2026 14:24
@NathanColosimo
Classify replay stream failures by ownership
f7a0c30
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@NathanColosimo
fix(world-vercel): resume truncated replay streams
2653939
@NathanColosimo
fix(world-vercel): require replay continuation cursor
cf646d9
@NathanColosimo
refactor(world-vercel): simplify replay recovery
bd05dca
@NathanColosimo
fix(world-vercel): append replay pages safely
b429134
@NathanColosimo
fix(world-vercel): bound partial stream continuations
32cd767
@NathanColosimo
fix(world-vercel): bound replay continuations
edeaf78
@NathanColosimo
fix(world-vercel): retain recovered replay cursors
7b8d0ff
@NathanColosimo
fix(world-vercel): validate each recovery cursor
c3aae5c
@NathanColosimo
test(world-vercel): include list response cursors
83e1ac7
@NathanColosimo
refactor(world-vercel): simplify replay recovery
b3095a5
@NathanColosimo@vercel
Update packages/world-vercel/src/events-v4.ts
e9d0c22
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Aug 31, 2026
 Open

@github-actions
github-actionsBot
commented
Aug 31, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for aaeb8e4 (AI decision).
This change is built on the streamed replay-log machinery that only exists on main: origin/stable's packages/world-vercel/src/events-v4.ts has no createWorkflowRunStartedEventV4, no createHookReceivedPreloadEventV4, no consumeEventFrameStream/EventStreamEndSchema cursor+hasMore result, and no prior truncated-stream resume path, so the resume-and-retry loop it introduces has nothing to attach to there. On stable a truncated frame stream already fails loudly rather than silently dropping events, so what remains is resiliency work (multi-GET recovery, retry bounds, cursor-progress guard) plus new typed WorkflowWorldError classification of existing throws — enhancements, not a defect fix for the maintenance line. If the typed-error classification in decodeCreateEventResponse/events.ts is wanted on stable, it should be split out and forced through separately.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
aaeb8e44633c1974f087ee7ce38666dbc8ad8a4d


This was referenced Aug 31, 2026
 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@alangenfeldalangenfeldalangenfeld approved these changes
@VaguelySeriousVaguelySeriousAwaiting requested review from VaguelySerious

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@alangenfeld@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
})();
[world-vercel] Classify incomplete replay streams as transport failures by NathanColosimo · Pull Request #3546 · vercel/workflow · GitHub
Skip to content
[world-vercel] Classify incomplete replay streams as transport failures - #3546
Merged
NathanColosimo merged 12 commits into
mainfrom 
codex/replay-stream-recovery
Aug 31, 2026
Merged
[world-vercel] Classify incomplete replay streams as transport failures#3546
NathanColosimo merged 12 commits into
mainfrom 
codex/replay-stream-recovery
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • require the replay sentinel before treating a framed response as complete
  • preserve validated prefix events and resume from the last event cursor until the sentinel arrives
  • recover truncated run_started, atomic-hook preload, and unbounded events.list responses through the same direct GET loop
  • allow an initial GET plus two retries per GET chain; a partial setup POST can therefore issue at most three suffix GETs
  • validate cursor progress on every response before accepting its events
  • keep explicitly paginated requests one-page operations
  • retry only transport truncation, never schema or observer failures
Once a valid cursor exists, recovery keeps the accepted prefix and fetches only the missing suffix. The implementation is one direct do … while loop with only the current cursor and retry count as recovery state.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Trace fresh workflow replay phases #3797 — replay phase tracing
  3. [core] Overlap workflow compilation with replay loading #3798 — overlap Node bundle compilation
  4. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
Validation
  • focused v4 event suite — 46 tests passed
  • full @workflow/world-vercel suite passed at the stack tip — 572 tests
  • pnpm --filter @workflow/world-vercel typecheck
  • git diff --check
@changeset-bot
changeset-botBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e9d0c22
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 28, 2026 9:29pm
example-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-astro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-express-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-hono-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-python-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workbench-vite-workflowReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-docsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-tarballsReadyReadyPreview, v0Aug 28, 2026 9:29pm
workflow-webReadyReadyPreview, v0Aug 28, 2026 9:29pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593 | 🔍 observability
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC | 🔍 observability
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A | 🔍 observability
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M154KVRDKTZR0KJW5AE7APMN
  • promiseAllWorkflow | wrun_41M1549RRB0GNGSV5HJV2YS593
  • sleepingWorkflow | wrun_41M154BH630GZYBDXMHX7JHYBC
  • parallelSleepWorkflow | wrun_41M154BRTY0GTE5FFD7QY1HA3A
  • nullByteWorkflow | wrun_41M154C27Q0GPHR7RAHQWX9HMJ
  • cancelRun - cancelling a running workflow | wrun_41M154H4BT0GTPVDAS0TNR30TB
  • cancelRun via CLI - cancelling a running workflow | wrun_41M154H7060GRQ7HJ4FXR93QP5
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M154HD3D0GVDGMVDXG6VWM5T
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M154HZG10GX1D13HEYFZ33A0
vercel-multi-region (1 failed)
nextjs-turbopack (1 failed):
  • multi-region (world-vercel) hooks on non-iad1 runs hook created by a sfo1 run resolves by token and resumes the run
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
14 flaky tests
  • addTenWorkflow (tanstack-start) — flaked in 2 jobs
  • addTenWorkflow (fastify)
  • addTenWorkflow (vite)
  • cancelRun via CLI - cancelling a running workflow (astro)
  • cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
  • cancelRun via CLI - cancelling a running workflow (nuxt)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (nitro)
  • hookWorkflow (express)
  • hookWorkflow (hono)
  • hookWorkflow (nest)
  • hookWorkflow (vite)
  • promiseAllWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (nest)
  • stepWinsRaceWorkflow (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:28:37Z · abandoned wrun_41M1548R2Z0GG885SKHCNYMCDZ · (+5 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYM0GNGD2N01DRJP6TR
  • run-pickup-stall · sleepingWorkflow (python) · at 21:28:57Z · abandoned wrun_41M154CKYT0GP5T99M2JK2RNB3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYV0GXMSR1D3YJTJ610
  • run-pickup-stall · nullByteWorkflow (python) · at 21:28:58Z · abandoned wrun_41M154CKYZ0GG00VR3DZ0YNP0N
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:28:59Z · abandoned wrun_41M154CRS10GXV4254Y9GX493H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:29:34Z · abandoned wrun_41M154DT4Z0GM156G1JA91CMSJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFRY0GJEJ1Z0B94BVCQG
  • run-pickup-stall · nullByteWorkflow (python) · at 21:29:54Z · abandoned wrun_41M154EFNB0GSXG82EEB0RQXDT
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EFTV0GQY9CA3YHZXKRHH
  • run-pickup-stall · sleepingWorkflow (python) · at 21:29:55Z · abandoned wrun_41M154EGAF0GJHMNKFRPRF44GY
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:30:22Z · abandoned wrun_01M154F0B7JB75RPZKHZ034NS7
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:30:45Z · abandoned wrun_41M154EXFW0GMM7P72ETDYBW6S
  • cold-start-warmup · suite warmup (python) · at 21:31:19Z · abandoned wrun_01M154DXAQ2WMDDXP5VZ5ZEACR · (+7 more)
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:31:31Z · abandoned wrun_41M154GCP60GWVR1RK1CJTWBQG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:31:34Z · abandoned wrun_01M154HJF8CHCNH305GH60575D
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJF9KX1602SFN1R769MG
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFG65FSMNG5KS7T8RYK
  • run-pickup-stall · sleepingWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFE6AFZ9RJVFSH4C697
  • run-pickup-stall · nullByteWorkflow (python) · at 21:31:34Z · abandoned wrun_01M154HJFJA17HEXHTG5HVYDWG
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:32:27Z · abandoned wrun_41M154HNCC0GH28PPPFDZAAJKZ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD39BMYM0PFJQ6F7CZXJ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:32:34Z · abandoned wrun_01M154KD31F5ND2T1R36J63R0S
  • run-pickup-stall · sleepingWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD37515KM9F5AXCPH0D9
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD339R9SBBJWEXZ9EHV1
  • run-pickup-stall · nullByteWorkflow (python) · at 21:32:34Z · abandoned wrun_01M154KD3CQSQ19C8YMZ3RYDA9
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:14Z · abandoned wrun_41M154K4JR0GK30D8JPG4B2594
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PM9FAFWKYWTDJ2K72C
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:33:34Z · abandoned wrun_01M154N7Q0PB9SRVATNJNVRQSK
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:33:34Z · abandoned wrun_01M154N7PX7F4R77XDCK1DMH1P
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P534814GPCJK014PHNT6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:34:04Z · abandoned wrun_01M154P532HAGPS0W06MY4MPSQ
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:34:19Z · abandoned wrun_01M154PKKN2JBV7KWZ5JPHB62B
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:34:34Z · abandoned wrun_01M154Q2A3KDYJRE3BW5MC7R37
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
❌ vercel-multi-region261027
✅ vercel-ws-transport553087640
Total1705218277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
❌ vercel-multi-region
AppPassedFailedSkipped
❌ nextjs-turbopack2610
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e9d0c22 · Fri, 28 Aug 2026 21:51:49 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1275 (+88%) 🔻1368 🔴 (+21%) 🔻1402 🔴 (+23%) 🔻1658 🔴 (+44%) 🔻30
TTFSstream271 (+18%) 🔻1322 🔴 (+23%) 🔻1336 🔴 (+21%) 🔻1454 🔴 (+24%) 🔻30
TTFShook + stream1184 (+142%) 🔻1548 🔴 (+15%)1630 🔴 (+18%) 🔻2027 🔴 (+39%) 🔻30
Fan-out TTFSPromise.all(100 steps)613 (+8.9%)2049 (+20%) 🔻2051 (+13%)5964 (+204%) 🔻10
Fan-out TTLSPromise.all(100 steps)2045 (-6.9%)4006 (+8.0%)4238 (+13%)12428 (+58%) 🔻10
STSO1020 steps (inline)110 (-2.7%)142 (+1.4%)161 (±0%)218 (+11%)1019
WO1020 steps141217 (±0%)141217 (±0%)141217 (±0%)141217 (±0%)1
CRTTfirst chunk (pooled)97 (-16%) 💚142 (-39%) 💚156 (-69%) 💚380 (-43%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)127 (-11%)144 (-71%)199 (-72%)343 (-65%)120 (-74%)10
size sweep (100/s, 160B-12KB)118 (-34%)158 (-32%)232 (-59%)430 (-52%)149 (-37%)10
replay gateway-gpt-5.4-nano-2000t (1x)121 (-38%)142 (-39%)179 (-60%)438 (-55%)245 (-66%)3
replay eve-gpt-5.6-sol-2000t (1x)117 (-70%)132 (-77%)164 (-91%)335 (-92%)245 (-89%)2
replay eve-gpt-5.6-sol-2000t (2x)153 (-3%)173 (-76%)232 (-78%)430 (-67%)289 (-49%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 139837ms → this run 140733ms (Δ +896ms, +1%)
100-150 ms ███████████████████████┃ main 857 this 858 +1
150-200 ms ███┃ main 152 this 148 -4
200-250 ms ┃ main 8 this 7 -1
250-300 ms ┃ main 0 this 2 +2
300-350 ms ┃ main 2 this 2 +0
350-400 ms ┃ main 0 this 1 +1
450-500 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▄█▁···· 123.8 (-56%) 119 (-38%) 199 (-72%) 343 (-65%) 3000
sweep ······▃█▁···· 133.2 (-36%) 121 (-30%) 232 (-59%) 430 (-52%) 3000
gw 1x ·····▁▆█▁▁··· 116.4 (-43%) 108 (-34%) 179 (-60%) 438 (-55%) 5295
eve 1x ·····▁▆█▁···· 115.6 (-73%) 106 (-42%) 164 (-91%) 335 (-92%) 5186
eve 2x ·····▁▂█▂▁··· 149.1 (-61%) 136 (-45%) 232 (-78%) 430 (-67%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▆▆▄▆█▄▄▂▃▁ 114–135ms
sweep █▄▅▂▇▅▄▇▁▁ 117–151ms
gw 1x ▅▄▃▁▃▁▆▃▆█ 106–131ms
eve 1x ▅▄▇▃▂▃█▅▃▁ 95–140ms
eve 2x ▃▁▄▂▁▁▄█▃▂ 127–209ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▆▅▂▆█▆ 132–134ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▁▅▃▅▃█▄▃▇▄ 32–41ms
sweep ▁▄▆▅█▆██▅▃ 37–63ms
gw 1x ▅▆▄▁▆▁▆▆█▆ 29–38ms
eve 1x ▇▅▆▃▄▄█▂▅▁ 19–25ms
eve 2x █▄▂▁▆▂▅▂▁▄ 20–30ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@NathanColosimoNathanColosimo
changed the title
Resume partial replay event streams[world-vercel] Resume partial replay streams after validated eventsAug 14, 2026

This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

@NathanColosimo
NathanColosimoforce-pushed
the
codex/replay-stream-recovery
branch
from
e9e7488 to
dbdc9a2CompareAugust 14, 2026 03:56
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, will let agent approve if no blockers found
VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

Comment threadpackages/world-vercel/src/storage.ts
Outdated

vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actions
github-actionsBot
commented
Aug 25, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.4 KiB (±0)1.76 MiB (+827 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)764.3 KiB (+412 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
e9d0c22 · run
@NathanColosimoNathanColosimo
mentioned this pull request
Aug 25, 2026
 Merged

vercel[bot]
vercelBot
reviewed
Aug 26, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

vercel[bot]
vercelBot
reviewed
Aug 27, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

NathanColosimoand others
added 12 commits
August 28, 2026 14:24
@NathanColosimo
Classify replay stream failures by ownership
f7a0c30
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@NathanColosimo
fix(world-vercel): resume truncated replay streams
2653939
@NathanColosimo
fix(world-vercel): require replay continuation cursor
cf646d9
@NathanColosimo
refactor(world-vercel): simplify replay recovery
bd05dca
@NathanColosimo
fix(world-vercel): append replay pages safely
b429134
@NathanColosimo
fix(world-vercel): bound partial stream continuations
32cd767
@NathanColosimo
fix(world-vercel): bound replay continuations
edeaf78
@NathanColosimo
fix(world-vercel): retain recovered replay cursors
7b8d0ff
@NathanColosimo
fix(world-vercel): validate each recovery cursor
c3aae5c
@NathanColosimo
test(world-vercel): include list response cursors
83e1ac7
@NathanColosimo
refactor(world-vercel): simplify replay recovery
b3095a5
@NathanColosimo@vercel
Update packages/world-vercel/src/events-v4.ts
e9d0c22
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Aug 31, 2026
 Open

@github-actions
github-actionsBot
commented
Aug 31, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for aaeb8e4 (AI decision).
This change is built on the streamed replay-log machinery that only exists on main: origin/stable's packages/world-vercel/src/events-v4.ts has no createWorkflowRunStartedEventV4, no createHookReceivedPreloadEventV4, no consumeEventFrameStream/EventStreamEndSchema cursor+hasMore result, and no prior truncated-stream resume path, so the resume-and-retry loop it introduces has nothing to attach to there. On stable a truncated frame stream already fails loudly rather than silently dropping events, so what remains is resiliency work (multi-GET recovery, retry bounds, cursor-progress guard) plus new typed WorkflowWorldError classification of existing throws — enhancements, not a defect fix for the maintenance line. If the typed-error classification in decodeCreateEventResponse/events.ts is wanted on stable, it should be split out and forced through separately.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
aaeb8e44633c1974f087ee7ce38666dbc8ad8a4d


This was referenced Aug 31, 2026
 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@alangenfeldalangenfeldalangenfeld approved these changes
@VaguelySeriousVaguelySeriousAwaiting requested review from VaguelySerious

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@alangenfeld@VaguelySerious