[world-vercel] Fail the run on a lost event payload instead of retrying forever - #3742

Merged
alangenfeld merged 5 commits into
mainfrom
peter/v4-terminal-payload-missing
Aug 27, 2026
Merged

[world-vercel] Fail the run on a lost event payload instead of retrying forever#3742
alangenfeld merged 5 commits into
mainfrom
peter/v4-terminal-payload-missing

Conversation

@VaguelySerious

@VaguelySeriousVaguelySerious commented Aug 22, 2026•
edited by alangenfeld
Loading

Copy link
Copy Markdown
Member

Summary & Motivation

The backend now closes a frame stream with a terminal {_error: 1, code} frame instead of a truncated body, which previously read as a dropped socket. payload-missing becomes a CorruptedEventLogError so the run fails with CORRUPTED_EVENT_LOG; any unrecognized code becomes a terminal WorkflowWorldError, so a future code can't turn into a redelivery loop.

Test Plan

Tests added for both codes in the v4 events reader and for the retry/terminal classification in core; typecheck clean.

Docs Preview

PagePreview
Errors and retrieshttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/foundations/errors-and-retries#error-codes
Corrupted event loghttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/errors/corrupted-event-log

@VaguelySerious@claude
fix(world-vercel): fail the run on a lost event payload instead of re…
3ea2135
…trying
A frame stream that dies mid-body reaches us as a truncated response, which
is exactly what a dropped socket looks like. So an event whose stored payload
is permanently gone was indistinguishable from a transient blip, and the
runtime kept redelivering a replay that could never succeed: one run re-read
a single missing payload 12,932 times in 26 minutes, and the backend query
behind each attempt throttled its table.
The World now sends a terminal `{_error: 1, code}` frame for failures that a
retry cannot fix. Handle it:
- `payload-missing` raises `CorruptedEventLogError`, so the run fails with
`CORRUPTED_EVENT_LOG` rather than looping. The log does reference a payload
nothing can produce.
- An unknown code raises a `WorkflowWorldError` with no retryable code and no
status, which is also terminal. A future code stays safe without needing a
client release first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@changeset-bot

changeset-botBot commented Aug 22, 2026•
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 9bdc6be

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Aug 22, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-astro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-express-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-hono-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-python-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-vite-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-docsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-swc-playgroundBuildingBuildingPreview, v0Aug 27, 2026 4:34pm
workflow-tarballsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-webReadyReadyPreview, v0Aug 27, 2026 4:34pm

@github-actions

github-actionsBot commented Aug 22, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (8 failed)

python-node (8 failed):

  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58 | 🔍 observability
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH | 🔍 observability
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3 | 🔍 observability
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099 | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M121D9TND7284XZVAK91FT03
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

  • cancelRun via CLI - cancelling a running workflow (hono)
  • cancelRun via CLI - cancelling a running workflow (nitro)
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously (astro)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (fastify)
  • hookWithSleepFinalStepWorkflow - step only on final payload (nextjs-webpack)
  • hookWorkflow (sveltekit)
  • hookWorkflow (tanstack-start)
  • hookWorkflow (vite)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

33 infra events
  • cold-start-warmup · suite warmup (python) · at 16:34:23Z · abandoned wrun_41M1211GA50GPGVFT9JA2YMHA7 · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KT
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KV
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:34:39Z · abandoned wrun_41M12155A60GPAMCTC4YE202Y9
  • run-pickup-stall · nullByteWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550R0GN4EKX2WNAX4YXZ
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:34:40Z · abandoned wrun_41M121550E0GV6HG2GXFK06K37
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:35:12Z · abandoned wrun_41M12164S20GVYBNPPWCTT4KRE
  • run-pickup-stall · nullByteWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170DC0GRDA5F94NK7TGBN
  • run-pickup-stall · sleepingWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170E40GP6HEPXA8WH6DX0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170EC0GKV255EZM11HP19
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170F10GZKFZN2ZR7A5EQV
  • cold-start-warmup · suite warmup (tanstack-start) · at 16:36:24Z · abandoned wrun_01M12180SGAV9AGMYMQX7C5F05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:36:30Z · abandoned wrun_41M12173XJ0GX076RMR9QNDHMY
  • cold-start-warmup · suite warmup (python) · at 16:37:35Z · abandoned wrun_01M1217BCRYX9VD93DFQCWR9NA · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ5
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HQCXM5BTW9C5SSE2GS
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ4
  • run-pickup-stall · sleepingWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HN6NW9FMYWRDEJM3MA
  • run-pickup-stall · nullByteWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HTQQ8ZEHEYY85Q1ZJF
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:38:11Z · abandoned wrun_41M1219VT30GWV53B51TW334GW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:38:11Z · abandoned wrun_41M1219R9Y0GW7ZQJG2A25VXFD
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5BTNYVW1SZDE3P3FJV
  • run-pickup-stall · sleepingWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5F8WY6822ANP1QM4A4
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:38:50Z · abandoned wrun_01M121CV59Y40CW71RTTB2SX1F
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5KAGK1871VP6HT1673
  • run-pickup-stall · nullByteWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5QQWXMDA3XS0XZR0SX
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENRVFWKZDQ6BC14ANRYM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENS367D49TBR1SKBS650
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:39:50Z · abandoned wrun_01M121ENS6XXHMNS0JY4DP59HN
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4C5D73E67TR9ZAKFW4
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4HM6QQWJ86GT0ZKCPH
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:40:50Z · abandoned wrun_01M121GGCAA4YW6KC5XBV80Q4H
  • run-pickup-stall · hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep (nextjs-webpack) · at 16:41:47Z · abandoned wrun_01M121J7RG4W57GGYQ2JYXETJA

E2E Test Summary

Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category

❌ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000

❌ 🌐 Cross-language Conformance

AppPassedFailedSkipped
❌ python09132

✅ vercel-http-transport

AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

✅ vercel-ws-transport

AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028

📋 View full workflow run

@github-actions

github-actionsBot commented Aug 22, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 9bdc6be · Thu, 27 Aug 2026 16:48:26 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep245 (+14%)1347 🔴 (+22%) 🔻1376 🔴 (+20%) 🔻1419 🔴 (+21%) 🔻30
TTFSstream1212 (+509%) 🔻1331 🔴 (+21%) 🔻1350 🔴 (+16%) 🔻1393 🔴 (-3.9%)30
TTFShook + stream1505 (+41%) 🔻1677 🔴 (+22%) 🔻1823 🔴 (+29%) 🔻5993 🔴 (+245%) 🔻30
Fan-out TTFSPromise.all(100 steps)593 (-8.1%)981 (-45%) 💚1742 (-4.4%)1950 (+5.8%)10
Fan-out TTLSPromise.all(100 steps)1796 (-13%)4594 (+15%)5159 (+25%) 🔻8031 (+83%) 🔻10
STSO1020 steps (inline)80 (-19%) 💚114 (-31%) 💚125 (-38%) 💚209 (-39%) 💚1019
WO1020 steps115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚1
CRTTfirst chunk (pooled)75 (-26%) 💚154 (-7.2%)351 (+74%) 🔻387 (-88%) 💚28

Streams

ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)98 (-34%)106 (-49%)141 (-53%)446 (-14%)85.5 (-52%)10
size sweep (100/s, 160B-12KB)97 (-32%)115 (-55%)199 (-50%)593 (-55%)102 (-62%)10
replay gateway-gpt-5.4-nano-2000t (1x)150 (-5%)101 (-61%)130 (-91%)461 (-85%)176 (-50%)3
replay eve-gpt-5.6-sol-2000t (1x)104 (-24%)126 (-34%)352 (+35%)3988 (+806%)2104 (+357%)2
replay eve-gpt-5.6-sol-2000t (2x)351 (+111%)164 (-99%)219 (-99%)397 (-98%)253 (-62%)3
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 160359ms → this run 115275ms (Δ -45084ms, -28%)

 50-100 ms ┃ main 1 this 44 +43
100-150 ms ███████████████░░░░░░░░┃ main 591 this 934 +343
150-200 ms ┃███████ main 314 this 25 -289
200-250 ms ┃█ main 78 this 13 -65
250-300 ms ┃ main 19 this 2 -17
300-350 ms ┃ main 7 this 1 -6
350-400 ms ┃ main 3 this 0 -3
400-450 ms ┃ main 2 this 0 -2
450-500 ms ┃ main 2 this 0 -2
500-550 ms ┃ main 1 this 0 -1
550-600 ms ┃ main 1 this 0 -1
📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ·····▁█▄▁···· 98.1 (-40%) 87 (-40%) 141 (-53%) 446 (-14%) 3000
sweep ·····▁█▅▁▁··· 103.4 (-50%) 92 (-43%) 199 (-50%) 593 (-55%) 3000
gw 1x ·····▁█▃▁▁··· 92.4 (-63%) 84 (-40%) 130 (-91%) 461 (-85%) 5295
eve 1x ·····▁█▄▁▁▁▁· 213.6 (+36%) 84 (-36%) 352 (+35%) 3988 (+806%) 5186
eve 2x ·····▁▇█▂▁··· 126.3 (-96%) 108 (-41%) 219 (-99%) 397 (-98%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control ▇▃█▆▄▂▂▂▂▁ 83–124ms
sweep ▇▅▂▇█▃▁▂▄▂ 88–123ms
gw 1x █▂▂▁▃▂▁▃▂▂ 82–129ms
eve 1x █▁▁▁▁▁▁▁▁▁ 78–1251ms
eve 2x ▄▂▂▂▂▃▄█▁▄ 113–156ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep █▇▅▂▁▁▃ 101–107ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▄▃▆█▃▃▃▃▄▁ 22–34ms
sweep ▄▂▄█▅▆▃▃▇▁ 32–43ms
gw 1x █▄▂▁▅▄▄▄▃▄ 20–31ms
eve 1x █▂▂▁▂▁▁▂▂▂ 17–60ms
eve 2x ▆▅▃▅█▄▄▆▁▆ 17–25ms
ℹ️ Metric definitions & methodology

Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.

The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@github-actions

github-actionsBot commented Aug 22, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0

Full trace: world-sim.txt

@VaguelySerious@claude
Revert the world-vercel URL override to empty
f68d1a4
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySerious
VaguelySerious marked this pull request as ready for review August 25, 2026 21:51
karthikscale3
karthikscale3 reviewed Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts Outdated
@github-actionsgithub-actionsBot mentioned this pull request Aug 27, 2026
Merged
@github-actions

github-actionsBot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Backport to stable failed for 1c44cc8 due to a workflow error (backport job run).

This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.

Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

1c44cc8c3f76f81553d04d21f56da1d2adbc1746

This was referenced Aug 27, 2026
Open
Open
Open
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alangenfeldalangenfeldalangenfeld approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@karthikscale3karthikscale3Awaiting requested review from karthikscale3

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@VaguelySerious@alangenfeld@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
[world-vercel] Fail the run on a lost event payload instead of retrying forever - #3742
Merged
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Aug 27, 2026
Merged
[world-vercel] Fail the run on a lost event payload instead of retrying forever#3742
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Conversation
@VaguelySerious
@VaguelySeriousVaguelySerious
commented
Aug 22, 2026•

edited by alangenfeld

Loading

Copy link


Copy Markdown

Member
Summary & Motivation
The backend now closes a frame stream with a terminal {_error: 1, code} frame instead of a truncated body, which previously read as a dropped socket. payload-missing becomes a CorruptedEventLogError so the run fails with CORRUPTED_EVENT_LOG; any unrecognized code becomes a terminal WorkflowWorldError, so a future code can't turn into a redelivery loop.
Test Plan
Tests added for both codes in the v4 events reader and for the retry/terminal classification in core; typecheck clean.
Docs Preview
PagePreview
Errors and retrieshttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/foundations/errors-and-retries#error-codes
Corrupted event loghttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/errors/corrupted-event-log
@VaguelySerious@claude
fix(world-vercel): fail the run on a lost event payload instead of re…
3ea2135
…trying
A frame stream that dies mid-body reaches us as a truncated response, which
is exactly what a dropped socket looks like. So an event whose stored payload
is permanently gone was indistinguishable from a transient blip, and the
runtime kept redelivering a replay that could never succeed: one run re-read
a single missing payload 12,932 times in 26 minutes, and the backend query
behind each attempt throttled its table.
The World now sends a terminal `{_error: 1, code}` frame for failures that a
retry cannot fix. Handle it:
- `payload-missing` raises `CorruptedEventLogError`, so the run fails with
`CORRUPTED_EVENT_LOG` rather than looping. The log does reference a payload
nothing can produce.
- An unknown code raises a `WorkflowWorldError` with no retryable code and no
status, which is also terminal. A future code stays safe without needing a
client release first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@changeset-bot
changeset-botBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 9bdc6be
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-astro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-express-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-hono-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-python-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-vite-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-docsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-swc-playgroundBuildingBuildingPreview, v0Aug 27, 2026 4:34pm
workflow-tarballsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-webReadyReadyPreview, v0Aug 27, 2026 4:34pm
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58 | 🔍 observability
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH | 🔍 observability
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3 | 🔍 observability
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M121D9TND7284XZVAK91FT03
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • cancelRun via CLI - cancelling a running workflow (hono)
  • cancelRun via CLI - cancelling a running workflow (nitro)
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously (astro)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (fastify)
  • hookWithSleepFinalStepWorkflow - step only on final payload (nextjs-webpack)
  • hookWorkflow (sveltekit)
  • hookWorkflow (tanstack-start)
  • hookWorkflow (vite)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (python) · at 16:34:23Z · abandoned wrun_41M1211GA50GPGVFT9JA2YMHA7 · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KT
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KV
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:34:39Z · abandoned wrun_41M12155A60GPAMCTC4YE202Y9
  • run-pickup-stall · nullByteWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550R0GN4EKX2WNAX4YXZ
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:34:40Z · abandoned wrun_41M121550E0GV6HG2GXFK06K37
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:35:12Z · abandoned wrun_41M12164S20GVYBNPPWCTT4KRE
  • run-pickup-stall · nullByteWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170DC0GRDA5F94NK7TGBN
  • run-pickup-stall · sleepingWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170E40GP6HEPXA8WH6DX0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170EC0GKV255EZM11HP19
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170F10GZKFZN2ZR7A5EQV
  • cold-start-warmup · suite warmup (tanstack-start) · at 16:36:24Z · abandoned wrun_01M12180SGAV9AGMYMQX7C5F05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:36:30Z · abandoned wrun_41M12173XJ0GX076RMR9QNDHMY
  • cold-start-warmup · suite warmup (python) · at 16:37:35Z · abandoned wrun_01M1217BCRYX9VD93DFQCWR9NA · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ5
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HQCXM5BTW9C5SSE2GS
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ4
  • run-pickup-stall · sleepingWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HN6NW9FMYWRDEJM3MA
  • run-pickup-stall · nullByteWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HTQQ8ZEHEYY85Q1ZJF
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:38:11Z · abandoned wrun_41M1219VT30GWV53B51TW334GW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:38:11Z · abandoned wrun_41M1219R9Y0GW7ZQJG2A25VXFD
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5BTNYVW1SZDE3P3FJV
  • run-pickup-stall · sleepingWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5F8WY6822ANP1QM4A4
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:38:50Z · abandoned wrun_01M121CV59Y40CW71RTTB2SX1F
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5KAGK1871VP6HT1673
  • run-pickup-stall · nullByteWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5QQWXMDA3XS0XZR0SX
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENRVFWKZDQ6BC14ANRYM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENS367D49TBR1SKBS650
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:39:50Z · abandoned wrun_01M121ENS6XXHMNS0JY4DP59HN
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4C5D73E67TR9ZAKFW4
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4HM6QQWJ86GT0ZKCPH
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:40:50Z · abandoned wrun_01M121GGCAA4YW6KC5XBV80Q4H
  • run-pickup-stall · hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep (nextjs-webpack) · at 16:41:47Z · abandoned wrun_01M121J7RG4W57GGYQ2JYXETJA
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 9bdc6be · Thu, 27 Aug 2026 16:48:26 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep245 (+14%)1347 🔴 (+22%) 🔻1376 🔴 (+20%) 🔻1419 🔴 (+21%) 🔻30
TTFSstream1212 (+509%) 🔻1331 🔴 (+21%) 🔻1350 🔴 (+16%) 🔻1393 🔴 (-3.9%)30
TTFShook + stream1505 (+41%) 🔻1677 🔴 (+22%) 🔻1823 🔴 (+29%) 🔻5993 🔴 (+245%) 🔻30
Fan-out TTFSPromise.all(100 steps)593 (-8.1%)981 (-45%) 💚1742 (-4.4%)1950 (+5.8%)10
Fan-out TTLSPromise.all(100 steps)1796 (-13%)4594 (+15%)5159 (+25%) 🔻8031 (+83%) 🔻10
STSO1020 steps (inline)80 (-19%) 💚114 (-31%) 💚125 (-38%) 💚209 (-39%) 💚1019
WO1020 steps115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚1
CRTTfirst chunk (pooled)75 (-26%) 💚154 (-7.2%)351 (+74%) 🔻387 (-88%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)98 (-34%)106 (-49%)141 (-53%)446 (-14%)85.5 (-52%)10
size sweep (100/s, 160B-12KB)97 (-32%)115 (-55%)199 (-50%)593 (-55%)102 (-62%)10
replay gateway-gpt-5.4-nano-2000t (1x)150 (-5%)101 (-61%)130 (-91%)461 (-85%)176 (-50%)3
replay eve-gpt-5.6-sol-2000t (1x)104 (-24%)126 (-34%)352 (+35%)3988 (+806%)2104 (+357%)2
replay eve-gpt-5.6-sol-2000t (2x)351 (+111%)164 (-99%)219 (-99%)397 (-98%)253 (-62%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 160359ms → this run 115275ms (Δ -45084ms, -28%)
 50-100 ms ┃ main 1 this 44 +43
100-150 ms ███████████████░░░░░░░░┃ main 591 this 934 +343
150-200 ms ┃███████ main 314 this 25 -289
200-250 ms ┃█ main 78 this 13 -65
250-300 ms ┃ main 19 this 2 -17
300-350 ms ┃ main 7 this 1 -6
350-400 ms ┃ main 3 this 0 -3
400-450 ms ┃ main 2 this 0 -2
450-500 ms ┃ main 2 this 0 -2
500-550 ms ┃ main 1 this 0 -1
550-600 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ·····▁█▄▁···· 98.1 (-40%) 87 (-40%) 141 (-53%) 446 (-14%) 3000
sweep ·····▁█▅▁▁··· 103.4 (-50%) 92 (-43%) 199 (-50%) 593 (-55%) 3000
gw 1x ·····▁█▃▁▁··· 92.4 (-63%) 84 (-40%) 130 (-91%) 461 (-85%) 5295
eve 1x ·····▁█▄▁▁▁▁· 213.6 (+36%) 84 (-36%) 352 (+35%) 3988 (+806%) 5186
eve 2x ·····▁▇█▂▁··· 126.3 (-96%) 108 (-41%) 219 (-99%) 397 (-98%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▇▃█▆▄▂▂▂▂▁ 83–124ms
sweep ▇▅▂▇█▃▁▂▄▂ 88–123ms
gw 1x █▂▂▁▃▂▁▃▂▂ 82–129ms
eve 1x █▁▁▁▁▁▁▁▁▁ 78–1251ms
eve 2x ▄▂▂▂▂▃▄█▁▄ 113–156ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep █▇▅▂▁▁▃ 101–107ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▃▆█▃▃▃▃▄▁ 22–34ms
sweep ▄▂▄█▅▆▃▃▇▁ 32–43ms
gw 1x █▄▂▁▅▄▄▄▃▄ 20–31ms
eve 1x █▂▂▁▂▁▁▂▂▂ 17–60ms
eve 2x ▆▅▃▅█▄▄▆▁▆ 17–25ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@VaguelySerious@claude
Revert the world-vercel URL override to empty
f68d1a4
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySerious
VaguelySerious
marked this pull request as ready for review
August 25, 2026 21:51
karthikscale3
karthikscale3
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 27, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 1c44cc8 due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
1c44cc8c3f76f81553d04d21f56da1d2adbc1746


This was referenced Aug 27, 2026
 Open

 Open

 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@alangenfeldalangenfeldalangenfeld approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@karthikscale3karthikscale3Awaiting requested review from karthikscale3

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@VaguelySerious@alangenfeld@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[world-vercel] Fail the run on a lost event payload instead of retrying forever - #3742
Merged
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Aug 27, 2026
Merged
[world-vercel] Fail the run on a lost event payload instead of retrying forever#3742
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Conversation
@VaguelySerious
@VaguelySeriousVaguelySerious
commented
Aug 22, 2026•

edited by alangenfeld

Loading

Copy link


Copy Markdown

Member
Summary & Motivation
The backend now closes a frame stream with a terminal {_error: 1, code} frame instead of a truncated body, which previously read as a dropped socket. payload-missing becomes a CorruptedEventLogError so the run fails with CORRUPTED_EVENT_LOG; any unrecognized code becomes a terminal WorkflowWorldError, so a future code can't turn into a redelivery loop.
Test Plan
Tests added for both codes in the v4 events reader and for the retry/terminal classification in core; typecheck clean.
Docs Preview
PagePreview
Errors and retrieshttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/foundations/errors-and-retries#error-codes
Corrupted event loghttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/errors/corrupted-event-log
@VaguelySerious@claude
fix(world-vercel): fail the run on a lost event payload instead of re…
3ea2135
…trying
A frame stream that dies mid-body reaches us as a truncated response, which
is exactly what a dropped socket looks like. So an event whose stored payload
is permanently gone was indistinguishable from a transient blip, and the
runtime kept redelivering a replay that could never succeed: one run re-read
a single missing payload 12,932 times in 26 minutes, and the backend query
behind each attempt throttled its table.
The World now sends a terminal `{_error: 1, code}` frame for failures that a
retry cannot fix. Handle it:
- `payload-missing` raises `CorruptedEventLogError`, so the run fails with
`CORRUPTED_EVENT_LOG` rather than looping. The log does reference a payload
nothing can produce.
- An unknown code raises a `WorkflowWorldError` with no retryable code and no
status, which is also terminal. A future code stays safe without needing a
client release first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@changeset-bot
changeset-botBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 9bdc6be
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-astro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-express-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-hono-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-python-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-vite-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-docsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-swc-playgroundBuildingBuildingPreview, v0Aug 27, 2026 4:34pm
workflow-tarballsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-webReadyReadyPreview, v0Aug 27, 2026 4:34pm
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58 | 🔍 observability
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH | 🔍 observability
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3 | 🔍 observability
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M121D9TND7284XZVAK91FT03
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • cancelRun via CLI - cancelling a running workflow (hono)
  • cancelRun via CLI - cancelling a running workflow (nitro)
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously (astro)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (fastify)
  • hookWithSleepFinalStepWorkflow - step only on final payload (nextjs-webpack)
  • hookWorkflow (sveltekit)
  • hookWorkflow (tanstack-start)
  • hookWorkflow (vite)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (python) · at 16:34:23Z · abandoned wrun_41M1211GA50GPGVFT9JA2YMHA7 · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KT
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KV
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:34:39Z · abandoned wrun_41M12155A60GPAMCTC4YE202Y9
  • run-pickup-stall · nullByteWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550R0GN4EKX2WNAX4YXZ
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:34:40Z · abandoned wrun_41M121550E0GV6HG2GXFK06K37
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:35:12Z · abandoned wrun_41M12164S20GVYBNPPWCTT4KRE
  • run-pickup-stall · nullByteWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170DC0GRDA5F94NK7TGBN
  • run-pickup-stall · sleepingWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170E40GP6HEPXA8WH6DX0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170EC0GKV255EZM11HP19
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170F10GZKFZN2ZR7A5EQV
  • cold-start-warmup · suite warmup (tanstack-start) · at 16:36:24Z · abandoned wrun_01M12180SGAV9AGMYMQX7C5F05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:36:30Z · abandoned wrun_41M12173XJ0GX076RMR9QNDHMY
  • cold-start-warmup · suite warmup (python) · at 16:37:35Z · abandoned wrun_01M1217BCRYX9VD93DFQCWR9NA · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ5
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HQCXM5BTW9C5SSE2GS
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ4
  • run-pickup-stall · sleepingWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HN6NW9FMYWRDEJM3MA
  • run-pickup-stall · nullByteWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HTQQ8ZEHEYY85Q1ZJF
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:38:11Z · abandoned wrun_41M1219VT30GWV53B51TW334GW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:38:11Z · abandoned wrun_41M1219R9Y0GW7ZQJG2A25VXFD
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5BTNYVW1SZDE3P3FJV
  • run-pickup-stall · sleepingWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5F8WY6822ANP1QM4A4
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:38:50Z · abandoned wrun_01M121CV59Y40CW71RTTB2SX1F
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5KAGK1871VP6HT1673
  • run-pickup-stall · nullByteWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5QQWXMDA3XS0XZR0SX
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENRVFWKZDQ6BC14ANRYM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENS367D49TBR1SKBS650
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:39:50Z · abandoned wrun_01M121ENS6XXHMNS0JY4DP59HN
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4C5D73E67TR9ZAKFW4
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4HM6QQWJ86GT0ZKCPH
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:40:50Z · abandoned wrun_01M121GGCAA4YW6KC5XBV80Q4H
  • run-pickup-stall · hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep (nextjs-webpack) · at 16:41:47Z · abandoned wrun_01M121J7RG4W57GGYQ2JYXETJA
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 9bdc6be · Thu, 27 Aug 2026 16:48:26 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep245 (+14%)1347 🔴 (+22%) 🔻1376 🔴 (+20%) 🔻1419 🔴 (+21%) 🔻30
TTFSstream1212 (+509%) 🔻1331 🔴 (+21%) 🔻1350 🔴 (+16%) 🔻1393 🔴 (-3.9%)30
TTFShook + stream1505 (+41%) 🔻1677 🔴 (+22%) 🔻1823 🔴 (+29%) 🔻5993 🔴 (+245%) 🔻30
Fan-out TTFSPromise.all(100 steps)593 (-8.1%)981 (-45%) 💚1742 (-4.4%)1950 (+5.8%)10
Fan-out TTLSPromise.all(100 steps)1796 (-13%)4594 (+15%)5159 (+25%) 🔻8031 (+83%) 🔻10
STSO1020 steps (inline)80 (-19%) 💚114 (-31%) 💚125 (-38%) 💚209 (-39%) 💚1019
WO1020 steps115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚1
CRTTfirst chunk (pooled)75 (-26%) 💚154 (-7.2%)351 (+74%) 🔻387 (-88%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)98 (-34%)106 (-49%)141 (-53%)446 (-14%)85.5 (-52%)10
size sweep (100/s, 160B-12KB)97 (-32%)115 (-55%)199 (-50%)593 (-55%)102 (-62%)10
replay gateway-gpt-5.4-nano-2000t (1x)150 (-5%)101 (-61%)130 (-91%)461 (-85%)176 (-50%)3
replay eve-gpt-5.6-sol-2000t (1x)104 (-24%)126 (-34%)352 (+35%)3988 (+806%)2104 (+357%)2
replay eve-gpt-5.6-sol-2000t (2x)351 (+111%)164 (-99%)219 (-99%)397 (-98%)253 (-62%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 160359ms → this run 115275ms (Δ -45084ms, -28%)
 50-100 ms ┃ main 1 this 44 +43
100-150 ms ███████████████░░░░░░░░┃ main 591 this 934 +343
150-200 ms ┃███████ main 314 this 25 -289
200-250 ms ┃█ main 78 this 13 -65
250-300 ms ┃ main 19 this 2 -17
300-350 ms ┃ main 7 this 1 -6
350-400 ms ┃ main 3 this 0 -3
400-450 ms ┃ main 2 this 0 -2
450-500 ms ┃ main 2 this 0 -2
500-550 ms ┃ main 1 this 0 -1
550-600 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ·····▁█▄▁···· 98.1 (-40%) 87 (-40%) 141 (-53%) 446 (-14%) 3000
sweep ·····▁█▅▁▁··· 103.4 (-50%) 92 (-43%) 199 (-50%) 593 (-55%) 3000
gw 1x ·····▁█▃▁▁··· 92.4 (-63%) 84 (-40%) 130 (-91%) 461 (-85%) 5295
eve 1x ·····▁█▄▁▁▁▁· 213.6 (+36%) 84 (-36%) 352 (+35%) 3988 (+806%) 5186
eve 2x ·····▁▇█▂▁··· 126.3 (-96%) 108 (-41%) 219 (-99%) 397 (-98%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▇▃█▆▄▂▂▂▂▁ 83–124ms
sweep ▇▅▂▇█▃▁▂▄▂ 88–123ms
gw 1x █▂▂▁▃▂▁▃▂▂ 82–129ms
eve 1x █▁▁▁▁▁▁▁▁▁ 78–1251ms
eve 2x ▄▂▂▂▂▃▄█▁▄ 113–156ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep █▇▅▂▁▁▃ 101–107ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▃▆█▃▃▃▃▄▁ 22–34ms
sweep ▄▂▄█▅▆▃▃▇▁ 32–43ms
gw 1x █▄▂▁▅▄▄▄▃▄ 20–31ms
eve 1x █▂▂▁▂▁▁▂▂▂ 17–60ms
eve 2x ▆▅▃▅█▄▄▆▁▆ 17–25ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@VaguelySerious@claude
Revert the world-vercel URL override to empty
f68d1a4
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySerious
VaguelySerious
marked this pull request as ready for review
August 25, 2026 21:51
karthikscale3
karthikscale3
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 27, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 1c44cc8 due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
1c44cc8c3f76f81553d04d21f56da1d2adbc1746


This was referenced Aug 27, 2026
 Open

 Open

 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@alangenfeldalangenfeldalangenfeld approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@karthikscale3karthikscale3Awaiting requested review from karthikscale3

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@VaguelySerious@alangenfeld@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[world-vercel] Fail the run on a lost event payload instead of retrying forever - #3742
Merged
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Aug 27, 2026
Merged
[world-vercel] Fail the run on a lost event payload instead of retrying forever#3742
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Conversation
@VaguelySerious
@VaguelySeriousVaguelySerious
commented
Aug 22, 2026•

edited by alangenfeld

Loading

Copy link


Copy Markdown

Member
Summary & Motivation
The backend now closes a frame stream with a terminal {_error: 1, code} frame instead of a truncated body, which previously read as a dropped socket. payload-missing becomes a CorruptedEventLogError so the run fails with CORRUPTED_EVENT_LOG; any unrecognized code becomes a terminal WorkflowWorldError, so a future code can't turn into a redelivery loop.
Test Plan
Tests added for both codes in the v4 events reader and for the retry/terminal classification in core; typecheck clean.
Docs Preview
PagePreview
Errors and retrieshttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/foundations/errors-and-retries#error-codes
Corrupted event loghttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/errors/corrupted-event-log
@VaguelySerious@claude
fix(world-vercel): fail the run on a lost event payload instead of re…
3ea2135
…trying
A frame stream that dies mid-body reaches us as a truncated response, which
is exactly what a dropped socket looks like. So an event whose stored payload
is permanently gone was indistinguishable from a transient blip, and the
runtime kept redelivering a replay that could never succeed: one run re-read
a single missing payload 12,932 times in 26 minutes, and the backend query
behind each attempt throttled its table.
The World now sends a terminal `{_error: 1, code}` frame for failures that a
retry cannot fix. Handle it:
- `payload-missing` raises `CorruptedEventLogError`, so the run fails with
`CORRUPTED_EVENT_LOG` rather than looping. The log does reference a payload
nothing can produce.
- An unknown code raises a `WorkflowWorldError` with no retryable code and no
status, which is also terminal. A future code stays safe without needing a
client release first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@changeset-bot
changeset-botBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 9bdc6be
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-astro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-express-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-hono-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-python-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-vite-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-docsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-swc-playgroundBuildingBuildingPreview, v0Aug 27, 2026 4:34pm
workflow-tarballsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-webReadyReadyPreview, v0Aug 27, 2026 4:34pm
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58 | 🔍 observability
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH | 🔍 observability
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3 | 🔍 observability
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M121D9TND7284XZVAK91FT03
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • cancelRun via CLI - cancelling a running workflow (hono)
  • cancelRun via CLI - cancelling a running workflow (nitro)
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously (astro)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (fastify)
  • hookWithSleepFinalStepWorkflow - step only on final payload (nextjs-webpack)
  • hookWorkflow (sveltekit)
  • hookWorkflow (tanstack-start)
  • hookWorkflow (vite)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (python) · at 16:34:23Z · abandoned wrun_41M1211GA50GPGVFT9JA2YMHA7 · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KT
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KV
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:34:39Z · abandoned wrun_41M12155A60GPAMCTC4YE202Y9
  • run-pickup-stall · nullByteWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550R0GN4EKX2WNAX4YXZ
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:34:40Z · abandoned wrun_41M121550E0GV6HG2GXFK06K37
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:35:12Z · abandoned wrun_41M12164S20GVYBNPPWCTT4KRE
  • run-pickup-stall · nullByteWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170DC0GRDA5F94NK7TGBN
  • run-pickup-stall · sleepingWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170E40GP6HEPXA8WH6DX0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170EC0GKV255EZM11HP19
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170F10GZKFZN2ZR7A5EQV
  • cold-start-warmup · suite warmup (tanstack-start) · at 16:36:24Z · abandoned wrun_01M12180SGAV9AGMYMQX7C5F05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:36:30Z · abandoned wrun_41M12173XJ0GX076RMR9QNDHMY
  • cold-start-warmup · suite warmup (python) · at 16:37:35Z · abandoned wrun_01M1217BCRYX9VD93DFQCWR9NA · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ5
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HQCXM5BTW9C5SSE2GS
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ4
  • run-pickup-stall · sleepingWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HN6NW9FMYWRDEJM3MA
  • run-pickup-stall · nullByteWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HTQQ8ZEHEYY85Q1ZJF
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:38:11Z · abandoned wrun_41M1219VT30GWV53B51TW334GW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:38:11Z · abandoned wrun_41M1219R9Y0GW7ZQJG2A25VXFD
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5BTNYVW1SZDE3P3FJV
  • run-pickup-stall · sleepingWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5F8WY6822ANP1QM4A4
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:38:50Z · abandoned wrun_01M121CV59Y40CW71RTTB2SX1F
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5KAGK1871VP6HT1673
  • run-pickup-stall · nullByteWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5QQWXMDA3XS0XZR0SX
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENRVFWKZDQ6BC14ANRYM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENS367D49TBR1SKBS650
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:39:50Z · abandoned wrun_01M121ENS6XXHMNS0JY4DP59HN
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4C5D73E67TR9ZAKFW4
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4HM6QQWJ86GT0ZKCPH
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:40:50Z · abandoned wrun_01M121GGCAA4YW6KC5XBV80Q4H
  • run-pickup-stall · hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep (nextjs-webpack) · at 16:41:47Z · abandoned wrun_01M121J7RG4W57GGYQ2JYXETJA
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 9bdc6be · Thu, 27 Aug 2026 16:48:26 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep245 (+14%)1347 🔴 (+22%) 🔻1376 🔴 (+20%) 🔻1419 🔴 (+21%) 🔻30
TTFSstream1212 (+509%) 🔻1331 🔴 (+21%) 🔻1350 🔴 (+16%) 🔻1393 🔴 (-3.9%)30
TTFShook + stream1505 (+41%) 🔻1677 🔴 (+22%) 🔻1823 🔴 (+29%) 🔻5993 🔴 (+245%) 🔻30
Fan-out TTFSPromise.all(100 steps)593 (-8.1%)981 (-45%) 💚1742 (-4.4%)1950 (+5.8%)10
Fan-out TTLSPromise.all(100 steps)1796 (-13%)4594 (+15%)5159 (+25%) 🔻8031 (+83%) 🔻10
STSO1020 steps (inline)80 (-19%) 💚114 (-31%) 💚125 (-38%) 💚209 (-39%) 💚1019
WO1020 steps115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚1
CRTTfirst chunk (pooled)75 (-26%) 💚154 (-7.2%)351 (+74%) 🔻387 (-88%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)98 (-34%)106 (-49%)141 (-53%)446 (-14%)85.5 (-52%)10
size sweep (100/s, 160B-12KB)97 (-32%)115 (-55%)199 (-50%)593 (-55%)102 (-62%)10
replay gateway-gpt-5.4-nano-2000t (1x)150 (-5%)101 (-61%)130 (-91%)461 (-85%)176 (-50%)3
replay eve-gpt-5.6-sol-2000t (1x)104 (-24%)126 (-34%)352 (+35%)3988 (+806%)2104 (+357%)2
replay eve-gpt-5.6-sol-2000t (2x)351 (+111%)164 (-99%)219 (-99%)397 (-98%)253 (-62%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 160359ms → this run 115275ms (Δ -45084ms, -28%)
 50-100 ms ┃ main 1 this 44 +43
100-150 ms ███████████████░░░░░░░░┃ main 591 this 934 +343
150-200 ms ┃███████ main 314 this 25 -289
200-250 ms ┃█ main 78 this 13 -65
250-300 ms ┃ main 19 this 2 -17
300-350 ms ┃ main 7 this 1 -6
350-400 ms ┃ main 3 this 0 -3
400-450 ms ┃ main 2 this 0 -2
450-500 ms ┃ main 2 this 0 -2
500-550 ms ┃ main 1 this 0 -1
550-600 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ·····▁█▄▁···· 98.1 (-40%) 87 (-40%) 141 (-53%) 446 (-14%) 3000
sweep ·····▁█▅▁▁··· 103.4 (-50%) 92 (-43%) 199 (-50%) 593 (-55%) 3000
gw 1x ·····▁█▃▁▁··· 92.4 (-63%) 84 (-40%) 130 (-91%) 461 (-85%) 5295
eve 1x ·····▁█▄▁▁▁▁· 213.6 (+36%) 84 (-36%) 352 (+35%) 3988 (+806%) 5186
eve 2x ·····▁▇█▂▁··· 126.3 (-96%) 108 (-41%) 219 (-99%) 397 (-98%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▇▃█▆▄▂▂▂▂▁ 83–124ms
sweep ▇▅▂▇█▃▁▂▄▂ 88–123ms
gw 1x █▂▂▁▃▂▁▃▂▂ 82–129ms
eve 1x █▁▁▁▁▁▁▁▁▁ 78–1251ms
eve 2x ▄▂▂▂▂▃▄█▁▄ 113–156ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep █▇▅▂▁▁▃ 101–107ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▃▆█▃▃▃▃▄▁ 22–34ms
sweep ▄▂▄█▅▆▃▃▇▁ 32–43ms
gw 1x █▄▂▁▅▄▄▄▃▄ 20–31ms
eve 1x █▂▂▁▂▁▁▂▂▂ 17–60ms
eve 2x ▆▅▃▅█▄▄▆▁▆ 17–25ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@VaguelySerious@claude
Revert the world-vercel URL override to empty
f68d1a4
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySerious
VaguelySerious
marked this pull request as ready for review
August 25, 2026 21:51
karthikscale3
karthikscale3
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 27, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 1c44cc8 due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
1c44cc8c3f76f81553d04d21f56da1d2adbc1746


This was referenced Aug 27, 2026
 Open

 Open

 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@alangenfeldalangenfeldalangenfeld approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@karthikscale3karthikscale3Awaiting requested review from karthikscale3

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@VaguelySerious@alangenfeld@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
[world-vercel] Fail the run on a lost event payload instead of retrying forever - #3742
Merged
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Aug 27, 2026
Merged
[world-vercel] Fail the run on a lost event payload instead of retrying forever#3742
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Conversation
@VaguelySerious
@VaguelySeriousVaguelySerious
commented
Aug 22, 2026•

edited by alangenfeld

Loading

Copy link


Copy Markdown

Member
Summary & Motivation
The backend now closes a frame stream with a terminal {_error: 1, code} frame instead of a truncated body, which previously read as a dropped socket. payload-missing becomes a CorruptedEventLogError so the run fails with CORRUPTED_EVENT_LOG; any unrecognized code becomes a terminal WorkflowWorldError, so a future code can't turn into a redelivery loop.
Test Plan
Tests added for both codes in the v4 events reader and for the retry/terminal classification in core; typecheck clean.
Docs Preview
PagePreview
Errors and retrieshttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/foundations/errors-and-retries#error-codes
Corrupted event loghttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/errors/corrupted-event-log
@VaguelySerious@claude
fix(world-vercel): fail the run on a lost event payload instead of re…
3ea2135
…trying
A frame stream that dies mid-body reaches us as a truncated response, which
is exactly what a dropped socket looks like. So an event whose stored payload
is permanently gone was indistinguishable from a transient blip, and the
runtime kept redelivering a replay that could never succeed: one run re-read
a single missing payload 12,932 times in 26 minutes, and the backend query
behind each attempt throttled its table.
The World now sends a terminal `{_error: 1, code}` frame for failures that a
retry cannot fix. Handle it:
- `payload-missing` raises `CorruptedEventLogError`, so the run fails with
`CORRUPTED_EVENT_LOG` rather than looping. The log does reference a payload
nothing can produce.
- An unknown code raises a `WorkflowWorldError` with no retryable code and no
status, which is also terminal. A future code stays safe without needing a
client release first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@changeset-bot
changeset-botBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 9bdc6be
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-astro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-express-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-hono-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-python-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-vite-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-docsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-swc-playgroundBuildingBuildingPreview, v0Aug 27, 2026 4:34pm
workflow-tarballsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-webReadyReadyPreview, v0Aug 27, 2026 4:34pm
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58 | 🔍 observability
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH | 🔍 observability
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3 | 🔍 observability
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M121D9TND7284XZVAK91FT03
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • cancelRun via CLI - cancelling a running workflow (hono)
  • cancelRun via CLI - cancelling a running workflow (nitro)
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously (astro)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (fastify)
  • hookWithSleepFinalStepWorkflow - step only on final payload (nextjs-webpack)
  • hookWorkflow (sveltekit)
  • hookWorkflow (tanstack-start)
  • hookWorkflow (vite)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (python) · at 16:34:23Z · abandoned wrun_41M1211GA50GPGVFT9JA2YMHA7 · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KT
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KV
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:34:39Z · abandoned wrun_41M12155A60GPAMCTC4YE202Y9
  • run-pickup-stall · nullByteWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550R0GN4EKX2WNAX4YXZ
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:34:40Z · abandoned wrun_41M121550E0GV6HG2GXFK06K37
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:35:12Z · abandoned wrun_41M12164S20GVYBNPPWCTT4KRE
  • run-pickup-stall · nullByteWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170DC0GRDA5F94NK7TGBN
  • run-pickup-stall · sleepingWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170E40GP6HEPXA8WH6DX0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170EC0GKV255EZM11HP19
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170F10GZKFZN2ZR7A5EQV
  • cold-start-warmup · suite warmup (tanstack-start) · at 16:36:24Z · abandoned wrun_01M12180SGAV9AGMYMQX7C5F05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:36:30Z · abandoned wrun_41M12173XJ0GX076RMR9QNDHMY
  • cold-start-warmup · suite warmup (python) · at 16:37:35Z · abandoned wrun_01M1217BCRYX9VD93DFQCWR9NA · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ5
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HQCXM5BTW9C5SSE2GS
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ4
  • run-pickup-stall · sleepingWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HN6NW9FMYWRDEJM3MA
  • run-pickup-stall · nullByteWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HTQQ8ZEHEYY85Q1ZJF
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:38:11Z · abandoned wrun_41M1219VT30GWV53B51TW334GW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:38:11Z · abandoned wrun_41M1219R9Y0GW7ZQJG2A25VXFD
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5BTNYVW1SZDE3P3FJV
  • run-pickup-stall · sleepingWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5F8WY6822ANP1QM4A4
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:38:50Z · abandoned wrun_01M121CV59Y40CW71RTTB2SX1F
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5KAGK1871VP6HT1673
  • run-pickup-stall · nullByteWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5QQWXMDA3XS0XZR0SX
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENRVFWKZDQ6BC14ANRYM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENS367D49TBR1SKBS650
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:39:50Z · abandoned wrun_01M121ENS6XXHMNS0JY4DP59HN
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4C5D73E67TR9ZAKFW4
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4HM6QQWJ86GT0ZKCPH
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:40:50Z · abandoned wrun_01M121GGCAA4YW6KC5XBV80Q4H
  • run-pickup-stall · hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep (nextjs-webpack) · at 16:41:47Z · abandoned wrun_01M121J7RG4W57GGYQ2JYXETJA
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 9bdc6be · Thu, 27 Aug 2026 16:48:26 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep245 (+14%)1347 🔴 (+22%) 🔻1376 🔴 (+20%) 🔻1419 🔴 (+21%) 🔻30
TTFSstream1212 (+509%) 🔻1331 🔴 (+21%) 🔻1350 🔴 (+16%) 🔻1393 🔴 (-3.9%)30
TTFShook + stream1505 (+41%) 🔻1677 🔴 (+22%) 🔻1823 🔴 (+29%) 🔻5993 🔴 (+245%) 🔻30
Fan-out TTFSPromise.all(100 steps)593 (-8.1%)981 (-45%) 💚1742 (-4.4%)1950 (+5.8%)10
Fan-out TTLSPromise.all(100 steps)1796 (-13%)4594 (+15%)5159 (+25%) 🔻8031 (+83%) 🔻10
STSO1020 steps (inline)80 (-19%) 💚114 (-31%) 💚125 (-38%) 💚209 (-39%) 💚1019
WO1020 steps115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚1
CRTTfirst chunk (pooled)75 (-26%) 💚154 (-7.2%)351 (+74%) 🔻387 (-88%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)98 (-34%)106 (-49%)141 (-53%)446 (-14%)85.5 (-52%)10
size sweep (100/s, 160B-12KB)97 (-32%)115 (-55%)199 (-50%)593 (-55%)102 (-62%)10
replay gateway-gpt-5.4-nano-2000t (1x)150 (-5%)101 (-61%)130 (-91%)461 (-85%)176 (-50%)3
replay eve-gpt-5.6-sol-2000t (1x)104 (-24%)126 (-34%)352 (+35%)3988 (+806%)2104 (+357%)2
replay eve-gpt-5.6-sol-2000t (2x)351 (+111%)164 (-99%)219 (-99%)397 (-98%)253 (-62%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 160359ms → this run 115275ms (Δ -45084ms, -28%)
 50-100 ms ┃ main 1 this 44 +43
100-150 ms ███████████████░░░░░░░░┃ main 591 this 934 +343
150-200 ms ┃███████ main 314 this 25 -289
200-250 ms ┃█ main 78 this 13 -65
250-300 ms ┃ main 19 this 2 -17
300-350 ms ┃ main 7 this 1 -6
350-400 ms ┃ main 3 this 0 -3
400-450 ms ┃ main 2 this 0 -2
450-500 ms ┃ main 2 this 0 -2
500-550 ms ┃ main 1 this 0 -1
550-600 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ·····▁█▄▁···· 98.1 (-40%) 87 (-40%) 141 (-53%) 446 (-14%) 3000
sweep ·····▁█▅▁▁··· 103.4 (-50%) 92 (-43%) 199 (-50%) 593 (-55%) 3000
gw 1x ·····▁█▃▁▁··· 92.4 (-63%) 84 (-40%) 130 (-91%) 461 (-85%) 5295
eve 1x ·····▁█▄▁▁▁▁· 213.6 (+36%) 84 (-36%) 352 (+35%) 3988 (+806%) 5186
eve 2x ·····▁▇█▂▁··· 126.3 (-96%) 108 (-41%) 219 (-99%) 397 (-98%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▇▃█▆▄▂▂▂▂▁ 83–124ms
sweep ▇▅▂▇█▃▁▂▄▂ 88–123ms
gw 1x █▂▂▁▃▂▁▃▂▂ 82–129ms
eve 1x █▁▁▁▁▁▁▁▁▁ 78–1251ms
eve 2x ▄▂▂▂▂▃▄█▁▄ 113–156ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep █▇▅▂▁▁▃ 101–107ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▃▆█▃▃▃▃▄▁ 22–34ms
sweep ▄▂▄█▅▆▃▃▇▁ 32–43ms
gw 1x █▄▂▁▅▄▄▄▃▄ 20–31ms
eve 1x █▂▂▁▂▁▁▂▂▂ 17–60ms
eve 2x ▆▅▃▅█▄▄▆▁▆ 17–25ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@VaguelySerious@claude
Revert the world-vercel URL override to empty
f68d1a4
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySerious
VaguelySerious
marked this pull request as ready for review
August 25, 2026 21:51
karthikscale3
karthikscale3
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 27, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 1c44cc8 due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
1c44cc8c3f76f81553d04d21f56da1d2adbc1746


This was referenced Aug 27, 2026
 Open

 Open

 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@alangenfeldalangenfeldalangenfeld approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@karthikscale3karthikscale3Awaiting requested review from karthikscale3

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@VaguelySerious@alangenfeld@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[world-vercel] Fail the run on a lost event payload instead of retrying forever - #3742
Merged
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Aug 27, 2026
Merged
[world-vercel] Fail the run on a lost event payload instead of retrying forever#3742
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Conversation
@VaguelySerious
@VaguelySeriousVaguelySerious
commented
Aug 22, 2026•

edited by alangenfeld

Loading

Copy link


Copy Markdown

Member
Summary & Motivation
The backend now closes a frame stream with a terminal {_error: 1, code} frame instead of a truncated body, which previously read as a dropped socket. payload-missing becomes a CorruptedEventLogError so the run fails with CORRUPTED_EVENT_LOG; any unrecognized code becomes a terminal WorkflowWorldError, so a future code can't turn into a redelivery loop.
Test Plan
Tests added for both codes in the v4 events reader and for the retry/terminal classification in core; typecheck clean.
Docs Preview
PagePreview
Errors and retrieshttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/foundations/errors-and-retries#error-codes
Corrupted event loghttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/errors/corrupted-event-log
@VaguelySerious@claude
fix(world-vercel): fail the run on a lost event payload instead of re…
3ea2135
…trying
A frame stream that dies mid-body reaches us as a truncated response, which
is exactly what a dropped socket looks like. So an event whose stored payload
is permanently gone was indistinguishable from a transient blip, and the
runtime kept redelivering a replay that could never succeed: one run re-read
a single missing payload 12,932 times in 26 minutes, and the backend query
behind each attempt throttled its table.
The World now sends a terminal `{_error: 1, code}` frame for failures that a
retry cannot fix. Handle it:
- `payload-missing` raises `CorruptedEventLogError`, so the run fails with
`CORRUPTED_EVENT_LOG` rather than looping. The log does reference a payload
nothing can produce.
- An unknown code raises a `WorkflowWorldError` with no retryable code and no
status, which is also terminal. A future code stays safe without needing a
client release first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@changeset-bot
changeset-botBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 9bdc6be
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-astro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-express-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-hono-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-python-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-vite-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-docsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-swc-playgroundBuildingBuildingPreview, v0Aug 27, 2026 4:34pm
workflow-tarballsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-webReadyReadyPreview, v0Aug 27, 2026 4:34pm
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58 | 🔍 observability
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH | 🔍 observability
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3 | 🔍 observability
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M121D9TND7284XZVAK91FT03
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • cancelRun via CLI - cancelling a running workflow (hono)
  • cancelRun via CLI - cancelling a running workflow (nitro)
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously (astro)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (fastify)
  • hookWithSleepFinalStepWorkflow - step only on final payload (nextjs-webpack)
  • hookWorkflow (sveltekit)
  • hookWorkflow (tanstack-start)
  • hookWorkflow (vite)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (python) · at 16:34:23Z · abandoned wrun_41M1211GA50GPGVFT9JA2YMHA7 · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KT
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KV
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:34:39Z · abandoned wrun_41M12155A60GPAMCTC4YE202Y9
  • run-pickup-stall · nullByteWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550R0GN4EKX2WNAX4YXZ
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:34:40Z · abandoned wrun_41M121550E0GV6HG2GXFK06K37
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:35:12Z · abandoned wrun_41M12164S20GVYBNPPWCTT4KRE
  • run-pickup-stall · nullByteWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170DC0GRDA5F94NK7TGBN
  • run-pickup-stall · sleepingWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170E40GP6HEPXA8WH6DX0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170EC0GKV255EZM11HP19
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170F10GZKFZN2ZR7A5EQV
  • cold-start-warmup · suite warmup (tanstack-start) · at 16:36:24Z · abandoned wrun_01M12180SGAV9AGMYMQX7C5F05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:36:30Z · abandoned wrun_41M12173XJ0GX076RMR9QNDHMY
  • cold-start-warmup · suite warmup (python) · at 16:37:35Z · abandoned wrun_01M1217BCRYX9VD93DFQCWR9NA · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ5
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HQCXM5BTW9C5SSE2GS
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ4
  • run-pickup-stall · sleepingWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HN6NW9FMYWRDEJM3MA
  • run-pickup-stall · nullByteWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HTQQ8ZEHEYY85Q1ZJF
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:38:11Z · abandoned wrun_41M1219VT30GWV53B51TW334GW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:38:11Z · abandoned wrun_41M1219R9Y0GW7ZQJG2A25VXFD
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5BTNYVW1SZDE3P3FJV
  • run-pickup-stall · sleepingWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5F8WY6822ANP1QM4A4
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:38:50Z · abandoned wrun_01M121CV59Y40CW71RTTB2SX1F
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5KAGK1871VP6HT1673
  • run-pickup-stall · nullByteWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5QQWXMDA3XS0XZR0SX
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENRVFWKZDQ6BC14ANRYM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENS367D49TBR1SKBS650
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:39:50Z · abandoned wrun_01M121ENS6XXHMNS0JY4DP59HN
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4C5D73E67TR9ZAKFW4
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4HM6QQWJ86GT0ZKCPH
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:40:50Z · abandoned wrun_01M121GGCAA4YW6KC5XBV80Q4H
  • run-pickup-stall · hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep (nextjs-webpack) · at 16:41:47Z · abandoned wrun_01M121J7RG4W57GGYQ2JYXETJA
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 9bdc6be · Thu, 27 Aug 2026 16:48:26 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep245 (+14%)1347 🔴 (+22%) 🔻1376 🔴 (+20%) 🔻1419 🔴 (+21%) 🔻30
TTFSstream1212 (+509%) 🔻1331 🔴 (+21%) 🔻1350 🔴 (+16%) 🔻1393 🔴 (-3.9%)30
TTFShook + stream1505 (+41%) 🔻1677 🔴 (+22%) 🔻1823 🔴 (+29%) 🔻5993 🔴 (+245%) 🔻30
Fan-out TTFSPromise.all(100 steps)593 (-8.1%)981 (-45%) 💚1742 (-4.4%)1950 (+5.8%)10
Fan-out TTLSPromise.all(100 steps)1796 (-13%)4594 (+15%)5159 (+25%) 🔻8031 (+83%) 🔻10
STSO1020 steps (inline)80 (-19%) 💚114 (-31%) 💚125 (-38%) 💚209 (-39%) 💚1019
WO1020 steps115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚1
CRTTfirst chunk (pooled)75 (-26%) 💚154 (-7.2%)351 (+74%) 🔻387 (-88%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)98 (-34%)106 (-49%)141 (-53%)446 (-14%)85.5 (-52%)10
size sweep (100/s, 160B-12KB)97 (-32%)115 (-55%)199 (-50%)593 (-55%)102 (-62%)10
replay gateway-gpt-5.4-nano-2000t (1x)150 (-5%)101 (-61%)130 (-91%)461 (-85%)176 (-50%)3
replay eve-gpt-5.6-sol-2000t (1x)104 (-24%)126 (-34%)352 (+35%)3988 (+806%)2104 (+357%)2
replay eve-gpt-5.6-sol-2000t (2x)351 (+111%)164 (-99%)219 (-99%)397 (-98%)253 (-62%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 160359ms → this run 115275ms (Δ -45084ms, -28%)
 50-100 ms ┃ main 1 this 44 +43
100-150 ms ███████████████░░░░░░░░┃ main 591 this 934 +343
150-200 ms ┃███████ main 314 this 25 -289
200-250 ms ┃█ main 78 this 13 -65
250-300 ms ┃ main 19 this 2 -17
300-350 ms ┃ main 7 this 1 -6
350-400 ms ┃ main 3 this 0 -3
400-450 ms ┃ main 2 this 0 -2
450-500 ms ┃ main 2 this 0 -2
500-550 ms ┃ main 1 this 0 -1
550-600 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ·····▁█▄▁···· 98.1 (-40%) 87 (-40%) 141 (-53%) 446 (-14%) 3000
sweep ·····▁█▅▁▁··· 103.4 (-50%) 92 (-43%) 199 (-50%) 593 (-55%) 3000
gw 1x ·····▁█▃▁▁··· 92.4 (-63%) 84 (-40%) 130 (-91%) 461 (-85%) 5295
eve 1x ·····▁█▄▁▁▁▁· 213.6 (+36%) 84 (-36%) 352 (+35%) 3988 (+806%) 5186
eve 2x ·····▁▇█▂▁··· 126.3 (-96%) 108 (-41%) 219 (-99%) 397 (-98%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▇▃█▆▄▂▂▂▂▁ 83–124ms
sweep ▇▅▂▇█▃▁▂▄▂ 88–123ms
gw 1x █▂▂▁▃▂▁▃▂▂ 82–129ms
eve 1x █▁▁▁▁▁▁▁▁▁ 78–1251ms
eve 2x ▄▂▂▂▂▃▄█▁▄ 113–156ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep █▇▅▂▁▁▃ 101–107ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▃▆█▃▃▃▃▄▁ 22–34ms
sweep ▄▂▄█▅▆▃▃▇▁ 32–43ms
gw 1x █▄▂▁▅▄▄▄▃▄ 20–31ms
eve 1x █▂▂▁▂▁▁▂▂▂ 17–60ms
eve 2x ▆▅▃▅█▄▄▆▁▆ 17–25ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@VaguelySerious@claude
Revert the world-vercel URL override to empty
f68d1a4
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySerious
VaguelySerious
marked this pull request as ready for review
August 25, 2026 21:51
karthikscale3
karthikscale3
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 27, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 1c44cc8 due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
1c44cc8c3f76f81553d04d21f56da1d2adbc1746


This was referenced Aug 27, 2026
 Open

 Open

 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@alangenfeldalangenfeldalangenfeld approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@karthikscale3karthikscale3Awaiting requested review from karthikscale3

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@VaguelySerious@alangenfeld@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[world-vercel] Fail the run on a lost event payload instead of retrying forever - #3742
Merged
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Aug 27, 2026
Merged
[world-vercel] Fail the run on a lost event payload instead of retrying forever#3742
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Conversation
@VaguelySerious
@VaguelySeriousVaguelySerious
commented
Aug 22, 2026•

edited by alangenfeld

Loading

Copy link


Copy Markdown

Member
Summary & Motivation
The backend now closes a frame stream with a terminal {_error: 1, code} frame instead of a truncated body, which previously read as a dropped socket. payload-missing becomes a CorruptedEventLogError so the run fails with CORRUPTED_EVENT_LOG; any unrecognized code becomes a terminal WorkflowWorldError, so a future code can't turn into a redelivery loop.
Test Plan
Tests added for both codes in the v4 events reader and for the retry/terminal classification in core; typecheck clean.
Docs Preview
PagePreview
Errors and retrieshttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/foundations/errors-and-retries#error-codes
Corrupted event loghttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/errors/corrupted-event-log
@VaguelySerious@claude
fix(world-vercel): fail the run on a lost event payload instead of re…
3ea2135
…trying
A frame stream that dies mid-body reaches us as a truncated response, which
is exactly what a dropped socket looks like. So an event whose stored payload
is permanently gone was indistinguishable from a transient blip, and the
runtime kept redelivering a replay that could never succeed: one run re-read
a single missing payload 12,932 times in 26 minutes, and the backend query
behind each attempt throttled its table.
The World now sends a terminal `{_error: 1, code}` frame for failures that a
retry cannot fix. Handle it:
- `payload-missing` raises `CorruptedEventLogError`, so the run fails with
`CORRUPTED_EVENT_LOG` rather than looping. The log does reference a payload
nothing can produce.
- An unknown code raises a `WorkflowWorldError` with no retryable code and no
status, which is also terminal. A future code stays safe without needing a
client release first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@changeset-bot
changeset-botBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 9bdc6be
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-astro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-express-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-hono-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-python-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-vite-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-docsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-swc-playgroundBuildingBuildingPreview, v0Aug 27, 2026 4:34pm
workflow-tarballsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-webReadyReadyPreview, v0Aug 27, 2026 4:34pm
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58 | 🔍 observability
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH | 🔍 observability
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3 | 🔍 observability
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M121D9TND7284XZVAK91FT03
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • cancelRun via CLI - cancelling a running workflow (hono)
  • cancelRun via CLI - cancelling a running workflow (nitro)
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously (astro)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (fastify)
  • hookWithSleepFinalStepWorkflow - step only on final payload (nextjs-webpack)
  • hookWorkflow (sveltekit)
  • hookWorkflow (tanstack-start)
  • hookWorkflow (vite)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (python) · at 16:34:23Z · abandoned wrun_41M1211GA50GPGVFT9JA2YMHA7 · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KT
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KV
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:34:39Z · abandoned wrun_41M12155A60GPAMCTC4YE202Y9
  • run-pickup-stall · nullByteWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550R0GN4EKX2WNAX4YXZ
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:34:40Z · abandoned wrun_41M121550E0GV6HG2GXFK06K37
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:35:12Z · abandoned wrun_41M12164S20GVYBNPPWCTT4KRE
  • run-pickup-stall · nullByteWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170DC0GRDA5F94NK7TGBN
  • run-pickup-stall · sleepingWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170E40GP6HEPXA8WH6DX0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170EC0GKV255EZM11HP19
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170F10GZKFZN2ZR7A5EQV
  • cold-start-warmup · suite warmup (tanstack-start) · at 16:36:24Z · abandoned wrun_01M12180SGAV9AGMYMQX7C5F05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:36:30Z · abandoned wrun_41M12173XJ0GX076RMR9QNDHMY
  • cold-start-warmup · suite warmup (python) · at 16:37:35Z · abandoned wrun_01M1217BCRYX9VD93DFQCWR9NA · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ5
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HQCXM5BTW9C5SSE2GS
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ4
  • run-pickup-stall · sleepingWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HN6NW9FMYWRDEJM3MA
  • run-pickup-stall · nullByteWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HTQQ8ZEHEYY85Q1ZJF
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:38:11Z · abandoned wrun_41M1219VT30GWV53B51TW334GW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:38:11Z · abandoned wrun_41M1219R9Y0GW7ZQJG2A25VXFD
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5BTNYVW1SZDE3P3FJV
  • run-pickup-stall · sleepingWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5F8WY6822ANP1QM4A4
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:38:50Z · abandoned wrun_01M121CV59Y40CW71RTTB2SX1F
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5KAGK1871VP6HT1673
  • run-pickup-stall · nullByteWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5QQWXMDA3XS0XZR0SX
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENRVFWKZDQ6BC14ANRYM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENS367D49TBR1SKBS650
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:39:50Z · abandoned wrun_01M121ENS6XXHMNS0JY4DP59HN
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4C5D73E67TR9ZAKFW4
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4HM6QQWJ86GT0ZKCPH
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:40:50Z · abandoned wrun_01M121GGCAA4YW6KC5XBV80Q4H
  • run-pickup-stall · hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep (nextjs-webpack) · at 16:41:47Z · abandoned wrun_01M121J7RG4W57GGYQ2JYXETJA
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 9bdc6be · Thu, 27 Aug 2026 16:48:26 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep245 (+14%)1347 🔴 (+22%) 🔻1376 🔴 (+20%) 🔻1419 🔴 (+21%) 🔻30
TTFSstream1212 (+509%) 🔻1331 🔴 (+21%) 🔻1350 🔴 (+16%) 🔻1393 🔴 (-3.9%)30
TTFShook + stream1505 (+41%) 🔻1677 🔴 (+22%) 🔻1823 🔴 (+29%) 🔻5993 🔴 (+245%) 🔻30
Fan-out TTFSPromise.all(100 steps)593 (-8.1%)981 (-45%) 💚1742 (-4.4%)1950 (+5.8%)10
Fan-out TTLSPromise.all(100 steps)1796 (-13%)4594 (+15%)5159 (+25%) 🔻8031 (+83%) 🔻10
STSO1020 steps (inline)80 (-19%) 💚114 (-31%) 💚125 (-38%) 💚209 (-39%) 💚1019
WO1020 steps115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚1
CRTTfirst chunk (pooled)75 (-26%) 💚154 (-7.2%)351 (+74%) 🔻387 (-88%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)98 (-34%)106 (-49%)141 (-53%)446 (-14%)85.5 (-52%)10
size sweep (100/s, 160B-12KB)97 (-32%)115 (-55%)199 (-50%)593 (-55%)102 (-62%)10
replay gateway-gpt-5.4-nano-2000t (1x)150 (-5%)101 (-61%)130 (-91%)461 (-85%)176 (-50%)3
replay eve-gpt-5.6-sol-2000t (1x)104 (-24%)126 (-34%)352 (+35%)3988 (+806%)2104 (+357%)2
replay eve-gpt-5.6-sol-2000t (2x)351 (+111%)164 (-99%)219 (-99%)397 (-98%)253 (-62%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 160359ms → this run 115275ms (Δ -45084ms, -28%)
 50-100 ms ┃ main 1 this 44 +43
100-150 ms ███████████████░░░░░░░░┃ main 591 this 934 +343
150-200 ms ┃███████ main 314 this 25 -289
200-250 ms ┃█ main 78 this 13 -65
250-300 ms ┃ main 19 this 2 -17
300-350 ms ┃ main 7 this 1 -6
350-400 ms ┃ main 3 this 0 -3
400-450 ms ┃ main 2 this 0 -2
450-500 ms ┃ main 2 this 0 -2
500-550 ms ┃ main 1 this 0 -1
550-600 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ·····▁█▄▁···· 98.1 (-40%) 87 (-40%) 141 (-53%) 446 (-14%) 3000
sweep ·····▁█▅▁▁··· 103.4 (-50%) 92 (-43%) 199 (-50%) 593 (-55%) 3000
gw 1x ·····▁█▃▁▁··· 92.4 (-63%) 84 (-40%) 130 (-91%) 461 (-85%) 5295
eve 1x ·····▁█▄▁▁▁▁· 213.6 (+36%) 84 (-36%) 352 (+35%) 3988 (+806%) 5186
eve 2x ·····▁▇█▂▁··· 126.3 (-96%) 108 (-41%) 219 (-99%) 397 (-98%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▇▃█▆▄▂▂▂▂▁ 83–124ms
sweep ▇▅▂▇█▃▁▂▄▂ 88–123ms
gw 1x █▂▂▁▃▂▁▃▂▂ 82–129ms
eve 1x █▁▁▁▁▁▁▁▁▁ 78–1251ms
eve 2x ▄▂▂▂▂▃▄█▁▄ 113–156ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep █▇▅▂▁▁▃ 101–107ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▃▆█▃▃▃▃▄▁ 22–34ms
sweep ▄▂▄█▅▆▃▃▇▁ 32–43ms
gw 1x █▄▂▁▅▄▄▄▃▄ 20–31ms
eve 1x █▂▂▁▂▁▁▂▂▂ 17–60ms
eve 2x ▆▅▃▅█▄▄▆▁▆ 17–25ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@VaguelySerious@claude
Revert the world-vercel URL override to empty
f68d1a4
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySerious
VaguelySerious
marked this pull request as ready for review
August 25, 2026 21:51
karthikscale3
karthikscale3
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 27, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 1c44cc8 due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
1c44cc8c3f76f81553d04d21f56da1d2adbc1746


This was referenced Aug 27, 2026
 Open

 Open

 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@alangenfeldalangenfeldalangenfeld approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@karthikscale3karthikscale3Awaiting requested review from karthikscale3

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@VaguelySerious@alangenfeld@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
[world-vercel] Fail the run on a lost event payload instead of retrying forever - #3742
Merged
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Aug 27, 2026
Merged
[world-vercel] Fail the run on a lost event payload instead of retrying forever#3742
alangenfeld merged 5 commits into
mainfrom 
peter/v4-terminal-payload-missing
Conversation
@VaguelySerious
@VaguelySeriousVaguelySerious
commented
Aug 22, 2026•

edited by alangenfeld

Loading

Copy link


Copy Markdown

Member
Summary & Motivation
The backend now closes a frame stream with a terminal {_error: 1, code} frame instead of a truncated body, which previously read as a dropped socket. payload-missing becomes a CorruptedEventLogError so the run fails with CORRUPTED_EVENT_LOG; any unrecognized code becomes a terminal WorkflowWorldError, so a future code can't turn into a redelivery loop.
Test Plan
Tests added for both codes in the v4 events reader and for the retry/terminal classification in core; typecheck clean.
Docs Preview
PagePreview
Errors and retrieshttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/foundations/errors-and-retries#error-codes
Corrupted event loghttps://workflow-docs-git-peter-v4-terminal-payload-missing.vercel.sh/docs/errors/corrupted-event-log
@VaguelySerious@claude
fix(world-vercel): fail the run on a lost event payload instead of re…
3ea2135
…trying
A frame stream that dies mid-body reaches us as a truncated response, which
is exactly what a dropped socket looks like. So an event whose stored payload
is permanently gone was indistinguishable from a transient blip, and the
runtime kept redelivering a replay that could never succeed: one run re-read
a single missing payload 12,932 times in 26 minutes, and the backend query
behind each attempt throttled its table.
The World now sends a terminal `{_error: 1, code}` frame for failures that a
retry cannot fix. Handle it:
- `payload-missing` raises `CorruptedEventLogError`, so the run fails with
`CORRUPTED_EVENT_LOG` rather than looping. The log does reference a payload
nothing can produce.
- An unknown code raises a `WorkflowWorldError` with no retryable code and no
status, which is also terminal. A future code stays safe without needing a
client release first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@changeset-bot
changeset-botBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 9bdc6be
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 17 packages
NameType
@workflow/world-vercelPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 27, 2026 4:34pm
example-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-astro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-express-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-hono-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-python-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workbench-vite-workflowReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-docsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-swc-playgroundBuildingBuildingPreview, v0Aug 27, 2026 4:34pm
workflow-tarballsReadyReadyPreview, v0Aug 27, 2026 4:34pm
workflow-webReadyReadyPreview, v0Aug 27, 2026 4:34pm
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58 | 🔍 observability
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH | 🔍 observability
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3 | 🔍 observability
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099 | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M121D9TND7284XZVAK91FT03
  • promiseAllWorkflow | wrun_41M1213GJS0GKEJZ508MYC3B58
  • sleepingWorkflow | wrun_41M12143770GP2H8E97SNB1KCH
  • parallelSleepWorkflow | wrun_41M12143MB0GK7TGB3TY5D1JS3
  • nullByteWorkflow | wrun_41M12149NG0GYB0P5SMXPR7F5W
  • cancelRun - cancelling a running workflow | wrun_41M1218AY40GJEVKXGGDZZXWHY
  • cancelRun via CLI - cancelling a running workflow | wrun_41M1218BMN0GH9RTJRD6QH8DEK
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M1218GYT0GJG8A4EH3RCGWQX
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M121923R0GW62HC7GV5V8099
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • cancelRun via CLI - cancelling a running workflow (hono)
  • cancelRun via CLI - cancelling a running workflow (nitro)
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously (astro)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (fastify)
  • hookWithSleepFinalStepWorkflow - step only on final payload (nextjs-webpack)
  • hookWorkflow (sveltekit)
  • hookWorkflow (tanstack-start)
  • hookWorkflow (vite)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (python) · at 16:34:23Z · abandoned wrun_41M1211GA50GPGVFT9JA2YMHA7 · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KT
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550M0GNDHJDY27BW71KV
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:34:39Z · abandoned wrun_41M12155A60GPAMCTC4YE202Y9
  • run-pickup-stall · nullByteWorkflow (python) · at 16:34:39Z · abandoned wrun_41M121550R0GN4EKX2WNAX4YXZ
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:34:40Z · abandoned wrun_41M121550E0GV6HG2GXFK06K37
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:35:12Z · abandoned wrun_41M12164S20GVYBNPPWCTT4KRE
  • run-pickup-stall · nullByteWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170DC0GRDA5F94NK7TGBN
  • run-pickup-stall · sleepingWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170E40GP6HEPXA8WH6DX0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170EC0GKV255EZM11HP19
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:35:40Z · abandoned wrun_41M12170F10GZKFZN2ZR7A5EQV
  • cold-start-warmup · suite warmup (tanstack-start) · at 16:36:24Z · abandoned wrun_01M12180SGAV9AGMYMQX7C5F05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:36:30Z · abandoned wrun_41M12173XJ0GX076RMR9QNDHMY
  • cold-start-warmup · suite warmup (python) · at 16:37:35Z · abandoned wrun_01M1217BCRYX9VD93DFQCWR9NA · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ5
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HQCXM5BTW9C5SSE2GS
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:37:50Z · abandoned wrun_01M121B0HG9PM2RF8XXXJTNFZ4
  • run-pickup-stall · sleepingWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HN6NW9FMYWRDEJM3MA
  • run-pickup-stall · nullByteWorkflow (python) · at 16:37:50Z · abandoned wrun_01M121B0HTQQ8ZEHEYY85Q1ZJF
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:38:11Z · abandoned wrun_41M1219VT30GWV53B51TW334GW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:38:11Z · abandoned wrun_41M1219R9Y0GW7ZQJG2A25VXFD
  • run-pickup-stall · promiseAllWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5BTNYVW1SZDE3P3FJV
  • run-pickup-stall · sleepingWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5F8WY6822ANP1QM4A4
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 16:38:50Z · abandoned wrun_01M121CV59Y40CW71RTTB2SX1F
  • run-pickup-stall · parallelSleepWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5KAGK1871VP6HT1673
  • run-pickup-stall · nullByteWorkflow (python) · at 16:38:50Z · abandoned wrun_01M121CV5QQWXMDA3XS0XZR0SX
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENRVFWKZDQ6BC14ANRYM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:39:50Z · abandoned wrun_01M121ENS367D49TBR1SKBS650
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:39:50Z · abandoned wrun_01M121ENS6XXHMNS0JY4DP59HN
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4C5D73E67TR9ZAKFW4
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 16:40:20Z · abandoned wrun_01M121FK4HM6QQWJ86GT0ZKCPH
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 16:40:50Z · abandoned wrun_01M121GGCAA4YW6KC5XBV80Q4H
  • run-pickup-stall · hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep (nextjs-webpack) · at 16:41:47Z · abandoned wrun_01M121J7RG4W57GGYQ2JYXETJA
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 9bdc6be · Thu, 27 Aug 2026 16:48:26 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep245 (+14%)1347 🔴 (+22%) 🔻1376 🔴 (+20%) 🔻1419 🔴 (+21%) 🔻30
TTFSstream1212 (+509%) 🔻1331 🔴 (+21%) 🔻1350 🔴 (+16%) 🔻1393 🔴 (-3.9%)30
TTFShook + stream1505 (+41%) 🔻1677 🔴 (+22%) 🔻1823 🔴 (+29%) 🔻5993 🔴 (+245%) 🔻30
Fan-out TTFSPromise.all(100 steps)593 (-8.1%)981 (-45%) 💚1742 (-4.4%)1950 (+5.8%)10
Fan-out TTLSPromise.all(100 steps)1796 (-13%)4594 (+15%)5159 (+25%) 🔻8031 (+83%) 🔻10
STSO1020 steps (inline)80 (-19%) 💚114 (-31%) 💚125 (-38%) 💚209 (-39%) 💚1019
WO1020 steps115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚115486 (-28%) 💚1
CRTTfirst chunk (pooled)75 (-26%) 💚154 (-7.2%)351 (+74%) 🔻387 (-88%) 💚28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)98 (-34%)106 (-49%)141 (-53%)446 (-14%)85.5 (-52%)10
size sweep (100/s, 160B-12KB)97 (-32%)115 (-55%)199 (-50%)593 (-55%)102 (-62%)10
replay gateway-gpt-5.4-nano-2000t (1x)150 (-5%)101 (-61%)130 (-91%)461 (-85%)176 (-50%)3
replay eve-gpt-5.6-sol-2000t (1x)104 (-24%)126 (-34%)352 (+35%)3988 (+806%)2104 (+357%)2
replay eve-gpt-5.6-sol-2000t (2x)351 (+111%)164 (-99%)219 (-99%)397 (-98%)253 (-62%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 160359ms → this run 115275ms (Δ -45084ms, -28%)
 50-100 ms ┃ main 1 this 44 +43
100-150 ms ███████████████░░░░░░░░┃ main 591 this 934 +343
150-200 ms ┃███████ main 314 this 25 -289
200-250 ms ┃█ main 78 this 13 -65
250-300 ms ┃ main 19 this 2 -17
300-350 ms ┃ main 7 this 1 -6
350-400 ms ┃ main 3 this 0 -3
400-450 ms ┃ main 2 this 0 -2
450-500 ms ┃ main 2 this 0 -2
500-550 ms ┃ main 1 this 0 -1
550-600 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ·····▁█▄▁···· 98.1 (-40%) 87 (-40%) 141 (-53%) 446 (-14%) 3000
sweep ·····▁█▅▁▁··· 103.4 (-50%) 92 (-43%) 199 (-50%) 593 (-55%) 3000
gw 1x ·····▁█▃▁▁··· 92.4 (-63%) 84 (-40%) 130 (-91%) 461 (-85%) 5295
eve 1x ·····▁█▄▁▁▁▁· 213.6 (+36%) 84 (-36%) 352 (+35%) 3988 (+806%) 5186
eve 2x ·····▁▇█▂▁··· 126.3 (-96%) 108 (-41%) 219 (-99%) 397 (-98%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▇▃█▆▄▂▂▂▂▁ 83–124ms
sweep ▇▅▂▇█▃▁▂▄▂ 88–123ms
gw 1x █▂▂▁▃▂▁▃▂▂ 82–129ms
eve 1x █▁▁▁▁▁▁▁▁▁ 78–1251ms
eve 2x ▄▂▂▂▂▃▄█▁▄ 113–156ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep █▇▅▂▁▁▃ 101–107ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▃▆█▃▃▃▃▄▁ 22–34ms
sweep ▄▂▄█▅▆▃▃▇▁ 32–43ms
gw 1x █▄▂▁▅▄▄▄▃▄ 20–31ms
eve 1x █▂▂▁▂▁▁▂▂▂ 17–60ms
eve 2x ▆▅▃▅█▄▄▆▁▆ 17–25ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@VaguelySerious@claude
Revert the world-vercel URL override to empty
f68d1a4
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySerious
VaguelySerious
marked this pull request as ready for review
August 25, 2026 21:51
karthikscale3
karthikscale3
reviewed
Aug 25, 2026
View reviewed changes
Comment threadpackages/world-vercel/src/events-v4.ts
Outdated

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 27, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 27, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 1c44cc8 due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
1c44cc8c3f76f81553d04d21f56da1d2adbc1746


This was referenced Aug 27, 2026
 Open

 Open

 Open

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@alangenfeldalangenfeldalangenfeld approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@karthikscale3karthikscale3Awaiting requested review from karthikscale3

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@VaguelySerious@alangenfeld@karthikscale3