[core] Retain workflow VMs across hooks - #3604

Merged
NathanColosimo merged 8 commits into
mainfrom
codex/retain-vm-with-hooks
Aug 25, 2026
Merged

[core] Retain workflow VMs across hooks#3604
NathanColosimo merged 8 commits into
mainfrom
codex/retain-vm-with-hooks

Conversation

@NathanColosimo

@NathanColosimoNathanColosimo commented Aug 17, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • retain a workflow VM across safe step boundaries that also contain hooks
  • generation-guard step and hook suspension signals so stale callbacks cannot demote a resumed session
  • include hook metadata and abort-reason serialization in the same passivity gate as step inputs
  • replay hook-only, wait, attribute, unsafe-serialization, and WORKFLOW_RETAINED_VM=0 boundaries

Why

Spec 6 event logs are monotone prefixes. A hook_received event that lands after the VM consumed its prefix extends that prefix on the next delta without invalidating the retained session, so an open hook does not itself require replay.

Retention still requires a step boundary. Inline step bodies keep awaiting an accepted step_started ownership claim when an open hook can wake a competing invocation, and resume keeps the existing exact-prefix extension check.

Verification

  • pnpm --filter @workflow/core test — 2,171 passed, 3 expected failures
  • pnpm --filter @workflow/core typecheck
  • pnpm turbo build --filter=@workflow/core... — 7 tasks passed
  • pnpm lint — passed with baseline advisory diagnostics
  • pnpm changeset status --since=origin/codex/speed-fresh-replay

Docs Preview

PagePreview
Runtime tuningRetained VM behavior
What's new in v5Faster and cheaper runs

@vercel

vercelBot commented Aug 17, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 9:07pm

@changeset-bot

changeset-botBot commented Aug 17, 2026•
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 80d157c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actionsBot commented Aug 17, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (8 failed)

python-node (8 failed):

  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5 | 🔍 observability
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0XC5XYM1YQE461MKEBB66M5
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

  • addTenWorkflow (astro)
  • cancelRun via CLI - cancelling a running workflow (express)
  • cancelRun via CLI - cancelling a running workflow (nextjs-webpack)
  • outputStreamInsideStepWorkflow - getWritable() called inside step functions (nest)
  • promiseAllWorkflow (nuxt)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:06:37Z · abandoned wrun_41M0XBTHM90GKEHFEBG0P3WRYH · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QZ0GWQ1VV3J60KTB3F
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QR0GS1012PB1XDMC78
  • run-pickup-stall · nullByteWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6R30GK1D1BKMT27DP2W
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY6R00GW8D3T8SG4KMXS6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY79W0GRS0AJ6WES3QYWB
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:07:28Z · abandoned wrun_41M0XBZ7TA0GHY54WQK17ZC84M
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02800GQAP9H12M3DZ100
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02DB0GKQ2ZVJDJQWTKTR
  • run-pickup-stall · nullByteWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02D70GRMT4K0ERWK650F
  • run-pickup-stall · sleepingWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02A30GHBD095MY9ZZM3N
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:08:45Z · abandoned wrun_41M0XC083N0GV495QNVSJHN6BM
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:08:47Z · abandoned wrun_01M0XC1ABA2NZRP3XR2YHZEZKK
  • run-pickup-stall · hookWorkflow (nuxt) · at 21:09:23Z · abandoned wrun_01M0XC2T3CVPEJCQH7GTE69QJG
  • cold-start-warmup · suite warmup (python) · at 21:09:36Z · abandoned wrun_01M0XBZZGYKN4RPTB7CJ3PNA52 · (+7 more)
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNHQ4KSAYN4F95P8NG2
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNJ4G7N6H02QPT80W0E
  • run-pickup-stall · sleepingWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNRZD1GW2CBRY37HGPY
  • run-pickup-stall · nullByteWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNW7S8NT7M3WCCBA2HA
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNT1GE0NWERH03VCCKW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:10:27Z · abandoned wrun_41M0XC2T4H0GTR8DGY9NS3Z2P7
  • run-pickup-stall · sleepingWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9GZCZCD7P53QG1HPAV
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9J7VNK34BM5NCRYV2K
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9KSPCVY5W465DZ8GW8
  • run-pickup-stall · nullByteWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9M06Y972WENA3NR7M1
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9EVY18HHV93RCRC3PN
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:07Z · abandoned wrun_41M0XC30070GGPHQ1573BT7VKH
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79X66B7VSG24T5PTYN6M
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79XC7S7S5P4CF68FPRGR
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:11:51Z · abandoned wrun_01M0XC79XKXT2S6ZT3ZYDHCK93
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879903Z8AF94AM73K33B
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879BZBMDVR5B8WTZK5KK
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:12:51Z · abandoned wrun_01M0XC94GQ8DS7QK6SQG7K5G1X
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:13:33Z · abandoned wrun_01M0XCADDAH0S0PB1KYMG5FWPZ

E2E Test Summary

Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category

❌ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000

❌ 🌐 Cross-language Conformance

AppPassedFailedSkipped
❌ python09132

✅ vercel-http-transport

AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

✅ vercel-ws-transport

AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028

📋 View full workflow run

@github-actions

github-actionsBot commented Aug 17, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0

Full trace: world-sim.txt

@NathanColosimo
NathanColosimoforce-pushed the codex/retain-vm-with-hooks branch from ce4631e to 56c6c60CompareAugust 18, 2026 22:27
@NathanColosimo
NathanColosimoforce-pushed the codex/retain-vm-with-hooks branch from 56c6c60 to 6f7a649CompareAugust 18, 2026 22:41
@NathanColosimo
NathanColosimoforce-pushed the codex/retain-vm-with-hooks branch from eb07b94 to 3c96fc3CompareAugust 21, 2026 10:51
This was referenced Aug 21, 2026
Merged
Merged
VaguelySerious
VaguelySerious reviewed Aug 21, 2026
View reviewed changes

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found

Comment threadpackages/core/src/global.ts
Comment threadpackages/core/src/runtime.ts
Comment threadpackages/core/src/runtime.ts Outdated
Comment threadpackages/core/src/retained-vm-loop.test.ts
@VaguelySeriousVaguelySerious mentioned this pull request Aug 21, 2026
Merged
@github-actions

github-actionsBot commented Aug 24, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Event Log Race Repro

  • vercel clean, 20 runs
  • local clean, 20 runs
  • postgres clean, 20 runs

Run History

RunLaneTotalCompleteCorruptStuckOther
08-25 00:13vercel2020000
local2020000
postgres2020000
08-25 03:04vercel2020000
local2020000
postgres2020000
08-25 19:31vercel1212000
local77000
postgres2019001
08-25 19:40vercel2020000
local2020000
postgres2020000
08-25 21:13vercel2020000
local2020000
postgres2020000
Config

vercel: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 240000ms

local, postgres: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 480000ms

VaguelySerious
VaguelySerious approved these changes Aug 24, 2026
View reviewed changes

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM assuming e2e passes (except python)

TooTallNate
TooTallNate reviewed Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/global.ts
TooTallNate
TooTallNate reviewed Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/runtime.ts
TooTallNate
TooTallNate previously requested changes Aug 24, 2026
View reviewed changes

@TooTallNateTooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI: Request changes. The PR currently breaks the public WorkflowSuspension.steps API under a patch changeset and leaves retained-VM documentation inconsistent with the new hook behavior. Details are inline.

vercel[bot]
vercelBot reviewed Aug 25, 2026
View reviewed changes

@vercelvercelBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Suggestion:

Docs incorrectly claim hook-involving suspensions always force a full replay, contradicting the new retention policy that retains the VM across step boundaries containing open hooks.

Fix on Vercel

Comment threadpackages/core/src/global.ts
@github-actions

github-actionsBot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.5 KiB (±0)1.76 MiB (-103 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)763.2 KiB (+351 B)
About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

80d157c · run

@github-actions

github-actionsBot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

No backport to stable for 556f3f0 (AI decision).

This is a performance enhancement that broadens the retained-VM fast path to step boundaries with open hooks, plus a new items alias on WorkflowSuspension and richer retention diagnostics — capability work, not a fix for a user-visible defect. It also builds directly on main-only infrastructure: origin/stable has no retained-vm-loop.test.ts, no suspensionGeneration on the suspension path, and no docs/content/docs/v5/ pages the commit updates.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

556f3f080a33f8673cf3c08805997dba4e4d865e

@github-actionsgithub-actionsBot mentioned this pull request Aug 26, 2026
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@TooTallNateTooTallNateAwaiting requested review from TooTallNate

Assignees

No one assigned

Labels

event-log-race-reproRun the event log race reproduction job

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@TooTallNate@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
[core] Retain workflow VMs across hooks - #3604
Merged
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Aug 25, 2026
Merged
[core] Retain workflow VMs across hooks#3604
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • retain a workflow VM across safe step boundaries that also contain hooks
  • generation-guard step and hook suspension signals so stale callbacks cannot demote a resumed session
  • include hook metadata and abort-reason serialization in the same passivity gate as step inputs
  • replay hook-only, wait, attribute, unsafe-serialization, and WORKFLOW_RETAINED_VM=0 boundaries
Why
Spec 6 event logs are monotone prefixes. A hook_received event that lands after the VM consumed its prefix extends that prefix on the next delta without invalidating the retained session, so an open hook does not itself require replay.
Retention still requires a step boundary. Inline step bodies keep awaiting an accepted step_started ownership claim when an open hook can wake a competing invocation, and resume keeps the existing exact-prefix extension check.
Verification
  • pnpm --filter @workflow/core test — 2,171 passed, 3 expected failures
  • pnpm --filter @workflow/core typecheck
  • pnpm turbo build --filter=@workflow/core... — 7 tasks passed
  • pnpm lint — passed with baseline advisory diagnostics
  • pnpm changeset status --since=origin/codex/speed-fresh-replay
Docs Preview
PagePreview
Runtime tuningRetained VM behavior
What's new in v5Faster and cheaper runs
@vercel
vercelBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 9:07pm
@changeset-bot
changeset-botBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 80d157c
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5 | 🔍 observability
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0XC5XYM1YQE461MKEBB66M5
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • addTenWorkflow (astro)
  • cancelRun via CLI - cancelling a running workflow (express)
  • cancelRun via CLI - cancelling a running workflow (nextjs-webpack)
  • outputStreamInsideStepWorkflow - getWritable() called inside step functions (nest)
  • promiseAllWorkflow (nuxt)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:06:37Z · abandoned wrun_41M0XBTHM90GKEHFEBG0P3WRYH · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QZ0GWQ1VV3J60KTB3F
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QR0GS1012PB1XDMC78
  • run-pickup-stall · nullByteWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6R30GK1D1BKMT27DP2W
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY6R00GW8D3T8SG4KMXS6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY79W0GRS0AJ6WES3QYWB
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:07:28Z · abandoned wrun_41M0XBZ7TA0GHY54WQK17ZC84M
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02800GQAP9H12M3DZ100
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02DB0GKQ2ZVJDJQWTKTR
  • run-pickup-stall · nullByteWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02D70GRMT4K0ERWK650F
  • run-pickup-stall · sleepingWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02A30GHBD095MY9ZZM3N
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:08:45Z · abandoned wrun_41M0XC083N0GV495QNVSJHN6BM
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:08:47Z · abandoned wrun_01M0XC1ABA2NZRP3XR2YHZEZKK
  • run-pickup-stall · hookWorkflow (nuxt) · at 21:09:23Z · abandoned wrun_01M0XC2T3CVPEJCQH7GTE69QJG
  • cold-start-warmup · suite warmup (python) · at 21:09:36Z · abandoned wrun_01M0XBZZGYKN4RPTB7CJ3PNA52 · (+7 more)
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNHQ4KSAYN4F95P8NG2
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNJ4G7N6H02QPT80W0E
  • run-pickup-stall · sleepingWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNRZD1GW2CBRY37HGPY
  • run-pickup-stall · nullByteWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNW7S8NT7M3WCCBA2HA
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNT1GE0NWERH03VCCKW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:10:27Z · abandoned wrun_41M0XC2T4H0GTR8DGY9NS3Z2P7
  • run-pickup-stall · sleepingWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9GZCZCD7P53QG1HPAV
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9J7VNK34BM5NCRYV2K
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9KSPCVY5W465DZ8GW8
  • run-pickup-stall · nullByteWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9M06Y972WENA3NR7M1
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9EVY18HHV93RCRC3PN
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:07Z · abandoned wrun_41M0XC30070GGPHQ1573BT7VKH
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79X66B7VSG24T5PTYN6M
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79XC7S7S5P4CF68FPRGR
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:11:51Z · abandoned wrun_01M0XC79XKXT2S6ZT3ZYDHCK93
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879903Z8AF94AM73K33B
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879BZBMDVR5B8WTZK5KK
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:12:51Z · abandoned wrun_01M0XC94GQ8DS7QK6SQG7K5G1X
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:13:33Z · abandoned wrun_01M0XCADDAH0S0PB1KYMG5FWPZ
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
ce4631e to
56c6c60CompareAugust 18, 2026 22:27
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
56c6c60 to
6f7a649CompareAugust 18, 2026 22:41
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
eb07b94 to
3c96fc3CompareAugust 21, 2026 10:51

This was referenced Aug 21, 2026
 Merged

 Merged

VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/core/src/global.ts
Comment threadpackages/core/src/runtime.ts
Comment threadpackages/core/src/runtime.ts
Outdated

Comment threadpackages/core/src/retained-vm-loop.test.ts
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 24, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Event Log Race Repro
  • vercel clean, 20 runs
  • local clean, 20 runs
  • postgres clean, 20 runs
Run History
RunLaneTotalCompleteCorruptStuckOther
08-25 00:13vercel2020000
local2020000
postgres2020000
08-25 03:04vercel2020000
local2020000
postgres2020000
08-25 19:31vercel1212000
local77000
postgres2019001
08-25 19:40vercel2020000
local2020000
postgres2020000
08-25 21:13vercel2020000
local2020000
postgres2020000
Config
vercel: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 240000ms
local, postgres: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 480000ms
VaguelySerious
VaguelySerious
approved these changes
Aug 24, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM assuming e2e passes (except python)
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/global.ts
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/runtime.ts
TooTallNate
TooTallNate
previously requested changes
Aug 24, 2026
View reviewed changes
@TooTallNateTooTallNate
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI: Request changes. The PR currently breaks the public WorkflowSuspension.steps API under a patch changeset and leaves retained-VM documentation inconsistent with the new hook behavior. Details are inline.
vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
@vercelvercelBot
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Suggestion:
Docs incorrectly claim hook-involving suspensions always force a full replay, contradicting the new retention policy that retains the VM across step boundaries containing open hooks.
Fix on Vercel
Comment threadpackages/core/src/global.ts
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.5 KiB (±0)1.76 MiB (-103 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)763.2 KiB (+351 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
80d157c · run
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 556f3f0 (AI decision).
This is a performance enhancement that broadens the retained-VM fast path to step boundaries with open hooks, plus a new items alias on WorkflowSuspension and richer retention diagnostics — capability work, not a fix for a user-visible defect. It also builds directly on main-only infrastructure: origin/stable has no retained-vm-loop.test.ts, no suspensionGeneration on the suspension path, and no docs/content/docs/v5/ pages the commit updates.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
556f3f080a33f8673cf3c08805997dba4e4d865e

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 26, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@TooTallNateTooTallNateAwaiting requested review from TooTallNate

Assignees


No one assigned


Labels

event-log-race-reproRun the event log race reproduction job

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@TooTallNate@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[core] Retain workflow VMs across hooks - #3604
Merged
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Aug 25, 2026
Merged
[core] Retain workflow VMs across hooks#3604
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • retain a workflow VM across safe step boundaries that also contain hooks
  • generation-guard step and hook suspension signals so stale callbacks cannot demote a resumed session
  • include hook metadata and abort-reason serialization in the same passivity gate as step inputs
  • replay hook-only, wait, attribute, unsafe-serialization, and WORKFLOW_RETAINED_VM=0 boundaries
Why
Spec 6 event logs are monotone prefixes. A hook_received event that lands after the VM consumed its prefix extends that prefix on the next delta without invalidating the retained session, so an open hook does not itself require replay.
Retention still requires a step boundary. Inline step bodies keep awaiting an accepted step_started ownership claim when an open hook can wake a competing invocation, and resume keeps the existing exact-prefix extension check.
Verification
  • pnpm --filter @workflow/core test — 2,171 passed, 3 expected failures
  • pnpm --filter @workflow/core typecheck
  • pnpm turbo build --filter=@workflow/core... — 7 tasks passed
  • pnpm lint — passed with baseline advisory diagnostics
  • pnpm changeset status --since=origin/codex/speed-fresh-replay
Docs Preview
PagePreview
Runtime tuningRetained VM behavior
What's new in v5Faster and cheaper runs
@vercel
vercelBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 9:07pm
@changeset-bot
changeset-botBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 80d157c
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5 | 🔍 observability
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0XC5XYM1YQE461MKEBB66M5
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • addTenWorkflow (astro)
  • cancelRun via CLI - cancelling a running workflow (express)
  • cancelRun via CLI - cancelling a running workflow (nextjs-webpack)
  • outputStreamInsideStepWorkflow - getWritable() called inside step functions (nest)
  • promiseAllWorkflow (nuxt)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:06:37Z · abandoned wrun_41M0XBTHM90GKEHFEBG0P3WRYH · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QZ0GWQ1VV3J60KTB3F
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QR0GS1012PB1XDMC78
  • run-pickup-stall · nullByteWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6R30GK1D1BKMT27DP2W
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY6R00GW8D3T8SG4KMXS6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY79W0GRS0AJ6WES3QYWB
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:07:28Z · abandoned wrun_41M0XBZ7TA0GHY54WQK17ZC84M
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02800GQAP9H12M3DZ100
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02DB0GKQ2ZVJDJQWTKTR
  • run-pickup-stall · nullByteWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02D70GRMT4K0ERWK650F
  • run-pickup-stall · sleepingWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02A30GHBD095MY9ZZM3N
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:08:45Z · abandoned wrun_41M0XC083N0GV495QNVSJHN6BM
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:08:47Z · abandoned wrun_01M0XC1ABA2NZRP3XR2YHZEZKK
  • run-pickup-stall · hookWorkflow (nuxt) · at 21:09:23Z · abandoned wrun_01M0XC2T3CVPEJCQH7GTE69QJG
  • cold-start-warmup · suite warmup (python) · at 21:09:36Z · abandoned wrun_01M0XBZZGYKN4RPTB7CJ3PNA52 · (+7 more)
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNHQ4KSAYN4F95P8NG2
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNJ4G7N6H02QPT80W0E
  • run-pickup-stall · sleepingWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNRZD1GW2CBRY37HGPY
  • run-pickup-stall · nullByteWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNW7S8NT7M3WCCBA2HA
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNT1GE0NWERH03VCCKW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:10:27Z · abandoned wrun_41M0XC2T4H0GTR8DGY9NS3Z2P7
  • run-pickup-stall · sleepingWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9GZCZCD7P53QG1HPAV
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9J7VNK34BM5NCRYV2K
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9KSPCVY5W465DZ8GW8
  • run-pickup-stall · nullByteWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9M06Y972WENA3NR7M1
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9EVY18HHV93RCRC3PN
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:07Z · abandoned wrun_41M0XC30070GGPHQ1573BT7VKH
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79X66B7VSG24T5PTYN6M
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79XC7S7S5P4CF68FPRGR
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:11:51Z · abandoned wrun_01M0XC79XKXT2S6ZT3ZYDHCK93
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879903Z8AF94AM73K33B
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879BZBMDVR5B8WTZK5KK
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:12:51Z · abandoned wrun_01M0XC94GQ8DS7QK6SQG7K5G1X
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:13:33Z · abandoned wrun_01M0XCADDAH0S0PB1KYMG5FWPZ
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
ce4631e to
56c6c60CompareAugust 18, 2026 22:27
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
56c6c60 to
6f7a649CompareAugust 18, 2026 22:41
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
eb07b94 to
3c96fc3CompareAugust 21, 2026 10:51

This was referenced Aug 21, 2026
 Merged

 Merged

VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/core/src/global.ts
Comment threadpackages/core/src/runtime.ts
Comment threadpackages/core/src/runtime.ts
Outdated

Comment threadpackages/core/src/retained-vm-loop.test.ts
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 24, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Event Log Race Repro
  • vercel clean, 20 runs
  • local clean, 20 runs
  • postgres clean, 20 runs
Run History
RunLaneTotalCompleteCorruptStuckOther
08-25 00:13vercel2020000
local2020000
postgres2020000
08-25 03:04vercel2020000
local2020000
postgres2020000
08-25 19:31vercel1212000
local77000
postgres2019001
08-25 19:40vercel2020000
local2020000
postgres2020000
08-25 21:13vercel2020000
local2020000
postgres2020000
Config
vercel: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 240000ms
local, postgres: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 480000ms
VaguelySerious
VaguelySerious
approved these changes
Aug 24, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM assuming e2e passes (except python)
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/global.ts
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/runtime.ts
TooTallNate
TooTallNate
previously requested changes
Aug 24, 2026
View reviewed changes
@TooTallNateTooTallNate
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI: Request changes. The PR currently breaks the public WorkflowSuspension.steps API under a patch changeset and leaves retained-VM documentation inconsistent with the new hook behavior. Details are inline.
vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
@vercelvercelBot
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Suggestion:
Docs incorrectly claim hook-involving suspensions always force a full replay, contradicting the new retention policy that retains the VM across step boundaries containing open hooks.
Fix on Vercel
Comment threadpackages/core/src/global.ts
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.5 KiB (±0)1.76 MiB (-103 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)763.2 KiB (+351 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
80d157c · run
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 556f3f0 (AI decision).
This is a performance enhancement that broadens the retained-VM fast path to step boundaries with open hooks, plus a new items alias on WorkflowSuspension and richer retention diagnostics — capability work, not a fix for a user-visible defect. It also builds directly on main-only infrastructure: origin/stable has no retained-vm-loop.test.ts, no suspensionGeneration on the suspension path, and no docs/content/docs/v5/ pages the commit updates.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
556f3f080a33f8673cf3c08805997dba4e4d865e

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 26, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@TooTallNateTooTallNateAwaiting requested review from TooTallNate

Assignees


No one assigned


Labels

event-log-race-reproRun the event log race reproduction job

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@TooTallNate@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[core] Retain workflow VMs across hooks - #3604
Merged
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Aug 25, 2026
Merged
[core] Retain workflow VMs across hooks#3604
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • retain a workflow VM across safe step boundaries that also contain hooks
  • generation-guard step and hook suspension signals so stale callbacks cannot demote a resumed session
  • include hook metadata and abort-reason serialization in the same passivity gate as step inputs
  • replay hook-only, wait, attribute, unsafe-serialization, and WORKFLOW_RETAINED_VM=0 boundaries
Why
Spec 6 event logs are monotone prefixes. A hook_received event that lands after the VM consumed its prefix extends that prefix on the next delta without invalidating the retained session, so an open hook does not itself require replay.
Retention still requires a step boundary. Inline step bodies keep awaiting an accepted step_started ownership claim when an open hook can wake a competing invocation, and resume keeps the existing exact-prefix extension check.
Verification
  • pnpm --filter @workflow/core test — 2,171 passed, 3 expected failures
  • pnpm --filter @workflow/core typecheck
  • pnpm turbo build --filter=@workflow/core... — 7 tasks passed
  • pnpm lint — passed with baseline advisory diagnostics
  • pnpm changeset status --since=origin/codex/speed-fresh-replay
Docs Preview
PagePreview
Runtime tuningRetained VM behavior
What's new in v5Faster and cheaper runs
@vercel
vercelBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 9:07pm
@changeset-bot
changeset-botBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 80d157c
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5 | 🔍 observability
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0XC5XYM1YQE461MKEBB66M5
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • addTenWorkflow (astro)
  • cancelRun via CLI - cancelling a running workflow (express)
  • cancelRun via CLI - cancelling a running workflow (nextjs-webpack)
  • outputStreamInsideStepWorkflow - getWritable() called inside step functions (nest)
  • promiseAllWorkflow (nuxt)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:06:37Z · abandoned wrun_41M0XBTHM90GKEHFEBG0P3WRYH · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QZ0GWQ1VV3J60KTB3F
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QR0GS1012PB1XDMC78
  • run-pickup-stall · nullByteWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6R30GK1D1BKMT27DP2W
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY6R00GW8D3T8SG4KMXS6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY79W0GRS0AJ6WES3QYWB
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:07:28Z · abandoned wrun_41M0XBZ7TA0GHY54WQK17ZC84M
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02800GQAP9H12M3DZ100
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02DB0GKQ2ZVJDJQWTKTR
  • run-pickup-stall · nullByteWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02D70GRMT4K0ERWK650F
  • run-pickup-stall · sleepingWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02A30GHBD095MY9ZZM3N
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:08:45Z · abandoned wrun_41M0XC083N0GV495QNVSJHN6BM
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:08:47Z · abandoned wrun_01M0XC1ABA2NZRP3XR2YHZEZKK
  • run-pickup-stall · hookWorkflow (nuxt) · at 21:09:23Z · abandoned wrun_01M0XC2T3CVPEJCQH7GTE69QJG
  • cold-start-warmup · suite warmup (python) · at 21:09:36Z · abandoned wrun_01M0XBZZGYKN4RPTB7CJ3PNA52 · (+7 more)
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNHQ4KSAYN4F95P8NG2
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNJ4G7N6H02QPT80W0E
  • run-pickup-stall · sleepingWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNRZD1GW2CBRY37HGPY
  • run-pickup-stall · nullByteWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNW7S8NT7M3WCCBA2HA
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNT1GE0NWERH03VCCKW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:10:27Z · abandoned wrun_41M0XC2T4H0GTR8DGY9NS3Z2P7
  • run-pickup-stall · sleepingWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9GZCZCD7P53QG1HPAV
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9J7VNK34BM5NCRYV2K
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9KSPCVY5W465DZ8GW8
  • run-pickup-stall · nullByteWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9M06Y972WENA3NR7M1
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9EVY18HHV93RCRC3PN
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:07Z · abandoned wrun_41M0XC30070GGPHQ1573BT7VKH
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79X66B7VSG24T5PTYN6M
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79XC7S7S5P4CF68FPRGR
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:11:51Z · abandoned wrun_01M0XC79XKXT2S6ZT3ZYDHCK93
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879903Z8AF94AM73K33B
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879BZBMDVR5B8WTZK5KK
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:12:51Z · abandoned wrun_01M0XC94GQ8DS7QK6SQG7K5G1X
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:13:33Z · abandoned wrun_01M0XCADDAH0S0PB1KYMG5FWPZ
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
ce4631e to
56c6c60CompareAugust 18, 2026 22:27
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
56c6c60 to
6f7a649CompareAugust 18, 2026 22:41
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
eb07b94 to
3c96fc3CompareAugust 21, 2026 10:51

This was referenced Aug 21, 2026
 Merged

 Merged

VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/core/src/global.ts
Comment threadpackages/core/src/runtime.ts
Comment threadpackages/core/src/runtime.ts
Outdated

Comment threadpackages/core/src/retained-vm-loop.test.ts
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 24, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Event Log Race Repro
  • vercel clean, 20 runs
  • local clean, 20 runs
  • postgres clean, 20 runs
Run History
RunLaneTotalCompleteCorruptStuckOther
08-25 00:13vercel2020000
local2020000
postgres2020000
08-25 03:04vercel2020000
local2020000
postgres2020000
08-25 19:31vercel1212000
local77000
postgres2019001
08-25 19:40vercel2020000
local2020000
postgres2020000
08-25 21:13vercel2020000
local2020000
postgres2020000
Config
vercel: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 240000ms
local, postgres: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 480000ms
VaguelySerious
VaguelySerious
approved these changes
Aug 24, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM assuming e2e passes (except python)
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/global.ts
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/runtime.ts
TooTallNate
TooTallNate
previously requested changes
Aug 24, 2026
View reviewed changes
@TooTallNateTooTallNate
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI: Request changes. The PR currently breaks the public WorkflowSuspension.steps API under a patch changeset and leaves retained-VM documentation inconsistent with the new hook behavior. Details are inline.
vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
@vercelvercelBot
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Suggestion:
Docs incorrectly claim hook-involving suspensions always force a full replay, contradicting the new retention policy that retains the VM across step boundaries containing open hooks.
Fix on Vercel
Comment threadpackages/core/src/global.ts
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.5 KiB (±0)1.76 MiB (-103 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)763.2 KiB (+351 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
80d157c · run
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 556f3f0 (AI decision).
This is a performance enhancement that broadens the retained-VM fast path to step boundaries with open hooks, plus a new items alias on WorkflowSuspension and richer retention diagnostics — capability work, not a fix for a user-visible defect. It also builds directly on main-only infrastructure: origin/stable has no retained-vm-loop.test.ts, no suspensionGeneration on the suspension path, and no docs/content/docs/v5/ pages the commit updates.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
556f3f080a33f8673cf3c08805997dba4e4d865e

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 26, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@TooTallNateTooTallNateAwaiting requested review from TooTallNate

Assignees


No one assigned


Labels

event-log-race-reproRun the event log race reproduction job

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@TooTallNate@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
[core] Retain workflow VMs across hooks - #3604
Merged
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Aug 25, 2026
Merged
[core] Retain workflow VMs across hooks#3604
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • retain a workflow VM across safe step boundaries that also contain hooks
  • generation-guard step and hook suspension signals so stale callbacks cannot demote a resumed session
  • include hook metadata and abort-reason serialization in the same passivity gate as step inputs
  • replay hook-only, wait, attribute, unsafe-serialization, and WORKFLOW_RETAINED_VM=0 boundaries
Why
Spec 6 event logs are monotone prefixes. A hook_received event that lands after the VM consumed its prefix extends that prefix on the next delta without invalidating the retained session, so an open hook does not itself require replay.
Retention still requires a step boundary. Inline step bodies keep awaiting an accepted step_started ownership claim when an open hook can wake a competing invocation, and resume keeps the existing exact-prefix extension check.
Verification
  • pnpm --filter @workflow/core test — 2,171 passed, 3 expected failures
  • pnpm --filter @workflow/core typecheck
  • pnpm turbo build --filter=@workflow/core... — 7 tasks passed
  • pnpm lint — passed with baseline advisory diagnostics
  • pnpm changeset status --since=origin/codex/speed-fresh-replay
Docs Preview
PagePreview
Runtime tuningRetained VM behavior
What's new in v5Faster and cheaper runs
@vercel
vercelBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 9:07pm
@changeset-bot
changeset-botBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 80d157c
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5 | 🔍 observability
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0XC5XYM1YQE461MKEBB66M5
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • addTenWorkflow (astro)
  • cancelRun via CLI - cancelling a running workflow (express)
  • cancelRun via CLI - cancelling a running workflow (nextjs-webpack)
  • outputStreamInsideStepWorkflow - getWritable() called inside step functions (nest)
  • promiseAllWorkflow (nuxt)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:06:37Z · abandoned wrun_41M0XBTHM90GKEHFEBG0P3WRYH · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QZ0GWQ1VV3J60KTB3F
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QR0GS1012PB1XDMC78
  • run-pickup-stall · nullByteWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6R30GK1D1BKMT27DP2W
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY6R00GW8D3T8SG4KMXS6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY79W0GRS0AJ6WES3QYWB
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:07:28Z · abandoned wrun_41M0XBZ7TA0GHY54WQK17ZC84M
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02800GQAP9H12M3DZ100
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02DB0GKQ2ZVJDJQWTKTR
  • run-pickup-stall · nullByteWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02D70GRMT4K0ERWK650F
  • run-pickup-stall · sleepingWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02A30GHBD095MY9ZZM3N
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:08:45Z · abandoned wrun_41M0XC083N0GV495QNVSJHN6BM
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:08:47Z · abandoned wrun_01M0XC1ABA2NZRP3XR2YHZEZKK
  • run-pickup-stall · hookWorkflow (nuxt) · at 21:09:23Z · abandoned wrun_01M0XC2T3CVPEJCQH7GTE69QJG
  • cold-start-warmup · suite warmup (python) · at 21:09:36Z · abandoned wrun_01M0XBZZGYKN4RPTB7CJ3PNA52 · (+7 more)
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNHQ4KSAYN4F95P8NG2
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNJ4G7N6H02QPT80W0E
  • run-pickup-stall · sleepingWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNRZD1GW2CBRY37HGPY
  • run-pickup-stall · nullByteWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNW7S8NT7M3WCCBA2HA
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNT1GE0NWERH03VCCKW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:10:27Z · abandoned wrun_41M0XC2T4H0GTR8DGY9NS3Z2P7
  • run-pickup-stall · sleepingWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9GZCZCD7P53QG1HPAV
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9J7VNK34BM5NCRYV2K
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9KSPCVY5W465DZ8GW8
  • run-pickup-stall · nullByteWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9M06Y972WENA3NR7M1
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9EVY18HHV93RCRC3PN
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:07Z · abandoned wrun_41M0XC30070GGPHQ1573BT7VKH
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79X66B7VSG24T5PTYN6M
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79XC7S7S5P4CF68FPRGR
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:11:51Z · abandoned wrun_01M0XC79XKXT2S6ZT3ZYDHCK93
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879903Z8AF94AM73K33B
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879BZBMDVR5B8WTZK5KK
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:12:51Z · abandoned wrun_01M0XC94GQ8DS7QK6SQG7K5G1X
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:13:33Z · abandoned wrun_01M0XCADDAH0S0PB1KYMG5FWPZ
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
ce4631e to
56c6c60CompareAugust 18, 2026 22:27
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
56c6c60 to
6f7a649CompareAugust 18, 2026 22:41
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
eb07b94 to
3c96fc3CompareAugust 21, 2026 10:51

This was referenced Aug 21, 2026
 Merged

 Merged

VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/core/src/global.ts
Comment threadpackages/core/src/runtime.ts
Comment threadpackages/core/src/runtime.ts
Outdated

Comment threadpackages/core/src/retained-vm-loop.test.ts
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 24, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Event Log Race Repro
  • vercel clean, 20 runs
  • local clean, 20 runs
  • postgres clean, 20 runs
Run History
RunLaneTotalCompleteCorruptStuckOther
08-25 00:13vercel2020000
local2020000
postgres2020000
08-25 03:04vercel2020000
local2020000
postgres2020000
08-25 19:31vercel1212000
local77000
postgres2019001
08-25 19:40vercel2020000
local2020000
postgres2020000
08-25 21:13vercel2020000
local2020000
postgres2020000
Config
vercel: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 240000ms
local, postgres: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 480000ms
VaguelySerious
VaguelySerious
approved these changes
Aug 24, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM assuming e2e passes (except python)
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/global.ts
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/runtime.ts
TooTallNate
TooTallNate
previously requested changes
Aug 24, 2026
View reviewed changes
@TooTallNateTooTallNate
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI: Request changes. The PR currently breaks the public WorkflowSuspension.steps API under a patch changeset and leaves retained-VM documentation inconsistent with the new hook behavior. Details are inline.
vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
@vercelvercelBot
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Suggestion:
Docs incorrectly claim hook-involving suspensions always force a full replay, contradicting the new retention policy that retains the VM across step boundaries containing open hooks.
Fix on Vercel
Comment threadpackages/core/src/global.ts
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.5 KiB (±0)1.76 MiB (-103 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)763.2 KiB (+351 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
80d157c · run
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 556f3f0 (AI decision).
This is a performance enhancement that broadens the retained-VM fast path to step boundaries with open hooks, plus a new items alias on WorkflowSuspension and richer retention diagnostics — capability work, not a fix for a user-visible defect. It also builds directly on main-only infrastructure: origin/stable has no retained-vm-loop.test.ts, no suspensionGeneration on the suspension path, and no docs/content/docs/v5/ pages the commit updates.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
556f3f080a33f8673cf3c08805997dba4e4d865e

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 26, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@TooTallNateTooTallNateAwaiting requested review from TooTallNate

Assignees


No one assigned


Labels

event-log-race-reproRun the event log race reproduction job

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@TooTallNate@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[core] Retain workflow VMs across hooks - #3604
Merged
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Aug 25, 2026
Merged
[core] Retain workflow VMs across hooks#3604
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • retain a workflow VM across safe step boundaries that also contain hooks
  • generation-guard step and hook suspension signals so stale callbacks cannot demote a resumed session
  • include hook metadata and abort-reason serialization in the same passivity gate as step inputs
  • replay hook-only, wait, attribute, unsafe-serialization, and WORKFLOW_RETAINED_VM=0 boundaries
Why
Spec 6 event logs are monotone prefixes. A hook_received event that lands after the VM consumed its prefix extends that prefix on the next delta without invalidating the retained session, so an open hook does not itself require replay.
Retention still requires a step boundary. Inline step bodies keep awaiting an accepted step_started ownership claim when an open hook can wake a competing invocation, and resume keeps the existing exact-prefix extension check.
Verification
  • pnpm --filter @workflow/core test — 2,171 passed, 3 expected failures
  • pnpm --filter @workflow/core typecheck
  • pnpm turbo build --filter=@workflow/core... — 7 tasks passed
  • pnpm lint — passed with baseline advisory diagnostics
  • pnpm changeset status --since=origin/codex/speed-fresh-replay
Docs Preview
PagePreview
Runtime tuningRetained VM behavior
What's new in v5Faster and cheaper runs
@vercel
vercelBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 9:07pm
@changeset-bot
changeset-botBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 80d157c
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5 | 🔍 observability
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0XC5XYM1YQE461MKEBB66M5
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • addTenWorkflow (astro)
  • cancelRun via CLI - cancelling a running workflow (express)
  • cancelRun via CLI - cancelling a running workflow (nextjs-webpack)
  • outputStreamInsideStepWorkflow - getWritable() called inside step functions (nest)
  • promiseAllWorkflow (nuxt)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:06:37Z · abandoned wrun_41M0XBTHM90GKEHFEBG0P3WRYH · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QZ0GWQ1VV3J60KTB3F
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QR0GS1012PB1XDMC78
  • run-pickup-stall · nullByteWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6R30GK1D1BKMT27DP2W
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY6R00GW8D3T8SG4KMXS6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY79W0GRS0AJ6WES3QYWB
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:07:28Z · abandoned wrun_41M0XBZ7TA0GHY54WQK17ZC84M
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02800GQAP9H12M3DZ100
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02DB0GKQ2ZVJDJQWTKTR
  • run-pickup-stall · nullByteWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02D70GRMT4K0ERWK650F
  • run-pickup-stall · sleepingWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02A30GHBD095MY9ZZM3N
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:08:45Z · abandoned wrun_41M0XC083N0GV495QNVSJHN6BM
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:08:47Z · abandoned wrun_01M0XC1ABA2NZRP3XR2YHZEZKK
  • run-pickup-stall · hookWorkflow (nuxt) · at 21:09:23Z · abandoned wrun_01M0XC2T3CVPEJCQH7GTE69QJG
  • cold-start-warmup · suite warmup (python) · at 21:09:36Z · abandoned wrun_01M0XBZZGYKN4RPTB7CJ3PNA52 · (+7 more)
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNHQ4KSAYN4F95P8NG2
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNJ4G7N6H02QPT80W0E
  • run-pickup-stall · sleepingWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNRZD1GW2CBRY37HGPY
  • run-pickup-stall · nullByteWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNW7S8NT7M3WCCBA2HA
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNT1GE0NWERH03VCCKW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:10:27Z · abandoned wrun_41M0XC2T4H0GTR8DGY9NS3Z2P7
  • run-pickup-stall · sleepingWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9GZCZCD7P53QG1HPAV
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9J7VNK34BM5NCRYV2K
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9KSPCVY5W465DZ8GW8
  • run-pickup-stall · nullByteWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9M06Y972WENA3NR7M1
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9EVY18HHV93RCRC3PN
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:07Z · abandoned wrun_41M0XC30070GGPHQ1573BT7VKH
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79X66B7VSG24T5PTYN6M
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79XC7S7S5P4CF68FPRGR
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:11:51Z · abandoned wrun_01M0XC79XKXT2S6ZT3ZYDHCK93
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879903Z8AF94AM73K33B
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879BZBMDVR5B8WTZK5KK
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:12:51Z · abandoned wrun_01M0XC94GQ8DS7QK6SQG7K5G1X
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:13:33Z · abandoned wrun_01M0XCADDAH0S0PB1KYMG5FWPZ
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
ce4631e to
56c6c60CompareAugust 18, 2026 22:27
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
56c6c60 to
6f7a649CompareAugust 18, 2026 22:41
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
eb07b94 to
3c96fc3CompareAugust 21, 2026 10:51

This was referenced Aug 21, 2026
 Merged

 Merged

VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/core/src/global.ts
Comment threadpackages/core/src/runtime.ts
Comment threadpackages/core/src/runtime.ts
Outdated

Comment threadpackages/core/src/retained-vm-loop.test.ts
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 24, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Event Log Race Repro
  • vercel clean, 20 runs
  • local clean, 20 runs
  • postgres clean, 20 runs
Run History
RunLaneTotalCompleteCorruptStuckOther
08-25 00:13vercel2020000
local2020000
postgres2020000
08-25 03:04vercel2020000
local2020000
postgres2020000
08-25 19:31vercel1212000
local77000
postgres2019001
08-25 19:40vercel2020000
local2020000
postgres2020000
08-25 21:13vercel2020000
local2020000
postgres2020000
Config
vercel: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 240000ms
local, postgres: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 480000ms
VaguelySerious
VaguelySerious
approved these changes
Aug 24, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM assuming e2e passes (except python)
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/global.ts
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/runtime.ts
TooTallNate
TooTallNate
previously requested changes
Aug 24, 2026
View reviewed changes
@TooTallNateTooTallNate
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI: Request changes. The PR currently breaks the public WorkflowSuspension.steps API under a patch changeset and leaves retained-VM documentation inconsistent with the new hook behavior. Details are inline.
vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
@vercelvercelBot
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Suggestion:
Docs incorrectly claim hook-involving suspensions always force a full replay, contradicting the new retention policy that retains the VM across step boundaries containing open hooks.
Fix on Vercel
Comment threadpackages/core/src/global.ts
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.5 KiB (±0)1.76 MiB (-103 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)763.2 KiB (+351 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
80d157c · run
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 556f3f0 (AI decision).
This is a performance enhancement that broadens the retained-VM fast path to step boundaries with open hooks, plus a new items alias on WorkflowSuspension and richer retention diagnostics — capability work, not a fix for a user-visible defect. It also builds directly on main-only infrastructure: origin/stable has no retained-vm-loop.test.ts, no suspensionGeneration on the suspension path, and no docs/content/docs/v5/ pages the commit updates.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
556f3f080a33f8673cf3c08805997dba4e4d865e

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 26, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@TooTallNateTooTallNateAwaiting requested review from TooTallNate

Assignees


No one assigned


Labels

event-log-race-reproRun the event log race reproduction job

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@TooTallNate@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[core] Retain workflow VMs across hooks - #3604
Merged
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Aug 25, 2026
Merged
[core] Retain workflow VMs across hooks#3604
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • retain a workflow VM across safe step boundaries that also contain hooks
  • generation-guard step and hook suspension signals so stale callbacks cannot demote a resumed session
  • include hook metadata and abort-reason serialization in the same passivity gate as step inputs
  • replay hook-only, wait, attribute, unsafe-serialization, and WORKFLOW_RETAINED_VM=0 boundaries
Why
Spec 6 event logs are monotone prefixes. A hook_received event that lands after the VM consumed its prefix extends that prefix on the next delta without invalidating the retained session, so an open hook does not itself require replay.
Retention still requires a step boundary. Inline step bodies keep awaiting an accepted step_started ownership claim when an open hook can wake a competing invocation, and resume keeps the existing exact-prefix extension check.
Verification
  • pnpm --filter @workflow/core test — 2,171 passed, 3 expected failures
  • pnpm --filter @workflow/core typecheck
  • pnpm turbo build --filter=@workflow/core... — 7 tasks passed
  • pnpm lint — passed with baseline advisory diagnostics
  • pnpm changeset status --since=origin/codex/speed-fresh-replay
Docs Preview
PagePreview
Runtime tuningRetained VM behavior
What's new in v5Faster and cheaper runs
@vercel
vercelBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 9:07pm
@changeset-bot
changeset-botBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 80d157c
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5 | 🔍 observability
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0XC5XYM1YQE461MKEBB66M5
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • addTenWorkflow (astro)
  • cancelRun via CLI - cancelling a running workflow (express)
  • cancelRun via CLI - cancelling a running workflow (nextjs-webpack)
  • outputStreamInsideStepWorkflow - getWritable() called inside step functions (nest)
  • promiseAllWorkflow (nuxt)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:06:37Z · abandoned wrun_41M0XBTHM90GKEHFEBG0P3WRYH · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QZ0GWQ1VV3J60KTB3F
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QR0GS1012PB1XDMC78
  • run-pickup-stall · nullByteWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6R30GK1D1BKMT27DP2W
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY6R00GW8D3T8SG4KMXS6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY79W0GRS0AJ6WES3QYWB
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:07:28Z · abandoned wrun_41M0XBZ7TA0GHY54WQK17ZC84M
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02800GQAP9H12M3DZ100
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02DB0GKQ2ZVJDJQWTKTR
  • run-pickup-stall · nullByteWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02D70GRMT4K0ERWK650F
  • run-pickup-stall · sleepingWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02A30GHBD095MY9ZZM3N
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:08:45Z · abandoned wrun_41M0XC083N0GV495QNVSJHN6BM
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:08:47Z · abandoned wrun_01M0XC1ABA2NZRP3XR2YHZEZKK
  • run-pickup-stall · hookWorkflow (nuxt) · at 21:09:23Z · abandoned wrun_01M0XC2T3CVPEJCQH7GTE69QJG
  • cold-start-warmup · suite warmup (python) · at 21:09:36Z · abandoned wrun_01M0XBZZGYKN4RPTB7CJ3PNA52 · (+7 more)
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNHQ4KSAYN4F95P8NG2
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNJ4G7N6H02QPT80W0E
  • run-pickup-stall · sleepingWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNRZD1GW2CBRY37HGPY
  • run-pickup-stall · nullByteWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNW7S8NT7M3WCCBA2HA
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNT1GE0NWERH03VCCKW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:10:27Z · abandoned wrun_41M0XC2T4H0GTR8DGY9NS3Z2P7
  • run-pickup-stall · sleepingWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9GZCZCD7P53QG1HPAV
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9J7VNK34BM5NCRYV2K
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9KSPCVY5W465DZ8GW8
  • run-pickup-stall · nullByteWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9M06Y972WENA3NR7M1
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9EVY18HHV93RCRC3PN
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:07Z · abandoned wrun_41M0XC30070GGPHQ1573BT7VKH
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79X66B7VSG24T5PTYN6M
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79XC7S7S5P4CF68FPRGR
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:11:51Z · abandoned wrun_01M0XC79XKXT2S6ZT3ZYDHCK93
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879903Z8AF94AM73K33B
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879BZBMDVR5B8WTZK5KK
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:12:51Z · abandoned wrun_01M0XC94GQ8DS7QK6SQG7K5G1X
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:13:33Z · abandoned wrun_01M0XCADDAH0S0PB1KYMG5FWPZ
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
ce4631e to
56c6c60CompareAugust 18, 2026 22:27
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
56c6c60 to
6f7a649CompareAugust 18, 2026 22:41
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
eb07b94 to
3c96fc3CompareAugust 21, 2026 10:51

This was referenced Aug 21, 2026
 Merged

 Merged

VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/core/src/global.ts
Comment threadpackages/core/src/runtime.ts
Comment threadpackages/core/src/runtime.ts
Outdated

Comment threadpackages/core/src/retained-vm-loop.test.ts
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 24, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Event Log Race Repro
  • vercel clean, 20 runs
  • local clean, 20 runs
  • postgres clean, 20 runs
Run History
RunLaneTotalCompleteCorruptStuckOther
08-25 00:13vercel2020000
local2020000
postgres2020000
08-25 03:04vercel2020000
local2020000
postgres2020000
08-25 19:31vercel1212000
local77000
postgres2019001
08-25 19:40vercel2020000
local2020000
postgres2020000
08-25 21:13vercel2020000
local2020000
postgres2020000
Config
vercel: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 240000ms
local, postgres: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 480000ms
VaguelySerious
VaguelySerious
approved these changes
Aug 24, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM assuming e2e passes (except python)
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/global.ts
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/runtime.ts
TooTallNate
TooTallNate
previously requested changes
Aug 24, 2026
View reviewed changes
@TooTallNateTooTallNate
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI: Request changes. The PR currently breaks the public WorkflowSuspension.steps API under a patch changeset and leaves retained-VM documentation inconsistent with the new hook behavior. Details are inline.
vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
@vercelvercelBot
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Suggestion:
Docs incorrectly claim hook-involving suspensions always force a full replay, contradicting the new retention policy that retains the VM across step boundaries containing open hooks.
Fix on Vercel
Comment threadpackages/core/src/global.ts
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.5 KiB (±0)1.76 MiB (-103 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)763.2 KiB (+351 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
80d157c · run
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 556f3f0 (AI decision).
This is a performance enhancement that broadens the retained-VM fast path to step boundaries with open hooks, plus a new items alias on WorkflowSuspension and richer retention diagnostics — capability work, not a fix for a user-visible defect. It also builds directly on main-only infrastructure: origin/stable has no retained-vm-loop.test.ts, no suspensionGeneration on the suspension path, and no docs/content/docs/v5/ pages the commit updates.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
556f3f080a33f8673cf3c08805997dba4e4d865e

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 26, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@TooTallNateTooTallNateAwaiting requested review from TooTallNate

Assignees


No one assigned


Labels

event-log-race-reproRun the event log race reproduction job

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@TooTallNate@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
[core] Retain workflow VMs across hooks - #3604
Merged
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Aug 25, 2026
Merged
[core] Retain workflow VMs across hooks#3604
NathanColosimo merged 8 commits into
mainfrom 
codex/retain-vm-with-hooks
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • retain a workflow VM across safe step boundaries that also contain hooks
  • generation-guard step and hook suspension signals so stale callbacks cannot demote a resumed session
  • include hook metadata and abort-reason serialization in the same passivity gate as step inputs
  • replay hook-only, wait, attribute, unsafe-serialization, and WORKFLOW_RETAINED_VM=0 boundaries
Why
Spec 6 event logs are monotone prefixes. A hook_received event that lands after the VM consumed its prefix extends that prefix on the next delta without invalidating the retained session, so an open hook does not itself require replay.
Retention still requires a step boundary. Inline step bodies keep awaiting an accepted step_started ownership claim when an open hook can wake a competing invocation, and resume keeps the existing exact-prefix extension check.
Verification
  • pnpm --filter @workflow/core test — 2,171 passed, 3 expected failures
  • pnpm --filter @workflow/core typecheck
  • pnpm turbo build --filter=@workflow/core... — 7 tasks passed
  • pnpm lint — passed with baseline advisory diagnostics
  • pnpm changeset status --since=origin/codex/speed-fresh-replay
Docs Preview
PagePreview
Runtime tuningRetained VM behavior
What's new in v5Faster and cheaper runs
@vercel
vercelBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 9:07pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 9:07pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 9:07pm
@changeset-bot
changeset-botBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 80d157c
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (8 failed)
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5 | 🔍 observability
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R | 🔍 observability
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0XC5XYM1YQE461MKEBB66M5
  • promiseAllWorkflow | wrun_41M0XBY3ZY0GWW3GKQANKWMTV3
  • sleepingWorkflow | wrun_41M0XBYSHY0GZSK5F9PEFNRMNS
  • parallelSleepWorkflow | wrun_41M0XBYSQC0GPEP2TTK573QQV5
  • nullByteWorkflow | wrun_41M0XBZ0X40GVNS4Q15T0N090C
  • cancelRun - cancelling a running workflow | wrun_41M0XC3RAN0GZV6KRDB67KFV89
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0XC3ZVB0GV12ECY3ZJPPB2F
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0XC43860GHM3P79AE87VE6N
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0XC4NG10GSH7BQAKT4TVV9R
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • addTenWorkflow (astro)
  • cancelRun via CLI - cancelling a running workflow (express)
  • cancelRun via CLI - cancelling a running workflow (nextjs-webpack)
  • outputStreamInsideStepWorkflow - getWritable() called inside step functions (nest)
  • promiseAllWorkflow (nuxt)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
34 infra events
  • cold-start-warmup · suite warmup (python) · at 21:06:37Z · abandoned wrun_41M0XBTHM90GKEHFEBG0P3WRYH · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QZ0GWQ1VV3J60KTB3F
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6QR0GS1012PB1XDMC78
  • run-pickup-stall · nullByteWorkflow (python) · at 21:06:54Z · abandoned wrun_41M0XBY6R30GK1D1BKMT27DP2W
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY6R00GW8D3T8SG4KMXS6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:06:55Z · abandoned wrun_41M0XBY79W0GRS0AJ6WES3QYWB
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:07:28Z · abandoned wrun_41M0XBZ7TA0GHY54WQK17ZC84M
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02800GQAP9H12M3DZ100
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02DB0GKQ2ZVJDJQWTKTR
  • run-pickup-stall · nullByteWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02D70GRMT4K0ERWK650F
  • run-pickup-stall · sleepingWorkflow (python) · at 21:07:55Z · abandoned wrun_41M0XC02A30GHBD095MY9ZZM3N
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:08:45Z · abandoned wrun_41M0XC083N0GV495QNVSJHN6BM
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:08:47Z · abandoned wrun_01M0XC1ABA2NZRP3XR2YHZEZKK
  • run-pickup-stall · hookWorkflow (nuxt) · at 21:09:23Z · abandoned wrun_01M0XC2T3CVPEJCQH7GTE69QJG
  • cold-start-warmup · suite warmup (python) · at 21:09:36Z · abandoned wrun_01M0XBZZGYKN4RPTB7CJ3PNA52 · (+7 more)
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNHQ4KSAYN4F95P8NG2
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNJ4G7N6H02QPT80W0E
  • run-pickup-stall · sleepingWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNRZD1GW2CBRY37HGPY
  • run-pickup-stall · nullByteWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNW7S8NT7M3WCCBA2HA
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:09:51Z · abandoned wrun_01M0XC3MNT1GE0NWERH03VCCKW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:10:27Z · abandoned wrun_41M0XC2T4H0GTR8DGY9NS3Z2P7
  • run-pickup-stall · sleepingWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9GZCZCD7P53QG1HPAV
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9J7VNK34BM5NCRYV2K
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9KSPCVY5W465DZ8GW8
  • run-pickup-stall · nullByteWorkflow (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9M06Y972WENA3NR7M1
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:10:51Z · abandoned wrun_01M0XC5F9EVY18HHV93RCRC3PN
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:07Z · abandoned wrun_41M0XC30070GGPHQ1573BT7VKH
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79X66B7VSG24T5PTYN6M
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:11:51Z · abandoned wrun_01M0XC79XC7S7S5P4CF68FPRGR
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:11:51Z · abandoned wrun_01M0XC79XKXT2S6ZT3ZYDHCK93
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879903Z8AF94AM73K33B
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:12:21Z · abandoned wrun_01M0XC879BZBMDVR5B8WTZK5KK
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:12:51Z · abandoned wrun_01M0XC94GQ8DS7QK6SQG7K5G1X
  • run-pickup-stall · hookCleanupTestWorkflow - hook token reuse after workflow completion (nextjs-webpack) · at 21:13:33Z · abandoned wrun_01M0XCADDAH0S0PB1KYMG5FWPZ
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production357087424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-http-transport8170143960
✅ vercel-multi-region270027
✅ vercel-ws-transport553087640
Total1705317277819848
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node132028
✅ astro-quickjs132028
✅ example-node132028
✅ example-quickjs132028
✅ express-node132028
✅ express-quickjs132028
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
✅ hono-quickjs132028
✅ nest-node132028
✅ nest-quickjs132028
✅ nextjs-turbopack-node15703
✅ nextjs-turbopack-quickjs15703
✅ nextjs-webpack-node15703
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
✅ nitro-quickjs132028
✅ nuxt-node132028
✅ nuxt-quickjs132028
❌ python-node08152
✅ sveltekit-node15109
✅ sveltekit-quickjs15109
✅ tanstack-start-node132028
✅ tanstack-start-quickjs132028
✅ vite-node132028
✅ vite-quickjs132028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-http-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ hono132028
✅ nextjs-turbopack15703
✅ nitro132028
✅ vite132028
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
✅ vercel-ws-transport
AppPassedFailedSkipped
✅ example132028
✅ express132028
✅ nextjs-turbopack15703
✅ vite132028
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 17, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
ce4631e to
56c6c60CompareAugust 18, 2026 22:27
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
56c6c60 to
6f7a649CompareAugust 18, 2026 22:41
@NathanColosimo
NathanColosimoforce-pushed
the
codex/retain-vm-with-hooks
branch
from
eb07b94 to
3c96fc3CompareAugust 21, 2026 10:51

This was referenced Aug 21, 2026
 Merged

 Merged

VaguelySerious
VaguelySerious
reviewed
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found
Comment threadpackages/core/src/global.ts
Comment threadpackages/core/src/runtime.ts
Comment threadpackages/core/src/runtime.ts
Outdated

Comment threadpackages/core/src/retained-vm-loop.test.ts
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

@github-actions
github-actionsBot
commented
Aug 24, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Event Log Race Repro
  • vercel clean, 20 runs
  • local clean, 20 runs
  • postgres clean, 20 runs
Run History
RunLaneTotalCompleteCorruptStuckOther
08-25 00:13vercel2020000
local2020000
postgres2020000
08-25 03:04vercel2020000
local2020000
postgres2020000
08-25 19:31vercel1212000
local77000
postgres2019001
08-25 19:40vercel2020000
local2020000
postgres2020000
08-25 21:13vercel2020000
local2020000
postgres2020000
Config
vercel: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 240000ms
local, postgres: 20 runs / step-storm 6, hook-storm 6, blocked-branch 6, hook-sleep 2 / c8 / 6x8 / watchdog 2500ms / step 2200±250ms / stagger 400ms / burst 4000+1200ms / poke 750ms / poke budget 64 then /8 / timeout 480000ms
VaguelySerious
VaguelySerious
approved these changes
Aug 24, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM assuming e2e passes (except python)
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/global.ts
TooTallNate
TooTallNate
reviewed
Aug 24, 2026
View reviewed changes
Comment threadpackages/core/src/runtime.ts
TooTallNate
TooTallNate
previously requested changes
Aug 24, 2026
View reviewed changes
@TooTallNateTooTallNate
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI: Request changes. The PR currently breaks the public WorkflowSuspension.steps API under a patch changeset and leaves retained-VM documentation inconsistent with the new hook behavior. Details are inline.
vercel[bot]
vercelBot
reviewed
Aug 25, 2026
View reviewed changes
@vercelvercelBot
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Suggestion:
Docs incorrectly claim hook-involving suspensions always force a full replay, contradicting the new retention policy that retains the VM across step boundaries containing open hooks.
Fix on Vercel
Comment threadpackages/core/src/global.ts
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
FrameworkFlow routeStep reg.Framework output
hono200.9 KiB (±0)40.5 KiB (±0)1.76 MiB (-103 B)
nextjs-turbopack206.3 KiB (±0)439 B (±0)763.2 KiB (+351 B)
About these numbers
Sizes are gzip; parentheses show the change against main.

Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.
80d157c · run
@github-actions
github-actionsBot
commented
Aug 25, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 556f3f0 (AI decision).
This is a performance enhancement that broadens the retained-VM fast path to step boundaries with open hooks, plus a new items alias on WorkflowSuspension and richer retention diagnostics — capability work, not a fix for a user-visible defect. It also builds directly on main-only infrastructure: origin/stable has no retained-vm-loop.test.ts, no suspensionGeneration on the suspension path, and no docs/content/docs/v5/ pages the commit updates.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
556f3f080a33f8673cf3c08805997dba4e4d865e

@github-actionsgithub-actionsBot
mentioned this pull request
Aug 26, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix
@TooTallNateTooTallNateAwaiting requested review from TooTallNate

Assignees


No one assigned


Labels

event-log-race-reproRun the event log race reproduction job

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo@TooTallNate@VaguelySerious