Run the test suites CI was silently skipping - #3733

Merged
pranaygp merged 2 commits into
mainfrom
pgp/workflow-world-tests
Aug 21, 2026
Merged

Run the test suites CI was silently skipping#3733
pranaygp merged 2 commits into
mainfrom
pgp/workflow-world-tests

Conversation

@pranaygp

@pranaygppranaygp commented Aug 21, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Fixes#3731.

turbo test runs a package's tests only if that package declares a test script. A suite without one is skipped in silence — the files look like coverage, and the assertions inside them rot with nobody watching. #3731 found @workflow/world in that state; checking the rest of the workspace found three more.

What was unrun

SuiteFilesTests
@workflow/world13160
@workflow/cli551
@workflow/nitro130
packages/core/e2e/utils.test.ts115
packages/core/e2e/manifest.test.ts136

All 292 pass as-is, so this adds no red. @workflow/world, @workflow/cli and @workflow/nitro get a test script (vitest run src) and a vitest devDependency, matching every other package.

The two packages/core/e2e files are a different shape of the same bug: that directory is run by dedicated workflow steps naming each file, and these two were named by none.

  • e2e/utils.test.ts tests the e2e helpers themselves and needs no deployment, so it joins the unit lane through @workflow/core's test script.
  • e2e/manifest.test.ts asserts the shape of a built workbench manifest, so it gets a step in E2E Local Prod immediately after local-build.test.ts — which is what runs pnpm build in the workbench app. It needs its own invocation rather than another file in that step, since vitest would otherwise run both concurrently, against a manifest that does not exist yet.

The spec-version.test.ts failure in the issue

Already gone. #3634 rewrote that assertion when it moved the current version to 7 (sealed log), so the file now pins SPEC_VERSION_CURRENT === SPEC_VERSION_SUPPORTS_SEALED_LOG rather than the stale compression version. Nothing to update here — but note that the fix landed blind, because the suite proving it still never ran.

Keeping it wired

scripts/check-no-unrun-tests.mjs walks the workspace and requires every test file to be reachable one of two ways:

  1. its package has a test script whose positional filters cover the file, so turbo test runs it — vitest treats positionals as filters, so vitest run src beside a test in e2e/ drops that file quietly; or
  2. a CI workflow, or a root package.json script, names the file (or a directory holding it) outright.

Anything else fails. It runs as a new No Unrun Tests job in lint.yml (node only, no install, alongside the other hygiene jobs). One waiver: @workflow/docs-typecheck, whose suite is far too slow for the unit lane and runs as pnpm test:docs from Docs Checks.

Path matching is boundary-aware on purpose. A plain substring search reads packages/world as present in every workflow mentioning packages/world-vercel, and a trailing / counts as continuation so that a lane naming packages/core/e2e/local-build.test.ts says nothing about the other files in that directory. Without both rules the check passes on the very state it exists to catch.

Verification

Each regression was re-introduced one at a time and the guard flagged it:

✗ @workflow/world (packages/world) has 13 test file(s) but no "test" script,
so `turbo test` skips the package entirely
✗ @workflow/core (packages/core) has test file(s) that neither its "test"
script nor any CI workflow runs: e2e/manifest.test.ts

turbo test after the fix:

@workflow/world:test: Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test: Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test: Test Files 1 passed (1) Tests 30 passed
@workflow/core:test: Test Files 104 passed Tests 2259 passed

manifest.test.ts was checked against all eight workbench apps with manifests, built locally: 36 passed.

CI confirms the suites actually execute rather than the job merely going green. Both unit lanes pass, and their logs show:

@workflow/world:test Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test Test Files 1 passed (1) Tests 30 passed
@workflow/core:test ✓ e2e/utils.test.ts (15 tests)

Windows matters here for two reasons, both now checked rather than assumed: nitro's assertions hard-code POSIX paths and hold because the source builds those specifiers with pathe, and the e2e/utils.test.ts positional filter does match there.

@workflow/world-postgres fails locally without Docker for testcontainers; it passes in CI (6 files / 193 tests).

🤖 Generated with Claude Code

@pranaygp@claude
Run the test suites CI was silently skipping
94c9088
`turbo test` runs a package's tests only if that package declares a `test`
script, so a suite can sit in the repo for months without ever running. Four
were in that state: @workflow/world (13 files, 160 tests), @workflow/cli (5 /
51), @workflow/nitro (1 / 30), and two files under packages/core/e2e that no
workflow named.
Wire each one up, and add scripts/check-test-suites-wired.mjs plus a lint job
so the next unwired suite fails CI instead of going unnoticed.
Fixes#3731
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pranaygp
pranaygp requested a review from a team as a code ownerAugust 21, 2026 22:45
CopilotAI lite review requested due to automatic review settings August 21, 2026 22:45
@vercel

vercelBot commented Aug 21, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-astro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-express-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-hono-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-python-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-vite-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-docsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-tarballsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-webReadyReadyPreview, v0Aug 21, 2026 10:52pm

CopilotAI reviewed Aug 21, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@changeset-bot

changeset-botBot commented Aug 21, 2026•
edited
Loading

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 023e049

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions

github-actionsBot commented Aug 21, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (32 failed)

astro-node (2 failed):

  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability

astro-quickjs (1 failed):

  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step

example-node (1 failed):

  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability

example-quickjs (1 failed):

  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step

express-node (1 failed):

  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability

express-quickjs (2 failed):

  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook | wrun_41M0K8E2780GKJZTMNTCF7TXRS | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step

hono-quickjs (1 failed):

  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals

nest-node (1 failed):

  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability

nest-quickjs (1 failed):

  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step

nextjs-turbopack-quickjs (2 failed):

  • webhookWorkflow | wrun_41M0K8ASG90GPZGENG5QRFKYZN | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step

nextjs-webpack-node (1 failed):

  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability

nitro-quickjs (2 failed):

  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step

nuxt-quickjs (2 failed):

  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals

python-node (8 failed):

  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC | 🔍 observability
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ | 🔍 observability
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X | 🔍 observability

sveltekit-node (1 failed):

  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability

sveltekit-quickjs (1 failed):

  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step

tanstack-start-node (2 failed):

  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability

vite-node (1 failed):

  • distributedAbortController - reconnect to existing controller | wrun_41M0K8HAHE0GVTB2BAFTQ84TC9 | 🔍 observability

vite-quickjs (1 failed):

  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals

🌐 Cross-language Conformance (9 failed)

python (9 failed):

  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K8H69RZBXWE62FS3BH7CAC
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

33 flaky tests
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (astro)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (nextjs-webpack)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (astro)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nuxt)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (vite)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (fastify)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-turbopack)
  • abortHookOrderingWorkflow [hook-first-abort-first]: hook.then → addEventListener → abort() → resumeHook (astro)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (fastify)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (hono)
  • abortReasonWorkflow: abort reason preserved across boundaries (astro)
  • abortReasonWorkflow: abort reason preserved across boundaries (fastify)
  • abortSurvivesReplayWorkflow: controller state consistent across replay (nuxt)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (astro)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (hono)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (nextjs-webpack)
  • addTenWorkflow (tanstack-start)
  • Calculator.calculate - static workflow method using static step methods from another class (nextjs-webpack)
  • distributedAbortController - manual abort triggers signal (nest)
  • distributedAbortController - manual abort triggers signal (sveltekit)
  • distributedAbortController - reconnect to existing controller (nitro)
  • distributedAbortController - reconnect to existing controller (sveltekit)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (express)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (hono)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nest)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (astro)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (hono)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (hono)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nitro)
  • webhookWorkflow (fastify)
  • workflowAndStepMetadataWorkflow (nextjs-webpack)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

33 infra events
  • cold-start-warmup · suite warmup (tanstack-start) · at 22:52:19Z · abandoned wrun_01M0K8C0WW5RQWS78EE4YVMKCW
  • cold-start-warmup · suite warmup (python) · at 22:52:39Z · abandoned wrun_41M0K89TTX0GY1APHXTS6PK89D · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H1
  • run-pickup-stall · nullByteWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFH10GX9K9PD2KZR3385
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGQ0GSPXK5H1GS7FCZ4
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFTX0GGSWRS14CZT9CRR
  • cold-start-warmup · suite warmup (python) · at 22:53:26Z · abandoned wrun_01M0K8B7VY7211258MVNH21EQQ · (+7 more)
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:53:28Z · abandoned wrun_41M0K8EF470GYTVH8A32M271NQ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0XWVW7FD6CWJSYS6GS
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX129NGA4EPDJ1E1XGGJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX13T6P1P1B6GBG27APD
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX15QXRX5S8P4TKAWE5R
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0Y1CK71YTJZ4PK2R91
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FASZ0GY8S51B3QA9804D
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FARV0GWX9AS5V5VKQ1N5
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FB1P0GHQ6SBP1AGW8S06
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FAW90GR7NW0F4HEBR5R9
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMHB4E9B13993WC73X5
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMFPFC849RM811TNR0E
  • run-pickup-stall · sleepingWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMM0TTN8V5VS10CRPEE
  • run-pickup-stall · nullByteWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMXA75AGR0ZG2ZQ5HST
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMNPJ85RDG23KCH8HJH
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:54:47Z · abandoned wrun_41M0K8FE9Q0GYH1F346WCF17T2
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ7XZQ9GXF1KSYT77GZ0
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ86FE14B8R6MNJRA9S7
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ89NC11EJ0YS9K4J66H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFJWRGDWW15C8M6ZXECB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFK7V61YSE03MDGAEDXM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:28Z · abandoned wrun_41M0K8J6690GQZ3TBQCZ186JCW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:28Z · abandoned wrun_41M0K8J2NE0GJF7X6RMVFPP1C0
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:41Z · abandoned wrun_01M0K8MCVBJKXXM7HC8W3AXS0V
  • run-pickup-stall · pathsAliasWorkflow - TypeScript path aliases resolve correctly (nextjs-webpack) · at 22:58:54Z · abandoned wrun_01M0K8REC1AXDVSBRCCMBQJJNM

E2E Test Summary

Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production3546327424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total1565941254818248
Details by Category

❌ ▲ Vercel Production

AppPassedFailedSkipped
❌ astro-node130228
❌ astro-quickjs131128
❌ example-node131128
❌ example-quickjs131128
❌ express-node131128
❌ express-quickjs130228
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
❌ hono-quickjs131128
❌ nest-node131128
❌ nest-quickjs131128
✅ nextjs-turbopack-node15703
❌ nextjs-turbopack-quickjs15523
❌ nextjs-webpack-node15613
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
❌ nitro-quickjs130228
✅ nuxt-node132028
❌ nuxt-quickjs130228
❌ python-node08152
❌ sveltekit-node15019
❌ sveltekit-quickjs15019
❌ tanstack-start-node130228
✅ tanstack-start-quickjs132028
❌ vite-node131128
❌ vite-quickjs131128

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000

❌ 🌐 Cross-language Conformance

AppPassedFailedSkipped
❌ python09132

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@github-actions

github-actionsBot commented Aug 21, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 023e049 · Fri, 21 Aug 2026 23:05:25 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1143 (+6.5%)1414 🔴 (+22%) 🔻1459 🔴 (+17%) 🔻1672 🔴 (+17%) 🔻30
TTFSstream1310 (+24%) 🔻1403 🔴 (+16%) 🔻1426 🔴 (+11%)1488 🔴 (+3.0%)30
TTFShook + stream1600 (+51%) 🔻1701 🔴 (+17%) 🔻1733 🔴 (+15%) 🔻1791 🔴 (+16%) 🔻30
Fan-out TTFSPromise.all(100 steps)540 (-8.8%)734 (-54%) 💚920 (-44%) 💚1060 (-36%) 💚10
Fan-out TTLSPromise.all(100 steps)1598 (-1.2%)2888 (-3.9%)7336 (+104%) 🔻7478 (+90%) 🔻10
STSO1020 steps (inline)129 (-14%)167 (-24%) 💚181 (-30%) 💚257 (-52%) 💚1019
WO1020 steps164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚1
CRTTfirst chunk (pooled)81 (-15%)163 (+10%)241 (+39%) 🔻466 (+101%) 🔻28

Streams

ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)114 (-12%)127 (-31%)188 (-26%)435 (+14%)127 (-25%)10
size sweep (100/s, 160B-12KB)109 (-12%)116 (-37%)172 (-49%)343 (-34%)107 (-19%)10
replay gateway-gpt-5.4-nano-2000t (1x)103 (-15%)107 (-85%)174 (-95%)509 (-89%)239 (-93%)3
replay eve-gpt-5.6-sol-2000t (1x)108 (-11%)108 (-33%)138 (-33%)315 (-12%)226 (-25%)2
replay eve-gpt-5.6-sol-2000t (2x)162 (+5%)212 (-35%)463 (-53%)3872 (+71%)723 (+83%)3
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 217526ms → this run 164269ms (Δ -53257ms, -24%)

 100-150 ms ░░░░░░░░┃ main 0 this 268 +268
150-200 ms ██████████████████░░░░░┃ main 523 this 717 +194
200-250 ms ┃████████████ main 379 this 23 -356
250-300 ms ┃█ main 65 this 6 -59
300-350 ms ┃ main 23 this 2 -21
350-400 ms ┃ main 10 this 2 -8
400-450 ms ┃ main 4 this 0 -4
450-500 ms ┃ main 1 this 0 -1
500-550 ms ┃ main 5 this 1 -4
550-600 ms ┃ main 4 this 0 -4
600-650 ms ┃ main 1 this 0 -1
650-700 ms ┃ main 1 this 0 -1
750-800 ms ┃ main 1 this 0 -1
900-950 ms ┃ main 1 this 0 -1
1050-1100 ms ┃ main 1 this 0 -1
📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······█▆▁···· 104.6 (-29%) 95 (-32%) 188 (-26%) 435 (+14%) 3000
sweep ·····▁█▅▁···· 99.5 (-33%) 92 (-29%) 172 (-49%) 343 (-34%) 3000
gw 1x ·····▁█▃▁▁··· 97.3 (-81%) 85 (-51%) 174 (-95%) 509 (-89%) 5295
eve 1x ·····▁█▄▁···· 93.4 (-32%) 86 (-31%) 138 (-33%) 315 (-12%) 5186
eve 2x ·····▁▆█▃▁·▁· 246.2 (-4%) 125 (-27%) 463 (-53%) 3872 (+71%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control ▅▂▁▁▂▆▅█▃▄ 95–120ms
sweep ▇█▆▂▂▁▂▂▁▁ 91–119ms
gw 1x █▂▁▂▁▂▁▂▁▃ 83–154ms
eve 1x ▂▂▃▃▁▂▆█▄▁ 82–117ms
eve 2x ▁▁▁▁▁▁▂▂▁█ 105–1109ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep ██▆▅▃▁▂ 96–103ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▄▁█▅▄▆▅▅▄▃ 25–34ms
sweep ▃█▅▆▆▆▆▄▁▂ 32–43ms
gw 1x █▃▂▁▂▂▂▂▂▃ 22–37ms
eve 1x ▁▁▁▄▂▃▆▇█▄ 18–25ms
eve 2x ▂▃▁▁▁▃▂▁▁█ 20–45ms
ℹ️ Metric definitions & methodology

Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.

The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

VaguelySerious
VaguelySerious approved these changes Aug 21, 2026
View reviewed changes
Comment thread.github/workflows/lint.yml Outdated
exit 1
fi

wired-test-suites:

@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get a better name?

@pranaygppranaygpAug 21, 2026

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to no-unrun-tests / No Unrun Tests, matching the no-test-overrides / No Test Overrides sibling right above it — both are short negative invariants. The script moved to scripts/check-no-unrun-tests.mjs to match, and the step now reads "Check every test file is run by turbo test or a CI lane".

Comment thread.changeset/run-e2e-utils-in-unit-lane.md Outdated
'@workflow/core': patch
---

Run `e2e/utils.test.ts` from the package's `test` script; it covers the e2e helpers themselves and needs no deployment, so nothing had been running it.

@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe don't need two changeset entries just for internal test changes

@pranaygppranaygpAug 21, 2026

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped both. Nothing here changes shipped behaviour — the only edits to published packages are a test script and a vitest devDependency — so there is nothing to release. (No status check gates on the changeset, so this stays green.)

@pranaygp@claude
Drop the changesets and rename the guard job
023e049
The PR only wires up existing suites and adds a CI check, so there is nothing
to release. Rename the job to match its `no-test-overrides` sibling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actionsBot commented Aug 21, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0

Full trace: world-sim.txt

@pranaygp
pranaygp enabled auto-merge (squash) August 21, 2026 22:49
@pranaygp
pranaygp disabled auto-merge August 21, 2026 22:59
@pranaygp
pranaygp merged commit 447013b into mainAug 21, 2026
147 of 167 checks passed
@pranaygp
pranaygp deleted the pgp/workflow-world-tests branch August 21, 2026 22:59
@github-actions

github-actionsBot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Backport to stable failed for 447013b due to a workflow error (backport job run).

This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.

Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

447013b73a5c240b4d17011faf45f5cfd7b4d956

pranaygp added a commit that referenced this pull request Aug 21, 2026
@pranaygp
Merge remote-tracking branch 'origin/main' into pgp/bundling-regressions
9979bb5
* origin/main:
Run the test suites CI was silently skipping (#3733)
[e2e] Change race repro hook poke to soft-degrade instead of hard-stop at budget (#3561)
@VaguelySeriousVaguelySerious mentioned this pull request Aug 21, 2026
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

@workflow/world has 13 test files and no test script, so none of them run in CI

3 participants

@pranaygp@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Run the test suites CI was silently skipping - #3733
Merged
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Aug 21, 2026
Merged
Run the test suites CI was silently skipping#3733
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Conversation
@pranaygp
@pranaygppranaygp
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#3731.
turbo test runs a package's tests only if that package declares a test script. A suite without one is skipped in silence — the files look like coverage, and the assertions inside them rot with nobody watching. #3731 found @workflow/world in that state; checking the rest of the workspace found three more.
What was unrun
SuiteFilesTests
@workflow/world13160
@workflow/cli551
@workflow/nitro130
packages/core/e2e/utils.test.ts115
packages/core/e2e/manifest.test.ts136
All 292 pass as-is, so this adds no red. @workflow/world, @workflow/cli and @workflow/nitro get a test script (vitest run src) and a vitest devDependency, matching every other package.
The two packages/core/e2e files are a different shape of the same bug: that directory is run by dedicated workflow steps naming each file, and these two were named by none.
  • e2e/utils.test.ts tests the e2e helpers themselves and needs no deployment, so it joins the unit lane through @workflow/core's test script.
  • e2e/manifest.test.ts asserts the shape of a built workbench manifest, so it gets a step in E2E Local Prod immediately after local-build.test.ts — which is what runs pnpm build in the workbench app. It needs its own invocation rather than another file in that step, since vitest would otherwise run both concurrently, against a manifest that does not exist yet.
The spec-version.test.ts failure in the issue
Already gone. #3634 rewrote that assertion when it moved the current version to 7 (sealed log), so the file now pins SPEC_VERSION_CURRENT === SPEC_VERSION_SUPPORTS_SEALED_LOG rather than the stale compression version. Nothing to update here — but note that the fix landed blind, because the suite proving it still never ran.
Keeping it wired
scripts/check-no-unrun-tests.mjs walks the workspace and requires every test file to be reachable one of two ways:
  1. its package has a test script whose positional filters cover the file, so turbo test runs it — vitest treats positionals as filters, so vitest run src beside a test in e2e/ drops that file quietly; or
  2. a CI workflow, or a root package.json script, names the file (or a directory holding it) outright.
Anything else fails. It runs as a new No Unrun Tests job in lint.yml (node only, no install, alongside the other hygiene jobs). One waiver: @workflow/docs-typecheck, whose suite is far too slow for the unit lane and runs as pnpm test:docs from Docs Checks.
Path matching is boundary-aware on purpose. A plain substring search reads packages/world as present in every workflow mentioning packages/world-vercel, and a trailing / counts as continuation so that a lane naming packages/core/e2e/local-build.test.ts says nothing about the other files in that directory. Without both rules the check passes on the very state it exists to catch.
Verification
Each regression was re-introduced one at a time and the guard flagged it:
✗ @workflow/world (packages/world) has 13 test file(s) but no "test" script,
so `turbo test` skips the package entirely
✗ @workflow/core (packages/core) has test file(s) that neither its "test"
script nor any CI workflow runs: e2e/manifest.test.ts

turbo test after the fix:
@workflow/world:test: Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test: Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test: Test Files 1 passed (1) Tests 30 passed
@workflow/core:test: Test Files 104 passed Tests 2259 passed

manifest.test.ts was checked against all eight workbench apps with manifests, built locally: 36 passed.
CI confirms the suites actually execute rather than the job merely going green. Both unit lanes pass, and their logs show:
@workflow/world:test Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test Test Files 1 passed (1) Tests 30 passed
@workflow/core:test ✓ e2e/utils.test.ts (15 tests)

Windows matters here for two reasons, both now checked rather than assumed: nitro's assertions hard-code POSIX paths and hold because the source builds those specifiers with pathe, and the e2e/utils.test.ts positional filter does match there.
@workflow/world-postgres fails locally without Docker for testcontainers; it passes in CI (6 files / 193 tests).
🤖 Generated with Claude Code
@pranaygp@claude
Run the test suites CI was silently skipping
94c9088
`turbo test` runs a package's tests only if that package declares a `test`
script, so a suite can sit in the repo for months without ever running. Four
were in that state: @workflow/world (13 files, 160 tests), @workflow/cli (5 /
51), @workflow/nitro (1 / 30), and two files under packages/core/e2e that no
workflow named.
Wire each one up, and add scripts/check-test-suites-wired.mjs plus a lint job
so the next unwired suite fails CI instead of going unnoticed.
Fixes#3731
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pranaygp
pranaygp
requested a review
from a team
as a code ownerAugust 21, 2026 22:45
CopilotAI
lite review requested due to automatic review settings
August 21, 2026 22:45
@vercel
vercelBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-astro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-express-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-hono-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-python-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-vite-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-docsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-tarballsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-webReadyReadyPreview, v0Aug 21, 2026 10:52pm
CopilotAI
reviewed
Aug 21, 2026
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
@changeset-bot
changeset-botBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: 023e049
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (32 failed)
astro-node (2 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
astro-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
example-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
example-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
express-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
express-quickjs (2 failed):
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook | wrun_41M0K8E2780GKJZTMNTCF7TXRS | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
hono-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
nest-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
nest-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
nextjs-turbopack-quickjs (2 failed):
  • webhookWorkflow | wrun_41M0K8ASG90GPZGENG5QRFKYZN | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nextjs-webpack-node (1 failed):
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
nitro-quickjs (2 failed):
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nuxt-quickjs (2 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC | 🔍 observability
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ | 🔍 observability
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X | 🔍 observability
sveltekit-node (1 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
sveltekit-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
tanstack-start-node (2 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
vite-node (1 failed):
  • distributedAbortController - reconnect to existing controller | wrun_41M0K8HAHE0GVTB2BAFTQ84TC9 | 🔍 observability
vite-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K8H69RZBXWE62FS3BH7CAC
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
33 flaky tests
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (astro)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (nextjs-webpack)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (astro)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nuxt)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (vite)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (fastify)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-turbopack)
  • abortHookOrderingWorkflow [hook-first-abort-first]: hook.then → addEventListener → abort() → resumeHook (astro)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (fastify)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (hono)
  • abortReasonWorkflow: abort reason preserved across boundaries (astro)
  • abortReasonWorkflow: abort reason preserved across boundaries (fastify)
  • abortSurvivesReplayWorkflow: controller state consistent across replay (nuxt)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (astro)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (hono)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (nextjs-webpack)
  • addTenWorkflow (tanstack-start)
  • Calculator.calculate - static workflow method using static step methods from another class (nextjs-webpack)
  • distributedAbortController - manual abort triggers signal (nest)
  • distributedAbortController - manual abort triggers signal (sveltekit)
  • distributedAbortController - reconnect to existing controller (nitro)
  • distributedAbortController - reconnect to existing controller (sveltekit)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (express)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (hono)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nest)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (astro)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (hono)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (hono)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nitro)
  • webhookWorkflow (fastify)
  • workflowAndStepMetadataWorkflow (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (tanstack-start) · at 22:52:19Z · abandoned wrun_01M0K8C0WW5RQWS78EE4YVMKCW
  • cold-start-warmup · suite warmup (python) · at 22:52:39Z · abandoned wrun_41M0K89TTX0GY1APHXTS6PK89D · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H1
  • run-pickup-stall · nullByteWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFH10GX9K9PD2KZR3385
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGQ0GSPXK5H1GS7FCZ4
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFTX0GGSWRS14CZT9CRR
  • cold-start-warmup · suite warmup (python) · at 22:53:26Z · abandoned wrun_01M0K8B7VY7211258MVNH21EQQ · (+7 more)
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:53:28Z · abandoned wrun_41M0K8EF470GYTVH8A32M271NQ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0XWVW7FD6CWJSYS6GS
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX129NGA4EPDJ1E1XGGJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX13T6P1P1B6GBG27APD
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX15QXRX5S8P4TKAWE5R
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0Y1CK71YTJZ4PK2R91
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FASZ0GY8S51B3QA9804D
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FARV0GWX9AS5V5VKQ1N5
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FB1P0GHQ6SBP1AGW8S06
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FAW90GR7NW0F4HEBR5R9
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMHB4E9B13993WC73X5
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMFPFC849RM811TNR0E
  • run-pickup-stall · sleepingWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMM0TTN8V5VS10CRPEE
  • run-pickup-stall · nullByteWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMXA75AGR0ZG2ZQ5HST
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMNPJ85RDG23KCH8HJH
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:54:47Z · abandoned wrun_41M0K8FE9Q0GYH1F346WCF17T2
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ7XZQ9GXF1KSYT77GZ0
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ86FE14B8R6MNJRA9S7
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ89NC11EJ0YS9K4J66H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFJWRGDWW15C8M6ZXECB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFK7V61YSE03MDGAEDXM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:28Z · abandoned wrun_41M0K8J6690GQZ3TBQCZ186JCW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:28Z · abandoned wrun_41M0K8J2NE0GJF7X6RMVFPP1C0
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:41Z · abandoned wrun_01M0K8MCVBJKXXM7HC8W3AXS0V
  • run-pickup-stall · pathsAliasWorkflow - TypeScript path aliases resolve correctly (nextjs-webpack) · at 22:58:54Z · abandoned wrun_01M0K8REC1AXDVSBRCCMBQJJNM
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production3546327424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total1565941254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node130228
❌ astro-quickjs131128
❌ example-node131128
❌ example-quickjs131128
❌ express-node131128
❌ express-quickjs130228
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
❌ hono-quickjs131128
❌ nest-node131128
❌ nest-quickjs131128
✅ nextjs-turbopack-node15703
❌ nextjs-turbopack-quickjs15523
❌ nextjs-webpack-node15613
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
❌ nitro-quickjs130228
✅ nuxt-node132028
❌ nuxt-quickjs130228
❌ python-node08152
❌ sveltekit-node15019
❌ sveltekit-quickjs15019
❌ tanstack-start-node130228
✅ tanstack-start-quickjs132028
❌ vite-node131128
❌ vite-quickjs131128
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 023e049 · Fri, 21 Aug 2026 23:05:25 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1143 (+6.5%)1414 🔴 (+22%) 🔻1459 🔴 (+17%) 🔻1672 🔴 (+17%) 🔻30
TTFSstream1310 (+24%) 🔻1403 🔴 (+16%) 🔻1426 🔴 (+11%)1488 🔴 (+3.0%)30
TTFShook + stream1600 (+51%) 🔻1701 🔴 (+17%) 🔻1733 🔴 (+15%) 🔻1791 🔴 (+16%) 🔻30
Fan-out TTFSPromise.all(100 steps)540 (-8.8%)734 (-54%) 💚920 (-44%) 💚1060 (-36%) 💚10
Fan-out TTLSPromise.all(100 steps)1598 (-1.2%)2888 (-3.9%)7336 (+104%) 🔻7478 (+90%) 🔻10
STSO1020 steps (inline)129 (-14%)167 (-24%) 💚181 (-30%) 💚257 (-52%) 💚1019
WO1020 steps164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚1
CRTTfirst chunk (pooled)81 (-15%)163 (+10%)241 (+39%) 🔻466 (+101%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)114 (-12%)127 (-31%)188 (-26%)435 (+14%)127 (-25%)10
size sweep (100/s, 160B-12KB)109 (-12%)116 (-37%)172 (-49%)343 (-34%)107 (-19%)10
replay gateway-gpt-5.4-nano-2000t (1x)103 (-15%)107 (-85%)174 (-95%)509 (-89%)239 (-93%)3
replay eve-gpt-5.6-sol-2000t (1x)108 (-11%)108 (-33%)138 (-33%)315 (-12%)226 (-25%)2
replay eve-gpt-5.6-sol-2000t (2x)162 (+5%)212 (-35%)463 (-53%)3872 (+71%)723 (+83%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 217526ms → this run 164269ms (Δ -53257ms, -24%)
 100-150 ms ░░░░░░░░┃ main 0 this 268 +268
150-200 ms ██████████████████░░░░░┃ main 523 this 717 +194
200-250 ms ┃████████████ main 379 this 23 -356
250-300 ms ┃█ main 65 this 6 -59
300-350 ms ┃ main 23 this 2 -21
350-400 ms ┃ main 10 this 2 -8
400-450 ms ┃ main 4 this 0 -4
450-500 ms ┃ main 1 this 0 -1
500-550 ms ┃ main 5 this 1 -4
550-600 ms ┃ main 4 this 0 -4
600-650 ms ┃ main 1 this 0 -1
650-700 ms ┃ main 1 this 0 -1
750-800 ms ┃ main 1 this 0 -1
900-950 ms ┃ main 1 this 0 -1
1050-1100 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······█▆▁···· 104.6 (-29%) 95 (-32%) 188 (-26%) 435 (+14%) 3000
sweep ·····▁█▅▁···· 99.5 (-33%) 92 (-29%) 172 (-49%) 343 (-34%) 3000
gw 1x ·····▁█▃▁▁··· 97.3 (-81%) 85 (-51%) 174 (-95%) 509 (-89%) 5295
eve 1x ·····▁█▄▁···· 93.4 (-32%) 86 (-31%) 138 (-33%) 315 (-12%) 5186
eve 2x ·····▁▆█▃▁·▁· 246.2 (-4%) 125 (-27%) 463 (-53%) 3872 (+71%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▂▁▁▂▆▅█▃▄ 95–120ms
sweep ▇█▆▂▂▁▂▂▁▁ 91–119ms
gw 1x █▂▁▂▁▂▁▂▁▃ 83–154ms
eve 1x ▂▂▃▃▁▂▆█▄▁ 82–117ms
eve 2x ▁▁▁▁▁▁▂▂▁█ 105–1109ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ██▆▅▃▁▂ 96–103ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▁█▅▄▆▅▅▄▃ 25–34ms
sweep ▃█▅▆▆▆▆▄▁▂ 32–43ms
gw 1x █▃▂▁▂▂▂▂▂▃ 22–37ms
eve 1x ▁▁▁▄▂▃▆▇█▄ 18–25ms
eve 2x ▂▃▁▁▁▃▂▁▁█ 20–45ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
Comment thread.github/workflows/lint.yml
Outdated

 exit 1
 fi

 wired-test-suites:
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get a better name?
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to no-unrun-tests / No Unrun Tests, matching the no-test-overrides / No Test Overrides sibling right above it — both are short negative invariants. The script moved to scripts/check-no-unrun-tests.mjs to match, and the step now reads "Check every test file is run by turbo test or a CI lane".
Comment thread.changeset/run-e2e-utils-in-unit-lane.md
Outdated

'@workflow/core': patch
---

Run `e2e/utils.test.ts` from the package's `test` script; it covers the e2e helpers themselves and needs no deployment, so nothing had been running it.
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe don't need two changeset entries just for internal test changes
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped both. Nothing here changes shipped behaviour — the only edits to published packages are a test script and a vitest devDependency — so there is nothing to release. (No status check gates on the changeset, so this stays green.)
@pranaygp@claude
Drop the changesets and rename the guard job
023e049
The PR only wires up existing suites and adds a CI check, so there is nothing
to release. Rename the job to match its `no-test-overrides` sibling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@pranaygp
pranaygp
enabled auto-merge (squash)
August 21, 2026 22:49
@pranaygp
pranaygp
disabled auto-merge
August 21, 2026 22:59
@pranaygp
pranaygp
merged commit 447013b
into
mainAug 21, 2026

147 of 167 checks passed

@pranaygp
pranaygp
deleted the

pgp/workflow-world-tests

branch
August 21, 2026 22:59
@github-actions
github-actionsBot
commented
Aug 21, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 447013b due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
447013b73a5c240b4d17011faf45f5cfd7b4d956

pranaygp added a commit
that referenced
this pull request
Aug 21, 2026
@pranaygp
Merge remote-tracking branch 'origin/main' into pgp/bundling-regressions
9979bb5
* origin/main:
Run the test suites CI was silently skipping (#3733)
[e2e] Change race repro hook poke to soft-degrade instead of hard-stop at budget (#3561)
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

@workflow/world has 13 test files and no test script, so none of them run in CI


3 participants

@pranaygp@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Run the test suites CI was silently skipping - #3733
Merged
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Aug 21, 2026
Merged
Run the test suites CI was silently skipping#3733
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Conversation
@pranaygp
@pranaygppranaygp
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#3731.
turbo test runs a package's tests only if that package declares a test script. A suite without one is skipped in silence — the files look like coverage, and the assertions inside them rot with nobody watching. #3731 found @workflow/world in that state; checking the rest of the workspace found three more.
What was unrun
SuiteFilesTests
@workflow/world13160
@workflow/cli551
@workflow/nitro130
packages/core/e2e/utils.test.ts115
packages/core/e2e/manifest.test.ts136
All 292 pass as-is, so this adds no red. @workflow/world, @workflow/cli and @workflow/nitro get a test script (vitest run src) and a vitest devDependency, matching every other package.
The two packages/core/e2e files are a different shape of the same bug: that directory is run by dedicated workflow steps naming each file, and these two were named by none.
  • e2e/utils.test.ts tests the e2e helpers themselves and needs no deployment, so it joins the unit lane through @workflow/core's test script.
  • e2e/manifest.test.ts asserts the shape of a built workbench manifest, so it gets a step in E2E Local Prod immediately after local-build.test.ts — which is what runs pnpm build in the workbench app. It needs its own invocation rather than another file in that step, since vitest would otherwise run both concurrently, against a manifest that does not exist yet.
The spec-version.test.ts failure in the issue
Already gone. #3634 rewrote that assertion when it moved the current version to 7 (sealed log), so the file now pins SPEC_VERSION_CURRENT === SPEC_VERSION_SUPPORTS_SEALED_LOG rather than the stale compression version. Nothing to update here — but note that the fix landed blind, because the suite proving it still never ran.
Keeping it wired
scripts/check-no-unrun-tests.mjs walks the workspace and requires every test file to be reachable one of two ways:
  1. its package has a test script whose positional filters cover the file, so turbo test runs it — vitest treats positionals as filters, so vitest run src beside a test in e2e/ drops that file quietly; or
  2. a CI workflow, or a root package.json script, names the file (or a directory holding it) outright.
Anything else fails. It runs as a new No Unrun Tests job in lint.yml (node only, no install, alongside the other hygiene jobs). One waiver: @workflow/docs-typecheck, whose suite is far too slow for the unit lane and runs as pnpm test:docs from Docs Checks.
Path matching is boundary-aware on purpose. A plain substring search reads packages/world as present in every workflow mentioning packages/world-vercel, and a trailing / counts as continuation so that a lane naming packages/core/e2e/local-build.test.ts says nothing about the other files in that directory. Without both rules the check passes on the very state it exists to catch.
Verification
Each regression was re-introduced one at a time and the guard flagged it:
✗ @workflow/world (packages/world) has 13 test file(s) but no "test" script,
so `turbo test` skips the package entirely
✗ @workflow/core (packages/core) has test file(s) that neither its "test"
script nor any CI workflow runs: e2e/manifest.test.ts

turbo test after the fix:
@workflow/world:test: Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test: Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test: Test Files 1 passed (1) Tests 30 passed
@workflow/core:test: Test Files 104 passed Tests 2259 passed

manifest.test.ts was checked against all eight workbench apps with manifests, built locally: 36 passed.
CI confirms the suites actually execute rather than the job merely going green. Both unit lanes pass, and their logs show:
@workflow/world:test Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test Test Files 1 passed (1) Tests 30 passed
@workflow/core:test ✓ e2e/utils.test.ts (15 tests)

Windows matters here for two reasons, both now checked rather than assumed: nitro's assertions hard-code POSIX paths and hold because the source builds those specifiers with pathe, and the e2e/utils.test.ts positional filter does match there.
@workflow/world-postgres fails locally without Docker for testcontainers; it passes in CI (6 files / 193 tests).
🤖 Generated with Claude Code
@pranaygp@claude
Run the test suites CI was silently skipping
94c9088
`turbo test` runs a package's tests only if that package declares a `test`
script, so a suite can sit in the repo for months without ever running. Four
were in that state: @workflow/world (13 files, 160 tests), @workflow/cli (5 /
51), @workflow/nitro (1 / 30), and two files under packages/core/e2e that no
workflow named.
Wire each one up, and add scripts/check-test-suites-wired.mjs plus a lint job
so the next unwired suite fails CI instead of going unnoticed.
Fixes#3731
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pranaygp
pranaygp
requested a review
from a team
as a code ownerAugust 21, 2026 22:45
CopilotAI
lite review requested due to automatic review settings
August 21, 2026 22:45
@vercel
vercelBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-astro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-express-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-hono-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-python-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-vite-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-docsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-tarballsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-webReadyReadyPreview, v0Aug 21, 2026 10:52pm
CopilotAI
reviewed
Aug 21, 2026
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
@changeset-bot
changeset-botBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: 023e049
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (32 failed)
astro-node (2 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
astro-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
example-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
example-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
express-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
express-quickjs (2 failed):
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook | wrun_41M0K8E2780GKJZTMNTCF7TXRS | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
hono-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
nest-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
nest-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
nextjs-turbopack-quickjs (2 failed):
  • webhookWorkflow | wrun_41M0K8ASG90GPZGENG5QRFKYZN | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nextjs-webpack-node (1 failed):
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
nitro-quickjs (2 failed):
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nuxt-quickjs (2 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC | 🔍 observability
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ | 🔍 observability
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X | 🔍 observability
sveltekit-node (1 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
sveltekit-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
tanstack-start-node (2 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
vite-node (1 failed):
  • distributedAbortController - reconnect to existing controller | wrun_41M0K8HAHE0GVTB2BAFTQ84TC9 | 🔍 observability
vite-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K8H69RZBXWE62FS3BH7CAC
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
33 flaky tests
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (astro)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (nextjs-webpack)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (astro)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nuxt)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (vite)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (fastify)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-turbopack)
  • abortHookOrderingWorkflow [hook-first-abort-first]: hook.then → addEventListener → abort() → resumeHook (astro)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (fastify)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (hono)
  • abortReasonWorkflow: abort reason preserved across boundaries (astro)
  • abortReasonWorkflow: abort reason preserved across boundaries (fastify)
  • abortSurvivesReplayWorkflow: controller state consistent across replay (nuxt)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (astro)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (hono)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (nextjs-webpack)
  • addTenWorkflow (tanstack-start)
  • Calculator.calculate - static workflow method using static step methods from another class (nextjs-webpack)
  • distributedAbortController - manual abort triggers signal (nest)
  • distributedAbortController - manual abort triggers signal (sveltekit)
  • distributedAbortController - reconnect to existing controller (nitro)
  • distributedAbortController - reconnect to existing controller (sveltekit)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (express)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (hono)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nest)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (astro)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (hono)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (hono)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nitro)
  • webhookWorkflow (fastify)
  • workflowAndStepMetadataWorkflow (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (tanstack-start) · at 22:52:19Z · abandoned wrun_01M0K8C0WW5RQWS78EE4YVMKCW
  • cold-start-warmup · suite warmup (python) · at 22:52:39Z · abandoned wrun_41M0K89TTX0GY1APHXTS6PK89D · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H1
  • run-pickup-stall · nullByteWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFH10GX9K9PD2KZR3385
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGQ0GSPXK5H1GS7FCZ4
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFTX0GGSWRS14CZT9CRR
  • cold-start-warmup · suite warmup (python) · at 22:53:26Z · abandoned wrun_01M0K8B7VY7211258MVNH21EQQ · (+7 more)
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:53:28Z · abandoned wrun_41M0K8EF470GYTVH8A32M271NQ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0XWVW7FD6CWJSYS6GS
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX129NGA4EPDJ1E1XGGJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX13T6P1P1B6GBG27APD
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX15QXRX5S8P4TKAWE5R
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0Y1CK71YTJZ4PK2R91
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FASZ0GY8S51B3QA9804D
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FARV0GWX9AS5V5VKQ1N5
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FB1P0GHQ6SBP1AGW8S06
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FAW90GR7NW0F4HEBR5R9
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMHB4E9B13993WC73X5
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMFPFC849RM811TNR0E
  • run-pickup-stall · sleepingWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMM0TTN8V5VS10CRPEE
  • run-pickup-stall · nullByteWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMXA75AGR0ZG2ZQ5HST
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMNPJ85RDG23KCH8HJH
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:54:47Z · abandoned wrun_41M0K8FE9Q0GYH1F346WCF17T2
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ7XZQ9GXF1KSYT77GZ0
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ86FE14B8R6MNJRA9S7
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ89NC11EJ0YS9K4J66H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFJWRGDWW15C8M6ZXECB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFK7V61YSE03MDGAEDXM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:28Z · abandoned wrun_41M0K8J6690GQZ3TBQCZ186JCW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:28Z · abandoned wrun_41M0K8J2NE0GJF7X6RMVFPP1C0
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:41Z · abandoned wrun_01M0K8MCVBJKXXM7HC8W3AXS0V
  • run-pickup-stall · pathsAliasWorkflow - TypeScript path aliases resolve correctly (nextjs-webpack) · at 22:58:54Z · abandoned wrun_01M0K8REC1AXDVSBRCCMBQJJNM
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production3546327424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total1565941254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node130228
❌ astro-quickjs131128
❌ example-node131128
❌ example-quickjs131128
❌ express-node131128
❌ express-quickjs130228
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
❌ hono-quickjs131128
❌ nest-node131128
❌ nest-quickjs131128
✅ nextjs-turbopack-node15703
❌ nextjs-turbopack-quickjs15523
❌ nextjs-webpack-node15613
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
❌ nitro-quickjs130228
✅ nuxt-node132028
❌ nuxt-quickjs130228
❌ python-node08152
❌ sveltekit-node15019
❌ sveltekit-quickjs15019
❌ tanstack-start-node130228
✅ tanstack-start-quickjs132028
❌ vite-node131128
❌ vite-quickjs131128
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 023e049 · Fri, 21 Aug 2026 23:05:25 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1143 (+6.5%)1414 🔴 (+22%) 🔻1459 🔴 (+17%) 🔻1672 🔴 (+17%) 🔻30
TTFSstream1310 (+24%) 🔻1403 🔴 (+16%) 🔻1426 🔴 (+11%)1488 🔴 (+3.0%)30
TTFShook + stream1600 (+51%) 🔻1701 🔴 (+17%) 🔻1733 🔴 (+15%) 🔻1791 🔴 (+16%) 🔻30
Fan-out TTFSPromise.all(100 steps)540 (-8.8%)734 (-54%) 💚920 (-44%) 💚1060 (-36%) 💚10
Fan-out TTLSPromise.all(100 steps)1598 (-1.2%)2888 (-3.9%)7336 (+104%) 🔻7478 (+90%) 🔻10
STSO1020 steps (inline)129 (-14%)167 (-24%) 💚181 (-30%) 💚257 (-52%) 💚1019
WO1020 steps164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚1
CRTTfirst chunk (pooled)81 (-15%)163 (+10%)241 (+39%) 🔻466 (+101%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)114 (-12%)127 (-31%)188 (-26%)435 (+14%)127 (-25%)10
size sweep (100/s, 160B-12KB)109 (-12%)116 (-37%)172 (-49%)343 (-34%)107 (-19%)10
replay gateway-gpt-5.4-nano-2000t (1x)103 (-15%)107 (-85%)174 (-95%)509 (-89%)239 (-93%)3
replay eve-gpt-5.6-sol-2000t (1x)108 (-11%)108 (-33%)138 (-33%)315 (-12%)226 (-25%)2
replay eve-gpt-5.6-sol-2000t (2x)162 (+5%)212 (-35%)463 (-53%)3872 (+71%)723 (+83%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 217526ms → this run 164269ms (Δ -53257ms, -24%)
 100-150 ms ░░░░░░░░┃ main 0 this 268 +268
150-200 ms ██████████████████░░░░░┃ main 523 this 717 +194
200-250 ms ┃████████████ main 379 this 23 -356
250-300 ms ┃█ main 65 this 6 -59
300-350 ms ┃ main 23 this 2 -21
350-400 ms ┃ main 10 this 2 -8
400-450 ms ┃ main 4 this 0 -4
450-500 ms ┃ main 1 this 0 -1
500-550 ms ┃ main 5 this 1 -4
550-600 ms ┃ main 4 this 0 -4
600-650 ms ┃ main 1 this 0 -1
650-700 ms ┃ main 1 this 0 -1
750-800 ms ┃ main 1 this 0 -1
900-950 ms ┃ main 1 this 0 -1
1050-1100 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······█▆▁···· 104.6 (-29%) 95 (-32%) 188 (-26%) 435 (+14%) 3000
sweep ·····▁█▅▁···· 99.5 (-33%) 92 (-29%) 172 (-49%) 343 (-34%) 3000
gw 1x ·····▁█▃▁▁··· 97.3 (-81%) 85 (-51%) 174 (-95%) 509 (-89%) 5295
eve 1x ·····▁█▄▁···· 93.4 (-32%) 86 (-31%) 138 (-33%) 315 (-12%) 5186
eve 2x ·····▁▆█▃▁·▁· 246.2 (-4%) 125 (-27%) 463 (-53%) 3872 (+71%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▂▁▁▂▆▅█▃▄ 95–120ms
sweep ▇█▆▂▂▁▂▂▁▁ 91–119ms
gw 1x █▂▁▂▁▂▁▂▁▃ 83–154ms
eve 1x ▂▂▃▃▁▂▆█▄▁ 82–117ms
eve 2x ▁▁▁▁▁▁▂▂▁█ 105–1109ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ██▆▅▃▁▂ 96–103ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▁█▅▄▆▅▅▄▃ 25–34ms
sweep ▃█▅▆▆▆▆▄▁▂ 32–43ms
gw 1x █▃▂▁▂▂▂▂▂▃ 22–37ms
eve 1x ▁▁▁▄▂▃▆▇█▄ 18–25ms
eve 2x ▂▃▁▁▁▃▂▁▁█ 20–45ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
Comment thread.github/workflows/lint.yml
Outdated

 exit 1
 fi

 wired-test-suites:
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get a better name?
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to no-unrun-tests / No Unrun Tests, matching the no-test-overrides / No Test Overrides sibling right above it — both are short negative invariants. The script moved to scripts/check-no-unrun-tests.mjs to match, and the step now reads "Check every test file is run by turbo test or a CI lane".
Comment thread.changeset/run-e2e-utils-in-unit-lane.md
Outdated

'@workflow/core': patch
---

Run `e2e/utils.test.ts` from the package's `test` script; it covers the e2e helpers themselves and needs no deployment, so nothing had been running it.
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe don't need two changeset entries just for internal test changes
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped both. Nothing here changes shipped behaviour — the only edits to published packages are a test script and a vitest devDependency — so there is nothing to release. (No status check gates on the changeset, so this stays green.)
@pranaygp@claude
Drop the changesets and rename the guard job
023e049
The PR only wires up existing suites and adds a CI check, so there is nothing
to release. Rename the job to match its `no-test-overrides` sibling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@pranaygp
pranaygp
enabled auto-merge (squash)
August 21, 2026 22:49
@pranaygp
pranaygp
disabled auto-merge
August 21, 2026 22:59
@pranaygp
pranaygp
merged commit 447013b
into
mainAug 21, 2026

147 of 167 checks passed

@pranaygp
pranaygp
deleted the

pgp/workflow-world-tests

branch
August 21, 2026 22:59
@github-actions
github-actionsBot
commented
Aug 21, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 447013b due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
447013b73a5c240b4d17011faf45f5cfd7b4d956

pranaygp added a commit
that referenced
this pull request
Aug 21, 2026
@pranaygp
Merge remote-tracking branch 'origin/main' into pgp/bundling-regressions
9979bb5
* origin/main:
Run the test suites CI was silently skipping (#3733)
[e2e] Change race repro hook poke to soft-degrade instead of hard-stop at budget (#3561)
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

@workflow/world has 13 test files and no test script, so none of them run in CI


3 participants

@pranaygp@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Run the test suites CI was silently skipping - #3733
Merged
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Aug 21, 2026
Merged
Run the test suites CI was silently skipping#3733
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Conversation
@pranaygp
@pranaygppranaygp
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#3731.
turbo test runs a package's tests only if that package declares a test script. A suite without one is skipped in silence — the files look like coverage, and the assertions inside them rot with nobody watching. #3731 found @workflow/world in that state; checking the rest of the workspace found three more.
What was unrun
SuiteFilesTests
@workflow/world13160
@workflow/cli551
@workflow/nitro130
packages/core/e2e/utils.test.ts115
packages/core/e2e/manifest.test.ts136
All 292 pass as-is, so this adds no red. @workflow/world, @workflow/cli and @workflow/nitro get a test script (vitest run src) and a vitest devDependency, matching every other package.
The two packages/core/e2e files are a different shape of the same bug: that directory is run by dedicated workflow steps naming each file, and these two were named by none.
  • e2e/utils.test.ts tests the e2e helpers themselves and needs no deployment, so it joins the unit lane through @workflow/core's test script.
  • e2e/manifest.test.ts asserts the shape of a built workbench manifest, so it gets a step in E2E Local Prod immediately after local-build.test.ts — which is what runs pnpm build in the workbench app. It needs its own invocation rather than another file in that step, since vitest would otherwise run both concurrently, against a manifest that does not exist yet.
The spec-version.test.ts failure in the issue
Already gone. #3634 rewrote that assertion when it moved the current version to 7 (sealed log), so the file now pins SPEC_VERSION_CURRENT === SPEC_VERSION_SUPPORTS_SEALED_LOG rather than the stale compression version. Nothing to update here — but note that the fix landed blind, because the suite proving it still never ran.
Keeping it wired
scripts/check-no-unrun-tests.mjs walks the workspace and requires every test file to be reachable one of two ways:
  1. its package has a test script whose positional filters cover the file, so turbo test runs it — vitest treats positionals as filters, so vitest run src beside a test in e2e/ drops that file quietly; or
  2. a CI workflow, or a root package.json script, names the file (or a directory holding it) outright.
Anything else fails. It runs as a new No Unrun Tests job in lint.yml (node only, no install, alongside the other hygiene jobs). One waiver: @workflow/docs-typecheck, whose suite is far too slow for the unit lane and runs as pnpm test:docs from Docs Checks.
Path matching is boundary-aware on purpose. A plain substring search reads packages/world as present in every workflow mentioning packages/world-vercel, and a trailing / counts as continuation so that a lane naming packages/core/e2e/local-build.test.ts says nothing about the other files in that directory. Without both rules the check passes on the very state it exists to catch.
Verification
Each regression was re-introduced one at a time and the guard flagged it:
✗ @workflow/world (packages/world) has 13 test file(s) but no "test" script,
so `turbo test` skips the package entirely
✗ @workflow/core (packages/core) has test file(s) that neither its "test"
script nor any CI workflow runs: e2e/manifest.test.ts

turbo test after the fix:
@workflow/world:test: Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test: Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test: Test Files 1 passed (1) Tests 30 passed
@workflow/core:test: Test Files 104 passed Tests 2259 passed

manifest.test.ts was checked against all eight workbench apps with manifests, built locally: 36 passed.
CI confirms the suites actually execute rather than the job merely going green. Both unit lanes pass, and their logs show:
@workflow/world:test Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test Test Files 1 passed (1) Tests 30 passed
@workflow/core:test ✓ e2e/utils.test.ts (15 tests)

Windows matters here for two reasons, both now checked rather than assumed: nitro's assertions hard-code POSIX paths and hold because the source builds those specifiers with pathe, and the e2e/utils.test.ts positional filter does match there.
@workflow/world-postgres fails locally without Docker for testcontainers; it passes in CI (6 files / 193 tests).
🤖 Generated with Claude Code
@pranaygp@claude
Run the test suites CI was silently skipping
94c9088
`turbo test` runs a package's tests only if that package declares a `test`
script, so a suite can sit in the repo for months without ever running. Four
were in that state: @workflow/world (13 files, 160 tests), @workflow/cli (5 /
51), @workflow/nitro (1 / 30), and two files under packages/core/e2e that no
workflow named.
Wire each one up, and add scripts/check-test-suites-wired.mjs plus a lint job
so the next unwired suite fails CI instead of going unnoticed.
Fixes#3731
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pranaygp
pranaygp
requested a review
from a team
as a code ownerAugust 21, 2026 22:45
CopilotAI
lite review requested due to automatic review settings
August 21, 2026 22:45
@vercel
vercelBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-astro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-express-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-hono-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-python-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-vite-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-docsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-tarballsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-webReadyReadyPreview, v0Aug 21, 2026 10:52pm
CopilotAI
reviewed
Aug 21, 2026
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
@changeset-bot
changeset-botBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: 023e049
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (32 failed)
astro-node (2 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
astro-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
example-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
example-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
express-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
express-quickjs (2 failed):
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook | wrun_41M0K8E2780GKJZTMNTCF7TXRS | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
hono-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
nest-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
nest-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
nextjs-turbopack-quickjs (2 failed):
  • webhookWorkflow | wrun_41M0K8ASG90GPZGENG5QRFKYZN | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nextjs-webpack-node (1 failed):
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
nitro-quickjs (2 failed):
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nuxt-quickjs (2 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC | 🔍 observability
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ | 🔍 observability
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X | 🔍 observability
sveltekit-node (1 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
sveltekit-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
tanstack-start-node (2 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
vite-node (1 failed):
  • distributedAbortController - reconnect to existing controller | wrun_41M0K8HAHE0GVTB2BAFTQ84TC9 | 🔍 observability
vite-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K8H69RZBXWE62FS3BH7CAC
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
33 flaky tests
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (astro)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (nextjs-webpack)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (astro)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nuxt)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (vite)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (fastify)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-turbopack)
  • abortHookOrderingWorkflow [hook-first-abort-first]: hook.then → addEventListener → abort() → resumeHook (astro)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (fastify)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (hono)
  • abortReasonWorkflow: abort reason preserved across boundaries (astro)
  • abortReasonWorkflow: abort reason preserved across boundaries (fastify)
  • abortSurvivesReplayWorkflow: controller state consistent across replay (nuxt)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (astro)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (hono)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (nextjs-webpack)
  • addTenWorkflow (tanstack-start)
  • Calculator.calculate - static workflow method using static step methods from another class (nextjs-webpack)
  • distributedAbortController - manual abort triggers signal (nest)
  • distributedAbortController - manual abort triggers signal (sveltekit)
  • distributedAbortController - reconnect to existing controller (nitro)
  • distributedAbortController - reconnect to existing controller (sveltekit)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (express)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (hono)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nest)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (astro)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (hono)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (hono)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nitro)
  • webhookWorkflow (fastify)
  • workflowAndStepMetadataWorkflow (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (tanstack-start) · at 22:52:19Z · abandoned wrun_01M0K8C0WW5RQWS78EE4YVMKCW
  • cold-start-warmup · suite warmup (python) · at 22:52:39Z · abandoned wrun_41M0K89TTX0GY1APHXTS6PK89D · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H1
  • run-pickup-stall · nullByteWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFH10GX9K9PD2KZR3385
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGQ0GSPXK5H1GS7FCZ4
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFTX0GGSWRS14CZT9CRR
  • cold-start-warmup · suite warmup (python) · at 22:53:26Z · abandoned wrun_01M0K8B7VY7211258MVNH21EQQ · (+7 more)
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:53:28Z · abandoned wrun_41M0K8EF470GYTVH8A32M271NQ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0XWVW7FD6CWJSYS6GS
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX129NGA4EPDJ1E1XGGJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX13T6P1P1B6GBG27APD
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX15QXRX5S8P4TKAWE5R
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0Y1CK71YTJZ4PK2R91
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FASZ0GY8S51B3QA9804D
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FARV0GWX9AS5V5VKQ1N5
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FB1P0GHQ6SBP1AGW8S06
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FAW90GR7NW0F4HEBR5R9
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMHB4E9B13993WC73X5
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMFPFC849RM811TNR0E
  • run-pickup-stall · sleepingWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMM0TTN8V5VS10CRPEE
  • run-pickup-stall · nullByteWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMXA75AGR0ZG2ZQ5HST
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMNPJ85RDG23KCH8HJH
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:54:47Z · abandoned wrun_41M0K8FE9Q0GYH1F346WCF17T2
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ7XZQ9GXF1KSYT77GZ0
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ86FE14B8R6MNJRA9S7
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ89NC11EJ0YS9K4J66H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFJWRGDWW15C8M6ZXECB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFK7V61YSE03MDGAEDXM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:28Z · abandoned wrun_41M0K8J6690GQZ3TBQCZ186JCW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:28Z · abandoned wrun_41M0K8J2NE0GJF7X6RMVFPP1C0
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:41Z · abandoned wrun_01M0K8MCVBJKXXM7HC8W3AXS0V
  • run-pickup-stall · pathsAliasWorkflow - TypeScript path aliases resolve correctly (nextjs-webpack) · at 22:58:54Z · abandoned wrun_01M0K8REC1AXDVSBRCCMBQJJNM
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production3546327424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total1565941254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node130228
❌ astro-quickjs131128
❌ example-node131128
❌ example-quickjs131128
❌ express-node131128
❌ express-quickjs130228
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
❌ hono-quickjs131128
❌ nest-node131128
❌ nest-quickjs131128
✅ nextjs-turbopack-node15703
❌ nextjs-turbopack-quickjs15523
❌ nextjs-webpack-node15613
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
❌ nitro-quickjs130228
✅ nuxt-node132028
❌ nuxt-quickjs130228
❌ python-node08152
❌ sveltekit-node15019
❌ sveltekit-quickjs15019
❌ tanstack-start-node130228
✅ tanstack-start-quickjs132028
❌ vite-node131128
❌ vite-quickjs131128
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 023e049 · Fri, 21 Aug 2026 23:05:25 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1143 (+6.5%)1414 🔴 (+22%) 🔻1459 🔴 (+17%) 🔻1672 🔴 (+17%) 🔻30
TTFSstream1310 (+24%) 🔻1403 🔴 (+16%) 🔻1426 🔴 (+11%)1488 🔴 (+3.0%)30
TTFShook + stream1600 (+51%) 🔻1701 🔴 (+17%) 🔻1733 🔴 (+15%) 🔻1791 🔴 (+16%) 🔻30
Fan-out TTFSPromise.all(100 steps)540 (-8.8%)734 (-54%) 💚920 (-44%) 💚1060 (-36%) 💚10
Fan-out TTLSPromise.all(100 steps)1598 (-1.2%)2888 (-3.9%)7336 (+104%) 🔻7478 (+90%) 🔻10
STSO1020 steps (inline)129 (-14%)167 (-24%) 💚181 (-30%) 💚257 (-52%) 💚1019
WO1020 steps164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚1
CRTTfirst chunk (pooled)81 (-15%)163 (+10%)241 (+39%) 🔻466 (+101%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)114 (-12%)127 (-31%)188 (-26%)435 (+14%)127 (-25%)10
size sweep (100/s, 160B-12KB)109 (-12%)116 (-37%)172 (-49%)343 (-34%)107 (-19%)10
replay gateway-gpt-5.4-nano-2000t (1x)103 (-15%)107 (-85%)174 (-95%)509 (-89%)239 (-93%)3
replay eve-gpt-5.6-sol-2000t (1x)108 (-11%)108 (-33%)138 (-33%)315 (-12%)226 (-25%)2
replay eve-gpt-5.6-sol-2000t (2x)162 (+5%)212 (-35%)463 (-53%)3872 (+71%)723 (+83%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 217526ms → this run 164269ms (Δ -53257ms, -24%)
 100-150 ms ░░░░░░░░┃ main 0 this 268 +268
150-200 ms ██████████████████░░░░░┃ main 523 this 717 +194
200-250 ms ┃████████████ main 379 this 23 -356
250-300 ms ┃█ main 65 this 6 -59
300-350 ms ┃ main 23 this 2 -21
350-400 ms ┃ main 10 this 2 -8
400-450 ms ┃ main 4 this 0 -4
450-500 ms ┃ main 1 this 0 -1
500-550 ms ┃ main 5 this 1 -4
550-600 ms ┃ main 4 this 0 -4
600-650 ms ┃ main 1 this 0 -1
650-700 ms ┃ main 1 this 0 -1
750-800 ms ┃ main 1 this 0 -1
900-950 ms ┃ main 1 this 0 -1
1050-1100 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······█▆▁···· 104.6 (-29%) 95 (-32%) 188 (-26%) 435 (+14%) 3000
sweep ·····▁█▅▁···· 99.5 (-33%) 92 (-29%) 172 (-49%) 343 (-34%) 3000
gw 1x ·····▁█▃▁▁··· 97.3 (-81%) 85 (-51%) 174 (-95%) 509 (-89%) 5295
eve 1x ·····▁█▄▁···· 93.4 (-32%) 86 (-31%) 138 (-33%) 315 (-12%) 5186
eve 2x ·····▁▆█▃▁·▁· 246.2 (-4%) 125 (-27%) 463 (-53%) 3872 (+71%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▂▁▁▂▆▅█▃▄ 95–120ms
sweep ▇█▆▂▂▁▂▂▁▁ 91–119ms
gw 1x █▂▁▂▁▂▁▂▁▃ 83–154ms
eve 1x ▂▂▃▃▁▂▆█▄▁ 82–117ms
eve 2x ▁▁▁▁▁▁▂▂▁█ 105–1109ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ██▆▅▃▁▂ 96–103ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▁█▅▄▆▅▅▄▃ 25–34ms
sweep ▃█▅▆▆▆▆▄▁▂ 32–43ms
gw 1x █▃▂▁▂▂▂▂▂▃ 22–37ms
eve 1x ▁▁▁▄▂▃▆▇█▄ 18–25ms
eve 2x ▂▃▁▁▁▃▂▁▁█ 20–45ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
Comment thread.github/workflows/lint.yml
Outdated

 exit 1
 fi

 wired-test-suites:
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get a better name?
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to no-unrun-tests / No Unrun Tests, matching the no-test-overrides / No Test Overrides sibling right above it — both are short negative invariants. The script moved to scripts/check-no-unrun-tests.mjs to match, and the step now reads "Check every test file is run by turbo test or a CI lane".
Comment thread.changeset/run-e2e-utils-in-unit-lane.md
Outdated

'@workflow/core': patch
---

Run `e2e/utils.test.ts` from the package's `test` script; it covers the e2e helpers themselves and needs no deployment, so nothing had been running it.
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe don't need two changeset entries just for internal test changes
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped both. Nothing here changes shipped behaviour — the only edits to published packages are a test script and a vitest devDependency — so there is nothing to release. (No status check gates on the changeset, so this stays green.)
@pranaygp@claude
Drop the changesets and rename the guard job
023e049
The PR only wires up existing suites and adds a CI check, so there is nothing
to release. Rename the job to match its `no-test-overrides` sibling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@pranaygp
pranaygp
enabled auto-merge (squash)
August 21, 2026 22:49
@pranaygp
pranaygp
disabled auto-merge
August 21, 2026 22:59
@pranaygp
pranaygp
merged commit 447013b
into
mainAug 21, 2026

147 of 167 checks passed

@pranaygp
pranaygp
deleted the

pgp/workflow-world-tests

branch
August 21, 2026 22:59
@github-actions
github-actionsBot
commented
Aug 21, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 447013b due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
447013b73a5c240b4d17011faf45f5cfd7b4d956

pranaygp added a commit
that referenced
this pull request
Aug 21, 2026
@pranaygp
Merge remote-tracking branch 'origin/main' into pgp/bundling-regressions
9979bb5
* origin/main:
Run the test suites CI was silently skipping (#3733)
[e2e] Change race repro hook poke to soft-degrade instead of hard-stop at budget (#3561)
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

@workflow/world has 13 test files and no test script, so none of them run in CI


3 participants

@pranaygp@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Run the test suites CI was silently skipping - #3733
Merged
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Aug 21, 2026
Merged
Run the test suites CI was silently skipping#3733
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Conversation
@pranaygp
@pranaygppranaygp
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#3731.
turbo test runs a package's tests only if that package declares a test script. A suite without one is skipped in silence — the files look like coverage, and the assertions inside them rot with nobody watching. #3731 found @workflow/world in that state; checking the rest of the workspace found three more.
What was unrun
SuiteFilesTests
@workflow/world13160
@workflow/cli551
@workflow/nitro130
packages/core/e2e/utils.test.ts115
packages/core/e2e/manifest.test.ts136
All 292 pass as-is, so this adds no red. @workflow/world, @workflow/cli and @workflow/nitro get a test script (vitest run src) and a vitest devDependency, matching every other package.
The two packages/core/e2e files are a different shape of the same bug: that directory is run by dedicated workflow steps naming each file, and these two were named by none.
  • e2e/utils.test.ts tests the e2e helpers themselves and needs no deployment, so it joins the unit lane through @workflow/core's test script.
  • e2e/manifest.test.ts asserts the shape of a built workbench manifest, so it gets a step in E2E Local Prod immediately after local-build.test.ts — which is what runs pnpm build in the workbench app. It needs its own invocation rather than another file in that step, since vitest would otherwise run both concurrently, against a manifest that does not exist yet.
The spec-version.test.ts failure in the issue
Already gone. #3634 rewrote that assertion when it moved the current version to 7 (sealed log), so the file now pins SPEC_VERSION_CURRENT === SPEC_VERSION_SUPPORTS_SEALED_LOG rather than the stale compression version. Nothing to update here — but note that the fix landed blind, because the suite proving it still never ran.
Keeping it wired
scripts/check-no-unrun-tests.mjs walks the workspace and requires every test file to be reachable one of two ways:
  1. its package has a test script whose positional filters cover the file, so turbo test runs it — vitest treats positionals as filters, so vitest run src beside a test in e2e/ drops that file quietly; or
  2. a CI workflow, or a root package.json script, names the file (or a directory holding it) outright.
Anything else fails. It runs as a new No Unrun Tests job in lint.yml (node only, no install, alongside the other hygiene jobs). One waiver: @workflow/docs-typecheck, whose suite is far too slow for the unit lane and runs as pnpm test:docs from Docs Checks.
Path matching is boundary-aware on purpose. A plain substring search reads packages/world as present in every workflow mentioning packages/world-vercel, and a trailing / counts as continuation so that a lane naming packages/core/e2e/local-build.test.ts says nothing about the other files in that directory. Without both rules the check passes on the very state it exists to catch.
Verification
Each regression was re-introduced one at a time and the guard flagged it:
✗ @workflow/world (packages/world) has 13 test file(s) but no "test" script,
so `turbo test` skips the package entirely
✗ @workflow/core (packages/core) has test file(s) that neither its "test"
script nor any CI workflow runs: e2e/manifest.test.ts

turbo test after the fix:
@workflow/world:test: Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test: Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test: Test Files 1 passed (1) Tests 30 passed
@workflow/core:test: Test Files 104 passed Tests 2259 passed

manifest.test.ts was checked against all eight workbench apps with manifests, built locally: 36 passed.
CI confirms the suites actually execute rather than the job merely going green. Both unit lanes pass, and their logs show:
@workflow/world:test Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test Test Files 1 passed (1) Tests 30 passed
@workflow/core:test ✓ e2e/utils.test.ts (15 tests)

Windows matters here for two reasons, both now checked rather than assumed: nitro's assertions hard-code POSIX paths and hold because the source builds those specifiers with pathe, and the e2e/utils.test.ts positional filter does match there.
@workflow/world-postgres fails locally without Docker for testcontainers; it passes in CI (6 files / 193 tests).
🤖 Generated with Claude Code
@pranaygp@claude
Run the test suites CI was silently skipping
94c9088
`turbo test` runs a package's tests only if that package declares a `test`
script, so a suite can sit in the repo for months without ever running. Four
were in that state: @workflow/world (13 files, 160 tests), @workflow/cli (5 /
51), @workflow/nitro (1 / 30), and two files under packages/core/e2e that no
workflow named.
Wire each one up, and add scripts/check-test-suites-wired.mjs plus a lint job
so the next unwired suite fails CI instead of going unnoticed.
Fixes#3731
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pranaygp
pranaygp
requested a review
from a team
as a code ownerAugust 21, 2026 22:45
CopilotAI
lite review requested due to automatic review settings
August 21, 2026 22:45
@vercel
vercelBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-astro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-express-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-hono-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-python-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-vite-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-docsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-tarballsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-webReadyReadyPreview, v0Aug 21, 2026 10:52pm
CopilotAI
reviewed
Aug 21, 2026
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
@changeset-bot
changeset-botBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: 023e049
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (32 failed)
astro-node (2 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
astro-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
example-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
example-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
express-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
express-quickjs (2 failed):
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook | wrun_41M0K8E2780GKJZTMNTCF7TXRS | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
hono-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
nest-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
nest-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
nextjs-turbopack-quickjs (2 failed):
  • webhookWorkflow | wrun_41M0K8ASG90GPZGENG5QRFKYZN | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nextjs-webpack-node (1 failed):
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
nitro-quickjs (2 failed):
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nuxt-quickjs (2 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC | 🔍 observability
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ | 🔍 observability
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X | 🔍 observability
sveltekit-node (1 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
sveltekit-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
tanstack-start-node (2 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
vite-node (1 failed):
  • distributedAbortController - reconnect to existing controller | wrun_41M0K8HAHE0GVTB2BAFTQ84TC9 | 🔍 observability
vite-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K8H69RZBXWE62FS3BH7CAC
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
33 flaky tests
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (astro)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (nextjs-webpack)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (astro)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nuxt)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (vite)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (fastify)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-turbopack)
  • abortHookOrderingWorkflow [hook-first-abort-first]: hook.then → addEventListener → abort() → resumeHook (astro)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (fastify)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (hono)
  • abortReasonWorkflow: abort reason preserved across boundaries (astro)
  • abortReasonWorkflow: abort reason preserved across boundaries (fastify)
  • abortSurvivesReplayWorkflow: controller state consistent across replay (nuxt)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (astro)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (hono)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (nextjs-webpack)
  • addTenWorkflow (tanstack-start)
  • Calculator.calculate - static workflow method using static step methods from another class (nextjs-webpack)
  • distributedAbortController - manual abort triggers signal (nest)
  • distributedAbortController - manual abort triggers signal (sveltekit)
  • distributedAbortController - reconnect to existing controller (nitro)
  • distributedAbortController - reconnect to existing controller (sveltekit)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (express)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (hono)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nest)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (astro)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (hono)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (hono)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nitro)
  • webhookWorkflow (fastify)
  • workflowAndStepMetadataWorkflow (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (tanstack-start) · at 22:52:19Z · abandoned wrun_01M0K8C0WW5RQWS78EE4YVMKCW
  • cold-start-warmup · suite warmup (python) · at 22:52:39Z · abandoned wrun_41M0K89TTX0GY1APHXTS6PK89D · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H1
  • run-pickup-stall · nullByteWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFH10GX9K9PD2KZR3385
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGQ0GSPXK5H1GS7FCZ4
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFTX0GGSWRS14CZT9CRR
  • cold-start-warmup · suite warmup (python) · at 22:53:26Z · abandoned wrun_01M0K8B7VY7211258MVNH21EQQ · (+7 more)
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:53:28Z · abandoned wrun_41M0K8EF470GYTVH8A32M271NQ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0XWVW7FD6CWJSYS6GS
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX129NGA4EPDJ1E1XGGJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX13T6P1P1B6GBG27APD
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX15QXRX5S8P4TKAWE5R
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0Y1CK71YTJZ4PK2R91
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FASZ0GY8S51B3QA9804D
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FARV0GWX9AS5V5VKQ1N5
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FB1P0GHQ6SBP1AGW8S06
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FAW90GR7NW0F4HEBR5R9
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMHB4E9B13993WC73X5
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMFPFC849RM811TNR0E
  • run-pickup-stall · sleepingWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMM0TTN8V5VS10CRPEE
  • run-pickup-stall · nullByteWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMXA75AGR0ZG2ZQ5HST
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMNPJ85RDG23KCH8HJH
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:54:47Z · abandoned wrun_41M0K8FE9Q0GYH1F346WCF17T2
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ7XZQ9GXF1KSYT77GZ0
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ86FE14B8R6MNJRA9S7
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ89NC11EJ0YS9K4J66H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFJWRGDWW15C8M6ZXECB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFK7V61YSE03MDGAEDXM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:28Z · abandoned wrun_41M0K8J6690GQZ3TBQCZ186JCW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:28Z · abandoned wrun_41M0K8J2NE0GJF7X6RMVFPP1C0
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:41Z · abandoned wrun_01M0K8MCVBJKXXM7HC8W3AXS0V
  • run-pickup-stall · pathsAliasWorkflow - TypeScript path aliases resolve correctly (nextjs-webpack) · at 22:58:54Z · abandoned wrun_01M0K8REC1AXDVSBRCCMBQJJNM
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production3546327424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total1565941254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node130228
❌ astro-quickjs131128
❌ example-node131128
❌ example-quickjs131128
❌ express-node131128
❌ express-quickjs130228
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
❌ hono-quickjs131128
❌ nest-node131128
❌ nest-quickjs131128
✅ nextjs-turbopack-node15703
❌ nextjs-turbopack-quickjs15523
❌ nextjs-webpack-node15613
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
❌ nitro-quickjs130228
✅ nuxt-node132028
❌ nuxt-quickjs130228
❌ python-node08152
❌ sveltekit-node15019
❌ sveltekit-quickjs15019
❌ tanstack-start-node130228
✅ tanstack-start-quickjs132028
❌ vite-node131128
❌ vite-quickjs131128
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 023e049 · Fri, 21 Aug 2026 23:05:25 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1143 (+6.5%)1414 🔴 (+22%) 🔻1459 🔴 (+17%) 🔻1672 🔴 (+17%) 🔻30
TTFSstream1310 (+24%) 🔻1403 🔴 (+16%) 🔻1426 🔴 (+11%)1488 🔴 (+3.0%)30
TTFShook + stream1600 (+51%) 🔻1701 🔴 (+17%) 🔻1733 🔴 (+15%) 🔻1791 🔴 (+16%) 🔻30
Fan-out TTFSPromise.all(100 steps)540 (-8.8%)734 (-54%) 💚920 (-44%) 💚1060 (-36%) 💚10
Fan-out TTLSPromise.all(100 steps)1598 (-1.2%)2888 (-3.9%)7336 (+104%) 🔻7478 (+90%) 🔻10
STSO1020 steps (inline)129 (-14%)167 (-24%) 💚181 (-30%) 💚257 (-52%) 💚1019
WO1020 steps164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚1
CRTTfirst chunk (pooled)81 (-15%)163 (+10%)241 (+39%) 🔻466 (+101%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)114 (-12%)127 (-31%)188 (-26%)435 (+14%)127 (-25%)10
size sweep (100/s, 160B-12KB)109 (-12%)116 (-37%)172 (-49%)343 (-34%)107 (-19%)10
replay gateway-gpt-5.4-nano-2000t (1x)103 (-15%)107 (-85%)174 (-95%)509 (-89%)239 (-93%)3
replay eve-gpt-5.6-sol-2000t (1x)108 (-11%)108 (-33%)138 (-33%)315 (-12%)226 (-25%)2
replay eve-gpt-5.6-sol-2000t (2x)162 (+5%)212 (-35%)463 (-53%)3872 (+71%)723 (+83%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 217526ms → this run 164269ms (Δ -53257ms, -24%)
 100-150 ms ░░░░░░░░┃ main 0 this 268 +268
150-200 ms ██████████████████░░░░░┃ main 523 this 717 +194
200-250 ms ┃████████████ main 379 this 23 -356
250-300 ms ┃█ main 65 this 6 -59
300-350 ms ┃ main 23 this 2 -21
350-400 ms ┃ main 10 this 2 -8
400-450 ms ┃ main 4 this 0 -4
450-500 ms ┃ main 1 this 0 -1
500-550 ms ┃ main 5 this 1 -4
550-600 ms ┃ main 4 this 0 -4
600-650 ms ┃ main 1 this 0 -1
650-700 ms ┃ main 1 this 0 -1
750-800 ms ┃ main 1 this 0 -1
900-950 ms ┃ main 1 this 0 -1
1050-1100 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······█▆▁···· 104.6 (-29%) 95 (-32%) 188 (-26%) 435 (+14%) 3000
sweep ·····▁█▅▁···· 99.5 (-33%) 92 (-29%) 172 (-49%) 343 (-34%) 3000
gw 1x ·····▁█▃▁▁··· 97.3 (-81%) 85 (-51%) 174 (-95%) 509 (-89%) 5295
eve 1x ·····▁█▄▁···· 93.4 (-32%) 86 (-31%) 138 (-33%) 315 (-12%) 5186
eve 2x ·····▁▆█▃▁·▁· 246.2 (-4%) 125 (-27%) 463 (-53%) 3872 (+71%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▂▁▁▂▆▅█▃▄ 95–120ms
sweep ▇█▆▂▂▁▂▂▁▁ 91–119ms
gw 1x █▂▁▂▁▂▁▂▁▃ 83–154ms
eve 1x ▂▂▃▃▁▂▆█▄▁ 82–117ms
eve 2x ▁▁▁▁▁▁▂▂▁█ 105–1109ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ██▆▅▃▁▂ 96–103ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▁█▅▄▆▅▅▄▃ 25–34ms
sweep ▃█▅▆▆▆▆▄▁▂ 32–43ms
gw 1x █▃▂▁▂▂▂▂▂▃ 22–37ms
eve 1x ▁▁▁▄▂▃▆▇█▄ 18–25ms
eve 2x ▂▃▁▁▁▃▂▁▁█ 20–45ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
Comment thread.github/workflows/lint.yml
Outdated

 exit 1
 fi

 wired-test-suites:
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get a better name?
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to no-unrun-tests / No Unrun Tests, matching the no-test-overrides / No Test Overrides sibling right above it — both are short negative invariants. The script moved to scripts/check-no-unrun-tests.mjs to match, and the step now reads "Check every test file is run by turbo test or a CI lane".
Comment thread.changeset/run-e2e-utils-in-unit-lane.md
Outdated

'@workflow/core': patch
---

Run `e2e/utils.test.ts` from the package's `test` script; it covers the e2e helpers themselves and needs no deployment, so nothing had been running it.
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe don't need two changeset entries just for internal test changes
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped both. Nothing here changes shipped behaviour — the only edits to published packages are a test script and a vitest devDependency — so there is nothing to release. (No status check gates on the changeset, so this stays green.)
@pranaygp@claude
Drop the changesets and rename the guard job
023e049
The PR only wires up existing suites and adds a CI check, so there is nothing
to release. Rename the job to match its `no-test-overrides` sibling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@pranaygp
pranaygp
enabled auto-merge (squash)
August 21, 2026 22:49
@pranaygp
pranaygp
disabled auto-merge
August 21, 2026 22:59
@pranaygp
pranaygp
merged commit 447013b
into
mainAug 21, 2026

147 of 167 checks passed

@pranaygp
pranaygp
deleted the

pgp/workflow-world-tests

branch
August 21, 2026 22:59
@github-actions
github-actionsBot
commented
Aug 21, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 447013b due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
447013b73a5c240b4d17011faf45f5cfd7b4d956

pranaygp added a commit
that referenced
this pull request
Aug 21, 2026
@pranaygp
Merge remote-tracking branch 'origin/main' into pgp/bundling-regressions
9979bb5
* origin/main:
Run the test suites CI was silently skipping (#3733)
[e2e] Change race repro hook poke to soft-degrade instead of hard-stop at budget (#3561)
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

@workflow/world has 13 test files and no test script, so none of them run in CI


3 participants

@pranaygp@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Run the test suites CI was silently skipping - #3733
Merged
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Aug 21, 2026
Merged
Run the test suites CI was silently skipping#3733
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Conversation
@pranaygp
@pranaygppranaygp
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#3731.
turbo test runs a package's tests only if that package declares a test script. A suite without one is skipped in silence — the files look like coverage, and the assertions inside them rot with nobody watching. #3731 found @workflow/world in that state; checking the rest of the workspace found three more.
What was unrun
SuiteFilesTests
@workflow/world13160
@workflow/cli551
@workflow/nitro130
packages/core/e2e/utils.test.ts115
packages/core/e2e/manifest.test.ts136
All 292 pass as-is, so this adds no red. @workflow/world, @workflow/cli and @workflow/nitro get a test script (vitest run src) and a vitest devDependency, matching every other package.
The two packages/core/e2e files are a different shape of the same bug: that directory is run by dedicated workflow steps naming each file, and these two were named by none.
  • e2e/utils.test.ts tests the e2e helpers themselves and needs no deployment, so it joins the unit lane through @workflow/core's test script.
  • e2e/manifest.test.ts asserts the shape of a built workbench manifest, so it gets a step in E2E Local Prod immediately after local-build.test.ts — which is what runs pnpm build in the workbench app. It needs its own invocation rather than another file in that step, since vitest would otherwise run both concurrently, against a manifest that does not exist yet.
The spec-version.test.ts failure in the issue
Already gone. #3634 rewrote that assertion when it moved the current version to 7 (sealed log), so the file now pins SPEC_VERSION_CURRENT === SPEC_VERSION_SUPPORTS_SEALED_LOG rather than the stale compression version. Nothing to update here — but note that the fix landed blind, because the suite proving it still never ran.
Keeping it wired
scripts/check-no-unrun-tests.mjs walks the workspace and requires every test file to be reachable one of two ways:
  1. its package has a test script whose positional filters cover the file, so turbo test runs it — vitest treats positionals as filters, so vitest run src beside a test in e2e/ drops that file quietly; or
  2. a CI workflow, or a root package.json script, names the file (or a directory holding it) outright.
Anything else fails. It runs as a new No Unrun Tests job in lint.yml (node only, no install, alongside the other hygiene jobs). One waiver: @workflow/docs-typecheck, whose suite is far too slow for the unit lane and runs as pnpm test:docs from Docs Checks.
Path matching is boundary-aware on purpose. A plain substring search reads packages/world as present in every workflow mentioning packages/world-vercel, and a trailing / counts as continuation so that a lane naming packages/core/e2e/local-build.test.ts says nothing about the other files in that directory. Without both rules the check passes on the very state it exists to catch.
Verification
Each regression was re-introduced one at a time and the guard flagged it:
✗ @workflow/world (packages/world) has 13 test file(s) but no "test" script,
so `turbo test` skips the package entirely
✗ @workflow/core (packages/core) has test file(s) that neither its "test"
script nor any CI workflow runs: e2e/manifest.test.ts

turbo test after the fix:
@workflow/world:test: Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test: Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test: Test Files 1 passed (1) Tests 30 passed
@workflow/core:test: Test Files 104 passed Tests 2259 passed

manifest.test.ts was checked against all eight workbench apps with manifests, built locally: 36 passed.
CI confirms the suites actually execute rather than the job merely going green. Both unit lanes pass, and their logs show:
@workflow/world:test Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test Test Files 1 passed (1) Tests 30 passed
@workflow/core:test ✓ e2e/utils.test.ts (15 tests)

Windows matters here for two reasons, both now checked rather than assumed: nitro's assertions hard-code POSIX paths and hold because the source builds those specifiers with pathe, and the e2e/utils.test.ts positional filter does match there.
@workflow/world-postgres fails locally without Docker for testcontainers; it passes in CI (6 files / 193 tests).
🤖 Generated with Claude Code
@pranaygp@claude
Run the test suites CI was silently skipping
94c9088
`turbo test` runs a package's tests only if that package declares a `test`
script, so a suite can sit in the repo for months without ever running. Four
were in that state: @workflow/world (13 files, 160 tests), @workflow/cli (5 /
51), @workflow/nitro (1 / 30), and two files under packages/core/e2e that no
workflow named.
Wire each one up, and add scripts/check-test-suites-wired.mjs plus a lint job
so the next unwired suite fails CI instead of going unnoticed.
Fixes#3731
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pranaygp
pranaygp
requested a review
from a team
as a code ownerAugust 21, 2026 22:45
CopilotAI
lite review requested due to automatic review settings
August 21, 2026 22:45
@vercel
vercelBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-astro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-express-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-hono-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-python-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-vite-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-docsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-tarballsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-webReadyReadyPreview, v0Aug 21, 2026 10:52pm
CopilotAI
reviewed
Aug 21, 2026
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
@changeset-bot
changeset-botBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: 023e049
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (32 failed)
astro-node (2 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
astro-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
example-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
example-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
express-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
express-quickjs (2 failed):
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook | wrun_41M0K8E2780GKJZTMNTCF7TXRS | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
hono-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
nest-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
nest-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
nextjs-turbopack-quickjs (2 failed):
  • webhookWorkflow | wrun_41M0K8ASG90GPZGENG5QRFKYZN | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nextjs-webpack-node (1 failed):
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
nitro-quickjs (2 failed):
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nuxt-quickjs (2 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC | 🔍 observability
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ | 🔍 observability
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X | 🔍 observability
sveltekit-node (1 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
sveltekit-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
tanstack-start-node (2 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
vite-node (1 failed):
  • distributedAbortController - reconnect to existing controller | wrun_41M0K8HAHE0GVTB2BAFTQ84TC9 | 🔍 observability
vite-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K8H69RZBXWE62FS3BH7CAC
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
33 flaky tests
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (astro)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (nextjs-webpack)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (astro)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nuxt)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (vite)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (fastify)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-turbopack)
  • abortHookOrderingWorkflow [hook-first-abort-first]: hook.then → addEventListener → abort() → resumeHook (astro)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (fastify)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (hono)
  • abortReasonWorkflow: abort reason preserved across boundaries (astro)
  • abortReasonWorkflow: abort reason preserved across boundaries (fastify)
  • abortSurvivesReplayWorkflow: controller state consistent across replay (nuxt)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (astro)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (hono)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (nextjs-webpack)
  • addTenWorkflow (tanstack-start)
  • Calculator.calculate - static workflow method using static step methods from another class (nextjs-webpack)
  • distributedAbortController - manual abort triggers signal (nest)
  • distributedAbortController - manual abort triggers signal (sveltekit)
  • distributedAbortController - reconnect to existing controller (nitro)
  • distributedAbortController - reconnect to existing controller (sveltekit)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (express)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (hono)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nest)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (astro)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (hono)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (hono)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nitro)
  • webhookWorkflow (fastify)
  • workflowAndStepMetadataWorkflow (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (tanstack-start) · at 22:52:19Z · abandoned wrun_01M0K8C0WW5RQWS78EE4YVMKCW
  • cold-start-warmup · suite warmup (python) · at 22:52:39Z · abandoned wrun_41M0K89TTX0GY1APHXTS6PK89D · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H1
  • run-pickup-stall · nullByteWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFH10GX9K9PD2KZR3385
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGQ0GSPXK5H1GS7FCZ4
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFTX0GGSWRS14CZT9CRR
  • cold-start-warmup · suite warmup (python) · at 22:53:26Z · abandoned wrun_01M0K8B7VY7211258MVNH21EQQ · (+7 more)
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:53:28Z · abandoned wrun_41M0K8EF470GYTVH8A32M271NQ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0XWVW7FD6CWJSYS6GS
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX129NGA4EPDJ1E1XGGJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX13T6P1P1B6GBG27APD
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX15QXRX5S8P4TKAWE5R
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0Y1CK71YTJZ4PK2R91
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FASZ0GY8S51B3QA9804D
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FARV0GWX9AS5V5VKQ1N5
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FB1P0GHQ6SBP1AGW8S06
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FAW90GR7NW0F4HEBR5R9
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMHB4E9B13993WC73X5
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMFPFC849RM811TNR0E
  • run-pickup-stall · sleepingWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMM0TTN8V5VS10CRPEE
  • run-pickup-stall · nullByteWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMXA75AGR0ZG2ZQ5HST
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMNPJ85RDG23KCH8HJH
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:54:47Z · abandoned wrun_41M0K8FE9Q0GYH1F346WCF17T2
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ7XZQ9GXF1KSYT77GZ0
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ86FE14B8R6MNJRA9S7
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ89NC11EJ0YS9K4J66H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFJWRGDWW15C8M6ZXECB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFK7V61YSE03MDGAEDXM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:28Z · abandoned wrun_41M0K8J6690GQZ3TBQCZ186JCW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:28Z · abandoned wrun_41M0K8J2NE0GJF7X6RMVFPP1C0
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:41Z · abandoned wrun_01M0K8MCVBJKXXM7HC8W3AXS0V
  • run-pickup-stall · pathsAliasWorkflow - TypeScript path aliases resolve correctly (nextjs-webpack) · at 22:58:54Z · abandoned wrun_01M0K8REC1AXDVSBRCCMBQJJNM
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production3546327424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total1565941254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node130228
❌ astro-quickjs131128
❌ example-node131128
❌ example-quickjs131128
❌ express-node131128
❌ express-quickjs130228
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
❌ hono-quickjs131128
❌ nest-node131128
❌ nest-quickjs131128
✅ nextjs-turbopack-node15703
❌ nextjs-turbopack-quickjs15523
❌ nextjs-webpack-node15613
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
❌ nitro-quickjs130228
✅ nuxt-node132028
❌ nuxt-quickjs130228
❌ python-node08152
❌ sveltekit-node15019
❌ sveltekit-quickjs15019
❌ tanstack-start-node130228
✅ tanstack-start-quickjs132028
❌ vite-node131128
❌ vite-quickjs131128
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 023e049 · Fri, 21 Aug 2026 23:05:25 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1143 (+6.5%)1414 🔴 (+22%) 🔻1459 🔴 (+17%) 🔻1672 🔴 (+17%) 🔻30
TTFSstream1310 (+24%) 🔻1403 🔴 (+16%) 🔻1426 🔴 (+11%)1488 🔴 (+3.0%)30
TTFShook + stream1600 (+51%) 🔻1701 🔴 (+17%) 🔻1733 🔴 (+15%) 🔻1791 🔴 (+16%) 🔻30
Fan-out TTFSPromise.all(100 steps)540 (-8.8%)734 (-54%) 💚920 (-44%) 💚1060 (-36%) 💚10
Fan-out TTLSPromise.all(100 steps)1598 (-1.2%)2888 (-3.9%)7336 (+104%) 🔻7478 (+90%) 🔻10
STSO1020 steps (inline)129 (-14%)167 (-24%) 💚181 (-30%) 💚257 (-52%) 💚1019
WO1020 steps164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚1
CRTTfirst chunk (pooled)81 (-15%)163 (+10%)241 (+39%) 🔻466 (+101%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)114 (-12%)127 (-31%)188 (-26%)435 (+14%)127 (-25%)10
size sweep (100/s, 160B-12KB)109 (-12%)116 (-37%)172 (-49%)343 (-34%)107 (-19%)10
replay gateway-gpt-5.4-nano-2000t (1x)103 (-15%)107 (-85%)174 (-95%)509 (-89%)239 (-93%)3
replay eve-gpt-5.6-sol-2000t (1x)108 (-11%)108 (-33%)138 (-33%)315 (-12%)226 (-25%)2
replay eve-gpt-5.6-sol-2000t (2x)162 (+5%)212 (-35%)463 (-53%)3872 (+71%)723 (+83%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 217526ms → this run 164269ms (Δ -53257ms, -24%)
 100-150 ms ░░░░░░░░┃ main 0 this 268 +268
150-200 ms ██████████████████░░░░░┃ main 523 this 717 +194
200-250 ms ┃████████████ main 379 this 23 -356
250-300 ms ┃█ main 65 this 6 -59
300-350 ms ┃ main 23 this 2 -21
350-400 ms ┃ main 10 this 2 -8
400-450 ms ┃ main 4 this 0 -4
450-500 ms ┃ main 1 this 0 -1
500-550 ms ┃ main 5 this 1 -4
550-600 ms ┃ main 4 this 0 -4
600-650 ms ┃ main 1 this 0 -1
650-700 ms ┃ main 1 this 0 -1
750-800 ms ┃ main 1 this 0 -1
900-950 ms ┃ main 1 this 0 -1
1050-1100 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······█▆▁···· 104.6 (-29%) 95 (-32%) 188 (-26%) 435 (+14%) 3000
sweep ·····▁█▅▁···· 99.5 (-33%) 92 (-29%) 172 (-49%) 343 (-34%) 3000
gw 1x ·····▁█▃▁▁··· 97.3 (-81%) 85 (-51%) 174 (-95%) 509 (-89%) 5295
eve 1x ·····▁█▄▁···· 93.4 (-32%) 86 (-31%) 138 (-33%) 315 (-12%) 5186
eve 2x ·····▁▆█▃▁·▁· 246.2 (-4%) 125 (-27%) 463 (-53%) 3872 (+71%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▂▁▁▂▆▅█▃▄ 95–120ms
sweep ▇█▆▂▂▁▂▂▁▁ 91–119ms
gw 1x █▂▁▂▁▂▁▂▁▃ 83–154ms
eve 1x ▂▂▃▃▁▂▆█▄▁ 82–117ms
eve 2x ▁▁▁▁▁▁▂▂▁█ 105–1109ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ██▆▅▃▁▂ 96–103ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▁█▅▄▆▅▅▄▃ 25–34ms
sweep ▃█▅▆▆▆▆▄▁▂ 32–43ms
gw 1x █▃▂▁▂▂▂▂▂▃ 22–37ms
eve 1x ▁▁▁▄▂▃▆▇█▄ 18–25ms
eve 2x ▂▃▁▁▁▃▂▁▁█ 20–45ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
Comment thread.github/workflows/lint.yml
Outdated

 exit 1
 fi

 wired-test-suites:
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get a better name?
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to no-unrun-tests / No Unrun Tests, matching the no-test-overrides / No Test Overrides sibling right above it — both are short negative invariants. The script moved to scripts/check-no-unrun-tests.mjs to match, and the step now reads "Check every test file is run by turbo test or a CI lane".
Comment thread.changeset/run-e2e-utils-in-unit-lane.md
Outdated

'@workflow/core': patch
---

Run `e2e/utils.test.ts` from the package's `test` script; it covers the e2e helpers themselves and needs no deployment, so nothing had been running it.
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe don't need two changeset entries just for internal test changes
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped both. Nothing here changes shipped behaviour — the only edits to published packages are a test script and a vitest devDependency — so there is nothing to release. (No status check gates on the changeset, so this stays green.)
@pranaygp@claude
Drop the changesets and rename the guard job
023e049
The PR only wires up existing suites and adds a CI check, so there is nothing
to release. Rename the job to match its `no-test-overrides` sibling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@pranaygp
pranaygp
enabled auto-merge (squash)
August 21, 2026 22:49
@pranaygp
pranaygp
disabled auto-merge
August 21, 2026 22:59
@pranaygp
pranaygp
merged commit 447013b
into
mainAug 21, 2026

147 of 167 checks passed

@pranaygp
pranaygp
deleted the

pgp/workflow-world-tests

branch
August 21, 2026 22:59
@github-actions
github-actionsBot
commented
Aug 21, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 447013b due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
447013b73a5c240b4d17011faf45f5cfd7b4d956

pranaygp added a commit
that referenced
this pull request
Aug 21, 2026
@pranaygp
Merge remote-tracking branch 'origin/main' into pgp/bundling-regressions
9979bb5
* origin/main:
Run the test suites CI was silently skipping (#3733)
[e2e] Change race repro hook poke to soft-degrade instead of hard-stop at budget (#3561)
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

@workflow/world has 13 test files and no test script, so none of them run in CI


3 participants

@pranaygp@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Run the test suites CI was silently skipping - #3733
Merged
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Aug 21, 2026
Merged
Run the test suites CI was silently skipping#3733
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Conversation
@pranaygp
@pranaygppranaygp
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#3731.
turbo test runs a package's tests only if that package declares a test script. A suite without one is skipped in silence — the files look like coverage, and the assertions inside them rot with nobody watching. #3731 found @workflow/world in that state; checking the rest of the workspace found three more.
What was unrun
SuiteFilesTests
@workflow/world13160
@workflow/cli551
@workflow/nitro130
packages/core/e2e/utils.test.ts115
packages/core/e2e/manifest.test.ts136
All 292 pass as-is, so this adds no red. @workflow/world, @workflow/cli and @workflow/nitro get a test script (vitest run src) and a vitest devDependency, matching every other package.
The two packages/core/e2e files are a different shape of the same bug: that directory is run by dedicated workflow steps naming each file, and these two were named by none.
  • e2e/utils.test.ts tests the e2e helpers themselves and needs no deployment, so it joins the unit lane through @workflow/core's test script.
  • e2e/manifest.test.ts asserts the shape of a built workbench manifest, so it gets a step in E2E Local Prod immediately after local-build.test.ts — which is what runs pnpm build in the workbench app. It needs its own invocation rather than another file in that step, since vitest would otherwise run both concurrently, against a manifest that does not exist yet.
The spec-version.test.ts failure in the issue
Already gone. #3634 rewrote that assertion when it moved the current version to 7 (sealed log), so the file now pins SPEC_VERSION_CURRENT === SPEC_VERSION_SUPPORTS_SEALED_LOG rather than the stale compression version. Nothing to update here — but note that the fix landed blind, because the suite proving it still never ran.
Keeping it wired
scripts/check-no-unrun-tests.mjs walks the workspace and requires every test file to be reachable one of two ways:
  1. its package has a test script whose positional filters cover the file, so turbo test runs it — vitest treats positionals as filters, so vitest run src beside a test in e2e/ drops that file quietly; or
  2. a CI workflow, or a root package.json script, names the file (or a directory holding it) outright.
Anything else fails. It runs as a new No Unrun Tests job in lint.yml (node only, no install, alongside the other hygiene jobs). One waiver: @workflow/docs-typecheck, whose suite is far too slow for the unit lane and runs as pnpm test:docs from Docs Checks.
Path matching is boundary-aware on purpose. A plain substring search reads packages/world as present in every workflow mentioning packages/world-vercel, and a trailing / counts as continuation so that a lane naming packages/core/e2e/local-build.test.ts says nothing about the other files in that directory. Without both rules the check passes on the very state it exists to catch.
Verification
Each regression was re-introduced one at a time and the guard flagged it:
✗ @workflow/world (packages/world) has 13 test file(s) but no "test" script,
so `turbo test` skips the package entirely
✗ @workflow/core (packages/core) has test file(s) that neither its "test"
script nor any CI workflow runs: e2e/manifest.test.ts

turbo test after the fix:
@workflow/world:test: Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test: Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test: Test Files 1 passed (1) Tests 30 passed
@workflow/core:test: Test Files 104 passed Tests 2259 passed

manifest.test.ts was checked against all eight workbench apps with manifests, built locally: 36 passed.
CI confirms the suites actually execute rather than the job merely going green. Both unit lanes pass, and their logs show:
@workflow/world:test Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test Test Files 1 passed (1) Tests 30 passed
@workflow/core:test ✓ e2e/utils.test.ts (15 tests)

Windows matters here for two reasons, both now checked rather than assumed: nitro's assertions hard-code POSIX paths and hold because the source builds those specifiers with pathe, and the e2e/utils.test.ts positional filter does match there.
@workflow/world-postgres fails locally without Docker for testcontainers; it passes in CI (6 files / 193 tests).
🤖 Generated with Claude Code
@pranaygp@claude
Run the test suites CI was silently skipping
94c9088
`turbo test` runs a package's tests only if that package declares a `test`
script, so a suite can sit in the repo for months without ever running. Four
were in that state: @workflow/world (13 files, 160 tests), @workflow/cli (5 /
51), @workflow/nitro (1 / 30), and two files under packages/core/e2e that no
workflow named.
Wire each one up, and add scripts/check-test-suites-wired.mjs plus a lint job
so the next unwired suite fails CI instead of going unnoticed.
Fixes#3731
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pranaygp
pranaygp
requested a review
from a team
as a code ownerAugust 21, 2026 22:45
CopilotAI
lite review requested due to automatic review settings
August 21, 2026 22:45
@vercel
vercelBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-astro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-express-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-hono-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-python-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-vite-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-docsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-tarballsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-webReadyReadyPreview, v0Aug 21, 2026 10:52pm
CopilotAI
reviewed
Aug 21, 2026
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
@changeset-bot
changeset-botBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: 023e049
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (32 failed)
astro-node (2 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
astro-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
example-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
example-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
express-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
express-quickjs (2 failed):
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook | wrun_41M0K8E2780GKJZTMNTCF7TXRS | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
hono-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
nest-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
nest-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
nextjs-turbopack-quickjs (2 failed):
  • webhookWorkflow | wrun_41M0K8ASG90GPZGENG5QRFKYZN | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nextjs-webpack-node (1 failed):
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
nitro-quickjs (2 failed):
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nuxt-quickjs (2 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC | 🔍 observability
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ | 🔍 observability
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X | 🔍 observability
sveltekit-node (1 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
sveltekit-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
tanstack-start-node (2 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
vite-node (1 failed):
  • distributedAbortController - reconnect to existing controller | wrun_41M0K8HAHE0GVTB2BAFTQ84TC9 | 🔍 observability
vite-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K8H69RZBXWE62FS3BH7CAC
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
33 flaky tests
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (astro)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (nextjs-webpack)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (astro)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nuxt)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (vite)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (fastify)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-turbopack)
  • abortHookOrderingWorkflow [hook-first-abort-first]: hook.then → addEventListener → abort() → resumeHook (astro)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (fastify)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (hono)
  • abortReasonWorkflow: abort reason preserved across boundaries (astro)
  • abortReasonWorkflow: abort reason preserved across boundaries (fastify)
  • abortSurvivesReplayWorkflow: controller state consistent across replay (nuxt)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (astro)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (hono)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (nextjs-webpack)
  • addTenWorkflow (tanstack-start)
  • Calculator.calculate - static workflow method using static step methods from another class (nextjs-webpack)
  • distributedAbortController - manual abort triggers signal (nest)
  • distributedAbortController - manual abort triggers signal (sveltekit)
  • distributedAbortController - reconnect to existing controller (nitro)
  • distributedAbortController - reconnect to existing controller (sveltekit)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (express)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (hono)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nest)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (astro)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (hono)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (hono)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nitro)
  • webhookWorkflow (fastify)
  • workflowAndStepMetadataWorkflow (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (tanstack-start) · at 22:52:19Z · abandoned wrun_01M0K8C0WW5RQWS78EE4YVMKCW
  • cold-start-warmup · suite warmup (python) · at 22:52:39Z · abandoned wrun_41M0K89TTX0GY1APHXTS6PK89D · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H1
  • run-pickup-stall · nullByteWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFH10GX9K9PD2KZR3385
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGQ0GSPXK5H1GS7FCZ4
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFTX0GGSWRS14CZT9CRR
  • cold-start-warmup · suite warmup (python) · at 22:53:26Z · abandoned wrun_01M0K8B7VY7211258MVNH21EQQ · (+7 more)
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:53:28Z · abandoned wrun_41M0K8EF470GYTVH8A32M271NQ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0XWVW7FD6CWJSYS6GS
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX129NGA4EPDJ1E1XGGJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX13T6P1P1B6GBG27APD
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX15QXRX5S8P4TKAWE5R
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0Y1CK71YTJZ4PK2R91
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FASZ0GY8S51B3QA9804D
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FARV0GWX9AS5V5VKQ1N5
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FB1P0GHQ6SBP1AGW8S06
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FAW90GR7NW0F4HEBR5R9
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMHB4E9B13993WC73X5
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMFPFC849RM811TNR0E
  • run-pickup-stall · sleepingWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMM0TTN8V5VS10CRPEE
  • run-pickup-stall · nullByteWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMXA75AGR0ZG2ZQ5HST
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMNPJ85RDG23KCH8HJH
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:54:47Z · abandoned wrun_41M0K8FE9Q0GYH1F346WCF17T2
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ7XZQ9GXF1KSYT77GZ0
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ86FE14B8R6MNJRA9S7
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ89NC11EJ0YS9K4J66H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFJWRGDWW15C8M6ZXECB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFK7V61YSE03MDGAEDXM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:28Z · abandoned wrun_41M0K8J6690GQZ3TBQCZ186JCW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:28Z · abandoned wrun_41M0K8J2NE0GJF7X6RMVFPP1C0
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:41Z · abandoned wrun_01M0K8MCVBJKXXM7HC8W3AXS0V
  • run-pickup-stall · pathsAliasWorkflow - TypeScript path aliases resolve correctly (nextjs-webpack) · at 22:58:54Z · abandoned wrun_01M0K8REC1AXDVSBRCCMBQJJNM
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production3546327424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total1565941254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node130228
❌ astro-quickjs131128
❌ example-node131128
❌ example-quickjs131128
❌ express-node131128
❌ express-quickjs130228
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
❌ hono-quickjs131128
❌ nest-node131128
❌ nest-quickjs131128
✅ nextjs-turbopack-node15703
❌ nextjs-turbopack-quickjs15523
❌ nextjs-webpack-node15613
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
❌ nitro-quickjs130228
✅ nuxt-node132028
❌ nuxt-quickjs130228
❌ python-node08152
❌ sveltekit-node15019
❌ sveltekit-quickjs15019
❌ tanstack-start-node130228
✅ tanstack-start-quickjs132028
❌ vite-node131128
❌ vite-quickjs131128
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 023e049 · Fri, 21 Aug 2026 23:05:25 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1143 (+6.5%)1414 🔴 (+22%) 🔻1459 🔴 (+17%) 🔻1672 🔴 (+17%) 🔻30
TTFSstream1310 (+24%) 🔻1403 🔴 (+16%) 🔻1426 🔴 (+11%)1488 🔴 (+3.0%)30
TTFShook + stream1600 (+51%) 🔻1701 🔴 (+17%) 🔻1733 🔴 (+15%) 🔻1791 🔴 (+16%) 🔻30
Fan-out TTFSPromise.all(100 steps)540 (-8.8%)734 (-54%) 💚920 (-44%) 💚1060 (-36%) 💚10
Fan-out TTLSPromise.all(100 steps)1598 (-1.2%)2888 (-3.9%)7336 (+104%) 🔻7478 (+90%) 🔻10
STSO1020 steps (inline)129 (-14%)167 (-24%) 💚181 (-30%) 💚257 (-52%) 💚1019
WO1020 steps164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚1
CRTTfirst chunk (pooled)81 (-15%)163 (+10%)241 (+39%) 🔻466 (+101%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)114 (-12%)127 (-31%)188 (-26%)435 (+14%)127 (-25%)10
size sweep (100/s, 160B-12KB)109 (-12%)116 (-37%)172 (-49%)343 (-34%)107 (-19%)10
replay gateway-gpt-5.4-nano-2000t (1x)103 (-15%)107 (-85%)174 (-95%)509 (-89%)239 (-93%)3
replay eve-gpt-5.6-sol-2000t (1x)108 (-11%)108 (-33%)138 (-33%)315 (-12%)226 (-25%)2
replay eve-gpt-5.6-sol-2000t (2x)162 (+5%)212 (-35%)463 (-53%)3872 (+71%)723 (+83%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 217526ms → this run 164269ms (Δ -53257ms, -24%)
 100-150 ms ░░░░░░░░┃ main 0 this 268 +268
150-200 ms ██████████████████░░░░░┃ main 523 this 717 +194
200-250 ms ┃████████████ main 379 this 23 -356
250-300 ms ┃█ main 65 this 6 -59
300-350 ms ┃ main 23 this 2 -21
350-400 ms ┃ main 10 this 2 -8
400-450 ms ┃ main 4 this 0 -4
450-500 ms ┃ main 1 this 0 -1
500-550 ms ┃ main 5 this 1 -4
550-600 ms ┃ main 4 this 0 -4
600-650 ms ┃ main 1 this 0 -1
650-700 ms ┃ main 1 this 0 -1
750-800 ms ┃ main 1 this 0 -1
900-950 ms ┃ main 1 this 0 -1
1050-1100 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······█▆▁···· 104.6 (-29%) 95 (-32%) 188 (-26%) 435 (+14%) 3000
sweep ·····▁█▅▁···· 99.5 (-33%) 92 (-29%) 172 (-49%) 343 (-34%) 3000
gw 1x ·····▁█▃▁▁··· 97.3 (-81%) 85 (-51%) 174 (-95%) 509 (-89%) 5295
eve 1x ·····▁█▄▁···· 93.4 (-32%) 86 (-31%) 138 (-33%) 315 (-12%) 5186
eve 2x ·····▁▆█▃▁·▁· 246.2 (-4%) 125 (-27%) 463 (-53%) 3872 (+71%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▂▁▁▂▆▅█▃▄ 95–120ms
sweep ▇█▆▂▂▁▂▂▁▁ 91–119ms
gw 1x █▂▁▂▁▂▁▂▁▃ 83–154ms
eve 1x ▂▂▃▃▁▂▆█▄▁ 82–117ms
eve 2x ▁▁▁▁▁▁▂▂▁█ 105–1109ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ██▆▅▃▁▂ 96–103ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▁█▅▄▆▅▅▄▃ 25–34ms
sweep ▃█▅▆▆▆▆▄▁▂ 32–43ms
gw 1x █▃▂▁▂▂▂▂▂▃ 22–37ms
eve 1x ▁▁▁▄▂▃▆▇█▄ 18–25ms
eve 2x ▂▃▁▁▁▃▂▁▁█ 20–45ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
Comment thread.github/workflows/lint.yml
Outdated

 exit 1
 fi

 wired-test-suites:
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get a better name?
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to no-unrun-tests / No Unrun Tests, matching the no-test-overrides / No Test Overrides sibling right above it — both are short negative invariants. The script moved to scripts/check-no-unrun-tests.mjs to match, and the step now reads "Check every test file is run by turbo test or a CI lane".
Comment thread.changeset/run-e2e-utils-in-unit-lane.md
Outdated

'@workflow/core': patch
---

Run `e2e/utils.test.ts` from the package's `test` script; it covers the e2e helpers themselves and needs no deployment, so nothing had been running it.
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe don't need two changeset entries just for internal test changes
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped both. Nothing here changes shipped behaviour — the only edits to published packages are a test script and a vitest devDependency — so there is nothing to release. (No status check gates on the changeset, so this stays green.)
@pranaygp@claude
Drop the changesets and rename the guard job
023e049
The PR only wires up existing suites and adds a CI check, so there is nothing
to release. Rename the job to match its `no-test-overrides` sibling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@pranaygp
pranaygp
enabled auto-merge (squash)
August 21, 2026 22:49
@pranaygp
pranaygp
disabled auto-merge
August 21, 2026 22:59
@pranaygp
pranaygp
merged commit 447013b
into
mainAug 21, 2026

147 of 167 checks passed

@pranaygp
pranaygp
deleted the

pgp/workflow-world-tests

branch
August 21, 2026 22:59
@github-actions
github-actionsBot
commented
Aug 21, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 447013b due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
447013b73a5c240b4d17011faf45f5cfd7b4d956

pranaygp added a commit
that referenced
this pull request
Aug 21, 2026
@pranaygp
Merge remote-tracking branch 'origin/main' into pgp/bundling-regressions
9979bb5
* origin/main:
Run the test suites CI was silently skipping (#3733)
[e2e] Change race repro hook poke to soft-degrade instead of hard-stop at budget (#3561)
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

@workflow/world has 13 test files and no test script, so none of them run in CI


3 participants

@pranaygp@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Run the test suites CI was silently skipping - #3733
Merged
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Aug 21, 2026
Merged
Run the test suites CI was silently skipping#3733
pranaygp merged 2 commits into
mainfrom 
pgp/workflow-world-tests
Conversation
@pranaygp
@pranaygppranaygp
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#3731.
turbo test runs a package's tests only if that package declares a test script. A suite without one is skipped in silence — the files look like coverage, and the assertions inside them rot with nobody watching. #3731 found @workflow/world in that state; checking the rest of the workspace found three more.
What was unrun
SuiteFilesTests
@workflow/world13160
@workflow/cli551
@workflow/nitro130
packages/core/e2e/utils.test.ts115
packages/core/e2e/manifest.test.ts136
All 292 pass as-is, so this adds no red. @workflow/world, @workflow/cli and @workflow/nitro get a test script (vitest run src) and a vitest devDependency, matching every other package.
The two packages/core/e2e files are a different shape of the same bug: that directory is run by dedicated workflow steps naming each file, and these two were named by none.
  • e2e/utils.test.ts tests the e2e helpers themselves and needs no deployment, so it joins the unit lane through @workflow/core's test script.
  • e2e/manifest.test.ts asserts the shape of a built workbench manifest, so it gets a step in E2E Local Prod immediately after local-build.test.ts — which is what runs pnpm build in the workbench app. It needs its own invocation rather than another file in that step, since vitest would otherwise run both concurrently, against a manifest that does not exist yet.
The spec-version.test.ts failure in the issue
Already gone. #3634 rewrote that assertion when it moved the current version to 7 (sealed log), so the file now pins SPEC_VERSION_CURRENT === SPEC_VERSION_SUPPORTS_SEALED_LOG rather than the stale compression version. Nothing to update here — but note that the fix landed blind, because the suite proving it still never ran.
Keeping it wired
scripts/check-no-unrun-tests.mjs walks the workspace and requires every test file to be reachable one of two ways:
  1. its package has a test script whose positional filters cover the file, so turbo test runs it — vitest treats positionals as filters, so vitest run src beside a test in e2e/ drops that file quietly; or
  2. a CI workflow, or a root package.json script, names the file (or a directory holding it) outright.
Anything else fails. It runs as a new No Unrun Tests job in lint.yml (node only, no install, alongside the other hygiene jobs). One waiver: @workflow/docs-typecheck, whose suite is far too slow for the unit lane and runs as pnpm test:docs from Docs Checks.
Path matching is boundary-aware on purpose. A plain substring search reads packages/world as present in every workflow mentioning packages/world-vercel, and a trailing / counts as continuation so that a lane naming packages/core/e2e/local-build.test.ts says nothing about the other files in that directory. Without both rules the check passes on the very state it exists to catch.
Verification
Each regression was re-introduced one at a time and the guard flagged it:
✗ @workflow/world (packages/world) has 13 test file(s) but no "test" script,
so `turbo test` skips the package entirely
✗ @workflow/core (packages/core) has test file(s) that neither its "test"
script nor any CI workflow runs: e2e/manifest.test.ts

turbo test after the fix:
@workflow/world:test: Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test: Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test: Test Files 1 passed (1) Tests 30 passed
@workflow/core:test: Test Files 104 passed Tests 2259 passed

manifest.test.ts was checked against all eight workbench apps with manifests, built locally: 36 passed.
CI confirms the suites actually execute rather than the job merely going green. Both unit lanes pass, and their logs show:
@workflow/world:test Test Files 13 passed (13) Tests 160 passed
@workflow/cli:test Test Files 5 passed (5) Tests 51 passed
@workflow/nitro:test Test Files 1 passed (1) Tests 30 passed
@workflow/core:test ✓ e2e/utils.test.ts (15 tests)

Windows matters here for two reasons, both now checked rather than assumed: nitro's assertions hard-code POSIX paths and hold because the source builds those specifiers with pathe, and the e2e/utils.test.ts positional filter does match there.
@workflow/world-postgres fails locally without Docker for testcontainers; it passes in CI (6 files / 193 tests).
🤖 Generated with Claude Code
@pranaygp@claude
Run the test suites CI was silently skipping
94c9088
`turbo test` runs a package's tests only if that package declares a `test`
script, so a suite can sit in the repo for months without ever running. Four
were in that state: @workflow/world (13 files, 160 tests), @workflow/cli (5 /
51), @workflow/nitro (1 / 30), and two files under packages/core/e2e that no
workflow named.
Wire each one up, and add scripts/check-test-suites-wired.mjs plus a lint job
so the next unwired suite fails CI instead of going unnoticed.
Fixes#3731
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pranaygp
pranaygp
requested a review
from a team
as a code ownerAugust 21, 2026 22:45
CopilotAI
lite review requested due to automatic review settings
August 21, 2026 22:45
@vercel
vercelBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 21, 2026 10:52pm
example-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-astro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-express-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-hono-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-python-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workbench-vite-workflowReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-docsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-tarballsReadyReadyPreview, v0Aug 21, 2026 10:52pm
workflow-webReadyReadyPreview, v0Aug 21, 2026 10:52pm
CopilotAI
reviewed
Aug 21, 2026
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
@changeset-bot
changeset-botBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: 023e049
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (32 failed)
astro-node (2 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
astro-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
example-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
example-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
express-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
express-quickjs (2 failed):
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook | wrun_41M0K8E2780GKJZTMNTCF7TXRS | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
hono-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
nest-node (1 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
nest-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
nextjs-turbopack-quickjs (2 failed):
  • webhookWorkflow | wrun_41M0K8ASG90GPZGENG5QRFKYZN | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nextjs-webpack-node (1 failed):
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
nitro-quickjs (2 failed):
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
nuxt-quickjs (2 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC | 🔍 observability
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ | 🔍 observability
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X | 🔍 observability
sveltekit-node (1 failed):
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K8E0GQ0GYBMJX24QGXF0R1 | 🔍 observability
sveltekit-quickjs (1 failed):
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
tanstack-start-node (2 failed):
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K8HSVT0GXPY1X309DF7CA4 | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K8JT7W0GN75S8J9J1GDYFD | 🔍 observability
vite-node (1 failed):
  • distributedAbortController - reconnect to existing controller | wrun_41M0K8HAHE0GVTB2BAFTQ84TC9 | 🔍 observability
vite-quickjs (1 failed):
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K8H69RZBXWE62FS3BH7CAC
  • promiseAllWorkflow | wrun_41M0K8A8YW0GW8JS022Q7CXMSC
  • sleepingWorkflow | wrun_41M0K8AY870GKZ4CX53H9VDGF7
  • parallelSleepWorkflow | wrun_41M0K8AY9Z0GP6SD0FZDFKCMXQ
  • nullByteWorkflow | wrun_41M0K8B54W0GH4JKJPARVP6JSV
  • cancelRun - cancelling a running workflow | wrun_41M0K8FWFB0GRA6240S3325B7G
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K8FYBD0GWVQG8552Q9E48Q
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K8G6QT0GM27FV2V16MEJ4W
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K8GQV80GMHQ2A3HEQ1RH9X
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
33 flaky tests
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (astro)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (nextjs-webpack)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (astro)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nuxt)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (vite)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (fastify)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-turbopack)
  • abortHookOrderingWorkflow [hook-first-abort-first]: hook.then → addEventListener → abort() → resumeHook (astro)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (fastify)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (hono)
  • abortReasonWorkflow: abort reason preserved across boundaries (astro)
  • abortReasonWorkflow: abort reason preserved across boundaries (fastify)
  • abortSurvivesReplayWorkflow: controller state consistent across replay (nuxt)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (astro)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (hono)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (nextjs-webpack)
  • addTenWorkflow (tanstack-start)
  • Calculator.calculate - static workflow method using static step methods from another class (nextjs-webpack)
  • distributedAbortController - manual abort triggers signal (nest)
  • distributedAbortController - manual abort triggers signal (sveltekit)
  • distributedAbortController - reconnect to existing controller (nitro)
  • distributedAbortController - reconnect to existing controller (sveltekit)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (express)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (hono)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nest)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (astro)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (hono)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (hono)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nitro)
  • webhookWorkflow (fastify)
  • workflowAndStepMetadataWorkflow (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
33 infra events
  • cold-start-warmup · suite warmup (tanstack-start) · at 22:52:19Z · abandoned wrun_01M0K8C0WW5RQWS78EE4YVMKCW
  • cold-start-warmup · suite warmup (python) · at 22:52:39Z · abandoned wrun_41M0K89TTX0GY1APHXTS6PK89D · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGX0GGFYTFBG8DFK7H1
  • run-pickup-stall · nullByteWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFH10GX9K9PD2KZR3385
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFGQ0GSPXK5H1GS7FCZ4
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:52:56Z · abandoned wrun_41M0K8DFTX0GGSWRS14CZT9CRR
  • cold-start-warmup · suite warmup (python) · at 22:53:26Z · abandoned wrun_01M0K8B7VY7211258MVNH21EQQ · (+7 more)
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:53:28Z · abandoned wrun_41M0K8EF470GYTVH8A32M271NQ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0XWVW7FD6CWJSYS6GS
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX129NGA4EPDJ1E1XGGJ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX13T6P1P1B6GBG27APD
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX15QXRX5S8P4TKAWE5R
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:41Z · abandoned wrun_01M0K8EX0Y1CK71YTJZ4PK2R91
  • run-pickup-stall · sleepingWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FASZ0GY8S51B3QA9804D
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FARV0GWX9AS5V5VKQ1N5
  • run-pickup-stall · nullByteWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FB1P0GHQ6SBP1AGW8S06
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:53:56Z · abandoned wrun_41M0K8FAW90GR7NW0F4HEBR5R9
  • run-pickup-stall · promiseAllWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMHB4E9B13993WC73X5
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMFPFC849RM811TNR0E
  • run-pickup-stall · sleepingWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMM0TTN8V5VS10CRPEE
  • run-pickup-stall · nullByteWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMXA75AGR0ZG2ZQ5HST
  • run-pickup-stall · parallelSleepWorkflow (python) · at 22:54:41Z · abandoned wrun_01M0K8GQMNPJ85RDG23KCH8HJH
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:54:47Z · abandoned wrun_41M0K8FE9Q0GYH1F346WCF17T2
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ7XZQ9GXF1KSYT77GZ0
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ86FE14B8R6MNJRA9S7
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:55:41Z · abandoned wrun_01M0K8JJ89NC11EJ0YS9K4J66H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFJWRGDWW15C8M6ZXECB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:11Z · abandoned wrun_01M0K8KFK7V61YSE03MDGAEDXM
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 22:56:28Z · abandoned wrun_41M0K8J6690GQZ3TBQCZ186JCW
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:28Z · abandoned wrun_41M0K8J2NE0GJF7X6RMVFPP1C0
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 22:56:41Z · abandoned wrun_01M0K8MCVBJKXXM7HC8W3AXS0V
  • run-pickup-stall · pathsAliasWorkflow - TypeScript path aliases resolve correctly (nextjs-webpack) · at 22:58:54Z · abandoned wrun_01M0K8REC1AXDVSBRCCMBQJJNM
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production3546327424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total1565941254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node130228
❌ astro-quickjs131128
❌ example-node131128
❌ example-quickjs131128
❌ express-node131128
❌ express-quickjs130228
✅ fastify-node132028
✅ fastify-quickjs132028
✅ hono-node132028
❌ hono-quickjs131128
❌ nest-node131128
❌ nest-quickjs131128
✅ nextjs-turbopack-node15703
❌ nextjs-turbopack-quickjs15523
❌ nextjs-webpack-node15613
✅ nextjs-webpack-quickjs15703
✅ nitro-node132028
❌ nitro-quickjs130228
✅ nuxt-node132028
❌ nuxt-quickjs130228
❌ python-node08152
❌ sveltekit-node15019
❌ sveltekit-quickjs15019
❌ tanstack-start-node130228
✅ tanstack-start-quickjs132028
❌ vite-node131128
❌ vite-quickjs131128
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit 023e049 · Fri, 21 Aug 2026 23:05:25 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1143 (+6.5%)1414 🔴 (+22%) 🔻1459 🔴 (+17%) 🔻1672 🔴 (+17%) 🔻30
TTFSstream1310 (+24%) 🔻1403 🔴 (+16%) 🔻1426 🔴 (+11%)1488 🔴 (+3.0%)30
TTFShook + stream1600 (+51%) 🔻1701 🔴 (+17%) 🔻1733 🔴 (+15%) 🔻1791 🔴 (+16%) 🔻30
Fan-out TTFSPromise.all(100 steps)540 (-8.8%)734 (-54%) 💚920 (-44%) 💚1060 (-36%) 💚10
Fan-out TTLSPromise.all(100 steps)1598 (-1.2%)2888 (-3.9%)7336 (+104%) 🔻7478 (+90%) 🔻10
STSO1020 steps (inline)129 (-14%)167 (-24%) 💚181 (-30%) 💚257 (-52%) 💚1019
WO1020 steps164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚164427 (-25%) 💚1
CRTTfirst chunk (pooled)81 (-15%)163 (+10%)241 (+39%) 🔻466 (+101%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)114 (-12%)127 (-31%)188 (-26%)435 (+14%)127 (-25%)10
size sweep (100/s, 160B-12KB)109 (-12%)116 (-37%)172 (-49%)343 (-34%)107 (-19%)10
replay gateway-gpt-5.4-nano-2000t (1x)103 (-15%)107 (-85%)174 (-95%)509 (-89%)239 (-93%)3
replay eve-gpt-5.6-sol-2000t (1x)108 (-11%)108 (-33%)138 (-33%)315 (-12%)226 (-25%)2
replay eve-gpt-5.6-sol-2000t (2x)162 (+5%)212 (-35%)463 (-53%)3872 (+71%)723 (+83%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 217526ms → this run 164269ms (Δ -53257ms, -24%)
 100-150 ms ░░░░░░░░┃ main 0 this 268 +268
150-200 ms ██████████████████░░░░░┃ main 523 this 717 +194
200-250 ms ┃████████████ main 379 this 23 -356
250-300 ms ┃█ main 65 this 6 -59
300-350 ms ┃ main 23 this 2 -21
350-400 ms ┃ main 10 this 2 -8
400-450 ms ┃ main 4 this 0 -4
450-500 ms ┃ main 1 this 0 -1
500-550 ms ┃ main 5 this 1 -4
550-600 ms ┃ main 4 this 0 -4
600-650 ms ┃ main 1 this 0 -1
650-700 ms ┃ main 1 this 0 -1
750-800 ms ┃ main 1 this 0 -1
900-950 ms ┃ main 1 this 0 -1
1050-1100 ms ┃ main 1 this 0 -1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······█▆▁···· 104.6 (-29%) 95 (-32%) 188 (-26%) 435 (+14%) 3000
sweep ·····▁█▅▁···· 99.5 (-33%) 92 (-29%) 172 (-49%) 343 (-34%) 3000
gw 1x ·····▁█▃▁▁··· 97.3 (-81%) 85 (-51%) 174 (-95%) 509 (-89%) 5295
eve 1x ·····▁█▄▁···· 93.4 (-32%) 86 (-31%) 138 (-33%) 315 (-12%) 5186
eve 2x ·····▁▆█▃▁·▁· 246.2 (-4%) 125 (-27%) 463 (-53%) 3872 (+71%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▂▁▁▂▆▅█▃▄ 95–120ms
sweep ▇█▆▂▂▁▂▂▁▁ 91–119ms
gw 1x █▂▁▂▁▂▁▂▁▃ 83–154ms
eve 1x ▂▂▃▃▁▂▆█▄▁ 82–117ms
eve 2x ▁▁▁▁▁▁▂▂▁█ 105–1109ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ██▆▅▃▁▂ 96–103ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▄▁█▅▄▆▅▅▄▃ 25–34ms
sweep ▃█▅▆▆▆▆▄▁▂ 32–43ms
gw 1x █▃▂▁▂▂▂▂▂▃ 22–37ms
eve 1x ▁▁▁▄▂▃▆▇█▄ 18–25ms
eve 2x ▂▃▁▁▁▃▂▁▁█ 20–45ms

ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
Comment thread.github/workflows/lint.yml
Outdated

 exit 1
 fi

 wired-test-suites:
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get a better name?
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to no-unrun-tests / No Unrun Tests, matching the no-test-overrides / No Test Overrides sibling right above it — both are short negative invariants. The script moved to scripts/check-no-unrun-tests.mjs to match, and the step now reads "Check every test file is run by turbo test or a CI lane".
Comment thread.changeset/run-e2e-utils-in-unit-lane.md
Outdated

'@workflow/core': patch
---

Run `e2e/utils.test.ts` from the package's `test` script; it covers the e2e helpers themselves and needs no deployment, so nothing had been running it.
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe don't need two changeset entries just for internal test changes
@pranaygppranaygpAug 21, 2026

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped both. Nothing here changes shipped behaviour — the only edits to published packages are a test script and a vitest devDependency — so there is nothing to release. (No status check gates on the changeset, so this stays green.)
@pranaygp@claude
Drop the changesets and rename the guard job
023e049
The PR only wires up existing suites and adds a CI check, so there is nothing
to release. Rename the job to match its `no-test-overrides` sibling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions
github-actionsBot
commented
Aug 21, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@pranaygp
pranaygp
enabled auto-merge (squash)
August 21, 2026 22:49
@pranaygp
pranaygp
disabled auto-merge
August 21, 2026 22:59
@pranaygp
pranaygp
merged commit 447013b
into
mainAug 21, 2026

147 of 167 checks passed

@pranaygp
pranaygp
deleted the

pgp/workflow-world-tests

branch
August 21, 2026 22:59
@github-actions
github-actionsBot
commented
Aug 21, 2026

Copy link


Copy Markdown

Contributor
Backport to stable failed for 447013b due to a workflow error (backport job run).
This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details.
Once the underlying issue is fixed, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
447013b73a5c240b4d17011faf45f5cfd7b4d956

pranaygp added a commit
that referenced
this pull request
Aug 21, 2026
@pranaygp
Merge remote-tracking branch 'origin/main' into pgp/bundling-regressions
9979bb5
* origin/main:
Run the test suites CI was silently skipping (#3733)
[e2e] Change race repro hook poke to soft-degrade instead of hard-stop at budget (#3561)
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

@workflow/world has 13 test files and no test script, so none of them run in CI


3 participants

@pranaygp@VaguelySerious