Skip to content

[core] Trace fresh workflow replay phases - #3523

Merged
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom
codex/speed-fresh-replay
Aug 25, 2026
Merged

[core] Trace fresh workflow replay phases#3523
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom
codex/speed-fresh-replay

Conversation

@NathanColosimo

@NathanColosimoNathanColosimo commented Aug 13, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • add focused spans for VM context creation, bundle compilation, bundle evaluation, input hydration, and replay execution
  • start the replay execution span before invoking user code so synchronous-before-first-await work is included
  • report whether the application bundle actually hit the compiled vm.Script cache
  • avoid a separate streamed-payload-preparation span: preparation intentionally overlaps event loading, while input hydration reports any residual blocking work
  • retain PR [core] Trace fresh workflow replay phases #3523 as the top layer so its deployed benchmark and trace history remain attached

Production trace shape

On 388 fresh-replay samples from the Next/Turbopack behavior run, median phase time was approximately:

  • 3.0 ms — workflow.vm.create_context
  • 33.0 ms — workflow.bundle.compile
  • 91.4 ms — workflow.bundle.evaluate
  • 1.2 ms — workflow.input.hydrate
  • 18.4 ms — workflow.replay.execute

The earlier 6.3 ms workflow.replay.prepare_payloads measurement represented an overlapping wall-time window rather than a blocking phase, so this revision removes that span instead of preserving a misleading trace shape.

This PR is intentionally observability-only against its base. The performance implementation is reviewed in #3547 and #3548.

Stack

  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Simplify replay payload preparation #3547 — synchronous replay decode
  3. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
  4. [core] Trace fresh workflow replay phases #3523 — replay phase tracing
  5. [core] Prune schema modules from workflow bundles #3550 — prune sandbox dependency graph
  6. [workflow] Load per-source replay bundles #3551 — per-source lazy VM bundles

Validation

  • core typecheck passed
  • full core suite: 2,116 tests passed, with 3 expected failures
  • focused tracing, script-cache, and replay-cache suites: 35 tests passed

@changeset-bot

changeset-botBot commented Aug 13, 2026•
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2996b0a

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Aug 13, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 11:37pm

@github-actions

github-actionsBot commented Aug 13, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (244 failed)

astro-node (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

astro-quickjs (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

example-node (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K23M5D0GK1235ZCKHS2R1Z | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

example-quickjs (12 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

express-node (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

express-quickjs (10 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

fastify-node (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

fastify-quickjs (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

hono-node (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

hono-quickjs (10 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

nest-node (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

nest-quickjs (13 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

nextjs-turbopack-node (3 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events

nextjs-turbopack-quickjs (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nextjs-webpack-node (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nextjs-webpack-quickjs (13 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

nitro-node (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

nitro-quickjs (10 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

nuxt-node (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nuxt-quickjs (10 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

python-node (8 failed):

  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1 | 🔍 observability
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z | 🔍 observability

sveltekit-node (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

sveltekit-quickjs (4 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

tanstack-start-node (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

tanstack-start-quickjs (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes Promise.all of disjoint-key writes: every key lands

vite-node (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

vite-quickjs (12 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

🌐 Cross-language Conformance (9 failed)

python (9 failed):

  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K1ZE1VVFB967MCQ01CD795
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

46 flaky tests
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nest)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nest)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (sveltekit)
  • addTenWorkflow (astro)
  • addTenWorkflow (fastify)
  • cancelRun via CLI - cancelling a running workflow (sveltekit)
  • distributedAbortController - manual abort triggers signal (express)
  • distributedAbortController - reconnect to existing controller (example)
  • distributedAbortController - reconnect to existing controller (nest)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (fastify)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nuxt)
  • hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data (nextjs-webpack)
  • hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (nextjs-webpack)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-webpack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (express)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • no startIndex (reads all chunks) (express)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (fastify)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (nextjs-webpack)
  • Promise.all of disjoint-key writes: every key lands (example)
  • Promise.all of disjoint-key writes: every key lands (fastify)
  • Promise.all of disjoint-key writes: every key lands (sveltekit)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (express)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (example)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (express)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (vite)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nextjs-turbopack)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nitro)
  • sleepWinsRaceWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (vite)
  • start: initial attributes are seeded on run creation (nextjs-turbopack)
  • start: reserved-prefix initial attributes are seeded with allowReservedAttributes (tanstack-start)
  • step throw of a non-Error value preserves it as cause on the wrapping FatalError (nextjs-turbopack)
  • stepWinsRaceWorkflow (vite)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (hono)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (nest)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (tanstack-start)
  • webhookWorkflow (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (astro)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nest)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nuxt)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (tanstack-start)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

32 infra events
  • cold-start-warmup · suite warmup (python) · at 20:54:40Z · abandoned wrun_41M0K1HSDJ0GME5BYMX59CNX6K · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFQ0GKFWCKBYBX5Q6TF
  • run-pickup-stall · nullByteWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEG00GYMTSZ31K3DAPBQ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1M
  • run-pickup-stall · sleepingWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1K
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NF1Q0GMAYMPP86V96QQ6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:29Z · abandoned wrun_41M0K1PEDW0GZE4QZ5CFW30S42
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:55:57Z · abandoned wrun_41M0K1Q9SP0GTHYAKV39VBFX6P
  • run-pickup-stall · sleepingWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9SV0GK5F06DGM51605Q
  • run-pickup-stall · nullByteWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9W10GKRAC9NR0E87KFY
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9V00GW4CNMJKV4AXAGS
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:56:48Z · abandoned wrun_41M0K1QDWZ0GXPT291HQ94V9K2
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:58:29Z · abandoned wrun_41M0K1T5SG0GRDC5WEMPDD22EV
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 20:58:29Z · abandoned wrun_41M0K1T1P70GHK6GE0X7XYSHXJ
  • cold-start-warmup · suite warmup (python) · at 20:58:53Z · abandoned wrun_01M0K1SFM5SAP36XBNNF6BAB06 · (+7 more)
  • cold-start-warmup · suite warmup (tanstack-start) · at 20:59:04Z · abandoned wrun_01M0K1WMSYDP0A4AN0EZ2YNE7B
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RSD3H8J0V61PC9W5MK
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RRH9X8HXPZKXSEWHR3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S01X7VQ0X3SWF6FY11
  • run-pickup-stall · sleepingWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RYSY1H6DS6DJP1AJCJ
  • run-pickup-stall · nullByteWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S3C0M5KZEW6DRBHZ83
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCR8A6B5DQXHZ0XTPEG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCTEEFAHZFWC64SGN5T
  • run-pickup-stall · nullByteWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCWJRW9Q8395REQRV4X
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCNNJ4CY13PGCBXDGSB
  • run-pickup-stall · sleepingWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCQJ9PV2MH9BM4DJF6A
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0BSDW3JADX703S5W05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0FWPYY292RKBY8CC9X
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:01:08Z · abandoned wrun_01M0K20T0JG89Z3DB7M0QF7DGB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBY519CZXPW964N7N4B
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBZXRG7XF8BEYE0CAHE
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:02:08Z · abandoned wrun_01M0K22MKP6JKDYT80XXGFSRPG

E2E Test Summary

Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production33342447424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total15447253254818248
Details by Category

❌ ▲ Vercel Production

AppPassedFailedSkipped
❌ astro-node124828
❌ astro-quickjs1211128
❌ example-node123928
❌ example-quickjs1201228
❌ express-node124828
❌ express-quickjs1221028
❌ fastify-node1211128
❌ fastify-quickjs124828
❌ hono-node123928
❌ hono-quickjs1221028
❌ nest-node123928
❌ nest-quickjs1191328
❌ nextjs-turbopack-node15433
❌ nextjs-turbopack-quickjs14893
❌ nextjs-webpack-node15073
❌ nextjs-webpack-quickjs144133
❌ nitro-node1211128
❌ nitro-quickjs1221028
❌ nuxt-node125728
❌ nuxt-quickjs1221028
❌ python-node08152
❌ sveltekit-node14479
❌ sveltekit-quickjs14749
❌ tanstack-start-node123928
❌ tanstack-start-quickjs123928
❌ vite-node125728
❌ vite-quickjs1201228

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000

❌ 🌐 Cross-language Conformance

AppPassedFailedSkipped
❌ python09132

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@github-actions

github-actionsBot commented Aug 13, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

❌ The benchmark run for 1a35510 failed. See the run logs for details.

commit d74f748 · Thu, 20 Aug 2026 18:02:44 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep248 (-76%) 💚1391 🔴 (+23%) 🔻1424 🔴 (+22%) 🔻1739 🔴 (+42%) 🔻30
TTFSstream277 (+56%) 🔻1424 🔴 (+27%) 🔻1473 🔴 (+31%) 🔻1861 🔴 (+59%) 🔻30
TTFShook + stream374 (-25%) 💚1623 🔴 (+13%)1687 🔴 (+14%)1835 🔴 (+19%) 🔻30
Fan-out TTFSPromise.all(100 steps)557 (+40%) 🔻967 (+8.0%)2100 (+20%) 🔻2302 (+22%) 🔻10
Fan-out TTLSPromise.all(100 steps)4980 (+4.0%)7820 (+27%) 🔻8029 (+28%) 🔻11536 (+60%) 🔻10
STSO1020 steps (inline)135 (+13%)274 (+11%)319 (+6.3%)575 (+14%)1019
WO1020 steps265663 (+12%)265663 (+12%)265663 (+12%)265663 (+12%)1
CRTTfirst chunk (pooled)101 (+9.8%)166 (+26%) 🔻411 (+172%) 🔻1142 (+592%) 🔻28

Streams

ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)134 (+9%)298 (+106%)368 (+39%)1053 (-5%)263 (+79%)10
size sweep (100/s, 160B-12KB)133 (+2%)203 (-8%)482 (-8%)884 (+14%)211 (-23%)10
replay gateway-gpt-5.4-nano-2000t (1x)138 (+14%)162 (+10%)300 (+17%)645 (-8%)461 (-11%)3
replay eve-gpt-5.6-sol-2000t (1x)314 (+146%)184 (+21%)351 (+38%)797 (-5%)696 (+39%)2
replay eve-gpt-5.6-sol-2000t (2x)147 (+20%)244 (+16%)397 (+23%)675 (-80%)544 (-10%)3
📜 Previous results (2)

50faa3b

Wed, 19 Aug 2026 17:28:21 GMT · run logs

vercel / nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1335 (+365%) 🔻1423 🔴 (+22%) 🔻1441 🔴 (+7.6%)1534 🔴 (-1.9%)30
TTFSstream1342 (+610%) 🔻1410 🔴 (+26%) 🔻1430 🔴 (+25%) 🔻1576 🔴 (+35%) 🔻30
TTFShook + stream1514 (+16%) 🔻1693 🔴 (+23%) 🔻1705 🔴 (+21%) 🔻2082 🔴 (+21%) 🔻30
Fan-out TTFSPromise.all(100 steps)1088 (-1.5%)2594 (+13%)2614 (-5.5%)2987 (+7.8%)10
Fan-out TTLSPromise.all(100 steps)5278 (±0%)6182 (-7.1%)6444 (-6.4%)8094 (+2.7%)10
STSO1020 steps (inline)120 (-9.1%)224 (-15%)249 (-18%) 💚336 (-33%) 💚1019
WO1020 steps216112 (-13%)216112 (-13%)216112 (-13%)216112 (-13%)1
CRTTfirst chunk (pooled)80 (-4.8%)126 (-0.8%)269 (-15%)567 (-9.6%)28

Streams

ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)99 (-12%)139 (-16%)221 (-5%)430 (-8%)109 (-32%)10
size sweep (100/s, 160B-12KB)108 (±0%)128 (-17%)220 (-22%)604 (-11%)104 (-36%)10
replay gateway-gpt-5.4-nano-2000t (1x)93 (-17%)159 (-43%)268 (-63%)424 (-72%)368 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)333 (+114%)129 (-8%)239 (+24%)447 (-29%)302 (-38%)2
replay eve-gpt-5.6-sol-2000t (2x)108 (-79%)344 (-15%)553 (-80%)751 (-82%)509 (-16%)3

50faa3b

Tue, 18 Aug 2026 23:23:51 GMT · run logs

vercel / nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1377 (+29%) 🔻1439 🔴 (+20%) 🔻1460 🔴 (+15%) 🔻1585 🔴 (-0.9%)30
TTFSstream225 (-27%) 💚1436 🔴 (+29%) 🔻1458 🔴 (+24%) 🔻1865 🔴 (+51%) 🔻30
TTFShook + stream346 (-74%) 💚1711 🔴 (+14%)1742 🔴 (+12%)1816 🔴 (-7.3%)30
Fan-out TTFSPromise.all(100 steps)1521 (+25%) 🔻3074 (+29%) 🔻3100 (+25%) 🔻3101 (+15%)10
Fan-out TTLSPromise.all(100 steps)7331 (+37%) 🔻27947 (+251%) 🔻310173 (+3739%) 🔻313908 (+3301%) 🔻10
CRTTfirst chunk (pooled)108 (-7.7%)164 (-19%) 💚213 (-34%) 💚416 (-34%) 💚25

Streams

ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)138 (-15%)197 (-45%)272 (-60%)1221 (-23%)181 (-52%)10
size sweep (100/s, 160B-12KB)147 (-2%)379 (+94%)642 (+146%)933 (-16%)247 (+54%)10
replay gateway-gpt-5.4-nano-2000t (1x)113 (-33%)196 (-5%)355 (+9%)821 (+42%)242 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)147 (-33%)156 (-20%)194 (-30%)424 (-14%)302 (-30%)2
ℹ️ Metric definitions & methodology

Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@github-actions

github-actionsBot commented Aug 13, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0

Full trace: world-sim.txt

vercel[bot]
vercelBot reviewed Aug 13, 2026
View reviewed changes
Comment threadpackages/core/src/replay-payload-cache.ts Outdated
This was referenced Aug 14, 2026
Merged
Merged
Draft
VaguelySerious
VaguelySerious approved these changes Aug 21, 2026
View reviewed changes

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

Comment threadpackages/core/src/workflow.ts
'workflow.bundle.compile',
async (span) => {
const bundle = getCachedWorkflowScript(workflowCode, filename);
const lookup = getCachedWorkflowScript(workflowLookupCode, filename);

@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

Giving the lookup script its own getCachedWorkflowScript call invalidates the bounding invariant documented in script-cache.ts. That cache is keyed by code at the top level and capped at MAX_BUNDLES = 8, and its doc block asserts: "In production this bound is never reached: a deployment is its own process serving exactly one build-time bundle literal ... so there is a single code key for the process lifetime."

workflowLookupCode embeds the workflow name, so each distinct workflow becomes its own top-level code entry. An app with 8 or more workflows now exceeds the cap in production, and MAX_BUNDLES effectively bounds bundle versions + distinct workflow names rather than bundle versions.

No correctness or real cost impact, and I do not think it needs restructuring: eviction only runs when a new code key is inserted and takes from the front, and touchBundle(workflowCode) moves the bundle to MRU at the start of every session, so the bundle is never the eviction candidate — only the one-line lookup scripts churn, and recompiling those is free. But that doc block is now wrong for any app with 8+ workflows and will mislead the next person tuning this cache. Either update it or key the lookup scripts separately from the bundle LRU.

The WorkflowBundleCompileCacheHit attribute correctly reports only the bundle's hit and the comment already calls out why the lookup script's own hit rate is not the signal — that part is right.

Comment threadpackages/core/src/telemetry.ts
}

/** Keeps a parked workflow's ambient trace context aligned with each resume. */
export async function createRefreshableTraceContext() {

@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

The returned context is a live proxy over a mutable current, which breaks OTel's immutable-Context contract. Re-parenting a resumed workflow's ambient context is the right goal — spans from a resume should not attach to the trace of the request that first parked the run — but the mechanism has an asymmetry worth documenting.

setValue forwards to current.setValue(...), and OTel contexts are immutable, so that returns a concrete derived context rather than another proxy. The consequence: work whose context is the proxy itself re-parents on refresh(), but anything that derives a context one level deeper (any nested trace() inside the workflow body, which captures the concrete context at creation) does not. So the re-parenting applies to direct children only, and refresh() retroactively changes the parent that already-pending continuations resolve to.

That is defensible for what this is for, but it is subtle enough that the next reader will assume uniform behaviour. A comment on the proxy naming the setValue boundary would save that.

The rest of the span lifecycle checks out: startTraceSpan's ended flag makes the vmTrace.fail in createWorkflowSession's catch a no-op once endVmTrace() has run, so a post-bootstrap throw such as WorkflowNotRegisteredError correctly leaves the VM span OK rather than marking it failed; and void execution.then(replayTrace.end, replayTrace.fail) handles the rejection without competing with the copy returned to the caller.

@NathanColosimo

NathanColosimo commented Aug 25, 2026

Copy link
Copy Markdown
ContributorAuthor

Stack repair note: this PR was marked merged by pushing its head into its stale circular base branch; no code from it reached main. The same tracing change now continues against the correct base in #3797.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@NathanColosimo@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
[core] Trace fresh workflow replay phases by NathanColosimo · Pull Request #3523 · vercel/workflow · GitHub
Skip to content
[core] Trace fresh workflow replay phases - #3523
Merged
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Aug 25, 2026
Merged
[core] Trace fresh workflow replay phases#3523
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • add focused spans for VM context creation, bundle compilation, bundle evaluation, input hydration, and replay execution
  • start the replay execution span before invoking user code so synchronous-before-first-await work is included
  • report whether the application bundle actually hit the compiled vm.Script cache
  • avoid a separate streamed-payload-preparation span: preparation intentionally overlaps event loading, while input hydration reports any residual blocking work
  • retain PR [core] Trace fresh workflow replay phases #3523 as the top layer so its deployed benchmark and trace history remain attached
Production trace shape
On 388 fresh-replay samples from the Next/Turbopack behavior run, median phase time was approximately:
  • 3.0 ms — workflow.vm.create_context
  • 33.0 ms — workflow.bundle.compile
  • 91.4 ms — workflow.bundle.evaluate
  • 1.2 ms — workflow.input.hydrate
  • 18.4 ms — workflow.replay.execute
The earlier 6.3 ms workflow.replay.prepare_payloads measurement represented an overlapping wall-time window rather than a blocking phase, so this revision removes that span instead of preserving a misleading trace shape.
This PR is intentionally observability-only against its base. The performance implementation is reviewed in #3547 and #3548.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Simplify replay payload preparation #3547 — synchronous replay decode
  3. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
  4. [core] Trace fresh workflow replay phases #3523 — replay phase tracing
  5. [core] Prune schema modules from workflow bundles #3550 — prune sandbox dependency graph
  6. [workflow] Load per-source replay bundles #3551 — per-source lazy VM bundles
Validation
  • core typecheck passed
  • full core suite: 2,116 tests passed, with 3 expected failures
  • focused tracing, script-cache, and replay-cache suites: 35 tests passed
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 2996b0a
The changes in this PR will be included in the next version bump.
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 11:37pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (244 failed)
astro-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
astro-quickjs (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
example-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K23M5D0GK1235ZCKHS2R1Z | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
example-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
express-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
express-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
fastify-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
fastify-quickjs (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
hono-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
hono-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nextjs-turbopack-node (3 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
nextjs-turbopack-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
nuxt-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nuxt-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1 | 🔍 observability
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z | 🔍 observability
sveltekit-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
sveltekit-quickjs (4 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
tanstack-start-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
tanstack-start-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes Promise.all of disjoint-key writes: every key lands
vite-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
vite-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K1ZE1VVFB967MCQ01CD795
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
46 flaky tests
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nest)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nest)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (sveltekit)
  • addTenWorkflow (astro)
  • addTenWorkflow (fastify)
  • cancelRun via CLI - cancelling a running workflow (sveltekit)
  • distributedAbortController - manual abort triggers signal (express)
  • distributedAbortController - reconnect to existing controller (example)
  • distributedAbortController - reconnect to existing controller (nest)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (fastify)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nuxt)
  • hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data (nextjs-webpack)
  • hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (nextjs-webpack)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-webpack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (express)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • no startIndex (reads all chunks) (express)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (fastify)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (nextjs-webpack)
  • Promise.all of disjoint-key writes: every key lands (example)
  • Promise.all of disjoint-key writes: every key lands (fastify)
  • Promise.all of disjoint-key writes: every key lands (sveltekit)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (express)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (example)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (express)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (vite)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nextjs-turbopack)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nitro)
  • sleepWinsRaceWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (vite)
  • start: initial attributes are seeded on run creation (nextjs-turbopack)
  • start: reserved-prefix initial attributes are seeded with allowReservedAttributes (tanstack-start)
  • step throw of a non-Error value preserves it as cause on the wrapping FatalError (nextjs-turbopack)
  • stepWinsRaceWorkflow (vite)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (hono)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (nest)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (tanstack-start)
  • webhookWorkflow (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (astro)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nest)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nuxt)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (tanstack-start)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
32 infra events
  • cold-start-warmup · suite warmup (python) · at 20:54:40Z · abandoned wrun_41M0K1HSDJ0GME5BYMX59CNX6K · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFQ0GKFWCKBYBX5Q6TF
  • run-pickup-stall · nullByteWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEG00GYMTSZ31K3DAPBQ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1M
  • run-pickup-stall · sleepingWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1K
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NF1Q0GMAYMPP86V96QQ6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:29Z · abandoned wrun_41M0K1PEDW0GZE4QZ5CFW30S42
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:55:57Z · abandoned wrun_41M0K1Q9SP0GTHYAKV39VBFX6P
  • run-pickup-stall · sleepingWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9SV0GK5F06DGM51605Q
  • run-pickup-stall · nullByteWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9W10GKRAC9NR0E87KFY
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9V00GW4CNMJKV4AXAGS
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:56:48Z · abandoned wrun_41M0K1QDWZ0GXPT291HQ94V9K2
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:58:29Z · abandoned wrun_41M0K1T5SG0GRDC5WEMPDD22EV
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 20:58:29Z · abandoned wrun_41M0K1T1P70GHK6GE0X7XYSHXJ
  • cold-start-warmup · suite warmup (python) · at 20:58:53Z · abandoned wrun_01M0K1SFM5SAP36XBNNF6BAB06 · (+7 more)
  • cold-start-warmup · suite warmup (tanstack-start) · at 20:59:04Z · abandoned wrun_01M0K1WMSYDP0A4AN0EZ2YNE7B
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RSD3H8J0V61PC9W5MK
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RRH9X8HXPZKXSEWHR3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S01X7VQ0X3SWF6FY11
  • run-pickup-stall · sleepingWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RYSY1H6DS6DJP1AJCJ
  • run-pickup-stall · nullByteWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S3C0M5KZEW6DRBHZ83
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCR8A6B5DQXHZ0XTPEG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCTEEFAHZFWC64SGN5T
  • run-pickup-stall · nullByteWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCWJRW9Q8395REQRV4X
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCNNJ4CY13PGCBXDGSB
  • run-pickup-stall · sleepingWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCQJ9PV2MH9BM4DJF6A
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0BSDW3JADX703S5W05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0FWPYY292RKBY8CC9X
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:01:08Z · abandoned wrun_01M0K20T0JG89Z3DB7M0QF7DGB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBY519CZXPW964N7N4B
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBZXRG7XF8BEYE0CAHE
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:02:08Z · abandoned wrun_01M0K22MKP6JKDYT80XXGFSRPG
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production33342447424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total15447253254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node124828
❌ astro-quickjs1211128
❌ example-node123928
❌ example-quickjs1201228
❌ express-node124828
❌ express-quickjs1221028
❌ fastify-node1211128
❌ fastify-quickjs124828
❌ hono-node123928
❌ hono-quickjs1221028
❌ nest-node123928
❌ nest-quickjs1191328
❌ nextjs-turbopack-node15433
❌ nextjs-turbopack-quickjs14893
❌ nextjs-webpack-node15073
❌ nextjs-webpack-quickjs144133
❌ nitro-node1211128
❌ nitro-quickjs1221028
❌ nuxt-node125728
❌ nuxt-quickjs1221028
❌ python-node08152
❌ sveltekit-node14479
❌ sveltekit-quickjs14749
❌ tanstack-start-node123928
❌ tanstack-start-quickjs123928
❌ vite-node125728
❌ vite-quickjs1201228
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
❌ The benchmark run for 1a35510 failed. See the run logs for details.
commit d74f748 · Thu, 20 Aug 2026 18:02:44 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep248 (-76%) 💚1391 🔴 (+23%) 🔻1424 🔴 (+22%) 🔻1739 🔴 (+42%) 🔻30
TTFSstream277 (+56%) 🔻1424 🔴 (+27%) 🔻1473 🔴 (+31%) 🔻1861 🔴 (+59%) 🔻30
TTFShook + stream374 (-25%) 💚1623 🔴 (+13%)1687 🔴 (+14%)1835 🔴 (+19%) 🔻30
Fan-out TTFSPromise.all(100 steps)557 (+40%) 🔻967 (+8.0%)2100 (+20%) 🔻2302 (+22%) 🔻10
Fan-out TTLSPromise.all(100 steps)4980 (+4.0%)7820 (+27%) 🔻8029 (+28%) 🔻11536 (+60%) 🔻10
STSO1020 steps (inline)135 (+13%)274 (+11%)319 (+6.3%)575 (+14%)1019
WO1020 steps265663 (+12%)265663 (+12%)265663 (+12%)265663 (+12%)1
CRTTfirst chunk (pooled)101 (+9.8%)166 (+26%) 🔻411 (+172%) 🔻1142 (+592%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)134 (+9%)298 (+106%)368 (+39%)1053 (-5%)263 (+79%)10
size sweep (100/s, 160B-12KB)133 (+2%)203 (-8%)482 (-8%)884 (+14%)211 (-23%)10
replay gateway-gpt-5.4-nano-2000t (1x)138 (+14%)162 (+10%)300 (+17%)645 (-8%)461 (-11%)3
replay eve-gpt-5.6-sol-2000t (1x)314 (+146%)184 (+21%)351 (+38%)797 (-5%)696 (+39%)2
replay eve-gpt-5.6-sol-2000t (2x)147 (+20%)244 (+16%)397 (+23%)675 (-80%)544 (-10%)3
📜 Previous results (2)
50faa3b
Wed, 19 Aug 2026 17:28:21 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1335 (+365%) 🔻1423 🔴 (+22%) 🔻1441 🔴 (+7.6%)1534 🔴 (-1.9%)30
TTFSstream1342 (+610%) 🔻1410 🔴 (+26%) 🔻1430 🔴 (+25%) 🔻1576 🔴 (+35%) 🔻30
TTFShook + stream1514 (+16%) 🔻1693 🔴 (+23%) 🔻1705 🔴 (+21%) 🔻2082 🔴 (+21%) 🔻30
Fan-out TTFSPromise.all(100 steps)1088 (-1.5%)2594 (+13%)2614 (-5.5%)2987 (+7.8%)10
Fan-out TTLSPromise.all(100 steps)5278 (±0%)6182 (-7.1%)6444 (-6.4%)8094 (+2.7%)10
STSO1020 steps (inline)120 (-9.1%)224 (-15%)249 (-18%) 💚336 (-33%) 💚1019
WO1020 steps216112 (-13%)216112 (-13%)216112 (-13%)216112 (-13%)1
CRTTfirst chunk (pooled)80 (-4.8%)126 (-0.8%)269 (-15%)567 (-9.6%)28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)99 (-12%)139 (-16%)221 (-5%)430 (-8%)109 (-32%)10
size sweep (100/s, 160B-12KB)108 (±0%)128 (-17%)220 (-22%)604 (-11%)104 (-36%)10
replay gateway-gpt-5.4-nano-2000t (1x)93 (-17%)159 (-43%)268 (-63%)424 (-72%)368 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)333 (+114%)129 (-8%)239 (+24%)447 (-29%)302 (-38%)2
replay eve-gpt-5.6-sol-2000t (2x)108 (-79%)344 (-15%)553 (-80%)751 (-82%)509 (-16%)3
50faa3b
Tue, 18 Aug 2026 23:23:51 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1377 (+29%) 🔻1439 🔴 (+20%) 🔻1460 🔴 (+15%) 🔻1585 🔴 (-0.9%)30
TTFSstream225 (-27%) 💚1436 🔴 (+29%) 🔻1458 🔴 (+24%) 🔻1865 🔴 (+51%) 🔻30
TTFShook + stream346 (-74%) 💚1711 🔴 (+14%)1742 🔴 (+12%)1816 🔴 (-7.3%)30
Fan-out TTFSPromise.all(100 steps)1521 (+25%) 🔻3074 (+29%) 🔻3100 (+25%) 🔻3101 (+15%)10
Fan-out TTLSPromise.all(100 steps)7331 (+37%) 🔻27947 (+251%) 🔻310173 (+3739%) 🔻313908 (+3301%) 🔻10
CRTTfirst chunk (pooled)108 (-7.7%)164 (-19%) 💚213 (-34%) 💚416 (-34%) 💚25
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)138 (-15%)197 (-45%)272 (-60%)1221 (-23%)181 (-52%)10
size sweep (100/s, 160B-12KB)147 (-2%)379 (+94%)642 (+146%)933 (-16%)247 (+54%)10
replay gateway-gpt-5.4-nano-2000t (1x)113 (-33%)196 (-5%)355 (+9%)821 (+42%)242 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)147 (-33%)156 (-20%)194 (-30%)424 (-14%)302 (-30%)2
ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
vercel[bot]
vercelBot
reviewed
Aug 13, 2026
View reviewed changes
Comment threadpackages/core/src/replay-payload-cache.ts
Outdated


This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues
Comment threadpackages/core/src/workflow.ts
 'workflow.bundle.compile',
 async (span) => {
 const bundle = getCachedWorkflowScript(workflowCode, filename);
 const lookup = getCachedWorkflowScript(workflowLookupCode, filename);
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
Giving the lookup script its own getCachedWorkflowScript call invalidates the bounding invariant documented in script-cache.ts. That cache is keyed by code at the top level and capped at MAX_BUNDLES = 8, and its doc block asserts: "In production this bound is never reached: a deployment is its own process serving exactly one build-time bundle literal ... so there is a single code key for the process lifetime."
workflowLookupCode embeds the workflow name, so each distinct workflow becomes its own top-level code entry. An app with 8 or more workflows now exceeds the cap in production, and MAX_BUNDLES effectively bounds bundle versions + distinct workflow names rather than bundle versions.
No correctness or real cost impact, and I do not think it needs restructuring: eviction only runs when a new code key is inserted and takes from the front, and touchBundle(workflowCode) moves the bundle to MRU at the start of every session, so the bundle is never the eviction candidate — only the one-line lookup scripts churn, and recompiling those is free. But that doc block is now wrong for any app with 8+ workflows and will mislead the next person tuning this cache. Either update it or key the lookup scripts separately from the bundle LRU.
The WorkflowBundleCompileCacheHit attribute correctly reports only the bundle's hit and the comment already calls out why the lookup script's own hit rate is not the signal — that part is right.
Comment threadpackages/core/src/telemetry.ts
}

/** Keeps a parked workflow's ambient trace context aligned with each resume. */
export async function createRefreshableTraceContext() {
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
The returned context is a live proxy over a mutable current, which breaks OTel's immutable-Context contract. Re-parenting a resumed workflow's ambient context is the right goal — spans from a resume should not attach to the trace of the request that first parked the run — but the mechanism has an asymmetry worth documenting.
setValue forwards to current.setValue(...), and OTel contexts are immutable, so that returns a concrete derived context rather than another proxy. The consequence: work whose context is the proxy itself re-parents on refresh(), but anything that derives a context one level deeper (any nested trace() inside the workflow body, which captures the concrete context at creation) does not. So the re-parenting applies to direct children only, and refresh() retroactively changes the parent that already-pending continuations resolve to.
That is defensible for what this is for, but it is subtle enough that the next reader will assume uniform behaviour. A comment on the proxy naming the setValue boundary would save that.
The rest of the span lifecycle checks out: startTraceSpan's ended flag makes the vmTrace.fail in createWorkflowSession's catch a no-op once endVmTrace() has run, so a post-bootstrap throw such as WorkflowNotRegisteredError correctly leaves the VM span OK rather than marking it failed; and void execution.then(replayTrace.end, replayTrace.fail) handles the rejection without competing with the copy returned to the caller.
@NathanColosimo
NathanColosimo
commented
Aug 25, 2026

Copy link


Copy Markdown

ContributorAuthor
Stack repair note: this PR was marked merged by pushing its head into its stale circular base branch; no code from it reached main. The same tracing change now continues against the correct base in #3797.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@NathanColosimo@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
[core] Trace fresh workflow replay phases by NathanColosimo · Pull Request #3523 · vercel/workflow · GitHub
Skip to content
[core] Trace fresh workflow replay phases - #3523
Merged
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Aug 25, 2026
Merged
[core] Trace fresh workflow replay phases#3523
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • add focused spans for VM context creation, bundle compilation, bundle evaluation, input hydration, and replay execution
  • start the replay execution span before invoking user code so synchronous-before-first-await work is included
  • report whether the application bundle actually hit the compiled vm.Script cache
  • avoid a separate streamed-payload-preparation span: preparation intentionally overlaps event loading, while input hydration reports any residual blocking work
  • retain PR [core] Trace fresh workflow replay phases #3523 as the top layer so its deployed benchmark and trace history remain attached
Production trace shape
On 388 fresh-replay samples from the Next/Turbopack behavior run, median phase time was approximately:
  • 3.0 ms — workflow.vm.create_context
  • 33.0 ms — workflow.bundle.compile
  • 91.4 ms — workflow.bundle.evaluate
  • 1.2 ms — workflow.input.hydrate
  • 18.4 ms — workflow.replay.execute
The earlier 6.3 ms workflow.replay.prepare_payloads measurement represented an overlapping wall-time window rather than a blocking phase, so this revision removes that span instead of preserving a misleading trace shape.
This PR is intentionally observability-only against its base. The performance implementation is reviewed in #3547 and #3548.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Simplify replay payload preparation #3547 — synchronous replay decode
  3. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
  4. [core] Trace fresh workflow replay phases #3523 — replay phase tracing
  5. [core] Prune schema modules from workflow bundles #3550 — prune sandbox dependency graph
  6. [workflow] Load per-source replay bundles #3551 — per-source lazy VM bundles
Validation
  • core typecheck passed
  • full core suite: 2,116 tests passed, with 3 expected failures
  • focused tracing, script-cache, and replay-cache suites: 35 tests passed
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 2996b0a
The changes in this PR will be included in the next version bump.
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 11:37pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (244 failed)
astro-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
astro-quickjs (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
example-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K23M5D0GK1235ZCKHS2R1Z | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
example-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
express-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
express-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
fastify-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
fastify-quickjs (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
hono-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
hono-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nextjs-turbopack-node (3 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
nextjs-turbopack-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
nuxt-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nuxt-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1 | 🔍 observability
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z | 🔍 observability
sveltekit-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
sveltekit-quickjs (4 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
tanstack-start-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
tanstack-start-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes Promise.all of disjoint-key writes: every key lands
vite-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
vite-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K1ZE1VVFB967MCQ01CD795
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
46 flaky tests
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nest)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nest)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (sveltekit)
  • addTenWorkflow (astro)
  • addTenWorkflow (fastify)
  • cancelRun via CLI - cancelling a running workflow (sveltekit)
  • distributedAbortController - manual abort triggers signal (express)
  • distributedAbortController - reconnect to existing controller (example)
  • distributedAbortController - reconnect to existing controller (nest)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (fastify)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nuxt)
  • hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data (nextjs-webpack)
  • hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (nextjs-webpack)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-webpack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (express)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • no startIndex (reads all chunks) (express)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (fastify)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (nextjs-webpack)
  • Promise.all of disjoint-key writes: every key lands (example)
  • Promise.all of disjoint-key writes: every key lands (fastify)
  • Promise.all of disjoint-key writes: every key lands (sveltekit)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (express)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (example)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (express)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (vite)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nextjs-turbopack)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nitro)
  • sleepWinsRaceWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (vite)
  • start: initial attributes are seeded on run creation (nextjs-turbopack)
  • start: reserved-prefix initial attributes are seeded with allowReservedAttributes (tanstack-start)
  • step throw of a non-Error value preserves it as cause on the wrapping FatalError (nextjs-turbopack)
  • stepWinsRaceWorkflow (vite)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (hono)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (nest)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (tanstack-start)
  • webhookWorkflow (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (astro)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nest)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nuxt)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (tanstack-start)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
32 infra events
  • cold-start-warmup · suite warmup (python) · at 20:54:40Z · abandoned wrun_41M0K1HSDJ0GME5BYMX59CNX6K · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFQ0GKFWCKBYBX5Q6TF
  • run-pickup-stall · nullByteWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEG00GYMTSZ31K3DAPBQ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1M
  • run-pickup-stall · sleepingWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1K
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NF1Q0GMAYMPP86V96QQ6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:29Z · abandoned wrun_41M0K1PEDW0GZE4QZ5CFW30S42
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:55:57Z · abandoned wrun_41M0K1Q9SP0GTHYAKV39VBFX6P
  • run-pickup-stall · sleepingWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9SV0GK5F06DGM51605Q
  • run-pickup-stall · nullByteWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9W10GKRAC9NR0E87KFY
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9V00GW4CNMJKV4AXAGS
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:56:48Z · abandoned wrun_41M0K1QDWZ0GXPT291HQ94V9K2
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:58:29Z · abandoned wrun_41M0K1T5SG0GRDC5WEMPDD22EV
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 20:58:29Z · abandoned wrun_41M0K1T1P70GHK6GE0X7XYSHXJ
  • cold-start-warmup · suite warmup (python) · at 20:58:53Z · abandoned wrun_01M0K1SFM5SAP36XBNNF6BAB06 · (+7 more)
  • cold-start-warmup · suite warmup (tanstack-start) · at 20:59:04Z · abandoned wrun_01M0K1WMSYDP0A4AN0EZ2YNE7B
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RSD3H8J0V61PC9W5MK
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RRH9X8HXPZKXSEWHR3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S01X7VQ0X3SWF6FY11
  • run-pickup-stall · sleepingWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RYSY1H6DS6DJP1AJCJ
  • run-pickup-stall · nullByteWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S3C0M5KZEW6DRBHZ83
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCR8A6B5DQXHZ0XTPEG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCTEEFAHZFWC64SGN5T
  • run-pickup-stall · nullByteWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCWJRW9Q8395REQRV4X
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCNNJ4CY13PGCBXDGSB
  • run-pickup-stall · sleepingWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCQJ9PV2MH9BM4DJF6A
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0BSDW3JADX703S5W05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0FWPYY292RKBY8CC9X
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:01:08Z · abandoned wrun_01M0K20T0JG89Z3DB7M0QF7DGB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBY519CZXPW964N7N4B
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBZXRG7XF8BEYE0CAHE
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:02:08Z · abandoned wrun_01M0K22MKP6JKDYT80XXGFSRPG
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production33342447424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total15447253254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node124828
❌ astro-quickjs1211128
❌ example-node123928
❌ example-quickjs1201228
❌ express-node124828
❌ express-quickjs1221028
❌ fastify-node1211128
❌ fastify-quickjs124828
❌ hono-node123928
❌ hono-quickjs1221028
❌ nest-node123928
❌ nest-quickjs1191328
❌ nextjs-turbopack-node15433
❌ nextjs-turbopack-quickjs14893
❌ nextjs-webpack-node15073
❌ nextjs-webpack-quickjs144133
❌ nitro-node1211128
❌ nitro-quickjs1221028
❌ nuxt-node125728
❌ nuxt-quickjs1221028
❌ python-node08152
❌ sveltekit-node14479
❌ sveltekit-quickjs14749
❌ tanstack-start-node123928
❌ tanstack-start-quickjs123928
❌ vite-node125728
❌ vite-quickjs1201228
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
❌ The benchmark run for 1a35510 failed. See the run logs for details.
commit d74f748 · Thu, 20 Aug 2026 18:02:44 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep248 (-76%) 💚1391 🔴 (+23%) 🔻1424 🔴 (+22%) 🔻1739 🔴 (+42%) 🔻30
TTFSstream277 (+56%) 🔻1424 🔴 (+27%) 🔻1473 🔴 (+31%) 🔻1861 🔴 (+59%) 🔻30
TTFShook + stream374 (-25%) 💚1623 🔴 (+13%)1687 🔴 (+14%)1835 🔴 (+19%) 🔻30
Fan-out TTFSPromise.all(100 steps)557 (+40%) 🔻967 (+8.0%)2100 (+20%) 🔻2302 (+22%) 🔻10
Fan-out TTLSPromise.all(100 steps)4980 (+4.0%)7820 (+27%) 🔻8029 (+28%) 🔻11536 (+60%) 🔻10
STSO1020 steps (inline)135 (+13%)274 (+11%)319 (+6.3%)575 (+14%)1019
WO1020 steps265663 (+12%)265663 (+12%)265663 (+12%)265663 (+12%)1
CRTTfirst chunk (pooled)101 (+9.8%)166 (+26%) 🔻411 (+172%) 🔻1142 (+592%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)134 (+9%)298 (+106%)368 (+39%)1053 (-5%)263 (+79%)10
size sweep (100/s, 160B-12KB)133 (+2%)203 (-8%)482 (-8%)884 (+14%)211 (-23%)10
replay gateway-gpt-5.4-nano-2000t (1x)138 (+14%)162 (+10%)300 (+17%)645 (-8%)461 (-11%)3
replay eve-gpt-5.6-sol-2000t (1x)314 (+146%)184 (+21%)351 (+38%)797 (-5%)696 (+39%)2
replay eve-gpt-5.6-sol-2000t (2x)147 (+20%)244 (+16%)397 (+23%)675 (-80%)544 (-10%)3
📜 Previous results (2)
50faa3b
Wed, 19 Aug 2026 17:28:21 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1335 (+365%) 🔻1423 🔴 (+22%) 🔻1441 🔴 (+7.6%)1534 🔴 (-1.9%)30
TTFSstream1342 (+610%) 🔻1410 🔴 (+26%) 🔻1430 🔴 (+25%) 🔻1576 🔴 (+35%) 🔻30
TTFShook + stream1514 (+16%) 🔻1693 🔴 (+23%) 🔻1705 🔴 (+21%) 🔻2082 🔴 (+21%) 🔻30
Fan-out TTFSPromise.all(100 steps)1088 (-1.5%)2594 (+13%)2614 (-5.5%)2987 (+7.8%)10
Fan-out TTLSPromise.all(100 steps)5278 (±0%)6182 (-7.1%)6444 (-6.4%)8094 (+2.7%)10
STSO1020 steps (inline)120 (-9.1%)224 (-15%)249 (-18%) 💚336 (-33%) 💚1019
WO1020 steps216112 (-13%)216112 (-13%)216112 (-13%)216112 (-13%)1
CRTTfirst chunk (pooled)80 (-4.8%)126 (-0.8%)269 (-15%)567 (-9.6%)28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)99 (-12%)139 (-16%)221 (-5%)430 (-8%)109 (-32%)10
size sweep (100/s, 160B-12KB)108 (±0%)128 (-17%)220 (-22%)604 (-11%)104 (-36%)10
replay gateway-gpt-5.4-nano-2000t (1x)93 (-17%)159 (-43%)268 (-63%)424 (-72%)368 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)333 (+114%)129 (-8%)239 (+24%)447 (-29%)302 (-38%)2
replay eve-gpt-5.6-sol-2000t (2x)108 (-79%)344 (-15%)553 (-80%)751 (-82%)509 (-16%)3
50faa3b
Tue, 18 Aug 2026 23:23:51 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1377 (+29%) 🔻1439 🔴 (+20%) 🔻1460 🔴 (+15%) 🔻1585 🔴 (-0.9%)30
TTFSstream225 (-27%) 💚1436 🔴 (+29%) 🔻1458 🔴 (+24%) 🔻1865 🔴 (+51%) 🔻30
TTFShook + stream346 (-74%) 💚1711 🔴 (+14%)1742 🔴 (+12%)1816 🔴 (-7.3%)30
Fan-out TTFSPromise.all(100 steps)1521 (+25%) 🔻3074 (+29%) 🔻3100 (+25%) 🔻3101 (+15%)10
Fan-out TTLSPromise.all(100 steps)7331 (+37%) 🔻27947 (+251%) 🔻310173 (+3739%) 🔻313908 (+3301%) 🔻10
CRTTfirst chunk (pooled)108 (-7.7%)164 (-19%) 💚213 (-34%) 💚416 (-34%) 💚25
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)138 (-15%)197 (-45%)272 (-60%)1221 (-23%)181 (-52%)10
size sweep (100/s, 160B-12KB)147 (-2%)379 (+94%)642 (+146%)933 (-16%)247 (+54%)10
replay gateway-gpt-5.4-nano-2000t (1x)113 (-33%)196 (-5%)355 (+9%)821 (+42%)242 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)147 (-33%)156 (-20%)194 (-30%)424 (-14%)302 (-30%)2
ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
vercel[bot]
vercelBot
reviewed
Aug 13, 2026
View reviewed changes
Comment threadpackages/core/src/replay-payload-cache.ts
Outdated


This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues
Comment threadpackages/core/src/workflow.ts
 'workflow.bundle.compile',
 async (span) => {
 const bundle = getCachedWorkflowScript(workflowCode, filename);
 const lookup = getCachedWorkflowScript(workflowLookupCode, filename);
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
Giving the lookup script its own getCachedWorkflowScript call invalidates the bounding invariant documented in script-cache.ts. That cache is keyed by code at the top level and capped at MAX_BUNDLES = 8, and its doc block asserts: "In production this bound is never reached: a deployment is its own process serving exactly one build-time bundle literal ... so there is a single code key for the process lifetime."
workflowLookupCode embeds the workflow name, so each distinct workflow becomes its own top-level code entry. An app with 8 or more workflows now exceeds the cap in production, and MAX_BUNDLES effectively bounds bundle versions + distinct workflow names rather than bundle versions.
No correctness or real cost impact, and I do not think it needs restructuring: eviction only runs when a new code key is inserted and takes from the front, and touchBundle(workflowCode) moves the bundle to MRU at the start of every session, so the bundle is never the eviction candidate — only the one-line lookup scripts churn, and recompiling those is free. But that doc block is now wrong for any app with 8+ workflows and will mislead the next person tuning this cache. Either update it or key the lookup scripts separately from the bundle LRU.
The WorkflowBundleCompileCacheHit attribute correctly reports only the bundle's hit and the comment already calls out why the lookup script's own hit rate is not the signal — that part is right.
Comment threadpackages/core/src/telemetry.ts
}

/** Keeps a parked workflow's ambient trace context aligned with each resume. */
export async function createRefreshableTraceContext() {
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
The returned context is a live proxy over a mutable current, which breaks OTel's immutable-Context contract. Re-parenting a resumed workflow's ambient context is the right goal — spans from a resume should not attach to the trace of the request that first parked the run — but the mechanism has an asymmetry worth documenting.
setValue forwards to current.setValue(...), and OTel contexts are immutable, so that returns a concrete derived context rather than another proxy. The consequence: work whose context is the proxy itself re-parents on refresh(), but anything that derives a context one level deeper (any nested trace() inside the workflow body, which captures the concrete context at creation) does not. So the re-parenting applies to direct children only, and refresh() retroactively changes the parent that already-pending continuations resolve to.
That is defensible for what this is for, but it is subtle enough that the next reader will assume uniform behaviour. A comment on the proxy naming the setValue boundary would save that.
The rest of the span lifecycle checks out: startTraceSpan's ended flag makes the vmTrace.fail in createWorkflowSession's catch a no-op once endVmTrace() has run, so a post-bootstrap throw such as WorkflowNotRegisteredError correctly leaves the VM span OK rather than marking it failed; and void execution.then(replayTrace.end, replayTrace.fail) handles the rejection without competing with the copy returned to the caller.
@NathanColosimo
NathanColosimo
commented
Aug 25, 2026

Copy link


Copy Markdown

ContributorAuthor
Stack repair note: this PR was marked merged by pushing its head into its stale circular base branch; no code from it reached main. The same tracing change now continues against the correct base in #3797.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@NathanColosimo@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
[core] Trace fresh workflow replay phases by NathanColosimo · Pull Request #3523 · vercel/workflow · GitHub
Skip to content
[core] Trace fresh workflow replay phases - #3523
Merged
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Aug 25, 2026
Merged
[core] Trace fresh workflow replay phases#3523
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • add focused spans for VM context creation, bundle compilation, bundle evaluation, input hydration, and replay execution
  • start the replay execution span before invoking user code so synchronous-before-first-await work is included
  • report whether the application bundle actually hit the compiled vm.Script cache
  • avoid a separate streamed-payload-preparation span: preparation intentionally overlaps event loading, while input hydration reports any residual blocking work
  • retain PR [core] Trace fresh workflow replay phases #3523 as the top layer so its deployed benchmark and trace history remain attached
Production trace shape
On 388 fresh-replay samples from the Next/Turbopack behavior run, median phase time was approximately:
  • 3.0 ms — workflow.vm.create_context
  • 33.0 ms — workflow.bundle.compile
  • 91.4 ms — workflow.bundle.evaluate
  • 1.2 ms — workflow.input.hydrate
  • 18.4 ms — workflow.replay.execute
The earlier 6.3 ms workflow.replay.prepare_payloads measurement represented an overlapping wall-time window rather than a blocking phase, so this revision removes that span instead of preserving a misleading trace shape.
This PR is intentionally observability-only against its base. The performance implementation is reviewed in #3547 and #3548.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Simplify replay payload preparation #3547 — synchronous replay decode
  3. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
  4. [core] Trace fresh workflow replay phases #3523 — replay phase tracing
  5. [core] Prune schema modules from workflow bundles #3550 — prune sandbox dependency graph
  6. [workflow] Load per-source replay bundles #3551 — per-source lazy VM bundles
Validation
  • core typecheck passed
  • full core suite: 2,116 tests passed, with 3 expected failures
  • focused tracing, script-cache, and replay-cache suites: 35 tests passed
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 2996b0a
The changes in this PR will be included in the next version bump.
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 11:37pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (244 failed)
astro-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
astro-quickjs (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
example-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K23M5D0GK1235ZCKHS2R1Z | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
example-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
express-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
express-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
fastify-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
fastify-quickjs (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
hono-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
hono-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nextjs-turbopack-node (3 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
nextjs-turbopack-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
nuxt-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nuxt-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1 | 🔍 observability
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z | 🔍 observability
sveltekit-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
sveltekit-quickjs (4 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
tanstack-start-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
tanstack-start-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes Promise.all of disjoint-key writes: every key lands
vite-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
vite-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K1ZE1VVFB967MCQ01CD795
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
46 flaky tests
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nest)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nest)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (sveltekit)
  • addTenWorkflow (astro)
  • addTenWorkflow (fastify)
  • cancelRun via CLI - cancelling a running workflow (sveltekit)
  • distributedAbortController - manual abort triggers signal (express)
  • distributedAbortController - reconnect to existing controller (example)
  • distributedAbortController - reconnect to existing controller (nest)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (fastify)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nuxt)
  • hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data (nextjs-webpack)
  • hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (nextjs-webpack)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-webpack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (express)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • no startIndex (reads all chunks) (express)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (fastify)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (nextjs-webpack)
  • Promise.all of disjoint-key writes: every key lands (example)
  • Promise.all of disjoint-key writes: every key lands (fastify)
  • Promise.all of disjoint-key writes: every key lands (sveltekit)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (express)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (example)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (express)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (vite)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nextjs-turbopack)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nitro)
  • sleepWinsRaceWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (vite)
  • start: initial attributes are seeded on run creation (nextjs-turbopack)
  • start: reserved-prefix initial attributes are seeded with allowReservedAttributes (tanstack-start)
  • step throw of a non-Error value preserves it as cause on the wrapping FatalError (nextjs-turbopack)
  • stepWinsRaceWorkflow (vite)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (hono)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (nest)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (tanstack-start)
  • webhookWorkflow (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (astro)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nest)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nuxt)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (tanstack-start)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
32 infra events
  • cold-start-warmup · suite warmup (python) · at 20:54:40Z · abandoned wrun_41M0K1HSDJ0GME5BYMX59CNX6K · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFQ0GKFWCKBYBX5Q6TF
  • run-pickup-stall · nullByteWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEG00GYMTSZ31K3DAPBQ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1M
  • run-pickup-stall · sleepingWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1K
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NF1Q0GMAYMPP86V96QQ6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:29Z · abandoned wrun_41M0K1PEDW0GZE4QZ5CFW30S42
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:55:57Z · abandoned wrun_41M0K1Q9SP0GTHYAKV39VBFX6P
  • run-pickup-stall · sleepingWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9SV0GK5F06DGM51605Q
  • run-pickup-stall · nullByteWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9W10GKRAC9NR0E87KFY
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9V00GW4CNMJKV4AXAGS
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:56:48Z · abandoned wrun_41M0K1QDWZ0GXPT291HQ94V9K2
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:58:29Z · abandoned wrun_41M0K1T5SG0GRDC5WEMPDD22EV
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 20:58:29Z · abandoned wrun_41M0K1T1P70GHK6GE0X7XYSHXJ
  • cold-start-warmup · suite warmup (python) · at 20:58:53Z · abandoned wrun_01M0K1SFM5SAP36XBNNF6BAB06 · (+7 more)
  • cold-start-warmup · suite warmup (tanstack-start) · at 20:59:04Z · abandoned wrun_01M0K1WMSYDP0A4AN0EZ2YNE7B
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RSD3H8J0V61PC9W5MK
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RRH9X8HXPZKXSEWHR3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S01X7VQ0X3SWF6FY11
  • run-pickup-stall · sleepingWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RYSY1H6DS6DJP1AJCJ
  • run-pickup-stall · nullByteWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S3C0M5KZEW6DRBHZ83
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCR8A6B5DQXHZ0XTPEG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCTEEFAHZFWC64SGN5T
  • run-pickup-stall · nullByteWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCWJRW9Q8395REQRV4X
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCNNJ4CY13PGCBXDGSB
  • run-pickup-stall · sleepingWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCQJ9PV2MH9BM4DJF6A
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0BSDW3JADX703S5W05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0FWPYY292RKBY8CC9X
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:01:08Z · abandoned wrun_01M0K20T0JG89Z3DB7M0QF7DGB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBY519CZXPW964N7N4B
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBZXRG7XF8BEYE0CAHE
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:02:08Z · abandoned wrun_01M0K22MKP6JKDYT80XXGFSRPG
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production33342447424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total15447253254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node124828
❌ astro-quickjs1211128
❌ example-node123928
❌ example-quickjs1201228
❌ express-node124828
❌ express-quickjs1221028
❌ fastify-node1211128
❌ fastify-quickjs124828
❌ hono-node123928
❌ hono-quickjs1221028
❌ nest-node123928
❌ nest-quickjs1191328
❌ nextjs-turbopack-node15433
❌ nextjs-turbopack-quickjs14893
❌ nextjs-webpack-node15073
❌ nextjs-webpack-quickjs144133
❌ nitro-node1211128
❌ nitro-quickjs1221028
❌ nuxt-node125728
❌ nuxt-quickjs1221028
❌ python-node08152
❌ sveltekit-node14479
❌ sveltekit-quickjs14749
❌ tanstack-start-node123928
❌ tanstack-start-quickjs123928
❌ vite-node125728
❌ vite-quickjs1201228
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
❌ The benchmark run for 1a35510 failed. See the run logs for details.
commit d74f748 · Thu, 20 Aug 2026 18:02:44 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep248 (-76%) 💚1391 🔴 (+23%) 🔻1424 🔴 (+22%) 🔻1739 🔴 (+42%) 🔻30
TTFSstream277 (+56%) 🔻1424 🔴 (+27%) 🔻1473 🔴 (+31%) 🔻1861 🔴 (+59%) 🔻30
TTFShook + stream374 (-25%) 💚1623 🔴 (+13%)1687 🔴 (+14%)1835 🔴 (+19%) 🔻30
Fan-out TTFSPromise.all(100 steps)557 (+40%) 🔻967 (+8.0%)2100 (+20%) 🔻2302 (+22%) 🔻10
Fan-out TTLSPromise.all(100 steps)4980 (+4.0%)7820 (+27%) 🔻8029 (+28%) 🔻11536 (+60%) 🔻10
STSO1020 steps (inline)135 (+13%)274 (+11%)319 (+6.3%)575 (+14%)1019
WO1020 steps265663 (+12%)265663 (+12%)265663 (+12%)265663 (+12%)1
CRTTfirst chunk (pooled)101 (+9.8%)166 (+26%) 🔻411 (+172%) 🔻1142 (+592%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)134 (+9%)298 (+106%)368 (+39%)1053 (-5%)263 (+79%)10
size sweep (100/s, 160B-12KB)133 (+2%)203 (-8%)482 (-8%)884 (+14%)211 (-23%)10
replay gateway-gpt-5.4-nano-2000t (1x)138 (+14%)162 (+10%)300 (+17%)645 (-8%)461 (-11%)3
replay eve-gpt-5.6-sol-2000t (1x)314 (+146%)184 (+21%)351 (+38%)797 (-5%)696 (+39%)2
replay eve-gpt-5.6-sol-2000t (2x)147 (+20%)244 (+16%)397 (+23%)675 (-80%)544 (-10%)3
📜 Previous results (2)
50faa3b
Wed, 19 Aug 2026 17:28:21 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1335 (+365%) 🔻1423 🔴 (+22%) 🔻1441 🔴 (+7.6%)1534 🔴 (-1.9%)30
TTFSstream1342 (+610%) 🔻1410 🔴 (+26%) 🔻1430 🔴 (+25%) 🔻1576 🔴 (+35%) 🔻30
TTFShook + stream1514 (+16%) 🔻1693 🔴 (+23%) 🔻1705 🔴 (+21%) 🔻2082 🔴 (+21%) 🔻30
Fan-out TTFSPromise.all(100 steps)1088 (-1.5%)2594 (+13%)2614 (-5.5%)2987 (+7.8%)10
Fan-out TTLSPromise.all(100 steps)5278 (±0%)6182 (-7.1%)6444 (-6.4%)8094 (+2.7%)10
STSO1020 steps (inline)120 (-9.1%)224 (-15%)249 (-18%) 💚336 (-33%) 💚1019
WO1020 steps216112 (-13%)216112 (-13%)216112 (-13%)216112 (-13%)1
CRTTfirst chunk (pooled)80 (-4.8%)126 (-0.8%)269 (-15%)567 (-9.6%)28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)99 (-12%)139 (-16%)221 (-5%)430 (-8%)109 (-32%)10
size sweep (100/s, 160B-12KB)108 (±0%)128 (-17%)220 (-22%)604 (-11%)104 (-36%)10
replay gateway-gpt-5.4-nano-2000t (1x)93 (-17%)159 (-43%)268 (-63%)424 (-72%)368 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)333 (+114%)129 (-8%)239 (+24%)447 (-29%)302 (-38%)2
replay eve-gpt-5.6-sol-2000t (2x)108 (-79%)344 (-15%)553 (-80%)751 (-82%)509 (-16%)3
50faa3b
Tue, 18 Aug 2026 23:23:51 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1377 (+29%) 🔻1439 🔴 (+20%) 🔻1460 🔴 (+15%) 🔻1585 🔴 (-0.9%)30
TTFSstream225 (-27%) 💚1436 🔴 (+29%) 🔻1458 🔴 (+24%) 🔻1865 🔴 (+51%) 🔻30
TTFShook + stream346 (-74%) 💚1711 🔴 (+14%)1742 🔴 (+12%)1816 🔴 (-7.3%)30
Fan-out TTFSPromise.all(100 steps)1521 (+25%) 🔻3074 (+29%) 🔻3100 (+25%) 🔻3101 (+15%)10
Fan-out TTLSPromise.all(100 steps)7331 (+37%) 🔻27947 (+251%) 🔻310173 (+3739%) 🔻313908 (+3301%) 🔻10
CRTTfirst chunk (pooled)108 (-7.7%)164 (-19%) 💚213 (-34%) 💚416 (-34%) 💚25
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)138 (-15%)197 (-45%)272 (-60%)1221 (-23%)181 (-52%)10
size sweep (100/s, 160B-12KB)147 (-2%)379 (+94%)642 (+146%)933 (-16%)247 (+54%)10
replay gateway-gpt-5.4-nano-2000t (1x)113 (-33%)196 (-5%)355 (+9%)821 (+42%)242 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)147 (-33%)156 (-20%)194 (-30%)424 (-14%)302 (-30%)2
ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
vercel[bot]
vercelBot
reviewed
Aug 13, 2026
View reviewed changes
Comment threadpackages/core/src/replay-payload-cache.ts
Outdated


This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues
Comment threadpackages/core/src/workflow.ts
 'workflow.bundle.compile',
 async (span) => {
 const bundle = getCachedWorkflowScript(workflowCode, filename);
 const lookup = getCachedWorkflowScript(workflowLookupCode, filename);
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
Giving the lookup script its own getCachedWorkflowScript call invalidates the bounding invariant documented in script-cache.ts. That cache is keyed by code at the top level and capped at MAX_BUNDLES = 8, and its doc block asserts: "In production this bound is never reached: a deployment is its own process serving exactly one build-time bundle literal ... so there is a single code key for the process lifetime."
workflowLookupCode embeds the workflow name, so each distinct workflow becomes its own top-level code entry. An app with 8 or more workflows now exceeds the cap in production, and MAX_BUNDLES effectively bounds bundle versions + distinct workflow names rather than bundle versions.
No correctness or real cost impact, and I do not think it needs restructuring: eviction only runs when a new code key is inserted and takes from the front, and touchBundle(workflowCode) moves the bundle to MRU at the start of every session, so the bundle is never the eviction candidate — only the one-line lookup scripts churn, and recompiling those is free. But that doc block is now wrong for any app with 8+ workflows and will mislead the next person tuning this cache. Either update it or key the lookup scripts separately from the bundle LRU.
The WorkflowBundleCompileCacheHit attribute correctly reports only the bundle's hit and the comment already calls out why the lookup script's own hit rate is not the signal — that part is right.
Comment threadpackages/core/src/telemetry.ts
}

/** Keeps a parked workflow's ambient trace context aligned with each resume. */
export async function createRefreshableTraceContext() {
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
The returned context is a live proxy over a mutable current, which breaks OTel's immutable-Context contract. Re-parenting a resumed workflow's ambient context is the right goal — spans from a resume should not attach to the trace of the request that first parked the run — but the mechanism has an asymmetry worth documenting.
setValue forwards to current.setValue(...), and OTel contexts are immutable, so that returns a concrete derived context rather than another proxy. The consequence: work whose context is the proxy itself re-parents on refresh(), but anything that derives a context one level deeper (any nested trace() inside the workflow body, which captures the concrete context at creation) does not. So the re-parenting applies to direct children only, and refresh() retroactively changes the parent that already-pending continuations resolve to.
That is defensible for what this is for, but it is subtle enough that the next reader will assume uniform behaviour. A comment on the proxy naming the setValue boundary would save that.
The rest of the span lifecycle checks out: startTraceSpan's ended flag makes the vmTrace.fail in createWorkflowSession's catch a no-op once endVmTrace() has run, so a post-bootstrap throw such as WorkflowNotRegisteredError correctly leaves the VM span OK rather than marking it failed; and void execution.then(replayTrace.end, replayTrace.fail) handles the rejection without competing with the copy returned to the caller.
@NathanColosimo
NathanColosimo
commented
Aug 25, 2026

Copy link


Copy Markdown

ContributorAuthor
Stack repair note: this PR was marked merged by pushing its head into its stale circular base branch; no code from it reached main. The same tracing change now continues against the correct base in #3797.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@NathanColosimo@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
[core] Trace fresh workflow replay phases by NathanColosimo · Pull Request #3523 · vercel/workflow · GitHub
Skip to content
[core] Trace fresh workflow replay phases - #3523
Merged
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Aug 25, 2026
Merged
[core] Trace fresh workflow replay phases#3523
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • add focused spans for VM context creation, bundle compilation, bundle evaluation, input hydration, and replay execution
  • start the replay execution span before invoking user code so synchronous-before-first-await work is included
  • report whether the application bundle actually hit the compiled vm.Script cache
  • avoid a separate streamed-payload-preparation span: preparation intentionally overlaps event loading, while input hydration reports any residual blocking work
  • retain PR [core] Trace fresh workflow replay phases #3523 as the top layer so its deployed benchmark and trace history remain attached
Production trace shape
On 388 fresh-replay samples from the Next/Turbopack behavior run, median phase time was approximately:
  • 3.0 ms — workflow.vm.create_context
  • 33.0 ms — workflow.bundle.compile
  • 91.4 ms — workflow.bundle.evaluate
  • 1.2 ms — workflow.input.hydrate
  • 18.4 ms — workflow.replay.execute
The earlier 6.3 ms workflow.replay.prepare_payloads measurement represented an overlapping wall-time window rather than a blocking phase, so this revision removes that span instead of preserving a misleading trace shape.
This PR is intentionally observability-only against its base. The performance implementation is reviewed in #3547 and #3548.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Simplify replay payload preparation #3547 — synchronous replay decode
  3. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
  4. [core] Trace fresh workflow replay phases #3523 — replay phase tracing
  5. [core] Prune schema modules from workflow bundles #3550 — prune sandbox dependency graph
  6. [workflow] Load per-source replay bundles #3551 — per-source lazy VM bundles
Validation
  • core typecheck passed
  • full core suite: 2,116 tests passed, with 3 expected failures
  • focused tracing, script-cache, and replay-cache suites: 35 tests passed
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 2996b0a
The changes in this PR will be included in the next version bump.
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 11:37pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (244 failed)
astro-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
astro-quickjs (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
example-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K23M5D0GK1235ZCKHS2R1Z | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
example-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
express-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
express-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
fastify-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
fastify-quickjs (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
hono-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
hono-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nextjs-turbopack-node (3 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
nextjs-turbopack-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
nuxt-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nuxt-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1 | 🔍 observability
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z | 🔍 observability
sveltekit-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
sveltekit-quickjs (4 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
tanstack-start-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
tanstack-start-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes Promise.all of disjoint-key writes: every key lands
vite-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
vite-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K1ZE1VVFB967MCQ01CD795
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
46 flaky tests
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nest)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nest)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (sveltekit)
  • addTenWorkflow (astro)
  • addTenWorkflow (fastify)
  • cancelRun via CLI - cancelling a running workflow (sveltekit)
  • distributedAbortController - manual abort triggers signal (express)
  • distributedAbortController - reconnect to existing controller (example)
  • distributedAbortController - reconnect to existing controller (nest)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (fastify)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nuxt)
  • hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data (nextjs-webpack)
  • hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (nextjs-webpack)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-webpack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (express)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • no startIndex (reads all chunks) (express)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (fastify)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (nextjs-webpack)
  • Promise.all of disjoint-key writes: every key lands (example)
  • Promise.all of disjoint-key writes: every key lands (fastify)
  • Promise.all of disjoint-key writes: every key lands (sveltekit)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (express)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (example)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (express)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (vite)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nextjs-turbopack)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nitro)
  • sleepWinsRaceWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (vite)
  • start: initial attributes are seeded on run creation (nextjs-turbopack)
  • start: reserved-prefix initial attributes are seeded with allowReservedAttributes (tanstack-start)
  • step throw of a non-Error value preserves it as cause on the wrapping FatalError (nextjs-turbopack)
  • stepWinsRaceWorkflow (vite)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (hono)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (nest)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (tanstack-start)
  • webhookWorkflow (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (astro)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nest)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nuxt)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (tanstack-start)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
32 infra events
  • cold-start-warmup · suite warmup (python) · at 20:54:40Z · abandoned wrun_41M0K1HSDJ0GME5BYMX59CNX6K · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFQ0GKFWCKBYBX5Q6TF
  • run-pickup-stall · nullByteWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEG00GYMTSZ31K3DAPBQ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1M
  • run-pickup-stall · sleepingWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1K
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NF1Q0GMAYMPP86V96QQ6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:29Z · abandoned wrun_41M0K1PEDW0GZE4QZ5CFW30S42
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:55:57Z · abandoned wrun_41M0K1Q9SP0GTHYAKV39VBFX6P
  • run-pickup-stall · sleepingWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9SV0GK5F06DGM51605Q
  • run-pickup-stall · nullByteWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9W10GKRAC9NR0E87KFY
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9V00GW4CNMJKV4AXAGS
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:56:48Z · abandoned wrun_41M0K1QDWZ0GXPT291HQ94V9K2
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:58:29Z · abandoned wrun_41M0K1T5SG0GRDC5WEMPDD22EV
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 20:58:29Z · abandoned wrun_41M0K1T1P70GHK6GE0X7XYSHXJ
  • cold-start-warmup · suite warmup (python) · at 20:58:53Z · abandoned wrun_01M0K1SFM5SAP36XBNNF6BAB06 · (+7 more)
  • cold-start-warmup · suite warmup (tanstack-start) · at 20:59:04Z · abandoned wrun_01M0K1WMSYDP0A4AN0EZ2YNE7B
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RSD3H8J0V61PC9W5MK
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RRH9X8HXPZKXSEWHR3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S01X7VQ0X3SWF6FY11
  • run-pickup-stall · sleepingWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RYSY1H6DS6DJP1AJCJ
  • run-pickup-stall · nullByteWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S3C0M5KZEW6DRBHZ83
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCR8A6B5DQXHZ0XTPEG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCTEEFAHZFWC64SGN5T
  • run-pickup-stall · nullByteWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCWJRW9Q8395REQRV4X
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCNNJ4CY13PGCBXDGSB
  • run-pickup-stall · sleepingWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCQJ9PV2MH9BM4DJF6A
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0BSDW3JADX703S5W05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0FWPYY292RKBY8CC9X
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:01:08Z · abandoned wrun_01M0K20T0JG89Z3DB7M0QF7DGB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBY519CZXPW964N7N4B
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBZXRG7XF8BEYE0CAHE
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:02:08Z · abandoned wrun_01M0K22MKP6JKDYT80XXGFSRPG
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production33342447424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total15447253254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node124828
❌ astro-quickjs1211128
❌ example-node123928
❌ example-quickjs1201228
❌ express-node124828
❌ express-quickjs1221028
❌ fastify-node1211128
❌ fastify-quickjs124828
❌ hono-node123928
❌ hono-quickjs1221028
❌ nest-node123928
❌ nest-quickjs1191328
❌ nextjs-turbopack-node15433
❌ nextjs-turbopack-quickjs14893
❌ nextjs-webpack-node15073
❌ nextjs-webpack-quickjs144133
❌ nitro-node1211128
❌ nitro-quickjs1221028
❌ nuxt-node125728
❌ nuxt-quickjs1221028
❌ python-node08152
❌ sveltekit-node14479
❌ sveltekit-quickjs14749
❌ tanstack-start-node123928
❌ tanstack-start-quickjs123928
❌ vite-node125728
❌ vite-quickjs1201228
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
❌ The benchmark run for 1a35510 failed. See the run logs for details.
commit d74f748 · Thu, 20 Aug 2026 18:02:44 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep248 (-76%) 💚1391 🔴 (+23%) 🔻1424 🔴 (+22%) 🔻1739 🔴 (+42%) 🔻30
TTFSstream277 (+56%) 🔻1424 🔴 (+27%) 🔻1473 🔴 (+31%) 🔻1861 🔴 (+59%) 🔻30
TTFShook + stream374 (-25%) 💚1623 🔴 (+13%)1687 🔴 (+14%)1835 🔴 (+19%) 🔻30
Fan-out TTFSPromise.all(100 steps)557 (+40%) 🔻967 (+8.0%)2100 (+20%) 🔻2302 (+22%) 🔻10
Fan-out TTLSPromise.all(100 steps)4980 (+4.0%)7820 (+27%) 🔻8029 (+28%) 🔻11536 (+60%) 🔻10
STSO1020 steps (inline)135 (+13%)274 (+11%)319 (+6.3%)575 (+14%)1019
WO1020 steps265663 (+12%)265663 (+12%)265663 (+12%)265663 (+12%)1
CRTTfirst chunk (pooled)101 (+9.8%)166 (+26%) 🔻411 (+172%) 🔻1142 (+592%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)134 (+9%)298 (+106%)368 (+39%)1053 (-5%)263 (+79%)10
size sweep (100/s, 160B-12KB)133 (+2%)203 (-8%)482 (-8%)884 (+14%)211 (-23%)10
replay gateway-gpt-5.4-nano-2000t (1x)138 (+14%)162 (+10%)300 (+17%)645 (-8%)461 (-11%)3
replay eve-gpt-5.6-sol-2000t (1x)314 (+146%)184 (+21%)351 (+38%)797 (-5%)696 (+39%)2
replay eve-gpt-5.6-sol-2000t (2x)147 (+20%)244 (+16%)397 (+23%)675 (-80%)544 (-10%)3
📜 Previous results (2)
50faa3b
Wed, 19 Aug 2026 17:28:21 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1335 (+365%) 🔻1423 🔴 (+22%) 🔻1441 🔴 (+7.6%)1534 🔴 (-1.9%)30
TTFSstream1342 (+610%) 🔻1410 🔴 (+26%) 🔻1430 🔴 (+25%) 🔻1576 🔴 (+35%) 🔻30
TTFShook + stream1514 (+16%) 🔻1693 🔴 (+23%) 🔻1705 🔴 (+21%) 🔻2082 🔴 (+21%) 🔻30
Fan-out TTFSPromise.all(100 steps)1088 (-1.5%)2594 (+13%)2614 (-5.5%)2987 (+7.8%)10
Fan-out TTLSPromise.all(100 steps)5278 (±0%)6182 (-7.1%)6444 (-6.4%)8094 (+2.7%)10
STSO1020 steps (inline)120 (-9.1%)224 (-15%)249 (-18%) 💚336 (-33%) 💚1019
WO1020 steps216112 (-13%)216112 (-13%)216112 (-13%)216112 (-13%)1
CRTTfirst chunk (pooled)80 (-4.8%)126 (-0.8%)269 (-15%)567 (-9.6%)28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)99 (-12%)139 (-16%)221 (-5%)430 (-8%)109 (-32%)10
size sweep (100/s, 160B-12KB)108 (±0%)128 (-17%)220 (-22%)604 (-11%)104 (-36%)10
replay gateway-gpt-5.4-nano-2000t (1x)93 (-17%)159 (-43%)268 (-63%)424 (-72%)368 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)333 (+114%)129 (-8%)239 (+24%)447 (-29%)302 (-38%)2
replay eve-gpt-5.6-sol-2000t (2x)108 (-79%)344 (-15%)553 (-80%)751 (-82%)509 (-16%)3
50faa3b
Tue, 18 Aug 2026 23:23:51 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1377 (+29%) 🔻1439 🔴 (+20%) 🔻1460 🔴 (+15%) 🔻1585 🔴 (-0.9%)30
TTFSstream225 (-27%) 💚1436 🔴 (+29%) 🔻1458 🔴 (+24%) 🔻1865 🔴 (+51%) 🔻30
TTFShook + stream346 (-74%) 💚1711 🔴 (+14%)1742 🔴 (+12%)1816 🔴 (-7.3%)30
Fan-out TTFSPromise.all(100 steps)1521 (+25%) 🔻3074 (+29%) 🔻3100 (+25%) 🔻3101 (+15%)10
Fan-out TTLSPromise.all(100 steps)7331 (+37%) 🔻27947 (+251%) 🔻310173 (+3739%) 🔻313908 (+3301%) 🔻10
CRTTfirst chunk (pooled)108 (-7.7%)164 (-19%) 💚213 (-34%) 💚416 (-34%) 💚25
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)138 (-15%)197 (-45%)272 (-60%)1221 (-23%)181 (-52%)10
size sweep (100/s, 160B-12KB)147 (-2%)379 (+94%)642 (+146%)933 (-16%)247 (+54%)10
replay gateway-gpt-5.4-nano-2000t (1x)113 (-33%)196 (-5%)355 (+9%)821 (+42%)242 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)147 (-33%)156 (-20%)194 (-30%)424 (-14%)302 (-30%)2
ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
vercel[bot]
vercelBot
reviewed
Aug 13, 2026
View reviewed changes
Comment threadpackages/core/src/replay-payload-cache.ts
Outdated


This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues
Comment threadpackages/core/src/workflow.ts
 'workflow.bundle.compile',
 async (span) => {
 const bundle = getCachedWorkflowScript(workflowCode, filename);
 const lookup = getCachedWorkflowScript(workflowLookupCode, filename);
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
Giving the lookup script its own getCachedWorkflowScript call invalidates the bounding invariant documented in script-cache.ts. That cache is keyed by code at the top level and capped at MAX_BUNDLES = 8, and its doc block asserts: "In production this bound is never reached: a deployment is its own process serving exactly one build-time bundle literal ... so there is a single code key for the process lifetime."
workflowLookupCode embeds the workflow name, so each distinct workflow becomes its own top-level code entry. An app with 8 or more workflows now exceeds the cap in production, and MAX_BUNDLES effectively bounds bundle versions + distinct workflow names rather than bundle versions.
No correctness or real cost impact, and I do not think it needs restructuring: eviction only runs when a new code key is inserted and takes from the front, and touchBundle(workflowCode) moves the bundle to MRU at the start of every session, so the bundle is never the eviction candidate — only the one-line lookup scripts churn, and recompiling those is free. But that doc block is now wrong for any app with 8+ workflows and will mislead the next person tuning this cache. Either update it or key the lookup scripts separately from the bundle LRU.
The WorkflowBundleCompileCacheHit attribute correctly reports only the bundle's hit and the comment already calls out why the lookup script's own hit rate is not the signal — that part is right.
Comment threadpackages/core/src/telemetry.ts
}

/** Keeps a parked workflow's ambient trace context aligned with each resume. */
export async function createRefreshableTraceContext() {
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
The returned context is a live proxy over a mutable current, which breaks OTel's immutable-Context contract. Re-parenting a resumed workflow's ambient context is the right goal — spans from a resume should not attach to the trace of the request that first parked the run — but the mechanism has an asymmetry worth documenting.
setValue forwards to current.setValue(...), and OTel contexts are immutable, so that returns a concrete derived context rather than another proxy. The consequence: work whose context is the proxy itself re-parents on refresh(), but anything that derives a context one level deeper (any nested trace() inside the workflow body, which captures the concrete context at creation) does not. So the re-parenting applies to direct children only, and refresh() retroactively changes the parent that already-pending continuations resolve to.
That is defensible for what this is for, but it is subtle enough that the next reader will assume uniform behaviour. A comment on the proxy naming the setValue boundary would save that.
The rest of the span lifecycle checks out: startTraceSpan's ended flag makes the vmTrace.fail in createWorkflowSession's catch a no-op once endVmTrace() has run, so a post-bootstrap throw such as WorkflowNotRegisteredError correctly leaves the VM span OK rather than marking it failed; and void execution.then(replayTrace.end, replayTrace.fail) handles the rejection without competing with the copy returned to the caller.
@NathanColosimo
NathanColosimo
commented
Aug 25, 2026

Copy link


Copy Markdown

ContributorAuthor
Stack repair note: this PR was marked merged by pushing its head into its stale circular base branch; no code from it reached main. The same tracing change now continues against the correct base in #3797.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@NathanColosimo@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
[core] Trace fresh workflow replay phases by NathanColosimo · Pull Request #3523 · vercel/workflow · GitHub
Skip to content
[core] Trace fresh workflow replay phases - #3523
Merged
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Aug 25, 2026
Merged
[core] Trace fresh workflow replay phases#3523
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • add focused spans for VM context creation, bundle compilation, bundle evaluation, input hydration, and replay execution
  • start the replay execution span before invoking user code so synchronous-before-first-await work is included
  • report whether the application bundle actually hit the compiled vm.Script cache
  • avoid a separate streamed-payload-preparation span: preparation intentionally overlaps event loading, while input hydration reports any residual blocking work
  • retain PR [core] Trace fresh workflow replay phases #3523 as the top layer so its deployed benchmark and trace history remain attached
Production trace shape
On 388 fresh-replay samples from the Next/Turbopack behavior run, median phase time was approximately:
  • 3.0 ms — workflow.vm.create_context
  • 33.0 ms — workflow.bundle.compile
  • 91.4 ms — workflow.bundle.evaluate
  • 1.2 ms — workflow.input.hydrate
  • 18.4 ms — workflow.replay.execute
The earlier 6.3 ms workflow.replay.prepare_payloads measurement represented an overlapping wall-time window rather than a blocking phase, so this revision removes that span instead of preserving a misleading trace shape.
This PR is intentionally observability-only against its base. The performance implementation is reviewed in #3547 and #3548.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Simplify replay payload preparation #3547 — synchronous replay decode
  3. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
  4. [core] Trace fresh workflow replay phases #3523 — replay phase tracing
  5. [core] Prune schema modules from workflow bundles #3550 — prune sandbox dependency graph
  6. [workflow] Load per-source replay bundles #3551 — per-source lazy VM bundles
Validation
  • core typecheck passed
  • full core suite: 2,116 tests passed, with 3 expected failures
  • focused tracing, script-cache, and replay-cache suites: 35 tests passed
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 2996b0a
The changes in this PR will be included in the next version bump.
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 11:37pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (244 failed)
astro-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
astro-quickjs (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
example-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K23M5D0GK1235ZCKHS2R1Z | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
example-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
express-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
express-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
fastify-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
fastify-quickjs (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
hono-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
hono-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nextjs-turbopack-node (3 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
nextjs-turbopack-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
nuxt-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nuxt-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1 | 🔍 observability
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z | 🔍 observability
sveltekit-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
sveltekit-quickjs (4 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
tanstack-start-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
tanstack-start-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes Promise.all of disjoint-key writes: every key lands
vite-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
vite-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K1ZE1VVFB967MCQ01CD795
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
46 flaky tests
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nest)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nest)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (sveltekit)
  • addTenWorkflow (astro)
  • addTenWorkflow (fastify)
  • cancelRun via CLI - cancelling a running workflow (sveltekit)
  • distributedAbortController - manual abort triggers signal (express)
  • distributedAbortController - reconnect to existing controller (example)
  • distributedAbortController - reconnect to existing controller (nest)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (fastify)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nuxt)
  • hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data (nextjs-webpack)
  • hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (nextjs-webpack)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-webpack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (express)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • no startIndex (reads all chunks) (express)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (fastify)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (nextjs-webpack)
  • Promise.all of disjoint-key writes: every key lands (example)
  • Promise.all of disjoint-key writes: every key lands (fastify)
  • Promise.all of disjoint-key writes: every key lands (sveltekit)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (express)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (example)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (express)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (vite)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nextjs-turbopack)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nitro)
  • sleepWinsRaceWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (vite)
  • start: initial attributes are seeded on run creation (nextjs-turbopack)
  • start: reserved-prefix initial attributes are seeded with allowReservedAttributes (tanstack-start)
  • step throw of a non-Error value preserves it as cause on the wrapping FatalError (nextjs-turbopack)
  • stepWinsRaceWorkflow (vite)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (hono)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (nest)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (tanstack-start)
  • webhookWorkflow (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (astro)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nest)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nuxt)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (tanstack-start)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
32 infra events
  • cold-start-warmup · suite warmup (python) · at 20:54:40Z · abandoned wrun_41M0K1HSDJ0GME5BYMX59CNX6K · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFQ0GKFWCKBYBX5Q6TF
  • run-pickup-stall · nullByteWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEG00GYMTSZ31K3DAPBQ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1M
  • run-pickup-stall · sleepingWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1K
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NF1Q0GMAYMPP86V96QQ6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:29Z · abandoned wrun_41M0K1PEDW0GZE4QZ5CFW30S42
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:55:57Z · abandoned wrun_41M0K1Q9SP0GTHYAKV39VBFX6P
  • run-pickup-stall · sleepingWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9SV0GK5F06DGM51605Q
  • run-pickup-stall · nullByteWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9W10GKRAC9NR0E87KFY
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9V00GW4CNMJKV4AXAGS
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:56:48Z · abandoned wrun_41M0K1QDWZ0GXPT291HQ94V9K2
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:58:29Z · abandoned wrun_41M0K1T5SG0GRDC5WEMPDD22EV
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 20:58:29Z · abandoned wrun_41M0K1T1P70GHK6GE0X7XYSHXJ
  • cold-start-warmup · suite warmup (python) · at 20:58:53Z · abandoned wrun_01M0K1SFM5SAP36XBNNF6BAB06 · (+7 more)
  • cold-start-warmup · suite warmup (tanstack-start) · at 20:59:04Z · abandoned wrun_01M0K1WMSYDP0A4AN0EZ2YNE7B
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RSD3H8J0V61PC9W5MK
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RRH9X8HXPZKXSEWHR3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S01X7VQ0X3SWF6FY11
  • run-pickup-stall · sleepingWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RYSY1H6DS6DJP1AJCJ
  • run-pickup-stall · nullByteWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S3C0M5KZEW6DRBHZ83
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCR8A6B5DQXHZ0XTPEG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCTEEFAHZFWC64SGN5T
  • run-pickup-stall · nullByteWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCWJRW9Q8395REQRV4X
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCNNJ4CY13PGCBXDGSB
  • run-pickup-stall · sleepingWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCQJ9PV2MH9BM4DJF6A
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0BSDW3JADX703S5W05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0FWPYY292RKBY8CC9X
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:01:08Z · abandoned wrun_01M0K20T0JG89Z3DB7M0QF7DGB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBY519CZXPW964N7N4B
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBZXRG7XF8BEYE0CAHE
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:02:08Z · abandoned wrun_01M0K22MKP6JKDYT80XXGFSRPG
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production33342447424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total15447253254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node124828
❌ astro-quickjs1211128
❌ example-node123928
❌ example-quickjs1201228
❌ express-node124828
❌ express-quickjs1221028
❌ fastify-node1211128
❌ fastify-quickjs124828
❌ hono-node123928
❌ hono-quickjs1221028
❌ nest-node123928
❌ nest-quickjs1191328
❌ nextjs-turbopack-node15433
❌ nextjs-turbopack-quickjs14893
❌ nextjs-webpack-node15073
❌ nextjs-webpack-quickjs144133
❌ nitro-node1211128
❌ nitro-quickjs1221028
❌ nuxt-node125728
❌ nuxt-quickjs1221028
❌ python-node08152
❌ sveltekit-node14479
❌ sveltekit-quickjs14749
❌ tanstack-start-node123928
❌ tanstack-start-quickjs123928
❌ vite-node125728
❌ vite-quickjs1201228
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
❌ The benchmark run for 1a35510 failed. See the run logs for details.
commit d74f748 · Thu, 20 Aug 2026 18:02:44 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep248 (-76%) 💚1391 🔴 (+23%) 🔻1424 🔴 (+22%) 🔻1739 🔴 (+42%) 🔻30
TTFSstream277 (+56%) 🔻1424 🔴 (+27%) 🔻1473 🔴 (+31%) 🔻1861 🔴 (+59%) 🔻30
TTFShook + stream374 (-25%) 💚1623 🔴 (+13%)1687 🔴 (+14%)1835 🔴 (+19%) 🔻30
Fan-out TTFSPromise.all(100 steps)557 (+40%) 🔻967 (+8.0%)2100 (+20%) 🔻2302 (+22%) 🔻10
Fan-out TTLSPromise.all(100 steps)4980 (+4.0%)7820 (+27%) 🔻8029 (+28%) 🔻11536 (+60%) 🔻10
STSO1020 steps (inline)135 (+13%)274 (+11%)319 (+6.3%)575 (+14%)1019
WO1020 steps265663 (+12%)265663 (+12%)265663 (+12%)265663 (+12%)1
CRTTfirst chunk (pooled)101 (+9.8%)166 (+26%) 🔻411 (+172%) 🔻1142 (+592%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)134 (+9%)298 (+106%)368 (+39%)1053 (-5%)263 (+79%)10
size sweep (100/s, 160B-12KB)133 (+2%)203 (-8%)482 (-8%)884 (+14%)211 (-23%)10
replay gateway-gpt-5.4-nano-2000t (1x)138 (+14%)162 (+10%)300 (+17%)645 (-8%)461 (-11%)3
replay eve-gpt-5.6-sol-2000t (1x)314 (+146%)184 (+21%)351 (+38%)797 (-5%)696 (+39%)2
replay eve-gpt-5.6-sol-2000t (2x)147 (+20%)244 (+16%)397 (+23%)675 (-80%)544 (-10%)3
📜 Previous results (2)
50faa3b
Wed, 19 Aug 2026 17:28:21 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1335 (+365%) 🔻1423 🔴 (+22%) 🔻1441 🔴 (+7.6%)1534 🔴 (-1.9%)30
TTFSstream1342 (+610%) 🔻1410 🔴 (+26%) 🔻1430 🔴 (+25%) 🔻1576 🔴 (+35%) 🔻30
TTFShook + stream1514 (+16%) 🔻1693 🔴 (+23%) 🔻1705 🔴 (+21%) 🔻2082 🔴 (+21%) 🔻30
Fan-out TTFSPromise.all(100 steps)1088 (-1.5%)2594 (+13%)2614 (-5.5%)2987 (+7.8%)10
Fan-out TTLSPromise.all(100 steps)5278 (±0%)6182 (-7.1%)6444 (-6.4%)8094 (+2.7%)10
STSO1020 steps (inline)120 (-9.1%)224 (-15%)249 (-18%) 💚336 (-33%) 💚1019
WO1020 steps216112 (-13%)216112 (-13%)216112 (-13%)216112 (-13%)1
CRTTfirst chunk (pooled)80 (-4.8%)126 (-0.8%)269 (-15%)567 (-9.6%)28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)99 (-12%)139 (-16%)221 (-5%)430 (-8%)109 (-32%)10
size sweep (100/s, 160B-12KB)108 (±0%)128 (-17%)220 (-22%)604 (-11%)104 (-36%)10
replay gateway-gpt-5.4-nano-2000t (1x)93 (-17%)159 (-43%)268 (-63%)424 (-72%)368 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)333 (+114%)129 (-8%)239 (+24%)447 (-29%)302 (-38%)2
replay eve-gpt-5.6-sol-2000t (2x)108 (-79%)344 (-15%)553 (-80%)751 (-82%)509 (-16%)3
50faa3b
Tue, 18 Aug 2026 23:23:51 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1377 (+29%) 🔻1439 🔴 (+20%) 🔻1460 🔴 (+15%) 🔻1585 🔴 (-0.9%)30
TTFSstream225 (-27%) 💚1436 🔴 (+29%) 🔻1458 🔴 (+24%) 🔻1865 🔴 (+51%) 🔻30
TTFShook + stream346 (-74%) 💚1711 🔴 (+14%)1742 🔴 (+12%)1816 🔴 (-7.3%)30
Fan-out TTFSPromise.all(100 steps)1521 (+25%) 🔻3074 (+29%) 🔻3100 (+25%) 🔻3101 (+15%)10
Fan-out TTLSPromise.all(100 steps)7331 (+37%) 🔻27947 (+251%) 🔻310173 (+3739%) 🔻313908 (+3301%) 🔻10
CRTTfirst chunk (pooled)108 (-7.7%)164 (-19%) 💚213 (-34%) 💚416 (-34%) 💚25
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)138 (-15%)197 (-45%)272 (-60%)1221 (-23%)181 (-52%)10
size sweep (100/s, 160B-12KB)147 (-2%)379 (+94%)642 (+146%)933 (-16%)247 (+54%)10
replay gateway-gpt-5.4-nano-2000t (1x)113 (-33%)196 (-5%)355 (+9%)821 (+42%)242 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)147 (-33%)156 (-20%)194 (-30%)424 (-14%)302 (-30%)2
ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
vercel[bot]
vercelBot
reviewed
Aug 13, 2026
View reviewed changes
Comment threadpackages/core/src/replay-payload-cache.ts
Outdated


This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues
Comment threadpackages/core/src/workflow.ts
 'workflow.bundle.compile',
 async (span) => {
 const bundle = getCachedWorkflowScript(workflowCode, filename);
 const lookup = getCachedWorkflowScript(workflowLookupCode, filename);
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
Giving the lookup script its own getCachedWorkflowScript call invalidates the bounding invariant documented in script-cache.ts. That cache is keyed by code at the top level and capped at MAX_BUNDLES = 8, and its doc block asserts: "In production this bound is never reached: a deployment is its own process serving exactly one build-time bundle literal ... so there is a single code key for the process lifetime."
workflowLookupCode embeds the workflow name, so each distinct workflow becomes its own top-level code entry. An app with 8 or more workflows now exceeds the cap in production, and MAX_BUNDLES effectively bounds bundle versions + distinct workflow names rather than bundle versions.
No correctness or real cost impact, and I do not think it needs restructuring: eviction only runs when a new code key is inserted and takes from the front, and touchBundle(workflowCode) moves the bundle to MRU at the start of every session, so the bundle is never the eviction candidate — only the one-line lookup scripts churn, and recompiling those is free. But that doc block is now wrong for any app with 8+ workflows and will mislead the next person tuning this cache. Either update it or key the lookup scripts separately from the bundle LRU.
The WorkflowBundleCompileCacheHit attribute correctly reports only the bundle's hit and the comment already calls out why the lookup script's own hit rate is not the signal — that part is right.
Comment threadpackages/core/src/telemetry.ts
}

/** Keeps a parked workflow's ambient trace context aligned with each resume. */
export async function createRefreshableTraceContext() {
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
The returned context is a live proxy over a mutable current, which breaks OTel's immutable-Context contract. Re-parenting a resumed workflow's ambient context is the right goal — spans from a resume should not attach to the trace of the request that first parked the run — but the mechanism has an asymmetry worth documenting.
setValue forwards to current.setValue(...), and OTel contexts are immutable, so that returns a concrete derived context rather than another proxy. The consequence: work whose context is the proxy itself re-parents on refresh(), but anything that derives a context one level deeper (any nested trace() inside the workflow body, which captures the concrete context at creation) does not. So the re-parenting applies to direct children only, and refresh() retroactively changes the parent that already-pending continuations resolve to.
That is defensible for what this is for, but it is subtle enough that the next reader will assume uniform behaviour. A comment on the proxy naming the setValue boundary would save that.
The rest of the span lifecycle checks out: startTraceSpan's ended flag makes the vmTrace.fail in createWorkflowSession's catch a no-op once endVmTrace() has run, so a post-bootstrap throw such as WorkflowNotRegisteredError correctly leaves the VM span OK rather than marking it failed; and void execution.then(replayTrace.end, replayTrace.fail) handles the rejection without competing with the copy returned to the caller.
@NathanColosimo
NathanColosimo
commented
Aug 25, 2026

Copy link


Copy Markdown

ContributorAuthor
Stack repair note: this PR was marked merged by pushing its head into its stale circular base branch; no code from it reached main. The same tracing change now continues against the correct base in #3797.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@NathanColosimo@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
[core] Trace fresh workflow replay phases by NathanColosimo · Pull Request #3523 · vercel/workflow · GitHub
Skip to content
[core] Trace fresh workflow replay phases - #3523
Merged
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Aug 25, 2026
Merged
[core] Trace fresh workflow replay phases#3523
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • add focused spans for VM context creation, bundle compilation, bundle evaluation, input hydration, and replay execution
  • start the replay execution span before invoking user code so synchronous-before-first-await work is included
  • report whether the application bundle actually hit the compiled vm.Script cache
  • avoid a separate streamed-payload-preparation span: preparation intentionally overlaps event loading, while input hydration reports any residual blocking work
  • retain PR [core] Trace fresh workflow replay phases #3523 as the top layer so its deployed benchmark and trace history remain attached
Production trace shape
On 388 fresh-replay samples from the Next/Turbopack behavior run, median phase time was approximately:
  • 3.0 ms — workflow.vm.create_context
  • 33.0 ms — workflow.bundle.compile
  • 91.4 ms — workflow.bundle.evaluate
  • 1.2 ms — workflow.input.hydrate
  • 18.4 ms — workflow.replay.execute
The earlier 6.3 ms workflow.replay.prepare_payloads measurement represented an overlapping wall-time window rather than a blocking phase, so this revision removes that span instead of preserving a misleading trace shape.
This PR is intentionally observability-only against its base. The performance implementation is reviewed in #3547 and #3548.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Simplify replay payload preparation #3547 — synchronous replay decode
  3. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
  4. [core] Trace fresh workflow replay phases #3523 — replay phase tracing
  5. [core] Prune schema modules from workflow bundles #3550 — prune sandbox dependency graph
  6. [workflow] Load per-source replay bundles #3551 — per-source lazy VM bundles
Validation
  • core typecheck passed
  • full core suite: 2,116 tests passed, with 3 expected failures
  • focused tracing, script-cache, and replay-cache suites: 35 tests passed
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 2996b0a
The changes in this PR will be included in the next version bump.
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 11:37pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (244 failed)
astro-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
astro-quickjs (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
example-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K23M5D0GK1235ZCKHS2R1Z | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
example-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
express-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
express-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
fastify-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
fastify-quickjs (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
hono-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
hono-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nextjs-turbopack-node (3 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
nextjs-turbopack-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
nuxt-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nuxt-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1 | 🔍 observability
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z | 🔍 observability
sveltekit-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
sveltekit-quickjs (4 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
tanstack-start-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
tanstack-start-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes Promise.all of disjoint-key writes: every key lands
vite-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
vite-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K1ZE1VVFB967MCQ01CD795
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
46 flaky tests
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nest)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nest)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (sveltekit)
  • addTenWorkflow (astro)
  • addTenWorkflow (fastify)
  • cancelRun via CLI - cancelling a running workflow (sveltekit)
  • distributedAbortController - manual abort triggers signal (express)
  • distributedAbortController - reconnect to existing controller (example)
  • distributedAbortController - reconnect to existing controller (nest)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (fastify)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nuxt)
  • hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data (nextjs-webpack)
  • hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (nextjs-webpack)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-webpack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (express)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • no startIndex (reads all chunks) (express)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (fastify)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (nextjs-webpack)
  • Promise.all of disjoint-key writes: every key lands (example)
  • Promise.all of disjoint-key writes: every key lands (fastify)
  • Promise.all of disjoint-key writes: every key lands (sveltekit)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (express)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (example)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (express)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (vite)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nextjs-turbopack)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nitro)
  • sleepWinsRaceWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (vite)
  • start: initial attributes are seeded on run creation (nextjs-turbopack)
  • start: reserved-prefix initial attributes are seeded with allowReservedAttributes (tanstack-start)
  • step throw of a non-Error value preserves it as cause on the wrapping FatalError (nextjs-turbopack)
  • stepWinsRaceWorkflow (vite)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (hono)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (nest)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (tanstack-start)
  • webhookWorkflow (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (astro)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nest)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nuxt)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (tanstack-start)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
32 infra events
  • cold-start-warmup · suite warmup (python) · at 20:54:40Z · abandoned wrun_41M0K1HSDJ0GME5BYMX59CNX6K · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFQ0GKFWCKBYBX5Q6TF
  • run-pickup-stall · nullByteWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEG00GYMTSZ31K3DAPBQ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1M
  • run-pickup-stall · sleepingWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1K
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NF1Q0GMAYMPP86V96QQ6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:29Z · abandoned wrun_41M0K1PEDW0GZE4QZ5CFW30S42
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:55:57Z · abandoned wrun_41M0K1Q9SP0GTHYAKV39VBFX6P
  • run-pickup-stall · sleepingWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9SV0GK5F06DGM51605Q
  • run-pickup-stall · nullByteWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9W10GKRAC9NR0E87KFY
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9V00GW4CNMJKV4AXAGS
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:56:48Z · abandoned wrun_41M0K1QDWZ0GXPT291HQ94V9K2
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:58:29Z · abandoned wrun_41M0K1T5SG0GRDC5WEMPDD22EV
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 20:58:29Z · abandoned wrun_41M0K1T1P70GHK6GE0X7XYSHXJ
  • cold-start-warmup · suite warmup (python) · at 20:58:53Z · abandoned wrun_01M0K1SFM5SAP36XBNNF6BAB06 · (+7 more)
  • cold-start-warmup · suite warmup (tanstack-start) · at 20:59:04Z · abandoned wrun_01M0K1WMSYDP0A4AN0EZ2YNE7B
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RSD3H8J0V61PC9W5MK
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RRH9X8HXPZKXSEWHR3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S01X7VQ0X3SWF6FY11
  • run-pickup-stall · sleepingWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RYSY1H6DS6DJP1AJCJ
  • run-pickup-stall · nullByteWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S3C0M5KZEW6DRBHZ83
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCR8A6B5DQXHZ0XTPEG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCTEEFAHZFWC64SGN5T
  • run-pickup-stall · nullByteWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCWJRW9Q8395REQRV4X
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCNNJ4CY13PGCBXDGSB
  • run-pickup-stall · sleepingWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCQJ9PV2MH9BM4DJF6A
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0BSDW3JADX703S5W05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0FWPYY292RKBY8CC9X
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:01:08Z · abandoned wrun_01M0K20T0JG89Z3DB7M0QF7DGB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBY519CZXPW964N7N4B
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBZXRG7XF8BEYE0CAHE
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:02:08Z · abandoned wrun_01M0K22MKP6JKDYT80XXGFSRPG
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production33342447424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total15447253254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node124828
❌ astro-quickjs1211128
❌ example-node123928
❌ example-quickjs1201228
❌ express-node124828
❌ express-quickjs1221028
❌ fastify-node1211128
❌ fastify-quickjs124828
❌ hono-node123928
❌ hono-quickjs1221028
❌ nest-node123928
❌ nest-quickjs1191328
❌ nextjs-turbopack-node15433
❌ nextjs-turbopack-quickjs14893
❌ nextjs-webpack-node15073
❌ nextjs-webpack-quickjs144133
❌ nitro-node1211128
❌ nitro-quickjs1221028
❌ nuxt-node125728
❌ nuxt-quickjs1221028
❌ python-node08152
❌ sveltekit-node14479
❌ sveltekit-quickjs14749
❌ tanstack-start-node123928
❌ tanstack-start-quickjs123928
❌ vite-node125728
❌ vite-quickjs1201228
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
❌ The benchmark run for 1a35510 failed. See the run logs for details.
commit d74f748 · Thu, 20 Aug 2026 18:02:44 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep248 (-76%) 💚1391 🔴 (+23%) 🔻1424 🔴 (+22%) 🔻1739 🔴 (+42%) 🔻30
TTFSstream277 (+56%) 🔻1424 🔴 (+27%) 🔻1473 🔴 (+31%) 🔻1861 🔴 (+59%) 🔻30
TTFShook + stream374 (-25%) 💚1623 🔴 (+13%)1687 🔴 (+14%)1835 🔴 (+19%) 🔻30
Fan-out TTFSPromise.all(100 steps)557 (+40%) 🔻967 (+8.0%)2100 (+20%) 🔻2302 (+22%) 🔻10
Fan-out TTLSPromise.all(100 steps)4980 (+4.0%)7820 (+27%) 🔻8029 (+28%) 🔻11536 (+60%) 🔻10
STSO1020 steps (inline)135 (+13%)274 (+11%)319 (+6.3%)575 (+14%)1019
WO1020 steps265663 (+12%)265663 (+12%)265663 (+12%)265663 (+12%)1
CRTTfirst chunk (pooled)101 (+9.8%)166 (+26%) 🔻411 (+172%) 🔻1142 (+592%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)134 (+9%)298 (+106%)368 (+39%)1053 (-5%)263 (+79%)10
size sweep (100/s, 160B-12KB)133 (+2%)203 (-8%)482 (-8%)884 (+14%)211 (-23%)10
replay gateway-gpt-5.4-nano-2000t (1x)138 (+14%)162 (+10%)300 (+17%)645 (-8%)461 (-11%)3
replay eve-gpt-5.6-sol-2000t (1x)314 (+146%)184 (+21%)351 (+38%)797 (-5%)696 (+39%)2
replay eve-gpt-5.6-sol-2000t (2x)147 (+20%)244 (+16%)397 (+23%)675 (-80%)544 (-10%)3
📜 Previous results (2)
50faa3b
Wed, 19 Aug 2026 17:28:21 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1335 (+365%) 🔻1423 🔴 (+22%) 🔻1441 🔴 (+7.6%)1534 🔴 (-1.9%)30
TTFSstream1342 (+610%) 🔻1410 🔴 (+26%) 🔻1430 🔴 (+25%) 🔻1576 🔴 (+35%) 🔻30
TTFShook + stream1514 (+16%) 🔻1693 🔴 (+23%) 🔻1705 🔴 (+21%) 🔻2082 🔴 (+21%) 🔻30
Fan-out TTFSPromise.all(100 steps)1088 (-1.5%)2594 (+13%)2614 (-5.5%)2987 (+7.8%)10
Fan-out TTLSPromise.all(100 steps)5278 (±0%)6182 (-7.1%)6444 (-6.4%)8094 (+2.7%)10
STSO1020 steps (inline)120 (-9.1%)224 (-15%)249 (-18%) 💚336 (-33%) 💚1019
WO1020 steps216112 (-13%)216112 (-13%)216112 (-13%)216112 (-13%)1
CRTTfirst chunk (pooled)80 (-4.8%)126 (-0.8%)269 (-15%)567 (-9.6%)28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)99 (-12%)139 (-16%)221 (-5%)430 (-8%)109 (-32%)10
size sweep (100/s, 160B-12KB)108 (±0%)128 (-17%)220 (-22%)604 (-11%)104 (-36%)10
replay gateway-gpt-5.4-nano-2000t (1x)93 (-17%)159 (-43%)268 (-63%)424 (-72%)368 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)333 (+114%)129 (-8%)239 (+24%)447 (-29%)302 (-38%)2
replay eve-gpt-5.6-sol-2000t (2x)108 (-79%)344 (-15%)553 (-80%)751 (-82%)509 (-16%)3
50faa3b
Tue, 18 Aug 2026 23:23:51 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1377 (+29%) 🔻1439 🔴 (+20%) 🔻1460 🔴 (+15%) 🔻1585 🔴 (-0.9%)30
TTFSstream225 (-27%) 💚1436 🔴 (+29%) 🔻1458 🔴 (+24%) 🔻1865 🔴 (+51%) 🔻30
TTFShook + stream346 (-74%) 💚1711 🔴 (+14%)1742 🔴 (+12%)1816 🔴 (-7.3%)30
Fan-out TTFSPromise.all(100 steps)1521 (+25%) 🔻3074 (+29%) 🔻3100 (+25%) 🔻3101 (+15%)10
Fan-out TTLSPromise.all(100 steps)7331 (+37%) 🔻27947 (+251%) 🔻310173 (+3739%) 🔻313908 (+3301%) 🔻10
CRTTfirst chunk (pooled)108 (-7.7%)164 (-19%) 💚213 (-34%) 💚416 (-34%) 💚25
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)138 (-15%)197 (-45%)272 (-60%)1221 (-23%)181 (-52%)10
size sweep (100/s, 160B-12KB)147 (-2%)379 (+94%)642 (+146%)933 (-16%)247 (+54%)10
replay gateway-gpt-5.4-nano-2000t (1x)113 (-33%)196 (-5%)355 (+9%)821 (+42%)242 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)147 (-33%)156 (-20%)194 (-30%)424 (-14%)302 (-30%)2
ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
vercel[bot]
vercelBot
reviewed
Aug 13, 2026
View reviewed changes
Comment threadpackages/core/src/replay-payload-cache.ts
Outdated


This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues
Comment threadpackages/core/src/workflow.ts
 'workflow.bundle.compile',
 async (span) => {
 const bundle = getCachedWorkflowScript(workflowCode, filename);
 const lookup = getCachedWorkflowScript(workflowLookupCode, filename);
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
Giving the lookup script its own getCachedWorkflowScript call invalidates the bounding invariant documented in script-cache.ts. That cache is keyed by code at the top level and capped at MAX_BUNDLES = 8, and its doc block asserts: "In production this bound is never reached: a deployment is its own process serving exactly one build-time bundle literal ... so there is a single code key for the process lifetime."
workflowLookupCode embeds the workflow name, so each distinct workflow becomes its own top-level code entry. An app with 8 or more workflows now exceeds the cap in production, and MAX_BUNDLES effectively bounds bundle versions + distinct workflow names rather than bundle versions.
No correctness or real cost impact, and I do not think it needs restructuring: eviction only runs when a new code key is inserted and takes from the front, and touchBundle(workflowCode) moves the bundle to MRU at the start of every session, so the bundle is never the eviction candidate — only the one-line lookup scripts churn, and recompiling those is free. But that doc block is now wrong for any app with 8+ workflows and will mislead the next person tuning this cache. Either update it or key the lookup scripts separately from the bundle LRU.
The WorkflowBundleCompileCacheHit attribute correctly reports only the bundle's hit and the comment already calls out why the lookup script's own hit rate is not the signal — that part is right.
Comment threadpackages/core/src/telemetry.ts
}

/** Keeps a parked workflow's ambient trace context aligned with each resume. */
export async function createRefreshableTraceContext() {
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
The returned context is a live proxy over a mutable current, which breaks OTel's immutable-Context contract. Re-parenting a resumed workflow's ambient context is the right goal — spans from a resume should not attach to the trace of the request that first parked the run — but the mechanism has an asymmetry worth documenting.
setValue forwards to current.setValue(...), and OTel contexts are immutable, so that returns a concrete derived context rather than another proxy. The consequence: work whose context is the proxy itself re-parents on refresh(), but anything that derives a context one level deeper (any nested trace() inside the workflow body, which captures the concrete context at creation) does not. So the re-parenting applies to direct children only, and refresh() retroactively changes the parent that already-pending continuations resolve to.
That is defensible for what this is for, but it is subtle enough that the next reader will assume uniform behaviour. A comment on the proxy naming the setValue boundary would save that.
The rest of the span lifecycle checks out: startTraceSpan's ended flag makes the vmTrace.fail in createWorkflowSession's catch a no-op once endVmTrace() has run, so a post-bootstrap throw such as WorkflowNotRegisteredError correctly leaves the VM span OK rather than marking it failed; and void execution.then(replayTrace.end, replayTrace.fail) handles the rejection without competing with the copy returned to the caller.
@NathanColosimo
NathanColosimo
commented
Aug 25, 2026

Copy link


Copy Markdown

ContributorAuthor
Stack repair note: this PR was marked merged by pushing its head into its stale circular base branch; no code from it reached main. The same tracing change now continues against the correct base in #3797.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@NathanColosimo@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
[core] Trace fresh workflow replay phases by NathanColosimo · Pull Request #3523 · vercel/workflow · GitHub
Skip to content
[core] Trace fresh workflow replay phases - #3523
Merged
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Aug 25, 2026
Merged
[core] Trace fresh workflow replay phases#3523
NathanColosimo merged 0 commit into
codex/replay-stream-preparationfrom 
codex/speed-fresh-replay
Conversation
@NathanColosimo
@NathanColosimoNathanColosimo
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
  • add focused spans for VM context creation, bundle compilation, bundle evaluation, input hydration, and replay execution
  • start the replay execution span before invoking user code so synchronous-before-first-await work is included
  • report whether the application bundle actually hit the compiled vm.Script cache
  • avoid a separate streamed-payload-preparation span: preparation intentionally overlaps event loading, while input hydration reports any residual blocking work
  • retain PR [core] Trace fresh workflow replay phases #3523 as the top layer so its deployed benchmark and trace history remain attached
Production trace shape
On 388 fresh-replay samples from the Next/Turbopack behavior run, median phase time was approximately:
  • 3.0 ms — workflow.vm.create_context
  • 33.0 ms — workflow.bundle.compile
  • 91.4 ms — workflow.bundle.evaluate
  • 1.2 ms — workflow.input.hydrate
  • 18.4 ms — workflow.replay.execute
The earlier 6.3 ms workflow.replay.prepare_payloads measurement represented an overlapping wall-time window rather than a blocking phase, so this revision removes that span instead of preserving a misleading trace shape.
This PR is intentionally observability-only against its base. The performance implementation is reviewed in #3547 and #3548.
Stack
  1. [world-vercel] Classify incomplete replay streams as transport failures #3546 — replay stream recovery
  2. [core] Simplify replay payload preparation #3547 — synchronous replay decode
  3. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
  4. [core] Trace fresh workflow replay phases #3523 — replay phase tracing
  5. [core] Prune schema modules from workflow bundles #3550 — prune sandbox dependency graph
  6. [workflow] Load per-source replay bundles #3551 — per-source lazy VM bundles
Validation
  • core typecheck passed
  • full core suite: 2,116 tests passed, with 3 expected failures
  • focused tracing, script-cache, and replay-cache suites: 35 tests passed
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: 2996b0a
The changes in this PR will be included in the next version bump.
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-nextjs-workflow-webpackReadyReadyPreview, v0Aug 25, 2026 11:37pm
example-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-astro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-express-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-fastify-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-hono-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nestjs-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nitro-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-nuxt-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-python-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-sveltekit-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-tanstack-start-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workbench-vite-workflowReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-docsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-swc-playgroundReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-tarballsReadyReadyPreview, v0Aug 25, 2026 11:37pm
workflow-webReadyReadyPreview, v0Aug 25, 2026 11:37pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
❌ Some tests failed
❌ Failed E2E Tests
▲ Vercel Production (244 failed)
astro-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
astro-quickjs (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
example-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K23M5D0GK1235ZCKHS2R1Z | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
example-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
express-node (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
express-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
fastify-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
fastify-quickjs (8 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
hono-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
hono-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nest-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
nextjs-turbopack-node (3 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
nextjs-turbopack-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nextjs-webpack-quickjs (13 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-node (11 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
nitro-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
nuxt-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
nuxt-quickjs (10 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit
python-node (8 failed):
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3 | 🔍 observability
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3 | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1 | 🔍 observability
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z | 🔍 observability
sveltekit-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
sveltekit-quickjs (4 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
tanstack-start-node (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
tanstack-start-quickjs (9 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes Promise.all of disjoint-key writes: every key lands
vite-node (7 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
vite-quickjs (12 failed):
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
🌐 Cross-language Conformance (9 failed)
python (9 failed):
  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K1ZE1VVFB967MCQ01CD795
  • promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3
  • sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3
  • parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1
  • nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD
  • cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
46 flaky tests
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nest)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nest)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (sveltekit)
  • addTenWorkflow (astro)
  • addTenWorkflow (fastify)
  • cancelRun via CLI - cancelling a running workflow (sveltekit)
  • distributedAbortController - manual abort triggers signal (express)
  • distributedAbortController - reconnect to existing controller (example)
  • distributedAbortController - reconnect to existing controller (nest)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (fastify)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nuxt)
  • hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data (nextjs-webpack)
  • hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (nextjs-webpack)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-webpack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (express)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • no startIndex (reads all chunks) (express)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (fastify)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (nextjs-webpack)
  • Promise.all of disjoint-key writes: every key lands (example)
  • Promise.all of disjoint-key writes: every key lands (fastify)
  • Promise.all of disjoint-key writes: every key lands (sveltekit)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (express)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (example)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (express)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (nextjs-webpack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (vite)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nextjs-turbopack)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nitro)
  • sleepWinsRaceWorkflow (tanstack-start)
  • sleepWinsRaceWorkflow (vite)
  • start: initial attributes are seeded on run creation (nextjs-turbopack)
  • start: reserved-prefix initial attributes are seeded with allowReservedAttributes (tanstack-start)
  • step throw of a non-Error value preserves it as cause on the wrapping FatalError (nextjs-turbopack)
  • stepWinsRaceWorkflow (vite)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (hono)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (nest)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (tanstack-start)
  • webhookWorkflow (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (astro)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nest)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nuxt)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (tanstack-start)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
32 infra events
  • cold-start-warmup · suite warmup (python) · at 20:54:40Z · abandoned wrun_41M0K1HSDJ0GME5BYMX59CNX6K · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFQ0GKFWCKBYBX5Q6TF
  • run-pickup-stall · nullByteWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEG00GYMTSZ31K3DAPBQ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1M
  • run-pickup-stall · sleepingWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1K
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NF1Q0GMAYMPP86V96QQ6
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:29Z · abandoned wrun_41M0K1PEDW0GZE4QZ5CFW30S42
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:55:57Z · abandoned wrun_41M0K1Q9SP0GTHYAKV39VBFX6P
  • run-pickup-stall · sleepingWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9SV0GK5F06DGM51605Q
  • run-pickup-stall · nullByteWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9W10GKRAC9NR0E87KFY
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9V00GW4CNMJKV4AXAGS
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:56:48Z · abandoned wrun_41M0K1QDWZ0GXPT291HQ94V9K2
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:58:29Z · abandoned wrun_41M0K1T5SG0GRDC5WEMPDD22EV
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 20:58:29Z · abandoned wrun_41M0K1T1P70GHK6GE0X7XYSHXJ
  • cold-start-warmup · suite warmup (python) · at 20:58:53Z · abandoned wrun_01M0K1SFM5SAP36XBNNF6BAB06 · (+7 more)
  • cold-start-warmup · suite warmup (tanstack-start) · at 20:59:04Z · abandoned wrun_01M0K1WMSYDP0A4AN0EZ2YNE7B
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RSD3H8J0V61PC9W5MK
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RRH9X8HXPZKXSEWHR3
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S01X7VQ0X3SWF6FY11
  • run-pickup-stall · sleepingWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RYSY1H6DS6DJP1AJCJ
  • run-pickup-stall · nullByteWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S3C0M5KZEW6DRBHZ83
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCR8A6B5DQXHZ0XTPEG
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCTEEFAHZFWC64SGN5T
  • run-pickup-stall · nullByteWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCWJRW9Q8395REQRV4X
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCNNJ4CY13PGCBXDGSB
  • run-pickup-stall · sleepingWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCQJ9PV2MH9BM4DJF6A
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0BSDW3JADX703S5W05
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0FWPYY292RKBY8CC9X
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:01:08Z · abandoned wrun_01M0K20T0JG89Z3DB7M0QF7DGB
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBY519CZXPW964N7N4B
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBZXRG7XF8BEYE0CAHE
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:02:08Z · abandoned wrun_01M0K22MKP6JKDYT80XXGFSRPG
E2E Test Summary
Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production33342447424320
✅ 💻 Local Development392205584480
✅ 📦 Local Production392205584480
✅ 🐘 Local Postgres392205584480
✅ 🪟 Windows32000320
❌ 🌐 Cross-language Conformance09132141
✅ vercel-multi-region270027
Total15447253254818248
Details by Category
❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro-node124828
❌ astro-quickjs1211128
❌ example-node123928
❌ example-quickjs1201228
❌ express-node124828
❌ express-quickjs1221028
❌ fastify-node1211128
❌ fastify-quickjs124828
❌ hono-node123928
❌ hono-quickjs1221028
❌ nest-node123928
❌ nest-quickjs1191328
❌ nextjs-turbopack-node15433
❌ nextjs-turbopack-quickjs14893
❌ nextjs-webpack-node15073
❌ nextjs-webpack-quickjs144133
❌ nitro-node1211128
❌ nitro-quickjs1221028
❌ nuxt-node125728
❌ nuxt-quickjs1221028
❌ python-node08152
❌ sveltekit-node14479
❌ sveltekit-quickjs14749
❌ tanstack-start-node123928
❌ tanstack-start-quickjs123928
❌ vite-node125728
❌ vite-quickjs1201228
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node134026
✅ astro-stable-quickjs134026
✅ express-stable-node134026
✅ express-stable-quickjs134026
✅ fastify-stable-node134026
✅ fastify-stable-quickjs134026
✅ hono-stable-node134026
✅ hono-stable-quickjs134026
✅ nest-stable-node134026
✅ nest-stable-quickjs134026
✅ nextjs-turbopack-canary-node141019
✅ nextjs-turbopack-canary-quickjs141019
✅ nextjs-turbopack-stable-node16000
✅ nextjs-turbopack-stable-quickjs16000
✅ nextjs-webpack-canary-node141019
✅ nextjs-webpack-canary-quickjs141019
✅ nextjs-webpack-stable-node16000
✅ nextjs-webpack-stable-quickjs16000
✅ nitro-stable-node134026
✅ nitro-stable-quickjs134026
✅ nuxt-stable-node134026
✅ nuxt-stable-quickjs134026
✅ sveltekit-stable-node15307
✅ sveltekit-stable-quickjs15307
✅ tanstack-start-node134026
✅ tanstack-start-quickjs134026
✅ vite-stable-node134026
✅ vite-stable-quickjs134026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node16000
✅ nextjs-turbopack-quickjs16000
❌ 🌐 Cross-language Conformance
AppPassedFailedSkipped
❌ python09132
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
❌ The benchmark run for 1a35510 failed. See the run logs for details.
commit d74f748 · Thu, 20 Aug 2026 18:02:44 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep248 (-76%) 💚1391 🔴 (+23%) 🔻1424 🔴 (+22%) 🔻1739 🔴 (+42%) 🔻30
TTFSstream277 (+56%) 🔻1424 🔴 (+27%) 🔻1473 🔴 (+31%) 🔻1861 🔴 (+59%) 🔻30
TTFShook + stream374 (-25%) 💚1623 🔴 (+13%)1687 🔴 (+14%)1835 🔴 (+19%) 🔻30
Fan-out TTFSPromise.all(100 steps)557 (+40%) 🔻967 (+8.0%)2100 (+20%) 🔻2302 (+22%) 🔻10
Fan-out TTLSPromise.all(100 steps)4980 (+4.0%)7820 (+27%) 🔻8029 (+28%) 🔻11536 (+60%) 🔻10
STSO1020 steps (inline)135 (+13%)274 (+11%)319 (+6.3%)575 (+14%)1019
WO1020 steps265663 (+12%)265663 (+12%)265663 (+12%)265663 (+12%)1
CRTTfirst chunk (pooled)101 (+9.8%)166 (+26%) 🔻411 (+172%) 🔻1142 (+592%) 🔻28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)134 (+9%)298 (+106%)368 (+39%)1053 (-5%)263 (+79%)10
size sweep (100/s, 160B-12KB)133 (+2%)203 (-8%)482 (-8%)884 (+14%)211 (-23%)10
replay gateway-gpt-5.4-nano-2000t (1x)138 (+14%)162 (+10%)300 (+17%)645 (-8%)461 (-11%)3
replay eve-gpt-5.6-sol-2000t (1x)314 (+146%)184 (+21%)351 (+38%)797 (-5%)696 (+39%)2
replay eve-gpt-5.6-sol-2000t (2x)147 (+20%)244 (+16%)397 (+23%)675 (-80%)544 (-10%)3
📜 Previous results (2)
50faa3b
Wed, 19 Aug 2026 17:28:21 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1335 (+365%) 🔻1423 🔴 (+22%) 🔻1441 🔴 (+7.6%)1534 🔴 (-1.9%)30
TTFSstream1342 (+610%) 🔻1410 🔴 (+26%) 🔻1430 🔴 (+25%) 🔻1576 🔴 (+35%) 🔻30
TTFShook + stream1514 (+16%) 🔻1693 🔴 (+23%) 🔻1705 🔴 (+21%) 🔻2082 🔴 (+21%) 🔻30
Fan-out TTFSPromise.all(100 steps)1088 (-1.5%)2594 (+13%)2614 (-5.5%)2987 (+7.8%)10
Fan-out TTLSPromise.all(100 steps)5278 (±0%)6182 (-7.1%)6444 (-6.4%)8094 (+2.7%)10
STSO1020 steps (inline)120 (-9.1%)224 (-15%)249 (-18%) 💚336 (-33%) 💚1019
WO1020 steps216112 (-13%)216112 (-13%)216112 (-13%)216112 (-13%)1
CRTTfirst chunk (pooled)80 (-4.8%)126 (-0.8%)269 (-15%)567 (-9.6%)28
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)99 (-12%)139 (-16%)221 (-5%)430 (-8%)109 (-32%)10
size sweep (100/s, 160B-12KB)108 (±0%)128 (-17%)220 (-22%)604 (-11%)104 (-36%)10
replay gateway-gpt-5.4-nano-2000t (1x)93 (-17%)159 (-43%)268 (-63%)424 (-72%)368 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)333 (+114%)129 (-8%)239 (+24%)447 (-29%)302 (-38%)2
replay eve-gpt-5.6-sol-2000t (2x)108 (-79%)344 (-15%)553 (-80%)751 (-82%)509 (-16%)3
50faa3b
Tue, 18 Aug 2026 23:23:51 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1377 (+29%) 🔻1439 🔴 (+20%) 🔻1460 🔴 (+15%) 🔻1585 🔴 (-0.9%)30
TTFSstream225 (-27%) 💚1436 🔴 (+29%) 🔻1458 🔴 (+24%) 🔻1865 🔴 (+51%) 🔻30
TTFShook + stream346 (-74%) 💚1711 🔴 (+14%)1742 🔴 (+12%)1816 🔴 (-7.3%)30
Fan-out TTFSPromise.all(100 steps)1521 (+25%) 🔻3074 (+29%) 🔻3100 (+25%) 🔻3101 (+15%)10
Fan-out TTLSPromise.all(100 steps)7331 (+37%) 🔻27947 (+251%) 🔻310173 (+3739%) 🔻313908 (+3301%) 🔻10
CRTTfirst chunk (pooled)108 (-7.7%)164 (-19%) 💚213 (-34%) 💚416 (-34%) 💚25
Streams
ScenarioCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)138 (-15%)197 (-45%)272 (-60%)1221 (-23%)181 (-52%)10
size sweep (100/s, 160B-12KB)147 (-2%)379 (+94%)642 (+146%)933 (-16%)247 (+54%)10
replay gateway-gpt-5.4-nano-2000t (1x)113 (-33%)196 (-5%)355 (+9%)821 (+42%)242 (-39%)3
replay eve-gpt-5.6-sol-2000t (1x)147 (-33%)156 (-20%)194 (-30%)424 (-14%)302 (-30%)2
ℹ️ Metric definitions & methodology
Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
vercel[bot]
vercelBot
reviewed
Aug 13, 2026
View reviewed changes
Comment threadpackages/core/src/replay-payload-cache.ts
Outdated


This was referenced Aug 14, 2026
 Merged

 Merged

 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 21, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues
Comment threadpackages/core/src/workflow.ts
 'workflow.bundle.compile',
 async (span) => {
 const bundle = getCachedWorkflowScript(workflowCode, filename);
 const lookup = getCachedWorkflowScript(workflowLookupCode, filename);
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
Giving the lookup script its own getCachedWorkflowScript call invalidates the bounding invariant documented in script-cache.ts. That cache is keyed by code at the top level and capped at MAX_BUNDLES = 8, and its doc block asserts: "In production this bound is never reached: a deployment is its own process serving exactly one build-time bundle literal ... so there is a single code key for the process lifetime."
workflowLookupCode embeds the workflow name, so each distinct workflow becomes its own top-level code entry. An app with 8 or more workflows now exceeds the cap in production, and MAX_BUNDLES effectively bounds bundle versions + distinct workflow names rather than bundle versions.
No correctness or real cost impact, and I do not think it needs restructuring: eviction only runs when a new code key is inserted and takes from the front, and touchBundle(workflowCode) moves the bundle to MRU at the start of every session, so the bundle is never the eviction candidate — only the one-line lookup scripts churn, and recompiling those is free. But that doc block is now wrong for any app with 8+ workflows and will mislead the next person tuning this cache. Either update it or key the lookup scripts separately from the bundle LRU.
The WorkflowBundleCompileCacheHit attribute correctly reports only the bundle's hit and the comment already calls out why the lookup script's own hit rate is not the signal — that part is right.
Comment threadpackages/core/src/telemetry.ts
}

/** Keeps a parked workflow's ambient trace context aligned with each resume. */
export async function createRefreshableTraceContext() {
@VaguelySeriousVaguelySeriousAug 21, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note
The returned context is a live proxy over a mutable current, which breaks OTel's immutable-Context contract. Re-parenting a resumed workflow's ambient context is the right goal — spans from a resume should not attach to the trace of the request that first parked the run — but the mechanism has an asymmetry worth documenting.
setValue forwards to current.setValue(...), and OTel contexts are immutable, so that returns a concrete derived context rather than another proxy. The consequence: work whose context is the proxy itself re-parents on refresh(), but anything that derives a context one level deeper (any nested trace() inside the workflow body, which captures the concrete context at creation) does not. So the re-parenting applies to direct children only, and refresh() retroactively changes the parent that already-pending continuations resolve to.
That is defensible for what this is for, but it is subtle enough that the next reader will assume uniform behaviour. A comment on the proxy naming the setValue boundary would save that.
The rest of the span lifecycle checks out: startTraceSpan's ended flag makes the vmTrace.fail in createWorkflowSession's catch a no-op once endVmTrace() has run, so a post-bootstrap throw such as WorkflowNotRegisteredError correctly leaves the VM span OK rather than marking it failed; and void execution.then(replayTrace.end, replayTrace.fail) handles the rejection without competing with the copy returned to the caller.
@NathanColosimo
NathanColosimo
commented
Aug 25, 2026

Copy link


Copy Markdown

ContributorAuthor
Stack repair note: this PR was marked merged by pushing its head into its stale circular base branch; no code from it reached main. The same tracing change now continues against the correct base in #3797.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes
@msullivanmsullivanAwaiting requested review from msullivan
@fantixfantixAwaiting requested review from fantix

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@NathanColosimo@VaguelySerious