[core] Record step execution duration metric - #3526

Merged
shalabhc merged 2 commits into
mainfrom
identify-datadog-span-duration-metrics
Aug 17, 2026
Merged

[core] Record step execution duration metric#3526
shalabhc merged 2 commits into
mainfrom
identify-datadog-span-duration-metrics

Conversation

@shalabhc

@shalabhcshalabhc commented Aug 13, 2026•
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • record workflow.step.execute.duration as an OpenTelemetry histogram around the inner user-code step span
  • rely on resource service.name for the service dimension and add only a bounded status attribute
  • cover the histogram name, unit, and attributes with a focused unit test

Validation

  • pnpm exec vitest run packages/core/src/telemetry-metrics.test.ts
  • pnpm --filter @workflow/core typecheck

@changeset-bot

changeset-botBot commented Aug 13, 2026•
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: f01d0bb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Aug 13, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 17, 2026 7:04pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 17, 2026 7:04pm
example-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-astro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-express-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-fastify-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-hono-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nestjs-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nitro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nuxt-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-python-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-sveltekit-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-vite-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workflow-docsReadyReadyPreview, v0Aug 17, 2026 7:04pm
workflow-swc-playgroundReadyReadyPreviewAug 17, 2026 7:04pm
workflow-tarballsReadyReadyPreviewAug 17, 2026 7:04pm
workflow-webReadyReadyPreviewAug 17, 2026 7:04pm

@github-actions

github-actionsBot commented Aug 13, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit f01d0bb · Mon, 17 Aug 2026 19:22:06 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep251 (-67%) 💚1415 🔴 (+26%) 🔻1502 🔴 (+25%) 🔻1821 🔴 (+18%) 🔻30
TTFSstream249 (-75%) 💚1406 🔴 (+31%) 🔻1422 🔴 (+31%) 🔻1529 🔴 (+33%) 🔻30
TTFShook + stream487 (-62%) 💚1741 🔴 (+22%) 🔻1834 🔴 (+23%) 🔻2044 🔴 (-3.6%)30
Fan-out TTFSPromise.all(100 steps)1356 (+13%)2767 (+22%) 🔻3169 (+33%) 🔻3171 (+20%) 🔻10
Fan-out TTLSPromise.all(100 steps)6422 (+5.8%)10032 (+21%) 🔻10054 (+19%) 🔻11482 (+23%) 🔻10
STSO1020 steps (inline)136 (+12%)263 (+12%)298 (+11%)456 (+6.5%)1019
WO1020 steps254953 (+15%)254953 (+15%)254953 (+15%)254953 (+15%)1
CRTTfirst chunk (pooled)97 (+10%)174 (+18%) 🔻508 (+189%) 🔻848 (+68%) 🔻28

Streams

Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)99.8 (-1%)5 (±0%)5 (±0%)145 (+38%)323 (+134%)537 (+214%)1402 (+454%)283 (+134%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.1 (±0%)334 (±0%)330 (±0%)140 (+16%)334 (+139%)550 (+151%)1226 (+111%)706 (+423%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)89.3 (±0%)16.2 (±0%)16.2 (±0%)120 (-10%)172 (+19%)244 (+27%)769 (+128%)682 (+134%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.6 (±0%)355 (±0%)354 (±0%)116 (-21%)185 (+33%)754 (+270%)3792 (+950%)2375 (+629%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (-1%)710 (±0%)709 (-1%)236 (+119%)317 (+28%)562 (+47%)1227 (+41%)786 (+13%)3
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 222159ms → this run 254711ms (Δ +32552ms, +15%)

 100-150 ms ┃ main 22 this 2 -20
150-200 ms ███┃████████████ main 386 this 92 -294
200-250 ms ███████████████████░░░░┃ main 450 this 566 +116
250-300 ms █████░░░░░┃ main 113 this 260 +147
300-350 ms █┃ main 29 this 54 +25
350-400 ms ┃ main 2 this 27 +25
400-450 ms ┃ main 7 this 7 +0
450-500 ms ┃ main 5 this 5 +0
500-550 ms ┃ main 2 this 4 +2
550-600 ms ┃ main 0 this 1 +1
600-650 ms ┃ main 2 this 0 -2
800-850 ms ┃ main 1 this 0 -1
3950-4000 ms ┃ main 0 this 1 +1
📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▄▁▁·· 234 (+112%) 152 (+46%) 537 (+214%) 1402 (+454%) 3000
sweep ······▂█▇▂▁·· 252.2 (+99%) 227 (+112%) 550 (+151%) 1226 (+111%) 3000
gw 1x ·····▁▃█▂▁··· 154.1 (+36%) 132 (+29%) 244 (+27%) 769 (+128%) 5295
eve 1x ·····▁▃█▂▁▁▁· 260.9 (+120%) 129 (+28%) 754 (+270%) 3792 (+950%) 5186
eve 2x ·····▁▁█▅▁▁·· 218.8 (+18%) 172 (+15%) 562 (+47%) 1227 (+41%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control █▇▆▄▃▄▂█▃▁ 170–295ms
sweep ▁▁█▇▆▆▇▄▅▄ 184–306ms
gw 1x █▁▂▄▂▁▁▁▆▂ 130–222ms
eve 1x ▁▁▁█▁▁▁▁▁▁ 127–1251ms
eve 2x ▄▄▁▁▃▅██▅▁ 149–311ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep ▅█▃▂▃▁█ 250–254ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▂▂▅▄▂▅▄█▁▁ 34–78ms
sweep ▃▇█▅▅▃▁▂▅▃ 38–91ms
gw 1x █▃▂▄▁▁▅▄▇▃ 41–52ms
eve 1x ▂▂▁█▂▂▂▂▁▂ 25–74ms
eve 2x ▅▅▁▁▄█▃▅▃▄ 24–40ms
📜 Previous results (1)

0287155

Thu, 13 Aug 2026 20:57:01 GMT · run logs

vercel / nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep262 (+42%) 🔻1391 🔴 (+26%) 🔻1418 🔴 (+17%) 🔻1486 🔴 (-6.6%)30
TTFSstream234 (-1.3%)1403 🔴 (+27%) 🔻1425 🔴 (+25%) 🔻1468 🔴 (+22%) 🔻30
TTFShook + stream442 (+15%) 🔻1678 🔴 (+22%) 🔻1746 🔴 (+25%) 🔻1835 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9107 (+1.0%)10476 (+1.2%)10715 (-11%)11193 (-96%) 💚10
Fan-out TTLSPromise.all(100 steps)17466 (-1.6%)19157 (-6.6%)19813 (-71%) 💚21056 (-93%) 💚10
STSO1020 steps (inline)140 (-7.3%)189 (-16%) 💚220 (-15%)340 (-35%) 💚1019
WO1020 steps188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚1
SLstream latency98 (+4.3%)381 🔴 (+154%) 🔻462 🔴 (+187%) 🔻5193 🔴 (+1367%) 🔻30
SOstream overhead (text)137 (-13%)265 🔴 (-10%)486 (+12%)4163 🔴 (+177%) 🔻30
SOstream overhead (structured)127 (+7.6%)344 🔴 (-15%) 💚366 (-34%) 💚812 (-68%) 💚30
ℹ️ Metric definitions & methodology

Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.

The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@github-actions

github-actionsBot commented Aug 13, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

✅ All tests passed

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

  • maxRetries=0 disables retries (nextjs-webpack)
  • plainModuleDoneHook resumed via plain API route (o2flow shape) (nextjs-webpack)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 19:04:53Z · abandoned wrun_01M08HS2Z9W5P8TX6GVABJVAWX

E2E Test Summary

Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category

✅ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600

✅ 🌐 Cross-language Conformance

AppPassedFailedSkipped
✅ python90128

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@github-actions

github-actionsBot commented Aug 13, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0

Full trace: world-sim.txt

@shalabhc
shalabhc marked this pull request as ready for review August 13, 2026 20:59
@shalabhc
shalabhc requested a review from a team as a code ownerAugust 13, 2026 20:59
karthikscale3
karthikscale3 approved these changes Aug 13, 2026
View reviewed changes

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

vercelBotand others added 2 commits August 17, 2026 18:59
@vercel@shalabhc
[core] Record step execution duration metric
64fccac
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@vercel@shalabhc
[core] Add changeset for the step execution duration metric
f01d0bb
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@shalabhc
shalabhcforce-pushed the identify-datadog-span-duration-metrics branch from 0287155 to f01d0bbCompareAugust 17, 2026 19:00
@shalabhc
shalabhc merged commit ac51658 into mainAug 17, 2026
301 of 303 checks passed
@shalabhc
shalabhc deleted the identify-datadog-span-duration-metrics branch August 17, 2026 21:49
@github-actions

github-actionsBot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

No backport to stable for ac51658 (AI decision).

This commit adds a new observability capability — a workflow.step.execute.duration OpenTelemetry histogram plus a new exported recordStepExecutionDuration API in @workflow/core — rather than fixing a defect on the maintenance line. New metrics and new exported API surface are feature work, so they stay on main regardless of how small or self-contained the change is.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

ac51658d39649e7d8e290924c3bf7b2c7df78b00

This was referenced Aug 17, 2026
Closed
Open
Closed
Open
Open
Merged
VaguelySerious added a commit that referenced this pull request Aug 21, 2026
@VaguelySerious@claude
docs: catch v5 up with main, and with the sealed log
a08bced
Sweep of everything that landed on main since the last content refresh,
plus the spec-7 sealed log from #3634, which is not merged but is the
default for v5.
Spec 7 changes what a World declares, so the upgrade guide's Spec
versions section is rewritten around `mintedSpecVersion()`. The accepted
range widens back to two versions, because a deployment with
`WORKFLOW_SEALED_LOG=0` stamps the one below, and both `SPEC_VERSION_CURRENT`
and `SPEC_VERSION_SUPPORTS_SLOT_IDENTITY` become literals by another name
for the purpose of declaring it: neither follows the setting. The new
section says what a sealed log costs an implementer, which for most is
nothing — a World allocating each position at the commit cannot leave a
hole, so it never seals and never writes a `noop`. The obligation belongs
to a World that pre-assigns positions, and the half that binds everyone
is the reader's.
`WORKFLOW_SEALED_LOG` is documented in runtime-tuning, taken verbatim
from #3634 so whichever lands second is a no-op merge. The deep
reference sections that PR writes, in event-sourcing and the World
allocation contract, are left to it.
Two optional World members shipped after the guide was written and were
missing everywhere:
- `events.createBatch` — in the World reference (interface and contract
note), the upgrade guide's optional surface, and the skill. Implementing
it is the declaration, so the note leads with atomicity per attempt and
what a World must reject.
- `runs.waitForTerminalStatus` — same three places. It is also what makes
`await run.returnValue` a long poll rather than an interval, which is
user-facing, so What's new gets it too.
What's new also gains the batched suspension write and the pre-assigned
positions behind it, both under faster-and-cheaper runs where they belong:
neither has any API surface.
Not addressed: `workflow.step.execute.duration`, the OpenTelemetry
histogram added in #3526, has nowhere to go. The docs have no metrics
reference at all, and inventing one for a single metric is a bigger call
than this sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySeriousVaguelySerious mentioned this pull request Aug 21, 2026
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@shalabhc@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
[core] Record step execution duration metric - #3526
Merged
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Aug 17, 2026
Merged
[core] Record step execution duration metric#3526
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Conversation
@shalabhc
@shalabhcshalabhc
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Summary
  • record workflow.step.execute.duration as an OpenTelemetry histogram around the inner user-code step span
  • rely on resource service.name for the service dimension and add only a bounded status attribute
  • cover the histogram name, unit, and attributes with a focused unit test
Validation
  • pnpm exec vitest run packages/core/src/telemetry-metrics.test.ts
  • pnpm --filter @workflow/core typecheck
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: f01d0bb
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 17, 2026 7:04pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 17, 2026 7:04pm
example-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-astro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-express-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-fastify-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-hono-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nestjs-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nitro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nuxt-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-python-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-sveltekit-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-vite-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workflow-docsReadyReadyPreview, v0Aug 17, 2026 7:04pm
workflow-swc-playgroundReadyReadyPreviewAug 17, 2026 7:04pm
workflow-tarballsReadyReadyPreviewAug 17, 2026 7:04pm
workflow-webReadyReadyPreviewAug 17, 2026 7:04pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit f01d0bb · Mon, 17 Aug 2026 19:22:06 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep251 (-67%) 💚1415 🔴 (+26%) 🔻1502 🔴 (+25%) 🔻1821 🔴 (+18%) 🔻30
TTFSstream249 (-75%) 💚1406 🔴 (+31%) 🔻1422 🔴 (+31%) 🔻1529 🔴 (+33%) 🔻30
TTFShook + stream487 (-62%) 💚1741 🔴 (+22%) 🔻1834 🔴 (+23%) 🔻2044 🔴 (-3.6%)30
Fan-out TTFSPromise.all(100 steps)1356 (+13%)2767 (+22%) 🔻3169 (+33%) 🔻3171 (+20%) 🔻10
Fan-out TTLSPromise.all(100 steps)6422 (+5.8%)10032 (+21%) 🔻10054 (+19%) 🔻11482 (+23%) 🔻10
STSO1020 steps (inline)136 (+12%)263 (+12%)298 (+11%)456 (+6.5%)1019
WO1020 steps254953 (+15%)254953 (+15%)254953 (+15%)254953 (+15%)1
CRTTfirst chunk (pooled)97 (+10%)174 (+18%) 🔻508 (+189%) 🔻848 (+68%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)99.8 (-1%)5 (±0%)5 (±0%)145 (+38%)323 (+134%)537 (+214%)1402 (+454%)283 (+134%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.1 (±0%)334 (±0%)330 (±0%)140 (+16%)334 (+139%)550 (+151%)1226 (+111%)706 (+423%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)89.3 (±0%)16.2 (±0%)16.2 (±0%)120 (-10%)172 (+19%)244 (+27%)769 (+128%)682 (+134%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.6 (±0%)355 (±0%)354 (±0%)116 (-21%)185 (+33%)754 (+270%)3792 (+950%)2375 (+629%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (-1%)710 (±0%)709 (-1%)236 (+119%)317 (+28%)562 (+47%)1227 (+41%)786 (+13%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 222159ms → this run 254711ms (Δ +32552ms, +15%)
 100-150 ms ┃ main 22 this 2 -20
150-200 ms ███┃████████████ main 386 this 92 -294
200-250 ms ███████████████████░░░░┃ main 450 this 566 +116
250-300 ms █████░░░░░┃ main 113 this 260 +147
300-350 ms █┃ main 29 this 54 +25
350-400 ms ┃ main 2 this 27 +25
400-450 ms ┃ main 7 this 7 +0
450-500 ms ┃ main 5 this 5 +0
500-550 ms ┃ main 2 this 4 +2
550-600 ms ┃ main 0 this 1 +1
600-650 ms ┃ main 2 this 0 -2
800-850 ms ┃ main 1 this 0 -1
3950-4000 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▄▁▁·· 234 (+112%) 152 (+46%) 537 (+214%) 1402 (+454%) 3000
sweep ······▂█▇▂▁·· 252.2 (+99%) 227 (+112%) 550 (+151%) 1226 (+111%) 3000
gw 1x ·····▁▃█▂▁··· 154.1 (+36%) 132 (+29%) 244 (+27%) 769 (+128%) 5295
eve 1x ·····▁▃█▂▁▁▁· 260.9 (+120%) 129 (+28%) 754 (+270%) 3792 (+950%) 5186
eve 2x ·····▁▁█▅▁▁·· 218.8 (+18%) 172 (+15%) 562 (+47%) 1227 (+41%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control █▇▆▄▃▄▂█▃▁ 170–295ms
sweep ▁▁█▇▆▆▇▄▅▄ 184–306ms
gw 1x █▁▂▄▂▁▁▁▆▂ 130–222ms
eve 1x ▁▁▁█▁▁▁▁▁▁ 127–1251ms
eve 2x ▄▄▁▁▃▅██▅▁ 149–311ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▅█▃▂▃▁█ 250–254ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂▅▄▂▅▄█▁▁ 34–78ms
sweep ▃▇█▅▅▃▁▂▅▃ 38–91ms
gw 1x █▃▂▄▁▁▅▄▇▃ 41–52ms
eve 1x ▂▂▁█▂▂▂▂▁▂ 25–74ms
eve 2x ▅▅▁▁▄█▃▅▃▄ 24–40ms

📜 Previous results (1)
0287155
Thu, 13 Aug 2026 20:57:01 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep262 (+42%) 🔻1391 🔴 (+26%) 🔻1418 🔴 (+17%) 🔻1486 🔴 (-6.6%)30
TTFSstream234 (-1.3%)1403 🔴 (+27%) 🔻1425 🔴 (+25%) 🔻1468 🔴 (+22%) 🔻30
TTFShook + stream442 (+15%) 🔻1678 🔴 (+22%) 🔻1746 🔴 (+25%) 🔻1835 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9107 (+1.0%)10476 (+1.2%)10715 (-11%)11193 (-96%) 💚10
Fan-out TTLSPromise.all(100 steps)17466 (-1.6%)19157 (-6.6%)19813 (-71%) 💚21056 (-93%) 💚10
STSO1020 steps (inline)140 (-7.3%)189 (-16%) 💚220 (-15%)340 (-35%) 💚1019
WO1020 steps188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚1
SLstream latency98 (+4.3%)381 🔴 (+154%) 🔻462 🔴 (+187%) 🔻5193 🔴 (+1367%) 🔻30
SOstream overhead (text)137 (-13%)265 🔴 (-10%)486 (+12%)4163 🔴 (+177%) 🔻30
SOstream overhead (structured)127 (+7.6%)344 🔴 (-15%) 💚366 (-34%) 💚812 (-68%) 💚30
ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • maxRetries=0 disables retries (nextjs-webpack)
  • plainModuleDoneHook resumed via plain API route (o2flow shape) (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 19:04:53Z · abandoned wrun_01M08HS2Z9W5P8TX6GVABJVAWX
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@shalabhc
shalabhc
marked this pull request as ready for review
August 13, 2026 20:59
@shalabhc
shalabhc
requested a review
from a team
as a code ownerAugust 13, 2026 20:59
karthikscale3
karthikscale3
approved these changes
Aug 13, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
vercelBotand others
added 2 commits
August 17, 2026 18:59
@vercel@shalabhc
[core] Record step execution duration metric
64fccac
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@vercel@shalabhc
[core] Add changeset for the step execution duration metric
f01d0bb
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@shalabhc
shalabhcforce-pushed
the
identify-datadog-span-duration-metrics
branch
from
0287155 to
f01d0bbCompareAugust 17, 2026 19:00
@shalabhc
shalabhc
merged commit ac51658
into
mainAug 17, 2026

301 of 303 checks passed

@shalabhc
shalabhc
deleted the

identify-datadog-span-duration-metrics

branch
August 17, 2026 21:49
@github-actions
github-actionsBot
commented
Aug 17, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for ac51658 (AI decision).
This commit adds a new observability capability — a workflow.step.execute.duration OpenTelemetry histogram plus a new exported recordStepExecutionDuration API in @workflow/core — rather than fixing a defect on the maintenance line. New metrics and new exported API surface are feature work, so they stay on main regardless of how small or self-contained the change is.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
ac51658d39649e7d8e290924c3bf7b2c7df78b00


This was referenced Aug 17, 2026
 Closed

 Open

 Closed

 Open

 Open

 Merged

VaguelySerious added a commit
that referenced
this pull request
Aug 21, 2026
@VaguelySerious@claude
docs: catch v5 up with main, and with the sealed log
a08bced
Sweep of everything that landed on main since the last content refresh,
plus the spec-7 sealed log from #3634, which is not merged but is the
default for v5.
Spec 7 changes what a World declares, so the upgrade guide's Spec
versions section is rewritten around `mintedSpecVersion()`. The accepted
range widens back to two versions, because a deployment with
`WORKFLOW_SEALED_LOG=0` stamps the one below, and both `SPEC_VERSION_CURRENT`
and `SPEC_VERSION_SUPPORTS_SLOT_IDENTITY` become literals by another name
for the purpose of declaring it: neither follows the setting. The new
section says what a sealed log costs an implementer, which for most is
nothing — a World allocating each position at the commit cannot leave a
hole, so it never seals and never writes a `noop`. The obligation belongs
to a World that pre-assigns positions, and the half that binds everyone
is the reader's.
`WORKFLOW_SEALED_LOG` is documented in runtime-tuning, taken verbatim
from #3634 so whichever lands second is a no-op merge. The deep
reference sections that PR writes, in event-sourcing and the World
allocation contract, are left to it.
Two optional World members shipped after the guide was written and were
missing everywhere:
- `events.createBatch` — in the World reference (interface and contract
note), the upgrade guide's optional surface, and the skill. Implementing
it is the declaration, so the note leads with atomicity per attempt and
what a World must reject.
- `runs.waitForTerminalStatus` — same three places. It is also what makes
`await run.returnValue` a long poll rather than an interval, which is
user-facing, so What's new gets it too.
What's new also gains the batched suspension write and the pre-assigned
positions behind it, both under faster-and-cheaper runs where they belong:
neither has any API surface.
Not addressed: `workflow.step.execute.duration`, the OpenTelemetry
histogram added in #3526, has nowhere to go. The docs have no metrics
reference at all, and inventing one for a single metric is a bigger call
than this sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@shalabhc@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[core] Record step execution duration metric - #3526
Merged
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Aug 17, 2026
Merged
[core] Record step execution duration metric#3526
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Conversation
@shalabhc
@shalabhcshalabhc
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Summary
  • record workflow.step.execute.duration as an OpenTelemetry histogram around the inner user-code step span
  • rely on resource service.name for the service dimension and add only a bounded status attribute
  • cover the histogram name, unit, and attributes with a focused unit test
Validation
  • pnpm exec vitest run packages/core/src/telemetry-metrics.test.ts
  • pnpm --filter @workflow/core typecheck
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: f01d0bb
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 17, 2026 7:04pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 17, 2026 7:04pm
example-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-astro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-express-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-fastify-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-hono-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nestjs-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nitro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nuxt-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-python-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-sveltekit-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-vite-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workflow-docsReadyReadyPreview, v0Aug 17, 2026 7:04pm
workflow-swc-playgroundReadyReadyPreviewAug 17, 2026 7:04pm
workflow-tarballsReadyReadyPreviewAug 17, 2026 7:04pm
workflow-webReadyReadyPreviewAug 17, 2026 7:04pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit f01d0bb · Mon, 17 Aug 2026 19:22:06 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep251 (-67%) 💚1415 🔴 (+26%) 🔻1502 🔴 (+25%) 🔻1821 🔴 (+18%) 🔻30
TTFSstream249 (-75%) 💚1406 🔴 (+31%) 🔻1422 🔴 (+31%) 🔻1529 🔴 (+33%) 🔻30
TTFShook + stream487 (-62%) 💚1741 🔴 (+22%) 🔻1834 🔴 (+23%) 🔻2044 🔴 (-3.6%)30
Fan-out TTFSPromise.all(100 steps)1356 (+13%)2767 (+22%) 🔻3169 (+33%) 🔻3171 (+20%) 🔻10
Fan-out TTLSPromise.all(100 steps)6422 (+5.8%)10032 (+21%) 🔻10054 (+19%) 🔻11482 (+23%) 🔻10
STSO1020 steps (inline)136 (+12%)263 (+12%)298 (+11%)456 (+6.5%)1019
WO1020 steps254953 (+15%)254953 (+15%)254953 (+15%)254953 (+15%)1
CRTTfirst chunk (pooled)97 (+10%)174 (+18%) 🔻508 (+189%) 🔻848 (+68%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)99.8 (-1%)5 (±0%)5 (±0%)145 (+38%)323 (+134%)537 (+214%)1402 (+454%)283 (+134%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.1 (±0%)334 (±0%)330 (±0%)140 (+16%)334 (+139%)550 (+151%)1226 (+111%)706 (+423%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)89.3 (±0%)16.2 (±0%)16.2 (±0%)120 (-10%)172 (+19%)244 (+27%)769 (+128%)682 (+134%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.6 (±0%)355 (±0%)354 (±0%)116 (-21%)185 (+33%)754 (+270%)3792 (+950%)2375 (+629%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (-1%)710 (±0%)709 (-1%)236 (+119%)317 (+28%)562 (+47%)1227 (+41%)786 (+13%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 222159ms → this run 254711ms (Δ +32552ms, +15%)
 100-150 ms ┃ main 22 this 2 -20
150-200 ms ███┃████████████ main 386 this 92 -294
200-250 ms ███████████████████░░░░┃ main 450 this 566 +116
250-300 ms █████░░░░░┃ main 113 this 260 +147
300-350 ms █┃ main 29 this 54 +25
350-400 ms ┃ main 2 this 27 +25
400-450 ms ┃ main 7 this 7 +0
450-500 ms ┃ main 5 this 5 +0
500-550 ms ┃ main 2 this 4 +2
550-600 ms ┃ main 0 this 1 +1
600-650 ms ┃ main 2 this 0 -2
800-850 ms ┃ main 1 this 0 -1
3950-4000 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▄▁▁·· 234 (+112%) 152 (+46%) 537 (+214%) 1402 (+454%) 3000
sweep ······▂█▇▂▁·· 252.2 (+99%) 227 (+112%) 550 (+151%) 1226 (+111%) 3000
gw 1x ·····▁▃█▂▁··· 154.1 (+36%) 132 (+29%) 244 (+27%) 769 (+128%) 5295
eve 1x ·····▁▃█▂▁▁▁· 260.9 (+120%) 129 (+28%) 754 (+270%) 3792 (+950%) 5186
eve 2x ·····▁▁█▅▁▁·· 218.8 (+18%) 172 (+15%) 562 (+47%) 1227 (+41%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control █▇▆▄▃▄▂█▃▁ 170–295ms
sweep ▁▁█▇▆▆▇▄▅▄ 184–306ms
gw 1x █▁▂▄▂▁▁▁▆▂ 130–222ms
eve 1x ▁▁▁█▁▁▁▁▁▁ 127–1251ms
eve 2x ▄▄▁▁▃▅██▅▁ 149–311ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▅█▃▂▃▁█ 250–254ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂▅▄▂▅▄█▁▁ 34–78ms
sweep ▃▇█▅▅▃▁▂▅▃ 38–91ms
gw 1x █▃▂▄▁▁▅▄▇▃ 41–52ms
eve 1x ▂▂▁█▂▂▂▂▁▂ 25–74ms
eve 2x ▅▅▁▁▄█▃▅▃▄ 24–40ms

📜 Previous results (1)
0287155
Thu, 13 Aug 2026 20:57:01 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep262 (+42%) 🔻1391 🔴 (+26%) 🔻1418 🔴 (+17%) 🔻1486 🔴 (-6.6%)30
TTFSstream234 (-1.3%)1403 🔴 (+27%) 🔻1425 🔴 (+25%) 🔻1468 🔴 (+22%) 🔻30
TTFShook + stream442 (+15%) 🔻1678 🔴 (+22%) 🔻1746 🔴 (+25%) 🔻1835 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9107 (+1.0%)10476 (+1.2%)10715 (-11%)11193 (-96%) 💚10
Fan-out TTLSPromise.all(100 steps)17466 (-1.6%)19157 (-6.6%)19813 (-71%) 💚21056 (-93%) 💚10
STSO1020 steps (inline)140 (-7.3%)189 (-16%) 💚220 (-15%)340 (-35%) 💚1019
WO1020 steps188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚1
SLstream latency98 (+4.3%)381 🔴 (+154%) 🔻462 🔴 (+187%) 🔻5193 🔴 (+1367%) 🔻30
SOstream overhead (text)137 (-13%)265 🔴 (-10%)486 (+12%)4163 🔴 (+177%) 🔻30
SOstream overhead (structured)127 (+7.6%)344 🔴 (-15%) 💚366 (-34%) 💚812 (-68%) 💚30
ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • maxRetries=0 disables retries (nextjs-webpack)
  • plainModuleDoneHook resumed via plain API route (o2flow shape) (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 19:04:53Z · abandoned wrun_01M08HS2Z9W5P8TX6GVABJVAWX
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@shalabhc
shalabhc
marked this pull request as ready for review
August 13, 2026 20:59
@shalabhc
shalabhc
requested a review
from a team
as a code ownerAugust 13, 2026 20:59
karthikscale3
karthikscale3
approved these changes
Aug 13, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
vercelBotand others
added 2 commits
August 17, 2026 18:59
@vercel@shalabhc
[core] Record step execution duration metric
64fccac
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@vercel@shalabhc
[core] Add changeset for the step execution duration metric
f01d0bb
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@shalabhc
shalabhcforce-pushed
the
identify-datadog-span-duration-metrics
branch
from
0287155 to
f01d0bbCompareAugust 17, 2026 19:00
@shalabhc
shalabhc
merged commit ac51658
into
mainAug 17, 2026

301 of 303 checks passed

@shalabhc
shalabhc
deleted the

identify-datadog-span-duration-metrics

branch
August 17, 2026 21:49
@github-actions
github-actionsBot
commented
Aug 17, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for ac51658 (AI decision).
This commit adds a new observability capability — a workflow.step.execute.duration OpenTelemetry histogram plus a new exported recordStepExecutionDuration API in @workflow/core — rather than fixing a defect on the maintenance line. New metrics and new exported API surface are feature work, so they stay on main regardless of how small or self-contained the change is.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
ac51658d39649e7d8e290924c3bf7b2c7df78b00


This was referenced Aug 17, 2026
 Closed

 Open

 Closed

 Open

 Open

 Merged

VaguelySerious added a commit
that referenced
this pull request
Aug 21, 2026
@VaguelySerious@claude
docs: catch v5 up with main, and with the sealed log
a08bced
Sweep of everything that landed on main since the last content refresh,
plus the spec-7 sealed log from #3634, which is not merged but is the
default for v5.
Spec 7 changes what a World declares, so the upgrade guide's Spec
versions section is rewritten around `mintedSpecVersion()`. The accepted
range widens back to two versions, because a deployment with
`WORKFLOW_SEALED_LOG=0` stamps the one below, and both `SPEC_VERSION_CURRENT`
and `SPEC_VERSION_SUPPORTS_SLOT_IDENTITY` become literals by another name
for the purpose of declaring it: neither follows the setting. The new
section says what a sealed log costs an implementer, which for most is
nothing — a World allocating each position at the commit cannot leave a
hole, so it never seals and never writes a `noop`. The obligation belongs
to a World that pre-assigns positions, and the half that binds everyone
is the reader's.
`WORKFLOW_SEALED_LOG` is documented in runtime-tuning, taken verbatim
from #3634 so whichever lands second is a no-op merge. The deep
reference sections that PR writes, in event-sourcing and the World
allocation contract, are left to it.
Two optional World members shipped after the guide was written and were
missing everywhere:
- `events.createBatch` — in the World reference (interface and contract
note), the upgrade guide's optional surface, and the skill. Implementing
it is the declaration, so the note leads with atomicity per attempt and
what a World must reject.
- `runs.waitForTerminalStatus` — same three places. It is also what makes
`await run.returnValue` a long poll rather than an interval, which is
user-facing, so What's new gets it too.
What's new also gains the batched suspension write and the pre-assigned
positions behind it, both under faster-and-cheaper runs where they belong:
neither has any API surface.
Not addressed: `workflow.step.execute.duration`, the OpenTelemetry
histogram added in #3526, has nowhere to go. The docs have no metrics
reference at all, and inventing one for a single metric is a bigger call
than this sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@shalabhc@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[core] Record step execution duration metric - #3526
Merged
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Aug 17, 2026
Merged
[core] Record step execution duration metric#3526
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Conversation
@shalabhc
@shalabhcshalabhc
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Summary
  • record workflow.step.execute.duration as an OpenTelemetry histogram around the inner user-code step span
  • rely on resource service.name for the service dimension and add only a bounded status attribute
  • cover the histogram name, unit, and attributes with a focused unit test
Validation
  • pnpm exec vitest run packages/core/src/telemetry-metrics.test.ts
  • pnpm --filter @workflow/core typecheck
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: f01d0bb
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 17, 2026 7:04pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 17, 2026 7:04pm
example-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-astro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-express-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-fastify-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-hono-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nestjs-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nitro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nuxt-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-python-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-sveltekit-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-vite-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workflow-docsReadyReadyPreview, v0Aug 17, 2026 7:04pm
workflow-swc-playgroundReadyReadyPreviewAug 17, 2026 7:04pm
workflow-tarballsReadyReadyPreviewAug 17, 2026 7:04pm
workflow-webReadyReadyPreviewAug 17, 2026 7:04pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit f01d0bb · Mon, 17 Aug 2026 19:22:06 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep251 (-67%) 💚1415 🔴 (+26%) 🔻1502 🔴 (+25%) 🔻1821 🔴 (+18%) 🔻30
TTFSstream249 (-75%) 💚1406 🔴 (+31%) 🔻1422 🔴 (+31%) 🔻1529 🔴 (+33%) 🔻30
TTFShook + stream487 (-62%) 💚1741 🔴 (+22%) 🔻1834 🔴 (+23%) 🔻2044 🔴 (-3.6%)30
Fan-out TTFSPromise.all(100 steps)1356 (+13%)2767 (+22%) 🔻3169 (+33%) 🔻3171 (+20%) 🔻10
Fan-out TTLSPromise.all(100 steps)6422 (+5.8%)10032 (+21%) 🔻10054 (+19%) 🔻11482 (+23%) 🔻10
STSO1020 steps (inline)136 (+12%)263 (+12%)298 (+11%)456 (+6.5%)1019
WO1020 steps254953 (+15%)254953 (+15%)254953 (+15%)254953 (+15%)1
CRTTfirst chunk (pooled)97 (+10%)174 (+18%) 🔻508 (+189%) 🔻848 (+68%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)99.8 (-1%)5 (±0%)5 (±0%)145 (+38%)323 (+134%)537 (+214%)1402 (+454%)283 (+134%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.1 (±0%)334 (±0%)330 (±0%)140 (+16%)334 (+139%)550 (+151%)1226 (+111%)706 (+423%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)89.3 (±0%)16.2 (±0%)16.2 (±0%)120 (-10%)172 (+19%)244 (+27%)769 (+128%)682 (+134%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.6 (±0%)355 (±0%)354 (±0%)116 (-21%)185 (+33%)754 (+270%)3792 (+950%)2375 (+629%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (-1%)710 (±0%)709 (-1%)236 (+119%)317 (+28%)562 (+47%)1227 (+41%)786 (+13%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 222159ms → this run 254711ms (Δ +32552ms, +15%)
 100-150 ms ┃ main 22 this 2 -20
150-200 ms ███┃████████████ main 386 this 92 -294
200-250 ms ███████████████████░░░░┃ main 450 this 566 +116
250-300 ms █████░░░░░┃ main 113 this 260 +147
300-350 ms █┃ main 29 this 54 +25
350-400 ms ┃ main 2 this 27 +25
400-450 ms ┃ main 7 this 7 +0
450-500 ms ┃ main 5 this 5 +0
500-550 ms ┃ main 2 this 4 +2
550-600 ms ┃ main 0 this 1 +1
600-650 ms ┃ main 2 this 0 -2
800-850 ms ┃ main 1 this 0 -1
3950-4000 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▄▁▁·· 234 (+112%) 152 (+46%) 537 (+214%) 1402 (+454%) 3000
sweep ······▂█▇▂▁·· 252.2 (+99%) 227 (+112%) 550 (+151%) 1226 (+111%) 3000
gw 1x ·····▁▃█▂▁··· 154.1 (+36%) 132 (+29%) 244 (+27%) 769 (+128%) 5295
eve 1x ·····▁▃█▂▁▁▁· 260.9 (+120%) 129 (+28%) 754 (+270%) 3792 (+950%) 5186
eve 2x ·····▁▁█▅▁▁·· 218.8 (+18%) 172 (+15%) 562 (+47%) 1227 (+41%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control █▇▆▄▃▄▂█▃▁ 170–295ms
sweep ▁▁█▇▆▆▇▄▅▄ 184–306ms
gw 1x █▁▂▄▂▁▁▁▆▂ 130–222ms
eve 1x ▁▁▁█▁▁▁▁▁▁ 127–1251ms
eve 2x ▄▄▁▁▃▅██▅▁ 149–311ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▅█▃▂▃▁█ 250–254ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂▅▄▂▅▄█▁▁ 34–78ms
sweep ▃▇█▅▅▃▁▂▅▃ 38–91ms
gw 1x █▃▂▄▁▁▅▄▇▃ 41–52ms
eve 1x ▂▂▁█▂▂▂▂▁▂ 25–74ms
eve 2x ▅▅▁▁▄█▃▅▃▄ 24–40ms

📜 Previous results (1)
0287155
Thu, 13 Aug 2026 20:57:01 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep262 (+42%) 🔻1391 🔴 (+26%) 🔻1418 🔴 (+17%) 🔻1486 🔴 (-6.6%)30
TTFSstream234 (-1.3%)1403 🔴 (+27%) 🔻1425 🔴 (+25%) 🔻1468 🔴 (+22%) 🔻30
TTFShook + stream442 (+15%) 🔻1678 🔴 (+22%) 🔻1746 🔴 (+25%) 🔻1835 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9107 (+1.0%)10476 (+1.2%)10715 (-11%)11193 (-96%) 💚10
Fan-out TTLSPromise.all(100 steps)17466 (-1.6%)19157 (-6.6%)19813 (-71%) 💚21056 (-93%) 💚10
STSO1020 steps (inline)140 (-7.3%)189 (-16%) 💚220 (-15%)340 (-35%) 💚1019
WO1020 steps188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚1
SLstream latency98 (+4.3%)381 🔴 (+154%) 🔻462 🔴 (+187%) 🔻5193 🔴 (+1367%) 🔻30
SOstream overhead (text)137 (-13%)265 🔴 (-10%)486 (+12%)4163 🔴 (+177%) 🔻30
SOstream overhead (structured)127 (+7.6%)344 🔴 (-15%) 💚366 (-34%) 💚812 (-68%) 💚30
ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • maxRetries=0 disables retries (nextjs-webpack)
  • plainModuleDoneHook resumed via plain API route (o2flow shape) (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 19:04:53Z · abandoned wrun_01M08HS2Z9W5P8TX6GVABJVAWX
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@shalabhc
shalabhc
marked this pull request as ready for review
August 13, 2026 20:59
@shalabhc
shalabhc
requested a review
from a team
as a code ownerAugust 13, 2026 20:59
karthikscale3
karthikscale3
approved these changes
Aug 13, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
vercelBotand others
added 2 commits
August 17, 2026 18:59
@vercel@shalabhc
[core] Record step execution duration metric
64fccac
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@vercel@shalabhc
[core] Add changeset for the step execution duration metric
f01d0bb
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@shalabhc
shalabhcforce-pushed
the
identify-datadog-span-duration-metrics
branch
from
0287155 to
f01d0bbCompareAugust 17, 2026 19:00
@shalabhc
shalabhc
merged commit ac51658
into
mainAug 17, 2026

301 of 303 checks passed

@shalabhc
shalabhc
deleted the

identify-datadog-span-duration-metrics

branch
August 17, 2026 21:49
@github-actions
github-actionsBot
commented
Aug 17, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for ac51658 (AI decision).
This commit adds a new observability capability — a workflow.step.execute.duration OpenTelemetry histogram plus a new exported recordStepExecutionDuration API in @workflow/core — rather than fixing a defect on the maintenance line. New metrics and new exported API surface are feature work, so they stay on main regardless of how small or self-contained the change is.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
ac51658d39649e7d8e290924c3bf7b2c7df78b00


This was referenced Aug 17, 2026
 Closed

 Open

 Closed

 Open

 Open

 Merged

VaguelySerious added a commit
that referenced
this pull request
Aug 21, 2026
@VaguelySerious@claude
docs: catch v5 up with main, and with the sealed log
a08bced
Sweep of everything that landed on main since the last content refresh,
plus the spec-7 sealed log from #3634, which is not merged but is the
default for v5.
Spec 7 changes what a World declares, so the upgrade guide's Spec
versions section is rewritten around `mintedSpecVersion()`. The accepted
range widens back to two versions, because a deployment with
`WORKFLOW_SEALED_LOG=0` stamps the one below, and both `SPEC_VERSION_CURRENT`
and `SPEC_VERSION_SUPPORTS_SLOT_IDENTITY` become literals by another name
for the purpose of declaring it: neither follows the setting. The new
section says what a sealed log costs an implementer, which for most is
nothing — a World allocating each position at the commit cannot leave a
hole, so it never seals and never writes a `noop`. The obligation belongs
to a World that pre-assigns positions, and the half that binds everyone
is the reader's.
`WORKFLOW_SEALED_LOG` is documented in runtime-tuning, taken verbatim
from #3634 so whichever lands second is a no-op merge. The deep
reference sections that PR writes, in event-sourcing and the World
allocation contract, are left to it.
Two optional World members shipped after the guide was written and were
missing everywhere:
- `events.createBatch` — in the World reference (interface and contract
note), the upgrade guide's optional surface, and the skill. Implementing
it is the declaration, so the note leads with atomicity per attempt and
what a World must reject.
- `runs.waitForTerminalStatus` — same three places. It is also what makes
`await run.returnValue` a long poll rather than an interval, which is
user-facing, so What's new gets it too.
What's new also gains the batched suspension write and the pre-assigned
positions behind it, both under faster-and-cheaper runs where they belong:
neither has any API surface.
Not addressed: `workflow.step.execute.duration`, the OpenTelemetry
histogram added in #3526, has nowhere to go. The docs have no metrics
reference at all, and inventing one for a single metric is a bigger call
than this sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@shalabhc@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
[core] Record step execution duration metric - #3526
Merged
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Aug 17, 2026
Merged
[core] Record step execution duration metric#3526
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Conversation
@shalabhc
@shalabhcshalabhc
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Summary
  • record workflow.step.execute.duration as an OpenTelemetry histogram around the inner user-code step span
  • rely on resource service.name for the service dimension and add only a bounded status attribute
  • cover the histogram name, unit, and attributes with a focused unit test
Validation
  • pnpm exec vitest run packages/core/src/telemetry-metrics.test.ts
  • pnpm --filter @workflow/core typecheck
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: f01d0bb
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 17, 2026 7:04pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 17, 2026 7:04pm
example-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-astro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-express-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-fastify-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-hono-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nestjs-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nitro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nuxt-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-python-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-sveltekit-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-vite-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workflow-docsReadyReadyPreview, v0Aug 17, 2026 7:04pm
workflow-swc-playgroundReadyReadyPreviewAug 17, 2026 7:04pm
workflow-tarballsReadyReadyPreviewAug 17, 2026 7:04pm
workflow-webReadyReadyPreviewAug 17, 2026 7:04pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit f01d0bb · Mon, 17 Aug 2026 19:22:06 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep251 (-67%) 💚1415 🔴 (+26%) 🔻1502 🔴 (+25%) 🔻1821 🔴 (+18%) 🔻30
TTFSstream249 (-75%) 💚1406 🔴 (+31%) 🔻1422 🔴 (+31%) 🔻1529 🔴 (+33%) 🔻30
TTFShook + stream487 (-62%) 💚1741 🔴 (+22%) 🔻1834 🔴 (+23%) 🔻2044 🔴 (-3.6%)30
Fan-out TTFSPromise.all(100 steps)1356 (+13%)2767 (+22%) 🔻3169 (+33%) 🔻3171 (+20%) 🔻10
Fan-out TTLSPromise.all(100 steps)6422 (+5.8%)10032 (+21%) 🔻10054 (+19%) 🔻11482 (+23%) 🔻10
STSO1020 steps (inline)136 (+12%)263 (+12%)298 (+11%)456 (+6.5%)1019
WO1020 steps254953 (+15%)254953 (+15%)254953 (+15%)254953 (+15%)1
CRTTfirst chunk (pooled)97 (+10%)174 (+18%) 🔻508 (+189%) 🔻848 (+68%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)99.8 (-1%)5 (±0%)5 (±0%)145 (+38%)323 (+134%)537 (+214%)1402 (+454%)283 (+134%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.1 (±0%)334 (±0%)330 (±0%)140 (+16%)334 (+139%)550 (+151%)1226 (+111%)706 (+423%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)89.3 (±0%)16.2 (±0%)16.2 (±0%)120 (-10%)172 (+19%)244 (+27%)769 (+128%)682 (+134%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.6 (±0%)355 (±0%)354 (±0%)116 (-21%)185 (+33%)754 (+270%)3792 (+950%)2375 (+629%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (-1%)710 (±0%)709 (-1%)236 (+119%)317 (+28%)562 (+47%)1227 (+41%)786 (+13%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 222159ms → this run 254711ms (Δ +32552ms, +15%)
 100-150 ms ┃ main 22 this 2 -20
150-200 ms ███┃████████████ main 386 this 92 -294
200-250 ms ███████████████████░░░░┃ main 450 this 566 +116
250-300 ms █████░░░░░┃ main 113 this 260 +147
300-350 ms █┃ main 29 this 54 +25
350-400 ms ┃ main 2 this 27 +25
400-450 ms ┃ main 7 this 7 +0
450-500 ms ┃ main 5 this 5 +0
500-550 ms ┃ main 2 this 4 +2
550-600 ms ┃ main 0 this 1 +1
600-650 ms ┃ main 2 this 0 -2
800-850 ms ┃ main 1 this 0 -1
3950-4000 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▄▁▁·· 234 (+112%) 152 (+46%) 537 (+214%) 1402 (+454%) 3000
sweep ······▂█▇▂▁·· 252.2 (+99%) 227 (+112%) 550 (+151%) 1226 (+111%) 3000
gw 1x ·····▁▃█▂▁··· 154.1 (+36%) 132 (+29%) 244 (+27%) 769 (+128%) 5295
eve 1x ·····▁▃█▂▁▁▁· 260.9 (+120%) 129 (+28%) 754 (+270%) 3792 (+950%) 5186
eve 2x ·····▁▁█▅▁▁·· 218.8 (+18%) 172 (+15%) 562 (+47%) 1227 (+41%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control █▇▆▄▃▄▂█▃▁ 170–295ms
sweep ▁▁█▇▆▆▇▄▅▄ 184–306ms
gw 1x █▁▂▄▂▁▁▁▆▂ 130–222ms
eve 1x ▁▁▁█▁▁▁▁▁▁ 127–1251ms
eve 2x ▄▄▁▁▃▅██▅▁ 149–311ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▅█▃▂▃▁█ 250–254ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂▅▄▂▅▄█▁▁ 34–78ms
sweep ▃▇█▅▅▃▁▂▅▃ 38–91ms
gw 1x █▃▂▄▁▁▅▄▇▃ 41–52ms
eve 1x ▂▂▁█▂▂▂▂▁▂ 25–74ms
eve 2x ▅▅▁▁▄█▃▅▃▄ 24–40ms

📜 Previous results (1)
0287155
Thu, 13 Aug 2026 20:57:01 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep262 (+42%) 🔻1391 🔴 (+26%) 🔻1418 🔴 (+17%) 🔻1486 🔴 (-6.6%)30
TTFSstream234 (-1.3%)1403 🔴 (+27%) 🔻1425 🔴 (+25%) 🔻1468 🔴 (+22%) 🔻30
TTFShook + stream442 (+15%) 🔻1678 🔴 (+22%) 🔻1746 🔴 (+25%) 🔻1835 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9107 (+1.0%)10476 (+1.2%)10715 (-11%)11193 (-96%) 💚10
Fan-out TTLSPromise.all(100 steps)17466 (-1.6%)19157 (-6.6%)19813 (-71%) 💚21056 (-93%) 💚10
STSO1020 steps (inline)140 (-7.3%)189 (-16%) 💚220 (-15%)340 (-35%) 💚1019
WO1020 steps188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚1
SLstream latency98 (+4.3%)381 🔴 (+154%) 🔻462 🔴 (+187%) 🔻5193 🔴 (+1367%) 🔻30
SOstream overhead (text)137 (-13%)265 🔴 (-10%)486 (+12%)4163 🔴 (+177%) 🔻30
SOstream overhead (structured)127 (+7.6%)344 🔴 (-15%) 💚366 (-34%) 💚812 (-68%) 💚30
ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • maxRetries=0 disables retries (nextjs-webpack)
  • plainModuleDoneHook resumed via plain API route (o2flow shape) (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 19:04:53Z · abandoned wrun_01M08HS2Z9W5P8TX6GVABJVAWX
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@shalabhc
shalabhc
marked this pull request as ready for review
August 13, 2026 20:59
@shalabhc
shalabhc
requested a review
from a team
as a code ownerAugust 13, 2026 20:59
karthikscale3
karthikscale3
approved these changes
Aug 13, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
vercelBotand others
added 2 commits
August 17, 2026 18:59
@vercel@shalabhc
[core] Record step execution duration metric
64fccac
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@vercel@shalabhc
[core] Add changeset for the step execution duration metric
f01d0bb
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@shalabhc
shalabhcforce-pushed
the
identify-datadog-span-duration-metrics
branch
from
0287155 to
f01d0bbCompareAugust 17, 2026 19:00
@shalabhc
shalabhc
merged commit ac51658
into
mainAug 17, 2026

301 of 303 checks passed

@shalabhc
shalabhc
deleted the

identify-datadog-span-duration-metrics

branch
August 17, 2026 21:49
@github-actions
github-actionsBot
commented
Aug 17, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for ac51658 (AI decision).
This commit adds a new observability capability — a workflow.step.execute.duration OpenTelemetry histogram plus a new exported recordStepExecutionDuration API in @workflow/core — rather than fixing a defect on the maintenance line. New metrics and new exported API surface are feature work, so they stay on main regardless of how small or self-contained the change is.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
ac51658d39649e7d8e290924c3bf7b2c7df78b00


This was referenced Aug 17, 2026
 Closed

 Open

 Closed

 Open

 Open

 Merged

VaguelySerious added a commit
that referenced
this pull request
Aug 21, 2026
@VaguelySerious@claude
docs: catch v5 up with main, and with the sealed log
a08bced
Sweep of everything that landed on main since the last content refresh,
plus the spec-7 sealed log from #3634, which is not merged but is the
default for v5.
Spec 7 changes what a World declares, so the upgrade guide's Spec
versions section is rewritten around `mintedSpecVersion()`. The accepted
range widens back to two versions, because a deployment with
`WORKFLOW_SEALED_LOG=0` stamps the one below, and both `SPEC_VERSION_CURRENT`
and `SPEC_VERSION_SUPPORTS_SLOT_IDENTITY` become literals by another name
for the purpose of declaring it: neither follows the setting. The new
section says what a sealed log costs an implementer, which for most is
nothing — a World allocating each position at the commit cannot leave a
hole, so it never seals and never writes a `noop`. The obligation belongs
to a World that pre-assigns positions, and the half that binds everyone
is the reader's.
`WORKFLOW_SEALED_LOG` is documented in runtime-tuning, taken verbatim
from #3634 so whichever lands second is a no-op merge. The deep
reference sections that PR writes, in event-sourcing and the World
allocation contract, are left to it.
Two optional World members shipped after the guide was written and were
missing everywhere:
- `events.createBatch` — in the World reference (interface and contract
note), the upgrade guide's optional surface, and the skill. Implementing
it is the declaration, so the note leads with atomicity per attempt and
what a World must reject.
- `runs.waitForTerminalStatus` — same three places. It is also what makes
`await run.returnValue` a long poll rather than an interval, which is
user-facing, so What's new gets it too.
What's new also gains the batched suspension write and the pre-assigned
positions behind it, both under faster-and-cheaper runs where they belong:
neither has any API surface.
Not addressed: `workflow.step.execute.duration`, the OpenTelemetry
histogram added in #3526, has nowhere to go. The docs have no metrics
reference at all, and inventing one for a single metric is a bigger call
than this sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@shalabhc@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[core] Record step execution duration metric - #3526
Merged
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Aug 17, 2026
Merged
[core] Record step execution duration metric#3526
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Conversation
@shalabhc
@shalabhcshalabhc
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Summary
  • record workflow.step.execute.duration as an OpenTelemetry histogram around the inner user-code step span
  • rely on resource service.name for the service dimension and add only a bounded status attribute
  • cover the histogram name, unit, and attributes with a focused unit test
Validation
  • pnpm exec vitest run packages/core/src/telemetry-metrics.test.ts
  • pnpm --filter @workflow/core typecheck
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: f01d0bb
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 17, 2026 7:04pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 17, 2026 7:04pm
example-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-astro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-express-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-fastify-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-hono-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nestjs-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nitro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nuxt-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-python-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-sveltekit-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-vite-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workflow-docsReadyReadyPreview, v0Aug 17, 2026 7:04pm
workflow-swc-playgroundReadyReadyPreviewAug 17, 2026 7:04pm
workflow-tarballsReadyReadyPreviewAug 17, 2026 7:04pm
workflow-webReadyReadyPreviewAug 17, 2026 7:04pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit f01d0bb · Mon, 17 Aug 2026 19:22:06 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep251 (-67%) 💚1415 🔴 (+26%) 🔻1502 🔴 (+25%) 🔻1821 🔴 (+18%) 🔻30
TTFSstream249 (-75%) 💚1406 🔴 (+31%) 🔻1422 🔴 (+31%) 🔻1529 🔴 (+33%) 🔻30
TTFShook + stream487 (-62%) 💚1741 🔴 (+22%) 🔻1834 🔴 (+23%) 🔻2044 🔴 (-3.6%)30
Fan-out TTFSPromise.all(100 steps)1356 (+13%)2767 (+22%) 🔻3169 (+33%) 🔻3171 (+20%) 🔻10
Fan-out TTLSPromise.all(100 steps)6422 (+5.8%)10032 (+21%) 🔻10054 (+19%) 🔻11482 (+23%) 🔻10
STSO1020 steps (inline)136 (+12%)263 (+12%)298 (+11%)456 (+6.5%)1019
WO1020 steps254953 (+15%)254953 (+15%)254953 (+15%)254953 (+15%)1
CRTTfirst chunk (pooled)97 (+10%)174 (+18%) 🔻508 (+189%) 🔻848 (+68%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)99.8 (-1%)5 (±0%)5 (±0%)145 (+38%)323 (+134%)537 (+214%)1402 (+454%)283 (+134%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.1 (±0%)334 (±0%)330 (±0%)140 (+16%)334 (+139%)550 (+151%)1226 (+111%)706 (+423%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)89.3 (±0%)16.2 (±0%)16.2 (±0%)120 (-10%)172 (+19%)244 (+27%)769 (+128%)682 (+134%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.6 (±0%)355 (±0%)354 (±0%)116 (-21%)185 (+33%)754 (+270%)3792 (+950%)2375 (+629%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (-1%)710 (±0%)709 (-1%)236 (+119%)317 (+28%)562 (+47%)1227 (+41%)786 (+13%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 222159ms → this run 254711ms (Δ +32552ms, +15%)
 100-150 ms ┃ main 22 this 2 -20
150-200 ms ███┃████████████ main 386 this 92 -294
200-250 ms ███████████████████░░░░┃ main 450 this 566 +116
250-300 ms █████░░░░░┃ main 113 this 260 +147
300-350 ms █┃ main 29 this 54 +25
350-400 ms ┃ main 2 this 27 +25
400-450 ms ┃ main 7 this 7 +0
450-500 ms ┃ main 5 this 5 +0
500-550 ms ┃ main 2 this 4 +2
550-600 ms ┃ main 0 this 1 +1
600-650 ms ┃ main 2 this 0 -2
800-850 ms ┃ main 1 this 0 -1
3950-4000 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▄▁▁·· 234 (+112%) 152 (+46%) 537 (+214%) 1402 (+454%) 3000
sweep ······▂█▇▂▁·· 252.2 (+99%) 227 (+112%) 550 (+151%) 1226 (+111%) 3000
gw 1x ·····▁▃█▂▁··· 154.1 (+36%) 132 (+29%) 244 (+27%) 769 (+128%) 5295
eve 1x ·····▁▃█▂▁▁▁· 260.9 (+120%) 129 (+28%) 754 (+270%) 3792 (+950%) 5186
eve 2x ·····▁▁█▅▁▁·· 218.8 (+18%) 172 (+15%) 562 (+47%) 1227 (+41%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control █▇▆▄▃▄▂█▃▁ 170–295ms
sweep ▁▁█▇▆▆▇▄▅▄ 184–306ms
gw 1x █▁▂▄▂▁▁▁▆▂ 130–222ms
eve 1x ▁▁▁█▁▁▁▁▁▁ 127–1251ms
eve 2x ▄▄▁▁▃▅██▅▁ 149–311ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▅█▃▂▃▁█ 250–254ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂▅▄▂▅▄█▁▁ 34–78ms
sweep ▃▇█▅▅▃▁▂▅▃ 38–91ms
gw 1x █▃▂▄▁▁▅▄▇▃ 41–52ms
eve 1x ▂▂▁█▂▂▂▂▁▂ 25–74ms
eve 2x ▅▅▁▁▄█▃▅▃▄ 24–40ms

📜 Previous results (1)
0287155
Thu, 13 Aug 2026 20:57:01 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep262 (+42%) 🔻1391 🔴 (+26%) 🔻1418 🔴 (+17%) 🔻1486 🔴 (-6.6%)30
TTFSstream234 (-1.3%)1403 🔴 (+27%) 🔻1425 🔴 (+25%) 🔻1468 🔴 (+22%) 🔻30
TTFShook + stream442 (+15%) 🔻1678 🔴 (+22%) 🔻1746 🔴 (+25%) 🔻1835 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9107 (+1.0%)10476 (+1.2%)10715 (-11%)11193 (-96%) 💚10
Fan-out TTLSPromise.all(100 steps)17466 (-1.6%)19157 (-6.6%)19813 (-71%) 💚21056 (-93%) 💚10
STSO1020 steps (inline)140 (-7.3%)189 (-16%) 💚220 (-15%)340 (-35%) 💚1019
WO1020 steps188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚1
SLstream latency98 (+4.3%)381 🔴 (+154%) 🔻462 🔴 (+187%) 🔻5193 🔴 (+1367%) 🔻30
SOstream overhead (text)137 (-13%)265 🔴 (-10%)486 (+12%)4163 🔴 (+177%) 🔻30
SOstream overhead (structured)127 (+7.6%)344 🔴 (-15%) 💚366 (-34%) 💚812 (-68%) 💚30
ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • maxRetries=0 disables retries (nextjs-webpack)
  • plainModuleDoneHook resumed via plain API route (o2flow shape) (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 19:04:53Z · abandoned wrun_01M08HS2Z9W5P8TX6GVABJVAWX
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@shalabhc
shalabhc
marked this pull request as ready for review
August 13, 2026 20:59
@shalabhc
shalabhc
requested a review
from a team
as a code ownerAugust 13, 2026 20:59
karthikscale3
karthikscale3
approved these changes
Aug 13, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
vercelBotand others
added 2 commits
August 17, 2026 18:59
@vercel@shalabhc
[core] Record step execution duration metric
64fccac
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@vercel@shalabhc
[core] Add changeset for the step execution duration metric
f01d0bb
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@shalabhc
shalabhcforce-pushed
the
identify-datadog-span-duration-metrics
branch
from
0287155 to
f01d0bbCompareAugust 17, 2026 19:00
@shalabhc
shalabhc
merged commit ac51658
into
mainAug 17, 2026

301 of 303 checks passed

@shalabhc
shalabhc
deleted the

identify-datadog-span-duration-metrics

branch
August 17, 2026 21:49
@github-actions
github-actionsBot
commented
Aug 17, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for ac51658 (AI decision).
This commit adds a new observability capability — a workflow.step.execute.duration OpenTelemetry histogram plus a new exported recordStepExecutionDuration API in @workflow/core — rather than fixing a defect on the maintenance line. New metrics and new exported API surface are feature work, so they stay on main regardless of how small or self-contained the change is.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
ac51658d39649e7d8e290924c3bf7b2c7df78b00


This was referenced Aug 17, 2026
 Closed

 Open

 Closed

 Open

 Open

 Merged

VaguelySerious added a commit
that referenced
this pull request
Aug 21, 2026
@VaguelySerious@claude
docs: catch v5 up with main, and with the sealed log
a08bced
Sweep of everything that landed on main since the last content refresh,
plus the spec-7 sealed log from #3634, which is not merged but is the
default for v5.
Spec 7 changes what a World declares, so the upgrade guide's Spec
versions section is rewritten around `mintedSpecVersion()`. The accepted
range widens back to two versions, because a deployment with
`WORKFLOW_SEALED_LOG=0` stamps the one below, and both `SPEC_VERSION_CURRENT`
and `SPEC_VERSION_SUPPORTS_SLOT_IDENTITY` become literals by another name
for the purpose of declaring it: neither follows the setting. The new
section says what a sealed log costs an implementer, which for most is
nothing — a World allocating each position at the commit cannot leave a
hole, so it never seals and never writes a `noop`. The obligation belongs
to a World that pre-assigns positions, and the half that binds everyone
is the reader's.
`WORKFLOW_SEALED_LOG` is documented in runtime-tuning, taken verbatim
from #3634 so whichever lands second is a no-op merge. The deep
reference sections that PR writes, in event-sourcing and the World
allocation contract, are left to it.
Two optional World members shipped after the guide was written and were
missing everywhere:
- `events.createBatch` — in the World reference (interface and contract
note), the upgrade guide's optional surface, and the skill. Implementing
it is the declaration, so the note leads with atomicity per attempt and
what a World must reject.
- `runs.waitForTerminalStatus` — same three places. It is also what makes
`await run.returnValue` a long poll rather than an interval, which is
user-facing, so What's new gets it too.
What's new also gains the batched suspension write and the pre-assigned
positions behind it, both under faster-and-cheaper runs where they belong:
neither has any API surface.
Not addressed: `workflow.step.execute.duration`, the OpenTelemetry
histogram added in #3526, has nowhere to go. The docs have no metrics
reference at all, and inventing one for a single metric is a bigger call
than this sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@shalabhc@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[core] Record step execution duration metric - #3526
Merged
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Aug 17, 2026
Merged
[core] Record step execution duration metric#3526
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Conversation
@shalabhc
@shalabhcshalabhc
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Summary
  • record workflow.step.execute.duration as an OpenTelemetry histogram around the inner user-code step span
  • rely on resource service.name for the service dimension and add only a bounded status attribute
  • cover the histogram name, unit, and attributes with a focused unit test
Validation
  • pnpm exec vitest run packages/core/src/telemetry-metrics.test.ts
  • pnpm --filter @workflow/core typecheck
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: f01d0bb
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 17, 2026 7:04pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 17, 2026 7:04pm
example-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-astro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-express-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-fastify-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-hono-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nestjs-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nitro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nuxt-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-python-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-sveltekit-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-vite-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workflow-docsReadyReadyPreview, v0Aug 17, 2026 7:04pm
workflow-swc-playgroundReadyReadyPreviewAug 17, 2026 7:04pm
workflow-tarballsReadyReadyPreviewAug 17, 2026 7:04pm
workflow-webReadyReadyPreviewAug 17, 2026 7:04pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit f01d0bb · Mon, 17 Aug 2026 19:22:06 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep251 (-67%) 💚1415 🔴 (+26%) 🔻1502 🔴 (+25%) 🔻1821 🔴 (+18%) 🔻30
TTFSstream249 (-75%) 💚1406 🔴 (+31%) 🔻1422 🔴 (+31%) 🔻1529 🔴 (+33%) 🔻30
TTFShook + stream487 (-62%) 💚1741 🔴 (+22%) 🔻1834 🔴 (+23%) 🔻2044 🔴 (-3.6%)30
Fan-out TTFSPromise.all(100 steps)1356 (+13%)2767 (+22%) 🔻3169 (+33%) 🔻3171 (+20%) 🔻10
Fan-out TTLSPromise.all(100 steps)6422 (+5.8%)10032 (+21%) 🔻10054 (+19%) 🔻11482 (+23%) 🔻10
STSO1020 steps (inline)136 (+12%)263 (+12%)298 (+11%)456 (+6.5%)1019
WO1020 steps254953 (+15%)254953 (+15%)254953 (+15%)254953 (+15%)1
CRTTfirst chunk (pooled)97 (+10%)174 (+18%) 🔻508 (+189%) 🔻848 (+68%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)99.8 (-1%)5 (±0%)5 (±0%)145 (+38%)323 (+134%)537 (+214%)1402 (+454%)283 (+134%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.1 (±0%)334 (±0%)330 (±0%)140 (+16%)334 (+139%)550 (+151%)1226 (+111%)706 (+423%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)89.3 (±0%)16.2 (±0%)16.2 (±0%)120 (-10%)172 (+19%)244 (+27%)769 (+128%)682 (+134%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.6 (±0%)355 (±0%)354 (±0%)116 (-21%)185 (+33%)754 (+270%)3792 (+950%)2375 (+629%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (-1%)710 (±0%)709 (-1%)236 (+119%)317 (+28%)562 (+47%)1227 (+41%)786 (+13%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 222159ms → this run 254711ms (Δ +32552ms, +15%)
 100-150 ms ┃ main 22 this 2 -20
150-200 ms ███┃████████████ main 386 this 92 -294
200-250 ms ███████████████████░░░░┃ main 450 this 566 +116
250-300 ms █████░░░░░┃ main 113 this 260 +147
300-350 ms █┃ main 29 this 54 +25
350-400 ms ┃ main 2 this 27 +25
400-450 ms ┃ main 7 this 7 +0
450-500 ms ┃ main 5 this 5 +0
500-550 ms ┃ main 2 this 4 +2
550-600 ms ┃ main 0 this 1 +1
600-650 ms ┃ main 2 this 0 -2
800-850 ms ┃ main 1 this 0 -1
3950-4000 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▄▁▁·· 234 (+112%) 152 (+46%) 537 (+214%) 1402 (+454%) 3000
sweep ······▂█▇▂▁·· 252.2 (+99%) 227 (+112%) 550 (+151%) 1226 (+111%) 3000
gw 1x ·····▁▃█▂▁··· 154.1 (+36%) 132 (+29%) 244 (+27%) 769 (+128%) 5295
eve 1x ·····▁▃█▂▁▁▁· 260.9 (+120%) 129 (+28%) 754 (+270%) 3792 (+950%) 5186
eve 2x ·····▁▁█▅▁▁·· 218.8 (+18%) 172 (+15%) 562 (+47%) 1227 (+41%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control █▇▆▄▃▄▂█▃▁ 170–295ms
sweep ▁▁█▇▆▆▇▄▅▄ 184–306ms
gw 1x █▁▂▄▂▁▁▁▆▂ 130–222ms
eve 1x ▁▁▁█▁▁▁▁▁▁ 127–1251ms
eve 2x ▄▄▁▁▃▅██▅▁ 149–311ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▅█▃▂▃▁█ 250–254ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂▅▄▂▅▄█▁▁ 34–78ms
sweep ▃▇█▅▅▃▁▂▅▃ 38–91ms
gw 1x █▃▂▄▁▁▅▄▇▃ 41–52ms
eve 1x ▂▂▁█▂▂▂▂▁▂ 25–74ms
eve 2x ▅▅▁▁▄█▃▅▃▄ 24–40ms

📜 Previous results (1)
0287155
Thu, 13 Aug 2026 20:57:01 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep262 (+42%) 🔻1391 🔴 (+26%) 🔻1418 🔴 (+17%) 🔻1486 🔴 (-6.6%)30
TTFSstream234 (-1.3%)1403 🔴 (+27%) 🔻1425 🔴 (+25%) 🔻1468 🔴 (+22%) 🔻30
TTFShook + stream442 (+15%) 🔻1678 🔴 (+22%) 🔻1746 🔴 (+25%) 🔻1835 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9107 (+1.0%)10476 (+1.2%)10715 (-11%)11193 (-96%) 💚10
Fan-out TTLSPromise.all(100 steps)17466 (-1.6%)19157 (-6.6%)19813 (-71%) 💚21056 (-93%) 💚10
STSO1020 steps (inline)140 (-7.3%)189 (-16%) 💚220 (-15%)340 (-35%) 💚1019
WO1020 steps188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚1
SLstream latency98 (+4.3%)381 🔴 (+154%) 🔻462 🔴 (+187%) 🔻5193 🔴 (+1367%) 🔻30
SOstream overhead (text)137 (-13%)265 🔴 (-10%)486 (+12%)4163 🔴 (+177%) 🔻30
SOstream overhead (structured)127 (+7.6%)344 🔴 (-15%) 💚366 (-34%) 💚812 (-68%) 💚30
ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • maxRetries=0 disables retries (nextjs-webpack)
  • plainModuleDoneHook resumed via plain API route (o2flow shape) (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 19:04:53Z · abandoned wrun_01M08HS2Z9W5P8TX6GVABJVAWX
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@shalabhc
shalabhc
marked this pull request as ready for review
August 13, 2026 20:59
@shalabhc
shalabhc
requested a review
from a team
as a code ownerAugust 13, 2026 20:59
karthikscale3
karthikscale3
approved these changes
Aug 13, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
vercelBotand others
added 2 commits
August 17, 2026 18:59
@vercel@shalabhc
[core] Record step execution duration metric
64fccac
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@vercel@shalabhc
[core] Add changeset for the step execution duration metric
f01d0bb
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@shalabhc
shalabhcforce-pushed
the
identify-datadog-span-duration-metrics
branch
from
0287155 to
f01d0bbCompareAugust 17, 2026 19:00
@shalabhc
shalabhc
merged commit ac51658
into
mainAug 17, 2026

301 of 303 checks passed

@shalabhc
shalabhc
deleted the

identify-datadog-span-duration-metrics

branch
August 17, 2026 21:49
@github-actions
github-actionsBot
commented
Aug 17, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for ac51658 (AI decision).
This commit adds a new observability capability — a workflow.step.execute.duration OpenTelemetry histogram plus a new exported recordStepExecutionDuration API in @workflow/core — rather than fixing a defect on the maintenance line. New metrics and new exported API surface are feature work, so they stay on main regardless of how small or self-contained the change is.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
ac51658d39649e7d8e290924c3bf7b2c7df78b00


This was referenced Aug 17, 2026
 Closed

 Open

 Closed

 Open

 Open

 Merged

VaguelySerious added a commit
that referenced
this pull request
Aug 21, 2026
@VaguelySerious@claude
docs: catch v5 up with main, and with the sealed log
a08bced
Sweep of everything that landed on main since the last content refresh,
plus the spec-7 sealed log from #3634, which is not merged but is the
default for v5.
Spec 7 changes what a World declares, so the upgrade guide's Spec
versions section is rewritten around `mintedSpecVersion()`. The accepted
range widens back to two versions, because a deployment with
`WORKFLOW_SEALED_LOG=0` stamps the one below, and both `SPEC_VERSION_CURRENT`
and `SPEC_VERSION_SUPPORTS_SLOT_IDENTITY` become literals by another name
for the purpose of declaring it: neither follows the setting. The new
section says what a sealed log costs an implementer, which for most is
nothing — a World allocating each position at the commit cannot leave a
hole, so it never seals and never writes a `noop`. The obligation belongs
to a World that pre-assigns positions, and the half that binds everyone
is the reader's.
`WORKFLOW_SEALED_LOG` is documented in runtime-tuning, taken verbatim
from #3634 so whichever lands second is a no-op merge. The deep
reference sections that PR writes, in event-sourcing and the World
allocation contract, are left to it.
Two optional World members shipped after the guide was written and were
missing everywhere:
- `events.createBatch` — in the World reference (interface and contract
note), the upgrade guide's optional surface, and the skill. Implementing
it is the declaration, so the note leads with atomicity per attempt and
what a World must reject.
- `runs.waitForTerminalStatus` — same three places. It is also what makes
`await run.returnValue` a long poll rather than an interval, which is
user-facing, so What's new gets it too.
What's new also gains the batched suspension write and the pre-assigned
positions behind it, both under faster-and-cheaper runs where they belong:
neither has any API surface.
Not addressed: `workflow.step.execute.duration`, the OpenTelemetry
histogram added in #3526, has nowhere to go. The docs have no metrics
reference at all, and inventing one for a single metric is a bigger call
than this sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@shalabhc@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
[core] Record step execution duration metric - #3526
Merged
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Aug 17, 2026
Merged
[core] Record step execution duration metric#3526
shalabhc merged 2 commits into
mainfrom 
identify-datadog-span-duration-metrics
Conversation
@shalabhc
@shalabhcshalabhc
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Collaborator
Summary
  • record workflow.step.execute.duration as an OpenTelemetry histogram around the inner user-code step span
  • rely on resource service.name for the service dimension and add only a bounded status attribute
  • cover the histogram name, unit, and attributes with a focused unit test
Validation
  • pnpm exec vitest run packages/core/src/telemetry-metrics.test.ts
  • pnpm --filter @workflow/core typecheck
@changeset-bot
changeset-botBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: f01d0bb
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 17, 2026 7:04pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 17, 2026 7:04pm
example-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-astro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-express-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-fastify-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-hono-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nestjs-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nitro-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-nuxt-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-python-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-sveltekit-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workbench-vite-workflowReadyReadyPreviewAug 17, 2026 7:04pm
workflow-docsReadyReadyPreview, v0Aug 17, 2026 7:04pm
workflow-swc-playgroundReadyReadyPreviewAug 17, 2026 7:04pm
workflow-tarballsReadyReadyPreviewAug 17, 2026 7:04pm
workflow-webReadyReadyPreviewAug 17, 2026 7:04pm
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit f01d0bb · Mon, 17 Aug 2026 19:22:06 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep251 (-67%) 💚1415 🔴 (+26%) 🔻1502 🔴 (+25%) 🔻1821 🔴 (+18%) 🔻30
TTFSstream249 (-75%) 💚1406 🔴 (+31%) 🔻1422 🔴 (+31%) 🔻1529 🔴 (+33%) 🔻30
TTFShook + stream487 (-62%) 💚1741 🔴 (+22%) 🔻1834 🔴 (+23%) 🔻2044 🔴 (-3.6%)30
Fan-out TTFSPromise.all(100 steps)1356 (+13%)2767 (+22%) 🔻3169 (+33%) 🔻3171 (+20%) 🔻10
Fan-out TTLSPromise.all(100 steps)6422 (+5.8%)10032 (+21%) 🔻10054 (+19%) 🔻11482 (+23%) 🔻10
STSO1020 steps (inline)136 (+12%)263 (+12%)298 (+11%)456 (+6.5%)1019
WO1020 steps254953 (+15%)254953 (+15%)254953 (+15%)254953 (+15%)1
CRTTfirst chunk (pooled)97 (+10%)174 (+18%) 🔻508 (+189%) 🔻848 (+68%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)99.8 (-1%)5 (±0%)5 (±0%)145 (+38%)323 (+134%)537 (+214%)1402 (+454%)283 (+134%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.1 (±0%)334 (±0%)330 (±0%)140 (+16%)334 (+139%)550 (+151%)1226 (+111%)706 (+423%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)89.3 (±0%)16.2 (±0%)16.2 (±0%)120 (-10%)172 (+19%)244 (+27%)769 (+128%)682 (+134%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.6 (±0%)355 (±0%)354 (±0%)116 (-21%)185 (+33%)754 (+270%)3792 (+950%)2375 (+629%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (-1%)710 (±0%)709 (-1%)236 (+119%)317 (+28%)562 (+47%)1227 (+41%)786 (+13%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 222159ms → this run 254711ms (Δ +32552ms, +15%)
 100-150 ms ┃ main 22 this 2 -20
150-200 ms ███┃████████████ main 386 this 92 -294
200-250 ms ███████████████████░░░░┃ main 450 this 566 +116
250-300 ms █████░░░░░┃ main 113 this 260 +147
300-350 ms █┃ main 29 this 54 +25
350-400 ms ┃ main 2 this 27 +25
400-450 ms ┃ main 7 this 7 +0
450-500 ms ┃ main 5 this 5 +0
500-550 ms ┃ main 2 this 4 +2
550-600 ms ┃ main 0 this 1 +1
600-650 ms ┃ main 2 this 0 -2
800-850 ms ┃ main 1 this 0 -1
3950-4000 ms ┃ main 0 this 1 +1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▄▁▁·· 234 (+112%) 152 (+46%) 537 (+214%) 1402 (+454%) 3000
sweep ······▂█▇▂▁·· 252.2 (+99%) 227 (+112%) 550 (+151%) 1226 (+111%) 3000
gw 1x ·····▁▃█▂▁··· 154.1 (+36%) 132 (+29%) 244 (+27%) 769 (+128%) 5295
eve 1x ·····▁▃█▂▁▁▁· 260.9 (+120%) 129 (+28%) 754 (+270%) 3792 (+950%) 5186
eve 2x ·····▁▁█▅▁▁·· 218.8 (+18%) 172 (+15%) 562 (+47%) 1227 (+41%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control █▇▆▄▃▄▂█▃▁ 170–295ms
sweep ▁▁█▇▆▆▇▄▅▄ 184–306ms
gw 1x █▁▂▄▂▁▁▁▆▂ 130–222ms
eve 1x ▁▁▁█▁▁▁▁▁▁ 127–1251ms
eve 2x ▄▄▁▁▃▅██▅▁ 149–311ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▅█▃▂▃▁█ 250–254ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂▅▄▂▅▄█▁▁ 34–78ms
sweep ▃▇█▅▅▃▁▂▅▃ 38–91ms
gw 1x █▃▂▄▁▁▅▄▇▃ 41–52ms
eve 1x ▂▂▁█▂▂▂▂▁▂ 25–74ms
eve 2x ▅▅▁▁▄█▃▅▃▄ 24–40ms

📜 Previous results (1)
0287155
Thu, 13 Aug 2026 20:57:01 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep262 (+42%) 🔻1391 🔴 (+26%) 🔻1418 🔴 (+17%) 🔻1486 🔴 (-6.6%)30
TTFSstream234 (-1.3%)1403 🔴 (+27%) 🔻1425 🔴 (+25%) 🔻1468 🔴 (+22%) 🔻30
TTFShook + stream442 (+15%) 🔻1678 🔴 (+22%) 🔻1746 🔴 (+25%) 🔻1835 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9107 (+1.0%)10476 (+1.2%)10715 (-11%)11193 (-96%) 💚10
Fan-out TTLSPromise.all(100 steps)17466 (-1.6%)19157 (-6.6%)19813 (-71%) 💚21056 (-93%) 💚10
STSO1020 steps (inline)140 (-7.3%)189 (-16%) 💚220 (-15%)340 (-35%) 💚1019
WO1020 steps188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚188071 (-18%) 💚1
SLstream latency98 (+4.3%)381 🔴 (+154%) 🔻462 🔴 (+187%) 🔻5193 🔴 (+1367%) 🔻30
SOstream overhead (text)137 (-13%)265 🔴 (-10%)486 (+12%)4163 🔴 (+177%) 🔻30
SOstream overhead (structured)127 (+7.6%)344 🔴 (-15%) 💚366 (-34%) 💚812 (-68%) 💚30
ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • maxRetries=0 disables retries (nextjs-webpack)
  • plainModuleDoneHook resumed via plain API route (o2flow shape) (nextjs-webpack)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 19:04:53Z · abandoned wrun_01M08HS2Z9W5P8TX6GVABJVAWX
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 13, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 world-sim scenario book — 1 fail of 41 total
fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
❌ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim.txt
@shalabhc
shalabhc
marked this pull request as ready for review
August 13, 2026 20:59
@shalabhc
shalabhc
requested a review
from a team
as a code ownerAugust 13, 2026 20:59
karthikscale3
karthikscale3
approved these changes
Aug 13, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
vercelBotand others
added 2 commits
August 17, 2026 18:59
@vercel@shalabhc
[core] Record step execution duration metric
64fccac
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@vercel@shalabhc
[core] Add changeset for the step execution duration metric
f01d0bb
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
@shalabhc
shalabhcforce-pushed
the
identify-datadog-span-duration-metrics
branch
from
0287155 to
f01d0bbCompareAugust 17, 2026 19:00
@shalabhc
shalabhc
merged commit ac51658
into
mainAug 17, 2026

301 of 303 checks passed

@shalabhc
shalabhc
deleted the

identify-datadog-span-duration-metrics

branch
August 17, 2026 21:49
@github-actions
github-actionsBot
commented
Aug 17, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for ac51658 (AI decision).
This commit adds a new observability capability — a workflow.step.execute.duration OpenTelemetry histogram plus a new exported recordStepExecutionDuration API in @workflow/core — rather than fixing a defect on the maintenance line. New metrics and new exported API surface are feature work, so they stay on main regardless of how small or self-contained the change is.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
ac51658d39649e7d8e290924c3bf7b2c7df78b00


This was referenced Aug 17, 2026
 Closed

 Open

 Closed

 Open

 Open

 Merged

VaguelySerious added a commit
that referenced
this pull request
Aug 21, 2026
@VaguelySerious@claude
docs: catch v5 up with main, and with the sealed log
a08bced
Sweep of everything that landed on main since the last content refresh,
plus the spec-7 sealed log from #3634, which is not merged but is the
default for v5.
Spec 7 changes what a World declares, so the upgrade guide's Spec
versions section is rewritten around `mintedSpecVersion()`. The accepted
range widens back to two versions, because a deployment with
`WORKFLOW_SEALED_LOG=0` stamps the one below, and both `SPEC_VERSION_CURRENT`
and `SPEC_VERSION_SUPPORTS_SLOT_IDENTITY` become literals by another name
for the purpose of declaring it: neither follows the setting. The new
section says what a sealed log costs an implementer, which for most is
nothing — a World allocating each position at the commit cannot leave a
hole, so it never seals and never writes a `noop`. The obligation belongs
to a World that pre-assigns positions, and the half that binds everyone
is the reader's.
`WORKFLOW_SEALED_LOG` is documented in runtime-tuning, taken verbatim
from #3634 so whichever lands second is a no-op merge. The deep
reference sections that PR writes, in event-sourcing and the World
allocation contract, are left to it.
Two optional World members shipped after the guide was written and were
missing everywhere:
- `events.createBatch` — in the World reference (interface and contract
note), the upgrade guide's optional surface, and the skill. Implementing
it is the declaration, so the note leads with atomicity per attempt and
what a World must reject.
- `runs.waitForTerminalStatus` — same three places. It is also what makes
`await run.returnValue` a long poll rather than an interval, which is
user-facing, so What's new gets it too.
What's new also gains the batched suspension write and the pre-assigned
positions behind it, both under faster-and-cheaper runs where they belong:
neither has any API surface.
Not addressed: `workflow.step.execute.duration`, the OpenTelemetry
histogram added in #3526, has nowhere to go. The docs have no metrics
reference at all, and inventing one for a single metric is a bigger call
than this sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VaguelySeriousVaguelySerious
mentioned this pull request
Aug 21, 2026
 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@shalabhc@karthikscale3