Stamp request provenance on step starts - #3567

Merged
alangenfeld merged 1 commit into
mainfrom
alangenfeld/workflow-step-request-provenance-plan
Aug 18, 2026
Merged

Stamp request provenance on step starts#3567
alangenfeld merged 1 commit into
mainfrom
alangenfeld/workflow-step-request-provenance-plan

Conversation

@alangenfeld

@alangenfeldalangenfeld commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Summary & Motivation

step_started now carries the invocation's request ID alongside the compute instance ID, so observability can show both dimensions of where an attempt ran. The two stay independent fields — world-vercel maps requestId onto its analytics vercelId, which the compute instance ID doesn't stand in for.

Test Plan

Unit tests added for the stamping on all three step-start paths, including when no request ID is available.

@alangenfeld
fix(core): stamp request IDs on step starts
ba2471a
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@changeset-bot

changeset-botBot commented Aug 14, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: ba2471a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Aug 14, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 11:15pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 11:15pm
example-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workflow-docsReadyReadyPreview, v0Aug 14, 2026 11:15pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 11:15pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 11:15pm
workflow-webReadyReadyPreviewAug 14, 2026 11:15pm

@github-actions

github-actionsBot commented Aug 14, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit ba2471a · Fri, 14 Aug 2026 23:37:10 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep241 (-70%) 💚1525 🔴 (+35%) 🔻1568 🔴 (+31%) 🔻1819 🔴 (+8.5%)30
TTFSstream286 (-72%) 💚1505 🔴 (+31%) 🔻1543 🔴 (+31%) 🔻1583 🔴 (+19%) 🔻30
TTFShook + stream382 (-27%) 💚1777 🔴 (+25%) 🔻1823 🔴 (+27%) 🔻1925 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9315 (+1.6%)10806 (+12%)10922 (+5.8%)11266 (+6.4%)10
Fan-out TTLSPromise.all(100 steps)18104 (+0.9%)20284 (+3.5%)20915 (+3.0%)21181 (+4.1%)10
STSO1020 steps (inline)128 (-9.2%)444 (-5.3%)484 (-6.7%)716 (+8.2%)1018
STSO1020 steps (queue-hop)21452145214521451
WO1020 steps395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)1
CRTTfirst chunk (pooled)101 (-1.0%)157 (-10%)251 (-38%) 💚967 (+48%) 🔻28

Streams

Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)100 (±0%)5 (±0%)5.1 (±0%)141 (-4%)291 (+68%)405 (-41%)1250 (+14%)217 (+59%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.9 (-2%)334 (±0%)333 (-2%)112 (-15%)189 (-28%)416 (-17%)1123 (+44%)152 (-26%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)88.9 (-1%)16.2 (±0%)16.2 (-1%)157 (-62%)151 (-66%)223 (-63%)583 (-44%)736 (+10%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.7 (±0%)355 (±0%)355 (±0%)152 (+5%)140 (-15%)210 (-3%)496 (+16%)861 (+142%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (±0%)710 (±0%)709 (±0%)124 (-35%)208 (-51%)357 (-47%)876 (-45%)520 (+51%)3
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 408504ms → this run 391566ms (Δ -16938ms, -4%)

 100-150 ms ┃ main 4 this 8 +4
150-200 ms ██┃ main 22 this 26 +4
200-250 ms ████┃█ main 58 this 46 -12
250-300 ms ██████████┃ main 95 this 97 +2
300-350 ms ███████████████░░░░░┃ main 141 this 194 +53
350-400 ms ███████████████████┃ main 186 this 187 +1
400-450 ms ███████████████████░░░░┃ main 176 this 221 +45
450-500 ms █████████████████┃██ main 185 this 163 -22
500-550 ms ███┃██████ main 93 this 38 -55
550-600 ms █┃█ main 31 this 14 -17
600-650 ms ┃█ main 16 this 8 -8
650-700 ms ┃ main 3 this 3 +0
700-750 ms ┃ main 5 this 5 +0
750-800 ms ┃ main 2 this 4 +2
800-850 ms ┃ main 0 this 1 +1
900-950 ms ┃ main 0 this 1 +1
950-1000 ms ┃ main 0 this 1 +1
1050-1100 ms ┃ main 1 this 1 +0
2300-2350 ms ┃ main 1 this 0 -1

1020 steps (queue-hop)

Cumulative STSO time: 2145ms over 1 samples

No main baseline with raw samples yet — showing this run's distribution on its own; the diff appears once a run on main has recorded them.

2000-2200 ms ████████████████████████ steps 1
📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▃▁▁·· 191 (+7%) 138 (+8%) 405 (-41%) 1250 (+14%) 3000
sweep ······▃█▃▁▁·· 171.6 (-12%) 124 (-10%) 416 (-17%) 1123 (+44%) 3000
gw 1x ·····▁▅█▂▁▁·· 132.8 (-50%) 111 (-47%) 223 (-63%) 583 (-44%) 5295
eve 1x ·····▁▇█▂▁▁·· 129.9 (-8%) 105 (-15%) 210 (-3%) 496 (+16%) 5186
eve 2x ·····▁▂█▃▁▁·· 168.4 (-32%) 144 (-28%) 357 (-47%) 876 (-45%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control ▅▃▂█▃▃▃▂▁▂ 144–290ms
sweep ▃▁▆▃▄▄█▇▄▆ 134–211ms
gw 1x ▃█▄▄▂▁▁▁▂▃ 106–204ms
eve 1x █▁▁▄▁▁▃▅▅▁ 108–182ms
eve 2x ▆▄▃▅▁▂█▆▃▁ 127–224ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep ▁▅█▄▃▁▁ 170–176ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control ▂▂█▆▁▁▁▃▂▃ 35–85ms
sweep ▁▅█▄▄▃█▄▄▄ 39–69ms
gw 1x ▂█▄▂▁▁▁▁▁▃ 34–53ms
eve 1x █▁▁▄▂▂▄▄▄▂ 20–39ms
eve 2x █▃▂▁▂▂▆▂▂▄ 25–35ms
ℹ️ Metric definitions & methodology

Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.

The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@github-actions

github-actionsBot commented Aug 14, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

✅ All tests passed

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (hono)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 23:17:44Z · abandoned wrun_01M0191WYSTZ28FPQ4WYYT5VJW

E2E Test Summary

Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development353605204056
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total149780250217480
Details by Category

✅ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600

✅ 🌐 Cross-language Conformance

AppPassedFailedSkipped
✅ python90128

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@github-actions

github-actionsBot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 Mint-ordered log — 3 fail of 41 total

log=mint-ordered · fence=per-spec

scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0

Full trace: world-sim-mint.txt

🟢 Append-only log — 0 fail of 41 total

log=append-only · fence=per-spec

scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0

Full trace: world-sim-append-only.txt

@alangenfeld
alangenfeld marked this pull request as ready for review August 17, 2026 17:00
@alangenfeld
alangenfeld requested a review from a team as a code ownerAugust 17, 2026 17:00
karthikscale3
karthikscale3 approved these changes Aug 18, 2026
View reviewed changes

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@alangenfeld
alangenfeld merged commit 880801b into mainAug 18, 2026
299 of 303 checks passed
@alangenfeld
alangenfeld deleted the alangenfeld/workflow-step-request-provenance-plan branch August 18, 2026 18:21
@github-actions

github-actionsBot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

No backport to stable for 880801b (AI decision).

This commit adds a new requestId field to step_started events (plus a new requestId param on StepExecutorParams and runWorkflowWithQuickJS) so observability can display request provenance alongside the compute instance ID. That is an additive observability enhancement — new data on an existing event and new API surface — not a fix for a defect, crash, or correctness problem on stable. Feature work stays on main, even though the changeset is marked patch and the change is self-contained.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

880801bd4521a0cfd76e8c46ebf560c388ecf30c

This was referenced Aug 18, 2026
Closed
Open
Closed
Open
Merged
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alangenfeld@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Stamp request provenance on step starts - #3567
Merged
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Aug 18, 2026
Merged
Stamp request provenance on step starts#3567
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Conversation
@alangenfeld
@alangenfeldalangenfeld
commented
Aug 14, 2026

Copy link


Copy Markdown

Collaborator
Summary & Motivation
step_started now carries the invocation's request ID alongside the compute instance ID, so observability can show both dimensions of where an attempt ran. The two stay independent fields — world-vercel maps requestId onto its analytics vercelId, which the compute instance ID doesn't stand in for.
Test Plan
Unit tests added for the stamping on all three step-start paths, including when no request ID is available.
@alangenfeld
fix(core): stamp request IDs on step starts
ba2471a
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@changeset-bot
changeset-botBot
commented
Aug 14, 2026

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: ba2471a
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 11:15pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 11:15pm
example-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workflow-docsReadyReadyPreview, v0Aug 14, 2026 11:15pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 11:15pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 11:15pm
workflow-webReadyReadyPreviewAug 14, 2026 11:15pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit ba2471a · Fri, 14 Aug 2026 23:37:10 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep241 (-70%) 💚1525 🔴 (+35%) 🔻1568 🔴 (+31%) 🔻1819 🔴 (+8.5%)30
TTFSstream286 (-72%) 💚1505 🔴 (+31%) 🔻1543 🔴 (+31%) 🔻1583 🔴 (+19%) 🔻30
TTFShook + stream382 (-27%) 💚1777 🔴 (+25%) 🔻1823 🔴 (+27%) 🔻1925 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9315 (+1.6%)10806 (+12%)10922 (+5.8%)11266 (+6.4%)10
Fan-out TTLSPromise.all(100 steps)18104 (+0.9%)20284 (+3.5%)20915 (+3.0%)21181 (+4.1%)10
STSO1020 steps (inline)128 (-9.2%)444 (-5.3%)484 (-6.7%)716 (+8.2%)1018
STSO1020 steps (queue-hop)21452145214521451
WO1020 steps395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)1
CRTTfirst chunk (pooled)101 (-1.0%)157 (-10%)251 (-38%) 💚967 (+48%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)100 (±0%)5 (±0%)5.1 (±0%)141 (-4%)291 (+68%)405 (-41%)1250 (+14%)217 (+59%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.9 (-2%)334 (±0%)333 (-2%)112 (-15%)189 (-28%)416 (-17%)1123 (+44%)152 (-26%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)88.9 (-1%)16.2 (±0%)16.2 (-1%)157 (-62%)151 (-66%)223 (-63%)583 (-44%)736 (+10%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.7 (±0%)355 (±0%)355 (±0%)152 (+5%)140 (-15%)210 (-3%)496 (+16%)861 (+142%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (±0%)710 (±0%)709 (±0%)124 (-35%)208 (-51%)357 (-47%)876 (-45%)520 (+51%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 408504ms → this run 391566ms (Δ -16938ms, -4%)
 100-150 ms ┃ main 4 this 8 +4
150-200 ms ██┃ main 22 this 26 +4
200-250 ms ████┃█ main 58 this 46 -12
250-300 ms ██████████┃ main 95 this 97 +2
300-350 ms ███████████████░░░░░┃ main 141 this 194 +53
350-400 ms ███████████████████┃ main 186 this 187 +1
400-450 ms ███████████████████░░░░┃ main 176 this 221 +45
450-500 ms █████████████████┃██ main 185 this 163 -22
500-550 ms ███┃██████ main 93 this 38 -55
550-600 ms █┃█ main 31 this 14 -17
600-650 ms ┃█ main 16 this 8 -8
650-700 ms ┃ main 3 this 3 +0
700-750 ms ┃ main 5 this 5 +0
750-800 ms ┃ main 2 this 4 +2
800-850 ms ┃ main 0 this 1 +1
900-950 ms ┃ main 0 this 1 +1
950-1000 ms ┃ main 0 this 1 +1
1050-1100 ms ┃ main 1 this 1 +0
2300-2350 ms ┃ main 1 this 0 -1

1020 steps (queue-hop)
Cumulative STSO time: 2145ms over 1 samples
No main baseline with raw samples yet — showing this run's distribution on its own; the diff appears once a run on main has recorded them.
2000-2200 ms ████████████████████████ steps 1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▃▁▁·· 191 (+7%) 138 (+8%) 405 (-41%) 1250 (+14%) 3000
sweep ······▃█▃▁▁·· 171.6 (-12%) 124 (-10%) 416 (-17%) 1123 (+44%) 3000
gw 1x ·····▁▅█▂▁▁·· 132.8 (-50%) 111 (-47%) 223 (-63%) 583 (-44%) 5295
eve 1x ·····▁▇█▂▁▁·· 129.9 (-8%) 105 (-15%) 210 (-3%) 496 (+16%) 5186
eve 2x ·····▁▂█▃▁▁·· 168.4 (-32%) 144 (-28%) 357 (-47%) 876 (-45%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▃▂█▃▃▃▂▁▂ 144–290ms
sweep ▃▁▆▃▄▄█▇▄▆ 134–211ms
gw 1x ▃█▄▄▂▁▁▁▂▃ 106–204ms
eve 1x █▁▁▄▁▁▃▅▅▁ 108–182ms
eve 2x ▆▄▃▅▁▂█▆▃▁ 127–224ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▅█▄▃▁▁ 170–176ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂█▆▁▁▁▃▂▃ 35–85ms
sweep ▁▅█▄▄▃█▄▄▄ 39–69ms
gw 1x ▂█▄▂▁▁▁▁▁▃ 34–53ms
eve 1x █▁▁▄▂▂▄▄▄▂ 20–39ms
eve 2x █▃▂▁▂▂▆▂▂▄ 25–35ms

ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 23:17:44Z · abandoned wrun_01M0191WYSTZ28FPQ4WYYT5VJW
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development353605204056
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total149780250217480
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@alangenfeld
alangenfeld
marked this pull request as ready for review
August 17, 2026 17:00
@alangenfeld
alangenfeld
requested a review
from a team
as a code ownerAugust 17, 2026 17:00
karthikscale3
karthikscale3
approved these changes
Aug 18, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
@alangenfeld
alangenfeld
merged commit 880801b
into
mainAug 18, 2026

299 of 303 checks passed

@alangenfeld
alangenfeld
deleted the

alangenfeld/workflow-step-request-provenance-plan

branch
August 18, 2026 18:21
@github-actions
github-actionsBot
commented
Aug 18, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 880801b (AI decision).
This commit adds a new requestId field to step_started events (plus a new requestId param on StepExecutorParams and runWorkflowWithQuickJS) so observability can display request provenance alongside the compute instance ID. That is an additive observability enhancement — new data on an existing event and new API surface — not a fix for a defect, crash, or correctness problem on stable. Feature work stays on main, even though the changeset is marked patch and the change is self-contained.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
880801bd4521a0cfd76e8c46ebf560c388ecf30c


This was referenced Aug 18, 2026
 Closed

 Open

 Closed

 Open

 Merged

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alangenfeld@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Stamp request provenance on step starts - #3567
Merged
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Aug 18, 2026
Merged
Stamp request provenance on step starts#3567
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Conversation
@alangenfeld
@alangenfeldalangenfeld
commented
Aug 14, 2026

Copy link


Copy Markdown

Collaborator
Summary & Motivation
step_started now carries the invocation's request ID alongside the compute instance ID, so observability can show both dimensions of where an attempt ran. The two stay independent fields — world-vercel maps requestId onto its analytics vercelId, which the compute instance ID doesn't stand in for.
Test Plan
Unit tests added for the stamping on all three step-start paths, including when no request ID is available.
@alangenfeld
fix(core): stamp request IDs on step starts
ba2471a
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@changeset-bot
changeset-botBot
commented
Aug 14, 2026

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: ba2471a
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 11:15pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 11:15pm
example-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workflow-docsReadyReadyPreview, v0Aug 14, 2026 11:15pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 11:15pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 11:15pm
workflow-webReadyReadyPreviewAug 14, 2026 11:15pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit ba2471a · Fri, 14 Aug 2026 23:37:10 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep241 (-70%) 💚1525 🔴 (+35%) 🔻1568 🔴 (+31%) 🔻1819 🔴 (+8.5%)30
TTFSstream286 (-72%) 💚1505 🔴 (+31%) 🔻1543 🔴 (+31%) 🔻1583 🔴 (+19%) 🔻30
TTFShook + stream382 (-27%) 💚1777 🔴 (+25%) 🔻1823 🔴 (+27%) 🔻1925 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9315 (+1.6%)10806 (+12%)10922 (+5.8%)11266 (+6.4%)10
Fan-out TTLSPromise.all(100 steps)18104 (+0.9%)20284 (+3.5%)20915 (+3.0%)21181 (+4.1%)10
STSO1020 steps (inline)128 (-9.2%)444 (-5.3%)484 (-6.7%)716 (+8.2%)1018
STSO1020 steps (queue-hop)21452145214521451
WO1020 steps395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)1
CRTTfirst chunk (pooled)101 (-1.0%)157 (-10%)251 (-38%) 💚967 (+48%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)100 (±0%)5 (±0%)5.1 (±0%)141 (-4%)291 (+68%)405 (-41%)1250 (+14%)217 (+59%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.9 (-2%)334 (±0%)333 (-2%)112 (-15%)189 (-28%)416 (-17%)1123 (+44%)152 (-26%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)88.9 (-1%)16.2 (±0%)16.2 (-1%)157 (-62%)151 (-66%)223 (-63%)583 (-44%)736 (+10%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.7 (±0%)355 (±0%)355 (±0%)152 (+5%)140 (-15%)210 (-3%)496 (+16%)861 (+142%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (±0%)710 (±0%)709 (±0%)124 (-35%)208 (-51%)357 (-47%)876 (-45%)520 (+51%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 408504ms → this run 391566ms (Δ -16938ms, -4%)
 100-150 ms ┃ main 4 this 8 +4
150-200 ms ██┃ main 22 this 26 +4
200-250 ms ████┃█ main 58 this 46 -12
250-300 ms ██████████┃ main 95 this 97 +2
300-350 ms ███████████████░░░░░┃ main 141 this 194 +53
350-400 ms ███████████████████┃ main 186 this 187 +1
400-450 ms ███████████████████░░░░┃ main 176 this 221 +45
450-500 ms █████████████████┃██ main 185 this 163 -22
500-550 ms ███┃██████ main 93 this 38 -55
550-600 ms █┃█ main 31 this 14 -17
600-650 ms ┃█ main 16 this 8 -8
650-700 ms ┃ main 3 this 3 +0
700-750 ms ┃ main 5 this 5 +0
750-800 ms ┃ main 2 this 4 +2
800-850 ms ┃ main 0 this 1 +1
900-950 ms ┃ main 0 this 1 +1
950-1000 ms ┃ main 0 this 1 +1
1050-1100 ms ┃ main 1 this 1 +0
2300-2350 ms ┃ main 1 this 0 -1

1020 steps (queue-hop)
Cumulative STSO time: 2145ms over 1 samples
No main baseline with raw samples yet — showing this run's distribution on its own; the diff appears once a run on main has recorded them.
2000-2200 ms ████████████████████████ steps 1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▃▁▁·· 191 (+7%) 138 (+8%) 405 (-41%) 1250 (+14%) 3000
sweep ······▃█▃▁▁·· 171.6 (-12%) 124 (-10%) 416 (-17%) 1123 (+44%) 3000
gw 1x ·····▁▅█▂▁▁·· 132.8 (-50%) 111 (-47%) 223 (-63%) 583 (-44%) 5295
eve 1x ·····▁▇█▂▁▁·· 129.9 (-8%) 105 (-15%) 210 (-3%) 496 (+16%) 5186
eve 2x ·····▁▂█▃▁▁·· 168.4 (-32%) 144 (-28%) 357 (-47%) 876 (-45%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▃▂█▃▃▃▂▁▂ 144–290ms
sweep ▃▁▆▃▄▄█▇▄▆ 134–211ms
gw 1x ▃█▄▄▂▁▁▁▂▃ 106–204ms
eve 1x █▁▁▄▁▁▃▅▅▁ 108–182ms
eve 2x ▆▄▃▅▁▂█▆▃▁ 127–224ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▅█▄▃▁▁ 170–176ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂█▆▁▁▁▃▂▃ 35–85ms
sweep ▁▅█▄▄▃█▄▄▄ 39–69ms
gw 1x ▂█▄▂▁▁▁▁▁▃ 34–53ms
eve 1x █▁▁▄▂▂▄▄▄▂ 20–39ms
eve 2x █▃▂▁▂▂▆▂▂▄ 25–35ms

ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 23:17:44Z · abandoned wrun_01M0191WYSTZ28FPQ4WYYT5VJW
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development353605204056
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total149780250217480
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@alangenfeld
alangenfeld
marked this pull request as ready for review
August 17, 2026 17:00
@alangenfeld
alangenfeld
requested a review
from a team
as a code ownerAugust 17, 2026 17:00
karthikscale3
karthikscale3
approved these changes
Aug 18, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
@alangenfeld
alangenfeld
merged commit 880801b
into
mainAug 18, 2026

299 of 303 checks passed

@alangenfeld
alangenfeld
deleted the

alangenfeld/workflow-step-request-provenance-plan

branch
August 18, 2026 18:21
@github-actions
github-actionsBot
commented
Aug 18, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 880801b (AI decision).
This commit adds a new requestId field to step_started events (plus a new requestId param on StepExecutorParams and runWorkflowWithQuickJS) so observability can display request provenance alongside the compute instance ID. That is an additive observability enhancement — new data on an existing event and new API surface — not a fix for a defect, crash, or correctness problem on stable. Feature work stays on main, even though the changeset is marked patch and the change is self-contained.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
880801bd4521a0cfd76e8c46ebf560c388ecf30c


This was referenced Aug 18, 2026
 Closed

 Open

 Closed

 Open

 Merged

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alangenfeld@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Stamp request provenance on step starts - #3567
Merged
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Aug 18, 2026
Merged
Stamp request provenance on step starts#3567
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Conversation
@alangenfeld
@alangenfeldalangenfeld
commented
Aug 14, 2026

Copy link


Copy Markdown

Collaborator
Summary & Motivation
step_started now carries the invocation's request ID alongside the compute instance ID, so observability can show both dimensions of where an attempt ran. The two stay independent fields — world-vercel maps requestId onto its analytics vercelId, which the compute instance ID doesn't stand in for.
Test Plan
Unit tests added for the stamping on all three step-start paths, including when no request ID is available.
@alangenfeld
fix(core): stamp request IDs on step starts
ba2471a
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@changeset-bot
changeset-botBot
commented
Aug 14, 2026

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: ba2471a
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 11:15pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 11:15pm
example-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workflow-docsReadyReadyPreview, v0Aug 14, 2026 11:15pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 11:15pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 11:15pm
workflow-webReadyReadyPreviewAug 14, 2026 11:15pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit ba2471a · Fri, 14 Aug 2026 23:37:10 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep241 (-70%) 💚1525 🔴 (+35%) 🔻1568 🔴 (+31%) 🔻1819 🔴 (+8.5%)30
TTFSstream286 (-72%) 💚1505 🔴 (+31%) 🔻1543 🔴 (+31%) 🔻1583 🔴 (+19%) 🔻30
TTFShook + stream382 (-27%) 💚1777 🔴 (+25%) 🔻1823 🔴 (+27%) 🔻1925 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9315 (+1.6%)10806 (+12%)10922 (+5.8%)11266 (+6.4%)10
Fan-out TTLSPromise.all(100 steps)18104 (+0.9%)20284 (+3.5%)20915 (+3.0%)21181 (+4.1%)10
STSO1020 steps (inline)128 (-9.2%)444 (-5.3%)484 (-6.7%)716 (+8.2%)1018
STSO1020 steps (queue-hop)21452145214521451
WO1020 steps395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)1
CRTTfirst chunk (pooled)101 (-1.0%)157 (-10%)251 (-38%) 💚967 (+48%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)100 (±0%)5 (±0%)5.1 (±0%)141 (-4%)291 (+68%)405 (-41%)1250 (+14%)217 (+59%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.9 (-2%)334 (±0%)333 (-2%)112 (-15%)189 (-28%)416 (-17%)1123 (+44%)152 (-26%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)88.9 (-1%)16.2 (±0%)16.2 (-1%)157 (-62%)151 (-66%)223 (-63%)583 (-44%)736 (+10%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.7 (±0%)355 (±0%)355 (±0%)152 (+5%)140 (-15%)210 (-3%)496 (+16%)861 (+142%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (±0%)710 (±0%)709 (±0%)124 (-35%)208 (-51%)357 (-47%)876 (-45%)520 (+51%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 408504ms → this run 391566ms (Δ -16938ms, -4%)
 100-150 ms ┃ main 4 this 8 +4
150-200 ms ██┃ main 22 this 26 +4
200-250 ms ████┃█ main 58 this 46 -12
250-300 ms ██████████┃ main 95 this 97 +2
300-350 ms ███████████████░░░░░┃ main 141 this 194 +53
350-400 ms ███████████████████┃ main 186 this 187 +1
400-450 ms ███████████████████░░░░┃ main 176 this 221 +45
450-500 ms █████████████████┃██ main 185 this 163 -22
500-550 ms ███┃██████ main 93 this 38 -55
550-600 ms █┃█ main 31 this 14 -17
600-650 ms ┃█ main 16 this 8 -8
650-700 ms ┃ main 3 this 3 +0
700-750 ms ┃ main 5 this 5 +0
750-800 ms ┃ main 2 this 4 +2
800-850 ms ┃ main 0 this 1 +1
900-950 ms ┃ main 0 this 1 +1
950-1000 ms ┃ main 0 this 1 +1
1050-1100 ms ┃ main 1 this 1 +0
2300-2350 ms ┃ main 1 this 0 -1

1020 steps (queue-hop)
Cumulative STSO time: 2145ms over 1 samples
No main baseline with raw samples yet — showing this run's distribution on its own; the diff appears once a run on main has recorded them.
2000-2200 ms ████████████████████████ steps 1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▃▁▁·· 191 (+7%) 138 (+8%) 405 (-41%) 1250 (+14%) 3000
sweep ······▃█▃▁▁·· 171.6 (-12%) 124 (-10%) 416 (-17%) 1123 (+44%) 3000
gw 1x ·····▁▅█▂▁▁·· 132.8 (-50%) 111 (-47%) 223 (-63%) 583 (-44%) 5295
eve 1x ·····▁▇█▂▁▁·· 129.9 (-8%) 105 (-15%) 210 (-3%) 496 (+16%) 5186
eve 2x ·····▁▂█▃▁▁·· 168.4 (-32%) 144 (-28%) 357 (-47%) 876 (-45%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▃▂█▃▃▃▂▁▂ 144–290ms
sweep ▃▁▆▃▄▄█▇▄▆ 134–211ms
gw 1x ▃█▄▄▂▁▁▁▂▃ 106–204ms
eve 1x █▁▁▄▁▁▃▅▅▁ 108–182ms
eve 2x ▆▄▃▅▁▂█▆▃▁ 127–224ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▅█▄▃▁▁ 170–176ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂█▆▁▁▁▃▂▃ 35–85ms
sweep ▁▅█▄▄▃█▄▄▄ 39–69ms
gw 1x ▂█▄▂▁▁▁▁▁▃ 34–53ms
eve 1x █▁▁▄▂▂▄▄▄▂ 20–39ms
eve 2x █▃▂▁▂▂▆▂▂▄ 25–35ms

ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 23:17:44Z · abandoned wrun_01M0191WYSTZ28FPQ4WYYT5VJW
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development353605204056
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total149780250217480
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@alangenfeld
alangenfeld
marked this pull request as ready for review
August 17, 2026 17:00
@alangenfeld
alangenfeld
requested a review
from a team
as a code ownerAugust 17, 2026 17:00
karthikscale3
karthikscale3
approved these changes
Aug 18, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
@alangenfeld
alangenfeld
merged commit 880801b
into
mainAug 18, 2026

299 of 303 checks passed

@alangenfeld
alangenfeld
deleted the

alangenfeld/workflow-step-request-provenance-plan

branch
August 18, 2026 18:21
@github-actions
github-actionsBot
commented
Aug 18, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 880801b (AI decision).
This commit adds a new requestId field to step_started events (plus a new requestId param on StepExecutorParams and runWorkflowWithQuickJS) so observability can display request provenance alongside the compute instance ID. That is an additive observability enhancement — new data on an existing event and new API surface — not a fix for a defect, crash, or correctness problem on stable. Feature work stays on main, even though the changeset is marked patch and the change is self-contained.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
880801bd4521a0cfd76e8c46ebf560c388ecf30c


This was referenced Aug 18, 2026
 Closed

 Open

 Closed

 Open

 Merged

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alangenfeld@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Stamp request provenance on step starts - #3567
Merged
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Aug 18, 2026
Merged
Stamp request provenance on step starts#3567
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Conversation
@alangenfeld
@alangenfeldalangenfeld
commented
Aug 14, 2026

Copy link


Copy Markdown

Collaborator
Summary & Motivation
step_started now carries the invocation's request ID alongside the compute instance ID, so observability can show both dimensions of where an attempt ran. The two stay independent fields — world-vercel maps requestId onto its analytics vercelId, which the compute instance ID doesn't stand in for.
Test Plan
Unit tests added for the stamping on all three step-start paths, including when no request ID is available.
@alangenfeld
fix(core): stamp request IDs on step starts
ba2471a
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@changeset-bot
changeset-botBot
commented
Aug 14, 2026

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: ba2471a
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 11:15pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 11:15pm
example-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workflow-docsReadyReadyPreview, v0Aug 14, 2026 11:15pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 11:15pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 11:15pm
workflow-webReadyReadyPreviewAug 14, 2026 11:15pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit ba2471a · Fri, 14 Aug 2026 23:37:10 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep241 (-70%) 💚1525 🔴 (+35%) 🔻1568 🔴 (+31%) 🔻1819 🔴 (+8.5%)30
TTFSstream286 (-72%) 💚1505 🔴 (+31%) 🔻1543 🔴 (+31%) 🔻1583 🔴 (+19%) 🔻30
TTFShook + stream382 (-27%) 💚1777 🔴 (+25%) 🔻1823 🔴 (+27%) 🔻1925 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9315 (+1.6%)10806 (+12%)10922 (+5.8%)11266 (+6.4%)10
Fan-out TTLSPromise.all(100 steps)18104 (+0.9%)20284 (+3.5%)20915 (+3.0%)21181 (+4.1%)10
STSO1020 steps (inline)128 (-9.2%)444 (-5.3%)484 (-6.7%)716 (+8.2%)1018
STSO1020 steps (queue-hop)21452145214521451
WO1020 steps395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)1
CRTTfirst chunk (pooled)101 (-1.0%)157 (-10%)251 (-38%) 💚967 (+48%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)100 (±0%)5 (±0%)5.1 (±0%)141 (-4%)291 (+68%)405 (-41%)1250 (+14%)217 (+59%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.9 (-2%)334 (±0%)333 (-2%)112 (-15%)189 (-28%)416 (-17%)1123 (+44%)152 (-26%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)88.9 (-1%)16.2 (±0%)16.2 (-1%)157 (-62%)151 (-66%)223 (-63%)583 (-44%)736 (+10%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.7 (±0%)355 (±0%)355 (±0%)152 (+5%)140 (-15%)210 (-3%)496 (+16%)861 (+142%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (±0%)710 (±0%)709 (±0%)124 (-35%)208 (-51%)357 (-47%)876 (-45%)520 (+51%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 408504ms → this run 391566ms (Δ -16938ms, -4%)
 100-150 ms ┃ main 4 this 8 +4
150-200 ms ██┃ main 22 this 26 +4
200-250 ms ████┃█ main 58 this 46 -12
250-300 ms ██████████┃ main 95 this 97 +2
300-350 ms ███████████████░░░░░┃ main 141 this 194 +53
350-400 ms ███████████████████┃ main 186 this 187 +1
400-450 ms ███████████████████░░░░┃ main 176 this 221 +45
450-500 ms █████████████████┃██ main 185 this 163 -22
500-550 ms ███┃██████ main 93 this 38 -55
550-600 ms █┃█ main 31 this 14 -17
600-650 ms ┃█ main 16 this 8 -8
650-700 ms ┃ main 3 this 3 +0
700-750 ms ┃ main 5 this 5 +0
750-800 ms ┃ main 2 this 4 +2
800-850 ms ┃ main 0 this 1 +1
900-950 ms ┃ main 0 this 1 +1
950-1000 ms ┃ main 0 this 1 +1
1050-1100 ms ┃ main 1 this 1 +0
2300-2350 ms ┃ main 1 this 0 -1

1020 steps (queue-hop)
Cumulative STSO time: 2145ms over 1 samples
No main baseline with raw samples yet — showing this run's distribution on its own; the diff appears once a run on main has recorded them.
2000-2200 ms ████████████████████████ steps 1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▃▁▁·· 191 (+7%) 138 (+8%) 405 (-41%) 1250 (+14%) 3000
sweep ······▃█▃▁▁·· 171.6 (-12%) 124 (-10%) 416 (-17%) 1123 (+44%) 3000
gw 1x ·····▁▅█▂▁▁·· 132.8 (-50%) 111 (-47%) 223 (-63%) 583 (-44%) 5295
eve 1x ·····▁▇█▂▁▁·· 129.9 (-8%) 105 (-15%) 210 (-3%) 496 (+16%) 5186
eve 2x ·····▁▂█▃▁▁·· 168.4 (-32%) 144 (-28%) 357 (-47%) 876 (-45%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▃▂█▃▃▃▂▁▂ 144–290ms
sweep ▃▁▆▃▄▄█▇▄▆ 134–211ms
gw 1x ▃█▄▄▂▁▁▁▂▃ 106–204ms
eve 1x █▁▁▄▁▁▃▅▅▁ 108–182ms
eve 2x ▆▄▃▅▁▂█▆▃▁ 127–224ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▅█▄▃▁▁ 170–176ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂█▆▁▁▁▃▂▃ 35–85ms
sweep ▁▅█▄▄▃█▄▄▄ 39–69ms
gw 1x ▂█▄▂▁▁▁▁▁▃ 34–53ms
eve 1x █▁▁▄▂▂▄▄▄▂ 20–39ms
eve 2x █▃▂▁▂▂▆▂▂▄ 25–35ms

ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 23:17:44Z · abandoned wrun_01M0191WYSTZ28FPQ4WYYT5VJW
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development353605204056
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total149780250217480
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@alangenfeld
alangenfeld
marked this pull request as ready for review
August 17, 2026 17:00
@alangenfeld
alangenfeld
requested a review
from a team
as a code ownerAugust 17, 2026 17:00
karthikscale3
karthikscale3
approved these changes
Aug 18, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
@alangenfeld
alangenfeld
merged commit 880801b
into
mainAug 18, 2026

299 of 303 checks passed

@alangenfeld
alangenfeld
deleted the

alangenfeld/workflow-step-request-provenance-plan

branch
August 18, 2026 18:21
@github-actions
github-actionsBot
commented
Aug 18, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 880801b (AI decision).
This commit adds a new requestId field to step_started events (plus a new requestId param on StepExecutorParams and runWorkflowWithQuickJS) so observability can display request provenance alongside the compute instance ID. That is an additive observability enhancement — new data on an existing event and new API surface — not a fix for a defect, crash, or correctness problem on stable. Feature work stays on main, even though the changeset is marked patch and the change is self-contained.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
880801bd4521a0cfd76e8c46ebf560c388ecf30c


This was referenced Aug 18, 2026
 Closed

 Open

 Closed

 Open

 Merged

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alangenfeld@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Stamp request provenance on step starts - #3567
Merged
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Aug 18, 2026
Merged
Stamp request provenance on step starts#3567
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Conversation
@alangenfeld
@alangenfeldalangenfeld
commented
Aug 14, 2026

Copy link


Copy Markdown

Collaborator
Summary & Motivation
step_started now carries the invocation's request ID alongside the compute instance ID, so observability can show both dimensions of where an attempt ran. The two stay independent fields — world-vercel maps requestId onto its analytics vercelId, which the compute instance ID doesn't stand in for.
Test Plan
Unit tests added for the stamping on all three step-start paths, including when no request ID is available.
@alangenfeld
fix(core): stamp request IDs on step starts
ba2471a
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@changeset-bot
changeset-botBot
commented
Aug 14, 2026

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: ba2471a
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 11:15pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 11:15pm
example-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workflow-docsReadyReadyPreview, v0Aug 14, 2026 11:15pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 11:15pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 11:15pm
workflow-webReadyReadyPreviewAug 14, 2026 11:15pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit ba2471a · Fri, 14 Aug 2026 23:37:10 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep241 (-70%) 💚1525 🔴 (+35%) 🔻1568 🔴 (+31%) 🔻1819 🔴 (+8.5%)30
TTFSstream286 (-72%) 💚1505 🔴 (+31%) 🔻1543 🔴 (+31%) 🔻1583 🔴 (+19%) 🔻30
TTFShook + stream382 (-27%) 💚1777 🔴 (+25%) 🔻1823 🔴 (+27%) 🔻1925 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9315 (+1.6%)10806 (+12%)10922 (+5.8%)11266 (+6.4%)10
Fan-out TTLSPromise.all(100 steps)18104 (+0.9%)20284 (+3.5%)20915 (+3.0%)21181 (+4.1%)10
STSO1020 steps (inline)128 (-9.2%)444 (-5.3%)484 (-6.7%)716 (+8.2%)1018
STSO1020 steps (queue-hop)21452145214521451
WO1020 steps395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)1
CRTTfirst chunk (pooled)101 (-1.0%)157 (-10%)251 (-38%) 💚967 (+48%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)100 (±0%)5 (±0%)5.1 (±0%)141 (-4%)291 (+68%)405 (-41%)1250 (+14%)217 (+59%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.9 (-2%)334 (±0%)333 (-2%)112 (-15%)189 (-28%)416 (-17%)1123 (+44%)152 (-26%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)88.9 (-1%)16.2 (±0%)16.2 (-1%)157 (-62%)151 (-66%)223 (-63%)583 (-44%)736 (+10%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.7 (±0%)355 (±0%)355 (±0%)152 (+5%)140 (-15%)210 (-3%)496 (+16%)861 (+142%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (±0%)710 (±0%)709 (±0%)124 (-35%)208 (-51%)357 (-47%)876 (-45%)520 (+51%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 408504ms → this run 391566ms (Δ -16938ms, -4%)
 100-150 ms ┃ main 4 this 8 +4
150-200 ms ██┃ main 22 this 26 +4
200-250 ms ████┃█ main 58 this 46 -12
250-300 ms ██████████┃ main 95 this 97 +2
300-350 ms ███████████████░░░░░┃ main 141 this 194 +53
350-400 ms ███████████████████┃ main 186 this 187 +1
400-450 ms ███████████████████░░░░┃ main 176 this 221 +45
450-500 ms █████████████████┃██ main 185 this 163 -22
500-550 ms ███┃██████ main 93 this 38 -55
550-600 ms █┃█ main 31 this 14 -17
600-650 ms ┃█ main 16 this 8 -8
650-700 ms ┃ main 3 this 3 +0
700-750 ms ┃ main 5 this 5 +0
750-800 ms ┃ main 2 this 4 +2
800-850 ms ┃ main 0 this 1 +1
900-950 ms ┃ main 0 this 1 +1
950-1000 ms ┃ main 0 this 1 +1
1050-1100 ms ┃ main 1 this 1 +0
2300-2350 ms ┃ main 1 this 0 -1

1020 steps (queue-hop)
Cumulative STSO time: 2145ms over 1 samples
No main baseline with raw samples yet — showing this run's distribution on its own; the diff appears once a run on main has recorded them.
2000-2200 ms ████████████████████████ steps 1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▃▁▁·· 191 (+7%) 138 (+8%) 405 (-41%) 1250 (+14%) 3000
sweep ······▃█▃▁▁·· 171.6 (-12%) 124 (-10%) 416 (-17%) 1123 (+44%) 3000
gw 1x ·····▁▅█▂▁▁·· 132.8 (-50%) 111 (-47%) 223 (-63%) 583 (-44%) 5295
eve 1x ·····▁▇█▂▁▁·· 129.9 (-8%) 105 (-15%) 210 (-3%) 496 (+16%) 5186
eve 2x ·····▁▂█▃▁▁·· 168.4 (-32%) 144 (-28%) 357 (-47%) 876 (-45%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▃▂█▃▃▃▂▁▂ 144–290ms
sweep ▃▁▆▃▄▄█▇▄▆ 134–211ms
gw 1x ▃█▄▄▂▁▁▁▂▃ 106–204ms
eve 1x █▁▁▄▁▁▃▅▅▁ 108–182ms
eve 2x ▆▄▃▅▁▂█▆▃▁ 127–224ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▅█▄▃▁▁ 170–176ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂█▆▁▁▁▃▂▃ 35–85ms
sweep ▁▅█▄▄▃█▄▄▄ 39–69ms
gw 1x ▂█▄▂▁▁▁▁▁▃ 34–53ms
eve 1x █▁▁▄▂▂▄▄▄▂ 20–39ms
eve 2x █▃▂▁▂▂▆▂▂▄ 25–35ms

ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 23:17:44Z · abandoned wrun_01M0191WYSTZ28FPQ4WYYT5VJW
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development353605204056
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total149780250217480
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@alangenfeld
alangenfeld
marked this pull request as ready for review
August 17, 2026 17:00
@alangenfeld
alangenfeld
requested a review
from a team
as a code ownerAugust 17, 2026 17:00
karthikscale3
karthikscale3
approved these changes
Aug 18, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
@alangenfeld
alangenfeld
merged commit 880801b
into
mainAug 18, 2026

299 of 303 checks passed

@alangenfeld
alangenfeld
deleted the

alangenfeld/workflow-step-request-provenance-plan

branch
August 18, 2026 18:21
@github-actions
github-actionsBot
commented
Aug 18, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 880801b (AI decision).
This commit adds a new requestId field to step_started events (plus a new requestId param on StepExecutorParams and runWorkflowWithQuickJS) so observability can display request provenance alongside the compute instance ID. That is an additive observability enhancement — new data on an existing event and new API surface — not a fix for a defect, crash, or correctness problem on stable. Feature work stays on main, even though the changeset is marked patch and the change is self-contained.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
880801bd4521a0cfd76e8c46ebf560c388ecf30c


This was referenced Aug 18, 2026
 Closed

 Open

 Closed

 Open

 Merged

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alangenfeld@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Stamp request provenance on step starts - #3567
Merged
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Aug 18, 2026
Merged
Stamp request provenance on step starts#3567
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Conversation
@alangenfeld
@alangenfeldalangenfeld
commented
Aug 14, 2026

Copy link


Copy Markdown

Collaborator
Summary & Motivation
step_started now carries the invocation's request ID alongside the compute instance ID, so observability can show both dimensions of where an attempt ran. The two stay independent fields — world-vercel maps requestId onto its analytics vercelId, which the compute instance ID doesn't stand in for.
Test Plan
Unit tests added for the stamping on all three step-start paths, including when no request ID is available.
@alangenfeld
fix(core): stamp request IDs on step starts
ba2471a
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@changeset-bot
changeset-botBot
commented
Aug 14, 2026

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: ba2471a
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 11:15pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 11:15pm
example-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workflow-docsReadyReadyPreview, v0Aug 14, 2026 11:15pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 11:15pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 11:15pm
workflow-webReadyReadyPreviewAug 14, 2026 11:15pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit ba2471a · Fri, 14 Aug 2026 23:37:10 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep241 (-70%) 💚1525 🔴 (+35%) 🔻1568 🔴 (+31%) 🔻1819 🔴 (+8.5%)30
TTFSstream286 (-72%) 💚1505 🔴 (+31%) 🔻1543 🔴 (+31%) 🔻1583 🔴 (+19%) 🔻30
TTFShook + stream382 (-27%) 💚1777 🔴 (+25%) 🔻1823 🔴 (+27%) 🔻1925 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9315 (+1.6%)10806 (+12%)10922 (+5.8%)11266 (+6.4%)10
Fan-out TTLSPromise.all(100 steps)18104 (+0.9%)20284 (+3.5%)20915 (+3.0%)21181 (+4.1%)10
STSO1020 steps (inline)128 (-9.2%)444 (-5.3%)484 (-6.7%)716 (+8.2%)1018
STSO1020 steps (queue-hop)21452145214521451
WO1020 steps395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)1
CRTTfirst chunk (pooled)101 (-1.0%)157 (-10%)251 (-38%) 💚967 (+48%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)100 (±0%)5 (±0%)5.1 (±0%)141 (-4%)291 (+68%)405 (-41%)1250 (+14%)217 (+59%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.9 (-2%)334 (±0%)333 (-2%)112 (-15%)189 (-28%)416 (-17%)1123 (+44%)152 (-26%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)88.9 (-1%)16.2 (±0%)16.2 (-1%)157 (-62%)151 (-66%)223 (-63%)583 (-44%)736 (+10%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.7 (±0%)355 (±0%)355 (±0%)152 (+5%)140 (-15%)210 (-3%)496 (+16%)861 (+142%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (±0%)710 (±0%)709 (±0%)124 (-35%)208 (-51%)357 (-47%)876 (-45%)520 (+51%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 408504ms → this run 391566ms (Δ -16938ms, -4%)
 100-150 ms ┃ main 4 this 8 +4
150-200 ms ██┃ main 22 this 26 +4
200-250 ms ████┃█ main 58 this 46 -12
250-300 ms ██████████┃ main 95 this 97 +2
300-350 ms ███████████████░░░░░┃ main 141 this 194 +53
350-400 ms ███████████████████┃ main 186 this 187 +1
400-450 ms ███████████████████░░░░┃ main 176 this 221 +45
450-500 ms █████████████████┃██ main 185 this 163 -22
500-550 ms ███┃██████ main 93 this 38 -55
550-600 ms █┃█ main 31 this 14 -17
600-650 ms ┃█ main 16 this 8 -8
650-700 ms ┃ main 3 this 3 +0
700-750 ms ┃ main 5 this 5 +0
750-800 ms ┃ main 2 this 4 +2
800-850 ms ┃ main 0 this 1 +1
900-950 ms ┃ main 0 this 1 +1
950-1000 ms ┃ main 0 this 1 +1
1050-1100 ms ┃ main 1 this 1 +0
2300-2350 ms ┃ main 1 this 0 -1

1020 steps (queue-hop)
Cumulative STSO time: 2145ms over 1 samples
No main baseline with raw samples yet — showing this run's distribution on its own; the diff appears once a run on main has recorded them.
2000-2200 ms ████████████████████████ steps 1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▃▁▁·· 191 (+7%) 138 (+8%) 405 (-41%) 1250 (+14%) 3000
sweep ······▃█▃▁▁·· 171.6 (-12%) 124 (-10%) 416 (-17%) 1123 (+44%) 3000
gw 1x ·····▁▅█▂▁▁·· 132.8 (-50%) 111 (-47%) 223 (-63%) 583 (-44%) 5295
eve 1x ·····▁▇█▂▁▁·· 129.9 (-8%) 105 (-15%) 210 (-3%) 496 (+16%) 5186
eve 2x ·····▁▂█▃▁▁·· 168.4 (-32%) 144 (-28%) 357 (-47%) 876 (-45%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▃▂█▃▃▃▂▁▂ 144–290ms
sweep ▃▁▆▃▄▄█▇▄▆ 134–211ms
gw 1x ▃█▄▄▂▁▁▁▂▃ 106–204ms
eve 1x █▁▁▄▁▁▃▅▅▁ 108–182ms
eve 2x ▆▄▃▅▁▂█▆▃▁ 127–224ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▅█▄▃▁▁ 170–176ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂█▆▁▁▁▃▂▃ 35–85ms
sweep ▁▅█▄▄▃█▄▄▄ 39–69ms
gw 1x ▂█▄▂▁▁▁▁▁▃ 34–53ms
eve 1x █▁▁▄▂▂▄▄▄▂ 20–39ms
eve 2x █▃▂▁▂▂▆▂▂▄ 25–35ms

ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 23:17:44Z · abandoned wrun_01M0191WYSTZ28FPQ4WYYT5VJW
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development353605204056
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total149780250217480
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@alangenfeld
alangenfeld
marked this pull request as ready for review
August 17, 2026 17:00
@alangenfeld
alangenfeld
requested a review
from a team
as a code ownerAugust 17, 2026 17:00
karthikscale3
karthikscale3
approved these changes
Aug 18, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
@alangenfeld
alangenfeld
merged commit 880801b
into
mainAug 18, 2026

299 of 303 checks passed

@alangenfeld
alangenfeld
deleted the

alangenfeld/workflow-step-request-provenance-plan

branch
August 18, 2026 18:21
@github-actions
github-actionsBot
commented
Aug 18, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 880801b (AI decision).
This commit adds a new requestId field to step_started events (plus a new requestId param on StepExecutorParams and runWorkflowWithQuickJS) so observability can display request provenance alongside the compute instance ID. That is an additive observability enhancement — new data on an existing event and new API surface — not a fix for a defect, crash, or correctness problem on stable. Feature work stays on main, even though the changeset is marked patch and the change is self-contained.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
880801bd4521a0cfd76e8c46ebf560c388ecf30c


This was referenced Aug 18, 2026
 Closed

 Open

 Closed

 Open

 Merged

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alangenfeld@karthikscale3

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Stamp request provenance on step starts - #3567
Merged
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Aug 18, 2026
Merged
Stamp request provenance on step starts#3567
alangenfeld merged 1 commit into
mainfrom 
alangenfeld/workflow-step-request-provenance-plan
Conversation
@alangenfeld
@alangenfeldalangenfeld
commented
Aug 14, 2026

Copy link


Copy Markdown

Collaborator
Summary & Motivation
step_started now carries the invocation's request ID alongside the compute instance ID, so observability can show both dimensions of where an attempt ran. The two stay independent fields — world-vercel maps requestId onto its analytics vercelId, which the compute instance ID doesn't stand in for.
Test Plan
Unit tests added for the stamping on all three step-start paths, including when no request ID is available.
@alangenfeld
fix(core): stamp request IDs on step starts
ba2471a
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@changeset-bot
changeset-botBot
commented
Aug 14, 2026

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: ba2471a
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 11:15pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 11:15pm
example-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 11:15pm
workflow-docsReadyReadyPreview, v0Aug 14, 2026 11:15pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 11:15pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 11:15pm
workflow-webReadyReadyPreviewAug 14, 2026 11:15pm
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit ba2471a · Fri, 14 Aug 2026 23:37:10 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep241 (-70%) 💚1525 🔴 (+35%) 🔻1568 🔴 (+31%) 🔻1819 🔴 (+8.5%)30
TTFSstream286 (-72%) 💚1505 🔴 (+31%) 🔻1543 🔴 (+31%) 🔻1583 🔴 (+19%) 🔻30
TTFShook + stream382 (-27%) 💚1777 🔴 (+25%) 🔻1823 🔴 (+27%) 🔻1925 🔴 (+20%) 🔻30
Fan-out TTFSPromise.all(100 steps)9315 (+1.6%)10806 (+12%)10922 (+5.8%)11266 (+6.4%)10
Fan-out TTLSPromise.all(100 steps)18104 (+0.9%)20284 (+3.5%)20915 (+3.0%)21181 (+4.1%)10
STSO1020 steps (inline)128 (-9.2%)444 (-5.3%)484 (-6.7%)716 (+8.2%)1018
STSO1020 steps (queue-hop)21452145214521451
WO1020 steps395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)395073 (-3.3%)1
CRTTfirst chunk (pooled)101 (-1.0%)157 (-10%)251 (-38%) 💚967 (+48%) 🔻28
Streams
Scenariowr c/srd c/swr KiB/srd KiB/sCRTT 1stp75p90p99CDV maxiters
paced control (100/s, 60B)100 (±0%)100 (±0%)5 (±0%)5.1 (±0%)141 (-4%)291 (+68%)405 (-41%)1250 (+14%)217 (+59%)10
size sweep (100/s, 160B-12KB)100 (±0%)99.9 (-2%)334 (±0%)333 (-2%)112 (-15%)189 (-28%)416 (-17%)1123 (+44%)152 (-26%)10
replay gateway-gpt-5.4-nano-2000t (1x)89.2 (±0%)88.9 (-1%)16.2 (±0%)16.2 (-1%)157 (-62%)151 (-66%)223 (-63%)583 (-44%)736 (+10%)3
replay eve-gpt-5.6-sol-2000t (1x)54.7 (±0%)54.7 (±0%)355 (±0%)355 (±0%)152 (+5%)140 (-15%)210 (-3%)496 (+16%)861 (+142%)2
replay eve-gpt-5.6-sol-2000t (2x)109 (±0%)109 (±0%)710 (±0%)709 (±0%)124 (-35%)208 (-51%)357 (-47%)876 (-45%)520 (+51%)3
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 408504ms → this run 391566ms (Δ -16938ms, -4%)
 100-150 ms ┃ main 4 this 8 +4
150-200 ms ██┃ main 22 this 26 +4
200-250 ms ████┃█ main 58 this 46 -12
250-300 ms ██████████┃ main 95 this 97 +2
300-350 ms ███████████████░░░░░┃ main 141 this 194 +53
350-400 ms ███████████████████┃ main 186 this 187 +1
400-450 ms ███████████████████░░░░┃ main 176 this 221 +45
450-500 ms █████████████████┃██ main 185 this 163 -22
500-550 ms ███┃██████ main 93 this 38 -55
550-600 ms █┃█ main 31 this 14 -17
600-650 ms ┃█ main 16 this 8 -8
650-700 ms ┃ main 3 this 3 +0
700-750 ms ┃ main 5 this 5 +0
750-800 ms ┃ main 2 this 4 +2
800-850 ms ┃ main 0 this 1 +1
900-950 ms ┃ main 0 this 1 +1
950-1000 ms ┃ main 0 this 1 +1
1050-1100 ms ┃ main 1 this 1 +0
2300-2350 ms ┃ main 1 this 0 -1

1020 steps (queue-hop)
Cumulative STSO time: 2145ms over 1 samples
No main baseline with raw samples yet — showing this run's distribution on its own; the diff appears once a run on main has recorded them.
2000-2200 ms ████████████████████████ steps 1

📈 CRTT drill-down vs main (RTT distributions & profiles)
variant RTT 1ms→5s+ avg p50 p90 p99 n
control ······▂█▃▁▁·· 191 (+7%) 138 (+8%) 405 (-41%) 1250 (+14%) 3000
sweep ······▃█▃▁▁·· 171.6 (-12%) 124 (-10%) 416 (-17%) 1123 (+44%) 3000
gw 1x ·····▁▅█▂▁▁·· 132.8 (-50%) 111 (-47%) 223 (-63%) 583 (-44%) 5295
eve 1x ·····▁▇█▂▁▁·· 129.9 (-8%) 105 (-15%) 210 (-3%) 496 (+16%) 5186
eve 2x ·····▁▂█▃▁▁·· 168.4 (-32%) 144 (-28%) 357 (-47%) 876 (-45%) 7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):
control ▅▃▂█▃▃▃▂▁▂ 144–290ms
sweep ▃▁▆▃▄▄█▇▄▆ 134–211ms
gw 1x ▃█▄▄▂▁▁▁▂▃ 106–204ms
eve 1x █▁▁▄▁▁▃▅▅▁ 108–182ms
eve 2x ▆▄▃▅▁▂█▆▃▁ 127–224ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):
sweep ▁▅█▄▃▁▁ 170–176ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):
control ▂▂█▆▁▁▁▃▂▃ 35–85ms
sweep ▁▅█▄▄▃█▄▄▄ 39–69ms
gw 1x ▂█▄▂▁▁▁▁▁▃ 34–53ms
eve 1x █▁▁▄▂▂▄▄▄▂ 20–39ms
eve 2x █▃▂▁▂▂▆▂▂▄ 25–35ms

ℹ️ Metric definitions & methodology
Streams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.
The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.
The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles
Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000teaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600
All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).
Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.
@github-actions
github-actionsBot
commented
Aug 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
⚠️ Flaky E2E Tests (passed on retry)
These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (hono)
🛠 Infra Events (absorbed by the harness)
Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.
  • run-pickup-stall · addTenWorkflow (tanstack-start) · at 23:17:44Z · abandoned wrun_01M0191WYSTZ28FPQ4WYYT5VJW
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development353605204056
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total149780250217480
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@alangenfeld
alangenfeld
marked this pull request as ready for review
August 17, 2026 17:00
@alangenfeld
alangenfeld
requested a review
from a team
as a code ownerAugust 17, 2026 17:00
karthikscale3
karthikscale3
approved these changes
Aug 18, 2026
View reviewed changes
@karthikscale3karthikscale3
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good
@alangenfeld
alangenfeld
merged commit 880801b
into
mainAug 18, 2026

299 of 303 checks passed

@alangenfeld
alangenfeld
deleted the

alangenfeld/workflow-step-request-provenance-plan

branch
August 18, 2026 18:21
@github-actions
github-actionsBot
commented
Aug 18, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 880801b (AI decision).
This commit adds a new requestId field to step_started events (plus a new requestId param on StepExecutorParams and runWorkflowWithQuickJS) so observability can display request provenance alongside the compute instance ID. That is an additive observability enhancement — new data on an existing event and new API surface — not a fix for a defect, crash, or correctness problem on stable. Feature work stays on main, even though the changeset is marked patch and the change is self-contained.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
880801bd4521a0cfd76e8c46ebf560c388ecf30c


This was referenced Aug 18, 2026
 Closed

 Open

 Closed

 Open

 Merged

 Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@karthikscale3karthikscale3karthikscale3 approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alangenfeld@karthikscale3