Skip to content

[do not merge] - #3447

Closed
TooTallNate wants to merge 1 commit into
mainfrom
events-consumer-clean-room-rewrite
Closed

[do not merge]#3447
TooTallNate wants to merge 1 commit into
mainfrom
events-consumer-clean-room-rewrite

Conversation

@TooTallNate

@TooTallNateTooTallNate commented Aug 11, 2026•
edited
Loading

Copy link
Copy Markdown
Member

No description provided.

@vercel

vercelBot commented Aug 11, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 11, 2026 3:55am
example-nextjs-workflow-webpackReadyReadyPreviewAug 11, 2026 3:55am
example-workflowReadyReadyPreviewAug 11, 2026 3:55am
workbench-astro-workflowReadyReadyPreviewAug 11, 2026 3:55am
workbench-express-workflowReadyReadyPreviewAug 11, 2026 3:55am
workbench-fastify-workflowReadyReadyPreviewAug 11, 2026 3:55am
workbench-hono-workflowReadyReadyPreviewAug 11, 2026 3:55am
workbench-nestjs-workflowReadyReadyPreviewAug 11, 2026 3:55am
workbench-nitro-workflowReadyReadyPreviewAug 11, 2026 3:55am
workbench-nuxt-workflowReadyReadyPreviewAug 11, 2026 3:55am
workbench-python-workflowErrorErrorAug 11, 2026 3:55am
workbench-sveltekit-workflowReadyReadyPreviewAug 11, 2026 3:55am
workbench-tanstack-start-workflowReadyReadyPreviewAug 11, 2026 3:55am
workbench-vite-workflowReadyReadyPreviewAug 11, 2026 3:55am
workflow-docsReadyReadyPreview, v0Aug 11, 2026 3:55am
workflow-swc-playgroundReadyReadyPreviewAug 11, 2026 3:55am
workflow-tarballsReadyReadyPreviewAug 11, 2026 3:55am
workflow-webReadyReadyPreviewAug 11, 2026 3:55am

@changeset-bot

changeset-botBot commented Aug 11, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 6dc5edd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@TooTallNateTooTallNate added the event-log-race-repro Run the event log race reproduction job label Aug 11, 2026
@github-actions

github-actionsBot commented Aug 11, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 6dc5edd · Tue, 11 Aug 2026 04:10:59 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1328 (+26%) 🔻1422 🔴 (+21%) 🔻1452 🔴 (+21%) 🔻1530 🔴 (-10%)30
TTFSstream278 (-73%) 💚1421 🔴 (+24%) 🔻1433 🔴 (+23%) 🔻1518 🔴 (+26%) 🔻30
TTFShook + stream1268 (+1.6%)1713 🔴 (+25%) 🔻1811 🔴 (+30%) 🔻2081 🔴 (+38%) 🔻30
STSO1020 steps (inline)92 (-4.2%)140 (-15%)158 (-22%) 💚251 (-68%) 💚1019
WO1020 steps139483 (-19%) 💚139483 (-19%) 💚139483 (-19%) 💚139483 (-19%) 💚1
SLstream latency100 (+9.9%)156 🔴 (+1.3%)183 🔴 (+6.4%)477 🔴 (-15%)30
SOstream overhead (text)109 (-17%) 💚175 (-37%) 💚186 (-44%) 💚217 (-72%) 💚30
SOstream overhead (structured)115 (-5.7%)160 (-35%) 💚206 (-34%) 💚273 (-39%) 💚30
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 170410ms → this run 138116ms (Δ -32294ms, -19%)

 50-100 ms ┃ main 1 this 2 +1
100-150 ms █████████████████░░░░░░┃ main 627 this 865 +238
150-200 ms ██┃█████ main 279 this 119 -160
200-250 ms ┃█ main 59 this 22 -37
250-300 ms ┃ main 17 this 7 -10
300-350 ms ┃ main 8 this 1 -7
350-400 ms ┃ main 3 this 1 -2
400-450 ms ┃ main 2 this 1 -1
450-500 ms ┃ main 1 this 0 -1
500-550 ms ┃ main 2 this 0 -2
600-650 ms ┃ main 2 this 0 -2
650-700 ms ┃ main 4 this 0 -4
700-750 ms ┃ main 3 this 0 -3
750-800 ms ┃ main 2 this 0 -2
800-850 ms ┃ main 3 this 1 -2
850-900 ms ┃ main 1 this 0 -1
950-1000 ms ┃ main 1 this 0 -1
1000-1050 ms ┃ main 1 this 0 -1
1250-1300 ms ┃ main 1 this 0 -1
1300-1350 ms ┃ main 1 this 0 -1
1650-1700 ms ┃ main 1 this 0 -1
ℹ️ Metric definitions & methodology

The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: █ is main, ┃ marks where this run lands, ░ bridges the gap when this run has more samples in a bucket.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@github-actions

github-actionsBot commented Aug 11, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

✅ All tests passed

E2E Test Summary

Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production346605904056
✅ 💻 Local Development353605204056
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ vercel-multi-region270027
Total149610222617187
Details by Category

✅ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@github-actions

github-actionsBot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Event Log Race Repro

1 of 14 latest repro runs hit event-log regressions.

Run History

Metric2026-08-11 03:57 UTC #1
logs / deploy
Result1/14 regressions
Total14
completed13
CORRUPTED_EVENT_LOG1
USER_ERROR0
RUNTIME_ERROR0
stuck0
other0
infra0
Config14 runs / step-storm 6, hook-storm 6, hook-sleep 2 / c8 / 6x8
Timingwatchdog 2500ms / step 2200±250ms / stagger 400ms / poke 750ms / timeout 240000ms

Latest Scenario Breakdown

ScenarioTotalcompletedCORRUPTED_EVENT_LOGUSER_ERRORRUNTIME_ERRORstuckotherinfra
step-storm66000000
hook-storm65100000
hook-sleep22000000

Latest Non-Completed Runs

ScenarioAttemptOutcomeStatusError codeRun
hook-storm1CORRUPTED_EVENT_LOGfailedCORRUPTED_EVENT_LOGwrun_41KZQFCK4Y0GZAB8Z80X598MRY

@TooTallNateTooTallNate changed the title core: clean-room rewrite of EventsConsumer[do not merge]Aug 11, 2026
@TooTallNate
TooTallNate deleted the events-consumer-clean-room-rewrite branch August 11, 2026 07:07
This was referenced Aug 11, 2026
Open
Draft
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

event-log-race-reproRun the event log race reproduction job

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@TooTallNate