Skip to content

[world-local] [test-only] Extend test budget on Windows runner for local storage test - #3503

Merged
VaguelySerious merged 1 commit into
mainfrom
peter/windows-preload-timeout
Aug 12, 2026
Merged

[world-local] [test-only] Extend test budget on Windows runner for local storage test#3503
VaguelySerious merged 1 commit into
mainfrom
peter/windows-preload-timeout

Conversation

@VaguelySerious

@VaguelySeriousVaguelySerious commented Aug 12, 2026

Copy link
Copy Markdown
Member

Unit Tests (windows-latest) has been red on main. One of its two independent
failures is packages/world-local/src/storage.test.ts, reported at lines 1284
and 1353 across main runs 31604115603, 31533009514 and 31616799260.

returns the complete preload when run_started is retried writes 999 events
sequentially, then asserts the preload. On the Windows runner each write costs
orders of magnitude more than locally, and the whole file drifted past its
budget: a green run took 143491ms and a failing run 154816ms with the same 245
tests, 8% apart. events.create is linear locally at ~0.95ms across n = 250,
500, 1000, 2000, so nothing got slower per event. It is a marginal budget, not
a step regression.

Two things that look like fixes and are not:

  • Batch the writes. At width 32 it is slower (1368ms vs 988ms sequential):
    the writers contend for the same event slot and re-probe.
  • Shrink the event count. At 501 the test still passes, which is the trap.
    The 1000 is load-bearing: it is the event cache's entry ceiling, so the test
    is the only thing covering the read-past-cache fallback. Below the ceiling it
    goes on passing while silently covering nothing.

So: export MAX_CACHED_EVENT_ENTRIES and size the test from it (ceiling + 1)
rather than restating the number, and raise the timeout. The comment records
why the count is what it is and why batching is not the cheaper option.

Windows CI on this branch confirms world-local now passes entirely,
Test Files 15 passed (15).

The other failure family, packages/core/src/events-consumer.test.ts at lines
1005 and 1123, is not addressed here. It has its own fix in a follow-up PR.

…ner can meet
`Unit Tests (windows-latest)` has been red on main since 2026-08-11 with
Error: Test timed out in 120000ms.
❯ src/storage.test.ts:1284:7
That is `returns the complete preload when run_started is retried`. It writes
a thousand events sequentially, one file write each, then asserts the retry
returns the whole log.
It is a marginal budget rather than a step regression. Same 245 tests either
side of the boundary, whole-file duration on the Windows runner went 143491ms
(last green) to 154816ms (first red), 8% apart. An added fs op per write would
show a far larger jump. On macOS the writes run at ~1ms each and the test
finishes in about a second, so it only ever bites on that runner.
Raise the budget to 300s and say why in the test, including that batching the
writes with `Promise.all` measures slower rather than faster: the writers then
contend for the same event slot and re-probe.
The count is load-bearing and was a bare 999/1001. It sits one past the event
cache so the preload cannot be served from cached entries alone and has to
read at least one back from disk. Below the ceiling the test still passes and
silently stops covering that fallback, so export the ceiling and size the test
from it.
Unit Tests (windows-latest) is an input to E2E Required Check, so this has been
failing the required aggregate on every open PR.
@VaguelySerious
VaguelySerious requested a review from a team as a code ownerAugust 12, 2026 20:01
@changeset-bot

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 104c2f9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 18 packages
NameType
@workflow/world-localPatch
@workflow/cliPatch
@workflow/corePatch
@workflow/vitestPatch
@workflow/webPatch
@workflow/world-postgresPatch
workflowPatch
@workflow/world-testingPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/web-sharedPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 12, 2026 8:05pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 12, 2026 8:05pm
example-workflowReadyReadyPreviewAug 12, 2026 8:05pm
workbench-astro-workflowReadyReadyPreviewAug 12, 2026 8:05pm
workbench-express-workflowReadyReadyPreviewAug 12, 2026 8:05pm
workbench-fastify-workflowReadyReadyPreviewAug 12, 2026 8:05pm
workbench-hono-workflowReadyReadyPreviewAug 12, 2026 8:05pm
workbench-nestjs-workflowReadyReadyPreviewAug 12, 2026 8:05pm
workbench-nitro-workflowReadyReadyPreviewAug 12, 2026 8:05pm
workbench-nuxt-workflowReadyReadyPreviewAug 12, 2026 8:05pm
workbench-python-workflowErrorErrorAug 12, 2026 8:05pm
workbench-sveltekit-workflowReadyReadyPreviewAug 12, 2026 8:05pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 12, 2026 8:05pm
workbench-vite-workflowReadyReadyPreviewAug 12, 2026 8:05pm
workflow-docsReadyReadyPreview, v0Aug 12, 2026 8:05pm
workflow-swc-playgroundReadyReadyPreviewAug 12, 2026 8:05pm
workflow-tarballsReadyReadyPreviewAug 12, 2026 8:05pm
workflow-webReadyReadyPreviewAug 12, 2026 8:05pm

@github-actions

github-actionsBot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (6 failed)

nextjs-turbopack-node (1 failed):

  • error handling catchability step throw of a non-Error value preserves it as cause on the wrapping FatalError

nextjs-webpack-node (1 failed):

  • error handling not registered WorkflowNotRegisteredError fails the run when workflow does not exist

nextjs-webpack-quickjs (1 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41KZVSV65X0GJAK6R5BJ8XM1MD | 🔍 observability

nuxt-quickjs (1 failed):

  • hookGetConflictWorkflow - awaiting hook.getConflict() registers hook without payload | wrun_41KZVSRDTX0GSTKS799S9XVST8 | 🔍 observability

sveltekit-node (1 failed):

  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41KZVSVQHN0GVMDJY9DXA0GAFX | 🔍 observability

sveltekit-quickjs (1 failed):

  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41KZVSVQHN0GVMDJY9DXA0GAFX | 🔍 observability

🐘 Local Postgres (1 failed)

sveltekit-stable-quickjs (1 failed):

  • DurableAgent e2e core tool error recovery

E2E Test Summary

Summary
PassedFailedSkippedTotal
❌ ▲ Vercel Production10036831092
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
❌ 🐘 Local Postgres380915584368
✅ 🪟 Windows31200312
✅ vercel-multi-region270027
Total127717175714535
Details by Category

❌ ▲ Vercel Production

AppPassedFailedSkipped
❌ nextjs-turbopack-node15213
❌ nextjs-webpack-node15213
❌ nextjs-webpack-quickjs15213
✅ nuxt-node128028
❌ nuxt-quickjs127128
❌ sveltekit-node14619
❌ sveltekit-quickjs14619

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

❌ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
❌ sveltekit-stable-quickjs14817
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@github-actions

github-actionsBot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 104c2f9 · Wed, 12 Aug 2026 20:27:47 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1348 (+507%) 🔻1468 🔴 (+31%) 🔻1543 🔴 (+32%) 🔻1895 🔴 (+8.5%)30
TTFSstream1370 (+431%) 🔻1423 🔴 (+29%) 🔻1434 🔴 (+27%) 🔻1517 🔴 (-1.1%)30
TTFShook + stream666 (+80%) 🔻1740 🔴 (+27%) 🔻1774 🔴 (+24%) 🔻1991 🔴 (-57%) 💚30
STSO1020 steps (inline)131 (-4.4%)179 (-15%) 💚207 (-16%) 💚366 (-8.5%)1019
WO1020 steps186478 (-9.1%)186478 (-9.1%)186478 (-9.1%)186478 (-9.1%)1
SLstream latency102 (+11%)129 🔴 (-12%)184 🔴 (+1.7%)289 🔴 (-35%) 💚30
SOstream overhead (text)120 (-1.6%)184 (-31%) 💚200 (-59%) 💚277 (-71%) 💚30
SOstream overhead (structured)129 (+4.0%)207 (-26%) 💚309 (-64%) 💚3074 🔴 (-89%) 💚30
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 204038ms → this run 185007ms (Δ -19031ms, -9%)

 100-150 ms █░░░░┃ main 11 this 184 +173
150-200 ms ██████████████████████░┃ main 643 this 716 +73
200-250 ms ██┃██████ main 277 this 85 -192
250-300 ms ┃█ main 55 this 14 -41
300-350 ms ┃ main 10 this 8 -2
350-400 ms ┃ main 12 this 4 -8
400-450 ms ┃ main 3 this 4 +1
450-500 ms ┃ main 1 this 1 +0
550-600 ms ┃ main 1 this 0 -1
600-650 ms ┃ main 1 this 0 -1
650-700 ms ┃ main 4 this 0 -4
750-800 ms ┃ main 0 this 1 +1
1000-1050 ms ┃ main 1 this 0 -1
4400-4450 ms ┃ main 0 this 1 +1
5000-5050 ms ┃ main 0 this 1 +1
ℹ️ Metric definitions & methodology

The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: is main, marks where this run lands, bridges the gap when this run has more samples in a bucket.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@github-actions

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 Mint-ordered log — 6 fail of 41 total

log=mint-ordered · fence=per-spec

scenariooutcomeeventsvirtreplayviolations
smoke-no-stepscompleted30msok0
smoke-one-stepcompleted60msok0
hook-at-step-startedcompleted120msok0
hook-at-step-completedcompleted120msok0
hook-at-hook-createdcompleted120msok0
deadline-hook-winscompleted71.0hok0
deadline-expirescompleted71.0hok0
long-sleepcompleted1130.0dok0
hook-never-arrivesstalled30msskipped0
step-retries-twicecompleted102.0sok0
parallel-stepscompleted90msok0
hook-on-execution-statecompleted120msok0
peek-hook-before-branchcompleted120msok0
peek-hook-after-branchcompleted120msok0
peek-hook-at-registrationcompleted120msok0
race-hook-before-probecompleted120msok0
race-hook-after-probecompleted120msok0
race-duplicate-deliverycompleted130msok0
attr-hook-before-stepcompleted110msok0
attr-hook-after-stepcompleted110msok0
attr-from-step-bodycompleted130msok0
fork-hook-after-timeoutcompleted141.0mok0
fork-hook-before-timeoutcompleted141.0mok0
count-hook-after-timeoutcompleted171.0mok0
count-hook-before-timeoutcompleted201.0mok0
stale-read-step-count-forkcompleted171.0mMISMATCH1
stale-read-equal-step-countscompleted141.0mMISMATCH1
step-vs-step-forkcompleted120msMISMATCH1
step-vs-step-fork-fencedcompleted120msMISMATCH1
fence-catches-benign-directioncompleted125msok0
in-flight-before-decisioncompleted171.0mMISMATCH1
in-flight-before-decision-countedcompleted201.0mok0
in-flight-after-decisionfailed142.0mMISMATCH1
stale-read-step-count-fork-fencedcompleted201.0mok0
fork-hook-winscompleted131.0mok0
fork-timeout-winscompleted131.0mok0
unclaimed-payload-under-forkcompleted171.0mok0
claimed-payload-under-forkcompleted171.0mok0
writers-independent-step-bodiescompleted120msok0
writers-scripted-tempocompleted120msok0
cancel-mid-stepcancelled70msskipped0

Full trace: world-sim-mint.txt

🟢 Append-only log — 0 fail of 41 total

log=append-only · fence=per-spec

scenariooutcomeeventsvirtreplayviolations
smoke-no-stepscompleted30msok0
smoke-one-stepcompleted60msok0
hook-at-step-startedcompleted120msok0
hook-at-step-completedcompleted120msok0
hook-at-hook-createdcompleted120msok0
deadline-hook-winscompleted71.0hok0
deadline-expirescompleted71.0hok0
long-sleepcompleted1130.0dok0
hook-never-arrivesstalled30msskipped0
step-retries-twicecompleted102.0sok0
parallel-stepscompleted90msok0
hook-on-execution-statecompleted120msok0
peek-hook-before-branchcompleted120msok0
peek-hook-after-branchcompleted120msok0
peek-hook-at-registrationcompleted120msok0
race-hook-before-probecompleted120msok0
race-hook-after-probecompleted120msok0
race-duplicate-deliverycompleted130msok0
attr-hook-before-stepcompleted110msok0
attr-hook-after-stepcompleted110msok0
attr-from-step-bodycompleted130msok0
fork-hook-after-timeoutcompleted141.0mok0
fork-hook-before-timeoutcompleted141.0mok0
count-hook-after-timeoutcompleted171.0mok0
count-hook-before-timeoutcompleted201.0mok0
stale-read-step-count-forkcompleted201.0mok0
stale-read-equal-step-countscompleted141.0mok0
step-vs-step-forkcompleted120msok0
step-vs-step-fork-fencedcompleted120msok0
fence-catches-benign-directioncompleted125msok0
in-flight-before-decisioncompleted171.0mok0
in-flight-before-decision-countedcompleted171.0mok0
in-flight-after-decisioncompleted192.0mok0
stale-read-step-count-fork-fencedcompleted201.0mok0
fork-hook-winscompleted131.0mok0
fork-timeout-winscompleted131.0mok0
unclaimed-payload-under-forkcompleted171.0mok0
claimed-payload-under-forkcompleted171.0mok0
writers-independent-step-bodiescompleted120msok0
writers-scripted-tempocompleted120msok0
cancel-mid-stepcancelled70msskipped0

Full trace: world-sim-append-only.txt

@VaguelySeriousVaguelySerious changed the title [world-local] Give the complete-preload test a budget the Windows runner can meet[world-local] [test-only] Give the complete-preload test a budget the Windows runner can meetAug 12, 2026
@VaguelySeriousVaguelySerious changed the title [world-local] [test-only] Give the complete-preload test a budget the Windows runner can meet[world-local] [test-only] Extend test budget on Windows runner for local storage testAug 12, 2026
@VaguelySerious
VaguelySerious merged commit 8931bee into mainAug 12, 2026
135 of 164 checks passed
@VaguelySerious
VaguelySerious deleted the peter/windows-preload-timeout branch August 12, 2026 20:09
@github-actionsgithub-actionsBot mentioned this pull request Aug 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

No backport to stable for 8931bee (AI decision).

This is a CI-stability fix, but the code it touches does not exist on stable: git show origin/stable:packages/world-local/src/storage.test.ts has no preload tests at all, and origin/stable's events-storage.ts has no event cache / maxCachedEventEntries. The timed-out test and the ceiling constant it now derives from are both main-only, so there is nothing on the maintenance line for this change to fix.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

8931bee41bbc5c2d1d94d923d841cd5413977cf4

@VaguelySeriousVaguelySerious changed the title [world-local] [test-only] Extend test budget on Windows runner for local storage test[world-local][core] Fix the two Windows unit-test failures blocking the required aggregateAug 12, 2026
@VaguelySeriousVaguelySerious changed the title [world-local][core] Fix the two Windows unit-test failures blocking the required aggregate[world-local] [test-only] Extend test budget on Windows runner for local storage testAug 12, 2026
shalabhc added a commit that referenced this pull request Aug 13, 2026
…ner can meet (#3503)
Co-Authored-By: shalabhchaturvedi-7802 <shalabh.chaturvedi@vercel.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@VaguelySerious