Uh oh!
There was an error while loading. Please reload this page.
[world-testing] Fix Unit Tests flake after adding event-limit test - #3055
Conversation
🦋 Changeset detectedLatest commit: 23b996c The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📜 Previous results (2)23b996cFri, 24 Jul 2026 18:56:45 GMT · run logs
ae73058Wed, 22 Jul 2026 16:50:59 GMT · run logs
ℹ️ Metric definitions & methodologyBest/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
karthikscale3
left a comment
There was a problem hiding this comment.
One cleanup/lifecycle issue found.
| }); | ||
| onTestFinished(() => { | ||
| proc.kill(); | ||
| void rm(dataDir, { recursive: true, force: true }); |
There was a problem hiding this comment.
proc.kill() only requests termination; it does not wait for the child to exit. On Windows the Local World process may still have files in dataDir open when this rm() runs, so it can reject with EPERM/EBUSY. Because the promise is discarded, that also becomes an unhandled rejection (and the temp directory may leak), potentially replacing the original flake with a teardown flake. Could this hook be made async, await the child's exit/close after killing it, and then await rm() (ideally with maxRetries/retryDelay for Windows)?
There was a problem hiding this comment.
Good catch — fixed in 23b996c, and the race was live on POSIX too, not just theoretical on Windows.
Teardown now waits for the child's exit before removing the dir (bounded at 5s so a child that refuses to die can't hang teardown), retries the removal (maxRetries: 10, retryDelay: 100) for Windows handle release, and awaits it with a .catch that warns instead of throwing — a leaked temp dir shouldn't fail an otherwise passing test.
Measured before/after on macOS, full world-testing suite:
| leaked temp dirs | teardown warnings | unhandled rejections | |
|---|---|---|---|
void rm() (previous) | 3 (one per test file) | — | 0 |
| awaited exit + retries | 0 | 0 | 0 |
The leaked dirs contained files the server wrote afterrm() ran (events/*.json, .locks/steps/*), which is the same race you identified — POSIX unlink just doesn't error on it, so it silently recreated the tree instead of throwing EPERM.
6x concurrent full-suite runs (90 tests) with the fix: all pass, 0 leaked dirs, 0 warnings, 0 unhandled rejections.
TooTallNate
left a comment
There was a problem hiding this comment.
Reviewed and reproduced the validation. Approving.
The diagnosis is the valuable part and it holds up. The untagged recovery sweep re-enqueuing every untagged run in the shared .workflow-data — including another live server's mid-flight runs — is exactly the mechanism that would produce spurious ReplayDivergenceErrors escalating to CORRUPTED_EVENT_LOG after the 3-retry divergence budget, and it squares with the earlier recovery-scoping fix (which excluded tagged runs from untagged sweeps but by design left untagged-vs-untagged sharing in place). The concurrency-dependence also explains the flake's signature perfectly: reproducible under CI's file-level parallelism, clean on solo re-runs.
The implementation gets the fiddly parts right:
- Isolation via
pid + UUIDtemp dirs, with the env spread ordered so an explicitopts.envoverride still wins. - The second commit's teardown ordering is the Windows-critical piece:
kill()only requests termination, so awaiting the actualexit(bounded at 5s with an unref'd timer so a wedged child can't hang teardown) beforermis what prevents the EPERM/EBUSY class — and the retried, warn-don't-failrmcovers the handles-linger-past-exit tail. - I checked for hidden shared-state dependencies: the only world construction lives in the spawned
server.mts(which inherits the env var); every test file interacts purely through the HTTP fetcher, so nothing reads the data dir out-of-band and no test expects state continuity across servers.
Verified locally: 15/15 on a single run, then 3× fully concurrent suite runs all green (a scaled-down version of the PR's 6× validation), and zero workflow-world-testing-* dirs leaked in the temp dir afterward — cleanup demonstrably works. CI is 100/100 including both Windows lanes, which are precisely the lanes this fixes.
One non-blocking thought: the harness is fixed, but the underlying product behavior remains — any two untagged local-world processes sharing a project directory (say, a dev server restarting while its predecessor is mid-drain, or two dev servers by mistake) can still recovery-sweep each other's live runs into a divergence race. If that's not already tracked, it may deserve an issue — e.g. a per-process lease on the data dir, or recovery limited to runs whose last event is older than some liveness horizon. The test flake was the canary; the harness isolation removes the canary rather than the hazard.
Uh oh!
There was an error while loading. Please reload this page.
Backport PR opened against |
) (#3104) Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Resolves conflicts with the precondition-guard backport (#3079) and the world-testing per-server data-dir isolation fix (#3055): - packages/core/src/runtime.ts: keep both error imports (MaxEventsExceededError + PreconditionFailedError); both are used. - packages/world-testing/src/util.mts: keep both spawn env entries; WORKFLOW_LOCAL_DATA_DIR first so opts.env can still override it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ry-2 * origin/main: (292 commits) feat(core): seal forwarded stream writes to the owner's public key (#3098) feat(core): seal hook payloads to the target run's public key (#3096) [e2e] Rebuild the event-log corruption repro around step-count divergence (#3147) feat: decrypt sealed payloads in the dashboard and CLI (#3146) Prewarm only appended replay payloads (#3131) feat: publish each run's X25519 public key on the run entity (#3095) feat(core): route sealed envelopes through the serialization layer (#3094) docs: redirect retired migration-guides URLs to comparisons (#3127) feat(core): add `encp` sealed-box encryption primitive (#3093) chore(core): clarify runtime comments (#3111) Remove obsolete world factory aliases (#3112) feat(core): deterministic sandbox hardening (#3045) Remove retired v1 step route plumbing (#3061) [core] Don't count racing invocations' duplicate step_started events toward the maxRetries ceiling (#3069) [world-testing] Isolate each spawned test server's data directory (#3055) fix: upgrade postcss to >=8.5.18 to address GHSA-r28c-9q8g-f849 (#3102) [next] Respect .gitignore in dev watcher to avoid EMFILE on large monorepos (#3085) [ci] Backport only stability fixes to `stable`, default to claude-opus-5 (#3092) perf(core): immediate leading-edge dispatch for idle streams (flush window default 0) (#3088) Optimize `processImportSpecifier` by computing `shouldFollowImportsFromFile` once per file (#3052) ... # Conflicts: # docs/components/geistdocs/desktop-menu.tsx # docs/components/geistdocs/mobile-menu.tsx # docs/content/docs/v5/cookbook/advanced/child-workflows.mdx # docs/content/docs/v5/cookbook/advanced/upgrading-workflows.mdx # docs/content/docs/v5/cookbook/agent-patterns/agent-cancellation.mdx # docs/content/docs/v5/cookbook/agent-patterns/durable-agent.mdx # docs/content/docs/v5/cookbook/agent-patterns/human-in-the-loop.mdx # docs/content/docs/v5/cookbook/common-patterns/batching.mdx # docs/content/docs/v5/cookbook/common-patterns/idempotency.mdx # docs/content/docs/v5/cookbook/common-patterns/rate-limiting.mdx # docs/content/docs/v5/cookbook/common-patterns/saga.mdx # docs/content/docs/v5/cookbook/common-patterns/scheduling.mdx # docs/content/docs/v5/cookbook/common-patterns/sequential-and-parallel.mdx # docs/content/docs/v5/cookbook/common-patterns/timeouts.mdx # docs/content/docs/v5/cookbook/common-patterns/webhooks.mdx # docs/content/docs/v5/cookbook/common-patterns/workflow-composition.mdx # docs/content/docs/v5/cookbook/index.mdx # docs/content/docs/v5/cookbook/integrations/ai-sdk.mdx # docs/content/docs/v5/cookbook/integrations/chat-sdk.mdx # docs/content/docs/v5/cookbook/integrations/sandbox.mdx # docs/next.config.ts # docs/proxy.ts # docs/scripts/lint.ts # pnpm-lock.yaml # pnpm-workspace.yaml
Summary
Windows Unit Tests failed on main's changeset-release PR (#3028) at the new
event-limit.test.tstest (added in #2986):CORRUPTED_EVENT_LOGmeans the run hitReplayDivergenceError3+ times (REPLAY_DIVERGENCE_MAX_RETRIES) before the max-events guard could cleanly fail it.Root cause
Every
startServer()call inworld-testingspawns a fresh Local World server process against the shared, untagged.workflow-datadirectory (notag, no isolateddataDir). The Local World'sstart()defaultsrecoverActiveRunstotrueand, when untagged, its recovery sweep lists every untagged run in that shared directory — not just the ones the current process created (see the comment inpackages/world-local/src/index.tsaround theisUntaggedfilter).So a server spawned by one test file, starting up while another test file's run is still mid-flight, re-enqueues that foreign run into its own in-process queue and dispatches it against the same on-disk event log — racing the run's actual owner. That produces spurious
ReplayDivergenceErrors under enough concurrency (the full suite, as CI runs it), which is why it reproduced on the Windows job's own head-to-head run but not on a solo re-run of the file.Confirmed locally: running the full
world-testingsuite concurrently (6x in parallel, simulating CI-level file-level concurrency) reliably producedidempotencytimeouts andinline-batches-debugfailures before the fix, and passed cleanly 6/6 after.Fix
Give each spawned server its own
WORKFLOW_LOCAL_DATA_DIR(a unique temp dir), removing the shared state entirely. Cleaned up inonTestFinished.Testing
pnpm build && npx vitest runinpackages/world-testing: 3/3 files, 15/15 tests pass.