Uh oh!
There was an error while loading. Please reload this page.
[world-local] Retry transient EPERM unlink failures on Windows - #3215
Conversation
deleteJSON was the one mutation path in the fs layer that neither used withWindowsRetry nor swallowed unlink errors. On Windows, unlink fails with a share-violation EPERM while a concurrent reader briefly holds the file open — hook polling races deleteAllHooksForRun by design — so a transient EPERM surfaced as a failed operation, e.g. a failed run.cancel(). Wrap the unlink in withWindowsRetry, matching the rename/link/unlink guards the write pipeline already has. ENOENT is not in the retryable set, so the already-deleted tolerance still short-circuits. Signed-off-by: Andrew Barba <barba@hey.com>
🦋 Changeset detectedLatest commit: c294ef9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 18 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
ℹ️ Metric definitions & methodologyBest/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
karthikscale3
left a comment
There was a problem hiding this comment.
No blocking findings. This reuses the existing Windows-only retry policy for deleteJSON, preserves ENOENT tolerance and permanent-error propagation, and adds focused coverage for the transient EPERM path.
Uh oh!
There was an error while loading. Please reload this page.
deleteJSON was the one mutation path in the fs layer that neither used withWindowsRetry nor swallowed unlink errors. On Windows, unlink fails with a share-violation EPERM while a concurrent reader briefly holds the file open — hook polling races deleteAllHooksForRun by design — so a transient EPERM surfaced as a failed operation, e.g. a failed run.cancel(). Wrap the unlink in withWindowsRetry, matching the rename/link/unlink guards the write pipeline already has. ENOENT is not in the retryable set, so the already-deleted tolerance still short-circuits. Signed-off-by: Andrew Barba <barba@hey.com>
Backport PR opened against |
Summary
run.cancel()on the local world can fail withEPERM: operation not permitted, unlinkon Windows when hook cleanup races a concurrent reader. The fs layer already owns the fix —withWindowsRetry, which retriesEPERM/EBUSY/EACCESwith exponential backoff — butdeleteJSONis the one mutation path that neither uses it nor swallows the error.Observed failure
eve CI,
test-integration (windows-latest)on vercel/eve#1387, world-local5.0.0-beta.32(the code is unchanged at current HEAD):Job: https://github.com/vercel/eve/actions/runs/30566760759/job/90953098171
The consuming test's assertions had all passed; a cleanup
run.cancel()threw becausedeleteAllHooksForRununlinked a hook JSON file while another handle (hook polling reading the same file) briefly held it open. On POSIX that unlink succeeds; on Windows an open handle is a share violation andunlinkthrowsEPERM(errno -4048).Why deleteJSON specifically
The write pipeline consistently guards the equivalent operations —
withWindowsRetry(() => fs.rename(...))(fs.ts:415),withWindowsRetry(() => fs.unlink(tempPath), 3)(fs.ts:422, 500),withWindowsRetry(() => fs.link(...))(fs.ts:489, 527) — and every other directfs.unlinkin the package swallows failures (helpers.ts:323,legacy.ts:71,events-storage.ts:2186,index.ts:222).deleteJSONis the only unlink that both skips the retry and propagates, and it sits under run cancellation, so a transient share violation surfaces as a failed cancel. Beyond CI, a realrun.cancel()on a Windows dev machine hits the same race.Change
deleteJSON's unlink inwithWindowsRetry.ENOENTis not in the retryable set, so the existing already-deleted tolerance still short-circuits.process.platformstubbed towin32, unlink rejectingEPERMtwice before succeeding).@workflow/world-localpatch.Testing
pnpm --filter @workflow/world-local test— 493/493pnpm --filter @workflow/world-local exec tsc --noEmitbiome checkclean on the touched files