Uh oh!
There was an error while loading. Please reload this page.
Fix concurrent wait_completed race condition in world-local - #1388
Closed
TooTallNate wants to merge 3 commits into
Closed
Fix concurrent wait_completed race condition in world-local#1388TooTallNate wants to merge 3 commits into
TooTallNate wants to merge 3 commits into
Conversation
Use writeExclusive (O_CREAT|O_EXCL) to atomically prevent concurrent invocations from both completing the same wait. Previously, two concurrent runtime invocations could both read the wait as 'waiting' and both create wait_completed events, causing duplicate events in the event log. On replay, the sleep callback consumed the first and the second was reported as an unconsumed event. Add wait test helpers (createWait, completeWait) and tests for: - Basic wait creation and completion - Duplicate wait_created rejection - Sequential duplicate wait_completed rejection (409) - Concurrent wait_completed race (Promise.allSettled, exactly one wins)
Contributor
🦋 Changeset detectedLatest commit: c3cac83 The changes in this PR will be included in the next version bump. This PR includes changesets to release 18 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Stream Benchmarks(includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
|
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests🐘 Local Postgres (1 failed)vite-stable (1 failed):
🌍 Community Worlds (55 failed)mongodb (3 failed):
redis (2 failed):
turso (50 failed):
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
❌ 🐘 Local Postgres
✅ 🪟 Windows
❌ 🌍 Community Worlds
✅ 📋 Other
❌ Some E2E test jobs failed:
Check the workflow run for details. |
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a concurrency bug in @workflow/world-local event storage where concurrent wait_completed writes could create duplicate events and break replay, by adding an atomic completion-claim mechanism and regression tests.
Changes:
- Add
createWait/completeWaithelpers for storage lifecycle tests. - Update
wait_completedhandling to atomically claim a completion lock file viawriteExclusive. - Add wait lifecycle + concurrent completion regression tests and a patch changeset.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| packages/world-local/src/test-helpers.ts | Adds wait-focused helper functions used by storage tests. |
| packages/world-local/src/storage/events-storage.ts | Introduces exclusive-create lock file to prevent concurrent double-completion. |
| packages/world-local/src/storage.test.ts | Adds wait lifecycle tests, including a concurrent completion regression test. |
| .changeset/fix-concurrent-wait-completed.md | Declares a patch release note for the race-condition fix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
731
to
749
| @@ -732,12 +747,6 @@ export function createEventsStorage( | |||
| status: 404, | |||
| }); | |||
| } | |||
Comment on lines
+725
to
+729
| const lockPath = taggedPath( | ||
| basedir, | ||
| 'waits', | ||
| `${waitCompositeKey}.completed`, | ||
| tag |
When a concurrent invocation already created a wait_completed event, the local events array was missing it (the 409 handler just continued). This could cause the workflow to not see the event during replay. Re-fetch the full event log from the source of truth when any 409 is encountered during wait completion, ensuring the events array has the correct ordering with all events from concurrent invocations.
| events.push(result.event!); | ||
| } catch (err) { | ||
| if (WorkflowAPIError.is(err) && err.status === 409) { | ||
| runtimeLogger.info('Wait already completed, skipping', { |
There was a problem hiding this comment.
If we see a 409 here, wouldn't it mean we can exit this flow early since there's a different flow that created the wait_completed event? 🤔 Unsure. Otherwise LGTM
MemberAuthor
There was a problem hiding this comment.
Ya I had that same optimization in mind. I kinda worry about if that earlier replay might not have the full event log though (i.e. if two steps completed ~simultaneously after the wait had elapsed, first replay only has the first step_completed event but the second one has both?)
There was a problem hiding this comment.
Mmh fair. I think this is likely fine to merge and maybe only leads to addition unnecessary replays. Can we add a TODO in here to maybe log or monitor how often this happen? Might also need to re-examine with all the changes in #1338
VaguelySerious
approved these changes
Mar 15, 2026
VaguelySerious
left a comment
Member
There was a problem hiding this comment.
LGTM let's add a comment or TODO though about re-examining this since it feels like a hack and it also applies to vercel/postgres world despite being a local world fix
pranaygp
commented
Mar 18, 2026
Contributor
I have a new PR here: #1434 where I'm checking for invalid state transitions across local and postgres world and ensuring all of them are tight. That should supersede this PR I think |
pranaygp added a commit
that referenced
this pull request
Mar 18, 2026
Local world fixes: - step_completed/step_failed: use writeExclusive lock to prevent concurrent duplicate terminal transitions - step_started: check for terminal lock file before allowing start - wait_completed: use writeExclusive lock (port from PR #1388) - isStepTerminal: include cancelled status Tests: - Concurrent step_completed race (exactly one succeeds, one gets 409) - Concurrent step_failed race - step_started rejection after concurrent step_completed - Concurrent wait_completed race Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5 tasks
pranaygp added a commit
that referenced
this pull request
Mar 18, 2026
Parallel vitest workers with different tags would collide on untagged lock files, causing spurious 409s. Include the tag in the lock filename (e.g. stepId.terminal.vitest-0) matching the pattern from PR #1388. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
pranaygp added a commit
that referenced
this pull request
Mar 18, 2026
…tions (#1434) * [world-postgres] Fix TOCTOU race in step_started that corrupts event log The step_started UPDATE had no conditional guard on step status, allowing a concurrent execution to revert a completed step back to 'running'. This caused duplicate step_completed events, triggering CORRUPTED_EVENT_LOG. Add notInArray guard to match the existing pattern on step_completed and the DynamoDB conditional expression used in the Vercel world. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [world-postgres] Add atomic terminal-state guards to all entity UPDATEs Add conditional WHERE clauses to match the Vercel world's DynamoDB conditional expressions, preventing TOCTOU races where concurrent requests could bypass pre-validation and write invalid state transitions. Changes: - step_started: add NOT IN (completed, failed, cancelled) guard - step_retrying: add terminal-state guard (was unguarded) - step_completed/step_failed: add cancelled to guard - run_completed/run_failed/run_cancelled: add terminal-state guards - isStepTerminal: include cancelled status Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [world-local] Add atomic terminal-state guards and concurrent race tests Local world fixes: - step_completed/step_failed: use writeExclusive lock to prevent concurrent duplicate terminal transitions - step_started: check for terminal lock file before allowing start - wait_completed: use writeExclusive lock (port from PR #1388) - isStepTerminal: include cancelled status Tests: - Concurrent step_completed race (exactly one succeeds, one gets 409) - Concurrent step_failed race - step_started rejection after concurrent step_completed - Concurrent wait_completed race Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Address PR review feedback - Fix lock file extension: use .lock instead of .json via taggedPath to avoid polluting entity directories with empty JSON files that cause SyntaxError during listing/parsing - Make startedAt update atomic using COALESCE in SQL instead of deriving isFirstStart from the TOCTOU validation read - Split changeset into separate entries per package - Remove unnecessary context from changeset text Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Address review feedback from @TooTallNate - step_started: replace fs.access() with a fresh re-read of the step entity — honest about being best-effort rather than claiming atomicity (local world is dev-only; postgres world has SQL-level atomic guards) - wait_completed: clean up lock file on 404 to avoid leaked lock files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: move lock files to .locks/ subdirectory off basedir Lock files in entity directories (steps/, waits/) broke tests that expect only tagged .json files. Move all locks to basedir/.locks/ and clean them up in clear(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: pass ISO string to COALESCE instead of Date object The postgres driver can't serialize a Date object inside a raw sql template literal. Convert to ISO string for proper parameterization. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: embed tag in lock file names for test isolation Parallel vitest workers with different tags would collide on untagged lock files, causing spurious 409s. Include the tag in the lock filename (e.g. stepId.terminal.vitest-0) matching the pattern from PR #1388. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
world-localwhere two concurrent runtime invocations could both createwait_completedevents for the same wait, causing duplicate events in the event logwait_completedcausedWorkflowRuntimeError: Unconsumed eventduring replay, since the sleep callback consumed the first event and was removed, leaving the second with no consumerwriteExclusive(O_CREAT|O_EXCL) to atomically claim a.completedlock file before transitioning a wait tocompleted— if a concurrent invocation already claimed it, the second gets a 409 which the runtime's existing conflict handler gracefully skipscreateWaitandcompleteWaittest helpersPromise.allSettledwith two simultaneous completions — exactly one succeeds, the other gets 409)