Uh oh!
There was an error while loading. Please reload this page.
core: write-ahead execution claims replace shutdown-hook suspension - #41800
Merged
Conversation
Restart continuity previously depended on a graceful-shutdown finalizer marking active Sessions as suspended. Any death without teardown (crash, SIGKILL, isolate eviction) skipped the finalizer and lost the turn. The claim is now written when execution starts, in the same transaction as the started event. Terminals release it on commit — except shutdown interruption, which preserves it so graceful restart and unclean death leave the same durable signature. The boot sweep resumes orphaned claims, defers Sessions with recent message activity (managed-server handoff overlap), and terminalizes a turn that exhausts its durable resume budget instead of crash-looping. suspendActiveSessions and store.suspend are deleted; graceful shutdown is an optimization, not a correctness requirement. One schema addition: session_v2.resume_attempts.
Review findings: consuming the claim before the resumed turn re-claimed it reopened the crash window this model exists to close, and let the sweep release the claim of a turn a user prompt had just revived. The claim now survives the entire resume path — only terminal events release it. The grace/redrive machinery defended against a successor server sweeping while its predecessor still drained, but the managed protocol already excludes that: kill/evict confirm the previous PID is dead before a contender spawns, the registration lock admits one managed server, and unregistered servers never sweep. Deleted along with lastActivityAt. Child (subagent) Sessions are excluded from the sweep: a resumed parent re-runs its tool call and spawns fresh children.
Pin time_updated in claim/release/countResume so claim bookkeeping never counts as user activity (matching the applyUsage precedent) — previously an interrupt or a boot sweep reordered the session list. Clear orphaned child claims at sweep time: children are never resumed, so no terminal would ever release them. Treat a missing row in countResume as a deleted Session and skip, instead of masking it as attempt zero. Hoist the exhausted-budget error literal, drop the unused configured() factory, and tidy the tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Restart continuity for in-flight turns currently depends on a graceful-shutdown finalizer:
suspendActiveSessionsmarks active Sessions moments before the process exits, and the next boot resumes whatever was marked. Any death without teardown — crash, SIGKILL, OOM, isolate eviction — skips the finalizer and silently loses the turn. For embedders whose runtime evicts mid-turn as steady state (workerd Durable Objects), that is the common case, not the edge case.This PR inverts when the durable mark is written. The claim is recorded when execution starts, in the same transaction as the
Execution.Startedevent, and terminals release it on commit. Recovery becomes a property of the database: a claim with no terminal is the signature of a dead process, regardless of how it died. No shutdown hook participates in correctness anymore.Before / After
Before: a turn is draining when the process dies uncleanly. The shutdown finalizer never runs, so
time_suspendedis never set. The next server boots,resumeSuspendedSessionsfinds nothing, and the turn is gone — the user's session sits idle with a half-finished assistant message and no continuation. Only gracefulSIGINT/SIGTERMrestarts recovered.Separately, a user-cancelled turn could resurrect:
Execution.Interruptednever cleared suspension, so a cancel that raced a shutdown left the suspension mark in place and the next boot resumed a turn the user had explicitly killed.After: the claim exists while the turn runs. Crash, SIGKILL, eviction, and graceful restart all leave the same durable state, and the next boot resumes the turn with a synthetic continuation. A user interrupt (or a superseding execution) releases the claim on commit, so a cancelled turn can never resurrect. A turn that keeps dying without completing exhausts a durable resume budget (default 10) and is terminalized with
Execution.Failedinstead of crash-looping forever.How
packages/core/src/session/execution.ts—clearSuspensionOnCommitbecomes aclaimOnCommit/releaseOnCommitpair.Startedclaims;SucceededandFailedrelease;Interruptedreleases foruserandsupersededreasons but deliberately preserves the claim forshutdown, so a graceful restart looks identical to an unclean death.packages/core/src/session/store.ts—store.suspend(bulk, shutdown-time) andconsumeSuspendedare deleted. New primitives:claim(null-guarded so re-claims are zero-write no-ops),release(clears claim + resets the attempt counter),countResume(durable increment; reports a deleted Session asundefinedso the sweep skips it),releaseChildClaims. All claim bookkeeping pinstime_updated(theapplyUsageprecedent) so it never counts as user activity for session ordering.listSuspendedreturns only top-level Sessions, and the sweep clears orphaned child claims outright: children are never resumed independently — a resumed parent re-runs its tool call and spawns fresh children.packages/core/src/session/execution/restart.ts— the sweep resumes orphaned claims without ever clearing them: only terminal events release a claim, so a death anywhere in the resume path leaves the same orphaned claim for the next boot. Per claim: skip if locally active → durably count the attempt before resuming (a crash inside the resumed turn cannot dodge the budget) → terminalize pastmaxAttempts→ otherwise publish the synthetic continuation and resume, forked so boot never waits on resumed turns (resuming an already-live Session joins its execution).suspendActiveSessionsis gone from the interface.The sweep assumes every orphaned claim's owner is dead. The managed-server protocol guarantees this:
kill/evictin the client service confirm the previous PID is gone before a contender is spawned, the registration lock admits one managed server at a time, and unregistered servers sharing the database never sweep.packages/server/src/process.ts— the shutdown finalizer is deleted. Boot-timeresumeSuspendedSessionsis the whole contract.Schema — one column,
session_v2.resume_attempts integer not null default 0(sql.ts, generated migration20260811161259_execution_claim_attempts).time_suspendedis reinterpreted as the claim timestamp; no rename, no data migration — an existing suspension from an old server reads as an orphaned claim and resumes exactly as before.Flow
Scope
Deliberately not covered here:
--standalonealongside the managed server) have no fencing for any concurrent session operation today; the sweep inherits that, it does not create it. If shared-database multi-server becomes real, the principled fix is an owner ID + liveness check on the claim, for the whole class at once.settleStaleToolCallsbounds the damage.progress: trueflag on event definitions, not event-name string matching.Testing
packages/core: full suite — 1645 pass / 0 fail (172 files).packages/server: 17 pass / 0 fail.bun turbo typecheck --filter=@opencode-ai/core --filter=@opencode-ai/serverclean;oxlintclean on changed files.test/session-execution.test.ts: claim exists while the drain runs and survives teardown interruption; natural completion releases; user interrupt releases (no resurrection); the sweep leaves locally-active Sessions untouched; the sweep never consumes the claim it recovers (attempt counted durably before the turn runs, claim intact through a second teardown); budget exhaustion terminalizes withExecution.Failedand resets the counter; child Sessions are excluded from the sweep.