Skip to content

core: write-ahead execution claims replace shutdown-hook suspension - #41800

Merged
kitlangton merged 4 commits into
v2from
execution-claim-journal
Aug 11, 2026
Merged

core: write-ahead execution claims replace shutdown-hook suspension#41800
kitlangton merged 4 commits into
v2from
execution-claim-journal

Conversation

@kitlangton

@kitlangtonkitlangton commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What

Restart continuity for in-flight turns currently depends on a graceful-shutdown finalizer: suspendActiveSessions marks active Sessions moments before the process exits, and the next boot resumes whatever was marked. Any death without teardown — crash, SIGKILL, OOM, isolate eviction — skips the finalizer and silently loses the turn. For embedders whose runtime evicts mid-turn as steady state (workerd Durable Objects), that is the common case, not the edge case.

This PR inverts when the durable mark is written. The claim is recorded when execution starts, in the same transaction as the Execution.Started event, and terminals release it on commit. Recovery becomes a property of the database: a claim with no terminal is the signature of a dead process, regardless of how it died. No shutdown hook participates in correctness anymore.

Before / After

Before: a turn is draining when the process dies uncleanly. The shutdown finalizer never runs, so time_suspended is never set. The next server boots, resumeSuspendedSessions finds nothing, and the turn is gone — the user's session sits idle with a half-finished assistant message and no continuation. Only graceful SIGINT/SIGTERM restarts recovered.

Separately, a user-cancelled turn could resurrect: Execution.Interrupted never cleared suspension, so a cancel that raced a shutdown left the suspension mark in place and the next boot resumed a turn the user had explicitly killed.

After: the claim exists while the turn runs. Crash, SIGKILL, eviction, and graceful restart all leave the same durable state, and the next boot resumes the turn with a synthetic continuation. A user interrupt (or a superseding execution) releases the claim on commit, so a cancelled turn can never resurrect. A turn that keeps dying without completing exhausts a durable resume budget (default 10) and is terminalized with Execution.Failed instead of crash-looping forever.

How

packages/core/src/session/execution.tsclearSuspensionOnCommit becomes a claimOnCommit / releaseOnCommit pair. Started claims; Succeeded and Failed release; Interrupted releases for user and superseded reasons but deliberately preserves the claim for shutdown, so a graceful restart looks identical to an unclean death.

packages/core/src/session/store.tsstore.suspend (bulk, shutdown-time) and consumeSuspended are deleted. New primitives: claim (null-guarded so re-claims are zero-write no-ops), release (clears claim + resets the attempt counter), countResume (durable increment; reports a deleted Session as undefined so the sweep skips it), releaseChildClaims. All claim bookkeeping pins time_updated (the applyUsage precedent) so it never counts as user activity for session ordering. listSuspended returns only top-level Sessions, and the sweep clears orphaned child claims outright: children are never resumed independently — a resumed parent re-runs its tool call and spawns fresh children.

packages/core/src/session/execution/restart.ts — the sweep resumes orphaned claims without ever clearing them: only terminal events release a claim, so a death anywhere in the resume path leaves the same orphaned claim for the next boot. Per claim: skip if locally active → durably count the attempt before resuming (a crash inside the resumed turn cannot dodge the budget) → terminalize past maxAttempts → otherwise publish the synthetic continuation and resume, forked so boot never waits on resumed turns (resuming an already-live Session joins its execution). suspendActiveSessions is gone from the interface.

The sweep assumes every orphaned claim's owner is dead. The managed-server protocol guarantees this: kill/evict in the client service confirm the previous PID is gone before a contender is spawned, the registration lock admits one managed server at a time, and unregistered servers sharing the database never sweep.

packages/server/src/process.ts — the shutdown finalizer is deleted. Boot-time resumeSuspendedSessions is the whole contract.

Schema — one column, session_v2.resume_attempts integer not null default 0 (sql.ts, generated migration 20260811161259_execution_claim_attempts). time_suspended is reinterpreted as the claim timestamp; no rename, no data migration — an existing suspension from an old server reads as an orphaned claim and resumes exactly as before.

Flow

sequenceDiagram
participant U as User
participant E as SessionExecution
participant DB as SQLite
participant R as SessionRestart (next boot)
U->>E: prompt
E->>DB: Execution.Started + claim (one tx)
Note over E: drain runs, claim held
alt completes / fails / user interrupt
E->>DB: terminal event + release (one tx)
else process dies (crash, eviction) or shutdown interrupt
Note over DB: claim survives, no terminal
R->>DB: sweep finds orphaned claim
alt attempts exhausted
R->>DB: Execution.Failed + release
else
R->>DB: count attempt (claim untouched)
R->>E: synthetic continuation + resume (forked)
end
end
Loading

Scope

Deliberately not covered here:

  • Cross-process session fencing — servers that share a database without registering (--standalone alongside the managed server) have no fencing for any concurrent session operation today; the sweep inherits that, it does not create it. If shared-database multi-server becomes real, the principled fix is an owner ID + liveness check on the claim, for the whole class at once.
  • Tool re-execution on replay — a resumed turn may re-run a tool whose first invocation completed but never recorded. Cloudflare's Agents SDK ships the same hole; settleStaleToolCalls bounds the damage.
  • Progress-based attempt reset — dropped for simplicity in favor of a higher budget. If it returns, it should be a progress: true flag on event definitions, not event-name string matching.

Testing

  • packages/core: full suite — 1645 pass / 0 fail (172 files).
  • packages/server: 17 pass / 0 fail.
  • bun turbo typecheck --filter=@opencode-ai/core --filter=@opencode-ai/server clean; oxlint clean on changed files.
  • Lifecycle coverage in test/session-execution.test.ts: claim exists while the drain runs and survives teardown interruption; natural completion releases; user interrupt releases (no resurrection); the sweep leaves locally-active Sessions untouched; the sweep never consumes the claim it recovers (attempt counted durably before the turn runs, claim intact through a second teardown); budget exhaustion terminalizes with Execution.Failed and resets the counter; child Sessions are excluded from the sweep.

Restart continuity previously depended on a graceful-shutdown finalizer
marking active Sessions as suspended. Any death without teardown (crash,
SIGKILL, isolate eviction) skipped the finalizer and lost the turn.
The claim is now written when execution starts, in the same transaction
as the started event. Terminals release it on commit — except shutdown
interruption, which preserves it so graceful restart and unclean death
leave the same durable signature. The boot sweep resumes orphaned claims,
defers Sessions with recent message activity (managed-server handoff
overlap), and terminalizes a turn that exhausts its durable resume budget
instead of crash-looping.
suspendActiveSessions and store.suspend are deleted; graceful shutdown is
an optimization, not a correctness requirement. One schema addition:
session_v2.resume_attempts.
Review findings: consuming the claim before the resumed turn re-claimed it
reopened the crash window this model exists to close, and let the sweep
release the claim of a turn a user prompt had just revived. The claim now
survives the entire resume path — only terminal events release it.
The grace/redrive machinery defended against a successor server sweeping
while its predecessor still drained, but the managed protocol already
excludes that: kill/evict confirm the previous PID is dead before a
contender spawns, the registration lock admits one managed server, and
unregistered servers never sweep. Deleted along with lastActivityAt.
Child (subagent) Sessions are excluded from the sweep: a resumed parent
re-runs its tool call and spawns fresh children.
Pin time_updated in claim/release/countResume so claim bookkeeping never
counts as user activity (matching the applyUsage precedent) — previously
an interrupt or a boot sweep reordered the session list. Clear orphaned
child claims at sweep time: children are never resumed, so no terminal
would ever release them. Treat a missing row in countResume as a deleted
Session and skip, instead of masking it as attempt zero. Hoist the
exhausted-budget error literal, drop the unused configured() factory, and
tidy the tests.
@kitlangton
kitlangton merged commit 7300e7e into v2Aug 11, 2026
11 of 12 checks passed
@kitlangton
kitlangton deleted the execution-claim-journal branch August 11, 2026 18:09
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@kitlangton