Skip to content

core: age cap on execution-claim resume - #41815

Closed
kitlangton wants to merge 1 commit into
v2from
execution-claim-max-age
Closed

core: age cap on execution-claim resume#41815
kitlangton wants to merge 1 commit into
v2from
execution-claim-max-age

Conversation

@kitlangton

Copy link
Copy Markdown
Contributor

Follow-up to #41800.

What

The restart sweep now refuses to resume a claim whose turn showed no sign of life for longer than maxAgeMs (default 10 minutes). Stale claims are terminalized through the same visible path as budget exhaustion: an Execution.Failed with "Execution was interrupted too long ago to resume automatically", the claim released and the counter reset atomically with the terminal, partial output preserved.

Why

#41800 resumed every orphaned claim regardless of age. The cases the mechanism was built for — managed upgrade, crash + respawn, isolate eviction — all reboot within seconds-to-minutes of the death. But a server that died mid-turn and boots hours or days later would silently continue a turn the user has long moved past: surprise token spend and a zombie continuation. Cloudflare's agents SDK hit exactly this in production (cloudflare/agents#1324 — stale continuations jamming agents behind new user messages); their documented recipe is an age guard.

The clock

Staleness is not measured from when the turn started — our agent turns legitimately run for hours, and a 2-hour turn interrupted mid-stream by a deploy is healthy. It is measured from the turn's last durable sign of life:

staleness = now − max(claimedAt, lastActivityAt ?? 0)
  • lastActivityAt = the session's newest message time_updated. Message rows update at durable part boundaries as a drain runs, so this ticks throughout a healthy turn. It deliberately reads messages, not the event log — event persistence is opt-in.
  • claimedAt is the floor for turns that died before producing any durable activity.
  • Both sides run on the wall clock the store writes claims with (Date.now).

Comparison points: Cloudflare's chat recovery recipe is ~2 minutes on the turn-start clock (would kill our long turns); their raw fiber recovery discards orphans after 24 h. Ours is 10 minutes on the activity clock — more generous on a more honest clock — and maxAgeMs is an Options field for embedders with long-silent workloads.

Known trade-off

A turn that was silent for >10 minutes inside one tool call (a monster build) and then crossed a crash reads stale and is skipped. The failure is gentle — visible terminal, partials preserved, one re-prompt — and the asymmetry favors skipping: a wrongly-skipped resume costs a re-prompt; a wrongly-taken resume costs surprise tokens and a zombie continuation.

Also in this PR

  • listSuspended returns { sessionID, claimedAt } (the sweep needs the claim time).
  • SessionStore.lastActivityAt — new read, documented as the staleness clock.
  • releaseChildClaims doc fix: a dead child's claim can be released by a terminal if the user prompts the child session directly; the accurate statement is that no terminal is otherwise coming.

Testing

Two new tests in session-execution.test.ts (12 total, all green):

  • a claim from an hour-dead process with no message activity terminalizes with the stale message, claim released, counter reset — and the drain never runs;
  • a claim from an hour-long turn with message activity moments before death resumes normally (the activity clock, not the start clock, decides).

Full core suite 1669 pass; repo typecheck clean; touched files lint 0/0.

A claim orphaned long ago should not silently continue a turn the user
has moved past — the failure mode Cloudflare hit in production when
stale continuations jammed agents behind new messages (agents#1324).
The sweep now terminalizes claims whose turn showed no sign of life for
longer than maxAgeMs (default 10 minutes) through the same visible path
as budget exhaustion. Staleness is measured from the turn's last durable
message activity, falling back to the claim time — NOT from when the
turn started, so a long healthy turn interrupted mid-stream is seconds
stale at the next boot and always resumes.
Also corrects the releaseChildClaims doc: a dead child's claim CAN be
released by a terminal if the user prompts the child directly; the point
is no terminal is otherwise coming.
@kitlangton

Copy link
Copy Markdown
ContributorAuthor

Closing after discussion: every failure mode here is already bounded and visible — maxAttempts durably prevents crash-loops, the continuation is interruptible, and a user booting their own CLI arguably wants their turn back regardless of age. Not worth the added store read and option surface. The branch stays around if a deployment ever demonstrates the need.

@kitlangton
kitlangton deleted the execution-claim-max-age branch August 11, 2026 19:05
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@kitlangton