Tower recovery: Claude architect crash-loops on stale --resume with no user escape

[amrmelsayed]

Problem

After a Tower recovery event (machine crash, tower start after downtime, network partition), some architect terminals enter a crash-loop displaying repeated "Could not resume session ..." errors. Users cannot cancel or dismiss the errors — the loop continues until Tower is externally killed or the offending row is manually removed from ~/.agent-farm/global.db.

Reported by users 2026-07-08.

Root cause (from subagent investigation 2026-07-08)

Both reconcile paths pre-compute restartOptions.args from the stored session id:

• packages/codev/src/agent-farm/servers/tower-terminals.ts:661-690 (Phase 1 startup reconcile)
• packages/codev/src/agent-farm/servers/tower-terminals.ts:898-925 (on-the-fly reconnect)

These args are handed to SessionManager.setupAutoRestart (packages/codev/src/terminal/session-manager.ts:845-891), which replays session.options.args verbatim after restartDelay (2s), up to maxRestarts (50 — tower-instances.ts:532).

If --resume <uuid> is unresumable (jsonl deleted, corrupted, or foreign), Claude Code prints Could not resume session ... and exits immediately. The exit re-triggers auto-restart with the same args. startRestartResetTimer fires unconditionally after each spawn() call (not after a "healthy" signal), so a fast-crashing process prevents the counter from ever resetting — count climbs monotonically to 50 (~100 seconds of errors) before removeDeadSession finally halts the loop.

No escape hatch exists — no env var, no CLI flag, no Tower HTTP endpoint. User must kill Tower or hand-edit global.db to break out.

Load-bearing context: the comment at packages/codev/src/agent-farm/utils/harness.ts:83-87 explicitly notes that codex/gemini omit session id to avoid this class of crash-loop. Claude has the vulnerability.

Fix sketch

Introduce a "recent failure" detector in setupAutoRestart (packages/codev/src/terminal/session-manager.ts:845-891):

1. Track restart timestamps + exit statuses on the session object
2. If ≥ 3 restarts fail within 30 seconds AND the failing args include --resume, strip --resume <uuid> from session.options.args on the next attempt
3. Log once: "resume session <uuid> unrecoverable; retrying without --resume"
4. Callback into state.ts to null out the stored session_id on the architect row so future reconciles don't relearn the poisoned id

Small change; matches the design intent of the harness.ts:83-87 comment (codex/gemini avoid this class; Claude should degrade gracefully into the same state).

Optional defense-in-depth: expose CODEV_SKIP_RESUME=1 env var read by resolveArchitectLaunch (tower-utils.ts:208-236) that strips --resume from launch args regardless of stored state. Emergency escape when the automatic fallback hasn't kicked in yet.

Scope

• packages/codev/src/terminal/session-manager.ts — fast-failure detector
• packages/codev/src/agent-farm/state.ts — helper to null-out session_id on architect row
• Tests: simulate 3 fast failing spawns with --resume <bad-uuid>, verify auto-restart drops the --resume flag on attempt 4 and clears the DB row

Related

• codev adopt / launchInstance: main architect hijacks an unrelated Claude Code session instead of spawning fresh #1145 (RC1 — buildResume has no ownership check; often triggers this crash-loop on fresh adopt)
• The RC3 bugfix filed alongside this one (removed sibling architects resurrect after crash) — that bug is another common trigger for this crash-loop because the resurrected architect carries a stale stored session_id

[amrmelsayed]

On it! Working on this with the PIR protocol (plan + dev-approval gates before PR).