Codev process fleet drives macOS memory-pressure kills: stranded shellper husks accumulate per restart + no idle-fleet memory policy

[amrmelsayed]

Evidence (2026-07-23, this machine — the recurring 'window terminated unexpectedly (killed, code 9)' report)

The VS Code window kills are macOS jetsam (memory-pressure SIGKILL) with codev's process fleet as the dominant memory load. Hard evidence, not inference:

1. JetsamEvent-2026-07-22-164753.ips (system diagnostic, same day as the first window-kill report): the top-resident-memory ranking at the event is dominated by claude CLI processes — 8+ processes at 400–580MB each, self-titled by version (2.1.207, 2.1.209, 2.1.214), i.e. long-lived survivors weeks old. Chrome (~580MB) and WindowServer round out the list. Under this pressure macOS SIGKILLs large renderers — the VS Code window (code 9) is the visible casualty.
2. Live process census (same machine, 2026-07-23): 41 shellper-main processes running, only 20 registered in Tower. 18 claude processes totaling ~6.3GB RSS.

Finding 1 — stranded shellper husks accumulate in waves that date-match Tower restarts

19 of the 21 unregistered shellpers are childless husks (~34MB each, ~657MB total), in two clean age cohorts:

• 6 husks aged ~6d15h → stranded 2026-07-16, the v3.2.3 release-day restarts
• 13 husks aged ~3d19–20h → stranded 2026-07-19, the Shellper reconnect error is swallowed: terminal becomes a silent zombie (no input/output, 'Message sent' logged for dropped frames) until next Tower restart #1198 live-deploy-incident / local-install churn day

This is empirical confirmation of the #1224 comment finding: a deregistered-but-alive shellper is never reaped by ANY path (killOrphanedShellpers skips responsive sockets; shellper-main idles forever awaiting a SPAWN). Every Tower restart/update strands a batch and nothing ever cleans them — the leak grows monotonically with update cadence, which is why the symptom 'keeps recurring and correlates with updates'. PR #1225's give-up/remove-architect reaps prevent NEW husks in its covered paths but nothing reaps the EXISTING population or strays outside those paths.

Finding 2 — the standing fleet itself is a multi-GB design reality with no policy

The tracked-and-legitimate fleet: ~13 architect claude processes across 5 active workspaces, resumed-alive since ~2026-07-13 (auto-restart + #832 resume keeps them permanent), at 175–583MB each ≈ ~4GB standing, plus active builders (e.g. 5 shannon builders ≈ 2.1GB at census time). An always-on multi-workspace fleet of ~20 claude processes is inherently ~5–6GB; on a laptop sharing memory with Chrome/VS Code this guarantees chronic pressure. There is currently no idle policy at all.

Proposed work

1. Husk sweep (bug-level): a reconciliation pass that reaps childless, unregistered shellper-main processes older than a grace period — as a Tower-startup step (extending killOrphanedShellpers's policy: a responsive socket should not protect a process that is BOTH unregistered AND childless for N hours) and/or a codev doctor/afx maintenance command so operators can recover without hand-rolled ps diffs. Must respect multi-Tower-instance socketDir scoping.
2. Idle-fleet memory policy (design-level, likely separate spec): options include architect hibernation (kill the claude child of an idle architect after N hours, keep the shellper + stored session id for instant resume — the infrastructure for this mostly exists post-Multi-architect conversation resume: disambiguate via per-architect session UUID #832/Tower/shellper: architect session-ID collisions + non-recovering crash loops after workspace restart #1224), per-workspace stop-on-idle, or at minimum surfacing fleet memory in afx status/dashboard so the cost is visible.
3. Observability: afx status (or doctor) should report total fleet RSS and flag unregistered shellpers — this census took manual ps/registry joining that the tooling should own.

Relation to existing work

• Tower/shellper: architect session-ID collisions + non-recovering crash loops after workspace restart #1224 / PR [Bugfix #1224] Skip architect session resume when a live process holds the id #1225: fixes husk-stranding in the give-up and remove-architect paths; this issue covers the existing population + paths outside those + the sweep.
• Tower upgrade stalls terminal adoption: waitForReplay burns 500ms per idle old-binary shellper (version skew from #1198) #1215 documented the window-kill as 'no causal link found, likely OS OOM' — correct on mechanism, but the census shows codev's own fleet is the primary load; this issue corrects that record.
• Web terminal: orphaned dotted-underline file-path decorations litter the screen after TUI repaints #1217 (web decoration DOM accumulation) remains a separate, minor renderer-side contributor.

[amrmelsayed]

Timing amendment: add a periodic in-process sweep as the third trigger

The two proposed triggers (Tower-startup pass + on-demand doctor/afx command) leave a timing gap: restarts are precisely the events that mint husks, so a startup-only sweep examines each husk cohort no earlier than the NEXT restart — measured at 4–7 days on the originating machine, and longer whenever update cadence slows. The on-demand command only helps an operator who already suspects a problem.

Third trigger: Tower re-runs the same reap predicate (unregistered AND childless AND older than grace period) on an internal timer (hourly is plenty). Cost is one ps snapshot + a registry diff. The aged condition makes it race-proof — a shellper momentarily childless between respawns, or mid-adoption at startup, cannot satisfy "unregistered and childless for hours" unless it is genuinely dead weight (childless+unregistered is a terminal state: respawns require a connected Tower client sending SPAWN, which an unregistered shellper can never receive).

Post-#1225 the give-up and remove-architect paths reap their own husks at stranding time, so the periodic sweep is the backstop for everything else: adoption failures, crash paths outside #1225's coverage, and the legacy population.

[amrmelsayed]

On it! Working on this with the PIR protocol (plan + dev-approval gates before PR).