Shellper leak: the spawn-gate test orphans one per spawn, and Tower leaks one per cap rejection

[pseudoseed]

Fix both halves of the shellper leak. They are separate defects with one shared symptom, and fixing either alone leaves the fleet filling up.

Evidence gathered tonight: 163 shellper processes holding 4.7 GB, of which 133 had a cwd inside a spawn-gate-profile temp directory and 126 of those pointed at a directory that no longer existed. Tower was wedged at Maximum 100 sessions reached and builders could not open terminals. Killing the 126 proven orphans took the fleet to 39 processes and 2.1 GB with no Tower restart.

Half 1 — the test leaks a shellper per spawn (#162)

packages/codev/src/agent-farm/__tests__/spawn-gate-profile.test.ts spawns builders into mkdtemp directories and its teardown removes the tree while the spawned shellper still has that directory as its working directory. Two consequences:

• The shellper survives the test, orphaned, holding a cwd that no longer exists. Roughly 13 leak per full-suite run.
• rmdir intermittently fails with ENOTEMPTY because the shellper is still writing into the tree. Reproduced on main 1 of 6 runs standalone, and about 2 of 4 inside the full suite.

Both symptoms are the same race. Fix the ordering: wait for the spawned session to exit before removing the tree. If a retry is used instead, retry rmdir only on ENOTEMPTY and let every other errno fail loudly, so a real permission or path bug is not swallowed.

The ENOTEMPTY failure lands on a test named "the default harness is unaffected", so when it fires during review of an unrelated branch the natural reading is "this PR broke the default harness". It cost one review round to rule that out.

Half 2 — Tower spawns before it checks the cap (#171)

[06:53:00.936] Creating session 7c7ce4ed…: command=/bin/bash, socket=…
[06:53:01.088] Session 7c7ce4ed… created: pid=22158
[06:53:01.089] Shellper creation failed for terminal, falling back: Maximum 100 sessions reached
[06:53:01.089] Failed to create terminal: Maximum 100 sessions reached

The process is spawned, then the cap is checked, then the failure is reported, and what was just started is never killed. pid=22158 was still running long after its request was reported as failed. So every request made while at the cap adds an orphan, and the fuller Tower gets the faster it fills. 148 cap-hit entries in about 8 hours.

Check the cap before spawning. If the check must stay where it is, kill the just-created session on the failure path. A failed create must leave nothing behind.

Second part: nothing reaps dead sessions. Tower counted 100 against a cap of 100 while most of those processes were gone or reparented to init. Reconcile the session table against live processes periodically and on every cap rejection, so the cap reflects reality rather than accumulated bookkeeping.

Also worth doing

The error surfaced to the caller names the symptom but not the owner. Include the top workspaces by session count in the message, so it points at the cause rather than at whoever asked next.

Verification

• A full agent-farm suite run must leave zero new shellper processes. Count before and after with ps aux | grep -c '[s]hellper' and assert the delta is 0.
• Run spawn-gate-profile.test.ts alone 10 times with no ENOTEMPTY.
• Before trusting a new test, verify it fails against a deliberately mutated source.

Note: the Tower half cannot take effect until Tower restarts, which cannot happen while builders are running. Land it anyway; it applies at the next restart.

[pseudoseed]

On it! Working on a fix now.