ACP bridge subprocesses are never reaped after an agent turn completes, leaking memory across all harnesses

[adroitandroid]

Describe the bug
When an agent (regardless of which ACP-compatible harness backs it) completes a turn and replies in a channel, the subprocess pool spawned to service that turn is not terminated or reaped. Each single message-to-agent interaction leaves behind roughly 10 idle child processes under the corresponding buzz-acp bridge, permanently, with no cleanup even minutes after the agent has finished responding and gone idle (confirmed via CPU dropping to ~0% and staying there).

This is harness-agnostic — reproduced identically with two different backing harnesses (a Claude-based agent and an opencode-based agent) in the same session. In each case, exactly one message produced exactly 10 leftover subprocesses that never exited.

Because this leak is per-message rather than per-session, it compounds continuously during normal use. Over roughly an hour of typical activity (multiple messages across multiple agents), leaked subprocess count reached ~88 processes consuming ~2.2GB RSS and ~2,100 open file descriptors. In a smaller two-message controlled test, 20 leaked processes alone accounted for ~3.78GB RSS and 627 open FDs — the two harnesses differ substantially in per-process footprint (~43MB/process for one harness's subprocesses vs. ~330MB/process for the other's), but both leak at the same ~10-processes-per-turn rate.

The resulting memory/FD pressure correlates with (and is the most likely cause of) several user-visible symptoms during sustained use:

• The relay WebSocket connection intermittently dropping, prompting a manual reconnect
• Sent/received messages failing to render in the UI
• Agents appearing unresponsive to new messages

Steps to reproduce

1. Launch Buzz desktop and note the baseline set of buzz-acp bridge child processes (e.g. ps -eo pid,ppid,command | awk '$2==<buzz-desktop-pid>') — this is a stable pool with no further children yet.
2. Send a single message to any agent and wait for its reply to appear in the channel.
3. Inspect the children of that agent's buzz-acp bridge process (pgrep -P <bridge-pid>) — observe ~10 new subprocesses spawned.
4. Wait several minutes after the agent's reply is visible in the UI. Observe via ps -o pid,%cpu,stat,etime that these subprocesses drop to near-0% CPU (idle) but remain alive indefinitely — they are never terminated.
5. Repeat with a different backing harness for a different agent — same ~10-subprocess leak pattern occurs.
6. Repeat step 2 across multiple agents/messages over an extended session — leaked process count, RSS, and FD usage grow linearly and without bound.

Expected behavior
Once an agent turn completes and the reply has been delivered, all subprocesses spawned to service that turn should be terminated and reaped by their parent buzz-acp bridge. Steady-state process/memory/FD usage should return close to baseline between turns, regardless of which ACP harness backs the agent, and regardless of how many messages are sent over a session's lifetime.

Version and platform

• Buzz version: 0.5.3 (0.5.3)
• OS: macOS

Logs / additional context
Quantified from a controlled two-message test (one message per harness, one harness-A agent + one harness-B agent):

• 20 leaked subprocesses total (10 per harness), all confirmed idle (~0% CPU) for 10+ minutes after their agent's reply was posted
• ~3.78GB combined RSS across just these 20 processes
• 627 open file descriptors across just these 20 processes
• System-wide memory compressor usage rose to 9.6GB with under 1GB "unused" RAM remaining, from this test alone

From the original (uncontrolled) session that surfaced the symptom: ~88 leaked processes, ~2.2GB RSS, ~2,100 FDs, accumulated over roughly an hour of normal multi-agent usage, including at least one process that became a <defunct> zombie never reaped by its parent. Fully quitting and relaunching the Buzz desktop app clears all leaked processes and temporarily resolves the disconnects/unresponsive-agent symptoms — until the leak reaccumulates with continued use.

---

Filed by Claude on behalf of the reporter.

[greenplasticmike]

Independent data from another deployment that bears on the third bullet in your symptom list ("agents appearing unresponsive", "messages failing to render"). We hit this today and instrumented it, so this is offered as a measured companion to the correlation you infer — not as a confirmation of the causal link, which we have not tested.

Per-send ledger for one agent session

One agent session issued 8 buzz messages send calls. The relay holds 8 of its events, one-to-one, nothing unaccounted for. Launch times are the harness's tool-call timestamps; publish times are relay created_at.

launched	event	published	content length
10:25:06	a8085585	10:25:06	3459
10:28:08	a8fee640	10:36:18	0
10:37:09	1fae930c	10:37:10	5171
10:40:08	203efd35	10:43:06	0
10:43:44	3ecaff4a	16:44:37	0
10:53:11	1d416e5d	10:53:12	2347
10:56:31	f431e726	10:56:31	2602
10:59:18	dbac5658	10:59:19	2461

The discriminator is stall time, not input method

Content survives if and only if the send returns promptly. The five that published within 0–1s of launch carried every byte. The three that stalled — 8m10s, 2m58s and 6h00m53s — all published with an empty body. No overlap and no exception, across both --content - reading from a file and printf | piping, so the input method is not what separates them.

Stated as a prediction rather than a count: a buzz messages send that does not return promptly will still publish, and what it publishes will be empty. That makes the failure detectable at the moment of stall instead of only afterwards by read-back.

One of those processes was never reaped, and it was not idle

The 10:43:44 send hung. The harness hit its 120s timeout and moved it to the background; nothing killed it. Its output file is still on disk, mtime 16:44:41Z:

{"accepted":true,"event_id":"3ecaff4a…","mention_pubkeys":["…","…"],"message":""}
EXIT=0

Launched 10:43:44.456Z, signed 16:44:37Z, exited 0 at 16:44:41Z — 6h 0m 57s for a single process making a single publish. It then landed in a channel threaded as a reply to a message from six hours earlier, because it held its 10:43 state in memory the whole time.

This is the part that may connect to your report. You count ~10 unreaped subprocesses per turn sitting idle at ~0% CPU. We had a long-lived child process that was not inert — it was holding an unsent message and published it empty on the way out. Whether it belongs to the population you are counting, we cannot say: ours was a buzz messages send that the harness backgrounded at its own 120-second timeout, we never enumerated what spawned it, and we never measured its CPU. It is no longer checkable in either direction — the process exited at 16:44:41Z. If it does belong, a leaked-process count may also be a count of writes in an indeterminate state.

Two things we can rule out, and one we cannot

• Not a signing-time rewrite. created_at is honest; 16:44:37Z is when the send actually reached signing.
• Not caused by killing the process. An earlier theory of ours blamed pkill. This one was never killed and still landed empty, so the body is lost somewhere inside the send path independently of how the process ends. The source file was intact on disk at 4613 bytes the entire time.
• Unresolved: whether those six hours were spent before signing or waiting on relay acknowledgement. Our side sees only launch and exit. The relay's receive time for that event would settle it.

Why the receipt cannot be trusted as written

EXIT=0 plus a well-formed receipt carrying a real event_id and correct mention_pubkeys is fully compatible with total content loss. Neither the sender nor the reader gets any signal — the sender sees success, the reader sees an empty message, and nothing reconciles the two. Any client that treats the receipt as delivery confirmation will silently drop content.

Happy to supply the raw transcript timestamps if useful.

[wesbillman]

Carl here, commenting on Wes's behalf after an independent three-agent review of the report and the relevant process lifecycle.

Thanks for the detailed measurements. We traced the lifecycle in Buzz Desktop v0.5.3 and current main. Desktop defaults managed-agent parallelism to 10 and passes that value as BUZZ_ACP_AGENTS; buzz-acp creates that many long-lived adapter workers and returns each worker to the pool after a turn so its ACP sessions can be reused. The controlled result in this report—one turn through each of two harnesses followed by 20 retained workers—therefore matches two configured pools of 10. It does not yet demonstrate 10 additional processes leaking on each successive turn.

Buzz performs cleanup at the bridge lifecycle boundary rather than after each turn: adapter workers are launched in process groups, shutdown kills the process group and awaits the direct child, and bridge shutdown explicitly drains both checked-out and idle workers. Killing these workers after every reply would discard session continuity, defeat configured concurrency, and repeatedly cold-start adapter/MCP processes, so we do not plan to make that change.

The retained pool can still create substantial memory pressure; we track that pool-sizing/default-parallelism problem separately in #2631.

To establish a distinct per-turn leak, could you provide PID, PPID, PGID, start time, and process state for the same bridge at three points: before turn 1, after turn 1, and after turn 2? If the second turn adds a new set of workers while the first set remains, we will reopen the cleanup investigation immediately.

The reported <defunct> process may also be actionable if you can provide its lineage and timing. The separate six-hour buzz messages send stall with empty content appears to involve the CLI publish path rather than ACP pool ownership; please file that independently so it can be investigated without conflating the two lifecycles.

[adroitandroid]

@wesbillman thanks for the context Wes (and Carl). is there any telemetry we can turn on in a local build that can help provide more useful logs/other info if such issues arise again? At the moment I had asked Claude to add logs at the relevant places for a dev build and that can be available if this issue recurs. but those would be coming from the logs I added in my fork. any existing telemetry that needs any flags to be turned on, would be happy to turn it on to provide useful bug submissions in future. thanks.

[ms9804]

Adding a variant of this that behaves differently from both #4577 as filed and #4708, because it defeats the usual mitigation.

The leaked workers are not orphans. #4708's orphans reparent to PID 1, and the standard sweep everyone writes (pgrep -P 1 -f 'claude-agent-acp') cleans those up. The ones I'm seeing keep a live buzz-acp parent, so a PPID=1 discriminator never matches them. Same symptom, different reachability.

They leak on the respawn path, not the completed-turn path. Every one traces to:

WARN buzz_acp: transport/protocol error — respawning agent
  agent=0 outcome="error" configured_model=sonnet pid=…
  error=Request timeout — agent did not respond within 60s

buzz-acp starts a replacement, but the previous claude-agent-acp and its claude child stay attached to the still-live parent. The correlation is exact: 23 transport/protocol error events in a day produced exactly 23 leaked claude --output-format stream-json workers. One per respawn, no exceptions.

Detection signature. Leaked workers stop accumulating CPU at ~10 minutes, then sit for hours with elapsed climbing:

pid=49334  elapsed=02:57:29  cputime=10:49  rss=16MB
pid=46208  elapsed=03:02:19  cputime=11:07  rss=30MB
pid=24073  elapsed=03:30:15  cputime=10:04  rss=98MB

Flat CPU against climbing elapsed is unambiguous — the turn was killed, the process was not. BUZZ_ACP_MAX_TURN_DURATION=1800 was set, and the oldest survivor was 440 minutes, 14× the cap. So the turn caps work; nothing bounds the process.

It is self-reinforcing, which is the part I'd flag hardest. The 60s "did not respond" threshold appears to be hardcoded. On a memory-pressured host, a paged-out worker can't answer inside 60s → transport error → respawn → another leaked worker → more pressure → more timeouts. Respawn rate tracked host memory across a week:

Window	Respawns	Host state
Aug 5–6	56, 62	swap exhausted
Aug 7–8	3, 4	healthy
Aug 10, 22:07–22:31	20 in 24 min	40 MB free, 92% swap
Aug 10, after reaping	0	5.7 GB freed

Reaping the leaked workers freed 5.7 GB of swap against only 1.2 GB resident — they were largely paged out, so ps RSS understates the cost substantially. Worth noting for anyone trying to size the impact from a process listing.

Workaround for anyone hitting this now. Sweeping on PPID=1 won't work. Threshold on elapsed time instead, derived from your own turn cap — past MAX_TURN_DURATION × 1.5 a process cannot be a live turn:

ps ax -o pid,etime,command \
  | grep -F 'claude --output-format stream-json' \
  | # kill those with elapsed > MAX_TURN_DURATION * 1.5

22 of 23 ignored SIGTERM and needed SIGKILL, which is further evidence they're wedged rather than idle. Twelve agents stayed up throughout; killing the stale workers doesn't disturb their parents.

One smaller observation: bridges accumulate too, at roughly 2 claude-agent-acp per buzz-acp (25 bridges for 13 parents here, ~11 MB each). Much smaller stakes than the workers, but the same blind spot — parented, so PPID=1 sweeps miss them.

Environment: macOS 15 (Apple Silicon), claude-agent-acp 0.64.2, 12 agents under launchd, relay self-hosted over Tailscale.

[ms9804]

Correction to my comment above — the workaround I posted is wrong and will take your roster down. Please don't use it.

I said "past MAX_TURN_DURATION × 1.5 a process cannot be a live turn." That rests on worker == turn, and it isn't. claude-agent-acp keeps one persistent claude worker per ACP session and serves every turn of that session through it, so a healthy worker lives as long as its session — until buzz-acp restarts. There is no age above which a worker is necessarily dead, so no elapsed-time threshold could have been safe.

What it costs when you kill a live worker: the adapter logs "Claude Agent process died" and deletes the session (acp-agent.js:2950). buzz-acp then keeps prompting the same, now-deleted sessionId indefinitely — it never calls session/new on that error — and every subsequent turn returns:

-32603  Internal error   data.details = "Session not found"

buzz-acp logs only the JSON-RPC message and discards data, so the actual cause is invisible in the logs. On my roster this produced fleet-wide wedges roughly an hour after every restart, and ~7,800 Session not found events before I traced it back. Two confirmed instances: a 16:04:16Z sweep of 13 workers produced a 16:07:20Z fleet-wide wedge; an 18:34:38Z sweep produced an 18:38:46Z wedge. The script I wrote to contain the leak was a considerably worse problem than the leak.

The discriminator that does work is the one my original comment already described without recognising what it meant: leaked workers "stop accumulating CPU at ~10 minutes and then sit for hours." A live session worker keeps burning CPU because it answers a heartbeat every few minutes. So compare CPU time between two runs spaced comfortably longer than your heartbeat interval, and reap only workers whose CPU has not moved at all:

# run on a schedule; persist `ps -o time` per worker between runs
# reap only when: elapsed > cap AND cputime identical to the previous run
# never on a single sample

A leaked worker survives one extra cycle before being collected. That costs memory, not correctness, and memory was never the part that hurt.

Two smaller traps found on the way: keying the CPU snapshot by pid alone lets a recycled pid inherit a dead process's history (key by pid + derived start time), and a first attempt using a bash 4 associative array failed open on macOS's bash 3.2 — reporting "0 stale workers" while silently doing nothing.

Everything in my original comment about the leak itself still stands: the 1:1 respawn-to-leak ratio, the live-parent-so-PPID=1-sweeps-miss-them finding, and the memory-pressure feedback loop. Only the workaround was wrong.