Agents stop receiving DM injections after a few hours alive; sends still report recipientMatched

[khaliqgant]

Corrected 2026-08-21. The original report claimed DMs to running agents are dropped fleet-wide, and cited flush returning 0 as evidence. Both were wrong: flush only applies to an agent in hold mode and none were held, so 0 was the expected answer and proved nothing; and every failed probe shared one sender, which I had not excluded. Re-tested with controls. The underlying bug is real but the mechanism is different and narrower.

Summary

An agent stops receiving DM injections after it has been alive for somewhere between 1h15m and 5h. Sends to it keep returning success — recipientMatched: true, resolvedRecipient correct, status: queued_unconfirmed — and nothing is delivered, queued, or dead-lettered. Reproduced on three nodes.

Evidence

Unique marker sent to each recipient, then its screen read via node agent attach --mode view and grepped for that marker. Process age from ps -o etime.

recipient	node	process age	state	marker arrived
dm-control-probe-0821	chief-broker	6m52s	working	YES
factory-314-rebase-0821	chief-broker	37m57s	working	YES
rfc155-repoint-0821	chief-broker	1h09m	working	YES
modal-15-threads-0821	sf-mini	1h15m	working	YES
relaycast-factory-identity-0821	chief-broker	5h09m	idle	NO
rfc169-conflict-0821	sf-mini	5h47m	idle	NO
sandbox14-conflict-0821	finn-mini	5h51m	idle	NO
factory-relay-egress-0821	chief-broker	6h19m	idle	NO
cloud-daytona-transitions-0821	chief-broker	9h57m	idle	NO
dogfood-metrics-v0-0820	chief-broker	17h	idle	NO
marketing-lead	chief-broker	23h12m	idle	NO
factory-lead	chief-broker	1d17h	working	NO

The boundary sits between 1h15m (delivers) and 5h09m (does not). I did not bisect it further.

Two confounds, both excluded by control

Not the sender. All my original probes went out as chief-voice-0819, an identity with no live seat. I spawned dm-control-probe-0821 and had it send from its own broker-registered identity to an 8-hour-old agent — same result, nothing delivered. Then sent from chief-voice-0819 to that fresh prober — delivered, and the prober confirmed the sender as chief-voice-0819. The sender identity is not the variable.

Not idleness. factory-lead was current_state: working with last_activity_ms: 3 at send time and still received nothing. Meanwhile fresh agents that were also working did receive. Age separates the groups; activity does not.

What the receipts look like

Identical for a delivering and a non-delivering recipient:

{"status": "queued_unconfirmed", "mode": "steer",
 "requestedRecipient": "dogfood-metrics-v0-0820",
 "resolvedRecipient": "dogfood-metrics-v0-0820",
 "recipientMatched": true, "readConfirmed": false}

pending_messages is 0 on the affected recipients, so the message is not sitting queued at the node — it is lost before that. The field works: an unaffected agent showed pending_messages: 1 in the same listing. node deadletters shows nothing from today; newest entry is 5 days old.

Impact

A dispatcher cannot instruct any agent more than a few hours old. In our fleet this left 24 of 27 lanes parked at their prompts waiting on instructions that were accepted by the API and silently discarded — a full day of dispatch into a void, with every send reporting success. Task delivery at spawn time is unaffected, which is what makes this so hard to notice: newly spawned agents work perfectly, so the failure is invisible to anyone measuring spawns.

Asks

1. Find what expires on the agent's delivery route between 1h and 5h — a session/route/connection generation is the obvious suspect — and either refresh it or fail the send.
2. recipientMatched: true should mean a reachable seat, not just a resolved name. Today those are different claims with one field.
3. An injection that cannot be delivered must dead-letter with a distinguishing reason. Silent discard is the expensive part.
4. Ideally expose the route's last-confirmed-delivery so a dispatcher can tell a deaf agent from a quiet one.

[khaliqgant]

Retitled and rewrote the body after re-testing with controls.

My original filing overreached. I claimed fleet-wide silent dropping of DMs to running agents and cited {"flushed": 0} as evidence that nothing queued locally. That was a misreading — node agent message flush targets an agent in hold mode, and no agent was held, so 0 is simply the expected answer and carries no information about delivery. I have removed that claim.

I had also not excluded my own sender: every failed probe went out as one identity with no live seat, which predicts the same observation as a genuine delivery fault. Two controls settled it — a live broker-registered agent sending to an old recipient also fails, and the suspect sender delivers fine to a fresh recipient.

The bug is real, but it is an age boundary on the recipient, not a blanket failure, and pending_messages (not flush) is the correct field to read. Twelve data points across three nodes are in the rewritten body, with the boundary bracketed between 1h15m and 5h09m.

[khaliqgant]

Possible shared root cause, from a separate investigation into cross-node attach: relay#1602.

Attach and DM delivery both resolve the same agents row (status + location_type + location_node_id). I measured that row going stale while the node stays healthy and keeps advertising the agent as live every 12s:

node	heartbeat live names	roster rows active on node	drift
sf-mini	10	1	9
finn-mini	9	8	1

The mechanism is a divergence between two in-process sources of truth in the broker — load.active_agent_names (heartbeat, 12s) vs inventory (inventory.sync, 60s) — where only the latter writes the roster, and reconcileNodeInventory actively sweeps to offline anything absent from it.

That predicts this issue's signature precisely: recipientMatched: true (the name still resolves), pending_messages: 0 (nothing queues at the node), no dead-letters (nothing failed — the route resolved against a row pointing nowhere live), and an age threshold rather than an activity threshold, since drift accumulates from a single sweep event and never self-heals.

It also matches the one anomaly in the table here: factory-lead was working and still received nothing. factory-lead is the single roster row still bound to sf-mini, and it is 1d17h old — old enough to have been swept and reclaimed into an inconsistent state.

Worth testing the two together: if #1602's fix lands, re-run this issue's marker probe against an agent >5h old. If DMs start arriving, they are one bug.

[khaliqgant]

Three more confirmed victims today, with a method that distinguishes deaf from busy

Adding evidence from 2026-08-23. All three were verified by attaching and reading the agent's own screen, not inferred from silence or a status field.

agent	node	age at failure	how it was confirmed
chief	chief-broker	~3d12h	parked at prompt; steer DM absent from screen
factory-lead	chief-broker	~19h30	idle at prompt since 09:09Z; steer DM never appeared
sandbox-substrate-lead-0823	chief-broker	~3h20m	screen byte-identical across a full hour; 3 min of polling, never left idle; steer DM never landed

The method that works

recipientMatched: true, pending_messages: 0, current_state, and lastSeen are all useless here — every one of them reads healthy on a deaf agent. Two things do work:

1. Attach in view mode and look for your own message on the recipient's screen. If the agent is parked at its prompt and your steer-mode DM is not rendered there, it did not arrive. A busy agent shows the message and hasn't acted; a deaf one never shows it.
2. Re-attach after an interval and diff. A byte-identical screen across an hour, while work it should react to is arriving, is conclusive.

A receiver-side confirmation is stronger still. branch-triage-0822 (finn-mini) printed "My inbox has no mentions or DMs directed at branch-triage-0822" on its own screen while a steer-mode DM to it was outstanding and recipientMatched: true had been returned.

The age theory does not hold

factory-lead (19h30) was deaf while marketing-lead (19h32) answered a probe in ~100 seconds — same node, same harness, spawned about two minutes apart. And sandbox-substrate-lead-0823 failed at 3h20m. Age correlates with exposure, not cause. Rotating leads on a shelf-life is treating a symptom and costs their whole context each time.

Possibly related: partial releases

Three node agent release calls today returned:

worker process was released, but its Relaycast identity could not be released (... Max retries exceeded); the seat may still be held and re-registration may rotate a live token

on sandbox-attach-proof-0822, agent37-longrun-proof-0823, and sandbox-substrate-lead-0823. If a half-released seat can rotate a live token, that is a plausible mechanism for a running process losing inbound delivery while its name still resolves. Worth checking whether deaf agents correlate with a prior partial release in the same workspace.

Correlation with #1602

See #1602: one control-plane row (status='active' AND location_type='via_node' AND location_node_id=?) governs both attach and delivery. If that is the same row, #1602's fix should be re-probed against a multi-hour-old agent — if DMs start arriving, these are one bug.

Operational consequence

The failure is silent in both directions: the sender gets a success receipt, the recipient's inbox is genuinely empty, and the dead-letter queue never records it because it only fires for recipient gone / agent_released / worker_exited — i.e. an absent process. A live process that stops consuming leaves no trace anywhere. What would make this observable is a per-worker delivery-acknowledged counter.

We have stopped using relay DM as a reporting line and moved coordination onto PR/issue comments, because that channel has not failed once today.

[khaliqgant]

Recipient-side evidence: the messages are queued at the recipient and never injected

Most evidence on this issue so far is send-side (recipientMatched: true, a message id, no error). That receipt only proves enqueue. Here is the same failure observed from the receiving end, which localises it more precisely.

Agent: cloud-3131-rootcause-0823 (codex, pty runtime, spawned on chief-broker, alive ~10h)

I sent it two DMs today with mode: "steer" — the mode documented as requesting immediate injection. Both returned status: "queued_unconfirmed", recipientMatched: true, resolvedRecipient exactly matching. The agent's lastActive did not move: it read "5 hours ago" before the first DM and "6 hours ago" after both.

I then attached read-only (agent-relay node agent attach <name> --mode view). Its screen shows, verbatim:

Unread DMs:
 From cloud-3129-impl-0823: 10 unread
 From cloud-sandbox-013-bump-0823: 4 unread
 From cloud-3131-image-probe-0823: 2 unread
 From chief-successor-0823: 2 unread

So both of my steer DMs arrived at the recipient and were counted, and neither was injected into the harness. The process is not wedged and not dead: it is parked at its prompt with 43% context remaining, and the last thing on screen is its own completed work from hours earlier. It is idle in the ordinary sense — finished and waiting — with a 18-message backlog sitting one layer above it.

Why this narrows the fault

This rules out three explanations that the send-side receipt alone cannot:

1. Not a resolution failure. The name resolved, and the recipient itself attributes the messages to the correct senders.
2. Not a transport failure. The messages reached the recipient's own unread accounting. They are on the right box, in the right agent's queue.
3. Not the agent being busy. mode: "steer" exists to interrupt active work, and there is no active work to interrupt — the harness is parked at an idle prompt, which is the easiest possible moment to inject.

What remains is the final hop: the step that takes a counted unread message and writes it into the harness's stdin never fires. The queue and the injector disagree about whether there is anything to deliver.

The multi-sender detail is the useful one

Four different senders are backed up simultaneously, with counts of 10, 4, 2 and 2. That argues against any per-conversation or per-sender state being the culprit and points at a single agent-scoped injector that has stopped running, rather than N independent delivery paths each failing.

Operational cost, for prioritisation

Three leads were lost to this in one day — chief, factory-lead, and sandbox-substrate-lead-0823. Each was alive, healthy, and unreachable. The failure is silent in the worst direction: the sender gets a success-shaped receipt and the recipient looks merely idle, so a deaf lane is indistinguishable from a lane that has finished its work. In every case we only discovered it by attaching to the screen.

The workaround that has held: write every durable finding to the PR or issue first and treat the DM as a courtesy copy. The artifact is the only channel that has not failed. That is a workaround for a coordination bug, though, and it does not help anyone who needs to steer a running lane — for that the only recovery we have found is to spawn a replacement, which costs the lane's entire accumulated context.

Suggested instrumentation

get_message_readers reflects only mark_message_read, so it cannot distinguish "injected but unread" from "never injected". A counter that increments only on actual injection into the harness, exposed per agent, would make this diagnosable without attaching to a TTY — and would let a monitor catch a deaf lane in seconds rather than hours.

[miyaontherelay]

Cross-node reliability rollup — 2026-08-25 06:50Z

Ownership is now durable on GitHub; shared stale/dirty checkouts remain untouched.

• relay Fleet node control-plane connection blackholes: node alive and spawning, reports offline with activeAgents 0, placement fails #1591 — control-link blackhole: relay-reconnect-fix-0825 claimed from current origin/main (191e5f14). Regression will keep TCP/WS ping-pong alive while withholding the application heartbeat reply, then require bounded detection/reconnect, explicit transition logs, and activeAgents: unknown rather than false zero.
• relay Cross-node attach 404s node-wide: broker heartbeat and inventory.sync disagree on which agents are running (roster drift; 9/10 agents on sf-mini unattachable) #1602 — roster drift: relay-roster-fix-0825 claimed from the same main. The older relay-1602-codex-sbx-0823 lane produced no code/branch/PR, so there is no duplicate implementation. The new lane will first reproduce live PTYs missing from inventory.sync, then converge heartbeat/inventory/adopted-worker state onto one authoritative broker set.
• relaycast-cloud V2 Dashboard and PTY update #30: relaycast-provider-route-0825 claimed the provider mismatch path. It will stay fail-closed—no cross-provider/socket fallback—and prove durable retry/error state plus recovery only after authoritative re-registration repairs provider_name.
• relaycast-cloud Spawn fixes #31/Integrate agent-trajectories for PDERO paradigm tracking #38: relaycast-node-liveness-0825 claimed both as one invariant. OPEN socket presence must not renew stale heartbeat state or make /send report delivery after liveness expires.
• relaycast-cloud PR feat: add first-class user messaging with channels and DMs #85: current head f0910327 is CI-green/mergeable but blocked by an unresolved current-head P1: an unbounded ready-agent array can exceed Cloudflare's 2,048-byte WebSocket attachment limit and leave readiness unpersisted. The blocker and required regression are recorded on the PR. This PR is only RFC relaycast#301's hosted-readiness slice, not its liveness/reaper or Relay registration-state-machine work.
• chief GitHub shared workspace and codex #66: chief-resident-fix-0825 claimed all three carried findings from ef17374, isolated from PR fix: update tests to match plan limits and skip unavailable trajectory feature #65. Regressions cover dead-PTY/stale-roster respawn, live-PTY/offline-roster no-duplicate, banner-safe JSON parsing, and selected-binary preflight. PR fix: update tests to match plan limits and skip unavailable trajectory feature #65 itself is config-only and non-overlapping.

#1593 interpretation: do not collapse all evidence into one root cause yet. PR #85 fixes a hosted readiness gap; #1602 can repair route/attach drift; but the latest recipient-side evidence here shows unread messages already queued at the agent while the final harness-injection hop stalls. Neither lane may claim this issue closed without a post-fix discriminator covering that queued-but-not-injected case.

Every lane is required to record the pre-fix regression, push a branch, open a PR, and report exact current-head checks. No merge, deploy, node restart, agent release, credential rotation, or production mutation is authorized.

[miyaontherelay]

Cross-node reliability rollup — 2026-08-25 07:18Z

Material progress since the ownership audit:

• relay Fleet node control-plane connection blackholes: node alive and spawning, reports offline with activeAgents 0, placement fails #1591: the red regression is durable: a continuously polled WS keeps pong traffic alive while application heartbeat replies are withheld; current main fails to declare the link dead (exit 101). Implementation/PR is still in progress.
• relay Cross-node attach 404s node-wide: broker heartbeat and inventory.sync disagree on which agents are running (roster drift; 9/10 agents on sf-mini unattachable) #1602: the red regression is durable: heartbeat advertises heartbeat-only-worker while reconnect inventory.sync omits it (exit 101). Implementation/PR is still in progress. This remains roster/attach state, not proof that Agents stop receiving DM injections after a few hours alive; sends still report recipientMatched #1593's final injector hop is fixed.
• relaycast-cloud V2 Dashboard and PTY update #30 → PR fix: add ping/pong keepalive to main and bridge WebSockets #86: open, clean/mergeable, CI green at 73d163fc. Regression-only because current engine already fails provider mismatch closed and permits correction only through authenticated recovery. Substantive coordinator review is recorded; independent approval is still required because the worker and coordinator share a GitHub identity.
• relaycast-cloud Spawn fixes #31/Integrate agent-trajectories for PDERO paradigm tracking #38 → PR Revoke handling #87: open, clean/mergeable, CI green and coordinator-approved at 22f034c4. It gates delivery on both aggregate-node and addressed-provider D1 liveness and closes stale sockets. Because it overlaps PR feat: add first-class user messaging with channels and DMs #85 in node.ts, the combined readiness/liveness suite must be rerun on the eventual landing head.
• relaycast-cloud PR feat: add first-class user messaging with channels and DMs #85: still at f0910327 and still blocked by its unresolved 2,048-byte attachment P1. A dedicated owner is adding exact Durable Object storage/hibernation/overflow coverage; no new head yet.
• chief GitHub shared workspace and codex #66 → PR Agent issues model not applied on spawn #67: open at af4d5d18, but the earlier approval has been replaced with changes-requested after current-head review found startup workerPid and recycled-PID ownership gaps. A dedicated owner claimed both with red-before-green regressions.
• chief PR fix: update tests to match plan limits and skip unavailable trajectory feature #65: unchanged; GitHub issue/PR comments must remain authoritative while this queued-but-not-injected defect persists. DMs are courtesy only and silence is not an availability signal.

No merge, deploy, node restart, agent release, credential rotation, or production mutation was performed.

[miyaontherelay]

Reliability rollup — updated 2026-08-25 08:03 UTC

• relay Fleet node control-plane connection blackholes: node alive and spawning, reports offline with activeAgents 0, placement fails #1591 / PR fix(broker): reconnect application-dead node links #1610 — exact head 44ddb17d is unchanged and is not ready. It resets reconnect delay after any transport-established disconnect, including before the first correlated application acknowledgement, enabling repeated 1s retries. Blocker: fix(broker): reconnect application-dead node links #1610 (comment). The active owner is amending with an application-ready-gated reset and regressions.
• relay Cross-node attach 404s node-wide: broker heartbeat and inventory.sync disagree on which agents are running (roster drift; 9/10 agents on sf-mini unattachable) #1602 / PR fix(broker): unify fleet live worker inventory #1611 — amended head 4a695b26 remains changes requested. The new tests manually construct the live candidate and later inject an already-repaired inventory, bypassing the actual WorkerRegistry/adoption seam. The reconciler it relies on predates the measured v11.8.1 incident, so an integrated production-path regression and escape explanation/fix remain required: fix(broker): unify fleet live worker inventory #1611 (review). Current CI also has one failed macOS standalone smoke caused by the hosted broker registration handshake receiving no response across all retries; all other reported checks passed.
• relaycast-cloud V2 Dashboard and PTY update #30 / PR fix: add ping/pong keepalive to main and bridge WebSockets #86 — exact head 73d163fc, CI green and independently approved; test-only fail-closed provider mismatch/recovery coverage.
• relaycast-cloud PR feat: add first-class user messaging with channels and DMs #85 — exact head 9559eb0b, CI green with two substantive current-head reviews. Exact readiness state is in synchronous DO storage with overflow/hibernation cleanup coverage. Final report: https://github.com/AgentWorkforce/relaycast-cloud/pull/85#issuecomment-5407116971.
• relaycast-cloud Spawn fixes #31/Integrate agent-trajectories for PDERO paradigm tracking #38 / PR Revoke handling #87 — exact head 22f034c4, CI green and approved; distinct requested reviewer still pending. Isolated, non-pushed feat: add first-class user messaging with channels and DMs #85+Revoke handling #87 composition preserved readiness plus aggregate/addressed-provider D1 gates and passed typecheck, NodeDO 56/56, and focused workerd 3/3: https://github.com/AgentWorkforce/relaycast-cloud/pull/87#issuecomment-5407070489.
• chief GitHub shared workspace and codex #66 / PR Agent issues model not applied on spawn #67 — externally merged by khaliqgant at 07:44:55 UTC as ece8edef, closing GitHub shared workspace and codex #66. The formal independent approval of exact merged head aafbcdfe was submitted 55 seconds later at 07:45:50 UTC after detached validation (17/17 focused, 66/66 config, 134/134 full library, syntax, and full diff checks). The coordinator did not perform or request the merge. chief PR fix: update tests to match plan limits and skip unavailable trajectory feature #65 is unchanged and still changes-requested until GitHub becomes the authoritative coordination plane.
• relaycast RFC Clear auth #301 — registration acceptance / provider-instance-conflict retry remains an explicit unowned gap. Neither Relay PR above may be credited for that Ready-state work. Coverage map: RFC: fail-closed provider reconnects and durable replay across hosted NodeDO relaycast#301 (comment).

The coordinator performed no merge, deploy, node restart, release, credential operation, production-row mutation, or shared dirty-checkout edit.

[khaliqgant]

The queued-but-never-injected case has a root cause and a fix: #1639

This addresses the gap called out earlier in this thread — "the latest recipient-side evidence here shows unread messages already queued at the agent while the final harness-injection hop stalls. Neither lane may claim this issue closed without a post-fix discriminator covering that queued-but-not-injected case." This is that discriminator. It is not a claim that #1593 is closed.

Mechanism

A parked (manual_flush) message is stamped with a RelaycastDeliveryReceipt carrying the agent_id that was live at queue time. The flush gate looked the ACK cursor up by that id:

// node_control.rs, before
let Some(cursor) = self.agents.get(receipt.agent_id.as_str()) else { return false };

That key is not stable. Two routine events invalidate it while messages sit parked:

1. bind_authoritative_identity — spawn register, token identity resolve, inventory repair. bind_identity does self.agents.remove(&previous.agent_id) and retires the id. It cannot come back: bind_identity refuses to re-adopt a retired agent_id.
2. seed_cursor on a node-control resume handshake sets acked == received == Relaycast's own position, so a parked seq at or below it can never satisfy seq == acked + 1.

can_ack_receipt returned false for both — indistinguishable from "not ACKable yet". So the flush stopped at the head message forever, returned flushed: 0, and a partial flush deliberately pins the worker back into manual_flush. Every later DM parked behind the poisoned head.

Why this matches the evidence in this thread

The recipient-side comment above is the sharpest data point and it fits exactly: the agent parked at its prompt, 18 unread counted at the recipient, none injected, and four different senders backed up simultaneously (10, 4, 2, 2). As noted there, that argues for "a single agent-scoped injector that has stopped running, rather than N independent delivery paths each failing." A head-of-line receipt that can never clear is precisely such an injector.

On the retracted flush evidence

The correction at the top of this issue withdrew the {"flushed": 0} observation as "simply the expected answer" that "carries no information about delivery."

That retraction was too generous to the code. FlushPending returned a bare count while flush_result.failure went only to a tracing::warn! — so a jammed queue and an empty queue were the same output. The 0 was a swallowed failure, not a null result. The original instinct to look there was right; the surface just could not tell you so.

#1639 makes the route return held and blocked_reason alongside flushed, so that reading is no longer ambiguous.

Fix

Classify Ready / Blocked / Orphaned. Orphaned receipts are injected and dropped without an ACK — Relaycast keeps ownership of the frame, so its redelivery policy is untouched. Withholding the ACK was always the safe half; withholding the message was the bug. A genuine ordering gap still classifies Blocked and still holds the queue.

Verified red → green with a byte-identical probe (left: 0, right: 2 → 8 passed; 0 failed), with both ordering guards passing in both directions so the fix is not "ACK everything and hope."

Scope

• Distinct from Cross-node attach 404s node-wide: broker heartbeat and inventory.sync disagree on which agents are running (roster drift; 9/10 agents on sf-mini unattachable) #1602 (roster/attach drift). Cross-node attach 404s node-wide: broker heartbeat and inventory.sync disagree on which agents are running (roster drift; 9/10 agents on sf-mini unattachable) #1602 explains routes resolving to nowhere live; this explains messages that reached the right agent's queue and were never written into the harness. Both can be true at once, and the age-correlation noted in this thread fits an accumulating identity-rebind count rather than a clock.
• This does not cover the send-side asks (2), (3), (4) in the original body — recipientMatched still means "name resolved", and an undeliverable injection still does not dead-letter with a distinguishing reason.
• The suggested per-agent injection counter remains the right instrument and is not in this PR.

Refs: #1559 (same failure seen as delivery-mode churn), #1639.