[Feature]: supervise long-running threads — turns complete while background subagents are still working, and nothing restarts the run

[eddy-curly]

Area

apps/server

Problem or use case

An overnight agent orchestration run stops silently and stays stopped until a human notices.

Evidence from a real run — thread 3a85bdd3-8264-474c-aabc-a1a62eaf5d3a "Overnight TDD agent orchestration" (project eddy-knowledge-hub), 31 turns, 2026-07-30T21:34Z → 2026-07-31T14:54Z:

	Stall A	Stall B
Turn marked completed	00:19:19Z	07:40:44Z
Background activities still arriving afterward	1,358 (until 00:52:43Z, +33m)	61 (until 07:48:47Z, +8m)
Then total silence for	3h 31m	6h 50m
Ended by	human: "Can you continue the process in a /loop?"	human: "You stopped again"

Stall A's 1,358 orphaned activities break down as 715 task.progress, 57 task.started, 58 task.completed, 527 context-window.updated, 1 checkpoint.captured. In other words, subagents kept working for 33 minutes inside a turn the server had already closed.

In Stall B a local_workflow task started activity fires at 07:40:32Z — 12 seconds before the turn closes at 07:40:44Z.

Both turns report state: completed with no error tone anywhere in the thread. The thread is not settled, snoozed, archived, or rate-limited: it has zero t3x.auto-resume.* activities and no entry in t3x-auto-resume.json, so auto-resume was never involved and could not have recovered it.

All 18 user messages in the thread are human (no message id carries the t3x-auto-resume: prefix). There is no automated continuation anywhere — the human is the loop, all night:

00:10:20Z Did you get started?
00:11:05Z Any updates?
00:13:53Z What happening here? Are you still working on the overnight tasks?
00:18:55Z So you are working on it now? No need to schedule for later
04:23:48Z Can you continue the process in a /loop?
05:49:31Z Continuing?
06:00:20Z Great. Continue working. I am going to bed now, ensure that your sub agents are not dying
06:42:14Z Are you working through it?
07:15:17Z Continuing right?
14:39:06Z You stopped again
14:53:19Z Can you note down your current progress in PROGRESS.md? I will have another agent take over

Proposed solution

1. Supervision. Let a thread be marked as a long-running/supervised run. When its turn completes but the run is not actually done, T3 Code re-prompts it instead of leaving it idle.

A stall signal is already derivable from existing projections: activities (task.progress, task.started, task.completed) continuing to arrive after projection_turns.completed_at, then going quiet with no new turn requested.

2. Make it visible and pinned. A distinct thread type that pins to the top of the sidebar and is visually differentiated, showing live state (running / stalled / last activity age) so an overnight run is identifiable at a glance rather than sinking into the list.

Why this matters

Unattended overnight work is the entire point of leaving an orchestration running. Today a stall costs the whole remaining window — 6h50m in the case above — and is only discoverable by manually opening the thread and reading its timeline. The user cannot tell a finished run from a dead one without reading it.

Smallest useful scope

• A per-thread "supervised" flag, plus pinned/distinct rendering in the web sidebar.
• A server-side stall detector that re-prompts once when a supervised thread has been idle for N minutes with no active turn.

Cron-scheduled thread creation is explicitly out of scope for a first pass.

Alternatives considered

Relying on the agent's own /loop self-scheduling. This is what was tried and it is what failed — the agent's wakeups do not survive the turn boundary, so once the turn closes there is nothing left to re-invoke it. Manual nudging works but requires a human awake at 4am.

Risks or tradeoffs

• Re-prompting a genuinely finished run wastes tokens and can confuse the agent. Needs a done-condition and an attempt cap — auto-resume's maxResumesPer24h is prior art worth reusing rather than reinventing.
• Interaction with settled / snoozed and with auto-resume's guards needs an explicit decision, not two mechanisms racing to revive the same thread.
• Per AGENTS.md: needs a decision for web, desktop, and mobile sidebars, and a reverse state (unpin / unsupervise) — a one-way door is a bug.

Examples or references

The mechanism described here — a turn reporting completed while its background tasks are still streaming activities — may deserve its own bug report. It is documented here because it is the direct cause of the stalls, and because a supervisor needs to know about it either way.

[radroid]

Correction: the premise "the agent cannot wake itself" is wrong

The Loop Watch design (docs/t3x/loop/DESIGN.md, commit cbb3a1373) was built on the conclusion that the Claude Agent SDK exposes no scheduling primitive, so a stall could only be detected externally. That conclusion was wrong, and it was wrong because it was drawn from the npm package rather than the binary the package spawns.

Verified today:

• The SDK ships the full Claude Code binary (v2.1.170, 222 MB, manifest.json). sdk.mjs/assistant.mjs/bridge.mjs contain zero occurrences of CronCreate/ScheduleWakeup — the implementations are compiled into the binary.
• That binary registers four model-callable tools unconditionally in its master tool list: CronCreate, CronDelete, CronList, ScheduleWakeup. They are also declared in the SDK's public type surface (sdk-tools.d.ts:9-40, ToolInputSchemas).
• The cron scheduler is constructed inside print.ts — the non-interactive stream-json (SDK) entrypoint — not only the Ink REPL hook. On fire it enqueues a meta user prompt into the live session and kicks the drain loop; the host harness is never consulted.
• Firing requires the input stream to stay open. ClaudeAdapter.ts:3181-3189 does exactly that — an unbounded Queue → long-lived AsyncIterable that is never closed between turns.
• A cron-fired turn arrives as assistant output with no active turn, which lands on the existing synthetic-turn path (ClaudeAdapter.ts:2468-2507) and closes normally on result (:2548-2563) — so it produces a complete turn.started → turn.completed pair.
• T3 passes no allowedTools, disallowedTools, toolAliases or hooks, and full-access mode auto-allows every tool. Nothing gates these today.

So /loop is reachable from a T3 thread right now, with zero adapter changes.

What this does and does not change for #38

It does not kill Loop Watch. Two constraints keep it necessary:

1. Session-lifetime only. tengu_kairos_cron_durable is false, so session crons live in the binary's in-process state. A server restart, a crashed session, or a provider respawn leaves nothing to fire — precisely the case where an overnight run dies silently.
2. tengu_kairos_loop_dynamic is a remote gate. It happens to be true in this machine's ~/.claude.json cache, but Anthropic can flip it, and ScheduleWakeup then degrades silently to a no-op. A self-paced loop that depends on it has no failure signal.

Loop Watch is the external backstop for both. But the design should be revised:

• options.hooks is entirely unused in T3 — 30 events, zero subscriptions. The Stop hook input carries session_crons and background_tasks, which is an authoritative answer to "is this session going to wake itself again, and is background work still in flight?" That is strictly better evidence than updated_at staleness, and it directly addresses the trap that made this issue hard (a turn marked completed while subagents stream).
• A thread that has armed its own wakeup should not be nudged. Loop Watch needs a "the session says it will wake at HH:MM" guard, or the two schedulers will fight — the exact hazard the design's own prompt line ("I am the scheduler for this thread") was written to prevent.

Filed as #42, which covers surfacing/bounding the binary's scheduling and proposes a durable T3-native wake_me MCP tool as the version that survives restarts. #38 should be re-scoped against it before Phase 1 is built.

Also worth recording: T3 Code already hands models app-native tools. apps/server/src/mcp/toolkits/preview/tools.ts on upstream/main ships 14 preview_* tools over a T3-hosted HTTP MCP server. Adding a toolkit is provider-agnostic and costs no fork seam — which is what #42 and #43 both build on.

[radroid]

Subagent tracking: the mechanism is now captured, not inferred

Ran claude 2.1.220 in --output-format stream-json — the same mode ClaudeAdapter drives — and captured a real subagent lifecycle both foreground and backgrounded. Full write-up and raw NDJSON: docs/t3x/loop/SUBAGENTS.md + docs/t3x/loop/captures/.

The #38 shape, reproduced

[29] assistant TOOL_USE Agent run_in_background=True
[30] system.background_tasks_changed
[31] system.task_started a7af6055981f2b9a0
...
[76] result.success <<< TURN ENDS
[77] result.success <<< TURN ENDS (two results — num_turns=2, then 1)
[78] system.background_tasks_changed ← after the turn ended
[79] system.task_updated {patch:{status:"killed", end_time:…}}
[80] system.task_notification {status:"stopped"}

Task lifecycle events keep arriving after result. That is not a T3 bug — it is how the protocol works, and run_in_background: true is the trigger. In this capture the task was killed because claude -p exits at turn end; in T3 the session is long-lived, so the work keeps running instead. That is how 3a85bdd3 produced 1,358 activities across 33 minutes inside a closed turn.

Two correlation keys, both already on the wire

task_started carries both task_id and tool_use_id, and every message produced inside a subagent carries parent_tool_use_id equal to that tool_use_id. Main-agent messages carry null. Complete subagent attribution, no hooks required:

task_id ←→ task_started.tool_use_id == parent_tool_use_id on nested messages

What T3 throws away

wire event	T3 today	file:line
task_started	mapped → task.started	ClaudeAdapter.ts:2681
task_progress	mapped → task.progress	:2692
task_updated	dropped (case "task_updated": return;)	:2716
task_notification	mapped → task.completed	:2718
background_tasks_changed	swallowed	:2597
parent_tool_use_id	read only to discard subagent token deltas	:2082

task_updated.patch is typed {status?: 'pending'|'running'|'completed'|'failed'|'killed'|'paused', description?, end_time?, total_paused_ms?, error?, is_backgrounded?} (sdk.d.ts:4086-4093). is_backgrounded is the flag that says "this will outlive the turn". killed/failed are terminal states task_notification may never report.

Consequence for this design

The current trigger infers liveness from updated_at staleness because the premise was that T3 cannot know whether background work is in flight. It can. An open-task set keyed on task_id gives an exact answer at turn-completion time, and needs zero new upstream surface — these are arms of a switch the fork already owns.

The idle timer stays, for two reasons: the roster is hot-stream state that a restart loses (the SQL column survives), and a task killed by a provider crash may never emit task_notification, so the set still needs a TTL.

Concrete revisions:

• Add "no open tasks in the roster" as guard chore(t3x): auto-build on a 12h cadence, not every main commit #15, as a veto on firing.
• Surface open-task count in the pill — Loop paused — 3 subagents working — instead of binary working/stalled.
• Map task_updated and background_tasks_changed rather than dropping them.

---

Second half: what the loop thread should look like when you open it

Requirement from use, not speculation: "I will be interacting with the loop at intervals to check if it is operating well. I will also have other chat sessions that check on the running loop. It would be nice if I could see a summary of how the loop has been running any time I open the loop chat, then have a button to expand and see the loop running in action right now and scroll through the chat history."

This is a real gap the current design does not cover. A 31-turn overnight transcript does not answer "is this going well?" — that is exactly why the night was spent typing "Are you still working on it?". The pill answers "alive or dead"; it does not answer "how has it been going".

Proposal: a run digest as the default view, with expand-to-transcript

Opening a supervised thread lands on a digest, not the raw transcript. One button — "Open live chat" — swaps to the normal ChatView with full history and live streaming. The toggle is sticky per thread. Nothing about the underlying thread changes; this is a second render of the same data.

Digest contents, all derived from projections (no model authorship, so it cannot lie):

• Status line: state, check-ins used/budget, deadline, elapsed since arm
• Right now: current turn state + open subagents by name/description, from the roster above
• Per-check-in rollup — one row per nudge: time, what moved (turns, tool calls, files touched, commits), and how long it worked before going quiet
• Terminal reason when stopped, with the sentinel text if done
• Last 3 assistant messages, collapsed

Optionally a short agent-authored summary per check-in, clearly labelled as agent-authored and visually separated from the derived metrics. The distinction matters — a run digest whose selling point is trust must not blend measured facts with a model's account of itself.

Mount cost is near zero. The thread route already carries the fork's existing 2-line seam (_chat.$environmentId.$threadId.tsx:18,92, currently AutoResumeOverlay, which the design already converts to a fork-owned ThreadT3xOverlay). The digest renders there, gated on "this thread is armed". No ChatView.tsx edit, no new ledger row.

The "other chat sessions check on the loop" part

That requirement is not a UI feature — it is an API one. A second thread asking "how is the overnight run going?" needs the digest as data.

This is the same MCP toolkit plumbing #46 Phase 2 introduces and #42/#43 depend on: a read-only t3x_loop_status tool returning the digest JSON for a given thread, provider-agnostic because all five adapters already attach the t3-code MCP server. Read-only, so no permission story is needed beyond what the MCP server already enforces.

That makes the supervisor observable three ways from one derivation: the pill (glance), the digest (open the thread), and the tool (ask from another thread, or from your phone).

Sequencing: the digest belongs in Phase 2, not Phase 1 — Phase 1 should stay the smallest thing that fixes the night. But the check-in rollup requires recording firedAtMs and the turn/activity cursor at nudge time, and that must go into Phase 1's store schema or the history is unreconstructable after the fact.

[radroid]

Priority triage — 2026-08-07: Rank 8 / 11 — Tier 5 (re-scope against #42)

Per this issue's own comments, the founding premise ("the agent cannot wake itself") was wrong, so the design needs re-scoping against #42 before Phase 1 is built. Sequenced right after the #42 design spike, not started independently.

[radroid]

Triage 2026-08-07 — hold until #49 lands; upstream has built a large part of this

upstream/main contains a2ca89aa feat: native subagent & workflow observability (#5219) — 6,871 lines across 46 files — which is in the 103-commit backlog this fork has not merged (#49). It overlaps this issue's second comment closely enough that the design should be re-cut rather than implemented.

What it ships that this issue proposes to build:

this issue's proposal	upstream a2ca89aa
"Map task_updated and background_tasks_changed rather than dropping them" (ClaudeAdapter.ts:2716, :2597)	ClaudeAdapter.ts +598, ProviderRuntimeIngestion.ts +177, providerRuntime.ts +175
An open-task roster keyed on task_id, as guard #15	apps/server/src/orchestration/ThreadBackgroundLiveness.ts (160 lines + 173 of tests)
"Surface open-task count in the pill — Loop paused — 3 subagents working"	a sidebar liveness state, Sidebar.logic.ts +54, SidebarV2.tsx +47
Subagent attribution via parent_tool_use_id	packages/client-runtime/src/state/subagentRuntime.ts (924 lines)
A per-run view distinct from the raw transcript	apps/web/src/components/AgentsPanel.tsx (568 lines) + RightPanelTabs.tsx

Building the roster fork-side now would be a fork path duplicating an upstream capability — the hazard SEAMS.md records, and this time it is visible in advance.

What survives, and is still worth building

Upstream's module doc is explicit about its boundaries:

in-memory per-thread background liveness for the sidebar status pill … no persistence, no migration. After a server restart the registry is empty until new task events arrive

So upstream observes; it does not act. It never re-prompts, never nudges, has no budget, no deadline, no attempt cap, and does not survive a restart. The thing that actually fixes the 6h50m night in the issue body — "re-prompts it instead of leaving it idle" — is untouched, and it is now a much smaller feature because the liveness signal arrives for free and provider-agnostically (the same commit covers Codex).

Two direct consequences for the design in docs/t3x/loop/DESIGN.md:

• The updated_at staleness trigger can probably be retired in favour of reading upstream's derived liveness, which is exact rather than inferred. That removes the design's most fragile part.
• Durability becomes the fork's whole contribution. Upstream's registry is empty after a restart by design; the design's persisted store is what makes an overnight run survivable, and it should be framed that way.

Recommend: leave open, do not start Phase 1, and re-cut the design against a2ca89aa once #49 lands. Sequenced after #49 and #39.

[radroid]

Closing this. The design is archived on main, and the half of it that upstream has since built is now in the fork — but the half that isn't is worth stating plainly rather than letting the close imply it's all handled.

What upstream landed

a2ca89aa1 feat: native subagent & workflow observability (#5219) is now an ancestor of main (arrived in the 2026-08-08 sync). Its own module doc describes this exact issue:

The turn can settle while native background work runs on (subagent fleets, workflow runs, Monitor watch loops); the shell previously showed nothing.

It ships apps/server/src/orchestration/ThreadBackgroundLiveness.ts, maps the task.started / task.progress / task.completed lifecycle this issue identified as being dropped, and surfaces it as a sidebar liveness pill (Sidebar.logic.ts) and a composer banner (ChatView.tsx).

That closes proposal 2 — "make it visible." A stall is now identifiable at a glance instead of requiring you to open the thread and read its timeline.

What it does not close

Proposal 1 — supervision — is untouched. I checked every consumer of the signal: all of them are presentation. Nothing re-prompts, nothing enqueues a turn, nothing acts. And the service is explicitly in-memory:

no persistence, no migration. After a server restart the registry is empty until new task events arrive

So on the run that motivated this issue, upstream's work would have shown a "working" pill for the 33 minutes of orphaned activity in Stall A — and then gone quiet for 3h31m exactly as before, because nothing continues the run. The human is still the loop.

The fork's auto-resume doesn't cover it either: it triggers on needs-input, which is why that thread had zero t3x.auto-resume.* activities. "Turn completed while background work continued" is a different trigger.

Surviving residue: durability + acting on the signal. That is a much smaller piece of work than this issue originally scoped, and it should be re-specified against what upstream now provides rather than resumed from the old design.

Why close rather than keep open

The design here (docs/t3x/loop/, from a 4-design / 12-judgement panel) was never built, and its premise has moved twice: pingdotgg#5219 supersedes the subagent-tracking mechanism it was built around, and upstream PR pingdotgg#3638 ships schedule_task / delegate_task MCP tools. Every codebase line number in it was measured against the 2026-08-02 merge-base and has drifted through several syncs since. It carries a status banner saying all of this. Keeping the issue open against a stale design is worse than closing it and re-opening a narrow one for the residue when it's actually wanted.

Research and captures are preserved at docs/t3x/loop/ — DESIGN.md, OPTIONS.md, RESEARCH.md, SUBAGENTS.md, and the captures/*.ndjson traces.