[bug] The board showed three dead runs as "completed" and kept "Needs you" empty while all three waited on a human — a coding card has no needs-human status and follows the session, not the run

[serge-ivo]

Split out of #541, which is about why those runs died. This one is about why the board said the
opposite of what happened.

Observed, production, 2026-08-13

Three Pilot runs on instance a1d3522f (AIPA coder) died on 2026-08-12 with
stuck not resolved in time — each after opening a human handoff and waiting 15 minutes for an
answer (#541). Here is that instance's board right now (instance_board):

columns: Waiting | Running | Needs you | Failed | Blocked | Done | Cancelled

Running: csess_22d08431… "Coding: dev/aipa (codex)" status: running
Done: csess_43be1f0e… "Coding: dev/aipa (codex)" status: completed
 csess_2dd3124c… "Coding: dev/aipa (claude)" status: completed
 csess_302422b7… "Coding: dev/aipa (claude)" status: completed
Needs you: (empty)
Failed: (empty)

• csess_2dd3124c is run 82739cb6, which failed. Board says completed.
• csess_302422b7 is run 1dbb3e54, which failed. Board says completed.
• csess_22d08431 is run ebc70ac1, which failed at 2026-08-12T22:56:23Z. Board says
  running, ~16 hours later.
• "Needs you" was empty the entire time all three sat waiting on a human, and it is empty now.
• "Failed" has never held any of them.

The board is the surface with a column literally named for this state, and it showed three
successes and one in-flight job for four dead runs.

Mechanism — the card tracks the SESSION, the run outcome never reaches it

1. There is no needs-human status a coding card can hold. lib/coding-board.ts:21:

export type CodingCardStatus = "running" | "completed" | "cancelled" | "failed";

Grep-verified as complete: upsertCodingSessionCard is only ever called with "running"
(coding-store.ts:472), and closeCodingSessionCards only with "cancelled" (:511) or
status === "error" ? "failed" : "completed" (:576). And:

$ grep -c needs_human workers/api/src/workflows/coding-session.ts
0

Its two peer workflows both have it — workflows/job-apply.ts parks a task in needs_human on
handoff, and lib/agent-loop.ts:204-207 maps escalated → needs_human. The Pilot is the only
autonomous workflow whose human handoff is invisible to the board.

2. The card's status is a function of the session's lifecycle, not the run's outcome.
coding-store.ts:576:

if (changed) await closeCodingSessionCards(env, instanceId, userId, [sessionId],
 status === "error" ? "failed" : "completed");

The workflow does pass "error" for a failed run (coding-session.ts:754), but only inside the
branch gated on shouldEndSessionAfterRun({ openedByRun: … }) (:743). A session the human
opened survives the run, so that branch is skipped and the card is never updated at all — which
is csess_22d08431, still "running" 16 hours after its run died.

I did not fully trace why the other two read completed rather than failed. The consistent
explanation is that the session had already left active by some other path (the End button, the
orphan reaper, or a --force takeover — endSession defaults to "ended" → completed), so the
workflow's own endSession(…, "error") matched no row, changed was false, and no card write
happened. Stated as the likely cause, not a verified one; the structural claim above does not
depend on it.

Why it matters more than a wrong badge

The owner's stated read of these runs is "they keep dying and I retype Retry". The board is where he
would look to see that a run wanted him — and it is the one surface that actively told him
everything was fine. The push notification did fire for all four handoffs (pushed_at set on
every row, verified), so the information existed; it was in the tray and nowhere durable.

What to do — cheapest first

(a) Add needs_human to CodingCardStatus and set it when the handoff opens. The handoff block
already exists at coding-session.ts:620-640 and already calls notifyUser; one
upsertCodingSessionCard(…, status: "needs_human") beside it, and one back to running on resume
(:659). This alone puts the card in the column that already renders.

(b) Make a failed run mark its card failed even when the session survives. Move the card write
out of the shouldEndSessionAfterRun branch — the card is about the work, and #271 deliberately
decoupled session lifetime from run lifetime. A session outliving its run is now normal, so a card
keyed only to session lifetime is structurally wrong, not just occasionally stale.

(c) A reconciler for stale running cards. sweepStaleRuns (lib/run-sweeper.ts) already
closes agent_loop_runs rows whose driver died after STALE_RUN_MS; it does not touch board cards.
csess_22d08431 is 16h stale. Cheapest version: have the existing sweep close the matching card.

Alternatives considered and rejected

• Derive the card from agent_loop_runs instead of the session. Rejected: [coordination] Ticket 2 — make coding work visible to a supervisor (domain writes) #206 put coding
  sessions on the board precisely so a human-driven session (no loop run at all) is visible to a
  supervisor. Keying to the run would delete that, which is the capability [coordination] Ticket 2 — make coding work visible to a supervisor (domain writes) #206 was filed to add.
• Drop the coding card and let the run's delegation card carry it. Rejected for the same reason,
  plus these three runs had taskId: null (verified in their error_log context) — they were not
  delegated, so there is no delegation card. The coding card is the only one they have.
• Only fix the notification. Rejected: the notification already worked. The defect is that the
  durable surface disagreed with it.

Acceptance criteria

1. While a Pilot run is parked in a handoff, its card is in Needs you; on resume it returns to
   Running.
2. A run that ends failed leaves its card in Failed, whether or not the session survives.
3. No card sits in Running more than STALE_RUN_MS after its run reached a terminal state.
4. A human-opened session with no loop run still shows as Running while live ([coordination] Ticket 2 — make coding work visible to a supervisor (domain writes) #206 preserved).

Regression risk

• (b) touches the one place a session leaves active (coding-store.ts:562), whose comment warns
  that every terminal path routes through it so the card "can reliably follow". Moving the write out
  must not create a second writer that can disagree — prefer calling
  upsertCodingSessionCard/closeCodingSessionCards explicitly from the workflow's end step and
  leaving endSession's write for the paths that have no run.
• (a) adds a status the console's board grouping must map to a column. defaultBoardColumns(["coding"])
  currently maps running/completed/cancelled/failed with no declaration (coding-board.ts:19-20) —
  check needs_human lands in "Needs you" and not in an "other" bucket before shipping.
• Test that would catch a regression: assert board state at three points of one Pilot run
  (start → handoff → terminal) rather than only at the end. No current test observes the mid-run
  card, which is why this survived.

Cross-refs: #541 (why the runs died), #206 (why the card exists at all), #271 (why sessions outlive
runs), #349 (the same "handoff notification lands somewhere other than the control" defect, fixed
for apply and browser-task — this is the coding instance of it).

[serge-ivo]

Fixed in b51fc83 — CI green, Deploy API Worker succeeded, api.proagentstore.online/health serving.

First: the thing you flagged as unverified, traced

"I did not fully trace why the other two read completed rather than failed… the End button, the orphan reaper, or a --force takeover."

It is not the End button and not --force. --force writes cancelled (suspendSessionsFromOtherNodes, coding-store.ts:515), and the End button is a deliberate human act. Grep-complete, there are exactly three card writers:

writer	status
createSession	running
suspendSessionsFromOtherNodes	cancelled
endSession	status === "error" ? "failed" : "completed" — and only when changed

So the terminal word belongs to whichever writer reaches the session first, and four of endSession's five callers pass the default "ended". Two of those four are crons: the 6-hour idle reaper (coding-session-sweeper.ts:181, via reapSession) and the orphan reconciler (coding-store.ts:610). Their "ended" is a statement about a session being untouched, not about how the work went, and it lands hours later with nobody watching.

That gives two structurally sufficient routes to completed on a failed run, both ordinary, neither needing a human:

1. the run does not own the session → it never calls endSession at all → a reaper closes it later as "ended";
2. the run owns it but a reaper moved the row first → the run's endSession(…, "error") matches nothing, changed is false, no card write.

Which of the two applied to csess_2dd3124c and csess_302422b7 is not determinable from code, and I could not settle it from data: this environment has no production D1 credential (~/dev/ops SOPS holds no pags.CLOUDFLARE_API_TOKEN; the live one is the CI repo secret). Recording that as a gap rather than guessing. The fix does not depend on it — both routes are the same defect, and it is the one your mechanism section already named: the run's verdict never reaches the card.

Two corrections to the issue that fall out of this: endSession is called from five places, not four (the idle reaper and the orphan reconciler are separate); and (c) is not just about a stale running card — a reaper actively writing completed over a dead run is the sharper half.

The fix is an ordering, not an extra write

Adding a card write in the workflow would not have been enough: a reaper firing six hours later would have overwritten it. So two authorities, ranked.

The RUN writes unconditionally (setCodingSessionCardStatus), at three points:

• tl-start → running. Not decoration: since [bug] Delegation is single-use — the Pilot ends the session its own driver requires, and the error blames the runner #271 a session outlives its run, so a second run on a session a previous run left failed would otherwise begin under a Failed card, and a resolved handoff could not come back out of "Needs you". createSession only opens a card for a session it CREATES.
• handoff opens → needs_human; handoff resolves → running. In coding-pause.ts, injected like every other effect there, so it still unit-tests with no D1, no relay and no Workflow. Written before the notify and the announce, so a throw in either cannot reproduce the incident (tray knows, board does not).
• end → statusFor(reason), outside the shouldEndSessionAfterRun branch, which is what left csess_22d08431 "running" for 16 hours.

The SESSION writes openOnly — closeWorkCards(..., { openOnly: true }) adds AND status IN ('running','needs_human'). It may fill a card the run left open; it may never overwrite one the run has terminated. This is the mirror of the rule setWorkCardProgress already states for the other direction ("it cannot resurrect a finished card"), and together they mean a card moves one way once a run has spoken. It is opt-in, so the pipeline domain (one writer) is untouched.

One table for the column

CodingCardStatus is now "running" | LoopRunStatus — where LoopRunStatus is what statusFor returns — rather than a hand-written union. So the coding card, the loop-run row and the delegation card cannot land in different columns for one run, and a stop reason added later cannot produce a status the board has no column for. A test walks LoopStopReason's own declaration and asserts every reason resolves to a real column.

That found a live second instance of this bug, which I fixed here: delegationTaskRecord took "running" | "completed" | "failed" off a local runSucceeded(outcome), so a delegated run that stopped waiting on its owner was closed failed on the supervisor's board while its own run row said needs_human. #541 made that reachable (an unanswered handoff now survives as stuck); #546 added a second way in.

needs_human is not terminal (isCodingCardOpen), so it carries no completedAt. And the engine-usage-limit park deliberately does not touch the card: #541 is explicit that it must not ping like a question, it reopens by itself, and a column that asks for attention it does not need stops being read.

Acceptance criteria

1. ✅ Parked in a handoff → Needs you; on resume → Running. Exercised for real against the pause machine, including the ordering.
2. ✅ A run ending failed leaves its card in Failed whether or not the session survives — and no later reaper can flip it.
3. ✅ sweepStaleRuns now closes the coding card too, joined through agent_loop_runs.session_id (0116, [enhancement] The run report cannot say the engine is working — a loop run is not linked to its coding session, so live runState never reaches describeLoopRun (#459 criterion 1) #465 — null for chat/pipeline runs, which is exactly the rows with no card), through the open-only close so a live workflow still wins.
4. ✅ Preserved. createSession still opens a human-driven session's card as running and the End route still closes it; [coordination] Ticket 2 — make coding work visible to a supervisor (domain writes) #206's case is untouched.

Regression risks you named

• "must not create a second writer that can disagree" — handled by the ranking above rather than by hoping. The run is unconditional, the session is open-only; that asymmetry is the rule.
• *"check needs_human lands in Needs you and not an other bucket"* — asserted against defaultBoardColumns(["coding"])+columnForStatus`, together with the four statuses that already worked, so the vocabulary was widened rather than swapped.
• "a test that asserts board state at three points of one Pilot run" — the test you asked for by name now exists. The two mid-run transitions run for real; the wiring is asserted over the workflow source with the call-site count as the denominator (ADR 0002 G1/G2), so a fourth writer has to be read rather than absorbed.

Every new assertion was proven by watching it fail (G4): closing the status union → 1 red; dropping openOnly → 2 red; moving the terminal write back inside the branch → 1 red; removing the pause machine's card call → 2 red; removing the sweeper's → 2 red.

workflows/coding-session.ts had to absorb three writes at 793/799, so startOnRunner is extracted to lib/coding-session-relaunch.ts — it never needed the Workflow. 792 after; the pin is unchanged.

Not done

• The idle reaper still writes completed for a session it reaps after 6h of silence with no run behind it. cancelled would be more honest, but it is a behaviour change for human-driven sessions with no incident behind it, and openOnly already stops it mattering wherever a run had an opinion. Worth its own ticket if you want the board to stop calling an abandoned session a success.
• Not verified against your live board. Confirming the four cards moved needs a signed-in read of instance_board on a1d3522f, and this environment has no PAGS API token; triggering a fresh handoff would mean driving one of your live agents, which I did not do. The existing four cards are terminal or stale, so they will not retro-fix — csess_22d08431 is the one to watch: it will close to Failed on the next sweepStaleRuns pass, and the next Pilot handoff should appear in Needs you while it waits.

[serge-ivo]

Heads-up for whoever finishes this: b51fc83 added needs_human to CodingCardStatus (lib/coding-board.ts:36, isCodingCardOpen at :47), and needs_human is the first status in the orphan sweep's SELECT — routes/instances-runtime.ts:414:

WHERE instance_id = ?1 AND user_id = ?2 AND status IN ('needs_human', 'running')
 AND type NOT IN ('job.apply_agent', 'browser.task')

coding.session is not in that exclusion list, so a Pilot parked in a handoff — the exact state this issue built the "Needs you" column for — is expired to failed by the next runner registration on that machine, with the message "its browser session is gone". Filed as #567 with the production evidence for the running path (a session marked complete at 01:31 that then pushed 14 more times to main until 03:31). Not observed for the needs_human path, which post-dates the evidence — it is a code reading of two files that landed independently.

#567 also argues the sweep's denylist should be inverted rather than extended, and specifically that narrowing it to status = 'running' would undo this issue.

[serge-ivo]

Verified as DELIVERED — closing on evidence, no further code written

Re-checked all four acceptance criteria against main today. b51fc83 and 37bb2ea (#592) are both
ancestors of HEAD; the ticket was simply never closed.

1. Parked in a handoff → "Needs you"; on resume → "Running". ✅
CodingCardStatus is "running" | LoopRunStatus (lib/coding-board.ts:38), and LoopRunStatus
includes needs_human (lib/agent-loop.ts:213) — structural, not a hand-written union. The writes
are real call sites, not stubs: lib/coding-pause.ts:145 await deps.card("needs_human") inside
waitForHuman, before notify (:148) and announce (:157); :203 await deps.card("running") on resolve. The dep is wired at workflows/coding-session.ts:568. Column
mapping confirmed at lib/agent-capabilities.ts:141.

2. A failed run leaves its card Failed whether or not the session survives. ✅
The terminal write is outside the branch — coding-session.ts:791 opens
shouldEndSessionAfterRun, :815 closes it, and the setCodingSessionCardStatus(…, statusFor(...)) write is at :831. Every session-side close is openOnly
(coding-board.ts:146), so no reaper can overwrite a run's verdict.

3. No card stuck in Running past the stale bound. ✅
lib/run-sweeper.ts:80-82 now selects session_id, and :109-112 closes the coding card of every
stale run that had one, through the open-only path so a live workflow still wins.

4. #206 preserved. ✅
createSession still opens a human-driven session's card as running (lib/coding-store.ts:476-484)
and the End route still closes it (:580). 37bb2ea's read-time reconciliation only touches
runStatus === "needs_human" && sessionEnded (lib/board-runs.ts:144), and lib/board.ts:386 keeps
item.userStatus outranking it.

Tests: coding-pause.test.ts (24), run-sweeper.test.ts (10), coding-board.test.ts (22),
board-runs.test.ts (15) — 71 passed. Named cases include "puts the card in Needs you the moment
the handoff opens, and back when it resolves (#553)" and "writes the run's verdict from statusFor,
OUTSIDE the end-the-session branch".

The open thread on this ticket is also closed

The comment warning that the orphan sweep at routes/instances-runtime.ts:414 would expire a parked
coding card is stale. That denylist is gone: :396-403 now binds AND type IN (…) from
ORPHANABLE_TASK_TYPES, an allowlist derived in lib/runtime-task-ownership.ts:91-94, where
coding.session is classified "cloud" (:73) and unknown types default to not-sweepable (:98).
#567 is closed.

What is NOT claimed

The four original production cards (csess_22d08431 et al.) were not re-read — they are terminal or
stale and will not retro-fix. And the self-declared "Not done" from the delivery comment stands: the
6-hour idle reaper still writes completed for a session it reaps with no run behind it. openOnly
means it cannot overwrite a run's verdict, so it affects none of the four criteria — worth its own
ticket only if the board should stop calling an abandoned session a success.