fix(provider): idle watchdog fires during long-running tool calls, injecting spurious recovery prompts and failing healthy runs

[jrob5756]

Summary

CopilotProvider's idle watchdog has no concept of an in-flight tool call. The Copilot SDK emits no events between tool.execution_start and tool.execution_complete, so the idle clock freezes for the entire duration of a tool call. Any tool that runs longer than idle_timeout_seconds (hardcoded 90.0) is misdiagnosed as a stuck session, and the provider injects a "you appear to have gotten stuck… please continue" prompt into a session that is working correctly.

Five of those in a row raise a non-retryable ProviderError and kill the workflow.

This is distinct from #483 (dead runtime / BrokenPipeError). Both fired in the same implement run, but this one has its own root cause and its own fix.

Observed failure

examples/implement.yaml, agent epic_reviewer. From the captured stderr log:

├─ [epic_reviewer] 🔧 read_agent
│ args: {"agent_id": "e11-r2-types", "wait": true, "timeout": 180}
├─ [epic_reviewer] ⚠️ Idle Recovery 1/5 - last: tool 'read_agent' ← fires mid-tool
│ result: Agent is still running after waiting 180s.
│ agent_id: e11-r2-types, status: running, elapsed: 2450s
│ [epic_reviewer] ⏳ Processing (turn 1)...
├─ [epic_reviewer] ⚠️ Idle Recovery 1/5 - last: tool 'read_agent'
├─ [epic_reviewer] ⚠️ Idle Recovery 2/5 - last: tool 'read_agent'
├─ [epic_reviewer] ⚠️ Idle Recovery 3/5 - last: tool 'read_agent'
├─ [epic_reviewer] ⚠️ Idle Recovery 4/5 - last: tool 'read_agent'
├─ [epic_reviewer] ⚠️ Idle Recovery 5/5 - last: tool 'read_agent'

workflow_failed ProviderError
Session appears stuck after 5 recovery attempts.
Last activity: tool 'read_agent' was executing.

The Idle Recovery 1/5 line sits between the tool's args and its result — the watchdog fired while the tool was legitimately blocked.

read_agent(wait=true, timeout=180) cannot return in under 180s when the polled agent is still running, so with a 90s idle timeout it is structurally guaranteed to trip the watchdog twice on every call. epic_reviewer polls long-lived sub-reviewers (elapsed: 1931s, 2450s), which is why this agent fails and the others in the same workflow do not.

Root cause

on_event bumps the idle clock for any non-ignored event (src/conductor/providers/copilot.py:1596-1598):

if event_type not in _IDLE_IGNORED_EVENTS:
 last_activity_ref[0] = event_type
 last_activity_ref[2] = time.monotonic()

tool.execution_start bumps it once (copilot.py:1649) — and then nothing else arrives until the tool finishes. _wait_with_idle_detection therefore sees a stale clock (copilot.py:2340):

if time_since_last_event < idle_timeout:
 recovery_attempts = 0 # "events still flowing" — never reached during a tool call
 ...
 continue

# Genuinely idle — no events for the full timeout period
recovery_attempts += 1

The "reset on progress" guard added for #4/#5 does not help here, because a running tool produces zero progress events to reset against.

Verified against the SDK event stream

Parsed from conductor-implement-…events.jsonl (20,548 events, 7,073 tool calls):

• Longest bash call: 298.0s, with 0 SDK events between its tool.execution_start and tool.execution_complete.
• 158 of 7,073 tool calls (2.2%) exceeded the 90s idle timeout — bash test-suite runs at 250–298s, read_agent polls at 183–193s, read_bash at 188–245s.
• Estimated ~232 spurious recovery prompts injected in that single run (bash 114, rg 51, view 44, read_bash 21).

Second-order damage: corrupted structured output

The spurious prompts are queued and delivered when the tool returns. The agent — which has already produced its answer — replies conversationally:

"Task already complete. … the final JSON report was already delivered in my previous message. No further action needed."

response_content is overwritten by every assistant.message (copilot.py:1601), so that prose replaces the valid JSON and structured-output extraction fails:

agent_parse_recovery coder attempt=1/5
 Could not extract JSON from response: 'Epic E10 is complete. All implementation,
 tests, and verification finished successfully — the final JSON report was already
 delivered in my previous message. No f…'

19 of the 21 agent_parse_recovery events in this run had a >90s tool call in the same agent turn:

agent	longest tool in turn
coder	298.0s bash
epic_fixer	272.6s bash
coder	263.2s bash
epic_fixer	265.5s bash
coder	222.1s read_bash

So the bug does not only kill runs outright — it silently burns parse-recovery budget and corrupts otherwise-valid agent output.

No workaround available

Only max_session_seconds is plumbed from YAML to IdleRecoveryConfig (src/conductor/providers/factory.py:115-118):

idle_recovery_config = None
if max_session_seconds is not None:
 idle_recovery_config = IdleRecoveryConfig(
 max_session_seconds=max_session_seconds,
 )

idle_timeout_seconds and max_recovery_attempts (copilot.py:163-164) are not reachable from a workflow file, so an author hitting this has no escape hatch short of editing provider source.

Secondary defect

last_activity_ref[1] is set on tool.execution_start (copilot.py:1649) and never cleared on tool.execution_complete. _build_stuck_info therefore always reports the last tool that started, so the error claims a tool "was executing" even when it completed long before. This actively misdirects diagnosis — the failure above names read_agent, whose call had already returned.

Suggested fix

1. Track in-flight tool calls in on_event — increment on tool.execution_start, decrement on tool.execution_complete — and suppress idle recovery while the count is non-zero. max_session_seconds and max_agent_iterations remain the backstop for a genuinely wedged tool, so nothing loses its guard.
2. Clear last_activity_ref[1] on tool.execution_complete so _build_stuck_info distinguishes "tool X is executing" from "tool X was the last thing that ran".
3. Expose idle_timeout_seconds (and probably max_recovery_attempts) in runtime: alongside max_session_seconds, so a workflow with legitimately long tool calls can tune it.

A regression test should assert that a simulated tool.execution_start with no following events for > idle_timeout_seconds does not produce a recovery prompt until tool.execution_complete arrives.

Environment

• conductor 0.1.33
• provider copilot, Copilot SDK CLI 1.0.78
• run id 3b0acfa3, examples/implement.yaml

[jrob5756]

Starting to take a look at this.