Make Loop resilient to network retry dead-ends and terminal spam

docs/SCHEDULING.md

@@ -42,6 +42,21 @@ If the assistant finishes a turn, Loop sends the prompt again. If that next turn
 
 Short continuation prompts such as `continue`, `keep going`, and `devam et` receive extra project-continuation guidance. The agent is told to treat the turn as continuation of the existing conversation/repository, inspect relevant files/TODO/progress/git state as needed, choose the next unfinished step, and avoid redoing completed work.
 
+**Plain `/loop devam et` is intentionally infinite.** A model saying "done" does not silently change that contract.
+
+If you explicitly want **continue until the project is actually done**, say so:
+
+```text
+/loop devam et bitene kadar devam tamamen projeyi bitir
+```
+
+Completion-bounded idle loops add two safeguards:
+
+- before declaring terminal completion, the agent is instructed to perform a fresh verification pass;
+- Loop auto-pauses only after **two consecutive current assistant turns** both say the project/work is complete **and** that no work remains. A reply that names a next/remaining task resets the terminal signal.
+
+This avoids post-completion spam without weakening the deliberately infinite `/loop devam et` form.
+
 For a real project, a stronger version is:
 
 ```text
@@ -137,45 +152,79 @@ Watch jobs remain dormant until their watch condition is triggered, then use the
 Before dispatching a Loop-owned turn, the runtime checks:
 
 1. no Loop run is already being dispatched for the session;
-2. OpenCode is not reporting a live running turn that still has unfinished assistant output;
+2. OpenCode is not reporting a live running or retrying turn that still owns the session;
 3. no active tool call is known for the session;
 4. no busy descendant/subtask session is known;
 5. `noOverlap` / active-run guards allow another turn;
 6. the job is still enabled, unpaused, and within its configured limits.
 
 If any of those checks fail, the job remains due and Loop retries later.
 
+## Network/provider outage recovery
+
+A provider/network outage is different from an ordinary stale `busy` acknowledgement.
+
+When OpenCode reports:
+
+```text
+session.status = retry
+```
+
+Loop treats the host as the current turn owner. It does **not** age that status into `idle`, and it does not inject another prompt on top of the retrying request.
+
+Likewise, if `session.status()` itself cannot be read because the network is down, Loop fails closed as busy/unknown rather than assuming idle.
+
+For Loop-owned turns:
+
+- transient errors such as `fetch failed`, connection loss/reset, DNS/transient socket failures, request timeouts, 429, and retryable 5xx/provider-unavailable errors are classified as infrastructure failures;
+- a failed infrastructure attempt does **not** consume the logical `runCount` or ordinary `failureCount`;
+- if that failed attempt had temporarily reached `--max-runs`, the job is re-enabled after the refund;
+- retries use exponential backoff (5s, 10s, 20s, 40s, capped at 60s);
+- if an explicit OpenCode `retry` remains stuck for 2 minutes, Loop aborts only the **Loop-owned active turn**, refunds that logical run, and returns it to backoff scheduling.
+
+The watchdog never aborts an unrelated foreground/user retry when no Loop-owned active run exists.
+
+Useful log events include:
+
+```text
+network-dispatch-error
+network-action-error
+provider-retry-recovery
+```
+
+This makes an outage visible without converting it into either a dead job or overlapping autonomous turns.
+
 ## Stale `busy` recovery
 
-Some OpenCode TUI builds can leave `session.status` at `busy` or `retry` after a plugin command acknowledgement even though the assistant message is already completed. This can otherwise produce the classic symptom:
+Some OpenCode TUI builds can leave `session.status` at `busy` after a plugin command acknowledgement even though the assistant message is already completed. This can otherwise produce the classic symptom:
 
 ```text
 Loop added
 runCount = 0
 lastRunAt = 0
 ```
 
-The runtime now cross-checks stale live status with the chronological session tail **before the first Loop run too**.
+The runtime cross-checks stale `busy` with the chronological session tail **before the first Loop run too**.
 
 Recovery is conservative:
 
-- latest assistant tail has a real completion timestamp -> stale busy may be recovered to idle;
+- latest assistant tail has a real completion timestamp -> stale `busy` may be recovered to idle;
 - latest assistant tail is unfinished -> remain busy;
 - latest message is user/non-assistant -> remain busy;
 - active tool or busy child session -> remain busy;
-- unknown completion -> remain busy.
+- unknown completion -> remain busy;
+- provider `retry` -> **never** use stale-busy age recovery; wait for host completion/idle or the Loop-owned retry watchdog;
+- status API read failed -> remain conservative; do not infer idle.
 
-So Loop can recover a stale host status without treating a genuinely running turn as finished.
-
-A recovery is written to `loop.log` as:
+A stale-busy recovery is written to `loop.log` as:
 
 ```text
 status-message-idle-recovery
 ```
 
 ## Busy deferral logging
 
-When a due job cannot run because the session is still busy, Loop now emits throttled diagnostics instead of silently leaving only the original `add` line.
+When a due job cannot run because the session is still busy, Loop emits throttled diagnostics instead of silently leaving only the original `add` line.
 
 Typical event:
 
@@ -193,7 +242,7 @@ Inspect recent events with:
 
 ## `/loop-status`
 
-Status now separates the schedule definition from its current state.
+Status separates the schedule definition from its current state.
 
 Examples:
 
@@ -203,7 +252,7 @@ schedule=every 5m, first after 5m | state=due in 3m
 schedule=once after 5m | state=due; waiting for idle
 ```
 
-This distinction is important: **due** is a timing fact; **waiting for idle** is an admission/safety fact.
+This distinction is important: **due** is a timing fact; **waiting for idle/retry** is an admission/safety fact.
 
 ## `/loop-doctor` and session-bound jobs
 
@@ -273,6 +322,12 @@ After the project state is established, a short continuation loop is enough:
 
 Because `devam et` is recognized as continuation shorthand, later turns are instructed to resume the existing project rather than start a new interpretation from scratch.
 
+If the desired contract is to stop when the project is demonstrably complete, make that explicit instead:
+
+```text
+/loop --safe --ask-never --progress-file progress.md devam et bitene kadar; projeyi bitir
+```
+
 ## Stopping and limits
 
 Idle loops are intentionally unlimited by default. Bound them when needed:

package.json

@@ -21,8 +21,8 @@
  "build:plugin": "bun build src/source/v1.js --outfile=src/index.js --target=bun --format=esm --external=@opencode-ai/plugin/tool",
  "build:plugin:npm": "npm run build:plugin",
  "prepack": "node --check src/index.js",
- "check": "node --check src/source/v1.js && node --check src/source/core/args.js && node --check src/source/core/continuation.js && node --check src/source/core/schedule-syntax.js && node --check src/source/core/state.js && node --check src/source/core/jobs.js && node --check src/source/core/process.js && node --check src/source/opencode/sdk.js && node --check src/source/opencode/session-context.js && node --check src/source/opencode/command-router.js && node --check src/source/opencode/goal-commands.js && node --check src/source/opencode/loop-commands.js && node --check src/source/opencode/loop-registration.js && node --check src/source/runtime/session-activity.js && node --check src/source/runtime/session-status.js && node --check src/source/runtime/schedule-policy.js && node --check src/source/runtime/scheduler-diagnostics.js && node --check src/source/runtime/companion-goal.js && node --check src/source/runtime/loop-diagnostics.js && node --check src/source/runtime/compaction.js && node --check src/source/runtime/action-dispatch.js && node --check src/source/runtime/run-finalization.js && node --check src/source/runtime/run-admission.js && node --check src/source/runtime/executor.js && node --check src/source/runtime/scheduler.js && node --check src/source/runtime/goal-prompt.js && node --check src/source/runtime/goal-report.js && node --check src/source/runtime/goal-evidence.js && node --check src/source/runtime/goal-runtime.js && node --check src/source/runtime/goal-policy.js && node --check src/source/runtime/goal-steering.js && node --check src/source/runtime/job-workspace.js && node --check src/source/opencode2/prompt-runtime.js && node --check src/source/opencode2/diagnostics.js && node --check src/source/opencode2/logging.js && node --check src/source/legacy-v1.js && node --check src/index.js && node --check scripts/install-node.mjs && node --check scripts/install-with-goals.mjs && node --check scripts/loopd.mjs && node --check scripts/install-test.mjs && node --check scripts/goal-companion-test.mjs && node --check scripts/loopd-test.mjs && node --check scripts/smoke-test.mjs && node --check scripts/host-adapter-contract-test.mjs && node --check scripts/command-router-test.mjs && node --check scripts/goal-command-handlers-test.mjs && node --check scripts/loop-command-handlers-test.mjs && node --check scripts/loop-registration-test.mjs && node --check scripts/schedule-syntax-test.mjs && node --check scripts/schedule-policy-test.mjs && node --check scripts/session-activity-test.mjs && node --check scripts/session-status-test.mjs && node --check scripts/session-status-idle-recovery-test.mjs && node --check scripts/compaction-runtime-test.mjs && node --check scripts/executor-runtime-test.mjs && node --check scripts/scheduler-runtime-test.mjs && node --check scripts/goal-runtime-test.mjs && node --check scripts/goal-policy-test.mjs && node --check scripts/goal-steering-test.mjs && node --check scripts/job-workspace-test.mjs && node --check scripts/v2-prompt-runtime-test.mjs && node --check scripts/v2-prompt-interval-test.mjs && node --check scripts/v2-command-runtime-test.mjs && node --check scripts/v2-command-adapter-test.mjs && node --check scripts/v2-diagnostics-test.mjs && node --check scripts/v2-logging-test.mjs && node --check scripts/comprehensive-watchdog.mjs && node --check scripts/comprehensive-test.mjs && node --check scripts/host-loop-canary.mjs && node --check scripts/host-goal-steering-canary.mjs && node --check scripts/publish-workflow-test.mjs",
- "test": "node scripts/publish-workflow-test.mjs && node scripts/command-router-test.mjs && node scripts/goal-command-handlers-test.mjs && node scripts/loop-command-handlers-test.mjs && node scripts/loop-registration-test.mjs && node scripts/schedule-syntax-test.mjs && node scripts/schedule-policy-test.mjs && node scripts/session-activity-test.mjs && node scripts/session-status-test.mjs && node scripts/session-status-idle-recovery-test.mjs && node scripts/compaction-runtime-test.mjs && node scripts/action-dispatch-test.mjs && node scripts/run-finalization-test.mjs && node scripts/run-admission-test.mjs && node scripts/executor-runtime-test.mjs && node scripts/scheduler-runtime-test.mjs && node scripts/goal-runtime-test.mjs && node scripts/goal-policy-test.mjs && node scripts/goal-steering-test.mjs && node scripts/job-workspace-test.mjs && node scripts/v2-prompt-runtime-test.mjs && node scripts/v2-prompt-interval-test.mjs && node scripts/v2-command-runtime-test.mjs && node scripts/v2-command-adapter-test.mjs && node scripts/v2-diagnostics-test.mjs && node scripts/v2-logging-test.mjs && node scripts/install-test.mjs && node scripts/goal-companion-test.mjs && node scripts/loopd-test.mjs && node scripts/smoke-test.mjs && node scripts/host-adapter-contract-test.mjs && node scripts/comprehensive-watchdog.mjs",
+ "check": "node --check src/source/v1.js && node --check src/source/core/args.js && node --check src/source/core/continuation.js && node --check src/source/core/schedule-syntax.js && node --check src/source/core/state.js && node --check src/source/core/jobs.js && node --check src/source/core/process.js && node --check src/source/opencode/sdk.js && node --check src/source/opencode/session-context.js && node --check src/source/opencode/command-router.js && node --check src/source/opencode/goal-commands.js && node --check src/source/opencode/loop-commands.js && node --check src/source/opencode/loop-registration.js && node --check src/source/runtime/session-activity.js && node --check src/source/runtime/session-status.js && node --check src/source/runtime/schedule-policy.js && node --check src/source/runtime/scheduler-diagnostics.js && node --check src/source/runtime/companion-goal.js && node --check src/source/runtime/loop-diagnostics.js && node --check src/source/runtime/compaction.js && node --check src/source/runtime/action-dispatch.js && node --check src/source/runtime/network-recovery.js && node --check src/source/runtime/terminal-guard.js && node --check src/source/runtime/run-finalization.js && node --check src/source/runtime/run-admission.js && node --check src/source/runtime/executor.js && node --check src/source/runtime/scheduler.js && node --check src/source/runtime/goal-prompt.js && node --check src/source/runtime/goal-report.js && node --check src/source/runtime/goal-evidence.js && node --check src/source/runtime/goal-runtime.js && node --check src/source/runtime/goal-policy.js && node --check src/source/runtime/goal-steering.js && node --check src/source/runtime/job-workspace.js && node --check src/source/opencode2/prompt-runtime.js && node --check src/source/opencode2/diagnostics.js && node --check src/source/opencode2/logging.js && node --check src/source/legacy-v1.js && node --check src/index.js && node --check scripts/install-node.mjs && node --check scripts/install-with-goals.mjs && node --check scripts/loopd.mjs && node --check scripts/install-test.mjs && node --check scripts/goal-companion-test.mjs && node --check scripts/loopd-test.mjs && node --check scripts/smoke-test.mjs && node --check scripts/host-adapter-contract-test.mjs && node --check scripts/command-router-test.mjs && node --check scripts/goal-command-handlers-test.mjs && node --check scripts/loop-command-handlers-test.mjs && node --check scripts/loop-registration-test.mjs && node --check scripts/schedule-syntax-test.mjs && node --check scripts/schedule-policy-test.mjs && node --check scripts/session-activity-test.mjs && node --check scripts/session-status-test.mjs && node --check scripts/session-status-idle-recovery-test.mjs && node --check scripts/network-recovery-test.mjs && node --check scripts/compaction-runtime-test.mjs && node --check scripts/executor-runtime-test.mjs && node --check scripts/scheduler-runtime-test.mjs && node --check scripts/goal-runtime-test.mjs && node --check scripts/goal-policy-test.mjs && node --check scripts/goal-steering-test.mjs && node --check scripts/job-workspace-test.mjs && node --check scripts/v2-prompt-runtime-test.mjs && node --check scripts/v2-prompt-interval-test.mjs && node --check scripts/v2-command-runtime-test.mjs && node --check scripts/v2-command-adapter-test.mjs && node --check scripts/v2-diagnostics-test.mjs && node --check scripts/v2-logging-test.mjs && node --check scripts/comprehensive-watchdog.mjs && node --check scripts/comprehensive-test.mjs && node --check scripts/host-loop-canary.mjs && node --check scripts/host-goal-steering-canary.mjs && node --check scripts/publish-workflow-test.mjs",
+ "test": "node scripts/publish-workflow-test.mjs && node scripts/command-router-test.mjs && node scripts/goal-command-handlers-test.mjs && node scripts/loop-command-handlers-test.mjs && node scripts/loop-registration-test.mjs && node scripts/schedule-syntax-test.mjs && node scripts/schedule-policy-test.mjs && node scripts/session-activity-test.mjs && node scripts/session-status-test.mjs && node scripts/session-status-idle-recovery-test.mjs && node scripts/network-recovery-test.mjs && node scripts/compaction-runtime-test.mjs && node scripts/action-dispatch-test.mjs && node scripts/run-finalization-test.mjs && node scripts/run-admission-test.mjs && node scripts/executor-runtime-test.mjs && node scripts/scheduler-runtime-test.mjs && node scripts/goal-runtime-test.mjs && node scripts/goal-policy-test.mjs && node scripts/goal-steering-test.mjs && node scripts/job-workspace-test.mjs && node scripts/v2-prompt-runtime-test.mjs && node scripts/v2-prompt-interval-test.mjs && node scripts/v2-command-runtime-test.mjs && node scripts/v2-command-adapter-test.mjs && node scripts/v2-diagnostics-test.mjs && node scripts/v2-logging-test.mjs && node scripts/install-test.mjs && node scripts/goal-companion-test.mjs && node scripts/loopd-test.mjs && node scripts/smoke-test.mjs && node scripts/host-adapter-contract-test.mjs && node scripts/comprehensive-watchdog.mjs",
  "canary:host": "node scripts/host-loop-canary.mjs && node scripts/host-goal-steering-canary.mjs",
  "install:global": "node scripts/install-with-goals.mjs",
  "pack:zip": "node scripts/make-zip.mjs"