diff --git a/docs/SCHEDULING.md b/docs/SCHEDULING.md index 3006fc5..be23329 100644 --- a/docs/SCHEDULING.md +++ b/docs/SCHEDULING.md @@ -42,6 +42,21 @@ If the assistant finishes a turn, Loop sends the prompt again. If that next turn Short continuation prompts such as `continue`, `keep going`, and `devam et` receive extra project-continuation guidance. The agent is told to treat the turn as continuation of the existing conversation/repository, inspect relevant files/TODO/progress/git state as needed, choose the next unfinished step, and avoid redoing completed work. +**Plain `/loop devam et` is intentionally infinite.** A model saying "done" does not silently change that contract. + +If you explicitly want **continue until the project is actually done**, say so: + +```text +/loop devam et bitene kadar devam tamamen projeyi bitir +``` + +Completion-bounded idle loops add two safeguards: + +- before declaring terminal completion, the agent is instructed to perform a fresh verification pass; +- Loop auto-pauses only after **two consecutive current assistant turns** both say the project/work is complete **and** that no work remains. A reply that names a next/remaining task resets the terminal signal. + +This avoids post-completion spam without weakening the deliberately infinite `/loop devam et` form. + For a real project, a stronger version is: ```text @@ -137,7 +152,7 @@ Watch jobs remain dormant until their watch condition is triggered, then use the Before dispatching a Loop-owned turn, the runtime checks: 1. no Loop run is already being dispatched for the session; -2. OpenCode is not reporting a live running turn that still has unfinished assistant output; +2. OpenCode is not reporting a live running or retrying turn that still owns the session; 3. no active tool call is known for the session; 4. no busy descendant/subtask session is known; 5. `noOverlap` / active-run guards allow another turn; @@ -145,9 +160,43 @@ Before dispatching a Loop-owned turn, the runtime checks: If any of those checks fail, the job remains due and Loop retries later. +## Network/provider outage recovery + +A provider/network outage is different from an ordinary stale `busy` acknowledgement. + +When OpenCode reports: + +```text +session.status = retry +``` + +Loop treats the host as the current turn owner. It does **not** age that status into `idle`, and it does not inject another prompt on top of the retrying request. + +Likewise, if `session.status()` itself cannot be read because the network is down, Loop fails closed as busy/unknown rather than assuming idle. + +For Loop-owned turns: + +- transient errors such as `fetch failed`, connection loss/reset, DNS/transient socket failures, request timeouts, 429, and retryable 5xx/provider-unavailable errors are classified as infrastructure failures; +- a failed infrastructure attempt does **not** consume the logical `runCount` or ordinary `failureCount`; +- if that failed attempt had temporarily reached `--max-runs`, the job is re-enabled after the refund; +- retries use exponential backoff (5s, 10s, 20s, 40s, capped at 60s); +- if an explicit OpenCode `retry` remains stuck for 2 minutes, Loop aborts only the **Loop-owned active turn**, refunds that logical run, and returns it to backoff scheduling. + +The watchdog never aborts an unrelated foreground/user retry when no Loop-owned active run exists. + +Useful log events include: + +```text +network-dispatch-error +network-action-error +provider-retry-recovery +``` + +This makes an outage visible without converting it into either a dead job or overlapping autonomous turns. + ## Stale `busy` recovery -Some OpenCode TUI builds can leave `session.status` at `busy` or `retry` after a plugin command acknowledgement even though the assistant message is already completed. This can otherwise produce the classic symptom: +Some OpenCode TUI builds can leave `session.status` at `busy` after a plugin command acknowledgement even though the assistant message is already completed. This can otherwise produce the classic symptom: ```text Loop added @@ -155,19 +204,19 @@ runCount = 0 lastRunAt = 0 ``` -The runtime now cross-checks stale live status with the chronological session tail **before the first Loop run too**. +The runtime cross-checks stale `busy` with the chronological session tail **before the first Loop run too**. Recovery is conservative: -- latest assistant tail has a real completion timestamp -> stale busy may be recovered to idle; +- latest assistant tail has a real completion timestamp -> stale `busy` may be recovered to idle; - latest assistant tail is unfinished -> remain busy; - latest message is user/non-assistant -> remain busy; - active tool or busy child session -> remain busy; -- unknown completion -> remain busy. +- unknown completion -> remain busy; +- provider `retry` -> **never** use stale-busy age recovery; wait for host completion/idle or the Loop-owned retry watchdog; +- status API read failed -> remain conservative; do not infer idle. -So Loop can recover a stale host status without treating a genuinely running turn as finished. - -A recovery is written to `loop.log` as: +A stale-busy recovery is written to `loop.log` as: ```text status-message-idle-recovery @@ -175,7 +224,7 @@ status-message-idle-recovery ## Busy deferral logging -When a due job cannot run because the session is still busy, Loop now emits throttled diagnostics instead of silently leaving only the original `add` line. +When a due job cannot run because the session is still busy, Loop emits throttled diagnostics instead of silently leaving only the original `add` line. Typical event: @@ -193,7 +242,7 @@ Inspect recent events with: ## `/loop-status` -Status now separates the schedule definition from its current state. +Status separates the schedule definition from its current state. Examples: @@ -203,7 +252,7 @@ schedule=every 5m, first after 5m | state=due in 3m schedule=once after 5m | state=due; waiting for idle ``` -This distinction is important: **due** is a timing fact; **waiting for idle** is an admission/safety fact. +This distinction is important: **due** is a timing fact; **waiting for idle/retry** is an admission/safety fact. ## `/loop-doctor` and session-bound jobs @@ -273,6 +322,12 @@ After the project state is established, a short continuation loop is enough: Because `devam et` is recognized as continuation shorthand, later turns are instructed to resume the existing project rather than start a new interpretation from scratch. +If the desired contract is to stop when the project is demonstrably complete, make that explicit instead: + +```text +/loop --safe --ask-never --progress-file progress.md devam et bitene kadar; projeyi bitir +``` + ## Stopping and limits Idle loops are intentionally unlimited by default. Bound them when needed: diff --git a/package.json b/package.json index 4a3df0b..3a9d85e 100644 --- a/package.json +++ b/package.json @@ -21,8 +21,8 @@ "build:plugin": "bun build src/source/v1.js --outfile=src/index.js --target=bun --format=esm --external=@opencode-ai/plugin/tool", "build:plugin:npm": "npm run build:plugin", "prepack": "node --check src/index.js", - "check": "node --check src/source/v1.js && node --check src/source/core/args.js && node --check src/source/core/continuation.js && node --check src/source/core/schedule-syntax.js && node --check src/source/core/state.js && node --check src/source/core/jobs.js && node --check src/source/core/process.js && node --check src/source/opencode/sdk.js && node --check src/source/opencode/session-context.js && node --check src/source/opencode/command-router.js && node --check src/source/opencode/goal-commands.js && node --check src/source/opencode/loop-commands.js && node --check src/source/opencode/loop-registration.js && node --check src/source/runtime/session-activity.js && node --check src/source/runtime/session-status.js && node --check src/source/runtime/schedule-policy.js && node --check src/source/runtime/scheduler-diagnostics.js && node --check src/source/runtime/companion-goal.js && node --check src/source/runtime/loop-diagnostics.js && node --check src/source/runtime/compaction.js && node --check src/source/runtime/action-dispatch.js && node --check src/source/runtime/run-finalization.js && node --check src/source/runtime/run-admission.js && node --check src/source/runtime/executor.js && node --check src/source/runtime/scheduler.js && node --check src/source/runtime/goal-prompt.js && node --check src/source/runtime/goal-report.js && node --check src/source/runtime/goal-evidence.js && node --check src/source/runtime/goal-runtime.js && node --check src/source/runtime/goal-policy.js && node --check src/source/runtime/goal-steering.js && node --check src/source/runtime/job-workspace.js && node --check src/source/opencode2/prompt-runtime.js && node --check src/source/opencode2/diagnostics.js && node --check src/source/opencode2/logging.js && node --check src/source/legacy-v1.js && node --check src/index.js && node --check scripts/install-node.mjs && node --check scripts/install-with-goals.mjs && node --check scripts/loopd.mjs && node --check scripts/install-test.mjs && node --check scripts/goal-companion-test.mjs && node --check scripts/loopd-test.mjs && node --check scripts/smoke-test.mjs && node --check scripts/host-adapter-contract-test.mjs && node --check scripts/command-router-test.mjs && node --check scripts/goal-command-handlers-test.mjs && node --check scripts/loop-command-handlers-test.mjs && node --check scripts/loop-registration-test.mjs && node --check scripts/schedule-syntax-test.mjs && node --check scripts/schedule-policy-test.mjs && node --check scripts/session-activity-test.mjs && node --check scripts/session-status-test.mjs && node --check scripts/session-status-idle-recovery-test.mjs && node --check scripts/compaction-runtime-test.mjs && node --check scripts/executor-runtime-test.mjs && node --check scripts/scheduler-runtime-test.mjs && node --check scripts/goal-runtime-test.mjs && node --check scripts/goal-policy-test.mjs && node --check scripts/goal-steering-test.mjs && node --check scripts/job-workspace-test.mjs && node --check scripts/v2-prompt-runtime-test.mjs && node --check scripts/v2-prompt-interval-test.mjs && node --check scripts/v2-command-runtime-test.mjs && node --check scripts/v2-command-adapter-test.mjs && node --check scripts/v2-diagnostics-test.mjs && node --check scripts/v2-logging-test.mjs && node --check scripts/comprehensive-watchdog.mjs && node --check scripts/comprehensive-test.mjs && node --check scripts/host-loop-canary.mjs && node --check scripts/host-goal-steering-canary.mjs && node --check scripts/publish-workflow-test.mjs", - "test": "node scripts/publish-workflow-test.mjs && node scripts/command-router-test.mjs && node scripts/goal-command-handlers-test.mjs && node scripts/loop-command-handlers-test.mjs && node scripts/loop-registration-test.mjs && node scripts/schedule-syntax-test.mjs && node scripts/schedule-policy-test.mjs && node scripts/session-activity-test.mjs && node scripts/session-status-test.mjs && node scripts/session-status-idle-recovery-test.mjs && node scripts/compaction-runtime-test.mjs && node scripts/action-dispatch-test.mjs && node scripts/run-finalization-test.mjs && node scripts/run-admission-test.mjs && node scripts/executor-runtime-test.mjs && node scripts/scheduler-runtime-test.mjs && node scripts/goal-runtime-test.mjs && node scripts/goal-policy-test.mjs && node scripts/goal-steering-test.mjs && node scripts/job-workspace-test.mjs && node scripts/v2-prompt-runtime-test.mjs && node scripts/v2-prompt-interval-test.mjs && node scripts/v2-command-runtime-test.mjs && node scripts/v2-command-adapter-test.mjs && node scripts/v2-diagnostics-test.mjs && node scripts/v2-logging-test.mjs && node scripts/install-test.mjs && node scripts/goal-companion-test.mjs && node scripts/loopd-test.mjs && node scripts/smoke-test.mjs && node scripts/host-adapter-contract-test.mjs && node scripts/comprehensive-watchdog.mjs", + "check": "node --check src/source/v1.js && node --check src/source/core/args.js && node --check src/source/core/continuation.js && node --check src/source/core/schedule-syntax.js && node --check src/source/core/state.js && node --check src/source/core/jobs.js && node --check src/source/core/process.js && node --check src/source/opencode/sdk.js && node --check src/source/opencode/session-context.js && node --check src/source/opencode/command-router.js && node --check src/source/opencode/goal-commands.js && node --check src/source/opencode/loop-commands.js && node --check src/source/opencode/loop-registration.js && node --check src/source/runtime/session-activity.js && node --check src/source/runtime/session-status.js && node --check src/source/runtime/schedule-policy.js && node --check src/source/runtime/scheduler-diagnostics.js && node --check src/source/runtime/companion-goal.js && node --check src/source/runtime/loop-diagnostics.js && node --check src/source/runtime/compaction.js && node --check src/source/runtime/action-dispatch.js && node --check src/source/runtime/network-recovery.js && node --check src/source/runtime/terminal-guard.js && node --check src/source/runtime/run-finalization.js && node --check src/source/runtime/run-admission.js && node --check src/source/runtime/executor.js && node --check src/source/runtime/scheduler.js && node --check src/source/runtime/goal-prompt.js && node --check src/source/runtime/goal-report.js && node --check src/source/runtime/goal-evidence.js && node --check src/source/runtime/goal-runtime.js && node --check src/source/runtime/goal-policy.js && node --check src/source/runtime/goal-steering.js && node --check src/source/runtime/job-workspace.js && node --check src/source/opencode2/prompt-runtime.js && node --check src/source/opencode2/diagnostics.js && node --check src/source/opencode2/logging.js && node --check src/source/legacy-v1.js && node --check src/index.js && node --check scripts/install-node.mjs && node --check scripts/install-with-goals.mjs && node --check scripts/loopd.mjs && node --check scripts/install-test.mjs && node --check scripts/goal-companion-test.mjs && node --check scripts/loopd-test.mjs && node --check scripts/smoke-test.mjs && node --check scripts/host-adapter-contract-test.mjs && node --check scripts/command-router-test.mjs && node --check scripts/goal-command-handlers-test.mjs && node --check scripts/loop-command-handlers-test.mjs && node --check scripts/loop-registration-test.mjs && node --check scripts/schedule-syntax-test.mjs && node --check scripts/schedule-policy-test.mjs && node --check scripts/session-activity-test.mjs && node --check scripts/session-status-test.mjs && node --check scripts/session-status-idle-recovery-test.mjs && node --check scripts/network-recovery-test.mjs && node --check scripts/compaction-runtime-test.mjs && node --check scripts/executor-runtime-test.mjs && node --check scripts/scheduler-runtime-test.mjs && node --check scripts/goal-runtime-test.mjs && node --check scripts/goal-policy-test.mjs && node --check scripts/goal-steering-test.mjs && node --check scripts/job-workspace-test.mjs && node --check scripts/v2-prompt-runtime-test.mjs && node --check scripts/v2-prompt-interval-test.mjs && node --check scripts/v2-command-runtime-test.mjs && node --check scripts/v2-command-adapter-test.mjs && node --check scripts/v2-diagnostics-test.mjs && node --check scripts/v2-logging-test.mjs && node --check scripts/comprehensive-watchdog.mjs && node --check scripts/comprehensive-test.mjs && node --check scripts/host-loop-canary.mjs && node --check scripts/host-goal-steering-canary.mjs && node --check scripts/publish-workflow-test.mjs", + "test": "node scripts/publish-workflow-test.mjs && node scripts/command-router-test.mjs && node scripts/goal-command-handlers-test.mjs && node scripts/loop-command-handlers-test.mjs && node scripts/loop-registration-test.mjs && node scripts/schedule-syntax-test.mjs && node scripts/schedule-policy-test.mjs && node scripts/session-activity-test.mjs && node scripts/session-status-test.mjs && node scripts/session-status-idle-recovery-test.mjs && node scripts/network-recovery-test.mjs && node scripts/compaction-runtime-test.mjs && node scripts/action-dispatch-test.mjs && node scripts/run-finalization-test.mjs && node scripts/run-admission-test.mjs && node scripts/executor-runtime-test.mjs && node scripts/scheduler-runtime-test.mjs && node scripts/goal-runtime-test.mjs && node scripts/goal-policy-test.mjs && node scripts/goal-steering-test.mjs && node scripts/job-workspace-test.mjs && node scripts/v2-prompt-runtime-test.mjs && node scripts/v2-prompt-interval-test.mjs && node scripts/v2-command-runtime-test.mjs && node scripts/v2-command-adapter-test.mjs && node scripts/v2-diagnostics-test.mjs && node scripts/v2-logging-test.mjs && node scripts/install-test.mjs && node scripts/goal-companion-test.mjs && node scripts/loopd-test.mjs && node scripts/smoke-test.mjs && node scripts/host-adapter-contract-test.mjs && node scripts/comprehensive-watchdog.mjs", "canary:host": "node scripts/host-loop-canary.mjs && node scripts/host-goal-steering-canary.mjs", "install:global": "node scripts/install-with-goals.mjs", "pack:zip": "node scripts/make-zip.mjs" diff --git a/scripts/network-recovery-test.mjs b/scripts/network-recovery-test.mjs new file mode 100644 index 0000000..05b16a7 --- /dev/null +++ b/scripts/network-recovery-test.mjs @@ -0,0 +1,218 @@ +import assert from "node:assert/strict" +import { isCompletionBoundedContinuation, isTerminalNoWorkReply } from "../src/source/core/continuation.js" +import { isTransientNetworkError, networkRetryDelayMs, refundInfrastructureRun } from "../src/source/runtime/network-recovery.js" +import { createSessionStatusRuntime } from "../src/source/runtime/session-status.js" +import { applyTerminalContinuationGuard } from "../src/source/runtime/terminal-guard.js" +import { createLoopExecutor } from "../src/source/runtime/executor.js" +import { clearSessionActivity, sessionStatuses, sessionStatusSeenAt } from "../src/source/runtime/session-activity.js" + +function clone(value) { return JSON.parse(JSON.stringify(value)) } + +for (const value of [ + "fetch failed", + "network connection lost", + "ECONNRESET", + "EAI_AGAIN", + "ETIMEDOUT", + "socket hang up", + "HTTP 503 service unavailable", +]) assert.equal(isTransientNetworkError(value), true, value) +assert.equal(isTransientNetworkError("HTTP 401 invalid API key"), false) +assert.equal(networkRetryDelayMs(1, 5_000, 60_000), 5_000) +assert.equal(networkRetryDelayMs(2, 5_000, 60_000), 10_000) +assert.equal(networkRetryDelayMs(9, 5_000, 60_000), 60_000) + +const refunded = { + id: "limited", + runCount: 3, + maxRuns: 3, + enabled: false, + lastRunAt: 500, + failureCount: 2, +} +refundInfrastructureRun(refunded, { + runCount: 3, + previousLastRunAt: 250, + disabledByMaxRuns: true, +}, { + reason: "dispatch_failed_transient", + error: new Error("fetch failed: ECONNRESET"), + now: 1_000, +}) +assert.equal(refunded.runCount, 2, "transient infrastructure attempt must not consume max-runs") +assert.equal(refunded.enabled, true, "max-runs disable caused by a transient attempt must be rolled back") +assert.equal(refunded.lastRunAt, 250) +assert.equal(refunded.failureCount, 2, "transient outage must not consume ordinary failure budget") +assert.equal(refunded.infrastructureFailureCount, 1) + +{ + let clock = 100_000 + const activeRuns = new Map([["retry-session", { startedAt: 1, jobId: "j", job: { id: "j" } }]]) + const logs = [] + const runtime = createSessionStatusRuntime({ + activeRuns, + now: () => clock, + staleActiveRecoveryMs: 1_000, + sessionStatusCacheMs: 0, + activeRunCompletionFromMessages: async () => "unknown", + appendLoopLog: async (...args) => logs.push(args), + }) + const retryClient = { session: { status: async () => ({ data: { "retry-session": { type: "retry" } } }) } } + assert.equal(await runtime.sessionStatusType(retryClient, "retry-session", "/repo"), "retry") + clock += 120_000 + assert.equal(await runtime.sessionStatusType(retryClient, "retry-session", "/repo"), "retry", "stale retry must never be converted to idle by age alone") + assert.equal(logs.some((entry) => entry[1] === "status-stale-recovery"), false) + clearSessionActivity("retry-session") + sessionStatuses.delete("retry-session") + sessionStatusSeenAt.delete("retry-session") +} + +{ + let clock = 100_000 + const activeRuns = new Map([["offline-session", { startedAt: 1, jobId: "j", job: { id: "j" } }]]) + const runtime = createSessionStatusRuntime({ + activeRuns, + now: () => clock, + staleActiveRecoveryMs: 1_000, + sessionStatusCacheMs: 0, + activeRunCompletionFromMessages: async () => "unknown", + appendLoopLog: async () => {}, + }) + const offlineClient = { session: { status: async () => { throw new Error("network connection lost") } } } + clock += 120_000 + assert.equal(await runtime.sessionStatusType(offlineClient, "offline-session", "/repo"), "busy", "failed status reads must fail closed, not become idle") + clearSessionActivity("offline-session") + sessionStatuses.delete("offline-session") + sessionStatusSeenAt.delete("offline-session") +} + +assert.equal(isCompletionBoundedContinuation("devam et"), false) +assert.equal(isCompletionBoundedContinuation("devam et bitene kadar devam tamamen projeyi bitir"), true) +assert.equal(isTerminalNoWorkReply("Proje tamamlandı — 28/28 test yeşil. Yapılacak iş yok. Sıfır bilinen hata."), true) +assert.equal(isTerminalNoWorkReply("Proje tamamlandı ama sıradaki iş build doğrulaması."), false) + +{ + const terminalMessages = [{ + info: { role: "assistant", time: { created: 2_000, completed: 2_100 } }, + parts: [{ type: "text", text: "Proje tamamlandı — 28/28 test yeşil. Yapılacak iş yok. Sıfır bilinen hata." }], + }] + const client = { session: { messages: async () => ({ data: terminalMessages }) } } + const bounded = { + id: "bounded", + action: "devam et bitene kadar devam tamamen projeyi bitir", + scheduleMode: "idle", + lastRunAt: 1_500, + paused: false, + } + let result = await applyTerminalContinuationGuard("/repo", client, "s", bounded) + assert.equal(result.terminal, true) + assert.equal(result.pausedNow, false) + assert.equal(result.job.terminalNoWorkCount, 1) + result = await applyTerminalContinuationGuard("/repo", client, "s", result.job) + assert.equal(result.pausedNow, true, "two current terminal replies should stop an explicit until-done loop") + assert.equal(result.job.paused, true) + + const infinite = { + id: "infinite", + action: "devam et", + scheduleMode: "idle", + lastRunAt: 1_500, + paused: false, + } + const untouched = await applyTerminalContinuationGuard("/repo", client, "s", infinite) + assert.equal(untouched.terminal, false) + assert.equal(untouched.job.paused, false, "plain /loop devam et must remain intentionally infinite") +} + +{ + let clock = 1_000_000 + let liveStatus = "idle" + const states = new Map() + const schedules = [] + const fireCalls = [] + const directory = "/repo" + const sessionID = "watchdog-session" + const stateKey = `${directory}:${sessionID}` + states.set(stateKey, { jobs: [{ + id: "job", + name: "job", + action: "devam et", + scheduleMode: "idle", + enabled: true, + paused: false, + intervalMs: 0, + runCount: 0, + maxRuns: 1, + maxRuntimeMs: 0, + timeoutMs: 0, + noOverlap: true, + }] }) + + const executor = createLoopExecutor({ + workspace: { + buildPrompt: async (_directory, job) => job.action, + ensureBranch: async (_directory, job) => job, + watchChanged: async () => false, + untilReached: async () => false, + createCheckpoint: async () => {}, + }, + goalPolicy: { + runGoalChecks: async (_directory, _sessionID, job) => job, + applyGoalNoProgressGuard: async (_directory, _client, _sessionID, job) => job, + }, + scheduler: { + rememberSession: () => {}, + scheduleDueWork: async (...args) => schedules.push(args), + }, + now: () => clock, + readState: async () => clone(states.get(stateKey)), + writeState: async (_directory, _sessionID, state) => states.set(stateKey, clone(state)), + appendLoopLog: async () => {}, + notifyJob: async () => {}, + toast: async () => {}, + runShellCommand: async () => ({ code: 0, stdout: "", stderr: "" }), + fireSdk: (...args) => { fireCalls.push(args); return Promise.resolve({ data: {} }) }, + sdkCall: async () => ({}), + normalizedModelRef: () => undefined, + compactTuiCommandName: () => undefined, + guardLoopOwnedUserMessage: () => {}, + dangerousShell: () => false, + activeRunCompletionFromMessages: async () => "unknown", + busyRetryMs: 5, + providerRetryWatchdogMs: 20, + networkRetryMaxMs: 100, + }) + + const client = { + session: { + status: async () => ({ data: liveStatus === "idle" ? {} : { [sessionID]: { type: liveStatus } } }), + prompt: async () => ({ data: {} }), + abort: async () => ({ data: {} }), + }, + } + + await executor.maybeRunDueJobs(directory, client, sessionID) + assert.ok(executor.getActiveRun(sessionID), "first logical run should be active") + assert.equal(states.get(stateKey).jobs[0].runCount, 1) + assert.equal(states.get(stateKey).jobs[0].enabled, false, "max-runs can temporarily disable while the real turn is in flight") + + liveStatus = "retry" + clock += 5 + await executor.maybeRunDueJobs(directory, client, sessionID) + assert.ok(executor.getActiveRun(sessionID), "first retry observation should remain host-owned") + clock += 25 + await executor.maybeRunDueJobs(directory, client, sessionID) + + assert.equal(executor.getActiveRun(sessionID), undefined, "stuck Loop-owned retry should be released after watchdog") + const job = states.get(stateKey).jobs[0] + assert.equal(job.runCount, 0, "network watchdog recovery must refund the logical run") + assert.equal(job.enabled, true, "refunded max-runs job should be runnable again") + assert.equal(job.infrastructureFailureCount, 1) + assert.ok(fireCalls.some((call) => String(call[1]).includes("provider retry watchdog")), "watchdog must abort only the Loop-owned stuck turn") + assert.ok(schedules.some((call) => call[3] >= 5), "recovery must reschedule with backoff") + clearSessionActivity(sessionID) + sessionStatuses.delete(sessionID) + sessionStatusSeenAt.delete(sessionID) +} + +console.log("network recovery + terminal continuation guard: ok") diff --git a/scripts/session-status-test.mjs b/scripts/session-status-test.mjs index a380de7..608c235 100644 --- a/scripts/session-status-test.mjs +++ b/scripts/session-status-test.mjs @@ -158,17 +158,31 @@ try { } { - const sessionID = "stale-recovery" + const sessionID = "retry-remains-owned" track(sessionID) runtime.clearSessionStatus(sessionID) activeRuns.set(sessionID, { jobId: "job-2", job: { id: "job-2" }, startedAt: 1_000 }) completion = "unknown" + const beforeLogs = logs.length const client = { session: { status: async () => ({ data: { [sessionID]: { type: "retry" } } }) } } - assert.equal(await runtime.sessionStatusType(client, sessionID, "/repo"), "idle") - assert.deepEqual(logs.at(-1), ["/repo", "status-stale-recovery", { sessionID, job: "job-2", startedAt: 1_000 }]) + assert.equal(await runtime.sessionStatusType(client, sessionID, "/repo"), "retry") + assert.equal(logs.length, beforeLogs, "unknown retry must not use stale-busy recovery") activeRuns.delete(sessionID) } + { + const sessionID = "retry-message-complete" + track(sessionID) + runtime.clearSessionStatus(sessionID) + activeRuns.set(sessionID, { jobId: "job-retry-complete", job: { id: "job-retry-complete", name: "retry-complete" }, startedAt: 1_000 }) + completion = "completed" + const client = { session: { status: async () => ({ data: { [sessionID]: { type: "retry" } } }) } } + assert.equal(await runtime.sessionStatusType(client, sessionID, "/repo"), "idle", "a genuinely completed current assistant turn may release stale retry") + assert.deepEqual(logs.at(-1), ["/repo", "status-message-complete-recovery", { sessionID, job: "retry-complete", startedAt: 1_000 }]) + activeRuns.delete(sessionID) + completion = "unknown" + } + { const sessionID = "fallback-fresh" track(sessionID) @@ -178,7 +192,7 @@ try { assert.equal(await runtime.sessionStatusType(client, sessionID, "/repo"), "busy") activeRuns.set(sessionID, { jobId: "stale", job: {}, startedAt: clock - 60_000 }) runtime.clearSessionStatus(sessionID) - assert.equal(await runtime.sessionStatusType(client, sessionID, "/repo"), "idle") + assert.equal(await runtime.sessionStatusType(client, sessionID, "/repo"), "busy", "status transport failure must fail closed even for an old active run") activeRuns.delete(sessionID) } diff --git a/src/index.js b/src/index.js index 6a23214..5b6bd48 100644 --- a/src/index.js +++ b/src/index.js @@ -1033,13 +1033,55 @@ var CONTINUATION_SHORTHANDS = new Set([ "devam et.", "devam et bakal\u0131m" ]); +var COMPLETION_BOUNDED_PATTERNS = [ + /\bbitene kadar\b/i, + /\b(?:tamamen|komple) projeyi bitir\b/i, + /\bi\u015Fi bitir\b/i, + /\buntil (?:it(?:'s| is) )?(?:done|complete|completed|finished)\b/i, + /\bfinish (?:the )?(?:project|task|work)\b/i, + /\bkeep going until\b/i +]; +var TERMINAL_COMPLETION_PATTERNS = [ + /proje tamamland[\u0131i](?=\s|[.,;:!\u2014-]|$)/i, + /\bproject (?:is )?(?:complete|completed|finished|done)\b/i, + /\b(?:task|work) (?:is )?(?:complete|completed|finished|done)\b/i +]; +var TERMINAL_NO_WORK_PATTERNS = [ + /yap[\u0131i]lacak (?:ba\u015Fka )?i\u015F yok(?=\s|[.,;:!\u2014-]|$)/i, + /ba\u015Fka (?:bir )?i\u015F (?:kalmad[\u0131i]|yok)(?=\s|[.,;:!\u2014-]|$)/i, + /\bnothing (?:else )?left to do\b/i, + /\bno (?:more|remaining) work\b/i, + /\bno known (?:bugs|issues)\b/i, + /\bzero known (?:bugs|issues)\b/i +]; +var NEXT_WORK_PATTERNS = [ + /\bnext(?: step| task)?\b/i, + /s[\u0131i]radaki(?=\s|[.,;:!\u2014-]|$)/i, + /sonraki(?=\s|[.,;:!\u2014-]|$)/i, + /kalan (?:i\u015F|i\u015Fler|todo|ad\u0131m)(?=\s|[.,;:!\u2014-]|$)/i, + /\bremaining (?:work|task|todo|step)/i, + /devam (?:edece\u011Fim|ediyorum|etmek gerek)(?=\s|[.,;:!\u2014-]|$)/i +]; function isContinuationShorthand(value) { return CONTINUATION_SHORTHANDS.has(String(value || "").trim().toLowerCase().replace(/\s+/g, " ")); } +function isCompletionBoundedContinuation(value) { + const text = String(value || "").trim(); + return COMPLETION_BOUNDED_PATTERNS.some((pattern) => pattern.test(text)); +} +function isTerminalNoWorkReply(value) { + const text = String(value || "").trim(); + if (!text || NEXT_WORK_PATTERNS.some((pattern) => pattern.test(text))) + return false; + const completed = TERMINAL_COMPLETION_PATTERNS.some((pattern) => pattern.test(text)); + const noWork = TERMINAL_NO_WORK_PATTERNS.some((pattern) => pattern.test(text)); + return completed && noWork; +} function continuationProjectInstruction(value) { - if (!isContinuationShorthand(value)) + if (!isContinuationShorthand(value) && !isCompletionBoundedContinuation(value)) return ""; - return "Treat this as continuation of the current project and conversation, not a fresh task. Inspect the repository state, relevant files, TODO/progress notes, recent changes, and git status as needed to identify the next unfinished step. Continue from existing work, do not redo completed work, and verify meaningful changes when practical."; + const finish = isCompletionBoundedContinuation(value) ? " If you believe the project is finished, perform a fresh verification pass before declaring completion; report both that the project is complete and that no work remains only when you have concrete current evidence." : ""; + return `Treat this as continuation of the current project and conversation, not a fresh task. Inspect the repository state, relevant files, TODO/progress notes, recent changes, and git status as needed to identify the next unfinished step. Continue from existing work, do not redo completed work, and verify meaningful changes when practical.${finish}`; } // src/source/core/jobs.js @@ -2865,7 +2907,9 @@ function createSessionStatusRuntime(options = {}) { if (directory) argsList.push({ query: { directory } }, { directory }, { workspace: directory }); argsList.push({}); + let attempted = false; for (const args of argsList) { + attempted = true; try { const result = await statusMethod.call(client.session, args); const error = sdkError(result); @@ -2895,7 +2939,7 @@ function createSessionStatusRuntime(options = {}) { return { type: "idle", source: "sdk" }; } catch {} } - return; + return attempted ? { type: "unknown", source: "sdk-error" } : undefined; } async function canFinalizeActiveRun(directory, client, sessionID, active, options2 = {}) { if (hasActiveToolCalls(sessionID) || hasBusyDescendant(sessionID)) @@ -2907,24 +2951,28 @@ function createSessionStatusRuntime(options = {}) { return true; if (!options2.requireIdle) return completion === "unknown" && staleActiveRun(sessionID); + const cached = sessionStatuses.get(sessionID); + const seenAt = sessionStatusSeenAt.get(sessionID) || 0; + const cachedIdleAfterRun = cached === "idle" && seenAt > (active.startedAt || 0); const live = await readLiveSessionStatus(client, sessionID, directory); if (live?.type === "idle") return true; + if (live?.type === "unknown" && cachedIdleAfterRun) { + return true; + } if (live?.type) { - if ((live.type === "busy" || live.type === "retry") && options2.forceStale && completion === "unknown" && staleActiveRun(sessionID)) + if (live.type === "busy" && options2.forceStale && completion === "unknown" && staleActiveRun(sessionID)) return true; return false; } if (options2.forceStale && completion === "unknown" && staleActiveRun(sessionID)) return true; - const cached = sessionStatuses.get(sessionID); - const seenAt = sessionStatusSeenAt.get(sessionID) || 0; - return cached === "idle" && seenAt > (active.startedAt || 0); + return cachedIdleAfterRun; } async function recoverCompletedTailWithoutActiveRun(directory, client, sessionID, liveType, seenAt) { if (activeRuns.has(sessionID)) return false; - if (liveType !== "busy" && liveType !== "retry") + if (liveType !== "busy") return false; if (!seenAt || now2() - seenAt < sessionStatusCacheMs) return false; @@ -2952,6 +3000,11 @@ function createSessionStatusRuntime(options = {}) { if (cached && now2() - seenAt < sessionStatusCacheMs) return cached; const live = await readLiveSessionStatus(client, sessionID, directory); + if (live?.type === "unknown") { + const conservative = cached === "retry" ? "retry" : "busy"; + markSessionStatus(sessionID, conservative); + return conservative; + } if (live?.type) { if (await recoverCompletedTailWithoutActiveRun(directory, client, sessionID, live.type, seenAt)) return "idle"; @@ -2959,13 +3012,15 @@ function createSessionStatusRuntime(options = {}) { const active = activeRuns.get(sessionID); if (active) { const completion = await activeRunCompletionFromMessages2(directory, client, sessionID, active); - if (completion === "completed" || completion === "unknown" && staleActiveRun(sessionID)) { + if (completion === "completed" || live.type === "busy" && completion === "unknown" && staleActiveRun(sessionID)) { markSessionStatus(sessionID, "idle"); - await appendLoopLog2(directory, completion === "completed" ? "status-message-complete-recovery" : "status-stale-recovery", { + const logDetails = { sessionID, job: active.job?.name || active.jobId, - startedAt: active.startedAt - }); + startedAt: active.startedAt, + ...completion === "completed" ? {} : { staleStatus: live.type } + }; + await appendLoopLog2(directory, completion === "completed" ? "status-message-complete-recovery" : "status-stale-recovery", logDetails); return "idle"; } } @@ -3181,6 +3236,58 @@ ${prompt}`; return { fireAction }; } +// src/source/runtime/terminal-guard.js +function messageText(message) { + const parts = Array.isArray(message?.parts) ? message.parts : []; + const fromParts = parts.filter((part) => part?.type === "text" && typeof part.text === "string").map((part) => part.text).join(` +`).trim(); + if (fromParts) + return fromParts; + const info = message?.info || message || {}; + for (const value of [info.text, info.content, info.summary]) { + if (typeof value === "string" && value.trim()) + return value.trim(); + } + return ""; +} +async function applyTerminalContinuationGuard(directory, client, sessionID, job, options = {}) { + if (job?.scheduleMode !== "idle" || !isCompletionBoundedContinuation(job?.action)) { + return { job, terminal: false, pausedNow: false }; + } + const messages = await readRecentSessionMessages(client, sessionID, directory, options.messageLimit || 8); + if (!messages) + return { job, terminal: false, pausedNow: false }; + const tail = orderedSessionMessages(messages).at(-1); + const info = tail?.info || tail || {}; + if (info.role !== "assistant") + return { job, terminal: false, pausedNow: false }; + const completed = Number(info?.time?.completed || 0); + const created = Number(info?.time?.created || 0); + const runStarted = Number(job?.lastRunAt || 0); + if (!Number.isFinite(completed) || completed <= 0) + return { job, terminal: false, pausedNow: false }; + if (runStarted > 0 && completed < runStarted && (!Number.isFinite(created) || created < runStarted)) { + return { job, terminal: false, pausedNow: false }; + } + const text = messageText(tail); + const terminal = isTerminalNoWorkReply(text); + if (!terminal) { + if (job.terminalNoWorkCount) + job.terminalNoWorkCount = 0; + return { job, terminal: false, pausedNow: false, text }; + } + job.terminalNoWorkCount = (job.terminalNoWorkCount || 0) + 1; + job.lastTerminalNoWorkAt = Date.now(); + job.lastTerminalNoWorkSummary = text.slice(0, 1000); + const threshold = Math.max(2, Number(options.threshold) || 2); + const pausedNow = job.terminalNoWorkCount >= threshold && !job.paused; + if (pausedNow) { + job.paused = true; + job.lastFailureReason = "terminal_no_work"; + } + return { job, terminal: true, pausedNow, text }; +} + // src/source/runtime/run-finalization.js function requireFunction8(value, label) { if (typeof value !== "function") @@ -3200,7 +3307,14 @@ function createRunFinalizationRuntime(options = {}) { const toast2 = typeof options.toast === "function" ? options.toast : toast; const writeGoalReport2 = typeof options.writeGoalReport === "function" ? options.writeGoalReport : writeGoalReport; const dangerousShell2 = typeof options.dangerousShell === "function" ? options.dangerousShell : dangerousShell; + const applyTerminalContinuationGuard2 = typeof options.applyTerminalContinuationGuard === "function" ? options.applyTerminalContinuationGuard : applyTerminalContinuationGuard; async function finalizeJob(directory, client, sessionID, state, job, previousJob) { + if (job.infrastructureFailureCount) { + job.infrastructureFailureCount = 0; + job.lastInfrastructureFailure = ""; + job.lastInfrastructureError = ""; + job.lastInfrastructureFailureAt = 0; + } if (job.verifyCommand) { const verify = await runShellCommand2(job.verifyCommand, directory, job.timeoutMs || 300000); job.lastVerifyAt = now2(); @@ -3263,6 +3377,18 @@ exit=` + postrun.code + ` job = await runGoalChecks(directory, sessionID, job, client); job = await applyGoalNoProgressGuard(directory, client, sessionID, job, previousJob); } + const terminal = await applyTerminalContinuationGuard2(directory, client, sessionID, job); + job = terminal.job; + if (terminal.pausedNow) { + await appendLoopLog2(directory, "terminal-no-work", { + sessionID, + job: job.name || job.id, + count: job.terminalNoWorkCount, + summary: String(terminal.text || "").slice(0, 1000) + }); + await notifyJob2(directory, job, "terminal_no_work"); + await toast2(client, "Loop paused: completion-bounded task reported complete with no work remaining twice.", "success"); + } state.jobs = (state.jobs || []).map((candidate) => candidate.id === job.id ? job : candidate).filter((candidate) => candidate.enabled !== false || isGoalJob(candidate)); await writeState2(directory, sessionID, state); if (isGoalJob(job)) @@ -3359,9 +3485,58 @@ exit=` + preflight.code + ` return { dueJobs: dueJobs2, admitJob }; } +// src/source/runtime/network-recovery.js +var TRANSIENT_NETWORK_PATTERNS = [ + /\b(?:408|425|429|500|502|503|504|524)\b/i, + /rate[\s_-]?limit|too many requests|overloaded|service[\s_-]?unavailable|provider[_ -]?unavailable/i, + /terminated|fetch failed|failed to fetch|network[\s_-]?error|network connection lost/i, + /connection (?:error|refused|lost)|socket (?:hang up|connection was closed)|reset before headers/i, + /\b(?:enotfound|eai_again|econnrefused|econnreset|etimedout|ehostunreach|enetunreach|epipe)\b/i, + /\b(?:request|response|connection|network|stream|read) (?:timeout|timed out|time out)\b/i, + /\btimeout(?:error)?\b/i +]; +function errorText(value) { + if (value instanceof Error) + return `${value.name}: ${value.message}`; + if (typeof value === "string") + return value; + try { + return JSON.stringify(value); + } catch { + return String(value); + } +} +function isTransientNetworkError(value) { + const text = errorText(value); + return TRANSIENT_NETWORK_PATTERNS.some((pattern) => pattern.test(text)); +} +function networkRetryDelayMs(attempt, baseMs = 5000, maxMs = 60000) { + const safeAttempt = Math.max(1, Math.floor(Number(attempt) || 1)); + const safeBase = Math.max(1, Math.floor(Number(baseMs) || 5000)); + const safeMax = Math.max(safeBase, Math.floor(Number(maxMs) || 60000)); + return Math.min(safeMax, safeBase * 2 ** Math.min(8, safeAttempt - 1)); +} +function refundInfrastructureRun(job, snapshot = {}, input = {}) { + const chargedCount = Number(snapshot.runCount ?? job.runCount ?? 0); + const currentCount = Number(job.runCount || 0); + if (chargedCount > 0 && currentCount >= chargedCount) + job.runCount = Math.max(0, currentCount - 1); + if (Number.isFinite(Number(snapshot.previousLastRunAt))) + job.lastRunAt = Number(snapshot.previousLastRunAt); + if (snapshot.disabledByMaxRuns && job.maxRuns > 0 && job.runCount < job.maxRuns) + job.enabled = true; + job.infrastructureFailureCount = (job.infrastructureFailureCount || 0) + 1; + job.lastInfrastructureFailure = String(input.reason || "transient_network_failure").slice(0, 120); + job.lastInfrastructureError = errorText(input.error).slice(0, 4000); + job.lastInfrastructureFailureAt = Number(input.now || Date.now()); + return job; +} + // src/source/runtime/executor.js var DEFAULT_ACTIVE_GUARD_MS = 45000; var DEFAULT_BUSY_RETRY_MS2 = 5000; +var DEFAULT_PROVIDER_RETRY_WATCHDOG_MS = 2 * 60000; +var DEFAULT_NETWORK_RETRY_MAX_MS = 60000; function requireFunction10(value, label) { if (typeof value !== "function") throw new TypeError(`createLoopExecutor requires ${label}`); @@ -3393,8 +3568,11 @@ function createLoopExecutor(options = {}) { const dangerousShell2 = typeof options.dangerousShell === "function" ? options.dangerousShell : dangerousShell; const activeGuardMs = Number.isFinite(Number(options.activeGuardMs)) && Number(options.activeGuardMs) > 0 ? Number(options.activeGuardMs) : DEFAULT_ACTIVE_GUARD_MS; const busyRetryMs = Number.isFinite(Number(options.busyRetryMs)) && Number(options.busyRetryMs) > 0 ? Number(options.busyRetryMs) : DEFAULT_BUSY_RETRY_MS2; + const providerRetryWatchdogMs = Number.isFinite(Number(options.providerRetryWatchdogMs)) && Number(options.providerRetryWatchdogMs) > 0 ? Number(options.providerRetryWatchdogMs) : DEFAULT_PROVIDER_RETRY_WATCHDOG_MS; + const networkRetryMaxMs = Number.isFinite(Number(options.networkRetryMaxMs)) && Number(options.networkRetryMaxMs) > 0 ? Number(options.networkRetryMaxMs) : DEFAULT_NETWORK_RETRY_MAX_MS; const activeRuns = new Map; const runLocks = new Map; + const retryRuns = new Map; const statusRuntime = createSessionStatusRuntime({ activeRuns, appendLoopLog: appendLoopLog2, @@ -3407,6 +3585,7 @@ function createLoopExecutor(options = {}) { updateSessionStatusFromEvent, staleActiveRun, canFinalizeActiveRun, + sessionStatusType, sessionIsIdle, markSessionStatus, clearSessionStatus @@ -3465,6 +3644,7 @@ function createLoopExecutor(options = {}) { clearTimeout(active.timer); compactionRuntime.clearForActiveRun(sessionID, active); activeRuns.delete(sessionID); + retryRuns.delete(sessionID); } function disposeSession(sessionID) { clearActiveRun(sessionID); @@ -3472,30 +3652,106 @@ function createLoopExecutor(options = {}) { compactionRuntime.clear(sessionID); clearSessionStatus(sessionID); } + async function persistInfrastructureRefund(directory, sessionID, active, input = {}) { + const state = await readState2(directory, sessionID); + const job = (state.jobs || []).find((candidate) => candidate.id === active.jobId); + if (!job) + return { job: undefined, delayMs: busyRetryMs }; + refundInfrastructureRun(job, { + runCount: active.job?.runCount, + previousLastRunAt: active.previousLastRunAt, + disabledByMaxRuns: active.disabledByMaxRuns + }, { + reason: input.reason, + error: input.error, + now: now2() + }); + state.jobs = (state.jobs || []).map((candidate) => candidate.id === job.id ? job : candidate); + await writeState2(directory, sessionID, state); + const delayMs = networkRetryDelayMs(job.infrastructureFailureCount || 1, busyRetryMs, networkRetryMaxMs); + return { job, delayMs }; + } async function recoverActiveDispatchFailure(directory, client, sessionID, jobId, runToken, error) { const active = activeRuns.get(sessionID); if (!active || active.jobId !== jobId || active.runToken !== runToken) return false; + const transient = isTransientNetworkError(error); + const snapshot = active; clearActiveRun(sessionID); clearSessionStatus(sessionID); const message = errorMessage(error); - const state = await readState2(directory, sessionID); - const job = (state.jobs || []).find((candidate) => candidate.id === jobId); + let state = await readState2(directory, sessionID); + let job = (state.jobs || []).find((candidate) => candidate.id === jobId); + let retryDelay = busyRetryMs; if (job) { - job.failureCount = (job.failureCount || 0) + 1; - job.lastFailureReason = "dispatch_failed"; - job.lastDispatchFailure = message.slice(0, 4000); - job.lastDispatchFailureAt = now2(); - if (job.maxFailures > 0 && job.failureCount >= job.maxFailures) { - job.paused = true; - await notifyJob2(directory, job, "dispatch_failed"); + if (transient) { + const refunded = await persistInfrastructureRefund(directory, sessionID, snapshot, { + reason: "dispatch_failed_transient", + error + }); + job = refunded.job; + retryDelay = refunded.delayMs; + state = await readState2(directory, sessionID); + } else { + job.failureCount = (job.failureCount || 0) + 1; + job.lastFailureReason = "dispatch_failed"; + job.lastDispatchFailure = message.slice(0, 4000); + job.lastDispatchFailureAt = now2(); + if (job.maxFailures > 0 && job.failureCount >= job.maxFailures) { + job.paused = true; + await notifyJob2(directory, job, "dispatch_failed"); + } + state.jobs = (state.jobs || []).map((candidate) => candidate.id === job.id ? job : candidate); + await writeState2(directory, sessionID, state); } - state.jobs = (state.jobs || []).map((candidate) => candidate.id === job.id ? job : candidate); - await writeState2(directory, sessionID, state); } - await appendLoopLog2(directory, "dispatch-error", { sessionID, job: job?.name || jobId, error: message }); - await toast2(client, `Loop dispatch failed${job?.paused ? " and paused" : ""}: ${message}`, job?.paused ? "error" : "warning"); - await scheduleDueWork(directory, client, sessionID, busyRetryMs); + await appendLoopLog2(directory, transient ? "network-dispatch-error" : "dispatch-error", { + sessionID, + job: job?.name || jobId, + error: message, + ...transient ? { retryInMs: retryDelay } : {} + }); + if (transient) { + await toast2(client, `Loop network dispatch failed; retrying when the session recovers: ${message}`, "warning"); + } else { + await toast2(client, `Loop dispatch failed${job?.paused ? " and paused" : ""}: ${message}`, job?.paused ? "error" : "warning"); + } + await scheduleDueWork(directory, client, sessionID, retryDelay); + return true; + } + async function recoverStuckProviderRetry(directory, client, sessionID) { + const active = activeRuns.get(sessionID); + if (!active || active.compactionOnly) { + retryRuns.delete(sessionID); + return false; + } + const current = retryRuns.get(sessionID); + if (!current || current.runToken !== active.runToken) { + retryRuns.set(sessionID, { runToken: active.runToken, since: now2() }); + return false; + } + if (now2() - current.since < providerRetryWatchdogMs) + return false; + const snapshot = active; + try { + if (client?.session?.abort) { + await fireSdk2(client, "session.abort provider retry watchdog", client.session.abort.bind(client.session), { path: { id: sessionID }, body: {} }, { path: { sessionID }, body: {} }, { sessionID }); + } + } catch {} + clearActiveRun(sessionID); + clearSessionStatus(sessionID); + const refunded = await persistInfrastructureRefund(directory, sessionID, snapshot, { + reason: "provider_retry_watchdog", + error: `OpenCode session.status stayed retry for ${providerRetryWatchdogMs}ms` + }); + await appendLoopLog2(directory, "provider-retry-recovery", { + sessionID, + job: refunded.job?.name || snapshot.jobId, + retryForMs: now2() - current.since, + retryInMs: refunded.delayMs + }); + await toast2(client, "Loop provider retry exceeded the watchdog; the Loop-owned turn was released and will retry with backoff.", "warning"); + await scheduleDueWork(directory, client, sessionID, refunded.delayMs); return true; } async function finalizeActiveRun(directory, client, sessionID, finalizeOptions = {}) { @@ -3546,14 +3802,24 @@ function createLoopExecutor(options = {}) { } runLocks.set(sessionID, now2()); let job; + let previousLastRunAt = 0; + let disabledByMaxRuns = false; try { await finalizeActiveRun(directory, client, sessionID, { requireIdle: true, forceStale: true }); - if (!await sessionIsIdle(client, sessionID, directory)) { + const statusType = await sessionStatusType(client, sessionID, directory); + if (statusType !== "idle") { + if (statusType === "retry") { + if (await recoverStuckProviderRetry(directory, client, sessionID)) + return; + } else { + retryRuns.delete(sessionID); + } if (runOptions.force) await toast2(client, "Loop queued: session is busy; it will run on the next idle check.", "info"); await reschedule(busyRetryMs); return; } + retryRuns.delete(sessionID); const active = activeRuns.get(sessionID); const activeAge = active ? now2() - (active.startedAt || 0) : 0; const activeGuard = active?.job?.timeoutMs || active?.job?.activeRecoveryMs || activeGuardMs; @@ -3607,9 +3873,11 @@ function createLoopExecutor(options = {}) { if (runNowRequested) delete job.runNowRequestedAt; job.watchTriggered = false; + previousLastRunAt = Number(job.lastRunAt || 0); job.lastRunAt = now2(); job.runCount = (job.runCount || 0) + 1; - if (job.maxRuns > 0 && job.runCount >= job.maxRuns) { + disabledByMaxRuns = job.maxRuns > 0 && job.runCount >= job.maxRuns; + if (disabledByMaxRuns) { job.enabled = false; await notifyJob2(directory, job, "max_runs_reached"); } @@ -3648,7 +3916,9 @@ function createLoopExecutor(options = {}) { startedAt: now2(), timer, runToken, - compactionAction: result.compaction === true + compactionAction: result.compaction === true, + previousLastRunAt, + disabledByMaxRuns }); if (result.compaction && compactionRuntime.isCompleted(sessionID, job.id)) { await compactionRuntime.finalize(directory, client, sessionID); @@ -3663,6 +3933,24 @@ function createLoopExecutor(options = {}) { await reschedule(busyRetryMs); } catch (error) { clearActiveRun(sessionID); + if (isTransientNetworkError(error) && job) { + const stateAfterFailure = await readState2(directory, sessionID); + const persisted = (stateAfterFailure.jobs || []).find((candidate) => candidate.id === job.id); + if (persisted) { + refundInfrastructureRun(persisted, { + runCount: job.runCount, + previousLastRunAt, + disabledByMaxRuns + }, { reason: "action_dispatch_transient", error, now: now2() }); + stateAfterFailure.jobs = (stateAfterFailure.jobs || []).map((candidate) => candidate.id === persisted.id ? persisted : candidate); + await writeState2(directory, sessionID, stateAfterFailure); + const delayMs = networkRetryDelayMs(persisted.infrastructureFailureCount || 1, busyRetryMs, networkRetryMaxMs); + await appendLoopLog2(directory, "network-action-error", { sessionID, job: persisted.name || persisted.id, error: errorMessage(error), retryInMs: delayMs }); + await toast2(client, "Loop action hit a transient network failure; the logical run was refunded and will retry.", "warning"); + await reschedule(delayMs); + return; + } + } await toast2(client, `Loop job failed: ${error instanceof Error ? error.message : String(error)}`, "error"); await appendLoopLog2(directory, "error", { sessionID, @@ -3680,9 +3968,11 @@ function createLoopExecutor(options = {}) { clearActiveRun, disposeSession, recoverActiveDispatchFailure, + recoverStuckProviderRetry, finalizeActiveRun, fireAction, maybeRunDueJobs, + sessionStatusType, sessionIsIdle, updateSessionStatusFromEvent, markSessionStatus, diff --git a/src/source/core/continuation.js b/src/source/core/continuation.js index ff50871..3b34fa1 100644 --- a/src/source/core/continuation.js +++ b/src/source/core/continuation.js @@ -10,11 +10,62 @@ const CONTINUATION_SHORTHANDS = new Set([ "devam et bakalım", ]) +const COMPLETION_BOUNDED_PATTERNS = [ + /\bbitene kadar\b/i, + /\b(?:tamamen|komple) projeyi bitir\b/i, + /\bişi bitir\b/i, + /\buntil (?:it(?:'s| is) )?(?:done|complete|completed|finished)\b/i, + /\bfinish (?:the )?(?:project|task|work)\b/i, + /\bkeep going until\b/i, +] + +const TERMINAL_COMPLETION_PATTERNS = [ + // JS \b is ASCII-oriented and does not see Turkish dotless ı as a word + // character. Use an explicit following delimiter for Turkish terminal forms. + /proje tamamland[ıi](?=\s|[.,;:!—-]|$)/i, + /\bproject (?:is )?(?:complete|completed|finished|done)\b/i, + /\b(?:task|work) (?:is )?(?:complete|completed|finished|done)\b/i, +] + +const TERMINAL_NO_WORK_PATTERNS = [ + /yap[ıi]lacak (?:başka )?iş yok(?=\s|[.,;:!—-]|$)/i, + /başka (?:bir )?iş (?:kalmad[ıi]|yok)(?=\s|[.,;:!—-]|$)/i, + /\bnothing (?:else )?left to do\b/i, + /\bno (?:more|remaining) work\b/i, + /\bno known (?:bugs|issues)\b/i, + /\bzero known (?:bugs|issues)\b/i, +] + +const NEXT_WORK_PATTERNS = [ + /\bnext(?: step| task)?\b/i, + /s[ıi]radaki(?=\s|[.,;:!—-]|$)/i, + /sonraki(?=\s|[.,;:!—-]|$)/i, + /kalan (?:iş|işler|todo|adım)(?=\s|[.,;:!—-]|$)/i, + /\bremaining (?:work|task|todo|step)/i, + /devam (?:edeceğim|ediyorum|etmek gerek)(?=\s|[.,;:!—-]|$)/i, +] + export function isContinuationShorthand(value) { return CONTINUATION_SHORTHANDS.has(String(value || "").trim().toLowerCase().replace(/\s+/g, " ")) } +export function isCompletionBoundedContinuation(value) { + const text = String(value || "").trim() + return COMPLETION_BOUNDED_PATTERNS.some((pattern) => pattern.test(text)) +} + +export function isTerminalNoWorkReply(value) { + const text = String(value || "").trim() + if (!text || NEXT_WORK_PATTERNS.some((pattern) => pattern.test(text))) return false + const completed = TERMINAL_COMPLETION_PATTERNS.some((pattern) => pattern.test(text)) + const noWork = TERMINAL_NO_WORK_PATTERNS.some((pattern) => pattern.test(text)) + return completed && noWork +} + export function continuationProjectInstruction(value) { - if (!isContinuationShorthand(value)) return "" - return "Treat this as continuation of the current project and conversation, not a fresh task. Inspect the repository state, relevant files, TODO/progress notes, recent changes, and git status as needed to identify the next unfinished step. Continue from existing work, do not redo completed work, and verify meaningful changes when practical." + if (!isContinuationShorthand(value) && !isCompletionBoundedContinuation(value)) return "" + const finish = isCompletionBoundedContinuation(value) + ? " If you believe the project is finished, perform a fresh verification pass before declaring completion; report both that the project is complete and that no work remains only when you have concrete current evidence." + : "" + return `Treat this as continuation of the current project and conversation, not a fresh task. Inspect the repository state, relevant files, TODO/progress notes, recent changes, and git status as needed to identify the next unfinished step. Continue from existing work, do not redo completed work, and verify meaningful changes when practical.${finish}` } diff --git a/src/source/runtime/executor.js b/src/source/runtime/executor.js index 5e647ae..6a5e0ee 100644 --- a/src/source/runtime/executor.js +++ b/src/source/runtime/executor.js @@ -10,9 +10,12 @@ import { createCompactionRuntime } from "./compaction.js" import { createActionDispatcher } from "./action-dispatch.js" import { createRunFinalizationRuntime } from "./run-finalization.js" import { createRunAdmissionRuntime } from "./run-admission.js" +import { isTransientNetworkError, networkRetryDelayMs, refundInfrastructureRun } from "./network-recovery.js" const DEFAULT_ACTIVE_GUARD_MS = 45_000 const DEFAULT_BUSY_RETRY_MS = 5_000 +const DEFAULT_PROVIDER_RETRY_WATCHDOG_MS = 2 * 60_000 +const DEFAULT_NETWORK_RETRY_MAX_MS = 60_000 function requireFunction(value, label) { if (typeof value !== "function") throw new TypeError(`createLoopExecutor requires ${label}`) @@ -51,9 +54,16 @@ export function createLoopExecutor(options = {}) { const busyRetryMs = Number.isFinite(Number(options.busyRetryMs)) && Number(options.busyRetryMs) > 0 ? Number(options.busyRetryMs) : DEFAULT_BUSY_RETRY_MS + const providerRetryWatchdogMs = Number.isFinite(Number(options.providerRetryWatchdogMs)) && Number(options.providerRetryWatchdogMs) > 0 + ? Number(options.providerRetryWatchdogMs) + : DEFAULT_PROVIDER_RETRY_WATCHDOG_MS + const networkRetryMaxMs = Number.isFinite(Number(options.networkRetryMaxMs)) && Number(options.networkRetryMaxMs) > 0 + ? Number(options.networkRetryMaxMs) + : DEFAULT_NETWORK_RETRY_MAX_MS const activeRuns = new Map() const runLocks = new Map() + const retryRuns = new Map() const statusRuntime = createSessionStatusRuntime({ activeRuns, @@ -67,6 +77,7 @@ export function createLoopExecutor(options = {}) { updateSessionStatusFromEvent, staleActiveRun, canFinalizeActiveRun, + sessionStatusType, sessionIsIdle, markSessionStatus, clearSessionStatus, @@ -129,6 +140,7 @@ export function createLoopExecutor(options = {}) { if (active?.timer) clearTimeout(active.timer) compactionRuntime.clearForActiveRun(sessionID, active) activeRuns.delete(sessionID) + retryRuns.delete(sessionID) } function disposeSession(sessionID) { @@ -138,32 +150,116 @@ export function createLoopExecutor(options = {}) { clearSessionStatus(sessionID) } + async function persistInfrastructureRefund(directory, sessionID, active, input = {}) { + const state = await readState(directory, sessionID) + const job = (state.jobs || []).find((candidate) => candidate.id === active.jobId) + if (!job) return { job: undefined, delayMs: busyRetryMs } + refundInfrastructureRun(job, { + runCount: active.job?.runCount, + previousLastRunAt: active.previousLastRunAt, + disabledByMaxRuns: active.disabledByMaxRuns, + }, { + reason: input.reason, + error: input.error, + now: now(), + }) + state.jobs = (state.jobs || []).map((candidate) => candidate.id === job.id ? job : candidate) + await writeState(directory, sessionID, state) + const delayMs = networkRetryDelayMs(job.infrastructureFailureCount || 1, busyRetryMs, networkRetryMaxMs) + return { job, delayMs } + } + async function recoverActiveDispatchFailure(directory, client, sessionID, jobId, runToken, error) { const active = activeRuns.get(sessionID) if (!active || active.jobId !== jobId || active.runToken !== runToken) return false + const transient = isTransientNetworkError(error) + const snapshot = active clearActiveRun(sessionID) clearSessionStatus(sessionID) const message = errorMessage(error) - const state = await readState(directory, sessionID) - const job = (state.jobs || []).find((candidate) => candidate.id === jobId) + let state = await readState(directory, sessionID) + let job = (state.jobs || []).find((candidate) => candidate.id === jobId) + let retryDelay = busyRetryMs if (job) { - job.failureCount = (job.failureCount || 0) + 1 - job.lastFailureReason = "dispatch_failed" - job.lastDispatchFailure = message.slice(0, 4000) - job.lastDispatchFailureAt = now() - if (job.maxFailures > 0 && job.failureCount >= job.maxFailures) { - job.paused = true - await notifyJob(directory, job, "dispatch_failed") + if (transient) { + const refunded = await persistInfrastructureRefund(directory, sessionID, snapshot, { + reason: "dispatch_failed_transient", + error, + }) + job = refunded.job + retryDelay = refunded.delayMs + state = await readState(directory, sessionID) + } else { + job.failureCount = (job.failureCount || 0) + 1 + job.lastFailureReason = "dispatch_failed" + job.lastDispatchFailure = message.slice(0, 4000) + job.lastDispatchFailureAt = now() + if (job.maxFailures > 0 && job.failureCount >= job.maxFailures) { + job.paused = true + await notifyJob(directory, job, "dispatch_failed") + } + state.jobs = (state.jobs || []).map((candidate) => candidate.id === job.id ? job : candidate) + await writeState(directory, sessionID, state) } - state.jobs = (state.jobs || []).map((candidate) => candidate.id === job.id ? job : candidate) - await writeState(directory, sessionID, state) } - await appendLoopLog(directory, "dispatch-error", { sessionID, job: job?.name || jobId, error: message }) - await toast(client, `Loop dispatch failed${job?.paused ? " and paused" : ""}: ${message}`, job?.paused ? "error" : "warning") - await scheduleDueWork(directory, client, sessionID, busyRetryMs) + await appendLoopLog(directory, transient ? "network-dispatch-error" : "dispatch-error", { + sessionID, + job: job?.name || jobId, + error: message, + ...(transient ? { retryInMs: retryDelay } : {}), + }) + if (transient) { + await toast(client, `Loop network dispatch failed; retrying when the session recovers: ${message}`, "warning") + } else { + await toast(client, `Loop dispatch failed${job?.paused ? " and paused" : ""}: ${message}`, job?.paused ? "error" : "warning") + } + await scheduleDueWork(directory, client, sessionID, retryDelay) + return true + } + + async function recoverStuckProviderRetry(directory, client, sessionID) { + const active = activeRuns.get(sessionID) + if (!active || active.compactionOnly) { + retryRuns.delete(sessionID) + return false + } + const current = retryRuns.get(sessionID) + if (!current || current.runToken !== active.runToken) { + retryRuns.set(sessionID, { runToken: active.runToken, since: now() }) + return false + } + if (now() - current.since < providerRetryWatchdogMs) return false + + const snapshot = active + try { + if (client?.session?.abort) { + await fireSdk( + client, + "session.abort provider retry watchdog", + client.session.abort.bind(client.session), + { path: { id: sessionID }, body: {} }, + { path: { sessionID }, body: {} }, + { sessionID }, + ) + } + } catch {} + clearActiveRun(sessionID) + clearSessionStatus(sessionID) + const refunded = await persistInfrastructureRefund(directory, sessionID, snapshot, { + reason: "provider_retry_watchdog", + error: `OpenCode session.status stayed retry for ${providerRetryWatchdogMs}ms`, + }) + await appendLoopLog(directory, "provider-retry-recovery", { + sessionID, + job: refunded.job?.name || snapshot.jobId, + retryForMs: now() - current.since, + retryInMs: refunded.delayMs, + }) + await toast(client, "Loop provider retry exceeded the watchdog; the Loop-owned turn was released and will retry with backoff.", "warning") + await scheduleDueWork(directory, client, sessionID, refunded.delayMs) return true } @@ -217,13 +313,22 @@ export function createLoopExecutor(options = {}) { } runLocks.set(sessionID, now()) let job + let previousLastRunAt = 0 + let disabledByMaxRuns = false try { await finalizeActiveRun(directory, client, sessionID, { requireIdle: true, forceStale: true }) - if (!await sessionIsIdle(client, sessionID, directory)) { + const statusType = await sessionStatusType(client, sessionID, directory) + if (statusType !== "idle") { + if (statusType === "retry") { + if (await recoverStuckProviderRetry(directory, client, sessionID)) return + } else { + retryRuns.delete(sessionID) + } if (runOptions.force) await toast(client, "Loop queued: session is busy; it will run on the next idle check.", "info") await reschedule(busyRetryMs) return } + retryRuns.delete(sessionID) const active = activeRuns.get(sessionID) const activeAge = active ? now() - (active.startedAt || 0) : 0 @@ -286,9 +391,11 @@ export function createLoopExecutor(options = {}) { if (runNowRequested) delete job.runNowRequestedAt job.watchTriggered = false + previousLastRunAt = Number(job.lastRunAt || 0) job.lastRunAt = now() job.runCount = (job.runCount || 0) + 1 - if (job.maxRuns > 0 && job.runCount >= job.maxRuns) { + disabledByMaxRuns = job.maxRuns > 0 && job.runCount >= job.maxRuns + if (disabledByMaxRuns) { job.enabled = false await notifyJob(directory, job, "max_runs_reached") } @@ -337,6 +444,8 @@ export function createLoopExecutor(options = {}) { timer, runToken, compactionAction: result.compaction === true, + previousLastRunAt, + disabledByMaxRuns, }) if (result.compaction && compactionRuntime.isCompleted(sessionID, job.id)) { await compactionRuntime.finalize(directory, client, sessionID) @@ -352,6 +461,24 @@ export function createLoopExecutor(options = {}) { await reschedule(busyRetryMs) } catch (error) { clearActiveRun(sessionID) + if (isTransientNetworkError(error) && job) { + const stateAfterFailure = await readState(directory, sessionID) + const persisted = (stateAfterFailure.jobs || []).find((candidate) => candidate.id === job.id) + if (persisted) { + refundInfrastructureRun(persisted, { + runCount: job.runCount, + previousLastRunAt, + disabledByMaxRuns, + }, { reason: "action_dispatch_transient", error, now: now() }) + stateAfterFailure.jobs = (stateAfterFailure.jobs || []).map((candidate) => candidate.id === persisted.id ? persisted : candidate) + await writeState(directory, sessionID, stateAfterFailure) + const delayMs = networkRetryDelayMs(persisted.infrastructureFailureCount || 1, busyRetryMs, networkRetryMaxMs) + await appendLoopLog(directory, "network-action-error", { sessionID, job: persisted.name || persisted.id, error: errorMessage(error), retryInMs: delayMs }) + await toast(client, "Loop action hit a transient network failure; the logical run was refunded and will retry.", "warning") + await reschedule(delayMs) + return + } + } await toast(client, `Loop job failed: ${error instanceof Error ? error.message : String(error)}`, "error") await appendLoopLog(directory, "error", { sessionID, @@ -370,9 +497,11 @@ export function createLoopExecutor(options = {}) { clearActiveRun, disposeSession, recoverActiveDispatchFailure, + recoverStuckProviderRetry, finalizeActiveRun, fireAction, maybeRunDueJobs, + sessionStatusType, sessionIsIdle, updateSessionStatusFromEvent, markSessionStatus, diff --git a/src/source/runtime/network-recovery.js b/src/source/runtime/network-recovery.js new file mode 100644 index 0000000..a1adaa0 --- /dev/null +++ b/src/source/runtime/network-recovery.js @@ -0,0 +1,40 @@ +const TRANSIENT_NETWORK_PATTERNS = [ + /\b(?:408|425|429|500|502|503|504|524)\b/i, + /rate[\s_-]?limit|too many requests|overloaded|service[\s_-]?unavailable|provider[_ -]?unavailable/i, + /terminated|fetch failed|failed to fetch|network[\s_-]?error|network connection lost/i, + /connection (?:error|refused|lost)|socket (?:hang up|connection was closed)|reset before headers/i, + /\b(?:enotfound|eai_again|econnrefused|econnreset|etimedout|ehostunreach|enetunreach|epipe)\b/i, + /\b(?:request|response|connection|network|stream|read) (?:timeout|timed out|time out)\b/i, + /\btimeout(?:error)?\b/i, +] + +function errorText(value) { + if (value instanceof Error) return `${value.name}: ${value.message}` + if (typeof value === "string") return value + try { return JSON.stringify(value) } catch { return String(value) } +} + +export function isTransientNetworkError(value) { + const text = errorText(value) + return TRANSIENT_NETWORK_PATTERNS.some((pattern) => pattern.test(text)) +} + +export function networkRetryDelayMs(attempt, baseMs = 5_000, maxMs = 60_000) { + const safeAttempt = Math.max(1, Math.floor(Number(attempt) || 1)) + const safeBase = Math.max(1, Math.floor(Number(baseMs) || 5_000)) + const safeMax = Math.max(safeBase, Math.floor(Number(maxMs) || 60_000)) + return Math.min(safeMax, safeBase * (2 ** Math.min(8, safeAttempt - 1))) +} + +export function refundInfrastructureRun(job, snapshot = {}, input = {}) { + const chargedCount = Number(snapshot.runCount ?? job.runCount ?? 0) + const currentCount = Number(job.runCount || 0) + if (chargedCount > 0 && currentCount >= chargedCount) job.runCount = Math.max(0, currentCount - 1) + if (Number.isFinite(Number(snapshot.previousLastRunAt))) job.lastRunAt = Number(snapshot.previousLastRunAt) + if (snapshot.disabledByMaxRuns && job.maxRuns > 0 && job.runCount < job.maxRuns) job.enabled = true + job.infrastructureFailureCount = (job.infrastructureFailureCount || 0) + 1 + job.lastInfrastructureFailure = String(input.reason || "transient_network_failure").slice(0, 120) + job.lastInfrastructureError = errorText(input.error).slice(0, 4000) + job.lastInfrastructureFailureAt = Number(input.now || Date.now()) + return job +} diff --git a/src/source/runtime/run-finalization.js b/src/source/runtime/run-finalization.js index 6441dc3..31d48dd 100644 --- a/src/source/runtime/run-finalization.js +++ b/src/source/runtime/run-finalization.js @@ -5,6 +5,7 @@ import { appendLoopLog as defaultAppendLoopLog, runShellCommand as defaultRunShe import { toast as defaultToast } from "../opencode/host.js" import { writeGoalReport as defaultWriteGoalReport } from "./goal-runtime.js" import { dangerousShell as defaultDangerousShell } from "./job-workspace.js" +import { applyTerminalContinuationGuard as defaultApplyTerminalContinuationGuard } from "./terminal-guard.js" function requireFunction(value, label) { if (typeof value !== "function") throw new TypeError(`createRunFinalizationRuntime requires ${label}`) @@ -25,8 +26,22 @@ export function createRunFinalizationRuntime(options = {}) { const toast = typeof options.toast === "function" ? options.toast : defaultToast const writeGoalReport = typeof options.writeGoalReport === "function" ? options.writeGoalReport : defaultWriteGoalReport const dangerousShell = typeof options.dangerousShell === "function" ? options.dangerousShell : defaultDangerousShell + const applyTerminalContinuationGuard = typeof options.applyTerminalContinuationGuard === "function" + ? options.applyTerminalContinuationGuard + : defaultApplyTerminalContinuationGuard async function finalizeJob(directory, client, sessionID, state, job, previousJob) { + // Infrastructure backoff is consecutive, not lifetime. Reaching normal + // finalization proves a real Loop-owned turn completed, so the next future + // outage should restart from the shortest backoff rather than inheriting an + // old 60s cap from unrelated historical network failures. + if (job.infrastructureFailureCount) { + job.infrastructureFailureCount = 0 + job.lastInfrastructureFailure = "" + job.lastInfrastructureError = "" + job.lastInfrastructureFailureAt = 0 + } + if (job.verifyCommand) { const verify = await runShellCommand(job.verifyCommand, directory, job.timeoutMs || 300_000) job.lastVerifyAt = now() @@ -86,6 +101,19 @@ export function createRunFinalizationRuntime(options = {}) { job = await applyGoalNoProgressGuard(directory, client, sessionID, job, previousJob) } + const terminal = await applyTerminalContinuationGuard(directory, client, sessionID, job) + job = terminal.job + if (terminal.pausedNow) { + await appendLoopLog(directory, "terminal-no-work", { + sessionID, + job: job.name || job.id, + count: job.terminalNoWorkCount, + summary: String(terminal.text || "").slice(0, 1000), + }) + await notifyJob(directory, job, "terminal_no_work") + await toast(client, "Loop paused: completion-bounded task reported complete with no work remaining twice.", "success") + } + state.jobs = (state.jobs || []) .map((candidate) => candidate.id === job.id ? job : candidate) .filter((candidate) => candidate.enabled !== false || isGoalJob(candidate)) diff --git a/src/source/runtime/session-status.js b/src/source/runtime/session-status.js index 1c28040..c14d72e 100644 --- a/src/source/runtime/session-status.js +++ b/src/source/runtime/session-status.js @@ -78,7 +78,9 @@ export function createSessionStatusRuntime(options = {}) { const argsList = [] if (directory) argsList.push({ query: { directory } }, { directory }, { workspace: directory }) argsList.push({}) + let attempted = false for (const args of argsList) { + attempted = true try { const result = await statusMethod.call(client.session, args) const error = sdkError(result) @@ -91,8 +93,6 @@ export function createSessionStatusRuntime(options = {}) { if (typeof observedType !== "string") continue markSessionStatus(observedSessionID, observedType, observedAt) } - // OpenCode's status list contains active sessions; idle sessions are - // normally omitted. Clear a completed descendant that was previously busy. for (const childID of sessionParents.keys()) { if (!isDescendantSession(childID, sessionID) || data[childID]) continue markSessionStatus(childID, "idle", observedAt) @@ -104,7 +104,7 @@ export function createSessionStatusRuntime(options = {}) { return { type: "idle", source: "sdk" } } catch {} } - return undefined + return attempted ? { type: "unknown", source: "sdk-error" } : undefined } async function canFinalizeActiveRun(directory, client, sessionID, active, options = {}) { @@ -117,24 +117,29 @@ export function createSessionStatusRuntime(options = {}) { if (completion === "completed") return true if (!options.requireIdle) return completion === "unknown" && staleActiveRun(sessionID) + const cached = sessionStatuses.get(sessionID) + const seenAt = sessionStatusSeenAt.get(sessionID) || 0 + const cachedIdleAfterRun = cached === "idle" && seenAt > (active.startedAt || 0) const live = await readLiveSessionStatus(client, sessionID, directory) if (live?.type === "idle") return true + if (live?.type === "unknown" && cachedIdleAfterRun) { + // A concrete idle observed after this active run is sufficient to finalize + // that run even if a later status read fails. This does not authorize a + // new prompt by itself; admission applies its own current status policy. + return true + } if (live?.type) { - if ((live.type === "busy" || live.type === "retry") && options.forceStale && completion === "unknown" && staleActiveRun(sessionID)) return true + if (live.type === "busy" && options.forceStale && completion === "unknown" && staleActiveRun(sessionID)) return true return false } if (options.forceStale && completion === "unknown" && staleActiveRun(sessionID)) return true - const cached = sessionStatuses.get(sessionID) - const seenAt = sessionStatusSeenAt.get(sessionID) || 0 - return cached === "idle" && seenAt > (active.startedAt || 0) + return cachedIdleAfterRun } async function recoverCompletedTailWithoutActiveRun(directory, client, sessionID, liveType, seenAt) { if (activeRuns.has(sessionID)) return false - if (liveType !== "busy" && liveType !== "retry") return false - // Never override the first fresh busy observation. Waiting at least one cache - // window gives the host time to expose a new user/assistant turn if it is real. + if (liveType !== "busy") return false if (!seenAt || now() - seenAt < sessionStatusCacheMs) return false const completion = await activeRunCompletionFromMessages(directory, client, sessionID, { startedAt: 0 }) if (completion !== "completed") return false @@ -149,9 +154,6 @@ export function createSessionStatusRuntime(options = {}) { } async function sessionStatusType(client, sessionID, directory, options = {}) { - // OpenCode can briefly report an idle session while a long-running tool or - // subtask is still executing. Tool lifecycle hooks are the more specific - // signal here, so never enqueue another turn until every active call ends. if (hasActiveToolCalls(sessionID) || hasBusyDescendant(sessionID)) { markSessionStatus(sessionID, "busy") return "busy" @@ -159,38 +161,35 @@ export function createSessionStatusRuntime(options = {}) { const cached = sessionStatuses.get(sessionID) const seenAt = sessionStatusSeenAt.get(sessionID) || 0 - - // Idle is safe to trust until OpenCode tells us otherwise. Busy/retry is only - // trusted briefly: OpenCode custom commands such as /loop-status create their - // own short assistant turn, and some TUI builds do not always emit the final - // idle event after that turn. If we cache busy forever, due loop work can get - // stuck at "due in every idle" until the user types another command. if (cached === "idle") return cached if (cached && now() - seenAt < sessionStatusCacheMs) return cached const live = await readLiveSessionStatus(client, sessionID, directory) + if (live?.type === "unknown") { + const conservative = cached === "retry" ? "retry" : "busy" + markSessionStatus(sessionID, conservative) + return conservative + } if (live?.type) { - // Some OpenCode 1.15.x/1.18.x TUI builds can leave session.status at busy - // after a plugin command acknowledgement. If no Loop run exists yet, a - // completed assistant tail plus an already-stale busy observation is a - // stronger signal than the unchanged live status. A genuinely running turn - // has an unfinished assistant tail (or latest user message) and is never - // force-recovered by this path. if (await recoverCompletedTailWithoutActiveRun(directory, client, sessionID, live.type, seenAt)) return "idle" - // When a Loop-owned turn exists, use its exact start boundary so an older - // completed assistant message can never finalize a newer active run. if ((live.type === "busy" || live.type === "retry") && options.recoverStaleActive !== false) { const active = activeRuns.get(sessionID) if (active) { const completion = await activeRunCompletionFromMessages(directory, client, sessionID, active) - if (completion === "completed" || (completion === "unknown" && staleActiveRun(sessionID))) { + if (completion === "completed" || (live.type === "busy" && completion === "unknown" && staleActiveRun(sessionID))) { markSessionStatus(sessionID, "idle") - await appendLoopLog(directory, completion === "completed" ? "status-message-complete-recovery" : "status-stale-recovery", { + const logDetails = { sessionID, job: active.job?.name || active.jobId, startedAt: active.startedAt, - }) + ...(completion === "completed" ? {} : { staleStatus: live.type }), + } + await appendLoopLog( + directory, + completion === "completed" ? "status-message-complete-recovery" : "status-stale-recovery", + logDetails, + ) return "idle" } } diff --git a/src/source/runtime/terminal-guard.js b/src/source/runtime/terminal-guard.js new file mode 100644 index 0000000..00c67c4 --- /dev/null +++ b/src/source/runtime/terminal-guard.js @@ -0,0 +1,56 @@ +import { isCompletionBoundedContinuation, isTerminalNoWorkReply } from "../core/continuation.js" +import { orderedSessionMessages, readRecentSessionMessages } from "../opencode/host.js" + +function messageText(message) { + const parts = Array.isArray(message?.parts) ? message.parts : [] + const fromParts = parts + .filter((part) => part?.type === "text" && typeof part.text === "string") + .map((part) => part.text) + .join("\n") + .trim() + if (fromParts) return fromParts + const info = message?.info || message || {} + for (const value of [info.text, info.content, info.summary]) { + if (typeof value === "string" && value.trim()) return value.trim() + } + return "" +} + +export async function applyTerminalContinuationGuard(directory, client, sessionID, job, options = {}) { + // `/loop devam et` is intentionally infinite. The guard is only for an + // explicit completion-bounded request such as "bitene kadar" / "until done". + if (job?.scheduleMode !== "idle" || !isCompletionBoundedContinuation(job?.action)) { + return { job, terminal: false, pausedNow: false } + } + + const messages = await readRecentSessionMessages(client, sessionID, directory, options.messageLimit || 8) + if (!messages) return { job, terminal: false, pausedNow: false } + const tail = orderedSessionMessages(messages).at(-1) + const info = tail?.info || tail || {} + if (info.role !== "assistant") return { job, terminal: false, pausedNow: false } + const completed = Number(info?.time?.completed || 0) + const created = Number(info?.time?.created || 0) + const runStarted = Number(job?.lastRunAt || 0) + if (!Number.isFinite(completed) || completed <= 0) return { job, terminal: false, pausedNow: false } + if (runStarted > 0 && completed < runStarted && (!Number.isFinite(created) || created < runStarted)) { + return { job, terminal: false, pausedNow: false } + } + + const text = messageText(tail) + const terminal = isTerminalNoWorkReply(text) + if (!terminal) { + if (job.terminalNoWorkCount) job.terminalNoWorkCount = 0 + return { job, terminal: false, pausedNow: false, text } + } + + job.terminalNoWorkCount = (job.terminalNoWorkCount || 0) + 1 + job.lastTerminalNoWorkAt = Date.now() + job.lastTerminalNoWorkSummary = text.slice(0, 1000) + const threshold = Math.max(2, Number(options.threshold) || 2) + const pausedNow = job.terminalNoWorkCount >= threshold && !job.paused + if (pausedNow) { + job.paused = true + job.lastFailureReason = "terminal_no_work" + } + return { job, terminal: true, pausedNow, text } +}