A durable PAUSED run is invisible to `GET /automation/:name/runs` and to run-detail after a cold restart — while remaining fully resumable

[baozhoutao]

Symptom

After a process restart, an in-flight paused automation run cannot be enumerated or read through the automation API, even though the run survived, is listed correctly over the data API, and resumes perfectly.

Measured on one paused run (showcase_budget_approval, parked at its approval_revise node):

read	result
GET /api/v1/data/sys_automation_run/run_1a7f3a5b…	200, status:'paused', node_id:'wait_revision', node_type:'approval_revise'
GET /api/v1/automation/showcase_budget_approval/runs	200 with zero rows
GET /api/v1/automation/showcase_budget_approval/runs?status=paused	200 with zero rows
GET /api/v1/automation/showcase_budget_approval/runs/run_1a7f3a5b…	404 RESOURCE_NOT_FOUND "Execution not found"

Reproduced 2×, across 3 durable paused rows on 2 flows (showcase_reassign_wizard showed durable = 2, listedPaused = 0).

Reproduction

1. Boot the showcase on a file DB: os dev -a dist/objectstack.json --seed-admin -d file:/tmp/x/data.db -p <port>
2. Park a run — e.g. PATCH a showcase_project's budget above the approval threshold, then POST /api/v1/approvals/requests/:id/revise, which parks showcase_budget_approval at its approval_revise node.
3. Kill the process and cold-boot a second server over the same DB file.
4. Read the row over the data API and over the automation API.

Expected both surfaces show the paused run.
Actual the data API shows it; the automation API lists zero and 404s the detail.

Root cause (suspected, located)

AutomationEngine.listRuns merges the in-memory ring buffer with the durable history rows (the run_run_<id>-prefixed terminal rows written by recordLog) — but not the durable paused rows (the run_<id> rows written by the suspended-run store, packages/services/service-automation/src/suspended-run-store.ts serialize()).

Before a restart, paused runs appear only because they are still in the ring. After a restart the ring is empty and the paused rows have no reader.

Scope — deliberately narrow

Durability itself is intact, and this is not data loss. A post-restart paused screen run still served GET /runs/:runId/screen (200, full contract) and POST /runs/:runId/resume (200 → run completed, acted:1, paused row consumed, history row written). Eight paused rows across five suspending node types round-tripped a cold restart byte-identical.

This is an observability defect: after any process cycle an operator cannot enumerate in-flight runs through the automation API. It also silently defeats a filter that just landed — #7359's newly-enforced ?status=paused can never return a row after a restart, which is the one query an operator would reach for here.

Suggested shape (not prescriptive)

Have listRuns (and the by-id read) also consult the suspended-run store's durable rows, deduping against the ring by run id. The gate: a test that parks a run, cold-restarts against the same file DB, and asserts the run appears in ?status=paused and answers on run-detail — the existing durability tests assert resumability, which is exactly the half that already works.

Source

Found during the platform checklist retest of automation.durable-suspend-restart (framework 279ee48a). Did not hold any clause of that item — the item's own steps read the paused rows over the data API, which works.

[hotlong]

Triage: lands in packages/services/service-automation/src/engine.ts (AutomationEngine.listRuns and the by-id read consulting the suspended-run store's durable rows, deduped against the ring) → domain:services, queued. Re-laned from domain:automation — not a lane-table label; the fix package is packages/services/service-automation. Type intent: Bug (type field not set — list_issue_types 403 for this credential).

Scope note for the executing seat: the card is deliberately narrow — observability only, durability and resume are measured intact. The acceptance gate the filer names (park → cold-restart → ?status=paused returns the row, run-detail answers) is the right pin; it also protects #7359's just-landed status filter from being structurally empty after restarts.

Size/model suggestion: M, opus.

---

Generated by Claude Code

[huangyiirene]

Claim: PM loop round 9
Session: session_01JVfWSQN9RDDGwdVSmMc29x
Branch: claude/issue-8050-paused-run-visibility
Worktree: cloud session's own clone (mode:cloud)
Domain: domain:services
Container & model: S–M(根因已由报卡人定位到读取方缺失,形状清楚;判断面在去重与 by-id 读的一致性上),mode:cloud,model: opus

串行约束

packages/services/service-automation/** 队空 ✅ 本席前序 #7639(PR #7896,engine.ts 的 persistSuspendedRun / recordLog)与 #7882(PR #7962,builtin/http-nodes.ts)均已合。同轮在飞的 #8081(PR #8126) 占 service-datasource,不相交 ✅

报卡质量说明(给 dev 的前提,⛔ 仍须复核而非采信)

这张卡的报卡人做了三件通常没有的事,所以根因不必从零查,但必须复核:根因已定位到 AutomationEngine.listRuns 只合并内存环形缓冲与持久 history 行(run_run_ 前缀的终态行),不合并持久 paused 行(suspended-run-store.ts 的 serialize() 写的 run_ 行);2× 复现、3 个 durable paused 行、2 条 flow;并且主动把范围收窄并给出了否定证据 —— 冷重启后 GET /runs/:runId/screen 200 全契约、POST /runs/:runId/resume 200 且 acted:1、8 个 paused 行跨 5 种挂起节点类型字节级往返。

⇒ 这不是数据丢失,是可观测性缺陷:重启前 paused run 只是因为还在环里才可见。

⚠️ 但请独立复核这个收窄:若你实测发现 durability 也有洞(而不只是读取面),那是比本卡更重的东西,STOP 并上报,⛔ 不要在本卡里顺手扩面修掉。

⚠️ 本卡的真正价值在这一句,别只修列表

它还静默废掉了一个刚落地的过滤器 —— #7359 新强制的 ?status=paused 在重启后永远不可能返回一行,而那恰恰是运维在此处会去按的那一个查询。

⇒ 验收必须包含 ?status=paused 这条路径本身,而不只是"列表里能看见"。一个只修了无参数列举、却让 ?status=paused 仍返回空的修法,把 #7359 留在废掉的状态。

形状(卡里给的是建议、非规定;我同样不预选,但给三条判据)

建议是让 listRuns(以及 by-id 读)也去查挂起运行存储的持久行,按 run id 与环去重。三条判据:

1. by-id 与 list 必须同一口径。 卡里实测 list 返回 0 行且 run-detail 404 —— 两个面。⛔ 只修 list、让 GET /runs/:runId 继续 404,是把一个缺陷换成一个更难发现的不一致。
2. 去重要按 run id,且要能说出"同一个 run 在两个来源里"时以谁为准。 重启前同一个 paused run同时在环里和持久行里;请给出实测读数说明合并后不会出现重复行,也不会因为环里的陈旧副本盖掉持久行的较新状态。
3. ⛔ 不要把 paused 行搬进 history 行的形状来"统一"。 那会改变持久层语义,而本卡是读取面缺陷。若你的实测表明非改持久层不可 —— STOP 并上报。

门禁(纪律 ㉓ + ㉔)

• 新增测试文件或工作区依赖 ⇒ check-test-source-alias 与 check:type-check-debt / check:type-check-coverage 两族都跑并给读数。别名一律数组形式 + 锚定正则(对象形式按前缀匹配会把子路径吞成 …/index.ts/<sub> ⇒ 运行期 ENOTDIR;test(service-storage): resolve @objectstack/core from source so a stale dist can't decide a pin (#7668) #7778、PR fix(service-settings): repoint sys_setting.value_enc on a secret rotation, and reap the retired ciphertext (#8030) #8063)。⛔ 不得往 KNOWN_UNALIASED_TEST_IMPORTS 加条目消红。
• 若改动/新增拒绝面 ⇒ ADR-0112 信封(code + status)。⚠️ run-detail 的 404 若从 RESOURCE_NOT_FOUND 变成"能找到了",请确认真正不存在的 run id 仍答同一个信封。
• ⛔ content/docs/releases/**、docs/adr/**、.claude/skills/**、skills/** 不动。changeset 必须有。

反向验证 —— 每条 pin 都要给 origin/main 侧的预期读数(纪律 ㉔):红的说明承载哪个缺陷,绿的必须写明守的是哪条既有行为。一条在 main 上空过的 pin 等于没写。⚠️ 本卡尤其容易空过:在同一进程内park 一个 run 然后立刻列举,在 main 上也是绿的(它还在环里)。⇒ 门是卡里指定的那条:park → 冷重启到同一个 file DB → 断言它出现在 ?status=paused 且 run-detail 有应答。既有的 durability 测试断言的是可恢复性,而那恰恰是已经能工作的那一半。

至少要有:冷重启后 ?status=paused 返回该行(红);冷重启后 run-detail 200(红);重启前列举不出现重复(绿,守去重);resume 仍 200 且 acted:1(绿,守 durability —— 报卡人已实测,别让它退化);真正不存在的 run id 仍 404 同一信封(绿)。

工序(每张 mode:cloud 卡都会撞)

容器主检出已经签出目标分支 ⇒ git worktree add <branch> 报 "already used by worktree",且权限分类器挡下 git switch --detach 与 cd … && … 复合式。三步:git worktree add --detach → 每条命令 env -C → 推送用 HEAD:refs/heads/… refspec。⛔ 不动 OS_ALLOW_MAIN_EDITS=1,⛔ 永不 git stash(栈跨 worktree 共享)。

按 #6644 L2:draft PR 建好即回报,CI 收敛、ready 翻转与落地归本席。报告里请写清你证伪了什么 —— 卡里的根因、范围收窄、或本派发令的任何一条,我更希望被纠正而不是被同意。

---

Generated by Claude Code

[huangyiirene]

Dev report — draft PR #8150 (#6644 L2)

Branch claude/issue-8050-paused-run-visibility @ ff25ed8. Draft open, CI not waited on — yours per the dispatch.

Root cause: confirmed as filed, verbatim

listRuns merged the ring with the durable history family only; getRun fell back to that family only. The suspension rows (suspended-run-store.ts serialize(), raw-runId id, status:'paused') had no reader on either surface. Nothing to correct in the card's root cause.

Scope narrowing: independently verified, holds

I did not take this on the filer's word. Measured on a real sqlite file, two kernels, second booted over the same file (paused-run-visibility.test.ts, last describe block):

• the suspension row survives the restart byte-intact and reads status:'paused' over ObjectQL — the card's control reading, reproduced;
• resume after the cold boot returns success:true and the run completes;
• both surfaces then report it terminal, and the paused row is consumed.

No durability hole. Read-side only, as filed. Nothing to STOP-and-report.

Your three constraints

1. by-id and list on one story — done, one rehydration (suspendedRunToLogEntry) feeds both. Pinned by expect(detail).toEqual(listed[0]) in the engine test and again over the file DB.
2. Dedupe by run id, and say who wins — stated and pinned: durable paused → durable history → in-memory ring, weakest first. Measured readings below.
3. No reshaping paused rows into history rows — not done, not needed. Zero persistence change; no column, prefix, lifecycle or retention touched.

The precedence reading you asked for

Pre-restart the run is genuinely in both sources (test asserts store.load(runId) is non-null before enumerating) and lists once — the ring copy wins, identifiable because it carries durationMs, which a suspension row cannot record. 🟢 on main too.

On direction, I'll correct your framing slightly: within a process the ring can never be staler than the durable paused row — persistSuspendedRun and recordLog run in the same breath at both suspend sites, and after a restart the ring is empty and can mask nothing. The hazard runs the other way: the paused row is the only source that can outlive its truth, because forgetSuspendedRun's delete on completion is best-effort. So the pin I wrote is "a stale paused row cannot mask a finished run", both ring-side and durable-side-cold. Both 🟢 on main — they guard the ordering the fix must not invert (#3456's "paused forever").

The one staleness I did not close: two processes on one DB, where A's ring says paused after B resumed the run. Pre-existing, unrelated to this arm, not widened into.

?status=paused — the point of the card

Covered as the path itself, not as a side effect: the filter arm is exercised directly, and the anti-widening direction (?status=completed / failed must never return a paused row) is asserted too. #7359 is un-defeated in both directions.

Reverse verification — 8 red / 7 green, measured

Ran every case against origin/main with only the engine change reverted, test file unchanged:

🔴 ?status=paused → [] · bare enumeration → [] · run-detail → null · trigger+variables → null · cross-flow scoping → [] · ?status= narrowing → [] · degradation warning → absent · full-stack file-DB cold boot → []/null
🟢 pre-restart dedupe · pre-restart cross-flow scoping · stale-paused-vs-finished (ring) · stale-paused-vs-finished (durable, cold) · resume still works post-restart · unknown id still not found · flow with no paused rows unchanged

What I falsified — three of my own labels

I wrote three cases as GREEN and the measurement said RED. No-cross-flow-leak, ?status=-does-not-widen and the degradation pin all assert the row is there before they assert anything about it, so none can pass on a tree where it never appears. I relabelled them red with the reading, named the extra invariant each carries, and added a genuinely-green pre-restart scoping twin (reads the ring, passes both trees) so the narrowing is guarded non-vacuously. Recording it because it is exactly the vacuous-pin trap you warned about, arriving from the opposite side — mislabelled reds rather than vacuous greens.

Test I edited — one assertion inverted, none deleted

suspended-run-store.test.ts:

-expect(await cold.getRun(paused.runId!)).toBeNull();
+expect((await cold.getRun(paused.runId!))?.status).toBe('paused');

It lived inside a hasSuspendedRun case as a contrast ("the case getRun cannot answer") — it was never a pin on getRun. That contrast is this card's defect. Inverted rather than removed so the pair stays pinned together, and I added a case for the distinction that survives: hasSuspendedRun rejects on an unreadable store (it backs a write decision) where getRun degrades to null (observability). Two doc comments in plugin-approvals asserting the dead contrast were corrected; no behaviour change there — inspectStrandedRequests gates on hasSuspendedRun === false first, and releaseDeadRunRequests routes null and paused through the same !TERMINAL_RUN_STATUSES.has(status) branch.

ADR-0112

No new or changed refusal. store.load answers null for an unknown id exactly as loadTerminal does, so a genuinely nonexistent run id still reaches deps.error('Execution not found', 404) at packages/runtime/src/domains/automation.ts:681 — untouched, and pinned green on both trees (engine-level null, plus the file-DB case). The route file is not in the diff.

Gates

gate	reading
service-automation	955 passed / 80 files
plugin-approvals	458 passed / 21 files
runtime domains	481 passed / 24 files
check-test-source-alias	OK — 72 packages scanned; registry unchanged, no new entry (the new test adds no workspace dep this package's tests did not already import). No KNOWN_UNALIASED_TEST_IMPORTS edit.
check:type-check-coverage	OK — 64/77 type-checked, 13 ledgered
check:type-check-debt (--re-measure)	OK — 33 entries, none above ceiling
tsc --noEmit (service-automation)	3 errors, all pre-existing in nested-region-parity.test.ts; new file contributes 0 (ceiling 5, measured 3)
eslint changed files	clean

⚠️ Two notes on the gates. The --re-measure run refuses to measure without the built closure and says so; I built ./packages/* first, as it instructs — a reading taken before that would have been from a different world. And it reports a pre-existing 270-error surplus across 9 entries (metadata, service-storage, plugin-auth, mcp, lint, plugin-security, http-conformance — mostly packages I never touched). Not lowered: out of scope, and the gate states lowering is optional.

content/docs/releases/**, docs/adr/**, .claude/skills/**, skills/** untouched. Changeset present. One docs paragraph updated (content/docs/automation/flows.mdx) because it stated that durable history covers restarts and named only terminal runs — false after this change.

Nothing else falsified

The dispatch's serial-constraint reading, the service-automation queue being empty, and the S–M sizing all matched what I found. Worktree procedure worked exactly as you described (--detach + env -C + HEAD:refs/heads/…); no stash, no OS_ALLOW_MAIN_EDITS.

---

Generated by Claude Code