fix(service-automation): route the last three suspended-run readers through the shared loader - #14650

Merged
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative
Sep 2, 2026
Merged

fix(service-automation): route the last three suspended-run readers through the shared loader#14650
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative

Conversation

@claude

@claudeclaudeBot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Fixes#14332

cancelRun, failAncestors and listSuspendedRunsDurable were the three readers of suspended-run state that #13617 deliberately left on the per-process suspendedRuns map. All three now take one answer to "where is this run parked", through the existing loadSuspendedRun / loadSuspendedRunStrict pair — with the degrading or strict loader chosen per site, so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived.

Ruling of record

No maintainer ruling is owed on this card — triage graded it dispatchable and fixed its shape. Quoted verbatim from triage (14332#issuecomment-5503565404):

Lands in:packages/services/service-automation/src/engine.ts — the three readers (cancelRun, failAncestors, listSuspendedRunsDurable). Ruled shape is the card's own: one answer to "where is this run parked" — route all three through loadSuspendedRun / loadSuspendedRunStrict, choosing the degrading or strict loader deliberately per site so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived; the getSuspendedScreen docblock already states the principle. p2 because site 2 (failAncestors) carries #13617's exact harm shape — failing an ancestor at a node it has already left — and site 1 tears down the wrong node executor's pause. ⛔ Site 3's "in-memory entries win" comment must be corrected in the same PR, not left as a true-sounding sentence that is backwards once several replicas share one store; its own docblock records it has no in-repo production consumer, so it is the cheapest of the three to make honest.

The per-site loader judgment

This is the deliverable, not the mechanical replacement. Each site's docblock now carries the same judgment in prose.

SiteLoader chosenNot found meansStore unreadable meansWhy that is the posture it already had
1 cancelRunstrict (loadSuspendedRunStrict)returns false — "already terminal / unknown", its documented idempotent successthrows out of the loader into this site's own catch, which keeps its #4632 DURABILITY record at errorThe degrading loader would have answered null under its own best-effort warn and silently downgraded the #4632 verdict. Its degradation decides a WRITE, so the strict form plus a local catch is the only shape that preserves both the return value and the level. Same shape resumeInternal uses.
2 failAncestorsdegrading (loadSuspendedRun)ends the walk (if (!parent) return;), unchangedreads as "no ancestor here" and ends the walk — never propagatesThis walk runs inside the catch arm already handling a run's failure, so it must not throw. "Degrade to null and stop" is exactly what the bare .catch(() => null) did. One thing gained: that silent swallow is now recorded, at the loader's declared best-effort warn.
3 listSuspendedRunsDurableneither — merge direction plus the corrected commentn/a (a listing, not a per-id read)unchanged: degrades to the in-memory cache alone at warn (#4632 FUNCTIONAL)A LIST is not a per-id answer. store.list() is capped and best-effort (ObjectStoreSuspendedRunStore reads at most 1000 paused rows) and the same merge is reached on the degraded path, so "absent from the list" is not the "the store answered and has no row" that loadSuspendedRunStrict rests on. The durable row wins a collision; map entries the listing does not carry are still included, deliberately.

Site 3's comment In-memory entries win — they are the freshest copy is corrected in place, together with the docblock sentence that repeated it (The in-memory cache takes precedence on id collisions).

Premise checks — verified on origin/main at 13bf05d, before the first edit

All hold; premise_still_valid: true.

  • P1cancelRun read this.suspendedRuns.get(runId) ?? null and only then store.load — map-first. HOLDS.
  • P2failAncestors walked $parentRunId with this.suspendedRuns.get(parentId) ?? (this.store ? await this.store.load(parentId).catch(() => null) : null). HOLDS.
  • P3listSuspendedRunsDurable merged the durable list under the map with the comment In-memory entries win — they are the freshest copy. HOLDS.
  • P4 The automation/approvals: 多副本集群下审批流每级节点(除首级)被重复创建 —— approve 后恢复读到滞后一拍的流运行态,同级要批两次(单副本零重复) #13617 pair exists and is unreshaped: loadSuspendedRunStrict is store-authoritative with the cacheOnlySuspensions fallback for a failed durable save; loadSuspendedRun is its degrading form; the getSuspendedScreen docblock still states one loader, two callers. HOLDS.
  • P5 Reproduced before the fix and kept as pins — see Ablation below, which measures all three reds per site on the committed tree. All three sites were defective; no partially false premise.
  • P6packages/spec/** and content/docs/releases/** untouched. HOLDS — the diff is 3 files: engine.ts, one new test file, one changeset.

PM hypotheses

  • H1 — holds. All three sites route through the existing pair with no third loader, and the degrading-or-strict choice is a judgment from each site's own docblock. Stated per site in the table above and in the code.
  • H2 — holds, with two named consequences rather than silent changes. No log site moved, no level changed, no thrown or returned shape changed, no row write changed. The finding(service-automation): engine.ts 还剩三处同形的 warn message 拼接 —— forgetSuspendedRun / cancelRun / listSuspendedRunsDurable,是 #5912+#6230 之后该文件的最后一批 #6299 byte-level pins for all three seams (suspended-run-store-consume-log-cause.test.ts, 20 tests) stay green untouched. Two consequences are stated in the docblocks, the changeset and the pins, not left to be discovered: (a) cancelRun — while a store is configured this process's map is no longer an answer, so a store outage reaches the existing error record even for a run this replica holds, where the old cache-first read cancelled from the local snapshot; that snapshot is the defect. (b) failAncestors — a store failure during the ancestor walk was swallowed in total silence by .catch(() => null) and is now recorded by the loader at warn. No new error-level site anywhere (the [Decision] plugin-sharing's refused-backfill report lands at warn where AGENTS.md puts it at error — and the card that was supposed to carry the level is CLOSED #13398-class ruling).
  • H3 — holds, verified independently with a positive control.git grep -n 'listSuspendedRunsDurable' -- 'packages/**' 'examples/**' 'apps/**' ':!*.test.ts' ':!*.spec.ts' returns only engine.ts's own definition and comments plus CHANGELOG prose — no consumer. Positive controls on the same command shape: hasSuspendedRun returns a real non-test consumer (packages/plugins/plugin-approvals/src/approval-service.ts:2704), and listSuspendedRuns returns its spec-contract declaration (packages/spec/src/contracts/automation-service.ts:589) — so the scan does find consumers where they exist.
  • H4 — holds. Clause-② is no. Declared from the actual diff: git diff -U0 origin/main...HEAD | grep export matches exactly one line, and it is prose inside the changeset (No signature, export or return-shape change...), not an export statement. No new exported symbol, no signature change on the three readers, no accept set moved. Changeset is patch. No needs:contract-review.
  • H5 — holds.service-automation: two concurrent resumes of one run on two replicas can both advance it — the idempotency guard is per-process #14333's concurrent-resume guard region is untouched: the diff in engine.ts is confined to cancelRun, failAncestors, listSuspendedRunsDurable and their docblocks.

Tests

New pins: packages/services/service-automation/src/multi-replica-suspended-run-readers.test.ts (11 tests), a sibling of the #13617 harness rather than more cases inside it. Reason recorded in its docblock: multi-replica-resume-staleness.test.ts carries a REVERT-PROOF ledger for one named mutation and its measured 4 red / 4 green split; adding these cases would falsify those counts and merge two different mutations' revert-proofs into one statement. The two files share only the two-engines-over-one-store shape.

Every site-1 and site-2 pin asserts on the onSuspensionReleasednotification (which node executor was told its pause is over), never on the row: the row delete is by id and is correct either way, which is the whole point of the card.

All runs below were taken at commit da4db89f5 and carry to the final head 89f5f8d because the two commits have a byte-identical tree: both are 514564fde37f75fd96c8f58e09ae139bc5ce0801, and git diff --stat da4db89f5 89f5f8d15 prints nothing. The extra commit exists only because this branch had already been pushed when a type-only import was corrected, and force-pushing is forbidden — so the amend was re-landed as a child commit rather than rewritten. Everything ran under scripts/pm/os-verify-lock.sh, exit captured before any pipe.

Full package suite — pnpm --filter @objectstack/service-automation test:

 Test Files 100 passed (100)
Tests 1182 passed (1182)
Duration 92.44s
os-verify-lock: VERDICT command-exit 0 · held the lock 92s

Type-check — the package has no typecheck script (it is a shrink-only DEBT ledger entry, frozen at 3), so a pnpm --filter ... typecheck would have matched zero scripts and exited 0 having measured nothing. Measured instead with the dependency closure built (pnpm --filter '@objectstack/service-automation^...' build, the world the ledger is measured in), tsc --noEmit -p tsconfig.json --listFiles:

src/nested-region-parity.test.ts(95,25): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(151,20): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(180,20): error TS2341: Property 'flows' is private ...
--- error count --- 3

Exactly the ledger's frozen 3, all pre-existing, none in the changed files; --listFiles confirms the new test file is in the program (this run is the reason a fourth error — a type imported from the wrong module — was caught and fixed before the head was final).

Repo-wide lint — no narrowing was needed, the whole population ran:

node --stack-size=4000 node_modules/eslint/bin/eslint.js . --no-inline-config
LINT EXIT=0 · os-verify-lock: VERDICT command-exit 0 · held the lock 116s

Gates

Derived on the final head from the merge-base change set, not a hand-written list: node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands at da4db89f5 (--repo asserted and held against this checkout's origin) — 36 commands over the 3 changed paths. The derivation is a property of the tree, which is byte-identical at the final head 89f5f8d15 (see Tests).

33 green, 0 red, 3 NOT MEASURED. The three are prerequisite refusals (exit code 3, which each script's own verdict text distinguishes from a finding's 1), not failures:

  • node scripts/check-test-completeness.mjsPREREQUISITE NOT MET — this gate grades a saved turbo run test log, and no log was named. Its own text says to record NOT MEASURED when the family is run locally.
  • pnpm check:dual-build-cjs-loadsPREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. (49 packages unbuilt; needs a full pnpm build.)
  • pnpm check:type-check-debtPREREQUISITE NOT MET ... --re-measure cannot run: 35 workspace dependenc(ies) ... have no built type entry point on disk. Its sibling check:type-check-coverage ran green (OK — 69/79 workspace packages type-checked ... 10 in the DEBT ledger), and the targeted tsc reading above answers this gate's substantive question for the one package this PR touches, in the built-closure world the ledger uses.

Also green outside the derived family: pnpm check:nul-bytes (OK (scanned 7992 text file(s) ... no raw ASCII control bytes)), plus a direct control-byte scan of both changed files.

Merge preflight before opening: git merge-tree --write-tree --name-only origin/main HEAD lists no conflicting paths — content/docs/permissions/system-context.mdx is not among them, so no regeneration is owed.

Ablation — three, one per site, on the committed tree

Each mutation restores that site's map-first read, is line-neutral, is proven on disk by anchored occurrence counts before the run, and is restored with git checkout HEAD -- ABSOLUTE_PATH inside a trap ... EXIT INT TERM, the restore proven by git hash-object against the HEAD blob and an empty git diff HEAD. Because the pins import ./engine.js relatively (vitest resolves that to src/engine.ts, not the package's dist/ through exports), no rebuild is interposed; the reds themselves are the proof the mutation reached the code under test.

HEAD blob for engine.ts = f2c33d8b57cea0dce4054de3b8f1ee0ac9b88e6a; every restore leg read back the same hash and an empty git diff HEAD.

MutationResultWhat went red
site 1 — run = this.suspendedRuns.get(runId) ?? await this.loadSuspendedRunStrict(runId)2 red / 9 greenTHE BUG: a stale replica must not tell the wrong node executor its pause is over (teardown names lv1 while the run is parked at lv2) and NEW REACH (a map hit means the unreadable store is never read, so the cancel proceeds from the snapshot)
site 2 — const parent = this.suspendedRuns.get(parentId) ?? await this.loadSuspendedRun(parentId)2 red / 9 greenTHE BUG: a stale parent must not be failed at a node it has already left and the unreadable-ancestor control, red for site 1's reason
site 3 — drop the byId.has(r.runId) guard so the map overwrites again1 red / 10 greenTHE BUG: the durable row wins an id collision, the stale in-process copy does not (the listing reports lv1, one level stale)

The controls that stay green under all three are the postures each site had before: the no-store cases where the map IS the authority, the failed-durable-save cases the store's silence says nothing about, and the unlistable store's documented short list.

One reading was voided rather than reported: the first attempt at the site-1 mutation used a one-line anchor that occurs twice in engine.ts (the other in resumeInternal), the guard refused at anchor count 2, and the run was re-anchored on a unique three-line block. A second attempt was voided by the on-disk proof itself before any test ran. Both voided attempts are named here because a silently re-run ablation is the defect one layer up.

Scope

Diff is 3 files. Untouched as required: packages/spec/**, content/docs/releases/**, skills/**, the store implementation, PR #14567's subflow delegation block, and the region #14333 is queued for. No out-of-scope findings were filed — nothing outside the card's own surface was found.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8

Generated by Claude Code


Generated by Claude Code

os-salesand others added 2 commits September 2, 2026 16:37
…ers through the shared loader
`cancelRun`, `failAncestors` and `listSuspendedRunsDurable` still preferred the
per-process `suspendedRuns` map over the shared `SuspendedRunStore`, which
#13617 had already made authoritative for the resume path. Each now takes one
answer to "where is this run parked", with the degrading or strict loader
chosen per site so the recorded #4632 / #6299 degradation posture is preserved
by choice rather than re-derived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
… module
`AutomationContext` is declared in `@objectstack/spec/contracts` and re-imported
by `engine.ts` as a type — it is not re-exported from there, so importing it
from `./engine.js` added a fourth error (TS2459) to a package whose type-check
debt ledger is frozen at 3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/service-automation, touching 4 documentable anchor(s).

2 release-owned page(s) name something this change touched. These are read-only:

  • content/docs/releases/v16.mdx(via AutomationEngine (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via AutomationEngine (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 5 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from da191b53496857088142d233a3b34da26653db98 — the merge of head 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin da191b53496857088142d233a3b34da26653db98 && git checkout da191b53496857088142d233a3b34da26653db98
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff 89f5f8d15dacac7b65cd1198ff48339d5bd36a18
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-automation: three more readers of suspended-run state still prefer the per-process map over the shared store

1 participant

@os-sales
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

fix(service-automation): route the last three suspended-run readers through the shared loader - #14650

Merged
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative
Sep 2, 2026
Merged

fix(service-automation): route the last three suspended-run readers through the shared loader#14650
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative

Conversation

@claude

@claudeclaudeBot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Fixes#14332

cancelRun, failAncestors and listSuspendedRunsDurable were the three readers of suspended-run state that #13617 deliberately left on the per-process suspendedRuns map. All three now take one answer to "where is this run parked", through the existing loadSuspendedRun / loadSuspendedRunStrict pair — with the degrading or strict loader chosen per site, so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived.

Ruling of record

No maintainer ruling is owed on this card — triage graded it dispatchable and fixed its shape. Quoted verbatim from triage (14332#issuecomment-5503565404):

Lands in:packages/services/service-automation/src/engine.ts — the three readers (cancelRun, failAncestors, listSuspendedRunsDurable). Ruled shape is the card's own: one answer to "where is this run parked" — route all three through loadSuspendedRun / loadSuspendedRunStrict, choosing the degrading or strict loader deliberately per site so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived; the getSuspendedScreen docblock already states the principle. p2 because site 2 (failAncestors) carries #13617's exact harm shape — failing an ancestor at a node it has already left — and site 1 tears down the wrong node executor's pause. ⛔ Site 3's "in-memory entries win" comment must be corrected in the same PR, not left as a true-sounding sentence that is backwards once several replicas share one store; its own docblock records it has no in-repo production consumer, so it is the cheapest of the three to make honest.

The per-site loader judgment

This is the deliverable, not the mechanical replacement. Each site's docblock now carries the same judgment in prose.

SiteLoader chosenNot found meansStore unreadable meansWhy that is the posture it already had
1 cancelRunstrict (loadSuspendedRunStrict)returns false — "already terminal / unknown", its documented idempotent successthrows out of the loader into this site's own catch, which keeps its #4632 DURABILITY record at errorThe degrading loader would have answered null under its own best-effort warn and silently downgraded the #4632 verdict. Its degradation decides a WRITE, so the strict form plus a local catch is the only shape that preserves both the return value and the level. Same shape resumeInternal uses.
2 failAncestorsdegrading (loadSuspendedRun)ends the walk (if (!parent) return;), unchangedreads as "no ancestor here" and ends the walk — never propagatesThis walk runs inside the catch arm already handling a run's failure, so it must not throw. "Degrade to null and stop" is exactly what the bare .catch(() => null) did. One thing gained: that silent swallow is now recorded, at the loader's declared best-effort warn.
3 listSuspendedRunsDurableneither — merge direction plus the corrected commentn/a (a listing, not a per-id read)unchanged: degrades to the in-memory cache alone at warn (#4632 FUNCTIONAL)A LIST is not a per-id answer. store.list() is capped and best-effort (ObjectStoreSuspendedRunStore reads at most 1000 paused rows) and the same merge is reached on the degraded path, so "absent from the list" is not the "the store answered and has no row" that loadSuspendedRunStrict rests on. The durable row wins a collision; map entries the listing does not carry are still included, deliberately.

Site 3's comment In-memory entries win — they are the freshest copy is corrected in place, together with the docblock sentence that repeated it (The in-memory cache takes precedence on id collisions).

Premise checks — verified on origin/main at 13bf05d, before the first edit

All hold; premise_still_valid: true.

  • P1cancelRun read this.suspendedRuns.get(runId) ?? null and only then store.load — map-first. HOLDS.
  • P2failAncestors walked $parentRunId with this.suspendedRuns.get(parentId) ?? (this.store ? await this.store.load(parentId).catch(() => null) : null). HOLDS.
  • P3listSuspendedRunsDurable merged the durable list under the map with the comment In-memory entries win — they are the freshest copy. HOLDS.
  • P4 The automation/approvals: 多副本集群下审批流每级节点(除首级)被重复创建 —— approve 后恢复读到滞后一拍的流运行态,同级要批两次(单副本零重复) #13617 pair exists and is unreshaped: loadSuspendedRunStrict is store-authoritative with the cacheOnlySuspensions fallback for a failed durable save; loadSuspendedRun is its degrading form; the getSuspendedScreen docblock still states one loader, two callers. HOLDS.
  • P5 Reproduced before the fix and kept as pins — see Ablation below, which measures all three reds per site on the committed tree. All three sites were defective; no partially false premise.
  • P6packages/spec/** and content/docs/releases/** untouched. HOLDS — the diff is 3 files: engine.ts, one new test file, one changeset.

PM hypotheses

  • H1 — holds. All three sites route through the existing pair with no third loader, and the degrading-or-strict choice is a judgment from each site's own docblock. Stated per site in the table above and in the code.
  • H2 — holds, with two named consequences rather than silent changes. No log site moved, no level changed, no thrown or returned shape changed, no row write changed. The finding(service-automation): engine.ts 还剩三处同形的 warn message 拼接 —— forgetSuspendedRun / cancelRun / listSuspendedRunsDurable,是 #5912+#6230 之后该文件的最后一批 #6299 byte-level pins for all three seams (suspended-run-store-consume-log-cause.test.ts, 20 tests) stay green untouched. Two consequences are stated in the docblocks, the changeset and the pins, not left to be discovered: (a) cancelRun — while a store is configured this process's map is no longer an answer, so a store outage reaches the existing error record even for a run this replica holds, where the old cache-first read cancelled from the local snapshot; that snapshot is the defect. (b) failAncestors — a store failure during the ancestor walk was swallowed in total silence by .catch(() => null) and is now recorded by the loader at warn. No new error-level site anywhere (the [Decision] plugin-sharing's refused-backfill report lands at warn where AGENTS.md puts it at error — and the card that was supposed to carry the level is CLOSED #13398-class ruling).
  • H3 — holds, verified independently with a positive control.git grep -n 'listSuspendedRunsDurable' -- 'packages/**' 'examples/**' 'apps/**' ':!*.test.ts' ':!*.spec.ts' returns only engine.ts's own definition and comments plus CHANGELOG prose — no consumer. Positive controls on the same command shape: hasSuspendedRun returns a real non-test consumer (packages/plugins/plugin-approvals/src/approval-service.ts:2704), and listSuspendedRuns returns its spec-contract declaration (packages/spec/src/contracts/automation-service.ts:589) — so the scan does find consumers where they exist.
  • H4 — holds. Clause-② is no. Declared from the actual diff: git diff -U0 origin/main...HEAD | grep export matches exactly one line, and it is prose inside the changeset (No signature, export or return-shape change...), not an export statement. No new exported symbol, no signature change on the three readers, no accept set moved. Changeset is patch. No needs:contract-review.
  • H5 — holds.service-automation: two concurrent resumes of one run on two replicas can both advance it — the idempotency guard is per-process #14333's concurrent-resume guard region is untouched: the diff in engine.ts is confined to cancelRun, failAncestors, listSuspendedRunsDurable and their docblocks.

Tests

New pins: packages/services/service-automation/src/multi-replica-suspended-run-readers.test.ts (11 tests), a sibling of the #13617 harness rather than more cases inside it. Reason recorded in its docblock: multi-replica-resume-staleness.test.ts carries a REVERT-PROOF ledger for one named mutation and its measured 4 red / 4 green split; adding these cases would falsify those counts and merge two different mutations' revert-proofs into one statement. The two files share only the two-engines-over-one-store shape.

Every site-1 and site-2 pin asserts on the onSuspensionReleasednotification (which node executor was told its pause is over), never on the row: the row delete is by id and is correct either way, which is the whole point of the card.

All runs below were taken at commit da4db89f5 and carry to the final head 89f5f8d because the two commits have a byte-identical tree: both are 514564fde37f75fd96c8f58e09ae139bc5ce0801, and git diff --stat da4db89f5 89f5f8d15 prints nothing. The extra commit exists only because this branch had already been pushed when a type-only import was corrected, and force-pushing is forbidden — so the amend was re-landed as a child commit rather than rewritten. Everything ran under scripts/pm/os-verify-lock.sh, exit captured before any pipe.

Full package suite — pnpm --filter @objectstack/service-automation test:

 Test Files 100 passed (100)
Tests 1182 passed (1182)
Duration 92.44s
os-verify-lock: VERDICT command-exit 0 · held the lock 92s

Type-check — the package has no typecheck script (it is a shrink-only DEBT ledger entry, frozen at 3), so a pnpm --filter ... typecheck would have matched zero scripts and exited 0 having measured nothing. Measured instead with the dependency closure built (pnpm --filter '@objectstack/service-automation^...' build, the world the ledger is measured in), tsc --noEmit -p tsconfig.json --listFiles:

src/nested-region-parity.test.ts(95,25): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(151,20): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(180,20): error TS2341: Property 'flows' is private ...
--- error count --- 3

Exactly the ledger's frozen 3, all pre-existing, none in the changed files; --listFiles confirms the new test file is in the program (this run is the reason a fourth error — a type imported from the wrong module — was caught and fixed before the head was final).

Repo-wide lint — no narrowing was needed, the whole population ran:

node --stack-size=4000 node_modules/eslint/bin/eslint.js . --no-inline-config
LINT EXIT=0 · os-verify-lock: VERDICT command-exit 0 · held the lock 116s

Gates

Derived on the final head from the merge-base change set, not a hand-written list: node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands at da4db89f5 (--repo asserted and held against this checkout's origin) — 36 commands over the 3 changed paths. The derivation is a property of the tree, which is byte-identical at the final head 89f5f8d15 (see Tests).

33 green, 0 red, 3 NOT MEASURED. The three are prerequisite refusals (exit code 3, which each script's own verdict text distinguishes from a finding's 1), not failures:

  • node scripts/check-test-completeness.mjsPREREQUISITE NOT MET — this gate grades a saved turbo run test log, and no log was named. Its own text says to record NOT MEASURED when the family is run locally.
  • pnpm check:dual-build-cjs-loadsPREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. (49 packages unbuilt; needs a full pnpm build.)
  • pnpm check:type-check-debtPREREQUISITE NOT MET ... --re-measure cannot run: 35 workspace dependenc(ies) ... have no built type entry point on disk. Its sibling check:type-check-coverage ran green (OK — 69/79 workspace packages type-checked ... 10 in the DEBT ledger), and the targeted tsc reading above answers this gate's substantive question for the one package this PR touches, in the built-closure world the ledger uses.

Also green outside the derived family: pnpm check:nul-bytes (OK (scanned 7992 text file(s) ... no raw ASCII control bytes)), plus a direct control-byte scan of both changed files.

Merge preflight before opening: git merge-tree --write-tree --name-only origin/main HEAD lists no conflicting paths — content/docs/permissions/system-context.mdx is not among them, so no regeneration is owed.

Ablation — three, one per site, on the committed tree

Each mutation restores that site's map-first read, is line-neutral, is proven on disk by anchored occurrence counts before the run, and is restored with git checkout HEAD -- ABSOLUTE_PATH inside a trap ... EXIT INT TERM, the restore proven by git hash-object against the HEAD blob and an empty git diff HEAD. Because the pins import ./engine.js relatively (vitest resolves that to src/engine.ts, not the package's dist/ through exports), no rebuild is interposed; the reds themselves are the proof the mutation reached the code under test.

HEAD blob for engine.ts = f2c33d8b57cea0dce4054de3b8f1ee0ac9b88e6a; every restore leg read back the same hash and an empty git diff HEAD.

MutationResultWhat went red
site 1 — run = this.suspendedRuns.get(runId) ?? await this.loadSuspendedRunStrict(runId)2 red / 9 greenTHE BUG: a stale replica must not tell the wrong node executor its pause is over (teardown names lv1 while the run is parked at lv2) and NEW REACH (a map hit means the unreadable store is never read, so the cancel proceeds from the snapshot)
site 2 — const parent = this.suspendedRuns.get(parentId) ?? await this.loadSuspendedRun(parentId)2 red / 9 greenTHE BUG: a stale parent must not be failed at a node it has already left and the unreadable-ancestor control, red for site 1's reason
site 3 — drop the byId.has(r.runId) guard so the map overwrites again1 red / 10 greenTHE BUG: the durable row wins an id collision, the stale in-process copy does not (the listing reports lv1, one level stale)

The controls that stay green under all three are the postures each site had before: the no-store cases where the map IS the authority, the failed-durable-save cases the store's silence says nothing about, and the unlistable store's documented short list.

One reading was voided rather than reported: the first attempt at the site-1 mutation used a one-line anchor that occurs twice in engine.ts (the other in resumeInternal), the guard refused at anchor count 2, and the run was re-anchored on a unique three-line block. A second attempt was voided by the on-disk proof itself before any test ran. Both voided attempts are named here because a silently re-run ablation is the defect one layer up.

Scope

Diff is 3 files. Untouched as required: packages/spec/**, content/docs/releases/**, skills/**, the store implementation, PR #14567's subflow delegation block, and the region #14333 is queued for. No out-of-scope findings were filed — nothing outside the card's own surface was found.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8

Generated by Claude Code


Generated by Claude Code

os-salesand others added 2 commits September 2, 2026 16:37
…ers through the shared loader
`cancelRun`, `failAncestors` and `listSuspendedRunsDurable` still preferred the
per-process `suspendedRuns` map over the shared `SuspendedRunStore`, which
#13617 had already made authoritative for the resume path. Each now takes one
answer to "where is this run parked", with the degrading or strict loader
chosen per site so the recorded #4632 / #6299 degradation posture is preserved
by choice rather than re-derived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
… module
`AutomationContext` is declared in `@objectstack/spec/contracts` and re-imported
by `engine.ts` as a type — it is not re-exported from there, so importing it
from `./engine.js` added a fourth error (TS2459) to a package whose type-check
debt ledger is frozen at 3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/service-automation, touching 4 documentable anchor(s).

2 release-owned page(s) name something this change touched. These are read-only:

  • content/docs/releases/v16.mdx(via AutomationEngine (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via AutomationEngine (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 5 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from da191b53496857088142d233a3b34da26653db98 — the merge of head 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin da191b53496857088142d233a3b34da26653db98 && git checkout da191b53496857088142d233a3b34da26653db98
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff 89f5f8d15dacac7b65cd1198ff48339d5bd36a18
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-automation: three more readers of suspended-run state still prefer the per-process map over the shared store

1 participant

@os-sales
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(service-automation): route the last three suspended-run readers through the shared loader - #14650

Merged
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative
Sep 2, 2026
Merged

fix(service-automation): route the last three suspended-run readers through the shared loader#14650
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative

Conversation

@claude

@claudeclaudeBot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Fixes#14332

cancelRun, failAncestors and listSuspendedRunsDurable were the three readers of suspended-run state that #13617 deliberately left on the per-process suspendedRuns map. All three now take one answer to "where is this run parked", through the existing loadSuspendedRun / loadSuspendedRunStrict pair — with the degrading or strict loader chosen per site, so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived.

Ruling of record

No maintainer ruling is owed on this card — triage graded it dispatchable and fixed its shape. Quoted verbatim from triage (14332#issuecomment-5503565404):

Lands in:packages/services/service-automation/src/engine.ts — the three readers (cancelRun, failAncestors, listSuspendedRunsDurable). Ruled shape is the card's own: one answer to "where is this run parked" — route all three through loadSuspendedRun / loadSuspendedRunStrict, choosing the degrading or strict loader deliberately per site so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived; the getSuspendedScreen docblock already states the principle. p2 because site 2 (failAncestors) carries #13617's exact harm shape — failing an ancestor at a node it has already left — and site 1 tears down the wrong node executor's pause. ⛔ Site 3's "in-memory entries win" comment must be corrected in the same PR, not left as a true-sounding sentence that is backwards once several replicas share one store; its own docblock records it has no in-repo production consumer, so it is the cheapest of the three to make honest.

The per-site loader judgment

This is the deliverable, not the mechanical replacement. Each site's docblock now carries the same judgment in prose.

SiteLoader chosenNot found meansStore unreadable meansWhy that is the posture it already had
1 cancelRunstrict (loadSuspendedRunStrict)returns false — "already terminal / unknown", its documented idempotent successthrows out of the loader into this site's own catch, which keeps its #4632 DURABILITY record at errorThe degrading loader would have answered null under its own best-effort warn and silently downgraded the #4632 verdict. Its degradation decides a WRITE, so the strict form plus a local catch is the only shape that preserves both the return value and the level. Same shape resumeInternal uses.
2 failAncestorsdegrading (loadSuspendedRun)ends the walk (if (!parent) return;), unchangedreads as "no ancestor here" and ends the walk — never propagatesThis walk runs inside the catch arm already handling a run's failure, so it must not throw. "Degrade to null and stop" is exactly what the bare .catch(() => null) did. One thing gained: that silent swallow is now recorded, at the loader's declared best-effort warn.
3 listSuspendedRunsDurableneither — merge direction plus the corrected commentn/a (a listing, not a per-id read)unchanged: degrades to the in-memory cache alone at warn (#4632 FUNCTIONAL)A LIST is not a per-id answer. store.list() is capped and best-effort (ObjectStoreSuspendedRunStore reads at most 1000 paused rows) and the same merge is reached on the degraded path, so "absent from the list" is not the "the store answered and has no row" that loadSuspendedRunStrict rests on. The durable row wins a collision; map entries the listing does not carry are still included, deliberately.

Site 3's comment In-memory entries win — they are the freshest copy is corrected in place, together with the docblock sentence that repeated it (The in-memory cache takes precedence on id collisions).

Premise checks — verified on origin/main at 13bf05d, before the first edit

All hold; premise_still_valid: true.

  • P1cancelRun read this.suspendedRuns.get(runId) ?? null and only then store.load — map-first. HOLDS.
  • P2failAncestors walked $parentRunId with this.suspendedRuns.get(parentId) ?? (this.store ? await this.store.load(parentId).catch(() => null) : null). HOLDS.
  • P3listSuspendedRunsDurable merged the durable list under the map with the comment In-memory entries win — they are the freshest copy. HOLDS.
  • P4 The automation/approvals: 多副本集群下审批流每级节点(除首级)被重复创建 —— approve 后恢复读到滞后一拍的流运行态,同级要批两次(单副本零重复) #13617 pair exists and is unreshaped: loadSuspendedRunStrict is store-authoritative with the cacheOnlySuspensions fallback for a failed durable save; loadSuspendedRun is its degrading form; the getSuspendedScreen docblock still states one loader, two callers. HOLDS.
  • P5 Reproduced before the fix and kept as pins — see Ablation below, which measures all three reds per site on the committed tree. All three sites were defective; no partially false premise.
  • P6packages/spec/** and content/docs/releases/** untouched. HOLDS — the diff is 3 files: engine.ts, one new test file, one changeset.

PM hypotheses

  • H1 — holds. All three sites route through the existing pair with no third loader, and the degrading-or-strict choice is a judgment from each site's own docblock. Stated per site in the table above and in the code.
  • H2 — holds, with two named consequences rather than silent changes. No log site moved, no level changed, no thrown or returned shape changed, no row write changed. The finding(service-automation): engine.ts 还剩三处同形的 warn message 拼接 —— forgetSuspendedRun / cancelRun / listSuspendedRunsDurable,是 #5912+#6230 之后该文件的最后一批 #6299 byte-level pins for all three seams (suspended-run-store-consume-log-cause.test.ts, 20 tests) stay green untouched. Two consequences are stated in the docblocks, the changeset and the pins, not left to be discovered: (a) cancelRun — while a store is configured this process's map is no longer an answer, so a store outage reaches the existing error record even for a run this replica holds, where the old cache-first read cancelled from the local snapshot; that snapshot is the defect. (b) failAncestors — a store failure during the ancestor walk was swallowed in total silence by .catch(() => null) and is now recorded by the loader at warn. No new error-level site anywhere (the [Decision] plugin-sharing's refused-backfill report lands at warn where AGENTS.md puts it at error — and the card that was supposed to carry the level is CLOSED #13398-class ruling).
  • H3 — holds, verified independently with a positive control.git grep -n 'listSuspendedRunsDurable' -- 'packages/**' 'examples/**' 'apps/**' ':!*.test.ts' ':!*.spec.ts' returns only engine.ts's own definition and comments plus CHANGELOG prose — no consumer. Positive controls on the same command shape: hasSuspendedRun returns a real non-test consumer (packages/plugins/plugin-approvals/src/approval-service.ts:2704), and listSuspendedRuns returns its spec-contract declaration (packages/spec/src/contracts/automation-service.ts:589) — so the scan does find consumers where they exist.
  • H4 — holds. Clause-② is no. Declared from the actual diff: git diff -U0 origin/main...HEAD | grep export matches exactly one line, and it is prose inside the changeset (No signature, export or return-shape change...), not an export statement. No new exported symbol, no signature change on the three readers, no accept set moved. Changeset is patch. No needs:contract-review.
  • H5 — holds.service-automation: two concurrent resumes of one run on two replicas can both advance it — the idempotency guard is per-process #14333's concurrent-resume guard region is untouched: the diff in engine.ts is confined to cancelRun, failAncestors, listSuspendedRunsDurable and their docblocks.

Tests

New pins: packages/services/service-automation/src/multi-replica-suspended-run-readers.test.ts (11 tests), a sibling of the #13617 harness rather than more cases inside it. Reason recorded in its docblock: multi-replica-resume-staleness.test.ts carries a REVERT-PROOF ledger for one named mutation and its measured 4 red / 4 green split; adding these cases would falsify those counts and merge two different mutations' revert-proofs into one statement. The two files share only the two-engines-over-one-store shape.

Every site-1 and site-2 pin asserts on the onSuspensionReleasednotification (which node executor was told its pause is over), never on the row: the row delete is by id and is correct either way, which is the whole point of the card.

All runs below were taken at commit da4db89f5 and carry to the final head 89f5f8d because the two commits have a byte-identical tree: both are 514564fde37f75fd96c8f58e09ae139bc5ce0801, and git diff --stat da4db89f5 89f5f8d15 prints nothing. The extra commit exists only because this branch had already been pushed when a type-only import was corrected, and force-pushing is forbidden — so the amend was re-landed as a child commit rather than rewritten. Everything ran under scripts/pm/os-verify-lock.sh, exit captured before any pipe.

Full package suite — pnpm --filter @objectstack/service-automation test:

 Test Files 100 passed (100)
Tests 1182 passed (1182)
Duration 92.44s
os-verify-lock: VERDICT command-exit 0 · held the lock 92s

Type-check — the package has no typecheck script (it is a shrink-only DEBT ledger entry, frozen at 3), so a pnpm --filter ... typecheck would have matched zero scripts and exited 0 having measured nothing. Measured instead with the dependency closure built (pnpm --filter '@objectstack/service-automation^...' build, the world the ledger is measured in), tsc --noEmit -p tsconfig.json --listFiles:

src/nested-region-parity.test.ts(95,25): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(151,20): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(180,20): error TS2341: Property 'flows' is private ...
--- error count --- 3

Exactly the ledger's frozen 3, all pre-existing, none in the changed files; --listFiles confirms the new test file is in the program (this run is the reason a fourth error — a type imported from the wrong module — was caught and fixed before the head was final).

Repo-wide lint — no narrowing was needed, the whole population ran:

node --stack-size=4000 node_modules/eslint/bin/eslint.js . --no-inline-config
LINT EXIT=0 · os-verify-lock: VERDICT command-exit 0 · held the lock 116s

Gates

Derived on the final head from the merge-base change set, not a hand-written list: node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands at da4db89f5 (--repo asserted and held against this checkout's origin) — 36 commands over the 3 changed paths. The derivation is a property of the tree, which is byte-identical at the final head 89f5f8d15 (see Tests).

33 green, 0 red, 3 NOT MEASURED. The three are prerequisite refusals (exit code 3, which each script's own verdict text distinguishes from a finding's 1), not failures:

  • node scripts/check-test-completeness.mjsPREREQUISITE NOT MET — this gate grades a saved turbo run test log, and no log was named. Its own text says to record NOT MEASURED when the family is run locally.
  • pnpm check:dual-build-cjs-loadsPREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. (49 packages unbuilt; needs a full pnpm build.)
  • pnpm check:type-check-debtPREREQUISITE NOT MET ... --re-measure cannot run: 35 workspace dependenc(ies) ... have no built type entry point on disk. Its sibling check:type-check-coverage ran green (OK — 69/79 workspace packages type-checked ... 10 in the DEBT ledger), and the targeted tsc reading above answers this gate's substantive question for the one package this PR touches, in the built-closure world the ledger uses.

Also green outside the derived family: pnpm check:nul-bytes (OK (scanned 7992 text file(s) ... no raw ASCII control bytes)), plus a direct control-byte scan of both changed files.

Merge preflight before opening: git merge-tree --write-tree --name-only origin/main HEAD lists no conflicting paths — content/docs/permissions/system-context.mdx is not among them, so no regeneration is owed.

Ablation — three, one per site, on the committed tree

Each mutation restores that site's map-first read, is line-neutral, is proven on disk by anchored occurrence counts before the run, and is restored with git checkout HEAD -- ABSOLUTE_PATH inside a trap ... EXIT INT TERM, the restore proven by git hash-object against the HEAD blob and an empty git diff HEAD. Because the pins import ./engine.js relatively (vitest resolves that to src/engine.ts, not the package's dist/ through exports), no rebuild is interposed; the reds themselves are the proof the mutation reached the code under test.

HEAD blob for engine.ts = f2c33d8b57cea0dce4054de3b8f1ee0ac9b88e6a; every restore leg read back the same hash and an empty git diff HEAD.

MutationResultWhat went red
site 1 — run = this.suspendedRuns.get(runId) ?? await this.loadSuspendedRunStrict(runId)2 red / 9 greenTHE BUG: a stale replica must not tell the wrong node executor its pause is over (teardown names lv1 while the run is parked at lv2) and NEW REACH (a map hit means the unreadable store is never read, so the cancel proceeds from the snapshot)
site 2 — const parent = this.suspendedRuns.get(parentId) ?? await this.loadSuspendedRun(parentId)2 red / 9 greenTHE BUG: a stale parent must not be failed at a node it has already left and the unreadable-ancestor control, red for site 1's reason
site 3 — drop the byId.has(r.runId) guard so the map overwrites again1 red / 10 greenTHE BUG: the durable row wins an id collision, the stale in-process copy does not (the listing reports lv1, one level stale)

The controls that stay green under all three are the postures each site had before: the no-store cases where the map IS the authority, the failed-durable-save cases the store's silence says nothing about, and the unlistable store's documented short list.

One reading was voided rather than reported: the first attempt at the site-1 mutation used a one-line anchor that occurs twice in engine.ts (the other in resumeInternal), the guard refused at anchor count 2, and the run was re-anchored on a unique three-line block. A second attempt was voided by the on-disk proof itself before any test ran. Both voided attempts are named here because a silently re-run ablation is the defect one layer up.

Scope

Diff is 3 files. Untouched as required: packages/spec/**, content/docs/releases/**, skills/**, the store implementation, PR #14567's subflow delegation block, and the region #14333 is queued for. No out-of-scope findings were filed — nothing outside the card's own surface was found.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8

Generated by Claude Code


Generated by Claude Code

os-salesand others added 2 commits September 2, 2026 16:37
…ers through the shared loader
`cancelRun`, `failAncestors` and `listSuspendedRunsDurable` still preferred the
per-process `suspendedRuns` map over the shared `SuspendedRunStore`, which
#13617 had already made authoritative for the resume path. Each now takes one
answer to "where is this run parked", with the degrading or strict loader
chosen per site so the recorded #4632 / #6299 degradation posture is preserved
by choice rather than re-derived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
… module
`AutomationContext` is declared in `@objectstack/spec/contracts` and re-imported
by `engine.ts` as a type — it is not re-exported from there, so importing it
from `./engine.js` added a fourth error (TS2459) to a package whose type-check
debt ledger is frozen at 3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/service-automation, touching 4 documentable anchor(s).

2 release-owned page(s) name something this change touched. These are read-only:

  • content/docs/releases/v16.mdx(via AutomationEngine (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via AutomationEngine (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 5 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from da191b53496857088142d233a3b34da26653db98 — the merge of head 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin da191b53496857088142d233a3b34da26653db98 && git checkout da191b53496857088142d233a3b34da26653db98
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff 89f5f8d15dacac7b65cd1198ff48339d5bd36a18
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-automation: three more readers of suspended-run state still prefer the per-process map over the shared store

1 participant

@os-sales
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(service-automation): route the last three suspended-run readers through the shared loader - #14650

Merged
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative
Sep 2, 2026
Merged

fix(service-automation): route the last three suspended-run readers through the shared loader#14650
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative

Conversation

@claude

@claudeclaudeBot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Fixes#14332

cancelRun, failAncestors and listSuspendedRunsDurable were the three readers of suspended-run state that #13617 deliberately left on the per-process suspendedRuns map. All three now take one answer to "where is this run parked", through the existing loadSuspendedRun / loadSuspendedRunStrict pair — with the degrading or strict loader chosen per site, so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived.

Ruling of record

No maintainer ruling is owed on this card — triage graded it dispatchable and fixed its shape. Quoted verbatim from triage (14332#issuecomment-5503565404):

Lands in:packages/services/service-automation/src/engine.ts — the three readers (cancelRun, failAncestors, listSuspendedRunsDurable). Ruled shape is the card's own: one answer to "where is this run parked" — route all three through loadSuspendedRun / loadSuspendedRunStrict, choosing the degrading or strict loader deliberately per site so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived; the getSuspendedScreen docblock already states the principle. p2 because site 2 (failAncestors) carries #13617's exact harm shape — failing an ancestor at a node it has already left — and site 1 tears down the wrong node executor's pause. ⛔ Site 3's "in-memory entries win" comment must be corrected in the same PR, not left as a true-sounding sentence that is backwards once several replicas share one store; its own docblock records it has no in-repo production consumer, so it is the cheapest of the three to make honest.

The per-site loader judgment

This is the deliverable, not the mechanical replacement. Each site's docblock now carries the same judgment in prose.

SiteLoader chosenNot found meansStore unreadable meansWhy that is the posture it already had
1 cancelRunstrict (loadSuspendedRunStrict)returns false — "already terminal / unknown", its documented idempotent successthrows out of the loader into this site's own catch, which keeps its #4632 DURABILITY record at errorThe degrading loader would have answered null under its own best-effort warn and silently downgraded the #4632 verdict. Its degradation decides a WRITE, so the strict form plus a local catch is the only shape that preserves both the return value and the level. Same shape resumeInternal uses.
2 failAncestorsdegrading (loadSuspendedRun)ends the walk (if (!parent) return;), unchangedreads as "no ancestor here" and ends the walk — never propagatesThis walk runs inside the catch arm already handling a run's failure, so it must not throw. "Degrade to null and stop" is exactly what the bare .catch(() => null) did. One thing gained: that silent swallow is now recorded, at the loader's declared best-effort warn.
3 listSuspendedRunsDurableneither — merge direction plus the corrected commentn/a (a listing, not a per-id read)unchanged: degrades to the in-memory cache alone at warn (#4632 FUNCTIONAL)A LIST is not a per-id answer. store.list() is capped and best-effort (ObjectStoreSuspendedRunStore reads at most 1000 paused rows) and the same merge is reached on the degraded path, so "absent from the list" is not the "the store answered and has no row" that loadSuspendedRunStrict rests on. The durable row wins a collision; map entries the listing does not carry are still included, deliberately.

Site 3's comment In-memory entries win — they are the freshest copy is corrected in place, together with the docblock sentence that repeated it (The in-memory cache takes precedence on id collisions).

Premise checks — verified on origin/main at 13bf05d, before the first edit

All hold; premise_still_valid: true.

  • P1cancelRun read this.suspendedRuns.get(runId) ?? null and only then store.load — map-first. HOLDS.
  • P2failAncestors walked $parentRunId with this.suspendedRuns.get(parentId) ?? (this.store ? await this.store.load(parentId).catch(() => null) : null). HOLDS.
  • P3listSuspendedRunsDurable merged the durable list under the map with the comment In-memory entries win — they are the freshest copy. HOLDS.
  • P4 The automation/approvals: 多副本集群下审批流每级节点(除首级)被重复创建 —— approve 后恢复读到滞后一拍的流运行态,同级要批两次(单副本零重复) #13617 pair exists and is unreshaped: loadSuspendedRunStrict is store-authoritative with the cacheOnlySuspensions fallback for a failed durable save; loadSuspendedRun is its degrading form; the getSuspendedScreen docblock still states one loader, two callers. HOLDS.
  • P5 Reproduced before the fix and kept as pins — see Ablation below, which measures all three reds per site on the committed tree. All three sites were defective; no partially false premise.
  • P6packages/spec/** and content/docs/releases/** untouched. HOLDS — the diff is 3 files: engine.ts, one new test file, one changeset.

PM hypotheses

  • H1 — holds. All three sites route through the existing pair with no third loader, and the degrading-or-strict choice is a judgment from each site's own docblock. Stated per site in the table above and in the code.
  • H2 — holds, with two named consequences rather than silent changes. No log site moved, no level changed, no thrown or returned shape changed, no row write changed. The finding(service-automation): engine.ts 还剩三处同形的 warn message 拼接 —— forgetSuspendedRun / cancelRun / listSuspendedRunsDurable,是 #5912+#6230 之后该文件的最后一批 #6299 byte-level pins for all three seams (suspended-run-store-consume-log-cause.test.ts, 20 tests) stay green untouched. Two consequences are stated in the docblocks, the changeset and the pins, not left to be discovered: (a) cancelRun — while a store is configured this process's map is no longer an answer, so a store outage reaches the existing error record even for a run this replica holds, where the old cache-first read cancelled from the local snapshot; that snapshot is the defect. (b) failAncestors — a store failure during the ancestor walk was swallowed in total silence by .catch(() => null) and is now recorded by the loader at warn. No new error-level site anywhere (the [Decision] plugin-sharing's refused-backfill report lands at warn where AGENTS.md puts it at error — and the card that was supposed to carry the level is CLOSED #13398-class ruling).
  • H3 — holds, verified independently with a positive control.git grep -n 'listSuspendedRunsDurable' -- 'packages/**' 'examples/**' 'apps/**' ':!*.test.ts' ':!*.spec.ts' returns only engine.ts's own definition and comments plus CHANGELOG prose — no consumer. Positive controls on the same command shape: hasSuspendedRun returns a real non-test consumer (packages/plugins/plugin-approvals/src/approval-service.ts:2704), and listSuspendedRuns returns its spec-contract declaration (packages/spec/src/contracts/automation-service.ts:589) — so the scan does find consumers where they exist.
  • H4 — holds. Clause-② is no. Declared from the actual diff: git diff -U0 origin/main...HEAD | grep export matches exactly one line, and it is prose inside the changeset (No signature, export or return-shape change...), not an export statement. No new exported symbol, no signature change on the three readers, no accept set moved. Changeset is patch. No needs:contract-review.
  • H5 — holds.service-automation: two concurrent resumes of one run on two replicas can both advance it — the idempotency guard is per-process #14333's concurrent-resume guard region is untouched: the diff in engine.ts is confined to cancelRun, failAncestors, listSuspendedRunsDurable and their docblocks.

Tests

New pins: packages/services/service-automation/src/multi-replica-suspended-run-readers.test.ts (11 tests), a sibling of the #13617 harness rather than more cases inside it. Reason recorded in its docblock: multi-replica-resume-staleness.test.ts carries a REVERT-PROOF ledger for one named mutation and its measured 4 red / 4 green split; adding these cases would falsify those counts and merge two different mutations' revert-proofs into one statement. The two files share only the two-engines-over-one-store shape.

Every site-1 and site-2 pin asserts on the onSuspensionReleasednotification (which node executor was told its pause is over), never on the row: the row delete is by id and is correct either way, which is the whole point of the card.

All runs below were taken at commit da4db89f5 and carry to the final head 89f5f8d because the two commits have a byte-identical tree: both are 514564fde37f75fd96c8f58e09ae139bc5ce0801, and git diff --stat da4db89f5 89f5f8d15 prints nothing. The extra commit exists only because this branch had already been pushed when a type-only import was corrected, and force-pushing is forbidden — so the amend was re-landed as a child commit rather than rewritten. Everything ran under scripts/pm/os-verify-lock.sh, exit captured before any pipe.

Full package suite — pnpm --filter @objectstack/service-automation test:

 Test Files 100 passed (100)
Tests 1182 passed (1182)
Duration 92.44s
os-verify-lock: VERDICT command-exit 0 · held the lock 92s

Type-check — the package has no typecheck script (it is a shrink-only DEBT ledger entry, frozen at 3), so a pnpm --filter ... typecheck would have matched zero scripts and exited 0 having measured nothing. Measured instead with the dependency closure built (pnpm --filter '@objectstack/service-automation^...' build, the world the ledger is measured in), tsc --noEmit -p tsconfig.json --listFiles:

src/nested-region-parity.test.ts(95,25): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(151,20): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(180,20): error TS2341: Property 'flows' is private ...
--- error count --- 3

Exactly the ledger's frozen 3, all pre-existing, none in the changed files; --listFiles confirms the new test file is in the program (this run is the reason a fourth error — a type imported from the wrong module — was caught and fixed before the head was final).

Repo-wide lint — no narrowing was needed, the whole population ran:

node --stack-size=4000 node_modules/eslint/bin/eslint.js . --no-inline-config
LINT EXIT=0 · os-verify-lock: VERDICT command-exit 0 · held the lock 116s

Gates

Derived on the final head from the merge-base change set, not a hand-written list: node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands at da4db89f5 (--repo asserted and held against this checkout's origin) — 36 commands over the 3 changed paths. The derivation is a property of the tree, which is byte-identical at the final head 89f5f8d15 (see Tests).

33 green, 0 red, 3 NOT MEASURED. The three are prerequisite refusals (exit code 3, which each script's own verdict text distinguishes from a finding's 1), not failures:

  • node scripts/check-test-completeness.mjsPREREQUISITE NOT MET — this gate grades a saved turbo run test log, and no log was named. Its own text says to record NOT MEASURED when the family is run locally.
  • pnpm check:dual-build-cjs-loadsPREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. (49 packages unbuilt; needs a full pnpm build.)
  • pnpm check:type-check-debtPREREQUISITE NOT MET ... --re-measure cannot run: 35 workspace dependenc(ies) ... have no built type entry point on disk. Its sibling check:type-check-coverage ran green (OK — 69/79 workspace packages type-checked ... 10 in the DEBT ledger), and the targeted tsc reading above answers this gate's substantive question for the one package this PR touches, in the built-closure world the ledger uses.

Also green outside the derived family: pnpm check:nul-bytes (OK (scanned 7992 text file(s) ... no raw ASCII control bytes)), plus a direct control-byte scan of both changed files.

Merge preflight before opening: git merge-tree --write-tree --name-only origin/main HEAD lists no conflicting paths — content/docs/permissions/system-context.mdx is not among them, so no regeneration is owed.

Ablation — three, one per site, on the committed tree

Each mutation restores that site's map-first read, is line-neutral, is proven on disk by anchored occurrence counts before the run, and is restored with git checkout HEAD -- ABSOLUTE_PATH inside a trap ... EXIT INT TERM, the restore proven by git hash-object against the HEAD blob and an empty git diff HEAD. Because the pins import ./engine.js relatively (vitest resolves that to src/engine.ts, not the package's dist/ through exports), no rebuild is interposed; the reds themselves are the proof the mutation reached the code under test.

HEAD blob for engine.ts = f2c33d8b57cea0dce4054de3b8f1ee0ac9b88e6a; every restore leg read back the same hash and an empty git diff HEAD.

MutationResultWhat went red
site 1 — run = this.suspendedRuns.get(runId) ?? await this.loadSuspendedRunStrict(runId)2 red / 9 greenTHE BUG: a stale replica must not tell the wrong node executor its pause is over (teardown names lv1 while the run is parked at lv2) and NEW REACH (a map hit means the unreadable store is never read, so the cancel proceeds from the snapshot)
site 2 — const parent = this.suspendedRuns.get(parentId) ?? await this.loadSuspendedRun(parentId)2 red / 9 greenTHE BUG: a stale parent must not be failed at a node it has already left and the unreadable-ancestor control, red for site 1's reason
site 3 — drop the byId.has(r.runId) guard so the map overwrites again1 red / 10 greenTHE BUG: the durable row wins an id collision, the stale in-process copy does not (the listing reports lv1, one level stale)

The controls that stay green under all three are the postures each site had before: the no-store cases where the map IS the authority, the failed-durable-save cases the store's silence says nothing about, and the unlistable store's documented short list.

One reading was voided rather than reported: the first attempt at the site-1 mutation used a one-line anchor that occurs twice in engine.ts (the other in resumeInternal), the guard refused at anchor count 2, and the run was re-anchored on a unique three-line block. A second attempt was voided by the on-disk proof itself before any test ran. Both voided attempts are named here because a silently re-run ablation is the defect one layer up.

Scope

Diff is 3 files. Untouched as required: packages/spec/**, content/docs/releases/**, skills/**, the store implementation, PR #14567's subflow delegation block, and the region #14333 is queued for. No out-of-scope findings were filed — nothing outside the card's own surface was found.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8

Generated by Claude Code


Generated by Claude Code

os-salesand others added 2 commits September 2, 2026 16:37
…ers through the shared loader
`cancelRun`, `failAncestors` and `listSuspendedRunsDurable` still preferred the
per-process `suspendedRuns` map over the shared `SuspendedRunStore`, which
#13617 had already made authoritative for the resume path. Each now takes one
answer to "where is this run parked", with the degrading or strict loader
chosen per site so the recorded #4632 / #6299 degradation posture is preserved
by choice rather than re-derived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
… module
`AutomationContext` is declared in `@objectstack/spec/contracts` and re-imported
by `engine.ts` as a type — it is not re-exported from there, so importing it
from `./engine.js` added a fourth error (TS2459) to a package whose type-check
debt ledger is frozen at 3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/service-automation, touching 4 documentable anchor(s).

2 release-owned page(s) name something this change touched. These are read-only:

  • content/docs/releases/v16.mdx(via AutomationEngine (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via AutomationEngine (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 5 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from da191b53496857088142d233a3b34da26653db98 — the merge of head 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin da191b53496857088142d233a3b34da26653db98 && git checkout da191b53496857088142d233a3b34da26653db98
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff 89f5f8d15dacac7b65cd1198ff48339d5bd36a18
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-automation: three more readers of suspended-run state still prefer the per-process map over the shared store

1 participant

@os-sales
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

fix(service-automation): route the last three suspended-run readers through the shared loader - #14650

Merged
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative
Sep 2, 2026
Merged

fix(service-automation): route the last three suspended-run readers through the shared loader#14650
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative

Conversation

@claude

@claudeclaudeBot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Fixes#14332

cancelRun, failAncestors and listSuspendedRunsDurable were the three readers of suspended-run state that #13617 deliberately left on the per-process suspendedRuns map. All three now take one answer to "where is this run parked", through the existing loadSuspendedRun / loadSuspendedRunStrict pair — with the degrading or strict loader chosen per site, so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived.

Ruling of record

No maintainer ruling is owed on this card — triage graded it dispatchable and fixed its shape. Quoted verbatim from triage (14332#issuecomment-5503565404):

Lands in:packages/services/service-automation/src/engine.ts — the three readers (cancelRun, failAncestors, listSuspendedRunsDurable). Ruled shape is the card's own: one answer to "where is this run parked" — route all three through loadSuspendedRun / loadSuspendedRunStrict, choosing the degrading or strict loader deliberately per site so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived; the getSuspendedScreen docblock already states the principle. p2 because site 2 (failAncestors) carries #13617's exact harm shape — failing an ancestor at a node it has already left — and site 1 tears down the wrong node executor's pause. ⛔ Site 3's "in-memory entries win" comment must be corrected in the same PR, not left as a true-sounding sentence that is backwards once several replicas share one store; its own docblock records it has no in-repo production consumer, so it is the cheapest of the three to make honest.

The per-site loader judgment

This is the deliverable, not the mechanical replacement. Each site's docblock now carries the same judgment in prose.

SiteLoader chosenNot found meansStore unreadable meansWhy that is the posture it already had
1 cancelRunstrict (loadSuspendedRunStrict)returns false — "already terminal / unknown", its documented idempotent successthrows out of the loader into this site's own catch, which keeps its #4632 DURABILITY record at errorThe degrading loader would have answered null under its own best-effort warn and silently downgraded the #4632 verdict. Its degradation decides a WRITE, so the strict form plus a local catch is the only shape that preserves both the return value and the level. Same shape resumeInternal uses.
2 failAncestorsdegrading (loadSuspendedRun)ends the walk (if (!parent) return;), unchangedreads as "no ancestor here" and ends the walk — never propagatesThis walk runs inside the catch arm already handling a run's failure, so it must not throw. "Degrade to null and stop" is exactly what the bare .catch(() => null) did. One thing gained: that silent swallow is now recorded, at the loader's declared best-effort warn.
3 listSuspendedRunsDurableneither — merge direction plus the corrected commentn/a (a listing, not a per-id read)unchanged: degrades to the in-memory cache alone at warn (#4632 FUNCTIONAL)A LIST is not a per-id answer. store.list() is capped and best-effort (ObjectStoreSuspendedRunStore reads at most 1000 paused rows) and the same merge is reached on the degraded path, so "absent from the list" is not the "the store answered and has no row" that loadSuspendedRunStrict rests on. The durable row wins a collision; map entries the listing does not carry are still included, deliberately.

Site 3's comment In-memory entries win — they are the freshest copy is corrected in place, together with the docblock sentence that repeated it (The in-memory cache takes precedence on id collisions).

Premise checks — verified on origin/main at 13bf05d, before the first edit

All hold; premise_still_valid: true.

  • P1cancelRun read this.suspendedRuns.get(runId) ?? null and only then store.load — map-first. HOLDS.
  • P2failAncestors walked $parentRunId with this.suspendedRuns.get(parentId) ?? (this.store ? await this.store.load(parentId).catch(() => null) : null). HOLDS.
  • P3listSuspendedRunsDurable merged the durable list under the map with the comment In-memory entries win — they are the freshest copy. HOLDS.
  • P4 The automation/approvals: 多副本集群下审批流每级节点(除首级)被重复创建 —— approve 后恢复读到滞后一拍的流运行态,同级要批两次(单副本零重复) #13617 pair exists and is unreshaped: loadSuspendedRunStrict is store-authoritative with the cacheOnlySuspensions fallback for a failed durable save; loadSuspendedRun is its degrading form; the getSuspendedScreen docblock still states one loader, two callers. HOLDS.
  • P5 Reproduced before the fix and kept as pins — see Ablation below, which measures all three reds per site on the committed tree. All three sites were defective; no partially false premise.
  • P6packages/spec/** and content/docs/releases/** untouched. HOLDS — the diff is 3 files: engine.ts, one new test file, one changeset.

PM hypotheses

  • H1 — holds. All three sites route through the existing pair with no third loader, and the degrading-or-strict choice is a judgment from each site's own docblock. Stated per site in the table above and in the code.
  • H2 — holds, with two named consequences rather than silent changes. No log site moved, no level changed, no thrown or returned shape changed, no row write changed. The finding(service-automation): engine.ts 还剩三处同形的 warn message 拼接 —— forgetSuspendedRun / cancelRun / listSuspendedRunsDurable,是 #5912+#6230 之后该文件的最后一批 #6299 byte-level pins for all three seams (suspended-run-store-consume-log-cause.test.ts, 20 tests) stay green untouched. Two consequences are stated in the docblocks, the changeset and the pins, not left to be discovered: (a) cancelRun — while a store is configured this process's map is no longer an answer, so a store outage reaches the existing error record even for a run this replica holds, where the old cache-first read cancelled from the local snapshot; that snapshot is the defect. (b) failAncestors — a store failure during the ancestor walk was swallowed in total silence by .catch(() => null) and is now recorded by the loader at warn. No new error-level site anywhere (the [Decision] plugin-sharing's refused-backfill report lands at warn where AGENTS.md puts it at error — and the card that was supposed to carry the level is CLOSED #13398-class ruling).
  • H3 — holds, verified independently with a positive control.git grep -n 'listSuspendedRunsDurable' -- 'packages/**' 'examples/**' 'apps/**' ':!*.test.ts' ':!*.spec.ts' returns only engine.ts's own definition and comments plus CHANGELOG prose — no consumer. Positive controls on the same command shape: hasSuspendedRun returns a real non-test consumer (packages/plugins/plugin-approvals/src/approval-service.ts:2704), and listSuspendedRuns returns its spec-contract declaration (packages/spec/src/contracts/automation-service.ts:589) — so the scan does find consumers where they exist.
  • H4 — holds. Clause-② is no. Declared from the actual diff: git diff -U0 origin/main...HEAD | grep export matches exactly one line, and it is prose inside the changeset (No signature, export or return-shape change...), not an export statement. No new exported symbol, no signature change on the three readers, no accept set moved. Changeset is patch. No needs:contract-review.
  • H5 — holds.service-automation: two concurrent resumes of one run on two replicas can both advance it — the idempotency guard is per-process #14333's concurrent-resume guard region is untouched: the diff in engine.ts is confined to cancelRun, failAncestors, listSuspendedRunsDurable and their docblocks.

Tests

New pins: packages/services/service-automation/src/multi-replica-suspended-run-readers.test.ts (11 tests), a sibling of the #13617 harness rather than more cases inside it. Reason recorded in its docblock: multi-replica-resume-staleness.test.ts carries a REVERT-PROOF ledger for one named mutation and its measured 4 red / 4 green split; adding these cases would falsify those counts and merge two different mutations' revert-proofs into one statement. The two files share only the two-engines-over-one-store shape.

Every site-1 and site-2 pin asserts on the onSuspensionReleasednotification (which node executor was told its pause is over), never on the row: the row delete is by id and is correct either way, which is the whole point of the card.

All runs below were taken at commit da4db89f5 and carry to the final head 89f5f8d because the two commits have a byte-identical tree: both are 514564fde37f75fd96c8f58e09ae139bc5ce0801, and git diff --stat da4db89f5 89f5f8d15 prints nothing. The extra commit exists only because this branch had already been pushed when a type-only import was corrected, and force-pushing is forbidden — so the amend was re-landed as a child commit rather than rewritten. Everything ran under scripts/pm/os-verify-lock.sh, exit captured before any pipe.

Full package suite — pnpm --filter @objectstack/service-automation test:

 Test Files 100 passed (100)
Tests 1182 passed (1182)
Duration 92.44s
os-verify-lock: VERDICT command-exit 0 · held the lock 92s

Type-check — the package has no typecheck script (it is a shrink-only DEBT ledger entry, frozen at 3), so a pnpm --filter ... typecheck would have matched zero scripts and exited 0 having measured nothing. Measured instead with the dependency closure built (pnpm --filter '@objectstack/service-automation^...' build, the world the ledger is measured in), tsc --noEmit -p tsconfig.json --listFiles:

src/nested-region-parity.test.ts(95,25): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(151,20): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(180,20): error TS2341: Property 'flows' is private ...
--- error count --- 3

Exactly the ledger's frozen 3, all pre-existing, none in the changed files; --listFiles confirms the new test file is in the program (this run is the reason a fourth error — a type imported from the wrong module — was caught and fixed before the head was final).

Repo-wide lint — no narrowing was needed, the whole population ran:

node --stack-size=4000 node_modules/eslint/bin/eslint.js . --no-inline-config
LINT EXIT=0 · os-verify-lock: VERDICT command-exit 0 · held the lock 116s

Gates

Derived on the final head from the merge-base change set, not a hand-written list: node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands at da4db89f5 (--repo asserted and held against this checkout's origin) — 36 commands over the 3 changed paths. The derivation is a property of the tree, which is byte-identical at the final head 89f5f8d15 (see Tests).

33 green, 0 red, 3 NOT MEASURED. The three are prerequisite refusals (exit code 3, which each script's own verdict text distinguishes from a finding's 1), not failures:

  • node scripts/check-test-completeness.mjsPREREQUISITE NOT MET — this gate grades a saved turbo run test log, and no log was named. Its own text says to record NOT MEASURED when the family is run locally.
  • pnpm check:dual-build-cjs-loadsPREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. (49 packages unbuilt; needs a full pnpm build.)
  • pnpm check:type-check-debtPREREQUISITE NOT MET ... --re-measure cannot run: 35 workspace dependenc(ies) ... have no built type entry point on disk. Its sibling check:type-check-coverage ran green (OK — 69/79 workspace packages type-checked ... 10 in the DEBT ledger), and the targeted tsc reading above answers this gate's substantive question for the one package this PR touches, in the built-closure world the ledger uses.

Also green outside the derived family: pnpm check:nul-bytes (OK (scanned 7992 text file(s) ... no raw ASCII control bytes)), plus a direct control-byte scan of both changed files.

Merge preflight before opening: git merge-tree --write-tree --name-only origin/main HEAD lists no conflicting paths — content/docs/permissions/system-context.mdx is not among them, so no regeneration is owed.

Ablation — three, one per site, on the committed tree

Each mutation restores that site's map-first read, is line-neutral, is proven on disk by anchored occurrence counts before the run, and is restored with git checkout HEAD -- ABSOLUTE_PATH inside a trap ... EXIT INT TERM, the restore proven by git hash-object against the HEAD blob and an empty git diff HEAD. Because the pins import ./engine.js relatively (vitest resolves that to src/engine.ts, not the package's dist/ through exports), no rebuild is interposed; the reds themselves are the proof the mutation reached the code under test.

HEAD blob for engine.ts = f2c33d8b57cea0dce4054de3b8f1ee0ac9b88e6a; every restore leg read back the same hash and an empty git diff HEAD.

MutationResultWhat went red
site 1 — run = this.suspendedRuns.get(runId) ?? await this.loadSuspendedRunStrict(runId)2 red / 9 greenTHE BUG: a stale replica must not tell the wrong node executor its pause is over (teardown names lv1 while the run is parked at lv2) and NEW REACH (a map hit means the unreadable store is never read, so the cancel proceeds from the snapshot)
site 2 — const parent = this.suspendedRuns.get(parentId) ?? await this.loadSuspendedRun(parentId)2 red / 9 greenTHE BUG: a stale parent must not be failed at a node it has already left and the unreadable-ancestor control, red for site 1's reason
site 3 — drop the byId.has(r.runId) guard so the map overwrites again1 red / 10 greenTHE BUG: the durable row wins an id collision, the stale in-process copy does not (the listing reports lv1, one level stale)

The controls that stay green under all three are the postures each site had before: the no-store cases where the map IS the authority, the failed-durable-save cases the store's silence says nothing about, and the unlistable store's documented short list.

One reading was voided rather than reported: the first attempt at the site-1 mutation used a one-line anchor that occurs twice in engine.ts (the other in resumeInternal), the guard refused at anchor count 2, and the run was re-anchored on a unique three-line block. A second attempt was voided by the on-disk proof itself before any test ran. Both voided attempts are named here because a silently re-run ablation is the defect one layer up.

Scope

Diff is 3 files. Untouched as required: packages/spec/**, content/docs/releases/**, skills/**, the store implementation, PR #14567's subflow delegation block, and the region #14333 is queued for. No out-of-scope findings were filed — nothing outside the card's own surface was found.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8

Generated by Claude Code


Generated by Claude Code

os-salesand others added 2 commits September 2, 2026 16:37
…ers through the shared loader
`cancelRun`, `failAncestors` and `listSuspendedRunsDurable` still preferred the
per-process `suspendedRuns` map over the shared `SuspendedRunStore`, which
#13617 had already made authoritative for the resume path. Each now takes one
answer to "where is this run parked", with the degrading or strict loader
chosen per site so the recorded #4632 / #6299 degradation posture is preserved
by choice rather than re-derived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
… module
`AutomationContext` is declared in `@objectstack/spec/contracts` and re-imported
by `engine.ts` as a type — it is not re-exported from there, so importing it
from `./engine.js` added a fourth error (TS2459) to a package whose type-check
debt ledger is frozen at 3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/service-automation, touching 4 documentable anchor(s).

2 release-owned page(s) name something this change touched. These are read-only:

  • content/docs/releases/v16.mdx(via AutomationEngine (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via AutomationEngine (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 5 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from da191b53496857088142d233a3b34da26653db98 — the merge of head 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin da191b53496857088142d233a3b34da26653db98 && git checkout da191b53496857088142d233a3b34da26653db98
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff 89f5f8d15dacac7b65cd1198ff48339d5bd36a18
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-automation: three more readers of suspended-run state still prefer the per-process map over the shared store

1 participant

@os-sales
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(service-automation): route the last three suspended-run readers through the shared loader - #14650

Merged
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative
Sep 2, 2026
Merged

fix(service-automation): route the last three suspended-run readers through the shared loader#14650
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative

Conversation

@claude

@claudeclaudeBot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Fixes#14332

cancelRun, failAncestors and listSuspendedRunsDurable were the three readers of suspended-run state that #13617 deliberately left on the per-process suspendedRuns map. All three now take one answer to "where is this run parked", through the existing loadSuspendedRun / loadSuspendedRunStrict pair — with the degrading or strict loader chosen per site, so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived.

Ruling of record

No maintainer ruling is owed on this card — triage graded it dispatchable and fixed its shape. Quoted verbatim from triage (14332#issuecomment-5503565404):

Lands in:packages/services/service-automation/src/engine.ts — the three readers (cancelRun, failAncestors, listSuspendedRunsDurable). Ruled shape is the card's own: one answer to "where is this run parked" — route all three through loadSuspendedRun / loadSuspendedRunStrict, choosing the degrading or strict loader deliberately per site so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived; the getSuspendedScreen docblock already states the principle. p2 because site 2 (failAncestors) carries #13617's exact harm shape — failing an ancestor at a node it has already left — and site 1 tears down the wrong node executor's pause. ⛔ Site 3's "in-memory entries win" comment must be corrected in the same PR, not left as a true-sounding sentence that is backwards once several replicas share one store; its own docblock records it has no in-repo production consumer, so it is the cheapest of the three to make honest.

The per-site loader judgment

This is the deliverable, not the mechanical replacement. Each site's docblock now carries the same judgment in prose.

SiteLoader chosenNot found meansStore unreadable meansWhy that is the posture it already had
1 cancelRunstrict (loadSuspendedRunStrict)returns false — "already terminal / unknown", its documented idempotent successthrows out of the loader into this site's own catch, which keeps its #4632 DURABILITY record at errorThe degrading loader would have answered null under its own best-effort warn and silently downgraded the #4632 verdict. Its degradation decides a WRITE, so the strict form plus a local catch is the only shape that preserves both the return value and the level. Same shape resumeInternal uses.
2 failAncestorsdegrading (loadSuspendedRun)ends the walk (if (!parent) return;), unchangedreads as "no ancestor here" and ends the walk — never propagatesThis walk runs inside the catch arm already handling a run's failure, so it must not throw. "Degrade to null and stop" is exactly what the bare .catch(() => null) did. One thing gained: that silent swallow is now recorded, at the loader's declared best-effort warn.
3 listSuspendedRunsDurableneither — merge direction plus the corrected commentn/a (a listing, not a per-id read)unchanged: degrades to the in-memory cache alone at warn (#4632 FUNCTIONAL)A LIST is not a per-id answer. store.list() is capped and best-effort (ObjectStoreSuspendedRunStore reads at most 1000 paused rows) and the same merge is reached on the degraded path, so "absent from the list" is not the "the store answered and has no row" that loadSuspendedRunStrict rests on. The durable row wins a collision; map entries the listing does not carry are still included, deliberately.

Site 3's comment In-memory entries win — they are the freshest copy is corrected in place, together with the docblock sentence that repeated it (The in-memory cache takes precedence on id collisions).

Premise checks — verified on origin/main at 13bf05d, before the first edit

All hold; premise_still_valid: true.

  • P1cancelRun read this.suspendedRuns.get(runId) ?? null and only then store.load — map-first. HOLDS.
  • P2failAncestors walked $parentRunId with this.suspendedRuns.get(parentId) ?? (this.store ? await this.store.load(parentId).catch(() => null) : null). HOLDS.
  • P3listSuspendedRunsDurable merged the durable list under the map with the comment In-memory entries win — they are the freshest copy. HOLDS.
  • P4 The automation/approvals: 多副本集群下审批流每级节点(除首级)被重复创建 —— approve 后恢复读到滞后一拍的流运行态,同级要批两次(单副本零重复) #13617 pair exists and is unreshaped: loadSuspendedRunStrict is store-authoritative with the cacheOnlySuspensions fallback for a failed durable save; loadSuspendedRun is its degrading form; the getSuspendedScreen docblock still states one loader, two callers. HOLDS.
  • P5 Reproduced before the fix and kept as pins — see Ablation below, which measures all three reds per site on the committed tree. All three sites were defective; no partially false premise.
  • P6packages/spec/** and content/docs/releases/** untouched. HOLDS — the diff is 3 files: engine.ts, one new test file, one changeset.

PM hypotheses

  • H1 — holds. All three sites route through the existing pair with no third loader, and the degrading-or-strict choice is a judgment from each site's own docblock. Stated per site in the table above and in the code.
  • H2 — holds, with two named consequences rather than silent changes. No log site moved, no level changed, no thrown or returned shape changed, no row write changed. The finding(service-automation): engine.ts 还剩三处同形的 warn message 拼接 —— forgetSuspendedRun / cancelRun / listSuspendedRunsDurable,是 #5912+#6230 之后该文件的最后一批 #6299 byte-level pins for all three seams (suspended-run-store-consume-log-cause.test.ts, 20 tests) stay green untouched. Two consequences are stated in the docblocks, the changeset and the pins, not left to be discovered: (a) cancelRun — while a store is configured this process's map is no longer an answer, so a store outage reaches the existing error record even for a run this replica holds, where the old cache-first read cancelled from the local snapshot; that snapshot is the defect. (b) failAncestors — a store failure during the ancestor walk was swallowed in total silence by .catch(() => null) and is now recorded by the loader at warn. No new error-level site anywhere (the [Decision] plugin-sharing's refused-backfill report lands at warn where AGENTS.md puts it at error — and the card that was supposed to carry the level is CLOSED #13398-class ruling).
  • H3 — holds, verified independently with a positive control.git grep -n 'listSuspendedRunsDurable' -- 'packages/**' 'examples/**' 'apps/**' ':!*.test.ts' ':!*.spec.ts' returns only engine.ts's own definition and comments plus CHANGELOG prose — no consumer. Positive controls on the same command shape: hasSuspendedRun returns a real non-test consumer (packages/plugins/plugin-approvals/src/approval-service.ts:2704), and listSuspendedRuns returns its spec-contract declaration (packages/spec/src/contracts/automation-service.ts:589) — so the scan does find consumers where they exist.
  • H4 — holds. Clause-② is no. Declared from the actual diff: git diff -U0 origin/main...HEAD | grep export matches exactly one line, and it is prose inside the changeset (No signature, export or return-shape change...), not an export statement. No new exported symbol, no signature change on the three readers, no accept set moved. Changeset is patch. No needs:contract-review.
  • H5 — holds.service-automation: two concurrent resumes of one run on two replicas can both advance it — the idempotency guard is per-process #14333's concurrent-resume guard region is untouched: the diff in engine.ts is confined to cancelRun, failAncestors, listSuspendedRunsDurable and their docblocks.

Tests

New pins: packages/services/service-automation/src/multi-replica-suspended-run-readers.test.ts (11 tests), a sibling of the #13617 harness rather than more cases inside it. Reason recorded in its docblock: multi-replica-resume-staleness.test.ts carries a REVERT-PROOF ledger for one named mutation and its measured 4 red / 4 green split; adding these cases would falsify those counts and merge two different mutations' revert-proofs into one statement. The two files share only the two-engines-over-one-store shape.

Every site-1 and site-2 pin asserts on the onSuspensionReleasednotification (which node executor was told its pause is over), never on the row: the row delete is by id and is correct either way, which is the whole point of the card.

All runs below were taken at commit da4db89f5 and carry to the final head 89f5f8d because the two commits have a byte-identical tree: both are 514564fde37f75fd96c8f58e09ae139bc5ce0801, and git diff --stat da4db89f5 89f5f8d15 prints nothing. The extra commit exists only because this branch had already been pushed when a type-only import was corrected, and force-pushing is forbidden — so the amend was re-landed as a child commit rather than rewritten. Everything ran under scripts/pm/os-verify-lock.sh, exit captured before any pipe.

Full package suite — pnpm --filter @objectstack/service-automation test:

 Test Files 100 passed (100)
Tests 1182 passed (1182)
Duration 92.44s
os-verify-lock: VERDICT command-exit 0 · held the lock 92s

Type-check — the package has no typecheck script (it is a shrink-only DEBT ledger entry, frozen at 3), so a pnpm --filter ... typecheck would have matched zero scripts and exited 0 having measured nothing. Measured instead with the dependency closure built (pnpm --filter '@objectstack/service-automation^...' build, the world the ledger is measured in), tsc --noEmit -p tsconfig.json --listFiles:

src/nested-region-parity.test.ts(95,25): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(151,20): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(180,20): error TS2341: Property 'flows' is private ...
--- error count --- 3

Exactly the ledger's frozen 3, all pre-existing, none in the changed files; --listFiles confirms the new test file is in the program (this run is the reason a fourth error — a type imported from the wrong module — was caught and fixed before the head was final).

Repo-wide lint — no narrowing was needed, the whole population ran:

node --stack-size=4000 node_modules/eslint/bin/eslint.js . --no-inline-config
LINT EXIT=0 · os-verify-lock: VERDICT command-exit 0 · held the lock 116s

Gates

Derived on the final head from the merge-base change set, not a hand-written list: node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands at da4db89f5 (--repo asserted and held against this checkout's origin) — 36 commands over the 3 changed paths. The derivation is a property of the tree, which is byte-identical at the final head 89f5f8d15 (see Tests).

33 green, 0 red, 3 NOT MEASURED. The three are prerequisite refusals (exit code 3, which each script's own verdict text distinguishes from a finding's 1), not failures:

  • node scripts/check-test-completeness.mjsPREREQUISITE NOT MET — this gate grades a saved turbo run test log, and no log was named. Its own text says to record NOT MEASURED when the family is run locally.
  • pnpm check:dual-build-cjs-loadsPREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. (49 packages unbuilt; needs a full pnpm build.)
  • pnpm check:type-check-debtPREREQUISITE NOT MET ... --re-measure cannot run: 35 workspace dependenc(ies) ... have no built type entry point on disk. Its sibling check:type-check-coverage ran green (OK — 69/79 workspace packages type-checked ... 10 in the DEBT ledger), and the targeted tsc reading above answers this gate's substantive question for the one package this PR touches, in the built-closure world the ledger uses.

Also green outside the derived family: pnpm check:nul-bytes (OK (scanned 7992 text file(s) ... no raw ASCII control bytes)), plus a direct control-byte scan of both changed files.

Merge preflight before opening: git merge-tree --write-tree --name-only origin/main HEAD lists no conflicting paths — content/docs/permissions/system-context.mdx is not among them, so no regeneration is owed.

Ablation — three, one per site, on the committed tree

Each mutation restores that site's map-first read, is line-neutral, is proven on disk by anchored occurrence counts before the run, and is restored with git checkout HEAD -- ABSOLUTE_PATH inside a trap ... EXIT INT TERM, the restore proven by git hash-object against the HEAD blob and an empty git diff HEAD. Because the pins import ./engine.js relatively (vitest resolves that to src/engine.ts, not the package's dist/ through exports), no rebuild is interposed; the reds themselves are the proof the mutation reached the code under test.

HEAD blob for engine.ts = f2c33d8b57cea0dce4054de3b8f1ee0ac9b88e6a; every restore leg read back the same hash and an empty git diff HEAD.

MutationResultWhat went red
site 1 — run = this.suspendedRuns.get(runId) ?? await this.loadSuspendedRunStrict(runId)2 red / 9 greenTHE BUG: a stale replica must not tell the wrong node executor its pause is over (teardown names lv1 while the run is parked at lv2) and NEW REACH (a map hit means the unreadable store is never read, so the cancel proceeds from the snapshot)
site 2 — const parent = this.suspendedRuns.get(parentId) ?? await this.loadSuspendedRun(parentId)2 red / 9 greenTHE BUG: a stale parent must not be failed at a node it has already left and the unreadable-ancestor control, red for site 1's reason
site 3 — drop the byId.has(r.runId) guard so the map overwrites again1 red / 10 greenTHE BUG: the durable row wins an id collision, the stale in-process copy does not (the listing reports lv1, one level stale)

The controls that stay green under all three are the postures each site had before: the no-store cases where the map IS the authority, the failed-durable-save cases the store's silence says nothing about, and the unlistable store's documented short list.

One reading was voided rather than reported: the first attempt at the site-1 mutation used a one-line anchor that occurs twice in engine.ts (the other in resumeInternal), the guard refused at anchor count 2, and the run was re-anchored on a unique three-line block. A second attempt was voided by the on-disk proof itself before any test ran. Both voided attempts are named here because a silently re-run ablation is the defect one layer up.

Scope

Diff is 3 files. Untouched as required: packages/spec/**, content/docs/releases/**, skills/**, the store implementation, PR #14567's subflow delegation block, and the region #14333 is queued for. No out-of-scope findings were filed — nothing outside the card's own surface was found.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8

Generated by Claude Code


Generated by Claude Code

os-salesand others added 2 commits September 2, 2026 16:37
…ers through the shared loader
`cancelRun`, `failAncestors` and `listSuspendedRunsDurable` still preferred the
per-process `suspendedRuns` map over the shared `SuspendedRunStore`, which
#13617 had already made authoritative for the resume path. Each now takes one
answer to "where is this run parked", with the degrading or strict loader
chosen per site so the recorded #4632 / #6299 degradation posture is preserved
by choice rather than re-derived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
… module
`AutomationContext` is declared in `@objectstack/spec/contracts` and re-imported
by `engine.ts` as a type — it is not re-exported from there, so importing it
from `./engine.js` added a fourth error (TS2459) to a package whose type-check
debt ledger is frozen at 3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/service-automation, touching 4 documentable anchor(s).

2 release-owned page(s) name something this change touched. These are read-only:

  • content/docs/releases/v16.mdx(via AutomationEngine (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via AutomationEngine (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 5 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from da191b53496857088142d233a3b34da26653db98 — the merge of head 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin da191b53496857088142d233a3b34da26653db98 && git checkout da191b53496857088142d233a3b34da26653db98
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff 89f5f8d15dacac7b65cd1198ff48339d5bd36a18
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-automation: three more readers of suspended-run state still prefer the per-process map over the shared store

1 participant

@os-sales
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(service-automation): route the last three suspended-run readers through the shared loader - #14650

Merged
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative
Sep 2, 2026
Merged

fix(service-automation): route the last three suspended-run readers through the shared loader#14650
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative

Conversation

@claude

@claudeclaudeBot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Fixes#14332

cancelRun, failAncestors and listSuspendedRunsDurable were the three readers of suspended-run state that #13617 deliberately left on the per-process suspendedRuns map. All three now take one answer to "where is this run parked", through the existing loadSuspendedRun / loadSuspendedRunStrict pair — with the degrading or strict loader chosen per site, so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived.

Ruling of record

No maintainer ruling is owed on this card — triage graded it dispatchable and fixed its shape. Quoted verbatim from triage (14332#issuecomment-5503565404):

Lands in:packages/services/service-automation/src/engine.ts — the three readers (cancelRun, failAncestors, listSuspendedRunsDurable). Ruled shape is the card's own: one answer to "where is this run parked" — route all three through loadSuspendedRun / loadSuspendedRunStrict, choosing the degrading or strict loader deliberately per site so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived; the getSuspendedScreen docblock already states the principle. p2 because site 2 (failAncestors) carries #13617's exact harm shape — failing an ancestor at a node it has already left — and site 1 tears down the wrong node executor's pause. ⛔ Site 3's "in-memory entries win" comment must be corrected in the same PR, not left as a true-sounding sentence that is backwards once several replicas share one store; its own docblock records it has no in-repo production consumer, so it is the cheapest of the three to make honest.

The per-site loader judgment

This is the deliverable, not the mechanical replacement. Each site's docblock now carries the same judgment in prose.

SiteLoader chosenNot found meansStore unreadable meansWhy that is the posture it already had
1 cancelRunstrict (loadSuspendedRunStrict)returns false — "already terminal / unknown", its documented idempotent successthrows out of the loader into this site's own catch, which keeps its #4632 DURABILITY record at errorThe degrading loader would have answered null under its own best-effort warn and silently downgraded the #4632 verdict. Its degradation decides a WRITE, so the strict form plus a local catch is the only shape that preserves both the return value and the level. Same shape resumeInternal uses.
2 failAncestorsdegrading (loadSuspendedRun)ends the walk (if (!parent) return;), unchangedreads as "no ancestor here" and ends the walk — never propagatesThis walk runs inside the catch arm already handling a run's failure, so it must not throw. "Degrade to null and stop" is exactly what the bare .catch(() => null) did. One thing gained: that silent swallow is now recorded, at the loader's declared best-effort warn.
3 listSuspendedRunsDurableneither — merge direction plus the corrected commentn/a (a listing, not a per-id read)unchanged: degrades to the in-memory cache alone at warn (#4632 FUNCTIONAL)A LIST is not a per-id answer. store.list() is capped and best-effort (ObjectStoreSuspendedRunStore reads at most 1000 paused rows) and the same merge is reached on the degraded path, so "absent from the list" is not the "the store answered and has no row" that loadSuspendedRunStrict rests on. The durable row wins a collision; map entries the listing does not carry are still included, deliberately.

Site 3's comment In-memory entries win — they are the freshest copy is corrected in place, together with the docblock sentence that repeated it (The in-memory cache takes precedence on id collisions).

Premise checks — verified on origin/main at 13bf05d, before the first edit

All hold; premise_still_valid: true.

  • P1cancelRun read this.suspendedRuns.get(runId) ?? null and only then store.load — map-first. HOLDS.
  • P2failAncestors walked $parentRunId with this.suspendedRuns.get(parentId) ?? (this.store ? await this.store.load(parentId).catch(() => null) : null). HOLDS.
  • P3listSuspendedRunsDurable merged the durable list under the map with the comment In-memory entries win — they are the freshest copy. HOLDS.
  • P4 The automation/approvals: 多副本集群下审批流每级节点(除首级)被重复创建 —— approve 后恢复读到滞后一拍的流运行态,同级要批两次(单副本零重复) #13617 pair exists and is unreshaped: loadSuspendedRunStrict is store-authoritative with the cacheOnlySuspensions fallback for a failed durable save; loadSuspendedRun is its degrading form; the getSuspendedScreen docblock still states one loader, two callers. HOLDS.
  • P5 Reproduced before the fix and kept as pins — see Ablation below, which measures all three reds per site on the committed tree. All three sites were defective; no partially false premise.
  • P6packages/spec/** and content/docs/releases/** untouched. HOLDS — the diff is 3 files: engine.ts, one new test file, one changeset.

PM hypotheses

  • H1 — holds. All three sites route through the existing pair with no third loader, and the degrading-or-strict choice is a judgment from each site's own docblock. Stated per site in the table above and in the code.
  • H2 — holds, with two named consequences rather than silent changes. No log site moved, no level changed, no thrown or returned shape changed, no row write changed. The finding(service-automation): engine.ts 还剩三处同形的 warn message 拼接 —— forgetSuspendedRun / cancelRun / listSuspendedRunsDurable,是 #5912+#6230 之后该文件的最后一批 #6299 byte-level pins for all three seams (suspended-run-store-consume-log-cause.test.ts, 20 tests) stay green untouched. Two consequences are stated in the docblocks, the changeset and the pins, not left to be discovered: (a) cancelRun — while a store is configured this process's map is no longer an answer, so a store outage reaches the existing error record even for a run this replica holds, where the old cache-first read cancelled from the local snapshot; that snapshot is the defect. (b) failAncestors — a store failure during the ancestor walk was swallowed in total silence by .catch(() => null) and is now recorded by the loader at warn. No new error-level site anywhere (the [Decision] plugin-sharing's refused-backfill report lands at warn where AGENTS.md puts it at error — and the card that was supposed to carry the level is CLOSED #13398-class ruling).
  • H3 — holds, verified independently with a positive control.git grep -n 'listSuspendedRunsDurable' -- 'packages/**' 'examples/**' 'apps/**' ':!*.test.ts' ':!*.spec.ts' returns only engine.ts's own definition and comments plus CHANGELOG prose — no consumer. Positive controls on the same command shape: hasSuspendedRun returns a real non-test consumer (packages/plugins/plugin-approvals/src/approval-service.ts:2704), and listSuspendedRuns returns its spec-contract declaration (packages/spec/src/contracts/automation-service.ts:589) — so the scan does find consumers where they exist.
  • H4 — holds. Clause-② is no. Declared from the actual diff: git diff -U0 origin/main...HEAD | grep export matches exactly one line, and it is prose inside the changeset (No signature, export or return-shape change...), not an export statement. No new exported symbol, no signature change on the three readers, no accept set moved. Changeset is patch. No needs:contract-review.
  • H5 — holds.service-automation: two concurrent resumes of one run on two replicas can both advance it — the idempotency guard is per-process #14333's concurrent-resume guard region is untouched: the diff in engine.ts is confined to cancelRun, failAncestors, listSuspendedRunsDurable and their docblocks.

Tests

New pins: packages/services/service-automation/src/multi-replica-suspended-run-readers.test.ts (11 tests), a sibling of the #13617 harness rather than more cases inside it. Reason recorded in its docblock: multi-replica-resume-staleness.test.ts carries a REVERT-PROOF ledger for one named mutation and its measured 4 red / 4 green split; adding these cases would falsify those counts and merge two different mutations' revert-proofs into one statement. The two files share only the two-engines-over-one-store shape.

Every site-1 and site-2 pin asserts on the onSuspensionReleasednotification (which node executor was told its pause is over), never on the row: the row delete is by id and is correct either way, which is the whole point of the card.

All runs below were taken at commit da4db89f5 and carry to the final head 89f5f8d because the two commits have a byte-identical tree: both are 514564fde37f75fd96c8f58e09ae139bc5ce0801, and git diff --stat da4db89f5 89f5f8d15 prints nothing. The extra commit exists only because this branch had already been pushed when a type-only import was corrected, and force-pushing is forbidden — so the amend was re-landed as a child commit rather than rewritten. Everything ran under scripts/pm/os-verify-lock.sh, exit captured before any pipe.

Full package suite — pnpm --filter @objectstack/service-automation test:

 Test Files 100 passed (100)
Tests 1182 passed (1182)
Duration 92.44s
os-verify-lock: VERDICT command-exit 0 · held the lock 92s

Type-check — the package has no typecheck script (it is a shrink-only DEBT ledger entry, frozen at 3), so a pnpm --filter ... typecheck would have matched zero scripts and exited 0 having measured nothing. Measured instead with the dependency closure built (pnpm --filter '@objectstack/service-automation^...' build, the world the ledger is measured in), tsc --noEmit -p tsconfig.json --listFiles:

src/nested-region-parity.test.ts(95,25): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(151,20): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(180,20): error TS2341: Property 'flows' is private ...
--- error count --- 3

Exactly the ledger's frozen 3, all pre-existing, none in the changed files; --listFiles confirms the new test file is in the program (this run is the reason a fourth error — a type imported from the wrong module — was caught and fixed before the head was final).

Repo-wide lint — no narrowing was needed, the whole population ran:

node --stack-size=4000 node_modules/eslint/bin/eslint.js . --no-inline-config
LINT EXIT=0 · os-verify-lock: VERDICT command-exit 0 · held the lock 116s

Gates

Derived on the final head from the merge-base change set, not a hand-written list: node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands at da4db89f5 (--repo asserted and held against this checkout's origin) — 36 commands over the 3 changed paths. The derivation is a property of the tree, which is byte-identical at the final head 89f5f8d15 (see Tests).

33 green, 0 red, 3 NOT MEASURED. The three are prerequisite refusals (exit code 3, which each script's own verdict text distinguishes from a finding's 1), not failures:

  • node scripts/check-test-completeness.mjsPREREQUISITE NOT MET — this gate grades a saved turbo run test log, and no log was named. Its own text says to record NOT MEASURED when the family is run locally.
  • pnpm check:dual-build-cjs-loadsPREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. (49 packages unbuilt; needs a full pnpm build.)
  • pnpm check:type-check-debtPREREQUISITE NOT MET ... --re-measure cannot run: 35 workspace dependenc(ies) ... have no built type entry point on disk. Its sibling check:type-check-coverage ran green (OK — 69/79 workspace packages type-checked ... 10 in the DEBT ledger), and the targeted tsc reading above answers this gate's substantive question for the one package this PR touches, in the built-closure world the ledger uses.

Also green outside the derived family: pnpm check:nul-bytes (OK (scanned 7992 text file(s) ... no raw ASCII control bytes)), plus a direct control-byte scan of both changed files.

Merge preflight before opening: git merge-tree --write-tree --name-only origin/main HEAD lists no conflicting paths — content/docs/permissions/system-context.mdx is not among them, so no regeneration is owed.

Ablation — three, one per site, on the committed tree

Each mutation restores that site's map-first read, is line-neutral, is proven on disk by anchored occurrence counts before the run, and is restored with git checkout HEAD -- ABSOLUTE_PATH inside a trap ... EXIT INT TERM, the restore proven by git hash-object against the HEAD blob and an empty git diff HEAD. Because the pins import ./engine.js relatively (vitest resolves that to src/engine.ts, not the package's dist/ through exports), no rebuild is interposed; the reds themselves are the proof the mutation reached the code under test.

HEAD blob for engine.ts = f2c33d8b57cea0dce4054de3b8f1ee0ac9b88e6a; every restore leg read back the same hash and an empty git diff HEAD.

MutationResultWhat went red
site 1 — run = this.suspendedRuns.get(runId) ?? await this.loadSuspendedRunStrict(runId)2 red / 9 greenTHE BUG: a stale replica must not tell the wrong node executor its pause is over (teardown names lv1 while the run is parked at lv2) and NEW REACH (a map hit means the unreadable store is never read, so the cancel proceeds from the snapshot)
site 2 — const parent = this.suspendedRuns.get(parentId) ?? await this.loadSuspendedRun(parentId)2 red / 9 greenTHE BUG: a stale parent must not be failed at a node it has already left and the unreadable-ancestor control, red for site 1's reason
site 3 — drop the byId.has(r.runId) guard so the map overwrites again1 red / 10 greenTHE BUG: the durable row wins an id collision, the stale in-process copy does not (the listing reports lv1, one level stale)

The controls that stay green under all three are the postures each site had before: the no-store cases where the map IS the authority, the failed-durable-save cases the store's silence says nothing about, and the unlistable store's documented short list.

One reading was voided rather than reported: the first attempt at the site-1 mutation used a one-line anchor that occurs twice in engine.ts (the other in resumeInternal), the guard refused at anchor count 2, and the run was re-anchored on a unique three-line block. A second attempt was voided by the on-disk proof itself before any test ran. Both voided attempts are named here because a silently re-run ablation is the defect one layer up.

Scope

Diff is 3 files. Untouched as required: packages/spec/**, content/docs/releases/**, skills/**, the store implementation, PR #14567's subflow delegation block, and the region #14333 is queued for. No out-of-scope findings were filed — nothing outside the card's own surface was found.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8

Generated by Claude Code


Generated by Claude Code

os-salesand others added 2 commits September 2, 2026 16:37
…ers through the shared loader
`cancelRun`, `failAncestors` and `listSuspendedRunsDurable` still preferred the
per-process `suspendedRuns` map over the shared `SuspendedRunStore`, which
#13617 had already made authoritative for the resume path. Each now takes one
answer to "where is this run parked", with the degrading or strict loader
chosen per site so the recorded #4632 / #6299 degradation posture is preserved
by choice rather than re-derived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
… module
`AutomationContext` is declared in `@objectstack/spec/contracts` and re-imported
by `engine.ts` as a type — it is not re-exported from there, so importing it
from `./engine.js` added a fourth error (TS2459) to a package whose type-check
debt ledger is frozen at 3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/service-automation, touching 4 documentable anchor(s).

2 release-owned page(s) name something this change touched. These are read-only:

  • content/docs/releases/v16.mdx(via AutomationEngine (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via AutomationEngine (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 5 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from da191b53496857088142d233a3b34da26653db98 — the merge of head 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin da191b53496857088142d233a3b34da26653db98 && git checkout da191b53496857088142d233a3b34da26653db98
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff 89f5f8d15dacac7b65cd1198ff48339d5bd36a18
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-automation: three more readers of suspended-run state still prefer the per-process map over the shared store

1 participant

@os-sales
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

fix(service-automation): route the last three suspended-run readers through the shared loader - #14650

Merged
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative
Sep 2, 2026
Merged

fix(service-automation): route the last three suspended-run readers through the shared loader#14650
os-sales merged 2 commits into
mainfrom
claude/issue-14332-suspended-run-readers-store-authoritative

Conversation

@claude

@claudeclaudeBot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Fixes#14332

cancelRun, failAncestors and listSuspendedRunsDurable were the three readers of suspended-run state that #13617 deliberately left on the per-process suspendedRuns map. All three now take one answer to "where is this run parked", through the existing loadSuspendedRun / loadSuspendedRunStrict pair — with the degrading or strict loader chosen per site, so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived.

Ruling of record

No maintainer ruling is owed on this card — triage graded it dispatchable and fixed its shape. Quoted verbatim from triage (14332#issuecomment-5503565404):

Lands in:packages/services/service-automation/src/engine.ts — the three readers (cancelRun, failAncestors, listSuspendedRunsDurable). Ruled shape is the card's own: one answer to "where is this run parked" — route all three through loadSuspendedRun / loadSuspendedRunStrict, choosing the degrading or strict loader deliberately per site so each site's recorded #4632 / #6299 degradation posture is preserved by choice rather than re-derived; the getSuspendedScreen docblock already states the principle. p2 because site 2 (failAncestors) carries #13617's exact harm shape — failing an ancestor at a node it has already left — and site 1 tears down the wrong node executor's pause. ⛔ Site 3's "in-memory entries win" comment must be corrected in the same PR, not left as a true-sounding sentence that is backwards once several replicas share one store; its own docblock records it has no in-repo production consumer, so it is the cheapest of the three to make honest.

The per-site loader judgment

This is the deliverable, not the mechanical replacement. Each site's docblock now carries the same judgment in prose.

SiteLoader chosenNot found meansStore unreadable meansWhy that is the posture it already had
1 cancelRunstrict (loadSuspendedRunStrict)returns false — "already terminal / unknown", its documented idempotent successthrows out of the loader into this site's own catch, which keeps its #4632 DURABILITY record at errorThe degrading loader would have answered null under its own best-effort warn and silently downgraded the #4632 verdict. Its degradation decides a WRITE, so the strict form plus a local catch is the only shape that preserves both the return value and the level. Same shape resumeInternal uses.
2 failAncestorsdegrading (loadSuspendedRun)ends the walk (if (!parent) return;), unchangedreads as "no ancestor here" and ends the walk — never propagatesThis walk runs inside the catch arm already handling a run's failure, so it must not throw. "Degrade to null and stop" is exactly what the bare .catch(() => null) did. One thing gained: that silent swallow is now recorded, at the loader's declared best-effort warn.
3 listSuspendedRunsDurableneither — merge direction plus the corrected commentn/a (a listing, not a per-id read)unchanged: degrades to the in-memory cache alone at warn (#4632 FUNCTIONAL)A LIST is not a per-id answer. store.list() is capped and best-effort (ObjectStoreSuspendedRunStore reads at most 1000 paused rows) and the same merge is reached on the degraded path, so "absent from the list" is not the "the store answered and has no row" that loadSuspendedRunStrict rests on. The durable row wins a collision; map entries the listing does not carry are still included, deliberately.

Site 3's comment In-memory entries win — they are the freshest copy is corrected in place, together with the docblock sentence that repeated it (The in-memory cache takes precedence on id collisions).

Premise checks — verified on origin/main at 13bf05d, before the first edit

All hold; premise_still_valid: true.

  • P1cancelRun read this.suspendedRuns.get(runId) ?? null and only then store.load — map-first. HOLDS.
  • P2failAncestors walked $parentRunId with this.suspendedRuns.get(parentId) ?? (this.store ? await this.store.load(parentId).catch(() => null) : null). HOLDS.
  • P3listSuspendedRunsDurable merged the durable list under the map with the comment In-memory entries win — they are the freshest copy. HOLDS.
  • P4 The automation/approvals: 多副本集群下审批流每级节点(除首级)被重复创建 —— approve 后恢复读到滞后一拍的流运行态,同级要批两次(单副本零重复) #13617 pair exists and is unreshaped: loadSuspendedRunStrict is store-authoritative with the cacheOnlySuspensions fallback for a failed durable save; loadSuspendedRun is its degrading form; the getSuspendedScreen docblock still states one loader, two callers. HOLDS.
  • P5 Reproduced before the fix and kept as pins — see Ablation below, which measures all three reds per site on the committed tree. All three sites were defective; no partially false premise.
  • P6packages/spec/** and content/docs/releases/** untouched. HOLDS — the diff is 3 files: engine.ts, one new test file, one changeset.

PM hypotheses

  • H1 — holds. All three sites route through the existing pair with no third loader, and the degrading-or-strict choice is a judgment from each site's own docblock. Stated per site in the table above and in the code.
  • H2 — holds, with two named consequences rather than silent changes. No log site moved, no level changed, no thrown or returned shape changed, no row write changed. The finding(service-automation): engine.ts 还剩三处同形的 warn message 拼接 —— forgetSuspendedRun / cancelRun / listSuspendedRunsDurable,是 #5912+#6230 之后该文件的最后一批 #6299 byte-level pins for all three seams (suspended-run-store-consume-log-cause.test.ts, 20 tests) stay green untouched. Two consequences are stated in the docblocks, the changeset and the pins, not left to be discovered: (a) cancelRun — while a store is configured this process's map is no longer an answer, so a store outage reaches the existing error record even for a run this replica holds, where the old cache-first read cancelled from the local snapshot; that snapshot is the defect. (b) failAncestors — a store failure during the ancestor walk was swallowed in total silence by .catch(() => null) and is now recorded by the loader at warn. No new error-level site anywhere (the [Decision] plugin-sharing's refused-backfill report lands at warn where AGENTS.md puts it at error — and the card that was supposed to carry the level is CLOSED #13398-class ruling).
  • H3 — holds, verified independently with a positive control.git grep -n 'listSuspendedRunsDurable' -- 'packages/**' 'examples/**' 'apps/**' ':!*.test.ts' ':!*.spec.ts' returns only engine.ts's own definition and comments plus CHANGELOG prose — no consumer. Positive controls on the same command shape: hasSuspendedRun returns a real non-test consumer (packages/plugins/plugin-approvals/src/approval-service.ts:2704), and listSuspendedRuns returns its spec-contract declaration (packages/spec/src/contracts/automation-service.ts:589) — so the scan does find consumers where they exist.
  • H4 — holds. Clause-② is no. Declared from the actual diff: git diff -U0 origin/main...HEAD | grep export matches exactly one line, and it is prose inside the changeset (No signature, export or return-shape change...), not an export statement. No new exported symbol, no signature change on the three readers, no accept set moved. Changeset is patch. No needs:contract-review.
  • H5 — holds.service-automation: two concurrent resumes of one run on two replicas can both advance it — the idempotency guard is per-process #14333's concurrent-resume guard region is untouched: the diff in engine.ts is confined to cancelRun, failAncestors, listSuspendedRunsDurable and their docblocks.

Tests

New pins: packages/services/service-automation/src/multi-replica-suspended-run-readers.test.ts (11 tests), a sibling of the #13617 harness rather than more cases inside it. Reason recorded in its docblock: multi-replica-resume-staleness.test.ts carries a REVERT-PROOF ledger for one named mutation and its measured 4 red / 4 green split; adding these cases would falsify those counts and merge two different mutations' revert-proofs into one statement. The two files share only the two-engines-over-one-store shape.

Every site-1 and site-2 pin asserts on the onSuspensionReleasednotification (which node executor was told its pause is over), never on the row: the row delete is by id and is correct either way, which is the whole point of the card.

All runs below were taken at commit da4db89f5 and carry to the final head 89f5f8d because the two commits have a byte-identical tree: both are 514564fde37f75fd96c8f58e09ae139bc5ce0801, and git diff --stat da4db89f5 89f5f8d15 prints nothing. The extra commit exists only because this branch had already been pushed when a type-only import was corrected, and force-pushing is forbidden — so the amend was re-landed as a child commit rather than rewritten. Everything ran under scripts/pm/os-verify-lock.sh, exit captured before any pipe.

Full package suite — pnpm --filter @objectstack/service-automation test:

 Test Files 100 passed (100)
Tests 1182 passed (1182)
Duration 92.44s
os-verify-lock: VERDICT command-exit 0 · held the lock 92s

Type-check — the package has no typecheck script (it is a shrink-only DEBT ledger entry, frozen at 3), so a pnpm --filter ... typecheck would have matched zero scripts and exited 0 having measured nothing. Measured instead with the dependency closure built (pnpm --filter '@objectstack/service-automation^...' build, the world the ledger is measured in), tsc --noEmit -p tsconfig.json --listFiles:

src/nested-region-parity.test.ts(95,25): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(151,20): error TS2341: Property 'flows' is private ...
src/nested-region-parity.test.ts(180,20): error TS2341: Property 'flows' is private ...
--- error count --- 3

Exactly the ledger's frozen 3, all pre-existing, none in the changed files; --listFiles confirms the new test file is in the program (this run is the reason a fourth error — a type imported from the wrong module — was caught and fixed before the head was final).

Repo-wide lint — no narrowing was needed, the whole population ran:

node --stack-size=4000 node_modules/eslint/bin/eslint.js . --no-inline-config
LINT EXIT=0 · os-verify-lock: VERDICT command-exit 0 · held the lock 116s

Gates

Derived on the final head from the merge-base change set, not a hand-written list: node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands at da4db89f5 (--repo asserted and held against this checkout's origin) — 36 commands over the 3 changed paths. The derivation is a property of the tree, which is byte-identical at the final head 89f5f8d15 (see Tests).

33 green, 0 red, 3 NOT MEASURED. The three are prerequisite refusals (exit code 3, which each script's own verdict text distinguishes from a finding's 1), not failures:

  • node scripts/check-test-completeness.mjsPREREQUISITE NOT MET — this gate grades a saved turbo run test log, and no log was named. Its own text says to record NOT MEASURED when the family is run locally.
  • pnpm check:dual-build-cjs-loadsPREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. (49 packages unbuilt; needs a full pnpm build.)
  • pnpm check:type-check-debtPREREQUISITE NOT MET ... --re-measure cannot run: 35 workspace dependenc(ies) ... have no built type entry point on disk. Its sibling check:type-check-coverage ran green (OK — 69/79 workspace packages type-checked ... 10 in the DEBT ledger), and the targeted tsc reading above answers this gate's substantive question for the one package this PR touches, in the built-closure world the ledger uses.

Also green outside the derived family: pnpm check:nul-bytes (OK (scanned 7992 text file(s) ... no raw ASCII control bytes)), plus a direct control-byte scan of both changed files.

Merge preflight before opening: git merge-tree --write-tree --name-only origin/main HEAD lists no conflicting paths — content/docs/permissions/system-context.mdx is not among them, so no regeneration is owed.

Ablation — three, one per site, on the committed tree

Each mutation restores that site's map-first read, is line-neutral, is proven on disk by anchored occurrence counts before the run, and is restored with git checkout HEAD -- ABSOLUTE_PATH inside a trap ... EXIT INT TERM, the restore proven by git hash-object against the HEAD blob and an empty git diff HEAD. Because the pins import ./engine.js relatively (vitest resolves that to src/engine.ts, not the package's dist/ through exports), no rebuild is interposed; the reds themselves are the proof the mutation reached the code under test.

HEAD blob for engine.ts = f2c33d8b57cea0dce4054de3b8f1ee0ac9b88e6a; every restore leg read back the same hash and an empty git diff HEAD.

MutationResultWhat went red
site 1 — run = this.suspendedRuns.get(runId) ?? await this.loadSuspendedRunStrict(runId)2 red / 9 greenTHE BUG: a stale replica must not tell the wrong node executor its pause is over (teardown names lv1 while the run is parked at lv2) and NEW REACH (a map hit means the unreadable store is never read, so the cancel proceeds from the snapshot)
site 2 — const parent = this.suspendedRuns.get(parentId) ?? await this.loadSuspendedRun(parentId)2 red / 9 greenTHE BUG: a stale parent must not be failed at a node it has already left and the unreadable-ancestor control, red for site 1's reason
site 3 — drop the byId.has(r.runId) guard so the map overwrites again1 red / 10 greenTHE BUG: the durable row wins an id collision, the stale in-process copy does not (the listing reports lv1, one level stale)

The controls that stay green under all three are the postures each site had before: the no-store cases where the map IS the authority, the failed-durable-save cases the store's silence says nothing about, and the unlistable store's documented short list.

One reading was voided rather than reported: the first attempt at the site-1 mutation used a one-line anchor that occurs twice in engine.ts (the other in resumeInternal), the guard refused at anchor count 2, and the run was re-anchored on a unique three-line block. A second attempt was voided by the on-disk proof itself before any test ran. Both voided attempts are named here because a silently re-run ablation is the defect one layer up.

Scope

Diff is 3 files. Untouched as required: packages/spec/**, content/docs/releases/**, skills/**, the store implementation, PR #14567's subflow delegation block, and the region #14333 is queued for. No out-of-scope findings were filed — nothing outside the card's own surface was found.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8

Generated by Claude Code


Generated by Claude Code

os-salesand others added 2 commits September 2, 2026 16:37
…ers through the shared loader
`cancelRun`, `failAncestors` and `listSuspendedRunsDurable` still preferred the
per-process `suspendedRuns` map over the shared `SuspendedRunStore`, which
#13617 had already made authoritative for the resume path. Each now takes one
answer to "where is this run parked", with the degrading or strict loader
chosen per site so the recorded #4632 / #6299 degradation posture is preserved
by choice rather than re-derived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
… module
`AutomationContext` is declared in `@objectstack/spec/contracts` and re-imported
by `engine.ts` as a type — it is not re-exported from there, so importing it
from `./engine.js` added a fourth error (TS2459) to a package whose type-check
debt ledger is frozen at 3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/service-automation, touching 4 documentable anchor(s).

2 release-owned page(s) name something this change touched. These are read-only:

  • content/docs/releases/v16.mdx(via AutomationEngine (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via AutomationEngine (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 5 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from da191b53496857088142d233a3b34da26653db98 — the merge of head 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin da191b53496857088142d233a3b34da26653db98 && git checkout da191b53496857088142d233a3b34da26653db98
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f 89f5f8d15dacac7b65cd1198ff48339d5bd36a18 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff 89f5f8d15dacac7b65cd1198ff48339d5bd36a18
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-automation: three more readers of suspended-run state still prefer the per-process map over the shared store

1 participant

@os-sales