fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run#14712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
os-project-manager merged 9 commits into from
Sep 3, 2026
+1,332 −41
Merged

fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run #14712

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
d9282c3
fix(service-automation): conditional advance claim on SuspendedRunSto…
claude Sep 2, 2026
5daab8d
test(service-automation): record the measured ablation ledger (#14333)
claude Sep 2, 2026
97ac4af
docs(permissions): re-certify the tenant-audit census at 218 write ca…
claude Sep 2, 2026
6f41e20
Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…
claude Sep 2, 2026
2b36945
test(service-automation): pin the compare-and-set condition and the p…
claude Sep 2, 2026
e3bf5cd
test(service-automation): record the three measured ablation ledgers …
claude Sep 2, 2026
74595e4
Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…
claude Sep 2, 2026
ad24042
chore(gates): record the newly pinned engine double and close its DEB…
claude Sep 2, 2026
feb213d
Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…
claude Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions .changeset/concurrent-replica-resume-advance-claim.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
---
"@objectstack/service-automation": minor
---

fix(service-automation): make the resume advance a conditional claim on the suspended-run store, so two replicas cannot both advance one run (#14333)

`AutomationEngine.resumeInternal` guarded a duplicate resume with `this.resuming`,
an in-process `Set`. That is a complete guard for exactly one deployment shape: a
single process. Behind a load balancer, two decisions on one run arriving in the
same instant on two replicas each passed their **own** `resuming` check, both read
the same fresh row out of the shared store, both consumed it, and both traversed
forward — so every downstream side effect ran twice. #13617 closed the sequential
half of this family (a replica resuming from a snapshot it had gone stale on); it
deliberately did not close the concurrent one.

Measured before the fix on the two-engines-over-one-shared-store harness, at
`packages/services/service-automation/src/concurrent-replica-resume-race.test.ts`:
**25 of 25** raced runs advanced twice — one action fired twice and one approval
level opened twice per run — for both reachable shapes the report named (parallel /
any-of approvers, and duplicated automated approve calls). A single approver per
level deciding sequentially does **not** race, and is pinned as the negative
control.

`SuspendedRunStore` therefore gains `claimSuspension(runId, parkedAt)`: consume the
durable record **only** if it is still parked at the node the caller read (and, when
the caller has one, still carrying that correlation), atomically, answering
`'claimed'` / `'lost'` / `'unsupported'`. The winner advances; the loser is refused
`RESUME_IN_PROGRESS` — the existing code, because the remedy is identical to the
in-process refusal's and `plugin-approvals` already branches on it that way — and
runs nothing. The per-process `resuming` set stays as the cheap first gate; it is
not replaced, and the single-replica path is unchanged.

Both shipped stores implement it: `InMemorySuspendedRunStore` tests and removes
with no `await` between the two, and `ObjectStoreSuspendedRunStore` issues one
`DELETE … WHERE id = ? AND node_id = ?` through the data engine's documented
compare-and-set route (`multi: true` with a full `where`), reading the affected-row
count. ⛔ No platform-object schema change: `node_id` and `correlation` are columns
`sys_automation_run` already carried.

The member is **optional**, so no existing implementation is broken, and its absence
is a declared degradation rather than a silent one: an engine whose store cannot
express the condition says once, at `warn`, that resume idempotency is in-process
only and what that costs — the same posture `AutomationEngine.claim()` already takes
when no persisted flow-dispatch ledger is attached.
34 changes: 17 additions & 17 deletions content/docs/permissions/tenant-audit-census.mdx
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -98,7 +98,7 @@ are reported as `undecidable` rather than assumed either way.

The same holds twice over for the context. An options argument spelled as a
literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
forwarding shim cannot, and **67 of the 217 sites are spelled that way**. A
forwarding shim cannot, and **67 of the 218 sites are spelled that way**. A
context resolved from an inline literal or a local `const` can be tested for
`isSystem`; one arriving from a helper call cannot.

Expand DownExpand Up@@ -147,10 +147,10 @@ reproduce them. Where it disagrees, it disagrees on the page:

| carried figure | where it survives | this census |
| :--- | :--- | ---: |
| 175 write call sites | quoted in the merged changeset | **217** |
| 175 write call sites | quoted in the merged changeset | **218** |
| 24 carrying no tenant context | quoted in the merged changeset | **9** provable and tenancy-enabled; **32** more whose options argument is unreadable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **145 of 217** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 99 decidably elevated, 0 decidably not, 101 undecidable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **146 of 218** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 100 decidably elevated, 0 decidably not, 101 undecidable |
| 141 and 132, two independent re-derivations | the card that filed this work | — |

**The differences are not reconciled, and deliberately so.** The old census's
Expand All@@ -161,17 +161,17 @@ at any commit.

Two structural facts do plausibly widen this reading against any hand or regex
one, and both are counted in the generated tables below: the 45 sites reached
through an erased (`any`) receiver, and the 37 that name their object through a
through an erased (`any`) receiver, and the 38 that name their object through a
`const` rather than inline. An instrument that read either the way a person does
would report a smaller number and would not say so.

The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
figure has no surviving corroboration anywhere in the tree.** This census reads
99 of 217 (45%) as decidably elevated, with 101 more whose elevation is a
100 of 218 (46%) as decidably elevated, with 101 more whose elevation is a
run-time fact — so the claim is neither confirmed nor refuted, and the honest
answer is that a static reading cannot settle it.

⇒ **Cite `9 / 217`, and say what it is**: the sites whose options argument was
⇒ **Cite `9 / 218`, and say what it is**: the sites whose options argument was
READ and holds no tenant context, against a decidably tenancy-enabled object.
That is the control's provable yield surface. ⛔ Do not cite it as "the sites
without tenant context" — **32 further sites** have an options argument this
Expand All@@ -183,29 +183,29 @@ cannot read, and they are neither in nor out.

| what | count |
| :--- | ---: |
| write call sites on the application surface | **217** |
| …whose object name is statically decidable | 145 |
| write call sites on the application surface | **218** |
| …whose object name is statically decidable | 146 |
| …whose object name is chosen at run time | 72 |
| …against an object with tenancy ENABLED | 145 |
| …against an object with tenancy ENABLED | 146 |
| …against an object that declares tenancy off | 0 |
| threading a tenant context | 133 |
| threading a tenant context | 134 |
| PROVABLY carrying none (options read, no context key) | **17** |
| …of those, against a decidably tenancy-enabled object | **9** |
| options argument UNREADABLE — may or may not carry one | 67 |
| …of those, against a decidably tenancy-enabled object | 32 |
| threading a decidably ELEVATED (`isSystem`) context | 99 |
| threading a decidably ELEVATED (`isSystem`) context | 100 |
| threading a context that is decidably NOT elevated | 0 |
| threading a context whose elevation is a run-time fact | 101 |

| how the instrument reached the site | count |
| :--- | ---: |
| receiver carried a readable engine type | 172 |
| receiver carried a readable engine type | 173 |
| receiver erased, placed by the object NAME | 19 |
| receiver erased, placed by an `object: string` PARAMETER | 15 |
| receiver erased, placed by an `UNTYPED_RECEIVERS` row | 11 |

| object name spelled inline | 108 |
| object name spelled through a `const` | 37 |
| object name spelled through a `const` | 38 |
| object name is an `object: string` parameter | 19 |
| object name is some other run-time expression | 53 |

Expand All@@ -224,13 +224,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

{/* END GENERATED: tenant-audit-census */}
18 changes: 9 additions & 9 deletions docs/audits/2026-08-tenant-audit-write-call-sites.counts.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -29,17 +29,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.

| Measure | Value |
|---|---:|
| Write call sites | 217 |
| Object name statically decidable | 145 |
| Write call sites | 218 |
| Object name statically decidable | 146 |
| Object name chosen at run time | 72 |
| Against a tenancy-enabled object | 145 |
| Against a tenancy-enabled object | 146 |
| Against an object declaring tenancy off | 0 |
| Threading a tenant context | 133 |
| Threading a tenant context | 134 |
| Provably carrying none | 17 |
| …and decidably tenancy-enabled | 9 |
| Options argument unreadable | 67 |
| …and decidably tenancy-enabled | 32 |
| Threading a decidably elevated context | 99 |
| Threading a decidably elevated context | 100 |
| Threading a decidably non-elevated context | 0 |
| Threading a context of undecidable elevation | 101 |

Expand All@@ -52,14 +52,14 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

## Every site

Expand DownExpand Up@@ -158,7 +158,7 @@ Measured on 2026-09-01 at `d3ebf3b55`.
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `update` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/flow-dispatch-store.ts` | `insert` | `sys_flow_dispatch` | enabled | elevated | 1 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 3 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `insert` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `update` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-datasource/src/datasource-admin-plugin.ts` | `delete` | `sys_metadata` | enabled | PROVABLY NONE | 1 |
Expand Down
Loading
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run#14712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-project-manager
merged 9 commits into

from

Sep 3, 2026

+1,332

−41

 Merged


fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run
#14712
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
d9282c3

fix(service-automation): conditional advance claim on SuspendedRunSto…


claude Sep 2, 2026
5daab8d

test(service-automation): record the measured ablation ledger (#14333)


claude Sep 2, 2026
97ac4af

docs(permissions): re-certify the tenant-audit census at 218 write ca…


claude Sep 2, 2026
6f41e20

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
2b36945

test(service-automation): pin the compare-and-set condition and the p…


claude Sep 2, 2026
e3bf5cd

test(service-automation): record the three measured ablation ledgers …


claude Sep 2, 2026
74595e4

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
ad24042

chore(gates): record the newly pinned engine double and close its DEB…


claude Sep 2, 2026
feb213d

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


44 changes: 44 additions & 0 deletions
.changeset/concurrent-replica-resume-advance-claim.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
---
"@objectstack/service-automation": minor
---

fix(service-automation): make the resume advance a conditional claim on the suspended-run store, so two replicas cannot both advance one run (#14333)

`AutomationEngine.resumeInternal` guarded a duplicate resume with `this.resuming`,
an in-process `Set`. That is a complete guard for exactly one deployment shape: a
single process. Behind a load balancer, two decisions on one run arriving in the
same instant on two replicas each passed their **own** `resuming` check, both read
the same fresh row out of the shared store, both consumed it, and both traversed
forward — so every downstream side effect ran twice. #13617 closed the sequential
half of this family (a replica resuming from a snapshot it had gone stale on); it
deliberately did not close the concurrent one.

Measured before the fix on the two-engines-over-one-shared-store harness, at
`packages/services/service-automation/src/concurrent-replica-resume-race.test.ts`:
**25 of 25** raced runs advanced twice — one action fired twice and one approval
level opened twice per run — for both reachable shapes the report named (parallel /
any-of approvers, and duplicated automated approve calls). A single approver per
level deciding sequentially does **not** race, and is pinned as the negative
control.

`SuspendedRunStore` therefore gains `claimSuspension(runId, parkedAt)`: consume the
durable record **only** if it is still parked at the node the caller read (and, when
the caller has one, still carrying that correlation), atomically, answering
`'claimed'` / `'lost'` / `'unsupported'`. The winner advances; the loser is refused
`RESUME_IN_PROGRESS` — the existing code, because the remedy is identical to the
in-process refusal's and `plugin-approvals` already branches on it that way — and
runs nothing. The per-process `resuming` set stays as the cheap first gate; it is
not replaced, and the single-replica path is unchanged.

Both shipped stores implement it: `InMemorySuspendedRunStore` tests and removes
with no `await` between the two, and `ObjectStoreSuspendedRunStore` issues one
`DELETE … WHERE id = ? AND node_id = ?` through the data engine's documented
compare-and-set route (`multi: true` with a full `where`), reading the affected-row
count. ⛔ No platform-object schema change: `node_id` and `correlation` are columns
`sys_automation_run` already carried.

The member is **optional**, so no existing implementation is broken, and its absence
is a declared degradation rather than a silent one: an engine whose store cannot
express the condition says once, at `warn`, that resume idempotency is in-process
only and what that costs — the same posture `AutomationEngine.claim()` already takes
when no persisted flow-dispatch ledger is attached.

34 changes: 17 additions & 17 deletions
content/docs/permissions/tenant-audit-census.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -98,7 +98,7 @@ are reported as `undecidable` rather than assumed either way.

The same holds twice over for the context. An options argument spelled as a
literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
forwarding shim cannot, and **67 of the 217 sites are spelled that way**. A
forwarding shim cannot, and **67 of the 218 sites are spelled that way**. A
context resolved from an inline literal or a local `const` can be tested for
`isSystem`; one arriving from a helper call cannot.

Expand DownExpand Up@@ -147,10 +147,10 @@ reproduce them. Where it disagrees, it disagrees on the page:

| carried figure | where it survives | this census |
| :--- | :--- | ---: |
| 175 write call sites | quoted in the merged changeset | **217** |
| 175 write call sites | quoted in the merged changeset | **218** |
| 24 carrying no tenant context | quoted in the merged changeset | **9** provable and tenancy-enabled; **32** more whose options argument is unreadable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **145 of 217** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 99 decidably elevated, 0 decidably not, 101 undecidable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **146 of 218** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 100 decidably elevated, 0 decidably not, 101 undecidable |
| 141 and 132, two independent re-derivations | the card that filed this work | — |

**The differences are not reconciled, and deliberately so.** The old census's
Expand All@@ -161,17 +161,17 @@ at any commit.

Two structural facts do plausibly widen this reading against any hand or regex
one, and both are counted in the generated tables below: the 45 sites reached
through an erased (`any`) receiver, and the 37 that name their object through a
through an erased (`any`) receiver, and the 38 that name their object through a
`const` rather than inline. An instrument that read either the way a person does
would report a smaller number and would not say so.

The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
figure has no surviving corroboration anywhere in the tree.** This census reads
99 of 217 (45%) as decidably elevated, with 101 more whose elevation is a
100 of 218 (46%) as decidably elevated, with 101 more whose elevation is a
run-time fact — so the claim is neither confirmed nor refuted, and the honest
answer is that a static reading cannot settle it.

⇒ **Cite `9 / 217`, and say what it is**: the sites whose options argument was
⇒ **Cite `9 / 218`, and say what it is**: the sites whose options argument was
READ and holds no tenant context, against a decidably tenancy-enabled object.
That is the control's provable yield surface. ⛔ Do not cite it as "the sites
without tenant context" — **32 further sites** have an options argument this
Expand All@@ -183,29 +183,29 @@ cannot read, and they are neither in nor out.

| what | count |
| :--- | ---: |
| write call sites on the application surface | **217** |
| …whose object name is statically decidable | 145 |
| write call sites on the application surface | **218** |
| …whose object name is statically decidable | 146 |
| …whose object name is chosen at run time | 72 |
| …against an object with tenancy ENABLED | 145 |
| …against an object with tenancy ENABLED | 146 |
| …against an object that declares tenancy off | 0 |
| threading a tenant context | 133 |
| threading a tenant context | 134 |
| PROVABLY carrying none (options read, no context key) | **17** |
| …of those, against a decidably tenancy-enabled object | **9** |
| options argument UNREADABLE — may or may not carry one | 67 |
| …of those, against a decidably tenancy-enabled object | 32 |
| threading a decidably ELEVATED (`isSystem`) context | 99 |
| threading a decidably ELEVATED (`isSystem`) context | 100 |
| threading a context that is decidably NOT elevated | 0 |
| threading a context whose elevation is a run-time fact | 101 |

| how the instrument reached the site | count |
| :--- | ---: |
| receiver carried a readable engine type | 172 |
| receiver carried a readable engine type | 173 |
| receiver erased, placed by the object NAME | 19 |
| receiver erased, placed by an `object: string` PARAMETER | 15 |
| receiver erased, placed by an `UNTYPED_RECEIVERS` row | 11 |

| object name spelled inline | 108 |
| object name spelled through a `const` | 37 |
| object name spelled through a `const` | 38 |
| object name is an `object: string` parameter | 19 |
| object name is some other run-time expression | 53 |

Expand All@@ -224,13 +224,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

{/* END GENERATED: tenant-audit-census */}

18 changes: 9 additions & 9 deletions
docs/audits/2026-08-tenant-audit-write-call-sites.counts.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,17 +29,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.

| Measure | Value |
|---|---:|
| Write call sites | 217 |
| Object name statically decidable | 145 |
| Write call sites | 218 |
| Object name statically decidable | 146 |
| Object name chosen at run time | 72 |
| Against a tenancy-enabled object | 145 |
| Against a tenancy-enabled object | 146 |
| Against an object declaring tenancy off | 0 |
| Threading a tenant context | 133 |
| Threading a tenant context | 134 |
| Provably carrying none | 17 |
| …and decidably tenancy-enabled | 9 |
| Options argument unreadable | 67 |
| …and decidably tenancy-enabled | 32 |
| Threading a decidably elevated context | 99 |
| Threading a decidably elevated context | 100 |
| Threading a decidably non-elevated context | 0 |
| Threading a context of undecidable elevation | 101 |

Expand All@@ -52,14 +52,14 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

## Every site

Expand DownExpand Up@@ -158,7 +158,7 @@ Measured on 2026-09-01 at `d3ebf3b55`.
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `update` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/flow-dispatch-store.ts` | `insert` | `sys_flow_dispatch` | enabled | elevated | 1 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 3 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `insert` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `update` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-datasource/src/datasource-admin-plugin.ts` | `delete` | `sys_metadata` | enabled | PROVABLY NONE | 1 |
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run#14712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-project-manager
merged 9 commits into

from

Sep 3, 2026

+1,332

−41

 Merged


fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run
#14712
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
d9282c3

fix(service-automation): conditional advance claim on SuspendedRunSto…


claude Sep 2, 2026
5daab8d

test(service-automation): record the measured ablation ledger (#14333)


claude Sep 2, 2026
97ac4af

docs(permissions): re-certify the tenant-audit census at 218 write ca…


claude Sep 2, 2026
6f41e20

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
2b36945

test(service-automation): pin the compare-and-set condition and the p…


claude Sep 2, 2026
e3bf5cd

test(service-automation): record the three measured ablation ledgers …


claude Sep 2, 2026
74595e4

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
ad24042

chore(gates): record the newly pinned engine double and close its DEB…


claude Sep 2, 2026
feb213d

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


44 changes: 44 additions & 0 deletions
.changeset/concurrent-replica-resume-advance-claim.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
---
"@objectstack/service-automation": minor
---

fix(service-automation): make the resume advance a conditional claim on the suspended-run store, so two replicas cannot both advance one run (#14333)

`AutomationEngine.resumeInternal` guarded a duplicate resume with `this.resuming`,
an in-process `Set`. That is a complete guard for exactly one deployment shape: a
single process. Behind a load balancer, two decisions on one run arriving in the
same instant on two replicas each passed their **own** `resuming` check, both read
the same fresh row out of the shared store, both consumed it, and both traversed
forward — so every downstream side effect ran twice. #13617 closed the sequential
half of this family (a replica resuming from a snapshot it had gone stale on); it
deliberately did not close the concurrent one.

Measured before the fix on the two-engines-over-one-shared-store harness, at
`packages/services/service-automation/src/concurrent-replica-resume-race.test.ts`:
**25 of 25** raced runs advanced twice — one action fired twice and one approval
level opened twice per run — for both reachable shapes the report named (parallel /
any-of approvers, and duplicated automated approve calls). A single approver per
level deciding sequentially does **not** race, and is pinned as the negative
control.

`SuspendedRunStore` therefore gains `claimSuspension(runId, parkedAt)`: consume the
durable record **only** if it is still parked at the node the caller read (and, when
the caller has one, still carrying that correlation), atomically, answering
`'claimed'` / `'lost'` / `'unsupported'`. The winner advances; the loser is refused
`RESUME_IN_PROGRESS` — the existing code, because the remedy is identical to the
in-process refusal's and `plugin-approvals` already branches on it that way — and
runs nothing. The per-process `resuming` set stays as the cheap first gate; it is
not replaced, and the single-replica path is unchanged.

Both shipped stores implement it: `InMemorySuspendedRunStore` tests and removes
with no `await` between the two, and `ObjectStoreSuspendedRunStore` issues one
`DELETE … WHERE id = ? AND node_id = ?` through the data engine's documented
compare-and-set route (`multi: true` with a full `where`), reading the affected-row
count. ⛔ No platform-object schema change: `node_id` and `correlation` are columns
`sys_automation_run` already carried.

The member is **optional**, so no existing implementation is broken, and its absence
is a declared degradation rather than a silent one: an engine whose store cannot
express the condition says once, at `warn`, that resume idempotency is in-process
only and what that costs — the same posture `AutomationEngine.claim()` already takes
when no persisted flow-dispatch ledger is attached.

34 changes: 17 additions & 17 deletions
content/docs/permissions/tenant-audit-census.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -98,7 +98,7 @@ are reported as `undecidable` rather than assumed either way.

The same holds twice over for the context. An options argument spelled as a
literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
forwarding shim cannot, and **67 of the 217 sites are spelled that way**. A
forwarding shim cannot, and **67 of the 218 sites are spelled that way**. A
context resolved from an inline literal or a local `const` can be tested for
`isSystem`; one arriving from a helper call cannot.

Expand DownExpand Up@@ -147,10 +147,10 @@ reproduce them. Where it disagrees, it disagrees on the page:

| carried figure | where it survives | this census |
| :--- | :--- | ---: |
| 175 write call sites | quoted in the merged changeset | **217** |
| 175 write call sites | quoted in the merged changeset | **218** |
| 24 carrying no tenant context | quoted in the merged changeset | **9** provable and tenancy-enabled; **32** more whose options argument is unreadable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **145 of 217** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 99 decidably elevated, 0 decidably not, 101 undecidable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **146 of 218** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 100 decidably elevated, 0 decidably not, 101 undecidable |
| 141 and 132, two independent re-derivations | the card that filed this work | — |

**The differences are not reconciled, and deliberately so.** The old census's
Expand All@@ -161,17 +161,17 @@ at any commit.

Two structural facts do plausibly widen this reading against any hand or regex
one, and both are counted in the generated tables below: the 45 sites reached
through an erased (`any`) receiver, and the 37 that name their object through a
through an erased (`any`) receiver, and the 38 that name their object through a
`const` rather than inline. An instrument that read either the way a person does
would report a smaller number and would not say so.

The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
figure has no surviving corroboration anywhere in the tree.** This census reads
99 of 217 (45%) as decidably elevated, with 101 more whose elevation is a
100 of 218 (46%) as decidably elevated, with 101 more whose elevation is a
run-time fact — so the claim is neither confirmed nor refuted, and the honest
answer is that a static reading cannot settle it.

⇒ **Cite `9 / 217`, and say what it is**: the sites whose options argument was
⇒ **Cite `9 / 218`, and say what it is**: the sites whose options argument was
READ and holds no tenant context, against a decidably tenancy-enabled object.
That is the control's provable yield surface. ⛔ Do not cite it as "the sites
without tenant context" — **32 further sites** have an options argument this
Expand All@@ -183,29 +183,29 @@ cannot read, and they are neither in nor out.

| what | count |
| :--- | ---: |
| write call sites on the application surface | **217** |
| …whose object name is statically decidable | 145 |
| write call sites on the application surface | **218** |
| …whose object name is statically decidable | 146 |
| …whose object name is chosen at run time | 72 |
| …against an object with tenancy ENABLED | 145 |
| …against an object with tenancy ENABLED | 146 |
| …against an object that declares tenancy off | 0 |
| threading a tenant context | 133 |
| threading a tenant context | 134 |
| PROVABLY carrying none (options read, no context key) | **17** |
| …of those, against a decidably tenancy-enabled object | **9** |
| options argument UNREADABLE — may or may not carry one | 67 |
| …of those, against a decidably tenancy-enabled object | 32 |
| threading a decidably ELEVATED (`isSystem`) context | 99 |
| threading a decidably ELEVATED (`isSystem`) context | 100 |
| threading a context that is decidably NOT elevated | 0 |
| threading a context whose elevation is a run-time fact | 101 |

| how the instrument reached the site | count |
| :--- | ---: |
| receiver carried a readable engine type | 172 |
| receiver carried a readable engine type | 173 |
| receiver erased, placed by the object NAME | 19 |
| receiver erased, placed by an `object: string` PARAMETER | 15 |
| receiver erased, placed by an `UNTYPED_RECEIVERS` row | 11 |

| object name spelled inline | 108 |
| object name spelled through a `const` | 37 |
| object name spelled through a `const` | 38 |
| object name is an `object: string` parameter | 19 |
| object name is some other run-time expression | 53 |

Expand All@@ -224,13 +224,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

{/* END GENERATED: tenant-audit-census */}

18 changes: 9 additions & 9 deletions
docs/audits/2026-08-tenant-audit-write-call-sites.counts.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,17 +29,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.

| Measure | Value |
|---|---:|
| Write call sites | 217 |
| Object name statically decidable | 145 |
| Write call sites | 218 |
| Object name statically decidable | 146 |
| Object name chosen at run time | 72 |
| Against a tenancy-enabled object | 145 |
| Against a tenancy-enabled object | 146 |
| Against an object declaring tenancy off | 0 |
| Threading a tenant context | 133 |
| Threading a tenant context | 134 |
| Provably carrying none | 17 |
| …and decidably tenancy-enabled | 9 |
| Options argument unreadable | 67 |
| …and decidably tenancy-enabled | 32 |
| Threading a decidably elevated context | 99 |
| Threading a decidably elevated context | 100 |
| Threading a decidably non-elevated context | 0 |
| Threading a context of undecidable elevation | 101 |

Expand All@@ -52,14 +52,14 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

## Every site

Expand DownExpand Up@@ -158,7 +158,7 @@ Measured on 2026-09-01 at `d3ebf3b55`.
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `update` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/flow-dispatch-store.ts` | `insert` | `sys_flow_dispatch` | enabled | elevated | 1 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 3 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `insert` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `update` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-datasource/src/datasource-admin-plugin.ts` | `delete` | `sys_metadata` | enabled | PROVABLY NONE | 1 |
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run#14712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-project-manager
merged 9 commits into

from

Sep 3, 2026

+1,332

−41

 Merged


fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run
#14712
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
d9282c3

fix(service-automation): conditional advance claim on SuspendedRunSto…


claude Sep 2, 2026
5daab8d

test(service-automation): record the measured ablation ledger (#14333)


claude Sep 2, 2026
97ac4af

docs(permissions): re-certify the tenant-audit census at 218 write ca…


claude Sep 2, 2026
6f41e20

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
2b36945

test(service-automation): pin the compare-and-set condition and the p…


claude Sep 2, 2026
e3bf5cd

test(service-automation): record the three measured ablation ledgers …


claude Sep 2, 2026
74595e4

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
ad24042

chore(gates): record the newly pinned engine double and close its DEB…


claude Sep 2, 2026
feb213d

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


44 changes: 44 additions & 0 deletions
.changeset/concurrent-replica-resume-advance-claim.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
---
"@objectstack/service-automation": minor
---

fix(service-automation): make the resume advance a conditional claim on the suspended-run store, so two replicas cannot both advance one run (#14333)

`AutomationEngine.resumeInternal` guarded a duplicate resume with `this.resuming`,
an in-process `Set`. That is a complete guard for exactly one deployment shape: a
single process. Behind a load balancer, two decisions on one run arriving in the
same instant on two replicas each passed their **own** `resuming` check, both read
the same fresh row out of the shared store, both consumed it, and both traversed
forward — so every downstream side effect ran twice. #13617 closed the sequential
half of this family (a replica resuming from a snapshot it had gone stale on); it
deliberately did not close the concurrent one.

Measured before the fix on the two-engines-over-one-shared-store harness, at
`packages/services/service-automation/src/concurrent-replica-resume-race.test.ts`:
**25 of 25** raced runs advanced twice — one action fired twice and one approval
level opened twice per run — for both reachable shapes the report named (parallel /
any-of approvers, and duplicated automated approve calls). A single approver per
level deciding sequentially does **not** race, and is pinned as the negative
control.

`SuspendedRunStore` therefore gains `claimSuspension(runId, parkedAt)`: consume the
durable record **only** if it is still parked at the node the caller read (and, when
the caller has one, still carrying that correlation), atomically, answering
`'claimed'` / `'lost'` / `'unsupported'`. The winner advances; the loser is refused
`RESUME_IN_PROGRESS` — the existing code, because the remedy is identical to the
in-process refusal's and `plugin-approvals` already branches on it that way — and
runs nothing. The per-process `resuming` set stays as the cheap first gate; it is
not replaced, and the single-replica path is unchanged.

Both shipped stores implement it: `InMemorySuspendedRunStore` tests and removes
with no `await` between the two, and `ObjectStoreSuspendedRunStore` issues one
`DELETE … WHERE id = ? AND node_id = ?` through the data engine's documented
compare-and-set route (`multi: true` with a full `where`), reading the affected-row
count. ⛔ No platform-object schema change: `node_id` and `correlation` are columns
`sys_automation_run` already carried.

The member is **optional**, so no existing implementation is broken, and its absence
is a declared degradation rather than a silent one: an engine whose store cannot
express the condition says once, at `warn`, that resume idempotency is in-process
only and what that costs — the same posture `AutomationEngine.claim()` already takes
when no persisted flow-dispatch ledger is attached.

34 changes: 17 additions & 17 deletions
content/docs/permissions/tenant-audit-census.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -98,7 +98,7 @@ are reported as `undecidable` rather than assumed either way.

The same holds twice over for the context. An options argument spelled as a
literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
forwarding shim cannot, and **67 of the 217 sites are spelled that way**. A
forwarding shim cannot, and **67 of the 218 sites are spelled that way**. A
context resolved from an inline literal or a local `const` can be tested for
`isSystem`; one arriving from a helper call cannot.

Expand DownExpand Up@@ -147,10 +147,10 @@ reproduce them. Where it disagrees, it disagrees on the page:

| carried figure | where it survives | this census |
| :--- | :--- | ---: |
| 175 write call sites | quoted in the merged changeset | **217** |
| 175 write call sites | quoted in the merged changeset | **218** |
| 24 carrying no tenant context | quoted in the merged changeset | **9** provable and tenancy-enabled; **32** more whose options argument is unreadable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **145 of 217** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 99 decidably elevated, 0 decidably not, 101 undecidable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **146 of 218** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 100 decidably elevated, 0 decidably not, 101 undecidable |
| 141 and 132, two independent re-derivations | the card that filed this work | — |

**The differences are not reconciled, and deliberately so.** The old census's
Expand All@@ -161,17 +161,17 @@ at any commit.

Two structural facts do plausibly widen this reading against any hand or regex
one, and both are counted in the generated tables below: the 45 sites reached
through an erased (`any`) receiver, and the 37 that name their object through a
through an erased (`any`) receiver, and the 38 that name their object through a
`const` rather than inline. An instrument that read either the way a person does
would report a smaller number and would not say so.

The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
figure has no surviving corroboration anywhere in the tree.** This census reads
99 of 217 (45%) as decidably elevated, with 101 more whose elevation is a
100 of 218 (46%) as decidably elevated, with 101 more whose elevation is a
run-time fact — so the claim is neither confirmed nor refuted, and the honest
answer is that a static reading cannot settle it.

⇒ **Cite `9 / 217`, and say what it is**: the sites whose options argument was
⇒ **Cite `9 / 218`, and say what it is**: the sites whose options argument was
READ and holds no tenant context, against a decidably tenancy-enabled object.
That is the control's provable yield surface. ⛔ Do not cite it as "the sites
without tenant context" — **32 further sites** have an options argument this
Expand All@@ -183,29 +183,29 @@ cannot read, and they are neither in nor out.

| what | count |
| :--- | ---: |
| write call sites on the application surface | **217** |
| …whose object name is statically decidable | 145 |
| write call sites on the application surface | **218** |
| …whose object name is statically decidable | 146 |
| …whose object name is chosen at run time | 72 |
| …against an object with tenancy ENABLED | 145 |
| …against an object with tenancy ENABLED | 146 |
| …against an object that declares tenancy off | 0 |
| threading a tenant context | 133 |
| threading a tenant context | 134 |
| PROVABLY carrying none (options read, no context key) | **17** |
| …of those, against a decidably tenancy-enabled object | **9** |
| options argument UNREADABLE — may or may not carry one | 67 |
| …of those, against a decidably tenancy-enabled object | 32 |
| threading a decidably ELEVATED (`isSystem`) context | 99 |
| threading a decidably ELEVATED (`isSystem`) context | 100 |
| threading a context that is decidably NOT elevated | 0 |
| threading a context whose elevation is a run-time fact | 101 |

| how the instrument reached the site | count |
| :--- | ---: |
| receiver carried a readable engine type | 172 |
| receiver carried a readable engine type | 173 |
| receiver erased, placed by the object NAME | 19 |
| receiver erased, placed by an `object: string` PARAMETER | 15 |
| receiver erased, placed by an `UNTYPED_RECEIVERS` row | 11 |

| object name spelled inline | 108 |
| object name spelled through a `const` | 37 |
| object name spelled through a `const` | 38 |
| object name is an `object: string` parameter | 19 |
| object name is some other run-time expression | 53 |

Expand All@@ -224,13 +224,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

{/* END GENERATED: tenant-audit-census */}

18 changes: 9 additions & 9 deletions
docs/audits/2026-08-tenant-audit-write-call-sites.counts.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,17 +29,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.

| Measure | Value |
|---|---:|
| Write call sites | 217 |
| Object name statically decidable | 145 |
| Write call sites | 218 |
| Object name statically decidable | 146 |
| Object name chosen at run time | 72 |
| Against a tenancy-enabled object | 145 |
| Against a tenancy-enabled object | 146 |
| Against an object declaring tenancy off | 0 |
| Threading a tenant context | 133 |
| Threading a tenant context | 134 |
| Provably carrying none | 17 |
| …and decidably tenancy-enabled | 9 |
| Options argument unreadable | 67 |
| …and decidably tenancy-enabled | 32 |
| Threading a decidably elevated context | 99 |
| Threading a decidably elevated context | 100 |
| Threading a decidably non-elevated context | 0 |
| Threading a context of undecidable elevation | 101 |

Expand All@@ -52,14 +52,14 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

## Every site

Expand DownExpand Up@@ -158,7 +158,7 @@ Measured on 2026-09-01 at `d3ebf3b55`.
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `update` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/flow-dispatch-store.ts` | `insert` | `sys_flow_dispatch` | enabled | elevated | 1 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 3 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `insert` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `update` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-datasource/src/datasource-admin-plugin.ts` | `delete` | `sys_metadata` | enabled | PROVABLY NONE | 1 |
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run#14712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-project-manager
merged 9 commits into

from

Sep 3, 2026

+1,332

−41

 Merged


fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run
#14712
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
d9282c3

fix(service-automation): conditional advance claim on SuspendedRunSto…


claude Sep 2, 2026
5daab8d

test(service-automation): record the measured ablation ledger (#14333)


claude Sep 2, 2026
97ac4af

docs(permissions): re-certify the tenant-audit census at 218 write ca…


claude Sep 2, 2026
6f41e20

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
2b36945

test(service-automation): pin the compare-and-set condition and the p…


claude Sep 2, 2026
e3bf5cd

test(service-automation): record the three measured ablation ledgers …


claude Sep 2, 2026
74595e4

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
ad24042

chore(gates): record the newly pinned engine double and close its DEB…


claude Sep 2, 2026
feb213d

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


44 changes: 44 additions & 0 deletions
.changeset/concurrent-replica-resume-advance-claim.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
---
"@objectstack/service-automation": minor
---

fix(service-automation): make the resume advance a conditional claim on the suspended-run store, so two replicas cannot both advance one run (#14333)

`AutomationEngine.resumeInternal` guarded a duplicate resume with `this.resuming`,
an in-process `Set`. That is a complete guard for exactly one deployment shape: a
single process. Behind a load balancer, two decisions on one run arriving in the
same instant on two replicas each passed their **own** `resuming` check, both read
the same fresh row out of the shared store, both consumed it, and both traversed
forward — so every downstream side effect ran twice. #13617 closed the sequential
half of this family (a replica resuming from a snapshot it had gone stale on); it
deliberately did not close the concurrent one.

Measured before the fix on the two-engines-over-one-shared-store harness, at
`packages/services/service-automation/src/concurrent-replica-resume-race.test.ts`:
**25 of 25** raced runs advanced twice — one action fired twice and one approval
level opened twice per run — for both reachable shapes the report named (parallel /
any-of approvers, and duplicated automated approve calls). A single approver per
level deciding sequentially does **not** race, and is pinned as the negative
control.

`SuspendedRunStore` therefore gains `claimSuspension(runId, parkedAt)`: consume the
durable record **only** if it is still parked at the node the caller read (and, when
the caller has one, still carrying that correlation), atomically, answering
`'claimed'` / `'lost'` / `'unsupported'`. The winner advances; the loser is refused
`RESUME_IN_PROGRESS` — the existing code, because the remedy is identical to the
in-process refusal's and `plugin-approvals` already branches on it that way — and
runs nothing. The per-process `resuming` set stays as the cheap first gate; it is
not replaced, and the single-replica path is unchanged.

Both shipped stores implement it: `InMemorySuspendedRunStore` tests and removes
with no `await` between the two, and `ObjectStoreSuspendedRunStore` issues one
`DELETE … WHERE id = ? AND node_id = ?` through the data engine's documented
compare-and-set route (`multi: true` with a full `where`), reading the affected-row
count. ⛔ No platform-object schema change: `node_id` and `correlation` are columns
`sys_automation_run` already carried.

The member is **optional**, so no existing implementation is broken, and its absence
is a declared degradation rather than a silent one: an engine whose store cannot
express the condition says once, at `warn`, that resume idempotency is in-process
only and what that costs — the same posture `AutomationEngine.claim()` already takes
when no persisted flow-dispatch ledger is attached.

34 changes: 17 additions & 17 deletions
content/docs/permissions/tenant-audit-census.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -98,7 +98,7 @@ are reported as `undecidable` rather than assumed either way.

The same holds twice over for the context. An options argument spelled as a
literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
forwarding shim cannot, and **67 of the 217 sites are spelled that way**. A
forwarding shim cannot, and **67 of the 218 sites are spelled that way**. A
context resolved from an inline literal or a local `const` can be tested for
`isSystem`; one arriving from a helper call cannot.

Expand DownExpand Up@@ -147,10 +147,10 @@ reproduce them. Where it disagrees, it disagrees on the page:

| carried figure | where it survives | this census |
| :--- | :--- | ---: |
| 175 write call sites | quoted in the merged changeset | **217** |
| 175 write call sites | quoted in the merged changeset | **218** |
| 24 carrying no tenant context | quoted in the merged changeset | **9** provable and tenancy-enabled; **32** more whose options argument is unreadable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **145 of 217** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 99 decidably elevated, 0 decidably not, 101 undecidable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **146 of 218** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 100 decidably elevated, 0 decidably not, 101 undecidable |
| 141 and 132, two independent re-derivations | the card that filed this work | — |

**The differences are not reconciled, and deliberately so.** The old census's
Expand All@@ -161,17 +161,17 @@ at any commit.

Two structural facts do plausibly widen this reading against any hand or regex
one, and both are counted in the generated tables below: the 45 sites reached
through an erased (`any`) receiver, and the 37 that name their object through a
through an erased (`any`) receiver, and the 38 that name their object through a
`const` rather than inline. An instrument that read either the way a person does
would report a smaller number and would not say so.

The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
figure has no surviving corroboration anywhere in the tree.** This census reads
99 of 217 (45%) as decidably elevated, with 101 more whose elevation is a
100 of 218 (46%) as decidably elevated, with 101 more whose elevation is a
run-time fact — so the claim is neither confirmed nor refuted, and the honest
answer is that a static reading cannot settle it.

⇒ **Cite `9 / 217`, and say what it is**: the sites whose options argument was
⇒ **Cite `9 / 218`, and say what it is**: the sites whose options argument was
READ and holds no tenant context, against a decidably tenancy-enabled object.
That is the control's provable yield surface. ⛔ Do not cite it as "the sites
without tenant context" — **32 further sites** have an options argument this
Expand All@@ -183,29 +183,29 @@ cannot read, and they are neither in nor out.

| what | count |
| :--- | ---: |
| write call sites on the application surface | **217** |
| …whose object name is statically decidable | 145 |
| write call sites on the application surface | **218** |
| …whose object name is statically decidable | 146 |
| …whose object name is chosen at run time | 72 |
| …against an object with tenancy ENABLED | 145 |
| …against an object with tenancy ENABLED | 146 |
| …against an object that declares tenancy off | 0 |
| threading a tenant context | 133 |
| threading a tenant context | 134 |
| PROVABLY carrying none (options read, no context key) | **17** |
| …of those, against a decidably tenancy-enabled object | **9** |
| options argument UNREADABLE — may or may not carry one | 67 |
| …of those, against a decidably tenancy-enabled object | 32 |
| threading a decidably ELEVATED (`isSystem`) context | 99 |
| threading a decidably ELEVATED (`isSystem`) context | 100 |
| threading a context that is decidably NOT elevated | 0 |
| threading a context whose elevation is a run-time fact | 101 |

| how the instrument reached the site | count |
| :--- | ---: |
| receiver carried a readable engine type | 172 |
| receiver carried a readable engine type | 173 |
| receiver erased, placed by the object NAME | 19 |
| receiver erased, placed by an `object: string` PARAMETER | 15 |
| receiver erased, placed by an `UNTYPED_RECEIVERS` row | 11 |

| object name spelled inline | 108 |
| object name spelled through a `const` | 37 |
| object name spelled through a `const` | 38 |
| object name is an `object: string` parameter | 19 |
| object name is some other run-time expression | 53 |

Expand All@@ -224,13 +224,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

{/* END GENERATED: tenant-audit-census */}

18 changes: 9 additions & 9 deletions
docs/audits/2026-08-tenant-audit-write-call-sites.counts.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,17 +29,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.

| Measure | Value |
|---|---:|
| Write call sites | 217 |
| Object name statically decidable | 145 |
| Write call sites | 218 |
| Object name statically decidable | 146 |
| Object name chosen at run time | 72 |
| Against a tenancy-enabled object | 145 |
| Against a tenancy-enabled object | 146 |
| Against an object declaring tenancy off | 0 |
| Threading a tenant context | 133 |
| Threading a tenant context | 134 |
| Provably carrying none | 17 |
| …and decidably tenancy-enabled | 9 |
| Options argument unreadable | 67 |
| …and decidably tenancy-enabled | 32 |
| Threading a decidably elevated context | 99 |
| Threading a decidably elevated context | 100 |
| Threading a decidably non-elevated context | 0 |
| Threading a context of undecidable elevation | 101 |

Expand All@@ -52,14 +52,14 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

## Every site

Expand DownExpand Up@@ -158,7 +158,7 @@ Measured on 2026-09-01 at `d3ebf3b55`.
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `update` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/flow-dispatch-store.ts` | `insert` | `sys_flow_dispatch` | enabled | elevated | 1 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 3 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `insert` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `update` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-datasource/src/datasource-admin-plugin.ts` | `delete` | `sys_metadata` | enabled | PROVABLY NONE | 1 |
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run#14712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-project-manager
merged 9 commits into

from

Sep 3, 2026

+1,332

−41

 Merged


fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run
#14712
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
d9282c3

fix(service-automation): conditional advance claim on SuspendedRunSto…


claude Sep 2, 2026
5daab8d

test(service-automation): record the measured ablation ledger (#14333)


claude Sep 2, 2026
97ac4af

docs(permissions): re-certify the tenant-audit census at 218 write ca…


claude Sep 2, 2026
6f41e20

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
2b36945

test(service-automation): pin the compare-and-set condition and the p…


claude Sep 2, 2026
e3bf5cd

test(service-automation): record the three measured ablation ledgers …


claude Sep 2, 2026
74595e4

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
ad24042

chore(gates): record the newly pinned engine double and close its DEB…


claude Sep 2, 2026
feb213d

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


44 changes: 44 additions & 0 deletions
.changeset/concurrent-replica-resume-advance-claim.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
---
"@objectstack/service-automation": minor
---

fix(service-automation): make the resume advance a conditional claim on the suspended-run store, so two replicas cannot both advance one run (#14333)

`AutomationEngine.resumeInternal` guarded a duplicate resume with `this.resuming`,
an in-process `Set`. That is a complete guard for exactly one deployment shape: a
single process. Behind a load balancer, two decisions on one run arriving in the
same instant on two replicas each passed their **own** `resuming` check, both read
the same fresh row out of the shared store, both consumed it, and both traversed
forward — so every downstream side effect ran twice. #13617 closed the sequential
half of this family (a replica resuming from a snapshot it had gone stale on); it
deliberately did not close the concurrent one.

Measured before the fix on the two-engines-over-one-shared-store harness, at
`packages/services/service-automation/src/concurrent-replica-resume-race.test.ts`:
**25 of 25** raced runs advanced twice — one action fired twice and one approval
level opened twice per run — for both reachable shapes the report named (parallel /
any-of approvers, and duplicated automated approve calls). A single approver per
level deciding sequentially does **not** race, and is pinned as the negative
control.

`SuspendedRunStore` therefore gains `claimSuspension(runId, parkedAt)`: consume the
durable record **only** if it is still parked at the node the caller read (and, when
the caller has one, still carrying that correlation), atomically, answering
`'claimed'` / `'lost'` / `'unsupported'`. The winner advances; the loser is refused
`RESUME_IN_PROGRESS` — the existing code, because the remedy is identical to the
in-process refusal's and `plugin-approvals` already branches on it that way — and
runs nothing. The per-process `resuming` set stays as the cheap first gate; it is
not replaced, and the single-replica path is unchanged.

Both shipped stores implement it: `InMemorySuspendedRunStore` tests and removes
with no `await` between the two, and `ObjectStoreSuspendedRunStore` issues one
`DELETE … WHERE id = ? AND node_id = ?` through the data engine's documented
compare-and-set route (`multi: true` with a full `where`), reading the affected-row
count. ⛔ No platform-object schema change: `node_id` and `correlation` are columns
`sys_automation_run` already carried.

The member is **optional**, so no existing implementation is broken, and its absence
is a declared degradation rather than a silent one: an engine whose store cannot
express the condition says once, at `warn`, that resume idempotency is in-process
only and what that costs — the same posture `AutomationEngine.claim()` already takes
when no persisted flow-dispatch ledger is attached.

34 changes: 17 additions & 17 deletions
content/docs/permissions/tenant-audit-census.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -98,7 +98,7 @@ are reported as `undecidable` rather than assumed either way.

The same holds twice over for the context. An options argument spelled as a
literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
forwarding shim cannot, and **67 of the 217 sites are spelled that way**. A
forwarding shim cannot, and **67 of the 218 sites are spelled that way**. A
context resolved from an inline literal or a local `const` can be tested for
`isSystem`; one arriving from a helper call cannot.

Expand DownExpand Up@@ -147,10 +147,10 @@ reproduce them. Where it disagrees, it disagrees on the page:

| carried figure | where it survives | this census |
| :--- | :--- | ---: |
| 175 write call sites | quoted in the merged changeset | **217** |
| 175 write call sites | quoted in the merged changeset | **218** |
| 24 carrying no tenant context | quoted in the merged changeset | **9** provable and tenancy-enabled; **32** more whose options argument is unreadable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **145 of 217** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 99 decidably elevated, 0 decidably not, 101 undecidable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **146 of 218** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 100 decidably elevated, 0 decidably not, 101 undecidable |
| 141 and 132, two independent re-derivations | the card that filed this work | — |

**The differences are not reconciled, and deliberately so.** The old census's
Expand All@@ -161,17 +161,17 @@ at any commit.

Two structural facts do plausibly widen this reading against any hand or regex
one, and both are counted in the generated tables below: the 45 sites reached
through an erased (`any`) receiver, and the 37 that name their object through a
through an erased (`any`) receiver, and the 38 that name their object through a
`const` rather than inline. An instrument that read either the way a person does
would report a smaller number and would not say so.

The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
figure has no surviving corroboration anywhere in the tree.** This census reads
99 of 217 (45%) as decidably elevated, with 101 more whose elevation is a
100 of 218 (46%) as decidably elevated, with 101 more whose elevation is a
run-time fact — so the claim is neither confirmed nor refuted, and the honest
answer is that a static reading cannot settle it.

⇒ **Cite `9 / 217`, and say what it is**: the sites whose options argument was
⇒ **Cite `9 / 218`, and say what it is**: the sites whose options argument was
READ and holds no tenant context, against a decidably tenancy-enabled object.
That is the control's provable yield surface. ⛔ Do not cite it as "the sites
without tenant context" — **32 further sites** have an options argument this
Expand All@@ -183,29 +183,29 @@ cannot read, and they are neither in nor out.

| what | count |
| :--- | ---: |
| write call sites on the application surface | **217** |
| …whose object name is statically decidable | 145 |
| write call sites on the application surface | **218** |
| …whose object name is statically decidable | 146 |
| …whose object name is chosen at run time | 72 |
| …against an object with tenancy ENABLED | 145 |
| …against an object with tenancy ENABLED | 146 |
| …against an object that declares tenancy off | 0 |
| threading a tenant context | 133 |
| threading a tenant context | 134 |
| PROVABLY carrying none (options read, no context key) | **17** |
| …of those, against a decidably tenancy-enabled object | **9** |
| options argument UNREADABLE — may or may not carry one | 67 |
| …of those, against a decidably tenancy-enabled object | 32 |
| threading a decidably ELEVATED (`isSystem`) context | 99 |
| threading a decidably ELEVATED (`isSystem`) context | 100 |
| threading a context that is decidably NOT elevated | 0 |
| threading a context whose elevation is a run-time fact | 101 |

| how the instrument reached the site | count |
| :--- | ---: |
| receiver carried a readable engine type | 172 |
| receiver carried a readable engine type | 173 |
| receiver erased, placed by the object NAME | 19 |
| receiver erased, placed by an `object: string` PARAMETER | 15 |
| receiver erased, placed by an `UNTYPED_RECEIVERS` row | 11 |

| object name spelled inline | 108 |
| object name spelled through a `const` | 37 |
| object name spelled through a `const` | 38 |
| object name is an `object: string` parameter | 19 |
| object name is some other run-time expression | 53 |

Expand All@@ -224,13 +224,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

{/* END GENERATED: tenant-audit-census */}

18 changes: 9 additions & 9 deletions
docs/audits/2026-08-tenant-audit-write-call-sites.counts.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,17 +29,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.

| Measure | Value |
|---|---:|
| Write call sites | 217 |
| Object name statically decidable | 145 |
| Write call sites | 218 |
| Object name statically decidable | 146 |
| Object name chosen at run time | 72 |
| Against a tenancy-enabled object | 145 |
| Against a tenancy-enabled object | 146 |
| Against an object declaring tenancy off | 0 |
| Threading a tenant context | 133 |
| Threading a tenant context | 134 |
| Provably carrying none | 17 |
| …and decidably tenancy-enabled | 9 |
| Options argument unreadable | 67 |
| …and decidably tenancy-enabled | 32 |
| Threading a decidably elevated context | 99 |
| Threading a decidably elevated context | 100 |
| Threading a decidably non-elevated context | 0 |
| Threading a context of undecidable elevation | 101 |

Expand All@@ -52,14 +52,14 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

## Every site

Expand DownExpand Up@@ -158,7 +158,7 @@ Measured on 2026-09-01 at `d3ebf3b55`.
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `update` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/flow-dispatch-store.ts` | `insert` | `sys_flow_dispatch` | enabled | elevated | 1 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 3 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `insert` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `update` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-datasource/src/datasource-admin-plugin.ts` | `delete` | `sys_metadata` | enabled | PROVABLY NONE | 1 |
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run#14712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-project-manager
merged 9 commits into

from

Sep 3, 2026

+1,332

−41

 Merged


fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run
#14712
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
d9282c3

fix(service-automation): conditional advance claim on SuspendedRunSto…


claude Sep 2, 2026
5daab8d

test(service-automation): record the measured ablation ledger (#14333)


claude Sep 2, 2026
97ac4af

docs(permissions): re-certify the tenant-audit census at 218 write ca…


claude Sep 2, 2026
6f41e20

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
2b36945

test(service-automation): pin the compare-and-set condition and the p…


claude Sep 2, 2026
e3bf5cd

test(service-automation): record the three measured ablation ledgers …


claude Sep 2, 2026
74595e4

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
ad24042

chore(gates): record the newly pinned engine double and close its DEB…


claude Sep 2, 2026
feb213d

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


44 changes: 44 additions & 0 deletions
.changeset/concurrent-replica-resume-advance-claim.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
---
"@objectstack/service-automation": minor
---

fix(service-automation): make the resume advance a conditional claim on the suspended-run store, so two replicas cannot both advance one run (#14333)

`AutomationEngine.resumeInternal` guarded a duplicate resume with `this.resuming`,
an in-process `Set`. That is a complete guard for exactly one deployment shape: a
single process. Behind a load balancer, two decisions on one run arriving in the
same instant on two replicas each passed their **own** `resuming` check, both read
the same fresh row out of the shared store, both consumed it, and both traversed
forward — so every downstream side effect ran twice. #13617 closed the sequential
half of this family (a replica resuming from a snapshot it had gone stale on); it
deliberately did not close the concurrent one.

Measured before the fix on the two-engines-over-one-shared-store harness, at
`packages/services/service-automation/src/concurrent-replica-resume-race.test.ts`:
**25 of 25** raced runs advanced twice — one action fired twice and one approval
level opened twice per run — for both reachable shapes the report named (parallel /
any-of approvers, and duplicated automated approve calls). A single approver per
level deciding sequentially does **not** race, and is pinned as the negative
control.

`SuspendedRunStore` therefore gains `claimSuspension(runId, parkedAt)`: consume the
durable record **only** if it is still parked at the node the caller read (and, when
the caller has one, still carrying that correlation), atomically, answering
`'claimed'` / `'lost'` / `'unsupported'`. The winner advances; the loser is refused
`RESUME_IN_PROGRESS` — the existing code, because the remedy is identical to the
in-process refusal's and `plugin-approvals` already branches on it that way — and
runs nothing. The per-process `resuming` set stays as the cheap first gate; it is
not replaced, and the single-replica path is unchanged.

Both shipped stores implement it: `InMemorySuspendedRunStore` tests and removes
with no `await` between the two, and `ObjectStoreSuspendedRunStore` issues one
`DELETE … WHERE id = ? AND node_id = ?` through the data engine's documented
compare-and-set route (`multi: true` with a full `where`), reading the affected-row
count. ⛔ No platform-object schema change: `node_id` and `correlation` are columns
`sys_automation_run` already carried.

The member is **optional**, so no existing implementation is broken, and its absence
is a declared degradation rather than a silent one: an engine whose store cannot
express the condition says once, at `warn`, that resume idempotency is in-process
only and what that costs — the same posture `AutomationEngine.claim()` already takes
when no persisted flow-dispatch ledger is attached.

34 changes: 17 additions & 17 deletions
content/docs/permissions/tenant-audit-census.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -98,7 +98,7 @@ are reported as `undecidable` rather than assumed either way.

The same holds twice over for the context. An options argument spelled as a
literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
forwarding shim cannot, and **67 of the 217 sites are spelled that way**. A
forwarding shim cannot, and **67 of the 218 sites are spelled that way**. A
context resolved from an inline literal or a local `const` can be tested for
`isSystem`; one arriving from a helper call cannot.

Expand DownExpand Up@@ -147,10 +147,10 @@ reproduce them. Where it disagrees, it disagrees on the page:

| carried figure | where it survives | this census |
| :--- | :--- | ---: |
| 175 write call sites | quoted in the merged changeset | **217** |
| 175 write call sites | quoted in the merged changeset | **218** |
| 24 carrying no tenant context | quoted in the merged changeset | **9** provable and tenancy-enabled; **32** more whose options argument is unreadable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **145 of 217** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 99 decidably elevated, 0 decidably not, 101 undecidable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **146 of 218** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 100 decidably elevated, 0 decidably not, 101 undecidable |
| 141 and 132, two independent re-derivations | the card that filed this work | — |

**The differences are not reconciled, and deliberately so.** The old census's
Expand All@@ -161,17 +161,17 @@ at any commit.

Two structural facts do plausibly widen this reading against any hand or regex
one, and both are counted in the generated tables below: the 45 sites reached
through an erased (`any`) receiver, and the 37 that name their object through a
through an erased (`any`) receiver, and the 38 that name their object through a
`const` rather than inline. An instrument that read either the way a person does
would report a smaller number and would not say so.

The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
figure has no surviving corroboration anywhere in the tree.** This census reads
99 of 217 (45%) as decidably elevated, with 101 more whose elevation is a
100 of 218 (46%) as decidably elevated, with 101 more whose elevation is a
run-time fact — so the claim is neither confirmed nor refuted, and the honest
answer is that a static reading cannot settle it.

⇒ **Cite `9 / 217`, and say what it is**: the sites whose options argument was
⇒ **Cite `9 / 218`, and say what it is**: the sites whose options argument was
READ and holds no tenant context, against a decidably tenancy-enabled object.
That is the control's provable yield surface. ⛔ Do not cite it as "the sites
without tenant context" — **32 further sites** have an options argument this
Expand All@@ -183,29 +183,29 @@ cannot read, and they are neither in nor out.

| what | count |
| :--- | ---: |
| write call sites on the application surface | **217** |
| …whose object name is statically decidable | 145 |
| write call sites on the application surface | **218** |
| …whose object name is statically decidable | 146 |
| …whose object name is chosen at run time | 72 |
| …against an object with tenancy ENABLED | 145 |
| …against an object with tenancy ENABLED | 146 |
| …against an object that declares tenancy off | 0 |
| threading a tenant context | 133 |
| threading a tenant context | 134 |
| PROVABLY carrying none (options read, no context key) | **17** |
| …of those, against a decidably tenancy-enabled object | **9** |
| options argument UNREADABLE — may or may not carry one | 67 |
| …of those, against a decidably tenancy-enabled object | 32 |
| threading a decidably ELEVATED (`isSystem`) context | 99 |
| threading a decidably ELEVATED (`isSystem`) context | 100 |
| threading a context that is decidably NOT elevated | 0 |
| threading a context whose elevation is a run-time fact | 101 |

| how the instrument reached the site | count |
| :--- | ---: |
| receiver carried a readable engine type | 172 |
| receiver carried a readable engine type | 173 |
| receiver erased, placed by the object NAME | 19 |
| receiver erased, placed by an `object: string` PARAMETER | 15 |
| receiver erased, placed by an `UNTYPED_RECEIVERS` row | 11 |

| object name spelled inline | 108 |
| object name spelled through a `const` | 37 |
| object name spelled through a `const` | 38 |
| object name is an `object: string` parameter | 19 |
| object name is some other run-time expression | 53 |

Expand All@@ -224,13 +224,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

{/* END GENERATED: tenant-audit-census */}

18 changes: 9 additions & 9 deletions
docs/audits/2026-08-tenant-audit-write-call-sites.counts.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,17 +29,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.

| Measure | Value |
|---|---:|
| Write call sites | 217 |
| Object name statically decidable | 145 |
| Write call sites | 218 |
| Object name statically decidable | 146 |
| Object name chosen at run time | 72 |
| Against a tenancy-enabled object | 145 |
| Against a tenancy-enabled object | 146 |
| Against an object declaring tenancy off | 0 |
| Threading a tenant context | 133 |
| Threading a tenant context | 134 |
| Provably carrying none | 17 |
| …and decidably tenancy-enabled | 9 |
| Options argument unreadable | 67 |
| …and decidably tenancy-enabled | 32 |
| Threading a decidably elevated context | 99 |
| Threading a decidably elevated context | 100 |
| Threading a decidably non-elevated context | 0 |
| Threading a context of undecidable elevation | 101 |

Expand All@@ -52,14 +52,14 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

## Every site

Expand DownExpand Up@@ -158,7 +158,7 @@ Measured on 2026-09-01 at `d3ebf3b55`.
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `update` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/flow-dispatch-store.ts` | `insert` | `sys_flow_dispatch` | enabled | elevated | 1 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 3 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `insert` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `update` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-datasource/src/datasource-admin-plugin.ts` | `delete` | `sys_metadata` | enabled | PROVABLY NONE | 1 |
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run#14712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-project-manager
merged 9 commits into

from

Sep 3, 2026

+1,332

−41

 Merged


fix(service-automation): a conditional advance claim on SuspendedRunStore, so two replicas cannot both advance one run
#14712
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


9 commits


Select commit
Hold shift + click to select a range
d9282c3

fix(service-automation): conditional advance claim on SuspendedRunSto…


claude Sep 2, 2026
5daab8d

test(service-automation): record the measured ablation ledger (#14333)


claude Sep 2, 2026
97ac4af

docs(permissions): re-certify the tenant-audit census at 218 write ca…


claude Sep 2, 2026
6f41e20

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
2b36945

test(service-automation): pin the compare-and-set condition and the p…


claude Sep 2, 2026
e3bf5cd

test(service-automation): record the three measured ablation ledgers …


claude Sep 2, 2026
74595e4

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026
ad24042

chore(gates): record the newly pinned engine double and close its DEB…


claude Sep 2, 2026
feb213d

Merge remote-tracking branch 'origin/main' into claude/issue-14333-co…


claude Sep 2, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


44 changes: 44 additions & 0 deletions
.changeset/concurrent-replica-resume-advance-claim.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
---
"@objectstack/service-automation": minor
---

fix(service-automation): make the resume advance a conditional claim on the suspended-run store, so two replicas cannot both advance one run (#14333)

`AutomationEngine.resumeInternal` guarded a duplicate resume with `this.resuming`,
an in-process `Set`. That is a complete guard for exactly one deployment shape: a
single process. Behind a load balancer, two decisions on one run arriving in the
same instant on two replicas each passed their **own** `resuming` check, both read
the same fresh row out of the shared store, both consumed it, and both traversed
forward — so every downstream side effect ran twice. #13617 closed the sequential
half of this family (a replica resuming from a snapshot it had gone stale on); it
deliberately did not close the concurrent one.

Measured before the fix on the two-engines-over-one-shared-store harness, at
`packages/services/service-automation/src/concurrent-replica-resume-race.test.ts`:
**25 of 25** raced runs advanced twice — one action fired twice and one approval
level opened twice per run — for both reachable shapes the report named (parallel /
any-of approvers, and duplicated automated approve calls). A single approver per
level deciding sequentially does **not** race, and is pinned as the negative
control.

`SuspendedRunStore` therefore gains `claimSuspension(runId, parkedAt)`: consume the
durable record **only** if it is still parked at the node the caller read (and, when
the caller has one, still carrying that correlation), atomically, answering
`'claimed'` / `'lost'` / `'unsupported'`. The winner advances; the loser is refused
`RESUME_IN_PROGRESS` — the existing code, because the remedy is identical to the
in-process refusal's and `plugin-approvals` already branches on it that way — and
runs nothing. The per-process `resuming` set stays as the cheap first gate; it is
not replaced, and the single-replica path is unchanged.

Both shipped stores implement it: `InMemorySuspendedRunStore` tests and removes
with no `await` between the two, and `ObjectStoreSuspendedRunStore` issues one
`DELETE … WHERE id = ? AND node_id = ?` through the data engine's documented
compare-and-set route (`multi: true` with a full `where`), reading the affected-row
count. ⛔ No platform-object schema change: `node_id` and `correlation` are columns
`sys_automation_run` already carried.

The member is **optional**, so no existing implementation is broken, and its absence
is a declared degradation rather than a silent one: an engine whose store cannot
express the condition says once, at `warn`, that resume idempotency is in-process
only and what that costs — the same posture `AutomationEngine.claim()` already takes
when no persisted flow-dispatch ledger is attached.

34 changes: 17 additions & 17 deletions
content/docs/permissions/tenant-audit-census.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -98,7 +98,7 @@ are reported as `undecidable` rather than assumed either way.

The same holds twice over for the context. An options argument spelled as a
literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
forwarding shim cannot, and **67 of the 217 sites are spelled that way**. A
forwarding shim cannot, and **67 of the 218 sites are spelled that way**. A
context resolved from an inline literal or a local `const` can be tested for
`isSystem`; one arriving from a helper call cannot.

Expand DownExpand Up@@ -147,10 +147,10 @@ reproduce them. Where it disagrees, it disagrees on the page:

| carried figure | where it survives | this census |
| :--- | :--- | ---: |
| 175 write call sites | quoted in the merged changeset | **217** |
| 175 write call sites | quoted in the merged changeset | **218** |
| 24 carrying no tenant context | quoted in the merged changeset | **9** provable and tenancy-enabled; **32** more whose options argument is unreadable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **145 of 217** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 99 decidably elevated, 0 decidably not, 101 undecidable |
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **146 of 218** decidable, **72** undecidable |
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 100 decidably elevated, 0 decidably not, 101 undecidable |
| 141 and 132, two independent re-derivations | the card that filed this work | — |

**The differences are not reconciled, and deliberately so.** The old census's
Expand All@@ -161,17 +161,17 @@ at any commit.

Two structural facts do plausibly widen this reading against any hand or regex
one, and both are counted in the generated tables below: the 45 sites reached
through an erased (`any`) receiver, and the 37 that name their object through a
through an erased (`any`) receiver, and the 38 that name their object through a
`const` rather than inline. An instrument that read either the way a person does
would report a smaller number and would not say so.

The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
figure has no surviving corroboration anywhere in the tree.** This census reads
99 of 217 (45%) as decidably elevated, with 101 more whose elevation is a
100 of 218 (46%) as decidably elevated, with 101 more whose elevation is a
run-time fact — so the claim is neither confirmed nor refuted, and the honest
answer is that a static reading cannot settle it.

⇒ **Cite `9 / 217`, and say what it is**: the sites whose options argument was
⇒ **Cite `9 / 218`, and say what it is**: the sites whose options argument was
READ and holds no tenant context, against a decidably tenancy-enabled object.
That is the control's provable yield surface. ⛔ Do not cite it as "the sites
without tenant context" — **32 further sites** have an options argument this
Expand All@@ -183,29 +183,29 @@ cannot read, and they are neither in nor out.

| what | count |
| :--- | ---: |
| write call sites on the application surface | **217** |
| …whose object name is statically decidable | 145 |
| write call sites on the application surface | **218** |
| …whose object name is statically decidable | 146 |
| …whose object name is chosen at run time | 72 |
| …against an object with tenancy ENABLED | 145 |
| …against an object with tenancy ENABLED | 146 |
| …against an object that declares tenancy off | 0 |
| threading a tenant context | 133 |
| threading a tenant context | 134 |
| PROVABLY carrying none (options read, no context key) | **17** |
| …of those, against a decidably tenancy-enabled object | **9** |
| options argument UNREADABLE — may or may not carry one | 67 |
| …of those, against a decidably tenancy-enabled object | 32 |
| threading a decidably ELEVATED (`isSystem`) context | 99 |
| threading a decidably ELEVATED (`isSystem`) context | 100 |
| threading a context that is decidably NOT elevated | 0 |
| threading a context whose elevation is a run-time fact | 101 |

| how the instrument reached the site | count |
| :--- | ---: |
| receiver carried a readable engine type | 172 |
| receiver carried a readable engine type | 173 |
| receiver erased, placed by the object NAME | 19 |
| receiver erased, placed by an `object: string` PARAMETER | 15 |
| receiver erased, placed by an `UNTYPED_RECEIVERS` row | 11 |

| object name spelled inline | 108 |
| object name spelled through a `const` | 37 |
| object name spelled through a `const` | 38 |
| object name is an `object: string` parameter | 19 |
| object name is some other run-time expression | 53 |

Expand All@@ -224,13 +224,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

{/* END GENERATED: tenant-audit-census */}

18 changes: 9 additions & 9 deletions
docs/audits/2026-08-tenant-audit-write-call-sites.counts.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,17 +29,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.

| Measure | Value |
|---|---:|
| Write call sites | 217 |
| Object name statically decidable | 145 |
| Write call sites | 218 |
| Object name statically decidable | 146 |
| Object name chosen at run time | 72 |
| Against a tenancy-enabled object | 145 |
| Against a tenancy-enabled object | 146 |
| Against an object declaring tenancy off | 0 |
| Threading a tenant context | 133 |
| Threading a tenant context | 134 |
| Provably carrying none | 17 |
| …and decidably tenancy-enabled | 9 |
| Options argument unreadable | 67 |
| …and decidably tenancy-enabled | 32 |
| Threading a decidably elevated context | 99 |
| Threading a decidably elevated context | 100 |
| Threading a decidably non-elevated context | 0 |
| Threading a context of undecidable elevation | 101 |

Expand All@@ -52,14 +52,14 @@ holds still. They are required to be HERE and to say WHEN they were true;
their values are not compared. The reasoning, and the measurement behind it,
are in `scripts/check-tenant-audit-census.mjs`.

Measured on 2026-09-01 at `d3ebf3b55`.
Measured on 2026-09-02 at `5daab8df0`.

| corpus scale (not enforced) | count |
| :--- | ---: |
| tracked non-test sources scanned | 534 |
| tracked non-test sources scanned | 539 |
| engine-shaped types recognised | 56 |
| declared objects in the registry | 297 |
| same-named calls subtracted as non-engine | 119 |
| same-named calls subtracted as non-engine | 130 |

## Every site

Expand DownExpand Up@@ -158,7 +158,7 @@ Measured on 2026-09-01 at `d3ebf3b55`.
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `insert` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/builtin/crud-nodes.ts` | `update` | `objectName` | undecidable | context, elevation undecidable | 1 |
| `packages/services/service-automation/src/flow-dispatch-store.ts` | `insert` | `sys_flow_dispatch` | enabled | elevated | 1 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `delete` | `sys_automation_run` | enabled | elevated | 3 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `insert` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-automation/src/suspended-run-store.ts` | `update` | `sys_automation_run` | enabled | elevated | 2 |
| `packages/services/service-datasource/src/datasource-admin-plugin.ts` | `delete` | `sys_metadata` | enabled | PROVABLY NONE | 1 |
Expand Down
Loading
Loading