Skip to content

fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained#13763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
os-steve merged 7 commits into from
Aug 31, 2026
Merged

fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained #13763

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
85c37f6
wip(runtime): partition /ready driver health into primary vs secondar…
os-steve Aug 31, 2026
e8bb34a
test(objectql): pin system-object-unbound to the reachable zero-drive…
os-steve Aug 31, 2026
ae226f6
chore: changeset for #13408
os-steve Aug 31, 2026
cbbc8d0
Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…
claude Aug 31, 2026
6a07781
docs(permissions): re-anchor the system-context census after the engi…
os-steve Aug 31, 2026
0e2fdb8
Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…
claude Aug 31, 2026
2aa73e8
fix(test): type the driver double and narrow the dispatch result so t…
os-steve Aug 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .changeset/ready-primary-datasource-drain.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
---
"@objectstack/objectql": minor
"@objectstack/runtime": minor
---

fix(runtime,objectql): `/api/v1/ready` drains only on the PRIMARY datasource's failure; a secondary/tenant datasource is reported, not drained (#13408)

On a multi-datasource deployment, one datasource whose driver could not start
pinned `/api/v1/ready` to 503 on **every replica** — so a readiness-checked load
balancer drained every upstream and took the whole deployment offline, while
Postgres and the app itself were healthy (`/api/v1/health` 200, direct reads
working). One tenant's misconfiguration = total outage. Observed on a live
3-replica EE deployment and recovered only by restarting every process.

Ruled 2026-08-31 (第 6 场总监席决裁批 #12, maintainer verbatim 「同意」), Option B:

> `/api/v1/ready` 只在**主/默认数据源**不健康时摘流量;次要/租户数据源的故障照常
> **上报**(`/ready` 响应 body、日志、告警)但不 drain 节点。

**What changed.** When a driver reports itself unhealthy, `/ready` now asks
which datasource is the deployment's primary one before choosing a status:

- primary unhealthy, or the criterion unresolvable ⇒ **503**, unchanged envelope;
- primary healthy, only a secondary down ⇒ **200**, with the failed drivers named
in a new `degraded` block: `{ status, state, degraded: { drivers, primaryDatasource } }`.

The failed driver is never hidden — the rejected alternative of filtering it out
of the response stays rejected. `degraded` appears **only** on that branch; an
all-healthy 200 body is byte-identical to before.

**The criterion is a readable fact, not a heuristic.** `ObjectQL.resolvePrimaryDatasource()`
(new, exported with `PrimaryDatasourceVerdict` / `PrimaryDatasourceUnresolvedReason`)
answers *where this deployment's platform system objects actually live*, resolved
through the same five-step routing order every query uses — never registration
order, never the driver flagged default at registration. The ADR-0057 §3.6 system
ledgers (`audit` / `telemetry` / `event`) are excluded because they are
deliberately routed off the primary. Disagreement, silence, or a name with no
driver behind it return an **unresolved verdict**, never a guess.

**Fail toward draining.** Every way of not knowing — an engine that predates the
probe, a probe that throws, a malformed verdict, a genuinely split deployment —
falls back to the old whole-node 503. Staying in rotation requires a *positive*
reading; the absence of a negative one is not permission. Pinned in both
directions, including an inversion ablation.

**framework#3756 is not overturned.** Its quantified reason — "a replica that
would fail 100% of its requests" — still holds in the single-datasource
deployment it was measured on, where the primary *is* the only source; that
deployment's answer is unchanged down to the response body. What #3756's
reasoning never covered is the multi-datasource shape, and that is the only
branch this carves out.

**Grade: `minor` for both, proposed rather than assumed** — this changes when a
published operational probe drains a node, so an operator whose alerting keys on
`/ready` returning 503 for *any* driver failure will see 200 + `degraded`
instead, and `degraded` is a new response field. Nothing is removed or renamed,
no declared contract key is added (the ruling explicitly declines that: 「⛔ 本裁不
加契约键」), single-datasource behaviour is bit-identical, and no migration is
required — so `major` overstates it and `patch` understates a deliberate change
to an availability control surface.

⚠️ Out of scope, tracked separately as **#13578**: `DELETE` of a datasource still
does not evict the stuck driver from the in-memory engine registry, so the
datasource keeps appearing in `/ready`'s report until the process restarts. That
half is a defect under either answer to this card and is queued independently.
24 changes: 12 additions & 12 deletions content/docs/permissions/system-context.mdx
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -109,18 +109,18 @@ that silently does not happen.

| # | Behaviour when `isSystem` | Package | What you get / what you lose | Anchor |
|:--|:---|:---|:---|:---|
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10616` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10778` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9509` |
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10712` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10874` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9605` |
| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1576` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9546`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5638` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3573`, `:3583`, `:3610` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9642`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5639` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3574`, `:3584`, `:3611` |
| 25 | Engine-owned / append-only write guard bypassed | plugin-security | Get: generic writes to `managedBy` engine-owned objects | `system-write-guard.ts:96`, `:120` |
| 26 | Identity write guard bypassed (ADR-0092) | plugin-auth | Get: direct writes to identity tables through the generic data path | `identity-write-guard.ts:98` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6336` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11364` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11293` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6337` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11460` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11389` |

### 3. Sharing (`plugin-sharing`)

Expand DownExpand Up@@ -179,8 +179,8 @@ a reader tracing where elevation travels needs them.

| # | Site | Package | What it does |
|:--|:---|:---|:---|
| 62 | `objectql/src/engine.ts:3405` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13705` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 62 | `objectql/src/engine.ts:3406` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13801` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 64 | `plugin-reports/src/report-service.ts:556` | plugin-reports | Threads the flag into the engine call that runs a report |
| 65 | `body-runner.ts:279` | runtime | Rebuilds an `ExecutionContext` from a hook session, carrying the flag across |

Expand All@@ -195,7 +195,7 @@ assuming `isSystem` covers it is a documented source of bugs.
|:---|:---|:---|
| "It suppresses triggers / record-change automation" | **No.** Only `skipTriggers` does. A bare `{ isSystem: true }` on a seed write re-fired automation on freshly seeded rows and wedged first boot | `metadata-protocol/src/seed-loader.ts:1909` (rationale at `:1819`–`1821`, #3760), `flow.zod.ts:685` |
| "It skips the state machine" | **No.** That is `skipStateMachine`, carried by seed replay and by `treatAsHistorical` imports | `objectql/src/engine.ts` FSM gate; see [State Machine](/docs/protocol/objectql/state-machine) |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9492`–`9509` |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9588`–`9605` |
| "It preserves a supplied `updated_at` / `updated_by`" | **No.** That is `preserveAudit`, a separate opt-in — and an UPDATE-path exemption only | `field.zod.ts:1451` (#3493 / #6640) |
| "It stamps `created_by`" | **No.** Audit stamping reads `userId` from the context. A user-less system write stamps nothing — that is today's behaviour, not an error | `runtime-identity.ts:280`–`281` |
| "It bypasses every guard" | **No.** The last-admin guard applies to **every** context, `isSystem` included — the deprovision path that actually locks an org out is the system one | `last-admin-guard.ts:286` |
Expand Down
48 changes: 48 additions & 0 deletions packages/objectql/src/driver-connect-errors.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -19,6 +19,54 @@ export interface DriverHealth {
skipped?: boolean;
}

/**
* Why {@link PrimaryDatasourceVerdict} could not name a primary datasource.
*
* Every member means the SAME thing to a readiness probe — *we cannot tell* —
* and the ruling on #13408 fixes what a probe must do with that answer: fall
* back to the old whole-node 503 (fail toward draining), never to "don't
* drain". They are distinguished only so the reason can be logged and reported;
* ⛔ a caller must not branch on them to keep a replica in rotation.
*/
export type PrimaryDatasourceUnresolvedReason =
/** No platform system object is registered here — nothing to read the fact off. */
| 'no-system-objects-registered'
/** The platform's system objects are split across more than one datasource. */
| 'system-objects-split'
/** A registered system object routes nowhere: no binding and no default driver. */
| 'system-object-unbound'
/** The name resolved, but no driver is registered under it, so nothing probes it. */
| 'no-driver-registered';

/**
* WHICH datasource carries this deployment's platform system objects — the
* machine-readable "primary/default datasource" fact the #13408 ruling requires
* (2026-08-31, 第 6 场总监席决裁批 #12), stated as a verdict rather than a
* `string | undefined` so "we could not tell" can never be mistaken for a name.
*
* ⛔ The ruling forbids deriving this from a heuristic such as "the first
* datasource registered". The one implementation is
* `ObjectQL.resolvePrimaryDatasource()`.
*/
export type PrimaryDatasourceVerdict =
| {
resolved: true;
/** The datasource name, which is also a registered driver's name. */
datasource: string;
/**
* How many registered platform system objects agreed on it. Never 0 — a
* verdict read off nothing is `no-system-objects-registered`, not a name,
* so a caller cannot act on a vacuous agreement.
*/
witnesses: number;
}
| {
resolved: false;
reason: PrimaryDatasourceUnresolvedReason;
/** The distinct datasource names seen, when more than one disagreed. */
candidates?: readonly string[];
};

/** `error.message` when it is an Error, its string form otherwise. */
function failureMessage(error: unknown): string {
if (error instanceof Error) return error.message || error.name;
Expand Down
Loading
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained by os-steve · Pull Request #13763 · objectstack-ai/objectstack · GitHub
Skip to content
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained#13763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-steve
merged 7 commits into

from

Aug 31, 2026
 Merged


fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained
#13763
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


7 commits


Select commit
Hold shift + click to select a range
85c37f6

wip(runtime): partition /ready driver health into primary vs secondar…


os-steve Aug 31, 2026
e8bb34a

test(objectql): pin system-object-unbound to the reachable zero-drive…


os-steve Aug 31, 2026
ae226f6

chore: changeset for #13408


os-steve Aug 31, 2026
cbbc8d0

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
6a07781

docs(permissions): re-anchor the system-context census after the engi…


os-steve Aug 31, 2026
0e2fdb8

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
2aa73e8

fix(test): type the driver double and narrow the dispatch result so t…


os-steve Aug 31, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


65 changes: 65 additions & 0 deletions
.changeset/ready-primary-datasource-drain.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
---
"@objectstack/objectql": minor
"@objectstack/runtime": minor
---

fix(runtime,objectql): `/api/v1/ready` drains only on the PRIMARY datasource's failure; a secondary/tenant datasource is reported, not drained (#13408)

On a multi-datasource deployment, one datasource whose driver could not start
pinned `/api/v1/ready` to 503 on **every replica** — so a readiness-checked load
balancer drained every upstream and took the whole deployment offline, while
Postgres and the app itself were healthy (`/api/v1/health` 200, direct reads
working). One tenant's misconfiguration = total outage. Observed on a live
3-replica EE deployment and recovered only by restarting every process.

Ruled 2026-08-31 (第 6 场总监席决裁批 #12, maintainer verbatim 「同意」), Option B:

> `/api/v1/ready` 只在**主/默认数据源**不健康时摘流量;次要/租户数据源的故障照常
> **上报**(`/ready` 响应 body、日志、告警)但不 drain 节点。

**What changed.** When a driver reports itself unhealthy, `/ready` now asks
which datasource is the deployment's primary one before choosing a status:

- primary unhealthy, or the criterion unresolvable ⇒ **503**, unchanged envelope;
- primary healthy, only a secondary down ⇒ **200**, with the failed drivers named
 in a new `degraded` block: `{ status, state, degraded: { drivers, primaryDatasource } }`.

The failed driver is never hidden — the rejected alternative of filtering it out
of the response stays rejected. `degraded` appears **only** on that branch; an
all-healthy 200 body is byte-identical to before.

**The criterion is a readable fact, not a heuristic.** `ObjectQL.resolvePrimaryDatasource()`
(new, exported with `PrimaryDatasourceVerdict` / `PrimaryDatasourceUnresolvedReason`)
answers *where this deployment's platform system objects actually live*, resolved
through the same five-step routing order every query uses — never registration
order, never the driver flagged default at registration. The ADR-0057 §3.6 system
ledgers (`audit` / `telemetry` / `event`) are excluded because they are
deliberately routed off the primary. Disagreement, silence, or a name with no
driver behind it return an **unresolved verdict**, never a guess.

**Fail toward draining.** Every way of not knowing — an engine that predates the
probe, a probe that throws, a malformed verdict, a genuinely split deployment —
falls back to the old whole-node 503. Staying in rotation requires a *positive*
reading; the absence of a negative one is not permission. Pinned in both
directions, including an inversion ablation.

**framework#3756 is not overturned.** Its quantified reason — "a replica that
would fail 100% of its requests" — still holds in the single-datasource
deployment it was measured on, where the primary *is* the only source; that
deployment's answer is unchanged down to the response body. What #3756's
reasoning never covered is the multi-datasource shape, and that is the only
branch this carves out.

**Grade: `minor` for both, proposed rather than assumed** — this changes when a
published operational probe drains a node, so an operator whose alerting keys on
`/ready` returning 503 for *any* driver failure will see 200 + `degraded`
instead, and `degraded` is a new response field. Nothing is removed or renamed,
no declared contract key is added (the ruling explicitly declines that: 「⛔ 本裁不
加契约键」), single-datasource behaviour is bit-identical, and no migration is
required — so `major` overstates it and `patch` understates a deliberate change
to an availability control surface.

⚠️ Out of scope, tracked separately as **#13578**: `DELETE` of a datasource still
does not evict the stuck driver from the in-memory engine registry, so the
datasource keeps appearing in `/ready`'s report until the process restarts. That
half is a defect under either answer to this card and is queued independently.

24 changes: 12 additions & 12 deletions
content/docs/permissions/system-context.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -109,18 +109,18 @@ that silently does not happen.

| # | Behaviour when `isSystem` | Package | What you get / what you lose | Anchor |
|:--|:---|:---|:---|:---|
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10616` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10778` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9509` |
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10712` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10874` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9605` |
| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1576` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9546`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5638` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3573`, `:3583`, `:3610` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9642`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5639` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3574`, `:3584`, `:3611` |
| 25 | Engine-owned / append-only write guard bypassed | plugin-security | Get: generic writes to `managedBy` engine-owned objects | `system-write-guard.ts:96`, `:120` |
| 26 | Identity write guard bypassed (ADR-0092) | plugin-auth | Get: direct writes to identity tables through the generic data path | `identity-write-guard.ts:98` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6336` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11364` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11293` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6337` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11460` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11389` |

### 3. Sharing (`plugin-sharing`)

Expand DownExpand Up@@ -179,8 +179,8 @@ a reader tracing where elevation travels needs them.

| # | Site | Package | What it does |
|:--|:---|:---|:---|
| 62 | `objectql/src/engine.ts:3405` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13705` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 62 | `objectql/src/engine.ts:3406` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13801` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 64 | `plugin-reports/src/report-service.ts:556` | plugin-reports | Threads the flag into the engine call that runs a report |
| 65 | `body-runner.ts:279` | runtime | Rebuilds an `ExecutionContext` from a hook session, carrying the flag across |

Expand All@@ -195,7 +195,7 @@ assuming `isSystem` covers it is a documented source of bugs.
|:---|:---|:---|
| "It suppresses triggers / record-change automation" | **No.** Only `skipTriggers` does. A bare `{ isSystem: true }` on a seed write re-fired automation on freshly seeded rows and wedged first boot | `metadata-protocol/src/seed-loader.ts:1909` (rationale at `:1819`–`1821`, #3760), `flow.zod.ts:685` |
| "It skips the state machine" | **No.** That is `skipStateMachine`, carried by seed replay and by `treatAsHistorical` imports | `objectql/src/engine.ts` FSM gate; see [State Machine](/docs/protocol/objectql/state-machine) |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9492`–`9509` |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9588`–`9605` |
| "It preserves a supplied `updated_at` / `updated_by`" | **No.** That is `preserveAudit`, a separate opt-in — and an UPDATE-path exemption only | `field.zod.ts:1451` (#3493 / #6640) |
| "It stamps `created_by`" | **No.** Audit stamping reads `userId` from the context. A user-less system write stamps nothing — that is today's behaviour, not an error | `runtime-identity.ts:280`–`281` |
| "It bypasses every guard" | **No.** The last-admin guard applies to **every** context, `isSystem` included — the deprovision path that actually locks an org out is the system one | `last-admin-guard.ts:286` |
Expand Down

48 changes: 48 additions & 0 deletions
packages/objectql/src/driver-connect-errors.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,6 +19,54 @@ export interface DriverHealth {
 skipped?: boolean;
}

/**
 * Why {@link PrimaryDatasourceVerdict} could not name a primary datasource.
 *
 * Every member means the SAME thing to a readiness probe — *we cannot tell* —
 * and the ruling on #13408 fixes what a probe must do with that answer: fall
 * back to the old whole-node 503 (fail toward draining), never to "don't
 * drain". They are distinguished only so the reason can be logged and reported;
 * ⛔ a caller must not branch on them to keep a replica in rotation.
 */
export type PrimaryDatasourceUnresolvedReason =
 /** No platform system object is registered here — nothing to read the fact off. */
 | 'no-system-objects-registered'
 /** The platform's system objects are split across more than one datasource. */
 | 'system-objects-split'
 /** A registered system object routes nowhere: no binding and no default driver. */
 | 'system-object-unbound'
 /** The name resolved, but no driver is registered under it, so nothing probes it. */
 | 'no-driver-registered';

/**
 * WHICH datasource carries this deployment's platform system objects — the
 * machine-readable "primary/default datasource" fact the #13408 ruling requires
 * (2026-08-31, 第 6 场总监席决裁批 #12), stated as a verdict rather than a
 * `string | undefined` so "we could not tell" can never be mistaken for a name.
 *
 * ⛔ The ruling forbids deriving this from a heuristic such as "the first
 * datasource registered". The one implementation is
 * `ObjectQL.resolvePrimaryDatasource()`.
 */
export type PrimaryDatasourceVerdict =
 | {
 resolved: true;
 /** The datasource name, which is also a registered driver's name. */
 datasource: string;
 /**
 * How many registered platform system objects agreed on it. Never 0 — a
 * verdict read off nothing is `no-system-objects-registered`, not a name,
 * so a caller cannot act on a vacuous agreement.
 */
 witnesses: number;
 }
 | {
 resolved: false;
 reason: PrimaryDatasourceUnresolvedReason;
 /** The distinct datasource names seen, when more than one disagreed. */
 candidates?: readonly string[];
 };

/** `error.message` when it is an Error, its string form otherwise. */
function failureMessage(error: unknown): string {
 if (error instanceof Error) return error.message || error.name;
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained by os-steve · Pull Request #13763 · objectstack-ai/objectstack · GitHub
Skip to content
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained#13763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-steve
merged 7 commits into

from

Aug 31, 2026
 Merged


fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained
#13763
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


7 commits


Select commit
Hold shift + click to select a range
85c37f6

wip(runtime): partition /ready driver health into primary vs secondar…


os-steve Aug 31, 2026
e8bb34a

test(objectql): pin system-object-unbound to the reachable zero-drive…


os-steve Aug 31, 2026
ae226f6

chore: changeset for #13408


os-steve Aug 31, 2026
cbbc8d0

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
6a07781

docs(permissions): re-anchor the system-context census after the engi…


os-steve Aug 31, 2026
0e2fdb8

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
2aa73e8

fix(test): type the driver double and narrow the dispatch result so t…


os-steve Aug 31, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


65 changes: 65 additions & 0 deletions
.changeset/ready-primary-datasource-drain.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
---
"@objectstack/objectql": minor
"@objectstack/runtime": minor
---

fix(runtime,objectql): `/api/v1/ready` drains only on the PRIMARY datasource's failure; a secondary/tenant datasource is reported, not drained (#13408)

On a multi-datasource deployment, one datasource whose driver could not start
pinned `/api/v1/ready` to 503 on **every replica** — so a readiness-checked load
balancer drained every upstream and took the whole deployment offline, while
Postgres and the app itself were healthy (`/api/v1/health` 200, direct reads
working). One tenant's misconfiguration = total outage. Observed on a live
3-replica EE deployment and recovered only by restarting every process.

Ruled 2026-08-31 (第 6 场总监席决裁批 #12, maintainer verbatim 「同意」), Option B:

> `/api/v1/ready` 只在**主/默认数据源**不健康时摘流量;次要/租户数据源的故障照常
> **上报**(`/ready` 响应 body、日志、告警)但不 drain 节点。

**What changed.** When a driver reports itself unhealthy, `/ready` now asks
which datasource is the deployment's primary one before choosing a status:

- primary unhealthy, or the criterion unresolvable ⇒ **503**, unchanged envelope;
- primary healthy, only a secondary down ⇒ **200**, with the failed drivers named
 in a new `degraded` block: `{ status, state, degraded: { drivers, primaryDatasource } }`.

The failed driver is never hidden — the rejected alternative of filtering it out
of the response stays rejected. `degraded` appears **only** on that branch; an
all-healthy 200 body is byte-identical to before.

**The criterion is a readable fact, not a heuristic.** `ObjectQL.resolvePrimaryDatasource()`
(new, exported with `PrimaryDatasourceVerdict` / `PrimaryDatasourceUnresolvedReason`)
answers *where this deployment's platform system objects actually live*, resolved
through the same five-step routing order every query uses — never registration
order, never the driver flagged default at registration. The ADR-0057 §3.6 system
ledgers (`audit` / `telemetry` / `event`) are excluded because they are
deliberately routed off the primary. Disagreement, silence, or a name with no
driver behind it return an **unresolved verdict**, never a guess.

**Fail toward draining.** Every way of not knowing — an engine that predates the
probe, a probe that throws, a malformed verdict, a genuinely split deployment —
falls back to the old whole-node 503. Staying in rotation requires a *positive*
reading; the absence of a negative one is not permission. Pinned in both
directions, including an inversion ablation.

**framework#3756 is not overturned.** Its quantified reason — "a replica that
would fail 100% of its requests" — still holds in the single-datasource
deployment it was measured on, where the primary *is* the only source; that
deployment's answer is unchanged down to the response body. What #3756's
reasoning never covered is the multi-datasource shape, and that is the only
branch this carves out.

**Grade: `minor` for both, proposed rather than assumed** — this changes when a
published operational probe drains a node, so an operator whose alerting keys on
`/ready` returning 503 for *any* driver failure will see 200 + `degraded`
instead, and `degraded` is a new response field. Nothing is removed or renamed,
no declared contract key is added (the ruling explicitly declines that: 「⛔ 本裁不
加契约键」), single-datasource behaviour is bit-identical, and no migration is
required — so `major` overstates it and `patch` understates a deliberate change
to an availability control surface.

⚠️ Out of scope, tracked separately as **#13578**: `DELETE` of a datasource still
does not evict the stuck driver from the in-memory engine registry, so the
datasource keeps appearing in `/ready`'s report until the process restarts. That
half is a defect under either answer to this card and is queued independently.

24 changes: 12 additions & 12 deletions
content/docs/permissions/system-context.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -109,18 +109,18 @@ that silently does not happen.

| # | Behaviour when `isSystem` | Package | What you get / what you lose | Anchor |
|:--|:---|:---|:---|:---|
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10616` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10778` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9509` |
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10712` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10874` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9605` |
| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1576` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9546`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5638` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3573`, `:3583`, `:3610` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9642`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5639` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3574`, `:3584`, `:3611` |
| 25 | Engine-owned / append-only write guard bypassed | plugin-security | Get: generic writes to `managedBy` engine-owned objects | `system-write-guard.ts:96`, `:120` |
| 26 | Identity write guard bypassed (ADR-0092) | plugin-auth | Get: direct writes to identity tables through the generic data path | `identity-write-guard.ts:98` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6336` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11364` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11293` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6337` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11460` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11389` |

### 3. Sharing (`plugin-sharing`)

Expand DownExpand Up@@ -179,8 +179,8 @@ a reader tracing where elevation travels needs them.

| # | Site | Package | What it does |
|:--|:---|:---|:---|
| 62 | `objectql/src/engine.ts:3405` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13705` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 62 | `objectql/src/engine.ts:3406` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13801` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 64 | `plugin-reports/src/report-service.ts:556` | plugin-reports | Threads the flag into the engine call that runs a report |
| 65 | `body-runner.ts:279` | runtime | Rebuilds an `ExecutionContext` from a hook session, carrying the flag across |

Expand All@@ -195,7 +195,7 @@ assuming `isSystem` covers it is a documented source of bugs.
|:---|:---|:---|
| "It suppresses triggers / record-change automation" | **No.** Only `skipTriggers` does. A bare `{ isSystem: true }` on a seed write re-fired automation on freshly seeded rows and wedged first boot | `metadata-protocol/src/seed-loader.ts:1909` (rationale at `:1819`–`1821`, #3760), `flow.zod.ts:685` |
| "It skips the state machine" | **No.** That is `skipStateMachine`, carried by seed replay and by `treatAsHistorical` imports | `objectql/src/engine.ts` FSM gate; see [State Machine](/docs/protocol/objectql/state-machine) |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9492`–`9509` |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9588`–`9605` |
| "It preserves a supplied `updated_at` / `updated_by`" | **No.** That is `preserveAudit`, a separate opt-in — and an UPDATE-path exemption only | `field.zod.ts:1451` (#3493 / #6640) |
| "It stamps `created_by`" | **No.** Audit stamping reads `userId` from the context. A user-less system write stamps nothing — that is today's behaviour, not an error | `runtime-identity.ts:280`–`281` |
| "It bypasses every guard" | **No.** The last-admin guard applies to **every** context, `isSystem` included — the deprovision path that actually locks an org out is the system one | `last-admin-guard.ts:286` |
Expand Down

48 changes: 48 additions & 0 deletions
packages/objectql/src/driver-connect-errors.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,6 +19,54 @@ export interface DriverHealth {
 skipped?: boolean;
}

/**
 * Why {@link PrimaryDatasourceVerdict} could not name a primary datasource.
 *
 * Every member means the SAME thing to a readiness probe — *we cannot tell* —
 * and the ruling on #13408 fixes what a probe must do with that answer: fall
 * back to the old whole-node 503 (fail toward draining), never to "don't
 * drain". They are distinguished only so the reason can be logged and reported;
 * ⛔ a caller must not branch on them to keep a replica in rotation.
 */
export type PrimaryDatasourceUnresolvedReason =
 /** No platform system object is registered here — nothing to read the fact off. */
 | 'no-system-objects-registered'
 /** The platform's system objects are split across more than one datasource. */
 | 'system-objects-split'
 /** A registered system object routes nowhere: no binding and no default driver. */
 | 'system-object-unbound'
 /** The name resolved, but no driver is registered under it, so nothing probes it. */
 | 'no-driver-registered';

/**
 * WHICH datasource carries this deployment's platform system objects — the
 * machine-readable "primary/default datasource" fact the #13408 ruling requires
 * (2026-08-31, 第 6 场总监席决裁批 #12), stated as a verdict rather than a
 * `string | undefined` so "we could not tell" can never be mistaken for a name.
 *
 * ⛔ The ruling forbids deriving this from a heuristic such as "the first
 * datasource registered". The one implementation is
 * `ObjectQL.resolvePrimaryDatasource()`.
 */
export type PrimaryDatasourceVerdict =
 | {
 resolved: true;
 /** The datasource name, which is also a registered driver's name. */
 datasource: string;
 /**
 * How many registered platform system objects agreed on it. Never 0 — a
 * verdict read off nothing is `no-system-objects-registered`, not a name,
 * so a caller cannot act on a vacuous agreement.
 */
 witnesses: number;
 }
 | {
 resolved: false;
 reason: PrimaryDatasourceUnresolvedReason;
 /** The distinct datasource names seen, when more than one disagreed. */
 candidates?: readonly string[];
 };

/** `error.message` when it is an Error, its string form otherwise. */
function failureMessage(error: unknown): string {
 if (error instanceof Error) return error.message || error.name;
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained by os-steve · Pull Request #13763 · objectstack-ai/objectstack · GitHub
Skip to content
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained#13763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-steve
merged 7 commits into

from

Aug 31, 2026
 Merged


fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained
#13763
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


7 commits


Select commit
Hold shift + click to select a range
85c37f6

wip(runtime): partition /ready driver health into primary vs secondar…


os-steve Aug 31, 2026
e8bb34a

test(objectql): pin system-object-unbound to the reachable zero-drive…


os-steve Aug 31, 2026
ae226f6

chore: changeset for #13408


os-steve Aug 31, 2026
cbbc8d0

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
6a07781

docs(permissions): re-anchor the system-context census after the engi…


os-steve Aug 31, 2026
0e2fdb8

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
2aa73e8

fix(test): type the driver double and narrow the dispatch result so t…


os-steve Aug 31, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


65 changes: 65 additions & 0 deletions
.changeset/ready-primary-datasource-drain.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
---
"@objectstack/objectql": minor
"@objectstack/runtime": minor
---

fix(runtime,objectql): `/api/v1/ready` drains only on the PRIMARY datasource's failure; a secondary/tenant datasource is reported, not drained (#13408)

On a multi-datasource deployment, one datasource whose driver could not start
pinned `/api/v1/ready` to 503 on **every replica** — so a readiness-checked load
balancer drained every upstream and took the whole deployment offline, while
Postgres and the app itself were healthy (`/api/v1/health` 200, direct reads
working). One tenant's misconfiguration = total outage. Observed on a live
3-replica EE deployment and recovered only by restarting every process.

Ruled 2026-08-31 (第 6 场总监席决裁批 #12, maintainer verbatim 「同意」), Option B:

> `/api/v1/ready` 只在**主/默认数据源**不健康时摘流量;次要/租户数据源的故障照常
> **上报**(`/ready` 响应 body、日志、告警)但不 drain 节点。

**What changed.** When a driver reports itself unhealthy, `/ready` now asks
which datasource is the deployment's primary one before choosing a status:

- primary unhealthy, or the criterion unresolvable ⇒ **503**, unchanged envelope;
- primary healthy, only a secondary down ⇒ **200**, with the failed drivers named
 in a new `degraded` block: `{ status, state, degraded: { drivers, primaryDatasource } }`.

The failed driver is never hidden — the rejected alternative of filtering it out
of the response stays rejected. `degraded` appears **only** on that branch; an
all-healthy 200 body is byte-identical to before.

**The criterion is a readable fact, not a heuristic.** `ObjectQL.resolvePrimaryDatasource()`
(new, exported with `PrimaryDatasourceVerdict` / `PrimaryDatasourceUnresolvedReason`)
answers *where this deployment's platform system objects actually live*, resolved
through the same five-step routing order every query uses — never registration
order, never the driver flagged default at registration. The ADR-0057 §3.6 system
ledgers (`audit` / `telemetry` / `event`) are excluded because they are
deliberately routed off the primary. Disagreement, silence, or a name with no
driver behind it return an **unresolved verdict**, never a guess.

**Fail toward draining.** Every way of not knowing — an engine that predates the
probe, a probe that throws, a malformed verdict, a genuinely split deployment —
falls back to the old whole-node 503. Staying in rotation requires a *positive*
reading; the absence of a negative one is not permission. Pinned in both
directions, including an inversion ablation.

**framework#3756 is not overturned.** Its quantified reason — "a replica that
would fail 100% of its requests" — still holds in the single-datasource
deployment it was measured on, where the primary *is* the only source; that
deployment's answer is unchanged down to the response body. What #3756's
reasoning never covered is the multi-datasource shape, and that is the only
branch this carves out.

**Grade: `minor` for both, proposed rather than assumed** — this changes when a
published operational probe drains a node, so an operator whose alerting keys on
`/ready` returning 503 for *any* driver failure will see 200 + `degraded`
instead, and `degraded` is a new response field. Nothing is removed or renamed,
no declared contract key is added (the ruling explicitly declines that: 「⛔ 本裁不
加契约键」), single-datasource behaviour is bit-identical, and no migration is
required — so `major` overstates it and `patch` understates a deliberate change
to an availability control surface.

⚠️ Out of scope, tracked separately as **#13578**: `DELETE` of a datasource still
does not evict the stuck driver from the in-memory engine registry, so the
datasource keeps appearing in `/ready`'s report until the process restarts. That
half is a defect under either answer to this card and is queued independently.

24 changes: 12 additions & 12 deletions
content/docs/permissions/system-context.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -109,18 +109,18 @@ that silently does not happen.

| # | Behaviour when `isSystem` | Package | What you get / what you lose | Anchor |
|:--|:---|:---|:---|:---|
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10616` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10778` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9509` |
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10712` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10874` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9605` |
| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1576` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9546`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5638` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3573`, `:3583`, `:3610` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9642`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5639` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3574`, `:3584`, `:3611` |
| 25 | Engine-owned / append-only write guard bypassed | plugin-security | Get: generic writes to `managedBy` engine-owned objects | `system-write-guard.ts:96`, `:120` |
| 26 | Identity write guard bypassed (ADR-0092) | plugin-auth | Get: direct writes to identity tables through the generic data path | `identity-write-guard.ts:98` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6336` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11364` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11293` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6337` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11460` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11389` |

### 3. Sharing (`plugin-sharing`)

Expand DownExpand Up@@ -179,8 +179,8 @@ a reader tracing where elevation travels needs them.

| # | Site | Package | What it does |
|:--|:---|:---|:---|
| 62 | `objectql/src/engine.ts:3405` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13705` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 62 | `objectql/src/engine.ts:3406` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13801` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 64 | `plugin-reports/src/report-service.ts:556` | plugin-reports | Threads the flag into the engine call that runs a report |
| 65 | `body-runner.ts:279` | runtime | Rebuilds an `ExecutionContext` from a hook session, carrying the flag across |

Expand All@@ -195,7 +195,7 @@ assuming `isSystem` covers it is a documented source of bugs.
|:---|:---|:---|
| "It suppresses triggers / record-change automation" | **No.** Only `skipTriggers` does. A bare `{ isSystem: true }` on a seed write re-fired automation on freshly seeded rows and wedged first boot | `metadata-protocol/src/seed-loader.ts:1909` (rationale at `:1819`–`1821`, #3760), `flow.zod.ts:685` |
| "It skips the state machine" | **No.** That is `skipStateMachine`, carried by seed replay and by `treatAsHistorical` imports | `objectql/src/engine.ts` FSM gate; see [State Machine](/docs/protocol/objectql/state-machine) |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9492`–`9509` |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9588`–`9605` |
| "It preserves a supplied `updated_at` / `updated_by`" | **No.** That is `preserveAudit`, a separate opt-in — and an UPDATE-path exemption only | `field.zod.ts:1451` (#3493 / #6640) |
| "It stamps `created_by`" | **No.** Audit stamping reads `userId` from the context. A user-less system write stamps nothing — that is today's behaviour, not an error | `runtime-identity.ts:280`–`281` |
| "It bypasses every guard" | **No.** The last-admin guard applies to **every** context, `isSystem` included — the deprovision path that actually locks an org out is the system one | `last-admin-guard.ts:286` |
Expand Down

48 changes: 48 additions & 0 deletions
packages/objectql/src/driver-connect-errors.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,6 +19,54 @@ export interface DriverHealth {
 skipped?: boolean;
}

/**
 * Why {@link PrimaryDatasourceVerdict} could not name a primary datasource.
 *
 * Every member means the SAME thing to a readiness probe — *we cannot tell* —
 * and the ruling on #13408 fixes what a probe must do with that answer: fall
 * back to the old whole-node 503 (fail toward draining), never to "don't
 * drain". They are distinguished only so the reason can be logged and reported;
 * ⛔ a caller must not branch on them to keep a replica in rotation.
 */
export type PrimaryDatasourceUnresolvedReason =
 /** No platform system object is registered here — nothing to read the fact off. */
 | 'no-system-objects-registered'
 /** The platform's system objects are split across more than one datasource. */
 | 'system-objects-split'
 /** A registered system object routes nowhere: no binding and no default driver. */
 | 'system-object-unbound'
 /** The name resolved, but no driver is registered under it, so nothing probes it. */
 | 'no-driver-registered';

/**
 * WHICH datasource carries this deployment's platform system objects — the
 * machine-readable "primary/default datasource" fact the #13408 ruling requires
 * (2026-08-31, 第 6 场总监席决裁批 #12), stated as a verdict rather than a
 * `string | undefined` so "we could not tell" can never be mistaken for a name.
 *
 * ⛔ The ruling forbids deriving this from a heuristic such as "the first
 * datasource registered". The one implementation is
 * `ObjectQL.resolvePrimaryDatasource()`.
 */
export type PrimaryDatasourceVerdict =
 | {
 resolved: true;
 /** The datasource name, which is also a registered driver's name. */
 datasource: string;
 /**
 * How many registered platform system objects agreed on it. Never 0 — a
 * verdict read off nothing is `no-system-objects-registered`, not a name,
 * so a caller cannot act on a vacuous agreement.
 */
 witnesses: number;
 }
 | {
 resolved: false;
 reason: PrimaryDatasourceUnresolvedReason;
 /** The distinct datasource names seen, when more than one disagreed. */
 candidates?: readonly string[];
 };

/** `error.message` when it is an Error, its string form otherwise. */
function failureMessage(error: unknown): string {
 if (error instanceof Error) return error.message || error.name;
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained by os-steve · Pull Request #13763 · objectstack-ai/objectstack · GitHub
Skip to content
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained#13763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-steve
merged 7 commits into

from

Aug 31, 2026
 Merged


fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained
#13763
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


7 commits


Select commit
Hold shift + click to select a range
85c37f6

wip(runtime): partition /ready driver health into primary vs secondar…


os-steve Aug 31, 2026
e8bb34a

test(objectql): pin system-object-unbound to the reachable zero-drive…


os-steve Aug 31, 2026
ae226f6

chore: changeset for #13408


os-steve Aug 31, 2026
cbbc8d0

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
6a07781

docs(permissions): re-anchor the system-context census after the engi…


os-steve Aug 31, 2026
0e2fdb8

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
2aa73e8

fix(test): type the driver double and narrow the dispatch result so t…


os-steve Aug 31, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


65 changes: 65 additions & 0 deletions
.changeset/ready-primary-datasource-drain.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
---
"@objectstack/objectql": minor
"@objectstack/runtime": minor
---

fix(runtime,objectql): `/api/v1/ready` drains only on the PRIMARY datasource's failure; a secondary/tenant datasource is reported, not drained (#13408)

On a multi-datasource deployment, one datasource whose driver could not start
pinned `/api/v1/ready` to 503 on **every replica** — so a readiness-checked load
balancer drained every upstream and took the whole deployment offline, while
Postgres and the app itself were healthy (`/api/v1/health` 200, direct reads
working). One tenant's misconfiguration = total outage. Observed on a live
3-replica EE deployment and recovered only by restarting every process.

Ruled 2026-08-31 (第 6 场总监席决裁批 #12, maintainer verbatim 「同意」), Option B:

> `/api/v1/ready` 只在**主/默认数据源**不健康时摘流量;次要/租户数据源的故障照常
> **上报**(`/ready` 响应 body、日志、告警)但不 drain 节点。

**What changed.** When a driver reports itself unhealthy, `/ready` now asks
which datasource is the deployment's primary one before choosing a status:

- primary unhealthy, or the criterion unresolvable ⇒ **503**, unchanged envelope;
- primary healthy, only a secondary down ⇒ **200**, with the failed drivers named
 in a new `degraded` block: `{ status, state, degraded: { drivers, primaryDatasource } }`.

The failed driver is never hidden — the rejected alternative of filtering it out
of the response stays rejected. `degraded` appears **only** on that branch; an
all-healthy 200 body is byte-identical to before.

**The criterion is a readable fact, not a heuristic.** `ObjectQL.resolvePrimaryDatasource()`
(new, exported with `PrimaryDatasourceVerdict` / `PrimaryDatasourceUnresolvedReason`)
answers *where this deployment's platform system objects actually live*, resolved
through the same five-step routing order every query uses — never registration
order, never the driver flagged default at registration. The ADR-0057 §3.6 system
ledgers (`audit` / `telemetry` / `event`) are excluded because they are
deliberately routed off the primary. Disagreement, silence, or a name with no
driver behind it return an **unresolved verdict**, never a guess.

**Fail toward draining.** Every way of not knowing — an engine that predates the
probe, a probe that throws, a malformed verdict, a genuinely split deployment —
falls back to the old whole-node 503. Staying in rotation requires a *positive*
reading; the absence of a negative one is not permission. Pinned in both
directions, including an inversion ablation.

**framework#3756 is not overturned.** Its quantified reason — "a replica that
would fail 100% of its requests" — still holds in the single-datasource
deployment it was measured on, where the primary *is* the only source; that
deployment's answer is unchanged down to the response body. What #3756's
reasoning never covered is the multi-datasource shape, and that is the only
branch this carves out.

**Grade: `minor` for both, proposed rather than assumed** — this changes when a
published operational probe drains a node, so an operator whose alerting keys on
`/ready` returning 503 for *any* driver failure will see 200 + `degraded`
instead, and `degraded` is a new response field. Nothing is removed or renamed,
no declared contract key is added (the ruling explicitly declines that: 「⛔ 本裁不
加契约键」), single-datasource behaviour is bit-identical, and no migration is
required — so `major` overstates it and `patch` understates a deliberate change
to an availability control surface.

⚠️ Out of scope, tracked separately as **#13578**: `DELETE` of a datasource still
does not evict the stuck driver from the in-memory engine registry, so the
datasource keeps appearing in `/ready`'s report until the process restarts. That
half is a defect under either answer to this card and is queued independently.

24 changes: 12 additions & 12 deletions
content/docs/permissions/system-context.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -109,18 +109,18 @@ that silently does not happen.

| # | Behaviour when `isSystem` | Package | What you get / what you lose | Anchor |
|:--|:---|:---|:---|:---|
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10616` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10778` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9509` |
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10712` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10874` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9605` |
| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1576` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9546`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5638` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3573`, `:3583`, `:3610` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9642`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5639` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3574`, `:3584`, `:3611` |
| 25 | Engine-owned / append-only write guard bypassed | plugin-security | Get: generic writes to `managedBy` engine-owned objects | `system-write-guard.ts:96`, `:120` |
| 26 | Identity write guard bypassed (ADR-0092) | plugin-auth | Get: direct writes to identity tables through the generic data path | `identity-write-guard.ts:98` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6336` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11364` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11293` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6337` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11460` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11389` |

### 3. Sharing (`plugin-sharing`)

Expand DownExpand Up@@ -179,8 +179,8 @@ a reader tracing where elevation travels needs them.

| # | Site | Package | What it does |
|:--|:---|:---|:---|
| 62 | `objectql/src/engine.ts:3405` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13705` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 62 | `objectql/src/engine.ts:3406` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13801` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 64 | `plugin-reports/src/report-service.ts:556` | plugin-reports | Threads the flag into the engine call that runs a report |
| 65 | `body-runner.ts:279` | runtime | Rebuilds an `ExecutionContext` from a hook session, carrying the flag across |

Expand All@@ -195,7 +195,7 @@ assuming `isSystem` covers it is a documented source of bugs.
|:---|:---|:---|
| "It suppresses triggers / record-change automation" | **No.** Only `skipTriggers` does. A bare `{ isSystem: true }` on a seed write re-fired automation on freshly seeded rows and wedged first boot | `metadata-protocol/src/seed-loader.ts:1909` (rationale at `:1819`–`1821`, #3760), `flow.zod.ts:685` |
| "It skips the state machine" | **No.** That is `skipStateMachine`, carried by seed replay and by `treatAsHistorical` imports | `objectql/src/engine.ts` FSM gate; see [State Machine](/docs/protocol/objectql/state-machine) |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9492`–`9509` |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9588`–`9605` |
| "It preserves a supplied `updated_at` / `updated_by`" | **No.** That is `preserveAudit`, a separate opt-in — and an UPDATE-path exemption only | `field.zod.ts:1451` (#3493 / #6640) |
| "It stamps `created_by`" | **No.** Audit stamping reads `userId` from the context. A user-less system write stamps nothing — that is today's behaviour, not an error | `runtime-identity.ts:280`–`281` |
| "It bypasses every guard" | **No.** The last-admin guard applies to **every** context, `isSystem` included — the deprovision path that actually locks an org out is the system one | `last-admin-guard.ts:286` |
Expand Down

48 changes: 48 additions & 0 deletions
packages/objectql/src/driver-connect-errors.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,6 +19,54 @@ export interface DriverHealth {
 skipped?: boolean;
}

/**
 * Why {@link PrimaryDatasourceVerdict} could not name a primary datasource.
 *
 * Every member means the SAME thing to a readiness probe — *we cannot tell* —
 * and the ruling on #13408 fixes what a probe must do with that answer: fall
 * back to the old whole-node 503 (fail toward draining), never to "don't
 * drain". They are distinguished only so the reason can be logged and reported;
 * ⛔ a caller must not branch on them to keep a replica in rotation.
 */
export type PrimaryDatasourceUnresolvedReason =
 /** No platform system object is registered here — nothing to read the fact off. */
 | 'no-system-objects-registered'
 /** The platform's system objects are split across more than one datasource. */
 | 'system-objects-split'
 /** A registered system object routes nowhere: no binding and no default driver. */
 | 'system-object-unbound'
 /** The name resolved, but no driver is registered under it, so nothing probes it. */
 | 'no-driver-registered';

/**
 * WHICH datasource carries this deployment's platform system objects — the
 * machine-readable "primary/default datasource" fact the #13408 ruling requires
 * (2026-08-31, 第 6 场总监席决裁批 #12), stated as a verdict rather than a
 * `string | undefined` so "we could not tell" can never be mistaken for a name.
 *
 * ⛔ The ruling forbids deriving this from a heuristic such as "the first
 * datasource registered". The one implementation is
 * `ObjectQL.resolvePrimaryDatasource()`.
 */
export type PrimaryDatasourceVerdict =
 | {
 resolved: true;
 /** The datasource name, which is also a registered driver's name. */
 datasource: string;
 /**
 * How many registered platform system objects agreed on it. Never 0 — a
 * verdict read off nothing is `no-system-objects-registered`, not a name,
 * so a caller cannot act on a vacuous agreement.
 */
 witnesses: number;
 }
 | {
 resolved: false;
 reason: PrimaryDatasourceUnresolvedReason;
 /** The distinct datasource names seen, when more than one disagreed. */
 candidates?: readonly string[];
 };

/** `error.message` when it is an Error, its string form otherwise. */
function failureMessage(error: unknown): string {
 if (error instanceof Error) return error.message || error.name;
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained by os-steve · Pull Request #13763 · objectstack-ai/objectstack · GitHub
Skip to content
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained#13763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-steve
merged 7 commits into

from

Aug 31, 2026
 Merged


fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained
#13763
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


7 commits


Select commit
Hold shift + click to select a range
85c37f6

wip(runtime): partition /ready driver health into primary vs secondar…


os-steve Aug 31, 2026
e8bb34a

test(objectql): pin system-object-unbound to the reachable zero-drive…


os-steve Aug 31, 2026
ae226f6

chore: changeset for #13408


os-steve Aug 31, 2026
cbbc8d0

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
6a07781

docs(permissions): re-anchor the system-context census after the engi…


os-steve Aug 31, 2026
0e2fdb8

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
2aa73e8

fix(test): type the driver double and narrow the dispatch result so t…


os-steve Aug 31, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


65 changes: 65 additions & 0 deletions
.changeset/ready-primary-datasource-drain.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
---
"@objectstack/objectql": minor
"@objectstack/runtime": minor
---

fix(runtime,objectql): `/api/v1/ready` drains only on the PRIMARY datasource's failure; a secondary/tenant datasource is reported, not drained (#13408)

On a multi-datasource deployment, one datasource whose driver could not start
pinned `/api/v1/ready` to 503 on **every replica** — so a readiness-checked load
balancer drained every upstream and took the whole deployment offline, while
Postgres and the app itself were healthy (`/api/v1/health` 200, direct reads
working). One tenant's misconfiguration = total outage. Observed on a live
3-replica EE deployment and recovered only by restarting every process.

Ruled 2026-08-31 (第 6 场总监席决裁批 #12, maintainer verbatim 「同意」), Option B:

> `/api/v1/ready` 只在**主/默认数据源**不健康时摘流量;次要/租户数据源的故障照常
> **上报**(`/ready` 响应 body、日志、告警)但不 drain 节点。

**What changed.** When a driver reports itself unhealthy, `/ready` now asks
which datasource is the deployment's primary one before choosing a status:

- primary unhealthy, or the criterion unresolvable ⇒ **503**, unchanged envelope;
- primary healthy, only a secondary down ⇒ **200**, with the failed drivers named
 in a new `degraded` block: `{ status, state, degraded: { drivers, primaryDatasource } }`.

The failed driver is never hidden — the rejected alternative of filtering it out
of the response stays rejected. `degraded` appears **only** on that branch; an
all-healthy 200 body is byte-identical to before.

**The criterion is a readable fact, not a heuristic.** `ObjectQL.resolvePrimaryDatasource()`
(new, exported with `PrimaryDatasourceVerdict` / `PrimaryDatasourceUnresolvedReason`)
answers *where this deployment's platform system objects actually live*, resolved
through the same five-step routing order every query uses — never registration
order, never the driver flagged default at registration. The ADR-0057 §3.6 system
ledgers (`audit` / `telemetry` / `event`) are excluded because they are
deliberately routed off the primary. Disagreement, silence, or a name with no
driver behind it return an **unresolved verdict**, never a guess.

**Fail toward draining.** Every way of not knowing — an engine that predates the
probe, a probe that throws, a malformed verdict, a genuinely split deployment —
falls back to the old whole-node 503. Staying in rotation requires a *positive*
reading; the absence of a negative one is not permission. Pinned in both
directions, including an inversion ablation.

**framework#3756 is not overturned.** Its quantified reason — "a replica that
would fail 100% of its requests" — still holds in the single-datasource
deployment it was measured on, where the primary *is* the only source; that
deployment's answer is unchanged down to the response body. What #3756's
reasoning never covered is the multi-datasource shape, and that is the only
branch this carves out.

**Grade: `minor` for both, proposed rather than assumed** — this changes when a
published operational probe drains a node, so an operator whose alerting keys on
`/ready` returning 503 for *any* driver failure will see 200 + `degraded`
instead, and `degraded` is a new response field. Nothing is removed or renamed,
no declared contract key is added (the ruling explicitly declines that: 「⛔ 本裁不
加契约键」), single-datasource behaviour is bit-identical, and no migration is
required — so `major` overstates it and `patch` understates a deliberate change
to an availability control surface.

⚠️ Out of scope, tracked separately as **#13578**: `DELETE` of a datasource still
does not evict the stuck driver from the in-memory engine registry, so the
datasource keeps appearing in `/ready`'s report until the process restarts. That
half is a defect under either answer to this card and is queued independently.

24 changes: 12 additions & 12 deletions
content/docs/permissions/system-context.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -109,18 +109,18 @@ that silently does not happen.

| # | Behaviour when `isSystem` | Package | What you get / what you lose | Anchor |
|:--|:---|:---|:---|:---|
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10616` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10778` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9509` |
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10712` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10874` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9605` |
| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1576` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9546`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5638` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3573`, `:3583`, `:3610` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9642`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5639` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3574`, `:3584`, `:3611` |
| 25 | Engine-owned / append-only write guard bypassed | plugin-security | Get: generic writes to `managedBy` engine-owned objects | `system-write-guard.ts:96`, `:120` |
| 26 | Identity write guard bypassed (ADR-0092) | plugin-auth | Get: direct writes to identity tables through the generic data path | `identity-write-guard.ts:98` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6336` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11364` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11293` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6337` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11460` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11389` |

### 3. Sharing (`plugin-sharing`)

Expand DownExpand Up@@ -179,8 +179,8 @@ a reader tracing where elevation travels needs them.

| # | Site | Package | What it does |
|:--|:---|:---|:---|
| 62 | `objectql/src/engine.ts:3405` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13705` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 62 | `objectql/src/engine.ts:3406` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13801` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 64 | `plugin-reports/src/report-service.ts:556` | plugin-reports | Threads the flag into the engine call that runs a report |
| 65 | `body-runner.ts:279` | runtime | Rebuilds an `ExecutionContext` from a hook session, carrying the flag across |

Expand All@@ -195,7 +195,7 @@ assuming `isSystem` covers it is a documented source of bugs.
|:---|:---|:---|
| "It suppresses triggers / record-change automation" | **No.** Only `skipTriggers` does. A bare `{ isSystem: true }` on a seed write re-fired automation on freshly seeded rows and wedged first boot | `metadata-protocol/src/seed-loader.ts:1909` (rationale at `:1819`–`1821`, #3760), `flow.zod.ts:685` |
| "It skips the state machine" | **No.** That is `skipStateMachine`, carried by seed replay and by `treatAsHistorical` imports | `objectql/src/engine.ts` FSM gate; see [State Machine](/docs/protocol/objectql/state-machine) |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9492`–`9509` |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9588`–`9605` |
| "It preserves a supplied `updated_at` / `updated_by`" | **No.** That is `preserveAudit`, a separate opt-in — and an UPDATE-path exemption only | `field.zod.ts:1451` (#3493 / #6640) |
| "It stamps `created_by`" | **No.** Audit stamping reads `userId` from the context. A user-less system write stamps nothing — that is today's behaviour, not an error | `runtime-identity.ts:280`–`281` |
| "It bypasses every guard" | **No.** The last-admin guard applies to **every** context, `isSystem` included — the deprovision path that actually locks an org out is the system one | `last-admin-guard.ts:286` |
Expand Down

48 changes: 48 additions & 0 deletions
packages/objectql/src/driver-connect-errors.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,6 +19,54 @@ export interface DriverHealth {
 skipped?: boolean;
}

/**
 * Why {@link PrimaryDatasourceVerdict} could not name a primary datasource.
 *
 * Every member means the SAME thing to a readiness probe — *we cannot tell* —
 * and the ruling on #13408 fixes what a probe must do with that answer: fall
 * back to the old whole-node 503 (fail toward draining), never to "don't
 * drain". They are distinguished only so the reason can be logged and reported;
 * ⛔ a caller must not branch on them to keep a replica in rotation.
 */
export type PrimaryDatasourceUnresolvedReason =
 /** No platform system object is registered here — nothing to read the fact off. */
 | 'no-system-objects-registered'
 /** The platform's system objects are split across more than one datasource. */
 | 'system-objects-split'
 /** A registered system object routes nowhere: no binding and no default driver. */
 | 'system-object-unbound'
 /** The name resolved, but no driver is registered under it, so nothing probes it. */
 | 'no-driver-registered';

/**
 * WHICH datasource carries this deployment's platform system objects — the
 * machine-readable "primary/default datasource" fact the #13408 ruling requires
 * (2026-08-31, 第 6 场总监席决裁批 #12), stated as a verdict rather than a
 * `string | undefined` so "we could not tell" can never be mistaken for a name.
 *
 * ⛔ The ruling forbids deriving this from a heuristic such as "the first
 * datasource registered". The one implementation is
 * `ObjectQL.resolvePrimaryDatasource()`.
 */
export type PrimaryDatasourceVerdict =
 | {
 resolved: true;
 /** The datasource name, which is also a registered driver's name. */
 datasource: string;
 /**
 * How many registered platform system objects agreed on it. Never 0 — a
 * verdict read off nothing is `no-system-objects-registered`, not a name,
 * so a caller cannot act on a vacuous agreement.
 */
 witnesses: number;
 }
 | {
 resolved: false;
 reason: PrimaryDatasourceUnresolvedReason;
 /** The distinct datasource names seen, when more than one disagreed. */
 candidates?: readonly string[];
 };

/** `error.message` when it is an Error, its string form otherwise. */
function failureMessage(error: unknown): string {
 if (error instanceof Error) return error.message || error.name;
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained by os-steve · Pull Request #13763 · objectstack-ai/objectstack · GitHub
Skip to content
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained#13763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-steve
merged 7 commits into

from

Aug 31, 2026
 Merged


fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained
#13763
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


7 commits


Select commit
Hold shift + click to select a range
85c37f6

wip(runtime): partition /ready driver health into primary vs secondar…


os-steve Aug 31, 2026
e8bb34a

test(objectql): pin system-object-unbound to the reachable zero-drive…


os-steve Aug 31, 2026
ae226f6

chore: changeset for #13408


os-steve Aug 31, 2026
cbbc8d0

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
6a07781

docs(permissions): re-anchor the system-context census after the engi…


os-steve Aug 31, 2026
0e2fdb8

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
2aa73e8

fix(test): type the driver double and narrow the dispatch result so t…


os-steve Aug 31, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


65 changes: 65 additions & 0 deletions
.changeset/ready-primary-datasource-drain.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
---
"@objectstack/objectql": minor
"@objectstack/runtime": minor
---

fix(runtime,objectql): `/api/v1/ready` drains only on the PRIMARY datasource's failure; a secondary/tenant datasource is reported, not drained (#13408)

On a multi-datasource deployment, one datasource whose driver could not start
pinned `/api/v1/ready` to 503 on **every replica** — so a readiness-checked load
balancer drained every upstream and took the whole deployment offline, while
Postgres and the app itself were healthy (`/api/v1/health` 200, direct reads
working). One tenant's misconfiguration = total outage. Observed on a live
3-replica EE deployment and recovered only by restarting every process.

Ruled 2026-08-31 (第 6 场总监席决裁批 #12, maintainer verbatim 「同意」), Option B:

> `/api/v1/ready` 只在**主/默认数据源**不健康时摘流量;次要/租户数据源的故障照常
> **上报**(`/ready` 响应 body、日志、告警)但不 drain 节点。

**What changed.** When a driver reports itself unhealthy, `/ready` now asks
which datasource is the deployment's primary one before choosing a status:

- primary unhealthy, or the criterion unresolvable ⇒ **503**, unchanged envelope;
- primary healthy, only a secondary down ⇒ **200**, with the failed drivers named
 in a new `degraded` block: `{ status, state, degraded: { drivers, primaryDatasource } }`.

The failed driver is never hidden — the rejected alternative of filtering it out
of the response stays rejected. `degraded` appears **only** on that branch; an
all-healthy 200 body is byte-identical to before.

**The criterion is a readable fact, not a heuristic.** `ObjectQL.resolvePrimaryDatasource()`
(new, exported with `PrimaryDatasourceVerdict` / `PrimaryDatasourceUnresolvedReason`)
answers *where this deployment's platform system objects actually live*, resolved
through the same five-step routing order every query uses — never registration
order, never the driver flagged default at registration. The ADR-0057 §3.6 system
ledgers (`audit` / `telemetry` / `event`) are excluded because they are
deliberately routed off the primary. Disagreement, silence, or a name with no
driver behind it return an **unresolved verdict**, never a guess.

**Fail toward draining.** Every way of not knowing — an engine that predates the
probe, a probe that throws, a malformed verdict, a genuinely split deployment —
falls back to the old whole-node 503. Staying in rotation requires a *positive*
reading; the absence of a negative one is not permission. Pinned in both
directions, including an inversion ablation.

**framework#3756 is not overturned.** Its quantified reason — "a replica that
would fail 100% of its requests" — still holds in the single-datasource
deployment it was measured on, where the primary *is* the only source; that
deployment's answer is unchanged down to the response body. What #3756's
reasoning never covered is the multi-datasource shape, and that is the only
branch this carves out.

**Grade: `minor` for both, proposed rather than assumed** — this changes when a
published operational probe drains a node, so an operator whose alerting keys on
`/ready` returning 503 for *any* driver failure will see 200 + `degraded`
instead, and `degraded` is a new response field. Nothing is removed or renamed,
no declared contract key is added (the ruling explicitly declines that: 「⛔ 本裁不
加契约键」), single-datasource behaviour is bit-identical, and no migration is
required — so `major` overstates it and `patch` understates a deliberate change
to an availability control surface.

⚠️ Out of scope, tracked separately as **#13578**: `DELETE` of a datasource still
does not evict the stuck driver from the in-memory engine registry, so the
datasource keeps appearing in `/ready`'s report until the process restarts. That
half is a defect under either answer to this card and is queued independently.

24 changes: 12 additions & 12 deletions
content/docs/permissions/system-context.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -109,18 +109,18 @@ that silently does not happen.

| # | Behaviour when `isSystem` | Package | What you get / what you lose | Anchor |
|:--|:---|:---|:---|:---|
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10616` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10778` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9509` |
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10712` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10874` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9605` |
| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1576` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9546`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5638` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3573`, `:3583`, `:3610` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9642`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5639` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3574`, `:3584`, `:3611` |
| 25 | Engine-owned / append-only write guard bypassed | plugin-security | Get: generic writes to `managedBy` engine-owned objects | `system-write-guard.ts:96`, `:120` |
| 26 | Identity write guard bypassed (ADR-0092) | plugin-auth | Get: direct writes to identity tables through the generic data path | `identity-write-guard.ts:98` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6336` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11364` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11293` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6337` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11460` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11389` |

### 3. Sharing (`plugin-sharing`)

Expand DownExpand Up@@ -179,8 +179,8 @@ a reader tracing where elevation travels needs them.

| # | Site | Package | What it does |
|:--|:---|:---|:---|
| 62 | `objectql/src/engine.ts:3405` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13705` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 62 | `objectql/src/engine.ts:3406` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13801` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 64 | `plugin-reports/src/report-service.ts:556` | plugin-reports | Threads the flag into the engine call that runs a report |
| 65 | `body-runner.ts:279` | runtime | Rebuilds an `ExecutionContext` from a hook session, carrying the flag across |

Expand All@@ -195,7 +195,7 @@ assuming `isSystem` covers it is a documented source of bugs.
|:---|:---|:---|
| "It suppresses triggers / record-change automation" | **No.** Only `skipTriggers` does. A bare `{ isSystem: true }` on a seed write re-fired automation on freshly seeded rows and wedged first boot | `metadata-protocol/src/seed-loader.ts:1909` (rationale at `:1819`–`1821`, #3760), `flow.zod.ts:685` |
| "It skips the state machine" | **No.** That is `skipStateMachine`, carried by seed replay and by `treatAsHistorical` imports | `objectql/src/engine.ts` FSM gate; see [State Machine](/docs/protocol/objectql/state-machine) |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9492`–`9509` |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9588`–`9605` |
| "It preserves a supplied `updated_at` / `updated_by`" | **No.** That is `preserveAudit`, a separate opt-in — and an UPDATE-path exemption only | `field.zod.ts:1451` (#3493 / #6640) |
| "It stamps `created_by`" | **No.** Audit stamping reads `userId` from the context. A user-less system write stamps nothing — that is today's behaviour, not an error | `runtime-identity.ts:280`–`281` |
| "It bypasses every guard" | **No.** The last-admin guard applies to **every** context, `isSystem` included — the deprovision path that actually locks an org out is the system one | `last-admin-guard.ts:286` |
Expand Down

48 changes: 48 additions & 0 deletions
packages/objectql/src/driver-connect-errors.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,6 +19,54 @@ export interface DriverHealth {
 skipped?: boolean;
}

/**
 * Why {@link PrimaryDatasourceVerdict} could not name a primary datasource.
 *
 * Every member means the SAME thing to a readiness probe — *we cannot tell* —
 * and the ruling on #13408 fixes what a probe must do with that answer: fall
 * back to the old whole-node 503 (fail toward draining), never to "don't
 * drain". They are distinguished only so the reason can be logged and reported;
 * ⛔ a caller must not branch on them to keep a replica in rotation.
 */
export type PrimaryDatasourceUnresolvedReason =
 /** No platform system object is registered here — nothing to read the fact off. */
 | 'no-system-objects-registered'
 /** The platform's system objects are split across more than one datasource. */
 | 'system-objects-split'
 /** A registered system object routes nowhere: no binding and no default driver. */
 | 'system-object-unbound'
 /** The name resolved, but no driver is registered under it, so nothing probes it. */
 | 'no-driver-registered';

/**
 * WHICH datasource carries this deployment's platform system objects — the
 * machine-readable "primary/default datasource" fact the #13408 ruling requires
 * (2026-08-31, 第 6 场总监席决裁批 #12), stated as a verdict rather than a
 * `string | undefined` so "we could not tell" can never be mistaken for a name.
 *
 * ⛔ The ruling forbids deriving this from a heuristic such as "the first
 * datasource registered". The one implementation is
 * `ObjectQL.resolvePrimaryDatasource()`.
 */
export type PrimaryDatasourceVerdict =
 | {
 resolved: true;
 /** The datasource name, which is also a registered driver's name. */
 datasource: string;
 /**
 * How many registered platform system objects agreed on it. Never 0 — a
 * verdict read off nothing is `no-system-objects-registered`, not a name,
 * so a caller cannot act on a vacuous agreement.
 */
 witnesses: number;
 }
 | {
 resolved: false;
 reason: PrimaryDatasourceUnresolvedReason;
 /** The distinct datasource names seen, when more than one disagreed. */
 candidates?: readonly string[];
 };

/** `error.message` when it is an Error, its string form otherwise. */
function failureMessage(error: unknown): string {
 if (error instanceof Error) return error.message || error.name;
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained by os-steve · Pull Request #13763 · objectstack-ai/objectstack · GitHub
Skip to content
fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained#13763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

os-steve
merged 7 commits into

from

Aug 31, 2026
 Merged


fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained
#13763
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


7 commits


Select commit
Hold shift + click to select a range
85c37f6

wip(runtime): partition /ready driver health into primary vs secondar…


os-steve Aug 31, 2026
e8bb34a

test(objectql): pin system-object-unbound to the reachable zero-drive…


os-steve Aug 31, 2026
ae226f6

chore: changeset for #13408


os-steve Aug 31, 2026
cbbc8d0

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
6a07781

docs(permissions): re-anchor the system-context census after the engi…


os-steve Aug 31, 2026
0e2fdb8

Merge remote-tracking branch 'origin/main' into claude/issue-13408-re…


claude Aug 31, 2026
2aa73e8

fix(test): type the driver double and narrow the dispatch result so t…


os-steve Aug 31, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


65 changes: 65 additions & 0 deletions
.changeset/ready-primary-datasource-drain.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
---
"@objectstack/objectql": minor
"@objectstack/runtime": minor
---

fix(runtime,objectql): `/api/v1/ready` drains only on the PRIMARY datasource's failure; a secondary/tenant datasource is reported, not drained (#13408)

On a multi-datasource deployment, one datasource whose driver could not start
pinned `/api/v1/ready` to 503 on **every replica** — so a readiness-checked load
balancer drained every upstream and took the whole deployment offline, while
Postgres and the app itself were healthy (`/api/v1/health` 200, direct reads
working). One tenant's misconfiguration = total outage. Observed on a live
3-replica EE deployment and recovered only by restarting every process.

Ruled 2026-08-31 (第 6 场总监席决裁批 #12, maintainer verbatim 「同意」), Option B:

> `/api/v1/ready` 只在**主/默认数据源**不健康时摘流量;次要/租户数据源的故障照常
> **上报**(`/ready` 响应 body、日志、告警)但不 drain 节点。

**What changed.** When a driver reports itself unhealthy, `/ready` now asks
which datasource is the deployment's primary one before choosing a status:

- primary unhealthy, or the criterion unresolvable ⇒ **503**, unchanged envelope;
- primary healthy, only a secondary down ⇒ **200**, with the failed drivers named
 in a new `degraded` block: `{ status, state, degraded: { drivers, primaryDatasource } }`.

The failed driver is never hidden — the rejected alternative of filtering it out
of the response stays rejected. `degraded` appears **only** on that branch; an
all-healthy 200 body is byte-identical to before.

**The criterion is a readable fact, not a heuristic.** `ObjectQL.resolvePrimaryDatasource()`
(new, exported with `PrimaryDatasourceVerdict` / `PrimaryDatasourceUnresolvedReason`)
answers *where this deployment's platform system objects actually live*, resolved
through the same five-step routing order every query uses — never registration
order, never the driver flagged default at registration. The ADR-0057 §3.6 system
ledgers (`audit` / `telemetry` / `event`) are excluded because they are
deliberately routed off the primary. Disagreement, silence, or a name with no
driver behind it return an **unresolved verdict**, never a guess.

**Fail toward draining.** Every way of not knowing — an engine that predates the
probe, a probe that throws, a malformed verdict, a genuinely split deployment —
falls back to the old whole-node 503. Staying in rotation requires a *positive*
reading; the absence of a negative one is not permission. Pinned in both
directions, including an inversion ablation.

**framework#3756 is not overturned.** Its quantified reason — "a replica that
would fail 100% of its requests" — still holds in the single-datasource
deployment it was measured on, where the primary *is* the only source; that
deployment's answer is unchanged down to the response body. What #3756's
reasoning never covered is the multi-datasource shape, and that is the only
branch this carves out.

**Grade: `minor` for both, proposed rather than assumed** — this changes when a
published operational probe drains a node, so an operator whose alerting keys on
`/ready` returning 503 for *any* driver failure will see 200 + `degraded`
instead, and `degraded` is a new response field. Nothing is removed or renamed,
no declared contract key is added (the ruling explicitly declines that: 「⛔ 本裁不
加契约键」), single-datasource behaviour is bit-identical, and no migration is
required — so `major` overstates it and `patch` understates a deliberate change
to an availability control surface.

⚠️ Out of scope, tracked separately as **#13578**: `DELETE` of a datasource still
does not evict the stuck driver from the in-memory engine registry, so the
datasource keeps appearing in `/ready`'s report until the process restarts. That
half is a defect under either answer to this card and is queued independently.

24 changes: 12 additions & 12 deletions
content/docs/permissions/system-context.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -109,18 +109,18 @@ that silently does not happen.

| # | Behaviour when `isSystem` | Package | What you get / what you lose | Anchor |
|:--|:---|:---|:---|:---|
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10616` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10778` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9509` |
| 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10712` |
| 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:10874` |
| 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9605` |
| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1576` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9546`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5638` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3573`, `:3583`, `:3610` |
| 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9642`, `readonly-strict-errors.ts:66` |
| 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5639` |
| 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3574`, `:3584`, `:3611` |
| 25 | Engine-owned / append-only write guard bypassed | plugin-security | Get: generic writes to `managedBy` engine-owned objects | `system-write-guard.ts:96`, `:120` |
| 26 | Identity write guard bypassed (ADR-0092) | plugin-auth | Get: direct writes to identity tables through the generic data path | `identity-write-guard.ts:98` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6336` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11364` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11293` |
| 27 | Search-companion column **kept** in a read's rows when it was explicitly requested | objectql | Get: the internal companion column is readable. Lose: nothing for app code — this is the engine reading its own index | `objectql/src/engine.ts:6337` |
| 28 | Dependent-count disclosure on a blocked delete | objectql | Get: the count of blocking children. Nothing was elevated past the caller, so nothing is withheld | `objectql/src/engine.ts:11460` |
| 29 | Reference-cleanup log attributes the write to `'system'` | objectql | Get: an honest actor label instead of `anonymous` when the context carries neither `userId` nor `actor` | `objectql/src/engine.ts:11389` |

### 3. Sharing (`plugin-sharing`)

Expand DownExpand Up@@ -179,8 +179,8 @@ a reader tracing where elevation travels needs them.

| # | Site | Package | What it does |
|:--|:---|:---|:---|
| 62 | `objectql/src/engine.ts:3405` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13705` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 62 | `objectql/src/engine.ts:3406` | objectql | Propagates `isSystem` into the hook session so hooks can tell engine self-writes from user writes |
| 63 | `objectql/src/engine.ts:13801` | objectql | `ScopedContext.isSystem` getter — re-exposes the underlying execution context's flag |
| 64 | `plugin-reports/src/report-service.ts:556` | plugin-reports | Threads the flag into the engine call that runs a report |
| 65 | `body-runner.ts:279` | runtime | Rebuilds an `ExecutionContext` from a hook session, carrying the flag across |

Expand All@@ -195,7 +195,7 @@ assuming `isSystem` covers it is a documented source of bugs.
|:---|:---|:---|
| "It suppresses triggers / record-change automation" | **No.** Only `skipTriggers` does. A bare `{ isSystem: true }` on a seed write re-fired automation on freshly seeded rows and wedged first boot | `metadata-protocol/src/seed-loader.ts:1909` (rationale at `:1819`–`1821`, #3760), `flow.zod.ts:685` |
| "It skips the state machine" | **No.** That is `skipStateMachine`, carried by seed replay and by `treatAsHistorical` imports | `objectql/src/engine.ts` FSM gate; see [State Machine](/docs/protocol/objectql/state-machine) |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9492`–`9509` |
| "It skips validation rules" | **No.** Field shape, `format`, `script` and the rest still run. The `readonly` strip runs *before* validation precisely so a discarded value is not judged | `objectql/src/engine.ts:9588`–`9605` |
| "It preserves a supplied `updated_at` / `updated_by`" | **No.** That is `preserveAudit`, a separate opt-in — and an UPDATE-path exemption only | `field.zod.ts:1451` (#3493 / #6640) |
| "It stamps `created_by`" | **No.** Audit stamping reads `userId` from the context. A user-less system write stamps nothing — that is today's behaviour, not an error | `runtime-identity.ts:280`–`281` |
| "It bypasses every guard" | **No.** The last-admin guard applies to **every** context, `isSystem` included — the deprovision path that actually locks an org out is the system one | `last-admin-guard.ts:286` |
Expand Down

48 changes: 48 additions & 0 deletions
packages/objectql/src/driver-connect-errors.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,6 +19,54 @@ export interface DriverHealth {
 skipped?: boolean;
}

/**
 * Why {@link PrimaryDatasourceVerdict} could not name a primary datasource.
 *
 * Every member means the SAME thing to a readiness probe — *we cannot tell* —
 * and the ruling on #13408 fixes what a probe must do with that answer: fall
 * back to the old whole-node 503 (fail toward draining), never to "don't
 * drain". They are distinguished only so the reason can be logged and reported;
 * ⛔ a caller must not branch on them to keep a replica in rotation.
 */
export type PrimaryDatasourceUnresolvedReason =
 /** No platform system object is registered here — nothing to read the fact off. */
 | 'no-system-objects-registered'
 /** The platform's system objects are split across more than one datasource. */
 | 'system-objects-split'
 /** A registered system object routes nowhere: no binding and no default driver. */
 | 'system-object-unbound'
 /** The name resolved, but no driver is registered under it, so nothing probes it. */
 | 'no-driver-registered';

/**
 * WHICH datasource carries this deployment's platform system objects — the
 * machine-readable "primary/default datasource" fact the #13408 ruling requires
 * (2026-08-31, 第 6 场总监席决裁批 #12), stated as a verdict rather than a
 * `string | undefined` so "we could not tell" can never be mistaken for a name.
 *
 * ⛔ The ruling forbids deriving this from a heuristic such as "the first
 * datasource registered". The one implementation is
 * `ObjectQL.resolvePrimaryDatasource()`.
 */
export type PrimaryDatasourceVerdict =
 | {
 resolved: true;
 /** The datasource name, which is also a registered driver's name. */
 datasource: string;
 /**
 * How many registered platform system objects agreed on it. Never 0 — a
 * verdict read off nothing is `no-system-objects-registered`, not a name,
 * so a caller cannot act on a vacuous agreement.
 */
 witnesses: number;
 }
 | {
 resolved: false;
 reason: PrimaryDatasourceUnresolvedReason;
 /** The distinct datasource names seen, when more than one disagreed. */
 candidates?: readonly string[];
 };

/** `error.message` when it is an Error, its string form otherwise. */
function failureMessage(error: unknown): string {
 if (error instanceof Error) return error.message || error.name;
Expand Down
Loading
Loading