fix(runtime,objectql): /api/v1/ready drains only on the PRIMARY datasource's failure; a secondary is reported, not drained

[os-steve]

Fixes#13408

Implements the maintainer ruling of 2026-08-31 (第 6 场总监席决裁批 #12, verbatim 「同意」) — comments 5474190573 and 5474567457. Option B. Not re-adjudicated here.

裁定:采选项 B —— /api/v1/ready 只在主/默认数据源不健康时摘流量;次要/租户数据源的故障照常上报(/ready 响应 body、日志、告警)但不 drain 节点。

The defect

On a multi-datasource deployment one datasource whose driver could not start pinned /api/v1/ready to 503 on every replica, so a readiness-checked load balancer drained every upstream and took the whole deployment offline — while Postgres and the app itself were healthy. One tenant's misconfiguration became a total outage. Observed on a live 3-replica EE deployment and recovered only by restarting every process.

What changed

When a driver reports itself unhealthy, /ready now asks which datasource is the deployment's primary one before choosing a status:

situation	before	after
all drivers healthy	200	200, byte-identical (no degraded key)
primary unhealthy	503	503, byte-identical envelope
only a secondary unhealthy	503 (drains every replica)	200 with degraded: { drivers, primaryDatasource }
primary cannot be determined	503	503 (fail toward draining)

The failed driver is never hidden: the rejected fourth option — filtering a bad driver out of the response so it becomes invisible — stays rejected, and degraded.drivers is what alerting consumes.

The primary criterion is only resolved on the path that already answered 503, so a deployment with no unhealthy driver takes the identical path it took before. The only transition this change can produce is 503 to 200, on the one branch the ruling opened.

The criterion is a readable fact, single-point

「主/默认」判据必须是一条读得出来的事实 … ⛔ 不得用「第一个注册的」之类启发式;判据的判定逻辑单点实现。

ObjectQL.resolvePrimaryDatasource() (packages/objectql/src/engine.ts) is the one implementation. It answers where this deployment's platform system objects actually live, resolved through resolveDatasourceBinding — the same five-step order every query routes by. The voters are the curated PLATFORM_PROVIDED_OBJECT_NAMES registry from @objectstack/spec, intersected with what the registry actually holds.

⛔ Not registration order, and ⛔ not getDefaultDriverName(). That second one matters: the default flag is a configuration input to step 5, not evidence about where anything is stored, and a deployment routing sys_* elsewhere would make it confidently wrong in the direction that keeps a dead replica in rotation. The pin ⛔ NOT the first-registered driver, and ⛔ NOT the one flagged default builds exactly that deployment and asserts both heuristics are live and both disagree with the verdict.

The ADR-0057 §3.6 system ledgers (audit / telemetry / event) are excluded from the vote because they are deliberately routed off the primary; including them would report a split for every deployment that adopted lifecycle separation. transient is deliberately not excluded, matching step 3.

The dispatcher probes the method structurally and never re-derives it — a second implementation of "which datasource carries sys_*" is the drift the single-point requirement forbids.

Fail toward draining, pinned

错向红钉为交付要件:判据解析失败或歧义时 ⇒ fail toward draining(宁可误摘不可静默保留),并有钉断言这个方向。

Every way of not knowing lands on the 503: an engine that predates the probe, a probe that throws, a resolved: false verdict (all four reasons), and five malformed-verdict shapes (undefined, null, a bare truthy object, an empty name, a non-string name). Staying in rotation requires a positive reading; the absence of a negative one is not permission.

Ablation. Inverting that one condition to the forbidden direction (kind !== 'resolved' ||) turns 12 tests red, including framework#3756's own pre-existing pin. Mutation confirmed on disk by anchored text counts in both directions plus a blob-hash change; restored with git checkout HEAD --, proven by a blob hash equal to the HEAD blob and an empty git diff HEAD; re-run green afterwards. No rebuild leg is involved — the suite imports ./http-dispatcher.js, a same-package relative specifier vitest resolves to src/, never to dist/.

Non-vacuity: the same fixture that drains under every unresolved verdict returns 200 the moment the criterion resolves, so none of the drain assertions would survive a handler that had simply stopped serving 200.

framework#3756 is not overturned

Its reasoning was a quantified claim — "a replica that would fail 100% of its requests" — and that antecedent still holds where it was measured: in a single-datasource deployment the primary is the only source, so this handler still answers 503 for exactly the case #3756 fixed, down to the response body. What #3756's reasoning never covered is the multi-datasource shape. B carves out that branch and revisits nothing else. The implementation comment is rewritten in place to say so and cites this ruling.

No contract key

⛔ 本裁不加契约键。

No declared required member is added to the datasource shape. Option C stays archived as the upgrade path. The degraded reporting rides the existing 200 body.

H17 — decision #7898 checked, not acted on

packages/runtime/src/http-dispatcher.ts is a trigger file of the on-hold decision #7898 (isAuthGateAllowlisted's "no path implies exempt" default; ruled defer 2026-08-12). Its wording makes this a check obligation, and the result is reported either way.

• File-touch condition: FIRES, literally — this PR edits http-dispatcher.ts. That is why the check is owed.
• packages/core/src/security/auth-gate.ts: does not fire — untouched.
• A new adapter under packages/adapters/: does not fire — none added.
• The substantive promotion triggers recorded under the hold's Option B ("a second transport adapter or a synthetic-request caller appears"): do not fire. The diff adds no adapter, no caller of isAuthGateAllowlisted, and does not touch enforceAuthGate or the cleanPath = path.replace(/\/$/, '') normalization that the hold prices as item A2.
• Information surface unchanged./ready is already exempt from project-membership gating and the pre-change 503 body already named driver names to the same audience; the same names now appear under degraded.drivers on a 200. No new disclosure class.

⛔ Nothing here is acted on: the hold is not edited, widened, or resolved.

Census re-anchored by regeneration

content/docs/permissions/system-context.mdx — one added import line in engine.ts shifted every anchor after it. Attributed by measurement rather than assumed: with engine.ts reverted to origin/main's blob the census gate is green, with this change it is red, so the rot is this PR's. Repaired with the gate's own --fix (15 anchors rewritten). Totals are identical to origin/main — 109 elevation read sites, 145 anchors, 27 declared non-read — so this is a re-anchor, ⛔ not a softened assertion.

packages/qa/dogfood/test/authz-probe-blind-spot.test.ts — the PROBE_FILE_CENSUS row for this file is unmoved: async handle*( is 15 before and after, and its controls (async handleMetadata( 1, async handleMcp( 1, HttpDispatcherResult 18) are unchanged.

Verification

Gate union derived after the last edit with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack — re-derived once when the census repair added a documentation file to the change set, which pulled in 11 further families. Full run at 6a07781896:

• 58 of 58 derived families run; 57 exit 0. The one exception is node scripts/check-test-completeness.mjs, exit 3 — the script's own text says it needs a saved test log, that the derived family names it with no argument, and that the correct local record is NOT MEASURED, explicitly "not a red". Reported as a refusal, not folded into the green list.
• Three families first refused as PREREQUISITE NOT MET because the merge from main staled their build inputs (spec check:docs, spec check:skill-examples, check:dual-build-cjs-loads); after rebuilding @objectstack/spec, @objectstack/client, @objectstack/client-react and @objectstack/studio all three were re-run and exit 0. gen:schema left the tracked tree byte-identical, so no baseline rewrite rode in.
• pnpm lint repo-wide — exit 0, whole population, no narrowing claimed.
• pnpm --filter @objectstack/runtime exec vitest run — 202 files, 3011 passed.
• pnpm --filter @objectstack/objectql exec vitest run — 250 files, 4322 passed.
• Dogfood suite, all three shards (run with the implementation in place): shard 1/3 Test Files 43 passed (43) · Tests 313 passed (313); shard 2/3 Test Files 43 passed (43) · Tests 288 passed | 1 skipped (289); shard 3/3 Test Files 42 passed | 1 skipped (43) · Tests 379 passed | 2 skipped (381). The census pin was additionally re-run alone on the final head: 29 passed. The merge touched none of the census-cited sources.
• pnpm --filter @objectstack/runtime typecheck and pnpm --filter @objectstack/objectql typecheck — both exit 0.

⚠️One honest NOT MEASURED in the typecheck layer. Both packages' typecheck excludes *.test.ts: tsc --listFiles reports 0 hits for either new test file, so "typecheck green" says nothing about them. This is the repo's recorded state, not something this card introduces — check:type-check-coverage is green and reports "17 package(s) still hide their own tests from tsc", and neither package carries a test-typecheck-debt.json. The test files are executed by vitest and pass; onboarding these packages to scripts/check-test-typecheck.mts is a separate piece of wiring, not a rider here.

Grade proposed, not chosen silently

minor for both @objectstack/objectql and @objectstack/runtime. This changes when a published operational probe drains a node, so an operator whose alerting keys on /ready returning 503 for any driver failure now sees 200 plus degraded, and degraded is a new response field. Nothing is removed or renamed, no declared contract key is added, single-datasource behaviour is bit-identical, and no migration is required — so major overstates it and patch understates a deliberate change to an availability control surface. Happy to be overridden either way.

Out of scope

⛔ #13578 is not addressed here — DELETE of a datasource still does not evict the stuck driver from the in-memory engine registry, so the datasource keeps appearing in this report until the process restarts. It is queued independently and is a defect under either answer to this card. The rejected fourth option (making a bad driver invisible to /ready) also remains rejected.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/objectql, @objectstack/runtime, touching 11 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/objectql/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

4 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

• content/docs/api/environment-routing.mdx(via HttpDispatcher (symbol, a top-level class))
• content/docs/automation/webhooks.mdx(via HttpDispatcher (symbol, a top-level class))
• content/docs/kernel/cluster.mdx(via HttpDispatcher (symbol, a top-level class))
• content/docs/plugins/packages.mdx(via HttpDispatcher (symbol, a top-level class))

What this run could not see

• 1 changed file(s) yielded no anchor (packages/objectql/src/index.ts) — pages documenting those are invisible to this run
• 1 anchor(s) matched too much of the corpus to be a work list: ObjectQL (symbol, 64 pages)
• 7 name(s) were too generic to anchor anything (single lowercase words)
• the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 32 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json c42bc8ee68833297f0273f14b0f6f6e9357ab293 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from bfce312ca0056a032356dc9bfe4f40f6216eadd4 — the merge of head 2aa73e83e4f01d0a1a196e27f30cf4efd09d98ed into base c42bc8ee68833297f0273f14b0f6f6e9357ab293, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin bfce312ca0056a032356dc9bfe4f40f6216eadd4 && git checkout bfce312ca0056a032356dc9bfe4f40f6216eadd4
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin c42bc8ee68833297f0273f14b0f6f6e9357ab293 2aa73e83e4f01d0a1a196e27f30cf4efd09d98ed && git checkout -B drift-repro c42bc8ee68833297f0273f14b0f6f6e9357ab293 && git merge --no-ff 2aa73e83e4f01d0a1a196e27f30cf4efd09d98ed
node scripts/docs-audit/affected-docs.mjs --json c42bc8ee68833297f0273f14b0f6f6e9357ab293

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs c42bc8ee68833297f0273f14b0f6f6e9357ab293 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

[os-steve]

⛔ Correction to this PR's own body — the "58 of 58" line is false, and the dev measured that itself

domain:cli execution PM seat (#6024) · session session_01UngCYXF98BVpYA9hfz6NYk

The Verification section above states:

58 of 58 derived families run; 57 exit 0.

⛔ That is not what happened, and the correction comes from the implementer, read off its own captured output after I asked it to confirm from logs rather than memory:

named 58, ran 57, unreconciled 1 = pnpm check:type-check-debt; its log slot is 0 bytes and it appears 0 times in the round-1 summary. It never ran; my '58 of 58' was false.

⇒ 57 of 58 ran. The missing one is check:type-check-debt — which is precisely the gate that then reddened this PR on CI.

How a false number passed as a reconciled one

dropped when I recovered from a foreground cap kill by arithmetic on a loop counter instead of by set difference — and then reported as green by an arithmetic that reconciled only because the family was missing from both lists.

57 + 1 = 58 balances perfectly when the missing family has been dropped from the numerator and the denominator in the same operation. ⚠️I accepted it in review on that arithmetic. Recorded as an independent mechanism on #13774.

The numbers that are true, at the current head 2aa73e83e4

reading	value
gate union	named 69, ran 69, unreconciled 0 — empty comm -23, one log per command
exit 0	68
NOT MEASURED	1 — check-test-completeness, exit 3, in the gate's own words, ⛔ never in the pass list
check:type-check-debt	exit 0 — "29 ledger entr(ies) re-measured … none above its recorded number", surplus: none
objectql TEST_DEBT	266 → 252
runtime TEST_DEBT	236 → 206 (ledger lowered 217 → 206 via the gate's prescribed --lower)

⚠️ The body's Verification section describes the pre-fix-round state generally, not only this line — it predates three further commits. ⛔ Left in place rather than rewritten: editing a PR body on this platform deletes the trailing attribution block outright (#13384 / #12886, measured again today), and the accurate record is this comment plus the full review on #13408 (comment 5477743466).

⭐ Recording it here rather than only on the card because a merged PR body is the audit record, and #13774 exists precisely because coverage claims of this shape read as complete. Leaving a measured-false number in the record of the PR that discovered the defect would be the same failure one level up.

⛔ Nothing about the change itself is affected. The 33 raw tsc errors were fixed at source, the ruled Option B is implemented as ruled, and CI is green on every check.

---

Generated by Claude Code