feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas - #14183

Merged
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout
Sep 1, 2026
Merged

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas#14183
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout

Conversation

@claude

@claudeclaudeBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes#13331

What this ships

A runtime-authored metadata mutation now reaches every replica's ObjectQL registry, not just the writer's. Measured defect on the shipped 3-replica EE compose (ADR-0018): PUT /api/v1/meta/object/... persisted to the shared sys_metadata (meta reads 200 fleet-wide) while only the writing replica registered the object — /api/v1/data/... answered a hard 404 OBJECT_NOT_FOUND on the other two, indefinitely; 200 concurrent creates through the LB gave 67x201 / 133x404 with a boot-loaded control object at 0 errors.

Mechanism (maintainer-ruled 2026-09-01, director batch A, Option A — verbatim adoption of the escalation's recommendation):

  • Publisher at the producer choke point. The protocol's post-persistence funnel (emitMetadataMutation, the seam onMetadataMutation subscribes — reached by saveMetaItem, runPublishSideEffects, deleteMetaItem) now also publishes the mutation's ADDRESS on a new cluster channel metadata.mutated (METADATA_MUTATION_CLUSTER_CHANNEL, payload ClusterMetadataMutationPayload, both exported from @objectstack/metadata-protocol). Drafts are never published — they enter no registry anywhere.
  • Peers converge from their own DB read. On receipt a replica re-reads the row from its OWN sys_metadata: active row present ⇒ re-run applyRegistryWriteThrough (package binding derived the way the recovery callers derive it); no active row ⇒ run the delete heal walk (restoreArtifactRegistryView). The payload is a signal, never trusted content; duplicates and out-of-order delivery converge to the row's current state by construction. After convergence the event replays into the replica's LOCAL onMetadataMutation listeners (never re-published), so authored hook/action re-binds re-sync on peers.
  • Attach seam + bridge lane.ObjectStackProtocolImplementation.attachMetadataMutationPubSub(pubsub, nodeId) — idempotent on the pair, loopback-suppressed via originNode, mirroring MetadataManager.attachClusterPubSub and the engine's attachAuthzInvalidationPubSub. MetadataClusterBridgePlugin late-binds it at kernel:ready as a second, independent lane: the boot shape that lacks a manager-backed metadata service (TS-config host-config — exactly the shipped EE shape) is the one that needs this lane most. The new lane skips the in-process memory driver from birth (the guard the authz sibling carries).
  • Option B (consumer-side self-heal at assertObjectRegistered) is not built — presented only as a possible stopgap; the maintainer did not order one.

The two fences, both held

Evidence

  • Two-arm, directions declared in the test-file header before running (protocol.cluster-mutation-fanout.test.ts, two replicas over one shared store): Arm B (attached) — the peer converges from its own read; Arm A (control, no bridge) — the identical write leaves the peer empty, so Arm B is the bridge's doing, not a harness artifact. Also pinned: address-only payload (key set equality), draft silence, loopback suppression, duplicate-delivery convergence, re-attach idempotency, detach, delete fan-out, draft-discard keeping the peer's active registration, and listener replay ordered after registry convergence.
  • Committed-tree ablation at 44a2e59 (script with trap restore, absolute paths): the single publisher call deleted — mutation proved on disk (anchor count 0, marker count 1) — turned exactly the 7 declared bus-crossing cases red with 15 staying green (controls held); restore proven by HEAD-blob equality (d7ba0225...) plus empty git diff HEAD plus marker count 0; restored run 22/22 green. Resolution note: these tests import ./protocol.js relative from src — the subject never resolves through dist, so no rebuild leg exists to skip; the on-disk grep is the falsifiable observation for both legs.
  • Suites at final head e460193: metadata-protocol 152 files passed / 2 skipped (2094 tests passed / 10 skipped), service-cluster 6 files / 79 tests passed, tsc --noEmit clean with both edited test files confirmed inside the program via --listFiles (2 hits).
  • Gate union at e460193 (node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands; provenance line names this repo at this commit and the --repo assertion holds): 44 derived; 41 exit 0; 3 NOT MEASURED by their own printed verdicts (check-test-completeness exit 3 — grades a CI turbo log; check:dual-build-cjs-loads exit 3 — needs the full workspace build; check:type-check-debt exit 3 — full re-measure). Two first-pass reds were fixed and re-run to their own OK lines: check:engine-double-contract ("OK — 741 pinned, 134 in the DEBT ledger, 3 exempt"; ledger learned the fanout file's pinned doubles via the gate's own --write) and check:objectql-double-limit ("conformance holds: 299 doubles graded"; the new find double now holds the caller's bound by presence).
  • The write-through caller pin re-opened deliberately: applyRegistryWriteThrough grew its FIFTH caller (the peer applier). The trace, the count pin, and a new route-5 spelling case (a plural delivered over the wire registers under the singular; fold via canonicalMetaType, the complete map) are updated in protocol.object-registry-write-through-spelling.test.ts.

Notes for contract review

CI rework — census re-anchor (cdbe5f0)

Lint & Repo Gates red at e460193: check-system-context-census — pure line rot from this PR's own insertion (the publisher block sits above stripReadonlyForInsert, moving its context.isSystem read from protocol.ts:1737 to :1741 while content/docs/permissions/system-context.mdx row 21 still anchored 1737). Repaired with the gate's own --fix and VERIFIED as a SHIFT, not a population change: the rewrite touches exactly one line — the anchor's line number — and zero prose, and the census population is unchanged (109 sites / 145 anchors before and after). The docs file joining this diff is that re-anchor, not scope creep. Reproduced red locally first, then the bare gate re-run green: "OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read."

Docs-drift assessment (stated, not rewritten, per dispatch): content/docs/kernel/cluster.mdx §6.2 is now INCOMPLETE — it says cross-node metadata invalidation "already works" through the manager's metadata.changed lane alone, which is exactly the account #13331 falsified for the runtime-authoring path (the protocol never reaches the manager, and host-config boots carry no manager at all). The page wants a paragraph on the metadata.mutated protocol lane and the bridge's second attach; its own closing principle there ("the peer re-reads the shared store, which is the source of truth") is the rule the new receipt implements, and its "Target spec (planned)" block already marks MetadataChangedEventPayloadSchema as not wired — corroborating #14180. Same incompleteness class applies to the §5/§7 bridge mentions and content/docs/concepts/metadata-lifecycle.mdx.

Generated by Claude Code


Generated by Claude Code


Generated by Claude Code

…ns out to peer replicas (#13331)
Publisher at the protocol's post-persistence choke point publishes the
mutation's address on the new metadata.mutated cluster channel; peers
converge their ObjectQL registry from their OWN sys_metadata read (write-
through when an active row exists, the delete heal walk when none does),
then replay the event into local onMetadataMutation listeners. The bridge
plugin late-binds attachMetadataMutationPubSub as a second independent
lane at kernel:ready, guarded off the in-process memory driver.
Ruled 2026-09-01 (director batch A, Option A): producer-side fan-out
mirroring the shipped AuthzClusterBridgePlugin shape; the payload is a
signal, never trusted content. Option B (consumer-side self-heal) is not
built.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
…ouble; ledger the new pinned engine doubles
check:objectql-double-limit named the new find double limit-blind — the
bound now applies after the filter, by presence. The engine-double ledger
learned the fanout file's pinned delete/findOne/update doubles via the
gate's own --write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@github-actions

github-actionsBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/metadata-protocol, @objectstack/service-cluster, touching 19 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

5 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/wire-format.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/concepts/metadata-lifecycle.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), ObjectStackProtocolImplementation (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/deployment/validating-metadata.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/kernel/cluster.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/kernel/services-checklist.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class))

3 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v15.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/releases/v16.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts) — pages documenting those are invisible to this run
  • 1 anchor(s) matched too much of the corpus to be a work list: /api/v1/data (route, 35 pages)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 11 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fcpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 1304ec033ce539d92069afd7ac66110cc1951b11 — the merge of head 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec into base e62129153f2aa3d7666eab9c1af2bdb18e7333fc, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 1304ec033ce539d92069afd7ac66110cc1951b11 && git checkout 1304ec033ce539d92069afd7ac66110cc1951b11
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin e62129153f2aa3d7666eab9c1af2bdb18e7333fc 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec && git checkout -B drift-repro e62129153f2aa3d7666eab9c1af2bdb18e7333fc && git merge --no-ff 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec
node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fc

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs e62129153f2aa3d7666eab9c1af2bdb18e7333fc → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 1, 2026
…he fan-out insertion
check-system-context-census red on CI: the metadata.mutated publisher
block inserted above stripReadonlyForInsert moved the context.isSystem
read 1737 -> 1741, and the census page still anchored the old line. The
gate's own --fix performed the shift (one line-number rewrite, zero
prose); bare re-run green: 109 sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
The auto-merge of origin/main kept this branch's pre-#14171 anchor rows
beside the row-21 edit, flunking the census 26 ways (paired stale
anchors / unanchored sites in engine.ts and share-link-service.ts —
pure adjacent-row line rot, zero population change). Resolution per the
tool, not by hand: took main's page wholesale, then re-derived with
check-system-context-census --fix, which rewrote exactly ONE anchor
(row 21, protocol.ts 1737 to 1741 — this branch's publisher-block
shift). Delta vs origin/main is that single line; bare gate green: 109
sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-aiClaude

Copy link
Copy Markdown
Collaborator

Contract review — CONTRACT_REVIEW_TIER — verdict: PASS

Delegated review (dispatching PM seat session_01Q5WBDtaUnoz5XuJ6jk8pQ5 is below tier). Reviewed against the ruling on #13331 (comment 5486837547, director batch A, 「同意。」 on Option A). This is a plain comment, not an approval; the reviewer seat does not flip ready, enqueue, or merge.

Instruments. The diff was measured through the API (get_files / get_diff), never through the shallow checkout's three-dot diff: 9 files, +1199/−45, which sums exactly to the PR's own changed_files/additions/deletions — the file list is complete by that control. File contents were read at the pinned SHAs 9dd022ae (base) and 0cf0a437 (head) via git show/git grep (both objects present locally, verified with cat-file -e). Every zero-reading below carried a positive control on the same instrument that fired non-zero. Locations below are by symbol.


Clause-② limbs, judged from the diff

Content limb: YES — declared honestly and completely. The minted public surface, measured:

  • METADATA_MUTATION_CLUSTER_CHANNEL = 'metadata.mutated' + ClusterMetadataMutationPayload, exported from the @objectstack/metadata-protocol barrel (the only two new exports in packages/metadata-protocol/src/index.ts).
  • attachMetadataMutationPubSub(pubsub, nodeId) / detachMetadataMutationPubSub() as public methods on the already-exported ObjectStackProtocolImplementation.
  • Base-tree zero, with control: git grep at 9dd022ae for metadata.mutated|attachMetadataMutationPubSub|METADATA_MUTATION_CLUSTER_CHANNEL → 0 hits (control on same ref, metadata.changed in metadata-manager.ts → 3). Nothing pre-existing is widened or shadowed; the PR mints exactly what it declares.
  • No undeclared widening found.service-cluster's barrel index.ts is untouched (so the anticipated fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 conflict genuinely cannot arise); the bridge's lane 2 is duck-typed plugin behavior, no new export; emitMetadataMutation's split into notifyMutationListenersLocal is private; the changeset, the pinned-doubles ledger entry, and the census row are process artifacts, not surface.

Path limb: NO — measured, not taken from the author. From the complete API file list: zero paths under packages/spec/**, zero writes to packages/objectql/src/engine.ts. That is both fences, held:

  • engine.ts fence: held (0 files touch it; the authz precedent at attachAuthzInvalidationPubSub was mirrored in shape only — see placement below).
  • packages/spec fence: held, and no contract-first split is owed — see placement.

Is the contract minimal and right-shaped?

Two lanes are correct, not a duplicate. Measured, all at head:

  • Lane 1's only production publisher is MetadataManager.notifyWatchers (metadata-manager.ts, the single non-doc publish(MetadataManager.CLUSTER_CHANNEL…) in the tree; the two other grep hits are JSDoc examples in service-cluster).
  • protocol.ts contains zero code references to the metadata service or notifyWatchers — the only 3 hits are doc comments this PR itself adds. The authoring path (saveMetaItemapplyRegistryWriteThrough) cannot reach lane 1 on any boot shape.
  • The lanes carry different disciplines for different state owners: metadata.changed replays the legacy watch event verbatim (content on the wire) into the metadata SERVICE's caches; metadata.mutated is address-only into the ObjectQL engine registry. Overloading lane 1 with a signal-only receipt would collide with its content-replay contract; and the host-config boot has no manager at all (CORE_FALLBACK_FACTORIES = metadata/cache/queue/i18n — the fallback has no cluster seam, and there is no protocol fallback either, so lane 2 duck-types the real implementation or skips). The author's argument tests out.
  • Wiring is real, not just harness-deep: runtime.ts mounts MetadataClusterBridgePlugin unconditionally, and 'protocol' is registered by exactly one production site (assembleMetadataProtocolctx.registerService('protocol', protocolShim) where the "shim" is the ObjectStackProtocolImplementation instance), reached via ObjectQLPlugin's default registerProtocol = true. The variable-key fallback blind spot that inverted the original card does not apply here.

The payload is genuinely address-only, and the receiver genuinely re-reads.MetadataMutationEvent at head is exactly {type, name, state, organizationId?} — no body field exists on the type; all three emit sites construct only those four fields (the item body goes to the awaited projector, which is a separate mechanism, not the event). On receipt, applyRemoteMetadataMutation consumes the wire only as an ADDRESS: type folded through canonicalMetaType, repo.get(ref, {state:'active'}) against the replica's own store, item taken from current.body (own DB), packageId from resolveOverlayPackageBinding (own DB, the #4636 recovery-caller derivation, verbatim). Wire state is consumed only as a draft-suppression guard. Nothing from the wire is applied. The key-set test pins this at runtime.

Attach/detach shape. Idempotent on the (pubsub, nodeId) pair with a different-pair re-attach detaching first — measured side-by-side identical to MetadataManager.attachClusterPubSub. Loopback via originNode equality. Detach idempotent. The bridge detaches both lanes independently on kernel:shutdown with error isolation (a throwing lane-1 detach doesn't strand lane 2 — pinned). Lane 2 carries isInProcessClusterDriver from birth, matching AuthzClusterBridgePlugin, and lane 1 stays byte-identical (its warn line verbatim — pinned), leaving #14021 unabsorbed as declared.

Scope parity of the receipt. The write-through's env gate and org-scope verdict (hydrateOverlayIntoRegistry, #6602) and the heal walk's org refusal (restoreArtifactRegistryView returns for organizationId !== null, #6780) are inherited, not re-decided — a peer applies exactly what the writer's own kernel would have applied locally. A forged org-scoped object signal fails safe: the peer's own read finds no such row, the heal walk's org gate refuses, no-op.

Failure directions

  • Duplicate delivery: same read, same idempotent registration — pinned.
  • Out-of-order: the DB read decides, not the event name — the draft-discard test pins the sharpest case (a "delete"-shaped signal whose read finds an active row re-registers instead of healing).
  • Lost signal: degrades to the pre-existing bound (stale until boot reload). This matches IPubSub's own contract doc at head verbatim ("no shipped driver exceeds at-most-once… Handlers MUST be idempotent and tolerate loss"); the PR promises exactly one hop of narrowing and nothing more, in the changeset and the attach docblock.
  • No row on the peer: heal walk, org-gated as above.
  • Draft: never published (publisher gate) plus a receiver guard.
  • Failed apply: caught, warned, dropped → fail-closed 404 staleness, never wrong data.

Beyond-minimum: the local listener replay — sound, and in scope

The ruling's clause 1 orders the shipped bridge shape mirrored, and the shipped template itself ships remote replay: MetadataManager.attachClusterPubSub's receipt handler invalidates first, then replays into notifyWatchersLocal (measured at head). The PR's replay is the same shape with the same #5109 invalidate-before-notify ordering (registry first, listeners second — pinned by a test that asserts registry state from inside the listener). Never re-published — the local/publish split plus the docblock's storm reasoning is correct, since the loopback guard only suppresses a node's own messages. Replay targets measured: ObjectQL's hook/action rebind and the i18n authored-translation sync are per-replica in-memory re-syncs from the replica's own reads — precisely the issue's second-order defect (runtime-authored automation being single-node). The consumers with shared-DB side effects (email template, permission-set projection) run through the awaited projector seam on the shipped protocol, which is invoked at the write sites only and is NOT replayed — so no fleet-wide duplication of projection writes. Verdict: not a design fork; it is inside the ruled shape, and it is what makes the fix complete for peers.

Placement: metadata-protocol is the right home

  • The state that goes stale (the engine-registry write-through) and the sys_metadata read the ruled receipt needs both live in the protocol; the engine template was shape-only.
  • Both shipped precedents put channel constants and payloads with their state owners, not in spec — measured: AUTHZ_INVALIDATED_CHANNEL in packages/core/src/security/authz-invalidation-channel.ts; ClusterMetadataChangedPayload in packages/metadata/src/metadata-manager.ts. IPubSub from spec/contracts is the already-shipped generic transport (type-only import; no new dependency edge — metadata-protocol already imports spec).
  • The spec's MetadataChangedEventPayloadSchema (cluster.zod.ts): measured dormant — the only reference outside packages/spec in the whole tree is a prose comment in metadata-manager.ts (control: ClusterCapabilityConfig, same instrument, 4+ consuming packages). Its declared semantics ("compare version with their cached value… out-of-order older versions are ignored") make the wire the thing trusted, the opposite of the ruled re-read receipt; and its version: z.bigint() cannot cross JSON.stringify at all. Filing [finding] spec: MetadataChangedEventPayloadSchema says every metadata persistence layer MUST emit it — zero producers or consumers in-tree, and its bigint version field cannot cross JSON #14180 for the spec lane instead of building on it was correct. No spec-lane contract is wearing local clothes here.

Findings

  1. Non-blocking — concurrent same-row mutations can interleave at the peer. The applier is fire-and-forget per message with no per-key serialization: writer saves v1 then v2 in quick succession; the peer's read for the v1 signal (dispatched first, completed before v2's commit) can — under DB-response jitter — deliver its continuation after the v2 applier has registered, leaving the peer's registry at v1 while sys_metadata holds v2, until the next signal or boot. partitionKey orders delivery, not apply completion. Why non-blocking: the registered body is a genuinely-persisted prior state (never fabricated, never wrong-tenant), the window is one DB-read RTT, any later mutation re-converges, the same interleave class already exists between two racing local writers, and the outcome sits inside the staleness bound the PR explicitly declares (equivalent to a lost signal). If it ever matters in practice, a per-type:name apply queue is a receiver-internal fix that touches no public contract.
  2. Non-blocking — one more silent seam of the metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 family.deleteMetaItem's legacy control-plane branch (code-only rows; no repo path) deletes the row and heals its own registry but never emitted the mutation event even locally — pre-existing and marked deliberate in code ("emits no watch event… pre-existing and deliberate") — so it fans out to no one. Same class as the recovery doors metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 records, but metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 names only rollbackMetaItem/revertCommit. Recommend appending this branch to metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 rather than widening this PR.
  3. Non-blocking — wording. The payload docblock's "at-least-once delivery and duplicates are harmless by construction" states a tolerance, while shipped drivers are at-most-once; a skimming reader could take it for a delivery guarantee. The attach docblock, changeset, and PR body all state the loss bound plainly, so no change is required — noted only so nobody later quotes the payload docblock as a promise.

Also verified

  • The census docs row in this diff is exactly the claimed anchor shift: context?.isSystem in stripReadonlyForInsert measured at protocol.ts:1737 at base and :1741 at head; the docs hunk changes that one number and zero prose. Not scope creep.
  • The write-through caller pin re-opened correctly: measured exactly five this.applyRegistryWriteThrough( sites at head (base: four), the fifth being applyRemoteMetadataMutation; the route-5 case drives the real subscribe path and pins the wire-plural fold.
  • emitMetadataMutation's callers at head are exactly saveMetaItem / runPublishSideEffects / deleteMetaItem — the claimed choke point — and the recovery doors run the write-through without emitting, confirming metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 as a pre-existing gap, not this PR's.
  • CI at head 0cf0a437: 33/33 check runs green (Lint & Repo Gates included). I did not re-run the suites locally; suite verdicts are read from CI and the dev reports, all other claims above are my own measurements.

Verdict: PASS. No blocking findings; no design fork the ruling leaves unsettled. The enqueue-gate decision returns to the dispatching seat.


Generated by Claude Code

Same mechanism as the previous merge: main's #14199 re-anchored 11 rows
of this machine-maintained table while this branch re-anchors row 21,
and a textually clean merge proves nothing about the line numbers.
Resolution per the tool: took main's page wholesale, re-derived with
check-system-context-census --fix on the merged tree — exactly ONE
anchor rewritten (row 21, protocol.ts 1737 to 1741; re-derived, not
carried — main has zero commits on protocol.ts, so the insertion shift
is unchanged). Delta vs origin/main is that single line; bare gate
green (109 sites, 145 anchors). engine-double-contract ledger: zero
main-side commits, gate OK on the merged tree — no regeneration owed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-ai
os-support-ai added this pull request to the merge queueSep 1, 2026
Merged via the queue into main with commit 1403d94Sep 1, 2026
34 checks passed
@os-support-ai
os-support-ai deleted the claude/issue-13331-metadata-registry-cluster-fanout branch September 1, 2026 14:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/xlteststooling

Projects

None yet

2 participants

@os-support-ai@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas - #14183

Merged
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout
Sep 1, 2026
Merged

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas#14183
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout

Conversation

@claude

@claudeclaudeBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes#13331

What this ships

A runtime-authored metadata mutation now reaches every replica's ObjectQL registry, not just the writer's. Measured defect on the shipped 3-replica EE compose (ADR-0018): PUT /api/v1/meta/object/... persisted to the shared sys_metadata (meta reads 200 fleet-wide) while only the writing replica registered the object — /api/v1/data/... answered a hard 404 OBJECT_NOT_FOUND on the other two, indefinitely; 200 concurrent creates through the LB gave 67x201 / 133x404 with a boot-loaded control object at 0 errors.

Mechanism (maintainer-ruled 2026-09-01, director batch A, Option A — verbatim adoption of the escalation's recommendation):

  • Publisher at the producer choke point. The protocol's post-persistence funnel (emitMetadataMutation, the seam onMetadataMutation subscribes — reached by saveMetaItem, runPublishSideEffects, deleteMetaItem) now also publishes the mutation's ADDRESS on a new cluster channel metadata.mutated (METADATA_MUTATION_CLUSTER_CHANNEL, payload ClusterMetadataMutationPayload, both exported from @objectstack/metadata-protocol). Drafts are never published — they enter no registry anywhere.
  • Peers converge from their own DB read. On receipt a replica re-reads the row from its OWN sys_metadata: active row present ⇒ re-run applyRegistryWriteThrough (package binding derived the way the recovery callers derive it); no active row ⇒ run the delete heal walk (restoreArtifactRegistryView). The payload is a signal, never trusted content; duplicates and out-of-order delivery converge to the row's current state by construction. After convergence the event replays into the replica's LOCAL onMetadataMutation listeners (never re-published), so authored hook/action re-binds re-sync on peers.
  • Attach seam + bridge lane.ObjectStackProtocolImplementation.attachMetadataMutationPubSub(pubsub, nodeId) — idempotent on the pair, loopback-suppressed via originNode, mirroring MetadataManager.attachClusterPubSub and the engine's attachAuthzInvalidationPubSub. MetadataClusterBridgePlugin late-binds it at kernel:ready as a second, independent lane: the boot shape that lacks a manager-backed metadata service (TS-config host-config — exactly the shipped EE shape) is the one that needs this lane most. The new lane skips the in-process memory driver from birth (the guard the authz sibling carries).
  • Option B (consumer-side self-heal at assertObjectRegistered) is not built — presented only as a possible stopgap; the maintainer did not order one.

The two fences, both held

Evidence

  • Two-arm, directions declared in the test-file header before running (protocol.cluster-mutation-fanout.test.ts, two replicas over one shared store): Arm B (attached) — the peer converges from its own read; Arm A (control, no bridge) — the identical write leaves the peer empty, so Arm B is the bridge's doing, not a harness artifact. Also pinned: address-only payload (key set equality), draft silence, loopback suppression, duplicate-delivery convergence, re-attach idempotency, detach, delete fan-out, draft-discard keeping the peer's active registration, and listener replay ordered after registry convergence.
  • Committed-tree ablation at 44a2e59 (script with trap restore, absolute paths): the single publisher call deleted — mutation proved on disk (anchor count 0, marker count 1) — turned exactly the 7 declared bus-crossing cases red with 15 staying green (controls held); restore proven by HEAD-blob equality (d7ba0225...) plus empty git diff HEAD plus marker count 0; restored run 22/22 green. Resolution note: these tests import ./protocol.js relative from src — the subject never resolves through dist, so no rebuild leg exists to skip; the on-disk grep is the falsifiable observation for both legs.
  • Suites at final head e460193: metadata-protocol 152 files passed / 2 skipped (2094 tests passed / 10 skipped), service-cluster 6 files / 79 tests passed, tsc --noEmit clean with both edited test files confirmed inside the program via --listFiles (2 hits).
  • Gate union at e460193 (node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands; provenance line names this repo at this commit and the --repo assertion holds): 44 derived; 41 exit 0; 3 NOT MEASURED by their own printed verdicts (check-test-completeness exit 3 — grades a CI turbo log; check:dual-build-cjs-loads exit 3 — needs the full workspace build; check:type-check-debt exit 3 — full re-measure). Two first-pass reds were fixed and re-run to their own OK lines: check:engine-double-contract ("OK — 741 pinned, 134 in the DEBT ledger, 3 exempt"; ledger learned the fanout file's pinned doubles via the gate's own --write) and check:objectql-double-limit ("conformance holds: 299 doubles graded"; the new find double now holds the caller's bound by presence).
  • The write-through caller pin re-opened deliberately: applyRegistryWriteThrough grew its FIFTH caller (the peer applier). The trace, the count pin, and a new route-5 spelling case (a plural delivered over the wire registers under the singular; fold via canonicalMetaType, the complete map) are updated in protocol.object-registry-write-through-spelling.test.ts.

Notes for contract review

CI rework — census re-anchor (cdbe5f0)

Lint & Repo Gates red at e460193: check-system-context-census — pure line rot from this PR's own insertion (the publisher block sits above stripReadonlyForInsert, moving its context.isSystem read from protocol.ts:1737 to :1741 while content/docs/permissions/system-context.mdx row 21 still anchored 1737). Repaired with the gate's own --fix and VERIFIED as a SHIFT, not a population change: the rewrite touches exactly one line — the anchor's line number — and zero prose, and the census population is unchanged (109 sites / 145 anchors before and after). The docs file joining this diff is that re-anchor, not scope creep. Reproduced red locally first, then the bare gate re-run green: "OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read."

Docs-drift assessment (stated, not rewritten, per dispatch): content/docs/kernel/cluster.mdx §6.2 is now INCOMPLETE — it says cross-node metadata invalidation "already works" through the manager's metadata.changed lane alone, which is exactly the account #13331 falsified for the runtime-authoring path (the protocol never reaches the manager, and host-config boots carry no manager at all). The page wants a paragraph on the metadata.mutated protocol lane and the bridge's second attach; its own closing principle there ("the peer re-reads the shared store, which is the source of truth") is the rule the new receipt implements, and its "Target spec (planned)" block already marks MetadataChangedEventPayloadSchema as not wired — corroborating #14180. Same incompleteness class applies to the §5/§7 bridge mentions and content/docs/concepts/metadata-lifecycle.mdx.

Generated by Claude Code


Generated by Claude Code


Generated by Claude Code

…ns out to peer replicas (#13331)
Publisher at the protocol's post-persistence choke point publishes the
mutation's address on the new metadata.mutated cluster channel; peers
converge their ObjectQL registry from their OWN sys_metadata read (write-
through when an active row exists, the delete heal walk when none does),
then replay the event into local onMetadataMutation listeners. The bridge
plugin late-binds attachMetadataMutationPubSub as a second independent
lane at kernel:ready, guarded off the in-process memory driver.
Ruled 2026-09-01 (director batch A, Option A): producer-side fan-out
mirroring the shipped AuthzClusterBridgePlugin shape; the payload is a
signal, never trusted content. Option B (consumer-side self-heal) is not
built.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
…ouble; ledger the new pinned engine doubles
check:objectql-double-limit named the new find double limit-blind — the
bound now applies after the filter, by presence. The engine-double ledger
learned the fanout file's pinned delete/findOne/update doubles via the
gate's own --write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@github-actions

github-actionsBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/metadata-protocol, @objectstack/service-cluster, touching 19 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

5 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/wire-format.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/concepts/metadata-lifecycle.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), ObjectStackProtocolImplementation (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/deployment/validating-metadata.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/kernel/cluster.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/kernel/services-checklist.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class))

3 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v15.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/releases/v16.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts) — pages documenting those are invisible to this run
  • 1 anchor(s) matched too much of the corpus to be a work list: /api/v1/data (route, 35 pages)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 11 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fcpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 1304ec033ce539d92069afd7ac66110cc1951b11 — the merge of head 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec into base e62129153f2aa3d7666eab9c1af2bdb18e7333fc, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 1304ec033ce539d92069afd7ac66110cc1951b11 && git checkout 1304ec033ce539d92069afd7ac66110cc1951b11
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin e62129153f2aa3d7666eab9c1af2bdb18e7333fc 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec && git checkout -B drift-repro e62129153f2aa3d7666eab9c1af2bdb18e7333fc && git merge --no-ff 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec
node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fc

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs e62129153f2aa3d7666eab9c1af2bdb18e7333fc → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 1, 2026
…he fan-out insertion
check-system-context-census red on CI: the metadata.mutated publisher
block inserted above stripReadonlyForInsert moved the context.isSystem
read 1737 -> 1741, and the census page still anchored the old line. The
gate's own --fix performed the shift (one line-number rewrite, zero
prose); bare re-run green: 109 sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
The auto-merge of origin/main kept this branch's pre-#14171 anchor rows
beside the row-21 edit, flunking the census 26 ways (paired stale
anchors / unanchored sites in engine.ts and share-link-service.ts —
pure adjacent-row line rot, zero population change). Resolution per the
tool, not by hand: took main's page wholesale, then re-derived with
check-system-context-census --fix, which rewrote exactly ONE anchor
(row 21, protocol.ts 1737 to 1741 — this branch's publisher-block
shift). Delta vs origin/main is that single line; bare gate green: 109
sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-aiClaude

Copy link
Copy Markdown
Collaborator

Contract review — CONTRACT_REVIEW_TIER — verdict: PASS

Delegated review (dispatching PM seat session_01Q5WBDtaUnoz5XuJ6jk8pQ5 is below tier). Reviewed against the ruling on #13331 (comment 5486837547, director batch A, 「同意。」 on Option A). This is a plain comment, not an approval; the reviewer seat does not flip ready, enqueue, or merge.

Instruments. The diff was measured through the API (get_files / get_diff), never through the shallow checkout's three-dot diff: 9 files, +1199/−45, which sums exactly to the PR's own changed_files/additions/deletions — the file list is complete by that control. File contents were read at the pinned SHAs 9dd022ae (base) and 0cf0a437 (head) via git show/git grep (both objects present locally, verified with cat-file -e). Every zero-reading below carried a positive control on the same instrument that fired non-zero. Locations below are by symbol.


Clause-② limbs, judged from the diff

Content limb: YES — declared honestly and completely. The minted public surface, measured:

  • METADATA_MUTATION_CLUSTER_CHANNEL = 'metadata.mutated' + ClusterMetadataMutationPayload, exported from the @objectstack/metadata-protocol barrel (the only two new exports in packages/metadata-protocol/src/index.ts).
  • attachMetadataMutationPubSub(pubsub, nodeId) / detachMetadataMutationPubSub() as public methods on the already-exported ObjectStackProtocolImplementation.
  • Base-tree zero, with control: git grep at 9dd022ae for metadata.mutated|attachMetadataMutationPubSub|METADATA_MUTATION_CLUSTER_CHANNEL → 0 hits (control on same ref, metadata.changed in metadata-manager.ts → 3). Nothing pre-existing is widened or shadowed; the PR mints exactly what it declares.
  • No undeclared widening found.service-cluster's barrel index.ts is untouched (so the anticipated fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 conflict genuinely cannot arise); the bridge's lane 2 is duck-typed plugin behavior, no new export; emitMetadataMutation's split into notifyMutationListenersLocal is private; the changeset, the pinned-doubles ledger entry, and the census row are process artifacts, not surface.

Path limb: NO — measured, not taken from the author. From the complete API file list: zero paths under packages/spec/**, zero writes to packages/objectql/src/engine.ts. That is both fences, held:

  • engine.ts fence: held (0 files touch it; the authz precedent at attachAuthzInvalidationPubSub was mirrored in shape only — see placement below).
  • packages/spec fence: held, and no contract-first split is owed — see placement.

Is the contract minimal and right-shaped?

Two lanes are correct, not a duplicate. Measured, all at head:

  • Lane 1's only production publisher is MetadataManager.notifyWatchers (metadata-manager.ts, the single non-doc publish(MetadataManager.CLUSTER_CHANNEL…) in the tree; the two other grep hits are JSDoc examples in service-cluster).
  • protocol.ts contains zero code references to the metadata service or notifyWatchers — the only 3 hits are doc comments this PR itself adds. The authoring path (saveMetaItemapplyRegistryWriteThrough) cannot reach lane 1 on any boot shape.
  • The lanes carry different disciplines for different state owners: metadata.changed replays the legacy watch event verbatim (content on the wire) into the metadata SERVICE's caches; metadata.mutated is address-only into the ObjectQL engine registry. Overloading lane 1 with a signal-only receipt would collide with its content-replay contract; and the host-config boot has no manager at all (CORE_FALLBACK_FACTORIES = metadata/cache/queue/i18n — the fallback has no cluster seam, and there is no protocol fallback either, so lane 2 duck-types the real implementation or skips). The author's argument tests out.
  • Wiring is real, not just harness-deep: runtime.ts mounts MetadataClusterBridgePlugin unconditionally, and 'protocol' is registered by exactly one production site (assembleMetadataProtocolctx.registerService('protocol', protocolShim) where the "shim" is the ObjectStackProtocolImplementation instance), reached via ObjectQLPlugin's default registerProtocol = true. The variable-key fallback blind spot that inverted the original card does not apply here.

The payload is genuinely address-only, and the receiver genuinely re-reads.MetadataMutationEvent at head is exactly {type, name, state, organizationId?} — no body field exists on the type; all three emit sites construct only those four fields (the item body goes to the awaited projector, which is a separate mechanism, not the event). On receipt, applyRemoteMetadataMutation consumes the wire only as an ADDRESS: type folded through canonicalMetaType, repo.get(ref, {state:'active'}) against the replica's own store, item taken from current.body (own DB), packageId from resolveOverlayPackageBinding (own DB, the #4636 recovery-caller derivation, verbatim). Wire state is consumed only as a draft-suppression guard. Nothing from the wire is applied. The key-set test pins this at runtime.

Attach/detach shape. Idempotent on the (pubsub, nodeId) pair with a different-pair re-attach detaching first — measured side-by-side identical to MetadataManager.attachClusterPubSub. Loopback via originNode equality. Detach idempotent. The bridge detaches both lanes independently on kernel:shutdown with error isolation (a throwing lane-1 detach doesn't strand lane 2 — pinned). Lane 2 carries isInProcessClusterDriver from birth, matching AuthzClusterBridgePlugin, and lane 1 stays byte-identical (its warn line verbatim — pinned), leaving #14021 unabsorbed as declared.

Scope parity of the receipt. The write-through's env gate and org-scope verdict (hydrateOverlayIntoRegistry, #6602) and the heal walk's org refusal (restoreArtifactRegistryView returns for organizationId !== null, #6780) are inherited, not re-decided — a peer applies exactly what the writer's own kernel would have applied locally. A forged org-scoped object signal fails safe: the peer's own read finds no such row, the heal walk's org gate refuses, no-op.

Failure directions

  • Duplicate delivery: same read, same idempotent registration — pinned.
  • Out-of-order: the DB read decides, not the event name — the draft-discard test pins the sharpest case (a "delete"-shaped signal whose read finds an active row re-registers instead of healing).
  • Lost signal: degrades to the pre-existing bound (stale until boot reload). This matches IPubSub's own contract doc at head verbatim ("no shipped driver exceeds at-most-once… Handlers MUST be idempotent and tolerate loss"); the PR promises exactly one hop of narrowing and nothing more, in the changeset and the attach docblock.
  • No row on the peer: heal walk, org-gated as above.
  • Draft: never published (publisher gate) plus a receiver guard.
  • Failed apply: caught, warned, dropped → fail-closed 404 staleness, never wrong data.

Beyond-minimum: the local listener replay — sound, and in scope

The ruling's clause 1 orders the shipped bridge shape mirrored, and the shipped template itself ships remote replay: MetadataManager.attachClusterPubSub's receipt handler invalidates first, then replays into notifyWatchersLocal (measured at head). The PR's replay is the same shape with the same #5109 invalidate-before-notify ordering (registry first, listeners second — pinned by a test that asserts registry state from inside the listener). Never re-published — the local/publish split plus the docblock's storm reasoning is correct, since the loopback guard only suppresses a node's own messages. Replay targets measured: ObjectQL's hook/action rebind and the i18n authored-translation sync are per-replica in-memory re-syncs from the replica's own reads — precisely the issue's second-order defect (runtime-authored automation being single-node). The consumers with shared-DB side effects (email template, permission-set projection) run through the awaited projector seam on the shipped protocol, which is invoked at the write sites only and is NOT replayed — so no fleet-wide duplication of projection writes. Verdict: not a design fork; it is inside the ruled shape, and it is what makes the fix complete for peers.

Placement: metadata-protocol is the right home

  • The state that goes stale (the engine-registry write-through) and the sys_metadata read the ruled receipt needs both live in the protocol; the engine template was shape-only.
  • Both shipped precedents put channel constants and payloads with their state owners, not in spec — measured: AUTHZ_INVALIDATED_CHANNEL in packages/core/src/security/authz-invalidation-channel.ts; ClusterMetadataChangedPayload in packages/metadata/src/metadata-manager.ts. IPubSub from spec/contracts is the already-shipped generic transport (type-only import; no new dependency edge — metadata-protocol already imports spec).
  • The spec's MetadataChangedEventPayloadSchema (cluster.zod.ts): measured dormant — the only reference outside packages/spec in the whole tree is a prose comment in metadata-manager.ts (control: ClusterCapabilityConfig, same instrument, 4+ consuming packages). Its declared semantics ("compare version with their cached value… out-of-order older versions are ignored") make the wire the thing trusted, the opposite of the ruled re-read receipt; and its version: z.bigint() cannot cross JSON.stringify at all. Filing [finding] spec: MetadataChangedEventPayloadSchema says every metadata persistence layer MUST emit it — zero producers or consumers in-tree, and its bigint version field cannot cross JSON #14180 for the spec lane instead of building on it was correct. No spec-lane contract is wearing local clothes here.

Findings

  1. Non-blocking — concurrent same-row mutations can interleave at the peer. The applier is fire-and-forget per message with no per-key serialization: writer saves v1 then v2 in quick succession; the peer's read for the v1 signal (dispatched first, completed before v2's commit) can — under DB-response jitter — deliver its continuation after the v2 applier has registered, leaving the peer's registry at v1 while sys_metadata holds v2, until the next signal or boot. partitionKey orders delivery, not apply completion. Why non-blocking: the registered body is a genuinely-persisted prior state (never fabricated, never wrong-tenant), the window is one DB-read RTT, any later mutation re-converges, the same interleave class already exists between two racing local writers, and the outcome sits inside the staleness bound the PR explicitly declares (equivalent to a lost signal). If it ever matters in practice, a per-type:name apply queue is a receiver-internal fix that touches no public contract.
  2. Non-blocking — one more silent seam of the metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 family.deleteMetaItem's legacy control-plane branch (code-only rows; no repo path) deletes the row and heals its own registry but never emitted the mutation event even locally — pre-existing and marked deliberate in code ("emits no watch event… pre-existing and deliberate") — so it fans out to no one. Same class as the recovery doors metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 records, but metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 names only rollbackMetaItem/revertCommit. Recommend appending this branch to metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 rather than widening this PR.
  3. Non-blocking — wording. The payload docblock's "at-least-once delivery and duplicates are harmless by construction" states a tolerance, while shipped drivers are at-most-once; a skimming reader could take it for a delivery guarantee. The attach docblock, changeset, and PR body all state the loss bound plainly, so no change is required — noted only so nobody later quotes the payload docblock as a promise.

Also verified

  • The census docs row in this diff is exactly the claimed anchor shift: context?.isSystem in stripReadonlyForInsert measured at protocol.ts:1737 at base and :1741 at head; the docs hunk changes that one number and zero prose. Not scope creep.
  • The write-through caller pin re-opened correctly: measured exactly five this.applyRegistryWriteThrough( sites at head (base: four), the fifth being applyRemoteMetadataMutation; the route-5 case drives the real subscribe path and pins the wire-plural fold.
  • emitMetadataMutation's callers at head are exactly saveMetaItem / runPublishSideEffects / deleteMetaItem — the claimed choke point — and the recovery doors run the write-through without emitting, confirming metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 as a pre-existing gap, not this PR's.
  • CI at head 0cf0a437: 33/33 check runs green (Lint & Repo Gates included). I did not re-run the suites locally; suite verdicts are read from CI and the dev reports, all other claims above are my own measurements.

Verdict: PASS. No blocking findings; no design fork the ruling leaves unsettled. The enqueue-gate decision returns to the dispatching seat.


Generated by Claude Code

Same mechanism as the previous merge: main's #14199 re-anchored 11 rows
of this machine-maintained table while this branch re-anchors row 21,
and a textually clean merge proves nothing about the line numbers.
Resolution per the tool: took main's page wholesale, re-derived with
check-system-context-census --fix on the merged tree — exactly ONE
anchor rewritten (row 21, protocol.ts 1737 to 1741; re-derived, not
carried — main has zero commits on protocol.ts, so the insertion shift
is unchanged). Delta vs origin/main is that single line; bare gate
green (109 sites, 145 anchors). engine-double-contract ledger: zero
main-side commits, gate OK on the merged tree — no regeneration owed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-ai
os-support-ai added this pull request to the merge queueSep 1, 2026
Merged via the queue into main with commit 1403d94Sep 1, 2026
34 checks passed
@os-support-ai
os-support-ai deleted the claude/issue-13331-metadata-registry-cluster-fanout branch September 1, 2026 14:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/xlteststooling

Projects

None yet

2 participants

@os-support-ai@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas - #14183

Merged
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout
Sep 1, 2026
Merged

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas#14183
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout

Conversation

@claude

@claudeclaudeBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes#13331

What this ships

A runtime-authored metadata mutation now reaches every replica's ObjectQL registry, not just the writer's. Measured defect on the shipped 3-replica EE compose (ADR-0018): PUT /api/v1/meta/object/... persisted to the shared sys_metadata (meta reads 200 fleet-wide) while only the writing replica registered the object — /api/v1/data/... answered a hard 404 OBJECT_NOT_FOUND on the other two, indefinitely; 200 concurrent creates through the LB gave 67x201 / 133x404 with a boot-loaded control object at 0 errors.

Mechanism (maintainer-ruled 2026-09-01, director batch A, Option A — verbatim adoption of the escalation's recommendation):

  • Publisher at the producer choke point. The protocol's post-persistence funnel (emitMetadataMutation, the seam onMetadataMutation subscribes — reached by saveMetaItem, runPublishSideEffects, deleteMetaItem) now also publishes the mutation's ADDRESS on a new cluster channel metadata.mutated (METADATA_MUTATION_CLUSTER_CHANNEL, payload ClusterMetadataMutationPayload, both exported from @objectstack/metadata-protocol). Drafts are never published — they enter no registry anywhere.
  • Peers converge from their own DB read. On receipt a replica re-reads the row from its OWN sys_metadata: active row present ⇒ re-run applyRegistryWriteThrough (package binding derived the way the recovery callers derive it); no active row ⇒ run the delete heal walk (restoreArtifactRegistryView). The payload is a signal, never trusted content; duplicates and out-of-order delivery converge to the row's current state by construction. After convergence the event replays into the replica's LOCAL onMetadataMutation listeners (never re-published), so authored hook/action re-binds re-sync on peers.
  • Attach seam + bridge lane.ObjectStackProtocolImplementation.attachMetadataMutationPubSub(pubsub, nodeId) — idempotent on the pair, loopback-suppressed via originNode, mirroring MetadataManager.attachClusterPubSub and the engine's attachAuthzInvalidationPubSub. MetadataClusterBridgePlugin late-binds it at kernel:ready as a second, independent lane: the boot shape that lacks a manager-backed metadata service (TS-config host-config — exactly the shipped EE shape) is the one that needs this lane most. The new lane skips the in-process memory driver from birth (the guard the authz sibling carries).
  • Option B (consumer-side self-heal at assertObjectRegistered) is not built — presented only as a possible stopgap; the maintainer did not order one.

The two fences, both held

Evidence

  • Two-arm, directions declared in the test-file header before running (protocol.cluster-mutation-fanout.test.ts, two replicas over one shared store): Arm B (attached) — the peer converges from its own read; Arm A (control, no bridge) — the identical write leaves the peer empty, so Arm B is the bridge's doing, not a harness artifact. Also pinned: address-only payload (key set equality), draft silence, loopback suppression, duplicate-delivery convergence, re-attach idempotency, detach, delete fan-out, draft-discard keeping the peer's active registration, and listener replay ordered after registry convergence.
  • Committed-tree ablation at 44a2e59 (script with trap restore, absolute paths): the single publisher call deleted — mutation proved on disk (anchor count 0, marker count 1) — turned exactly the 7 declared bus-crossing cases red with 15 staying green (controls held); restore proven by HEAD-blob equality (d7ba0225...) plus empty git diff HEAD plus marker count 0; restored run 22/22 green. Resolution note: these tests import ./protocol.js relative from src — the subject never resolves through dist, so no rebuild leg exists to skip; the on-disk grep is the falsifiable observation for both legs.
  • Suites at final head e460193: metadata-protocol 152 files passed / 2 skipped (2094 tests passed / 10 skipped), service-cluster 6 files / 79 tests passed, tsc --noEmit clean with both edited test files confirmed inside the program via --listFiles (2 hits).
  • Gate union at e460193 (node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands; provenance line names this repo at this commit and the --repo assertion holds): 44 derived; 41 exit 0; 3 NOT MEASURED by their own printed verdicts (check-test-completeness exit 3 — grades a CI turbo log; check:dual-build-cjs-loads exit 3 — needs the full workspace build; check:type-check-debt exit 3 — full re-measure). Two first-pass reds were fixed and re-run to their own OK lines: check:engine-double-contract ("OK — 741 pinned, 134 in the DEBT ledger, 3 exempt"; ledger learned the fanout file's pinned doubles via the gate's own --write) and check:objectql-double-limit ("conformance holds: 299 doubles graded"; the new find double now holds the caller's bound by presence).
  • The write-through caller pin re-opened deliberately: applyRegistryWriteThrough grew its FIFTH caller (the peer applier). The trace, the count pin, and a new route-5 spelling case (a plural delivered over the wire registers under the singular; fold via canonicalMetaType, the complete map) are updated in protocol.object-registry-write-through-spelling.test.ts.

Notes for contract review

CI rework — census re-anchor (cdbe5f0)

Lint & Repo Gates red at e460193: check-system-context-census — pure line rot from this PR's own insertion (the publisher block sits above stripReadonlyForInsert, moving its context.isSystem read from protocol.ts:1737 to :1741 while content/docs/permissions/system-context.mdx row 21 still anchored 1737). Repaired with the gate's own --fix and VERIFIED as a SHIFT, not a population change: the rewrite touches exactly one line — the anchor's line number — and zero prose, and the census population is unchanged (109 sites / 145 anchors before and after). The docs file joining this diff is that re-anchor, not scope creep. Reproduced red locally first, then the bare gate re-run green: "OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read."

Docs-drift assessment (stated, not rewritten, per dispatch): content/docs/kernel/cluster.mdx §6.2 is now INCOMPLETE — it says cross-node metadata invalidation "already works" through the manager's metadata.changed lane alone, which is exactly the account #13331 falsified for the runtime-authoring path (the protocol never reaches the manager, and host-config boots carry no manager at all). The page wants a paragraph on the metadata.mutated protocol lane and the bridge's second attach; its own closing principle there ("the peer re-reads the shared store, which is the source of truth") is the rule the new receipt implements, and its "Target spec (planned)" block already marks MetadataChangedEventPayloadSchema as not wired — corroborating #14180. Same incompleteness class applies to the §5/§7 bridge mentions and content/docs/concepts/metadata-lifecycle.mdx.

Generated by Claude Code


Generated by Claude Code


Generated by Claude Code

…ns out to peer replicas (#13331)
Publisher at the protocol's post-persistence choke point publishes the
mutation's address on the new metadata.mutated cluster channel; peers
converge their ObjectQL registry from their OWN sys_metadata read (write-
through when an active row exists, the delete heal walk when none does),
then replay the event into local onMetadataMutation listeners. The bridge
plugin late-binds attachMetadataMutationPubSub as a second independent
lane at kernel:ready, guarded off the in-process memory driver.
Ruled 2026-09-01 (director batch A, Option A): producer-side fan-out
mirroring the shipped AuthzClusterBridgePlugin shape; the payload is a
signal, never trusted content. Option B (consumer-side self-heal) is not
built.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
…ouble; ledger the new pinned engine doubles
check:objectql-double-limit named the new find double limit-blind — the
bound now applies after the filter, by presence. The engine-double ledger
learned the fanout file's pinned delete/findOne/update doubles via the
gate's own --write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@github-actions

github-actionsBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/metadata-protocol, @objectstack/service-cluster, touching 19 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

5 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/wire-format.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/concepts/metadata-lifecycle.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), ObjectStackProtocolImplementation (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/deployment/validating-metadata.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/kernel/cluster.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/kernel/services-checklist.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class))

3 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v15.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/releases/v16.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts) — pages documenting those are invisible to this run
  • 1 anchor(s) matched too much of the corpus to be a work list: /api/v1/data (route, 35 pages)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 11 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fcpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 1304ec033ce539d92069afd7ac66110cc1951b11 — the merge of head 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec into base e62129153f2aa3d7666eab9c1af2bdb18e7333fc, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 1304ec033ce539d92069afd7ac66110cc1951b11 && git checkout 1304ec033ce539d92069afd7ac66110cc1951b11
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin e62129153f2aa3d7666eab9c1af2bdb18e7333fc 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec && git checkout -B drift-repro e62129153f2aa3d7666eab9c1af2bdb18e7333fc && git merge --no-ff 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec
node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fc

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs e62129153f2aa3d7666eab9c1af2bdb18e7333fc → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 1, 2026
…he fan-out insertion
check-system-context-census red on CI: the metadata.mutated publisher
block inserted above stripReadonlyForInsert moved the context.isSystem
read 1737 -> 1741, and the census page still anchored the old line. The
gate's own --fix performed the shift (one line-number rewrite, zero
prose); bare re-run green: 109 sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
The auto-merge of origin/main kept this branch's pre-#14171 anchor rows
beside the row-21 edit, flunking the census 26 ways (paired stale
anchors / unanchored sites in engine.ts and share-link-service.ts —
pure adjacent-row line rot, zero population change). Resolution per the
tool, not by hand: took main's page wholesale, then re-derived with
check-system-context-census --fix, which rewrote exactly ONE anchor
(row 21, protocol.ts 1737 to 1741 — this branch's publisher-block
shift). Delta vs origin/main is that single line; bare gate green: 109
sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-aiClaude

Copy link
Copy Markdown
Collaborator

Contract review — CONTRACT_REVIEW_TIER — verdict: PASS

Delegated review (dispatching PM seat session_01Q5WBDtaUnoz5XuJ6jk8pQ5 is below tier). Reviewed against the ruling on #13331 (comment 5486837547, director batch A, 「同意。」 on Option A). This is a plain comment, not an approval; the reviewer seat does not flip ready, enqueue, or merge.

Instruments. The diff was measured through the API (get_files / get_diff), never through the shallow checkout's three-dot diff: 9 files, +1199/−45, which sums exactly to the PR's own changed_files/additions/deletions — the file list is complete by that control. File contents were read at the pinned SHAs 9dd022ae (base) and 0cf0a437 (head) via git show/git grep (both objects present locally, verified with cat-file -e). Every zero-reading below carried a positive control on the same instrument that fired non-zero. Locations below are by symbol.


Clause-② limbs, judged from the diff

Content limb: YES — declared honestly and completely. The minted public surface, measured:

  • METADATA_MUTATION_CLUSTER_CHANNEL = 'metadata.mutated' + ClusterMetadataMutationPayload, exported from the @objectstack/metadata-protocol barrel (the only two new exports in packages/metadata-protocol/src/index.ts).
  • attachMetadataMutationPubSub(pubsub, nodeId) / detachMetadataMutationPubSub() as public methods on the already-exported ObjectStackProtocolImplementation.
  • Base-tree zero, with control: git grep at 9dd022ae for metadata.mutated|attachMetadataMutationPubSub|METADATA_MUTATION_CLUSTER_CHANNEL → 0 hits (control on same ref, metadata.changed in metadata-manager.ts → 3). Nothing pre-existing is widened or shadowed; the PR mints exactly what it declares.
  • No undeclared widening found.service-cluster's barrel index.ts is untouched (so the anticipated fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 conflict genuinely cannot arise); the bridge's lane 2 is duck-typed plugin behavior, no new export; emitMetadataMutation's split into notifyMutationListenersLocal is private; the changeset, the pinned-doubles ledger entry, and the census row are process artifacts, not surface.

Path limb: NO — measured, not taken from the author. From the complete API file list: zero paths under packages/spec/**, zero writes to packages/objectql/src/engine.ts. That is both fences, held:

  • engine.ts fence: held (0 files touch it; the authz precedent at attachAuthzInvalidationPubSub was mirrored in shape only — see placement below).
  • packages/spec fence: held, and no contract-first split is owed — see placement.

Is the contract minimal and right-shaped?

Two lanes are correct, not a duplicate. Measured, all at head:

  • Lane 1's only production publisher is MetadataManager.notifyWatchers (metadata-manager.ts, the single non-doc publish(MetadataManager.CLUSTER_CHANNEL…) in the tree; the two other grep hits are JSDoc examples in service-cluster).
  • protocol.ts contains zero code references to the metadata service or notifyWatchers — the only 3 hits are doc comments this PR itself adds. The authoring path (saveMetaItemapplyRegistryWriteThrough) cannot reach lane 1 on any boot shape.
  • The lanes carry different disciplines for different state owners: metadata.changed replays the legacy watch event verbatim (content on the wire) into the metadata SERVICE's caches; metadata.mutated is address-only into the ObjectQL engine registry. Overloading lane 1 with a signal-only receipt would collide with its content-replay contract; and the host-config boot has no manager at all (CORE_FALLBACK_FACTORIES = metadata/cache/queue/i18n — the fallback has no cluster seam, and there is no protocol fallback either, so lane 2 duck-types the real implementation or skips). The author's argument tests out.
  • Wiring is real, not just harness-deep: runtime.ts mounts MetadataClusterBridgePlugin unconditionally, and 'protocol' is registered by exactly one production site (assembleMetadataProtocolctx.registerService('protocol', protocolShim) where the "shim" is the ObjectStackProtocolImplementation instance), reached via ObjectQLPlugin's default registerProtocol = true. The variable-key fallback blind spot that inverted the original card does not apply here.

The payload is genuinely address-only, and the receiver genuinely re-reads.MetadataMutationEvent at head is exactly {type, name, state, organizationId?} — no body field exists on the type; all three emit sites construct only those four fields (the item body goes to the awaited projector, which is a separate mechanism, not the event). On receipt, applyRemoteMetadataMutation consumes the wire only as an ADDRESS: type folded through canonicalMetaType, repo.get(ref, {state:'active'}) against the replica's own store, item taken from current.body (own DB), packageId from resolveOverlayPackageBinding (own DB, the #4636 recovery-caller derivation, verbatim). Wire state is consumed only as a draft-suppression guard. Nothing from the wire is applied. The key-set test pins this at runtime.

Attach/detach shape. Idempotent on the (pubsub, nodeId) pair with a different-pair re-attach detaching first — measured side-by-side identical to MetadataManager.attachClusterPubSub. Loopback via originNode equality. Detach idempotent. The bridge detaches both lanes independently on kernel:shutdown with error isolation (a throwing lane-1 detach doesn't strand lane 2 — pinned). Lane 2 carries isInProcessClusterDriver from birth, matching AuthzClusterBridgePlugin, and lane 1 stays byte-identical (its warn line verbatim — pinned), leaving #14021 unabsorbed as declared.

Scope parity of the receipt. The write-through's env gate and org-scope verdict (hydrateOverlayIntoRegistry, #6602) and the heal walk's org refusal (restoreArtifactRegistryView returns for organizationId !== null, #6780) are inherited, not re-decided — a peer applies exactly what the writer's own kernel would have applied locally. A forged org-scoped object signal fails safe: the peer's own read finds no such row, the heal walk's org gate refuses, no-op.

Failure directions

  • Duplicate delivery: same read, same idempotent registration — pinned.
  • Out-of-order: the DB read decides, not the event name — the draft-discard test pins the sharpest case (a "delete"-shaped signal whose read finds an active row re-registers instead of healing).
  • Lost signal: degrades to the pre-existing bound (stale until boot reload). This matches IPubSub's own contract doc at head verbatim ("no shipped driver exceeds at-most-once… Handlers MUST be idempotent and tolerate loss"); the PR promises exactly one hop of narrowing and nothing more, in the changeset and the attach docblock.
  • No row on the peer: heal walk, org-gated as above.
  • Draft: never published (publisher gate) plus a receiver guard.
  • Failed apply: caught, warned, dropped → fail-closed 404 staleness, never wrong data.

Beyond-minimum: the local listener replay — sound, and in scope

The ruling's clause 1 orders the shipped bridge shape mirrored, and the shipped template itself ships remote replay: MetadataManager.attachClusterPubSub's receipt handler invalidates first, then replays into notifyWatchersLocal (measured at head). The PR's replay is the same shape with the same #5109 invalidate-before-notify ordering (registry first, listeners second — pinned by a test that asserts registry state from inside the listener). Never re-published — the local/publish split plus the docblock's storm reasoning is correct, since the loopback guard only suppresses a node's own messages. Replay targets measured: ObjectQL's hook/action rebind and the i18n authored-translation sync are per-replica in-memory re-syncs from the replica's own reads — precisely the issue's second-order defect (runtime-authored automation being single-node). The consumers with shared-DB side effects (email template, permission-set projection) run through the awaited projector seam on the shipped protocol, which is invoked at the write sites only and is NOT replayed — so no fleet-wide duplication of projection writes. Verdict: not a design fork; it is inside the ruled shape, and it is what makes the fix complete for peers.

Placement: metadata-protocol is the right home

  • The state that goes stale (the engine-registry write-through) and the sys_metadata read the ruled receipt needs both live in the protocol; the engine template was shape-only.
  • Both shipped precedents put channel constants and payloads with their state owners, not in spec — measured: AUTHZ_INVALIDATED_CHANNEL in packages/core/src/security/authz-invalidation-channel.ts; ClusterMetadataChangedPayload in packages/metadata/src/metadata-manager.ts. IPubSub from spec/contracts is the already-shipped generic transport (type-only import; no new dependency edge — metadata-protocol already imports spec).
  • The spec's MetadataChangedEventPayloadSchema (cluster.zod.ts): measured dormant — the only reference outside packages/spec in the whole tree is a prose comment in metadata-manager.ts (control: ClusterCapabilityConfig, same instrument, 4+ consuming packages). Its declared semantics ("compare version with their cached value… out-of-order older versions are ignored") make the wire the thing trusted, the opposite of the ruled re-read receipt; and its version: z.bigint() cannot cross JSON.stringify at all. Filing [finding] spec: MetadataChangedEventPayloadSchema says every metadata persistence layer MUST emit it — zero producers or consumers in-tree, and its bigint version field cannot cross JSON #14180 for the spec lane instead of building on it was correct. No spec-lane contract is wearing local clothes here.

Findings

  1. Non-blocking — concurrent same-row mutations can interleave at the peer. The applier is fire-and-forget per message with no per-key serialization: writer saves v1 then v2 in quick succession; the peer's read for the v1 signal (dispatched first, completed before v2's commit) can — under DB-response jitter — deliver its continuation after the v2 applier has registered, leaving the peer's registry at v1 while sys_metadata holds v2, until the next signal or boot. partitionKey orders delivery, not apply completion. Why non-blocking: the registered body is a genuinely-persisted prior state (never fabricated, never wrong-tenant), the window is one DB-read RTT, any later mutation re-converges, the same interleave class already exists between two racing local writers, and the outcome sits inside the staleness bound the PR explicitly declares (equivalent to a lost signal). If it ever matters in practice, a per-type:name apply queue is a receiver-internal fix that touches no public contract.
  2. Non-blocking — one more silent seam of the metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 family.deleteMetaItem's legacy control-plane branch (code-only rows; no repo path) deletes the row and heals its own registry but never emitted the mutation event even locally — pre-existing and marked deliberate in code ("emits no watch event… pre-existing and deliberate") — so it fans out to no one. Same class as the recovery doors metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 records, but metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 names only rollbackMetaItem/revertCommit. Recommend appending this branch to metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 rather than widening this PR.
  3. Non-blocking — wording. The payload docblock's "at-least-once delivery and duplicates are harmless by construction" states a tolerance, while shipped drivers are at-most-once; a skimming reader could take it for a delivery guarantee. The attach docblock, changeset, and PR body all state the loss bound plainly, so no change is required — noted only so nobody later quotes the payload docblock as a promise.

Also verified

  • The census docs row in this diff is exactly the claimed anchor shift: context?.isSystem in stripReadonlyForInsert measured at protocol.ts:1737 at base and :1741 at head; the docs hunk changes that one number and zero prose. Not scope creep.
  • The write-through caller pin re-opened correctly: measured exactly five this.applyRegistryWriteThrough( sites at head (base: four), the fifth being applyRemoteMetadataMutation; the route-5 case drives the real subscribe path and pins the wire-plural fold.
  • emitMetadataMutation's callers at head are exactly saveMetaItem / runPublishSideEffects / deleteMetaItem — the claimed choke point — and the recovery doors run the write-through without emitting, confirming metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 as a pre-existing gap, not this PR's.
  • CI at head 0cf0a437: 33/33 check runs green (Lint & Repo Gates included). I did not re-run the suites locally; suite verdicts are read from CI and the dev reports, all other claims above are my own measurements.

Verdict: PASS. No blocking findings; no design fork the ruling leaves unsettled. The enqueue-gate decision returns to the dispatching seat.


Generated by Claude Code

Same mechanism as the previous merge: main's #14199 re-anchored 11 rows
of this machine-maintained table while this branch re-anchors row 21,
and a textually clean merge proves nothing about the line numbers.
Resolution per the tool: took main's page wholesale, re-derived with
check-system-context-census --fix on the merged tree — exactly ONE
anchor rewritten (row 21, protocol.ts 1737 to 1741; re-derived, not
carried — main has zero commits on protocol.ts, so the insertion shift
is unchanged). Delta vs origin/main is that single line; bare gate
green (109 sites, 145 anchors). engine-double-contract ledger: zero
main-side commits, gate OK on the merged tree — no regeneration owed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-ai
os-support-ai added this pull request to the merge queueSep 1, 2026
Merged via the queue into main with commit 1403d94Sep 1, 2026
34 checks passed
@os-support-ai
os-support-ai deleted the claude/issue-13331-metadata-registry-cluster-fanout branch September 1, 2026 14:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/xlteststooling

Projects

None yet

2 participants

@os-support-ai@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas - #14183

Merged
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout
Sep 1, 2026
Merged

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas#14183
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout

Conversation

@claude

@claudeclaudeBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes#13331

What this ships

A runtime-authored metadata mutation now reaches every replica's ObjectQL registry, not just the writer's. Measured defect on the shipped 3-replica EE compose (ADR-0018): PUT /api/v1/meta/object/... persisted to the shared sys_metadata (meta reads 200 fleet-wide) while only the writing replica registered the object — /api/v1/data/... answered a hard 404 OBJECT_NOT_FOUND on the other two, indefinitely; 200 concurrent creates through the LB gave 67x201 / 133x404 with a boot-loaded control object at 0 errors.

Mechanism (maintainer-ruled 2026-09-01, director batch A, Option A — verbatim adoption of the escalation's recommendation):

  • Publisher at the producer choke point. The protocol's post-persistence funnel (emitMetadataMutation, the seam onMetadataMutation subscribes — reached by saveMetaItem, runPublishSideEffects, deleteMetaItem) now also publishes the mutation's ADDRESS on a new cluster channel metadata.mutated (METADATA_MUTATION_CLUSTER_CHANNEL, payload ClusterMetadataMutationPayload, both exported from @objectstack/metadata-protocol). Drafts are never published — they enter no registry anywhere.
  • Peers converge from their own DB read. On receipt a replica re-reads the row from its OWN sys_metadata: active row present ⇒ re-run applyRegistryWriteThrough (package binding derived the way the recovery callers derive it); no active row ⇒ run the delete heal walk (restoreArtifactRegistryView). The payload is a signal, never trusted content; duplicates and out-of-order delivery converge to the row's current state by construction. After convergence the event replays into the replica's LOCAL onMetadataMutation listeners (never re-published), so authored hook/action re-binds re-sync on peers.
  • Attach seam + bridge lane.ObjectStackProtocolImplementation.attachMetadataMutationPubSub(pubsub, nodeId) — idempotent on the pair, loopback-suppressed via originNode, mirroring MetadataManager.attachClusterPubSub and the engine's attachAuthzInvalidationPubSub. MetadataClusterBridgePlugin late-binds it at kernel:ready as a second, independent lane: the boot shape that lacks a manager-backed metadata service (TS-config host-config — exactly the shipped EE shape) is the one that needs this lane most. The new lane skips the in-process memory driver from birth (the guard the authz sibling carries).
  • Option B (consumer-side self-heal at assertObjectRegistered) is not built — presented only as a possible stopgap; the maintainer did not order one.

The two fences, both held

Evidence

  • Two-arm, directions declared in the test-file header before running (protocol.cluster-mutation-fanout.test.ts, two replicas over one shared store): Arm B (attached) — the peer converges from its own read; Arm A (control, no bridge) — the identical write leaves the peer empty, so Arm B is the bridge's doing, not a harness artifact. Also pinned: address-only payload (key set equality), draft silence, loopback suppression, duplicate-delivery convergence, re-attach idempotency, detach, delete fan-out, draft-discard keeping the peer's active registration, and listener replay ordered after registry convergence.
  • Committed-tree ablation at 44a2e59 (script with trap restore, absolute paths): the single publisher call deleted — mutation proved on disk (anchor count 0, marker count 1) — turned exactly the 7 declared bus-crossing cases red with 15 staying green (controls held); restore proven by HEAD-blob equality (d7ba0225...) plus empty git diff HEAD plus marker count 0; restored run 22/22 green. Resolution note: these tests import ./protocol.js relative from src — the subject never resolves through dist, so no rebuild leg exists to skip; the on-disk grep is the falsifiable observation for both legs.
  • Suites at final head e460193: metadata-protocol 152 files passed / 2 skipped (2094 tests passed / 10 skipped), service-cluster 6 files / 79 tests passed, tsc --noEmit clean with both edited test files confirmed inside the program via --listFiles (2 hits).
  • Gate union at e460193 (node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands; provenance line names this repo at this commit and the --repo assertion holds): 44 derived; 41 exit 0; 3 NOT MEASURED by their own printed verdicts (check-test-completeness exit 3 — grades a CI turbo log; check:dual-build-cjs-loads exit 3 — needs the full workspace build; check:type-check-debt exit 3 — full re-measure). Two first-pass reds were fixed and re-run to their own OK lines: check:engine-double-contract ("OK — 741 pinned, 134 in the DEBT ledger, 3 exempt"; ledger learned the fanout file's pinned doubles via the gate's own --write) and check:objectql-double-limit ("conformance holds: 299 doubles graded"; the new find double now holds the caller's bound by presence).
  • The write-through caller pin re-opened deliberately: applyRegistryWriteThrough grew its FIFTH caller (the peer applier). The trace, the count pin, and a new route-5 spelling case (a plural delivered over the wire registers under the singular; fold via canonicalMetaType, the complete map) are updated in protocol.object-registry-write-through-spelling.test.ts.

Notes for contract review

CI rework — census re-anchor (cdbe5f0)

Lint & Repo Gates red at e460193: check-system-context-census — pure line rot from this PR's own insertion (the publisher block sits above stripReadonlyForInsert, moving its context.isSystem read from protocol.ts:1737 to :1741 while content/docs/permissions/system-context.mdx row 21 still anchored 1737). Repaired with the gate's own --fix and VERIFIED as a SHIFT, not a population change: the rewrite touches exactly one line — the anchor's line number — and zero prose, and the census population is unchanged (109 sites / 145 anchors before and after). The docs file joining this diff is that re-anchor, not scope creep. Reproduced red locally first, then the bare gate re-run green: "OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read."

Docs-drift assessment (stated, not rewritten, per dispatch): content/docs/kernel/cluster.mdx §6.2 is now INCOMPLETE — it says cross-node metadata invalidation "already works" through the manager's metadata.changed lane alone, which is exactly the account #13331 falsified for the runtime-authoring path (the protocol never reaches the manager, and host-config boots carry no manager at all). The page wants a paragraph on the metadata.mutated protocol lane and the bridge's second attach; its own closing principle there ("the peer re-reads the shared store, which is the source of truth") is the rule the new receipt implements, and its "Target spec (planned)" block already marks MetadataChangedEventPayloadSchema as not wired — corroborating #14180. Same incompleteness class applies to the §5/§7 bridge mentions and content/docs/concepts/metadata-lifecycle.mdx.

Generated by Claude Code


Generated by Claude Code


Generated by Claude Code

…ns out to peer replicas (#13331)
Publisher at the protocol's post-persistence choke point publishes the
mutation's address on the new metadata.mutated cluster channel; peers
converge their ObjectQL registry from their OWN sys_metadata read (write-
through when an active row exists, the delete heal walk when none does),
then replay the event into local onMetadataMutation listeners. The bridge
plugin late-binds attachMetadataMutationPubSub as a second independent
lane at kernel:ready, guarded off the in-process memory driver.
Ruled 2026-09-01 (director batch A, Option A): producer-side fan-out
mirroring the shipped AuthzClusterBridgePlugin shape; the payload is a
signal, never trusted content. Option B (consumer-side self-heal) is not
built.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
…ouble; ledger the new pinned engine doubles
check:objectql-double-limit named the new find double limit-blind — the
bound now applies after the filter, by presence. The engine-double ledger
learned the fanout file's pinned delete/findOne/update doubles via the
gate's own --write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@github-actions

github-actionsBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/metadata-protocol, @objectstack/service-cluster, touching 19 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

5 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/wire-format.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/concepts/metadata-lifecycle.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), ObjectStackProtocolImplementation (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/deployment/validating-metadata.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/kernel/cluster.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/kernel/services-checklist.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class))

3 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v15.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/releases/v16.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts) — pages documenting those are invisible to this run
  • 1 anchor(s) matched too much of the corpus to be a work list: /api/v1/data (route, 35 pages)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 11 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fcpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 1304ec033ce539d92069afd7ac66110cc1951b11 — the merge of head 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec into base e62129153f2aa3d7666eab9c1af2bdb18e7333fc, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 1304ec033ce539d92069afd7ac66110cc1951b11 && git checkout 1304ec033ce539d92069afd7ac66110cc1951b11
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin e62129153f2aa3d7666eab9c1af2bdb18e7333fc 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec && git checkout -B drift-repro e62129153f2aa3d7666eab9c1af2bdb18e7333fc && git merge --no-ff 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec
node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fc

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs e62129153f2aa3d7666eab9c1af2bdb18e7333fc → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 1, 2026
…he fan-out insertion
check-system-context-census red on CI: the metadata.mutated publisher
block inserted above stripReadonlyForInsert moved the context.isSystem
read 1737 -> 1741, and the census page still anchored the old line. The
gate's own --fix performed the shift (one line-number rewrite, zero
prose); bare re-run green: 109 sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
The auto-merge of origin/main kept this branch's pre-#14171 anchor rows
beside the row-21 edit, flunking the census 26 ways (paired stale
anchors / unanchored sites in engine.ts and share-link-service.ts —
pure adjacent-row line rot, zero population change). Resolution per the
tool, not by hand: took main's page wholesale, then re-derived with
check-system-context-census --fix, which rewrote exactly ONE anchor
(row 21, protocol.ts 1737 to 1741 — this branch's publisher-block
shift). Delta vs origin/main is that single line; bare gate green: 109
sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-aiClaude

Copy link
Copy Markdown
Collaborator

Contract review — CONTRACT_REVIEW_TIER — verdict: PASS

Delegated review (dispatching PM seat session_01Q5WBDtaUnoz5XuJ6jk8pQ5 is below tier). Reviewed against the ruling on #13331 (comment 5486837547, director batch A, 「同意。」 on Option A). This is a plain comment, not an approval; the reviewer seat does not flip ready, enqueue, or merge.

Instruments. The diff was measured through the API (get_files / get_diff), never through the shallow checkout's three-dot diff: 9 files, +1199/−45, which sums exactly to the PR's own changed_files/additions/deletions — the file list is complete by that control. File contents were read at the pinned SHAs 9dd022ae (base) and 0cf0a437 (head) via git show/git grep (both objects present locally, verified with cat-file -e). Every zero-reading below carried a positive control on the same instrument that fired non-zero. Locations below are by symbol.


Clause-② limbs, judged from the diff

Content limb: YES — declared honestly and completely. The minted public surface, measured:

  • METADATA_MUTATION_CLUSTER_CHANNEL = 'metadata.mutated' + ClusterMetadataMutationPayload, exported from the @objectstack/metadata-protocol barrel (the only two new exports in packages/metadata-protocol/src/index.ts).
  • attachMetadataMutationPubSub(pubsub, nodeId) / detachMetadataMutationPubSub() as public methods on the already-exported ObjectStackProtocolImplementation.
  • Base-tree zero, with control: git grep at 9dd022ae for metadata.mutated|attachMetadataMutationPubSub|METADATA_MUTATION_CLUSTER_CHANNEL → 0 hits (control on same ref, metadata.changed in metadata-manager.ts → 3). Nothing pre-existing is widened or shadowed; the PR mints exactly what it declares.
  • No undeclared widening found.service-cluster's barrel index.ts is untouched (so the anticipated fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 conflict genuinely cannot arise); the bridge's lane 2 is duck-typed plugin behavior, no new export; emitMetadataMutation's split into notifyMutationListenersLocal is private; the changeset, the pinned-doubles ledger entry, and the census row are process artifacts, not surface.

Path limb: NO — measured, not taken from the author. From the complete API file list: zero paths under packages/spec/**, zero writes to packages/objectql/src/engine.ts. That is both fences, held:

  • engine.ts fence: held (0 files touch it; the authz precedent at attachAuthzInvalidationPubSub was mirrored in shape only — see placement below).
  • packages/spec fence: held, and no contract-first split is owed — see placement.

Is the contract minimal and right-shaped?

Two lanes are correct, not a duplicate. Measured, all at head:

  • Lane 1's only production publisher is MetadataManager.notifyWatchers (metadata-manager.ts, the single non-doc publish(MetadataManager.CLUSTER_CHANNEL…) in the tree; the two other grep hits are JSDoc examples in service-cluster).
  • protocol.ts contains zero code references to the metadata service or notifyWatchers — the only 3 hits are doc comments this PR itself adds. The authoring path (saveMetaItemapplyRegistryWriteThrough) cannot reach lane 1 on any boot shape.
  • The lanes carry different disciplines for different state owners: metadata.changed replays the legacy watch event verbatim (content on the wire) into the metadata SERVICE's caches; metadata.mutated is address-only into the ObjectQL engine registry. Overloading lane 1 with a signal-only receipt would collide with its content-replay contract; and the host-config boot has no manager at all (CORE_FALLBACK_FACTORIES = metadata/cache/queue/i18n — the fallback has no cluster seam, and there is no protocol fallback either, so lane 2 duck-types the real implementation or skips). The author's argument tests out.
  • Wiring is real, not just harness-deep: runtime.ts mounts MetadataClusterBridgePlugin unconditionally, and 'protocol' is registered by exactly one production site (assembleMetadataProtocolctx.registerService('protocol', protocolShim) where the "shim" is the ObjectStackProtocolImplementation instance), reached via ObjectQLPlugin's default registerProtocol = true. The variable-key fallback blind spot that inverted the original card does not apply here.

The payload is genuinely address-only, and the receiver genuinely re-reads.MetadataMutationEvent at head is exactly {type, name, state, organizationId?} — no body field exists on the type; all three emit sites construct only those four fields (the item body goes to the awaited projector, which is a separate mechanism, not the event). On receipt, applyRemoteMetadataMutation consumes the wire only as an ADDRESS: type folded through canonicalMetaType, repo.get(ref, {state:'active'}) against the replica's own store, item taken from current.body (own DB), packageId from resolveOverlayPackageBinding (own DB, the #4636 recovery-caller derivation, verbatim). Wire state is consumed only as a draft-suppression guard. Nothing from the wire is applied. The key-set test pins this at runtime.

Attach/detach shape. Idempotent on the (pubsub, nodeId) pair with a different-pair re-attach detaching first — measured side-by-side identical to MetadataManager.attachClusterPubSub. Loopback via originNode equality. Detach idempotent. The bridge detaches both lanes independently on kernel:shutdown with error isolation (a throwing lane-1 detach doesn't strand lane 2 — pinned). Lane 2 carries isInProcessClusterDriver from birth, matching AuthzClusterBridgePlugin, and lane 1 stays byte-identical (its warn line verbatim — pinned), leaving #14021 unabsorbed as declared.

Scope parity of the receipt. The write-through's env gate and org-scope verdict (hydrateOverlayIntoRegistry, #6602) and the heal walk's org refusal (restoreArtifactRegistryView returns for organizationId !== null, #6780) are inherited, not re-decided — a peer applies exactly what the writer's own kernel would have applied locally. A forged org-scoped object signal fails safe: the peer's own read finds no such row, the heal walk's org gate refuses, no-op.

Failure directions

  • Duplicate delivery: same read, same idempotent registration — pinned.
  • Out-of-order: the DB read decides, not the event name — the draft-discard test pins the sharpest case (a "delete"-shaped signal whose read finds an active row re-registers instead of healing).
  • Lost signal: degrades to the pre-existing bound (stale until boot reload). This matches IPubSub's own contract doc at head verbatim ("no shipped driver exceeds at-most-once… Handlers MUST be idempotent and tolerate loss"); the PR promises exactly one hop of narrowing and nothing more, in the changeset and the attach docblock.
  • No row on the peer: heal walk, org-gated as above.
  • Draft: never published (publisher gate) plus a receiver guard.
  • Failed apply: caught, warned, dropped → fail-closed 404 staleness, never wrong data.

Beyond-minimum: the local listener replay — sound, and in scope

The ruling's clause 1 orders the shipped bridge shape mirrored, and the shipped template itself ships remote replay: MetadataManager.attachClusterPubSub's receipt handler invalidates first, then replays into notifyWatchersLocal (measured at head). The PR's replay is the same shape with the same #5109 invalidate-before-notify ordering (registry first, listeners second — pinned by a test that asserts registry state from inside the listener). Never re-published — the local/publish split plus the docblock's storm reasoning is correct, since the loopback guard only suppresses a node's own messages. Replay targets measured: ObjectQL's hook/action rebind and the i18n authored-translation sync are per-replica in-memory re-syncs from the replica's own reads — precisely the issue's second-order defect (runtime-authored automation being single-node). The consumers with shared-DB side effects (email template, permission-set projection) run through the awaited projector seam on the shipped protocol, which is invoked at the write sites only and is NOT replayed — so no fleet-wide duplication of projection writes. Verdict: not a design fork; it is inside the ruled shape, and it is what makes the fix complete for peers.

Placement: metadata-protocol is the right home

  • The state that goes stale (the engine-registry write-through) and the sys_metadata read the ruled receipt needs both live in the protocol; the engine template was shape-only.
  • Both shipped precedents put channel constants and payloads with their state owners, not in spec — measured: AUTHZ_INVALIDATED_CHANNEL in packages/core/src/security/authz-invalidation-channel.ts; ClusterMetadataChangedPayload in packages/metadata/src/metadata-manager.ts. IPubSub from spec/contracts is the already-shipped generic transport (type-only import; no new dependency edge — metadata-protocol already imports spec).
  • The spec's MetadataChangedEventPayloadSchema (cluster.zod.ts): measured dormant — the only reference outside packages/spec in the whole tree is a prose comment in metadata-manager.ts (control: ClusterCapabilityConfig, same instrument, 4+ consuming packages). Its declared semantics ("compare version with their cached value… out-of-order older versions are ignored") make the wire the thing trusted, the opposite of the ruled re-read receipt; and its version: z.bigint() cannot cross JSON.stringify at all. Filing [finding] spec: MetadataChangedEventPayloadSchema says every metadata persistence layer MUST emit it — zero producers or consumers in-tree, and its bigint version field cannot cross JSON #14180 for the spec lane instead of building on it was correct. No spec-lane contract is wearing local clothes here.

Findings

  1. Non-blocking — concurrent same-row mutations can interleave at the peer. The applier is fire-and-forget per message with no per-key serialization: writer saves v1 then v2 in quick succession; the peer's read for the v1 signal (dispatched first, completed before v2's commit) can — under DB-response jitter — deliver its continuation after the v2 applier has registered, leaving the peer's registry at v1 while sys_metadata holds v2, until the next signal or boot. partitionKey orders delivery, not apply completion. Why non-blocking: the registered body is a genuinely-persisted prior state (never fabricated, never wrong-tenant), the window is one DB-read RTT, any later mutation re-converges, the same interleave class already exists between two racing local writers, and the outcome sits inside the staleness bound the PR explicitly declares (equivalent to a lost signal). If it ever matters in practice, a per-type:name apply queue is a receiver-internal fix that touches no public contract.
  2. Non-blocking — one more silent seam of the metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 family.deleteMetaItem's legacy control-plane branch (code-only rows; no repo path) deletes the row and heals its own registry but never emitted the mutation event even locally — pre-existing and marked deliberate in code ("emits no watch event… pre-existing and deliberate") — so it fans out to no one. Same class as the recovery doors metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 records, but metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 names only rollbackMetaItem/revertCommit. Recommend appending this branch to metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 rather than widening this PR.
  3. Non-blocking — wording. The payload docblock's "at-least-once delivery and duplicates are harmless by construction" states a tolerance, while shipped drivers are at-most-once; a skimming reader could take it for a delivery guarantee. The attach docblock, changeset, and PR body all state the loss bound plainly, so no change is required — noted only so nobody later quotes the payload docblock as a promise.

Also verified

  • The census docs row in this diff is exactly the claimed anchor shift: context?.isSystem in stripReadonlyForInsert measured at protocol.ts:1737 at base and :1741 at head; the docs hunk changes that one number and zero prose. Not scope creep.
  • The write-through caller pin re-opened correctly: measured exactly five this.applyRegistryWriteThrough( sites at head (base: four), the fifth being applyRemoteMetadataMutation; the route-5 case drives the real subscribe path and pins the wire-plural fold.
  • emitMetadataMutation's callers at head are exactly saveMetaItem / runPublishSideEffects / deleteMetaItem — the claimed choke point — and the recovery doors run the write-through without emitting, confirming metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 as a pre-existing gap, not this PR's.
  • CI at head 0cf0a437: 33/33 check runs green (Lint & Repo Gates included). I did not re-run the suites locally; suite verdicts are read from CI and the dev reports, all other claims above are my own measurements.

Verdict: PASS. No blocking findings; no design fork the ruling leaves unsettled. The enqueue-gate decision returns to the dispatching seat.


Generated by Claude Code

Same mechanism as the previous merge: main's #14199 re-anchored 11 rows
of this machine-maintained table while this branch re-anchors row 21,
and a textually clean merge proves nothing about the line numbers.
Resolution per the tool: took main's page wholesale, re-derived with
check-system-context-census --fix on the merged tree — exactly ONE
anchor rewritten (row 21, protocol.ts 1737 to 1741; re-derived, not
carried — main has zero commits on protocol.ts, so the insertion shift
is unchanged). Delta vs origin/main is that single line; bare gate
green (109 sites, 145 anchors). engine-double-contract ledger: zero
main-side commits, gate OK on the merged tree — no regeneration owed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-ai
os-support-ai added this pull request to the merge queueSep 1, 2026
Merged via the queue into main with commit 1403d94Sep 1, 2026
34 checks passed
@os-support-ai
os-support-ai deleted the claude/issue-13331-metadata-registry-cluster-fanout branch September 1, 2026 14:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/xlteststooling

Projects

None yet

2 participants

@os-support-ai@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas - #14183

Merged
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout
Sep 1, 2026
Merged

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas#14183
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout

Conversation

@claude

@claudeclaudeBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes#13331

What this ships

A runtime-authored metadata mutation now reaches every replica's ObjectQL registry, not just the writer's. Measured defect on the shipped 3-replica EE compose (ADR-0018): PUT /api/v1/meta/object/... persisted to the shared sys_metadata (meta reads 200 fleet-wide) while only the writing replica registered the object — /api/v1/data/... answered a hard 404 OBJECT_NOT_FOUND on the other two, indefinitely; 200 concurrent creates through the LB gave 67x201 / 133x404 with a boot-loaded control object at 0 errors.

Mechanism (maintainer-ruled 2026-09-01, director batch A, Option A — verbatim adoption of the escalation's recommendation):

  • Publisher at the producer choke point. The protocol's post-persistence funnel (emitMetadataMutation, the seam onMetadataMutation subscribes — reached by saveMetaItem, runPublishSideEffects, deleteMetaItem) now also publishes the mutation's ADDRESS on a new cluster channel metadata.mutated (METADATA_MUTATION_CLUSTER_CHANNEL, payload ClusterMetadataMutationPayload, both exported from @objectstack/metadata-protocol). Drafts are never published — they enter no registry anywhere.
  • Peers converge from their own DB read. On receipt a replica re-reads the row from its OWN sys_metadata: active row present ⇒ re-run applyRegistryWriteThrough (package binding derived the way the recovery callers derive it); no active row ⇒ run the delete heal walk (restoreArtifactRegistryView). The payload is a signal, never trusted content; duplicates and out-of-order delivery converge to the row's current state by construction. After convergence the event replays into the replica's LOCAL onMetadataMutation listeners (never re-published), so authored hook/action re-binds re-sync on peers.
  • Attach seam + bridge lane.ObjectStackProtocolImplementation.attachMetadataMutationPubSub(pubsub, nodeId) — idempotent on the pair, loopback-suppressed via originNode, mirroring MetadataManager.attachClusterPubSub and the engine's attachAuthzInvalidationPubSub. MetadataClusterBridgePlugin late-binds it at kernel:ready as a second, independent lane: the boot shape that lacks a manager-backed metadata service (TS-config host-config — exactly the shipped EE shape) is the one that needs this lane most. The new lane skips the in-process memory driver from birth (the guard the authz sibling carries).
  • Option B (consumer-side self-heal at assertObjectRegistered) is not built — presented only as a possible stopgap; the maintainer did not order one.

The two fences, both held

Evidence

  • Two-arm, directions declared in the test-file header before running (protocol.cluster-mutation-fanout.test.ts, two replicas over one shared store): Arm B (attached) — the peer converges from its own read; Arm A (control, no bridge) — the identical write leaves the peer empty, so Arm B is the bridge's doing, not a harness artifact. Also pinned: address-only payload (key set equality), draft silence, loopback suppression, duplicate-delivery convergence, re-attach idempotency, detach, delete fan-out, draft-discard keeping the peer's active registration, and listener replay ordered after registry convergence.
  • Committed-tree ablation at 44a2e59 (script with trap restore, absolute paths): the single publisher call deleted — mutation proved on disk (anchor count 0, marker count 1) — turned exactly the 7 declared bus-crossing cases red with 15 staying green (controls held); restore proven by HEAD-blob equality (d7ba0225...) plus empty git diff HEAD plus marker count 0; restored run 22/22 green. Resolution note: these tests import ./protocol.js relative from src — the subject never resolves through dist, so no rebuild leg exists to skip; the on-disk grep is the falsifiable observation for both legs.
  • Suites at final head e460193: metadata-protocol 152 files passed / 2 skipped (2094 tests passed / 10 skipped), service-cluster 6 files / 79 tests passed, tsc --noEmit clean with both edited test files confirmed inside the program via --listFiles (2 hits).
  • Gate union at e460193 (node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands; provenance line names this repo at this commit and the --repo assertion holds): 44 derived; 41 exit 0; 3 NOT MEASURED by their own printed verdicts (check-test-completeness exit 3 — grades a CI turbo log; check:dual-build-cjs-loads exit 3 — needs the full workspace build; check:type-check-debt exit 3 — full re-measure). Two first-pass reds were fixed and re-run to their own OK lines: check:engine-double-contract ("OK — 741 pinned, 134 in the DEBT ledger, 3 exempt"; ledger learned the fanout file's pinned doubles via the gate's own --write) and check:objectql-double-limit ("conformance holds: 299 doubles graded"; the new find double now holds the caller's bound by presence).
  • The write-through caller pin re-opened deliberately: applyRegistryWriteThrough grew its FIFTH caller (the peer applier). The trace, the count pin, and a new route-5 spelling case (a plural delivered over the wire registers under the singular; fold via canonicalMetaType, the complete map) are updated in protocol.object-registry-write-through-spelling.test.ts.

Notes for contract review

CI rework — census re-anchor (cdbe5f0)

Lint & Repo Gates red at e460193: check-system-context-census — pure line rot from this PR's own insertion (the publisher block sits above stripReadonlyForInsert, moving its context.isSystem read from protocol.ts:1737 to :1741 while content/docs/permissions/system-context.mdx row 21 still anchored 1737). Repaired with the gate's own --fix and VERIFIED as a SHIFT, not a population change: the rewrite touches exactly one line — the anchor's line number — and zero prose, and the census population is unchanged (109 sites / 145 anchors before and after). The docs file joining this diff is that re-anchor, not scope creep. Reproduced red locally first, then the bare gate re-run green: "OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read."

Docs-drift assessment (stated, not rewritten, per dispatch): content/docs/kernel/cluster.mdx §6.2 is now INCOMPLETE — it says cross-node metadata invalidation "already works" through the manager's metadata.changed lane alone, which is exactly the account #13331 falsified for the runtime-authoring path (the protocol never reaches the manager, and host-config boots carry no manager at all). The page wants a paragraph on the metadata.mutated protocol lane and the bridge's second attach; its own closing principle there ("the peer re-reads the shared store, which is the source of truth") is the rule the new receipt implements, and its "Target spec (planned)" block already marks MetadataChangedEventPayloadSchema as not wired — corroborating #14180. Same incompleteness class applies to the §5/§7 bridge mentions and content/docs/concepts/metadata-lifecycle.mdx.

Generated by Claude Code


Generated by Claude Code


Generated by Claude Code

…ns out to peer replicas (#13331)
Publisher at the protocol's post-persistence choke point publishes the
mutation's address on the new metadata.mutated cluster channel; peers
converge their ObjectQL registry from their OWN sys_metadata read (write-
through when an active row exists, the delete heal walk when none does),
then replay the event into local onMetadataMutation listeners. The bridge
plugin late-binds attachMetadataMutationPubSub as a second independent
lane at kernel:ready, guarded off the in-process memory driver.
Ruled 2026-09-01 (director batch A, Option A): producer-side fan-out
mirroring the shipped AuthzClusterBridgePlugin shape; the payload is a
signal, never trusted content. Option B (consumer-side self-heal) is not
built.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
…ouble; ledger the new pinned engine doubles
check:objectql-double-limit named the new find double limit-blind — the
bound now applies after the filter, by presence. The engine-double ledger
learned the fanout file's pinned delete/findOne/update doubles via the
gate's own --write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@github-actions

github-actionsBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/metadata-protocol, @objectstack/service-cluster, touching 19 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

5 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/wire-format.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/concepts/metadata-lifecycle.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), ObjectStackProtocolImplementation (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/deployment/validating-metadata.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/kernel/cluster.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/kernel/services-checklist.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class))

3 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v15.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/releases/v16.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts) — pages documenting those are invisible to this run
  • 1 anchor(s) matched too much of the corpus to be a work list: /api/v1/data (route, 35 pages)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 11 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fcpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 1304ec033ce539d92069afd7ac66110cc1951b11 — the merge of head 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec into base e62129153f2aa3d7666eab9c1af2bdb18e7333fc, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 1304ec033ce539d92069afd7ac66110cc1951b11 && git checkout 1304ec033ce539d92069afd7ac66110cc1951b11
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin e62129153f2aa3d7666eab9c1af2bdb18e7333fc 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec && git checkout -B drift-repro e62129153f2aa3d7666eab9c1af2bdb18e7333fc && git merge --no-ff 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec
node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fc

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs e62129153f2aa3d7666eab9c1af2bdb18e7333fc → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 1, 2026
…he fan-out insertion
check-system-context-census red on CI: the metadata.mutated publisher
block inserted above stripReadonlyForInsert moved the context.isSystem
read 1737 -> 1741, and the census page still anchored the old line. The
gate's own --fix performed the shift (one line-number rewrite, zero
prose); bare re-run green: 109 sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
The auto-merge of origin/main kept this branch's pre-#14171 anchor rows
beside the row-21 edit, flunking the census 26 ways (paired stale
anchors / unanchored sites in engine.ts and share-link-service.ts —
pure adjacent-row line rot, zero population change). Resolution per the
tool, not by hand: took main's page wholesale, then re-derived with
check-system-context-census --fix, which rewrote exactly ONE anchor
(row 21, protocol.ts 1737 to 1741 — this branch's publisher-block
shift). Delta vs origin/main is that single line; bare gate green: 109
sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-aiClaude

Copy link
Copy Markdown
Collaborator

Contract review — CONTRACT_REVIEW_TIER — verdict: PASS

Delegated review (dispatching PM seat session_01Q5WBDtaUnoz5XuJ6jk8pQ5 is below tier). Reviewed against the ruling on #13331 (comment 5486837547, director batch A, 「同意。」 on Option A). This is a plain comment, not an approval; the reviewer seat does not flip ready, enqueue, or merge.

Instruments. The diff was measured through the API (get_files / get_diff), never through the shallow checkout's three-dot diff: 9 files, +1199/−45, which sums exactly to the PR's own changed_files/additions/deletions — the file list is complete by that control. File contents were read at the pinned SHAs 9dd022ae (base) and 0cf0a437 (head) via git show/git grep (both objects present locally, verified with cat-file -e). Every zero-reading below carried a positive control on the same instrument that fired non-zero. Locations below are by symbol.


Clause-② limbs, judged from the diff

Content limb: YES — declared honestly and completely. The minted public surface, measured:

  • METADATA_MUTATION_CLUSTER_CHANNEL = 'metadata.mutated' + ClusterMetadataMutationPayload, exported from the @objectstack/metadata-protocol barrel (the only two new exports in packages/metadata-protocol/src/index.ts).
  • attachMetadataMutationPubSub(pubsub, nodeId) / detachMetadataMutationPubSub() as public methods on the already-exported ObjectStackProtocolImplementation.
  • Base-tree zero, with control: git grep at 9dd022ae for metadata.mutated|attachMetadataMutationPubSub|METADATA_MUTATION_CLUSTER_CHANNEL → 0 hits (control on same ref, metadata.changed in metadata-manager.ts → 3). Nothing pre-existing is widened or shadowed; the PR mints exactly what it declares.
  • No undeclared widening found.service-cluster's barrel index.ts is untouched (so the anticipated fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 conflict genuinely cannot arise); the bridge's lane 2 is duck-typed plugin behavior, no new export; emitMetadataMutation's split into notifyMutationListenersLocal is private; the changeset, the pinned-doubles ledger entry, and the census row are process artifacts, not surface.

Path limb: NO — measured, not taken from the author. From the complete API file list: zero paths under packages/spec/**, zero writes to packages/objectql/src/engine.ts. That is both fences, held:

  • engine.ts fence: held (0 files touch it; the authz precedent at attachAuthzInvalidationPubSub was mirrored in shape only — see placement below).
  • packages/spec fence: held, and no contract-first split is owed — see placement.

Is the contract minimal and right-shaped?

Two lanes are correct, not a duplicate. Measured, all at head:

  • Lane 1's only production publisher is MetadataManager.notifyWatchers (metadata-manager.ts, the single non-doc publish(MetadataManager.CLUSTER_CHANNEL…) in the tree; the two other grep hits are JSDoc examples in service-cluster).
  • protocol.ts contains zero code references to the metadata service or notifyWatchers — the only 3 hits are doc comments this PR itself adds. The authoring path (saveMetaItemapplyRegistryWriteThrough) cannot reach lane 1 on any boot shape.
  • The lanes carry different disciplines for different state owners: metadata.changed replays the legacy watch event verbatim (content on the wire) into the metadata SERVICE's caches; metadata.mutated is address-only into the ObjectQL engine registry. Overloading lane 1 with a signal-only receipt would collide with its content-replay contract; and the host-config boot has no manager at all (CORE_FALLBACK_FACTORIES = metadata/cache/queue/i18n — the fallback has no cluster seam, and there is no protocol fallback either, so lane 2 duck-types the real implementation or skips). The author's argument tests out.
  • Wiring is real, not just harness-deep: runtime.ts mounts MetadataClusterBridgePlugin unconditionally, and 'protocol' is registered by exactly one production site (assembleMetadataProtocolctx.registerService('protocol', protocolShim) where the "shim" is the ObjectStackProtocolImplementation instance), reached via ObjectQLPlugin's default registerProtocol = true. The variable-key fallback blind spot that inverted the original card does not apply here.

The payload is genuinely address-only, and the receiver genuinely re-reads.MetadataMutationEvent at head is exactly {type, name, state, organizationId?} — no body field exists on the type; all three emit sites construct only those four fields (the item body goes to the awaited projector, which is a separate mechanism, not the event). On receipt, applyRemoteMetadataMutation consumes the wire only as an ADDRESS: type folded through canonicalMetaType, repo.get(ref, {state:'active'}) against the replica's own store, item taken from current.body (own DB), packageId from resolveOverlayPackageBinding (own DB, the #4636 recovery-caller derivation, verbatim). Wire state is consumed only as a draft-suppression guard. Nothing from the wire is applied. The key-set test pins this at runtime.

Attach/detach shape. Idempotent on the (pubsub, nodeId) pair with a different-pair re-attach detaching first — measured side-by-side identical to MetadataManager.attachClusterPubSub. Loopback via originNode equality. Detach idempotent. The bridge detaches both lanes independently on kernel:shutdown with error isolation (a throwing lane-1 detach doesn't strand lane 2 — pinned). Lane 2 carries isInProcessClusterDriver from birth, matching AuthzClusterBridgePlugin, and lane 1 stays byte-identical (its warn line verbatim — pinned), leaving #14021 unabsorbed as declared.

Scope parity of the receipt. The write-through's env gate and org-scope verdict (hydrateOverlayIntoRegistry, #6602) and the heal walk's org refusal (restoreArtifactRegistryView returns for organizationId !== null, #6780) are inherited, not re-decided — a peer applies exactly what the writer's own kernel would have applied locally. A forged org-scoped object signal fails safe: the peer's own read finds no such row, the heal walk's org gate refuses, no-op.

Failure directions

  • Duplicate delivery: same read, same idempotent registration — pinned.
  • Out-of-order: the DB read decides, not the event name — the draft-discard test pins the sharpest case (a "delete"-shaped signal whose read finds an active row re-registers instead of healing).
  • Lost signal: degrades to the pre-existing bound (stale until boot reload). This matches IPubSub's own contract doc at head verbatim ("no shipped driver exceeds at-most-once… Handlers MUST be idempotent and tolerate loss"); the PR promises exactly one hop of narrowing and nothing more, in the changeset and the attach docblock.
  • No row on the peer: heal walk, org-gated as above.
  • Draft: never published (publisher gate) plus a receiver guard.
  • Failed apply: caught, warned, dropped → fail-closed 404 staleness, never wrong data.

Beyond-minimum: the local listener replay — sound, and in scope

The ruling's clause 1 orders the shipped bridge shape mirrored, and the shipped template itself ships remote replay: MetadataManager.attachClusterPubSub's receipt handler invalidates first, then replays into notifyWatchersLocal (measured at head). The PR's replay is the same shape with the same #5109 invalidate-before-notify ordering (registry first, listeners second — pinned by a test that asserts registry state from inside the listener). Never re-published — the local/publish split plus the docblock's storm reasoning is correct, since the loopback guard only suppresses a node's own messages. Replay targets measured: ObjectQL's hook/action rebind and the i18n authored-translation sync are per-replica in-memory re-syncs from the replica's own reads — precisely the issue's second-order defect (runtime-authored automation being single-node). The consumers with shared-DB side effects (email template, permission-set projection) run through the awaited projector seam on the shipped protocol, which is invoked at the write sites only and is NOT replayed — so no fleet-wide duplication of projection writes. Verdict: not a design fork; it is inside the ruled shape, and it is what makes the fix complete for peers.

Placement: metadata-protocol is the right home

  • The state that goes stale (the engine-registry write-through) and the sys_metadata read the ruled receipt needs both live in the protocol; the engine template was shape-only.
  • Both shipped precedents put channel constants and payloads with their state owners, not in spec — measured: AUTHZ_INVALIDATED_CHANNEL in packages/core/src/security/authz-invalidation-channel.ts; ClusterMetadataChangedPayload in packages/metadata/src/metadata-manager.ts. IPubSub from spec/contracts is the already-shipped generic transport (type-only import; no new dependency edge — metadata-protocol already imports spec).
  • The spec's MetadataChangedEventPayloadSchema (cluster.zod.ts): measured dormant — the only reference outside packages/spec in the whole tree is a prose comment in metadata-manager.ts (control: ClusterCapabilityConfig, same instrument, 4+ consuming packages). Its declared semantics ("compare version with their cached value… out-of-order older versions are ignored") make the wire the thing trusted, the opposite of the ruled re-read receipt; and its version: z.bigint() cannot cross JSON.stringify at all. Filing [finding] spec: MetadataChangedEventPayloadSchema says every metadata persistence layer MUST emit it — zero producers or consumers in-tree, and its bigint version field cannot cross JSON #14180 for the spec lane instead of building on it was correct. No spec-lane contract is wearing local clothes here.

Findings

  1. Non-blocking — concurrent same-row mutations can interleave at the peer. The applier is fire-and-forget per message with no per-key serialization: writer saves v1 then v2 in quick succession; the peer's read for the v1 signal (dispatched first, completed before v2's commit) can — under DB-response jitter — deliver its continuation after the v2 applier has registered, leaving the peer's registry at v1 while sys_metadata holds v2, until the next signal or boot. partitionKey orders delivery, not apply completion. Why non-blocking: the registered body is a genuinely-persisted prior state (never fabricated, never wrong-tenant), the window is one DB-read RTT, any later mutation re-converges, the same interleave class already exists between two racing local writers, and the outcome sits inside the staleness bound the PR explicitly declares (equivalent to a lost signal). If it ever matters in practice, a per-type:name apply queue is a receiver-internal fix that touches no public contract.
  2. Non-blocking — one more silent seam of the metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 family.deleteMetaItem's legacy control-plane branch (code-only rows; no repo path) deletes the row and heals its own registry but never emitted the mutation event even locally — pre-existing and marked deliberate in code ("emits no watch event… pre-existing and deliberate") — so it fans out to no one. Same class as the recovery doors metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 records, but metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 names only rollbackMetaItem/revertCommit. Recommend appending this branch to metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 rather than widening this PR.
  3. Non-blocking — wording. The payload docblock's "at-least-once delivery and duplicates are harmless by construction" states a tolerance, while shipped drivers are at-most-once; a skimming reader could take it for a delivery guarantee. The attach docblock, changeset, and PR body all state the loss bound plainly, so no change is required — noted only so nobody later quotes the payload docblock as a promise.

Also verified

  • The census docs row in this diff is exactly the claimed anchor shift: context?.isSystem in stripReadonlyForInsert measured at protocol.ts:1737 at base and :1741 at head; the docs hunk changes that one number and zero prose. Not scope creep.
  • The write-through caller pin re-opened correctly: measured exactly five this.applyRegistryWriteThrough( sites at head (base: four), the fifth being applyRemoteMetadataMutation; the route-5 case drives the real subscribe path and pins the wire-plural fold.
  • emitMetadataMutation's callers at head are exactly saveMetaItem / runPublishSideEffects / deleteMetaItem — the claimed choke point — and the recovery doors run the write-through without emitting, confirming metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 as a pre-existing gap, not this PR's.
  • CI at head 0cf0a437: 33/33 check runs green (Lint & Repo Gates included). I did not re-run the suites locally; suite verdicts are read from CI and the dev reports, all other claims above are my own measurements.

Verdict: PASS. No blocking findings; no design fork the ruling leaves unsettled. The enqueue-gate decision returns to the dispatching seat.


Generated by Claude Code

Same mechanism as the previous merge: main's #14199 re-anchored 11 rows
of this machine-maintained table while this branch re-anchors row 21,
and a textually clean merge proves nothing about the line numbers.
Resolution per the tool: took main's page wholesale, re-derived with
check-system-context-census --fix on the merged tree — exactly ONE
anchor rewritten (row 21, protocol.ts 1737 to 1741; re-derived, not
carried — main has zero commits on protocol.ts, so the insertion shift
is unchanged). Delta vs origin/main is that single line; bare gate
green (109 sites, 145 anchors). engine-double-contract ledger: zero
main-side commits, gate OK on the merged tree — no regeneration owed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-ai
os-support-ai added this pull request to the merge queueSep 1, 2026
Merged via the queue into main with commit 1403d94Sep 1, 2026
34 checks passed
@os-support-ai
os-support-ai deleted the claude/issue-13331-metadata-registry-cluster-fanout branch September 1, 2026 14:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/xlteststooling

Projects

None yet

2 participants

@os-support-ai@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas - #14183

Merged
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout
Sep 1, 2026
Merged

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas#14183
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout

Conversation

@claude

@claudeclaudeBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes#13331

What this ships

A runtime-authored metadata mutation now reaches every replica's ObjectQL registry, not just the writer's. Measured defect on the shipped 3-replica EE compose (ADR-0018): PUT /api/v1/meta/object/... persisted to the shared sys_metadata (meta reads 200 fleet-wide) while only the writing replica registered the object — /api/v1/data/... answered a hard 404 OBJECT_NOT_FOUND on the other two, indefinitely; 200 concurrent creates through the LB gave 67x201 / 133x404 with a boot-loaded control object at 0 errors.

Mechanism (maintainer-ruled 2026-09-01, director batch A, Option A — verbatim adoption of the escalation's recommendation):

  • Publisher at the producer choke point. The protocol's post-persistence funnel (emitMetadataMutation, the seam onMetadataMutation subscribes — reached by saveMetaItem, runPublishSideEffects, deleteMetaItem) now also publishes the mutation's ADDRESS on a new cluster channel metadata.mutated (METADATA_MUTATION_CLUSTER_CHANNEL, payload ClusterMetadataMutationPayload, both exported from @objectstack/metadata-protocol). Drafts are never published — they enter no registry anywhere.
  • Peers converge from their own DB read. On receipt a replica re-reads the row from its OWN sys_metadata: active row present ⇒ re-run applyRegistryWriteThrough (package binding derived the way the recovery callers derive it); no active row ⇒ run the delete heal walk (restoreArtifactRegistryView). The payload is a signal, never trusted content; duplicates and out-of-order delivery converge to the row's current state by construction. After convergence the event replays into the replica's LOCAL onMetadataMutation listeners (never re-published), so authored hook/action re-binds re-sync on peers.
  • Attach seam + bridge lane.ObjectStackProtocolImplementation.attachMetadataMutationPubSub(pubsub, nodeId) — idempotent on the pair, loopback-suppressed via originNode, mirroring MetadataManager.attachClusterPubSub and the engine's attachAuthzInvalidationPubSub. MetadataClusterBridgePlugin late-binds it at kernel:ready as a second, independent lane: the boot shape that lacks a manager-backed metadata service (TS-config host-config — exactly the shipped EE shape) is the one that needs this lane most. The new lane skips the in-process memory driver from birth (the guard the authz sibling carries).
  • Option B (consumer-side self-heal at assertObjectRegistered) is not built — presented only as a possible stopgap; the maintainer did not order one.

The two fences, both held

Evidence

  • Two-arm, directions declared in the test-file header before running (protocol.cluster-mutation-fanout.test.ts, two replicas over one shared store): Arm B (attached) — the peer converges from its own read; Arm A (control, no bridge) — the identical write leaves the peer empty, so Arm B is the bridge's doing, not a harness artifact. Also pinned: address-only payload (key set equality), draft silence, loopback suppression, duplicate-delivery convergence, re-attach idempotency, detach, delete fan-out, draft-discard keeping the peer's active registration, and listener replay ordered after registry convergence.
  • Committed-tree ablation at 44a2e59 (script with trap restore, absolute paths): the single publisher call deleted — mutation proved on disk (anchor count 0, marker count 1) — turned exactly the 7 declared bus-crossing cases red with 15 staying green (controls held); restore proven by HEAD-blob equality (d7ba0225...) plus empty git diff HEAD plus marker count 0; restored run 22/22 green. Resolution note: these tests import ./protocol.js relative from src — the subject never resolves through dist, so no rebuild leg exists to skip; the on-disk grep is the falsifiable observation for both legs.
  • Suites at final head e460193: metadata-protocol 152 files passed / 2 skipped (2094 tests passed / 10 skipped), service-cluster 6 files / 79 tests passed, tsc --noEmit clean with both edited test files confirmed inside the program via --listFiles (2 hits).
  • Gate union at e460193 (node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands; provenance line names this repo at this commit and the --repo assertion holds): 44 derived; 41 exit 0; 3 NOT MEASURED by their own printed verdicts (check-test-completeness exit 3 — grades a CI turbo log; check:dual-build-cjs-loads exit 3 — needs the full workspace build; check:type-check-debt exit 3 — full re-measure). Two first-pass reds were fixed and re-run to their own OK lines: check:engine-double-contract ("OK — 741 pinned, 134 in the DEBT ledger, 3 exempt"; ledger learned the fanout file's pinned doubles via the gate's own --write) and check:objectql-double-limit ("conformance holds: 299 doubles graded"; the new find double now holds the caller's bound by presence).
  • The write-through caller pin re-opened deliberately: applyRegistryWriteThrough grew its FIFTH caller (the peer applier). The trace, the count pin, and a new route-5 spelling case (a plural delivered over the wire registers under the singular; fold via canonicalMetaType, the complete map) are updated in protocol.object-registry-write-through-spelling.test.ts.

Notes for contract review

CI rework — census re-anchor (cdbe5f0)

Lint & Repo Gates red at e460193: check-system-context-census — pure line rot from this PR's own insertion (the publisher block sits above stripReadonlyForInsert, moving its context.isSystem read from protocol.ts:1737 to :1741 while content/docs/permissions/system-context.mdx row 21 still anchored 1737). Repaired with the gate's own --fix and VERIFIED as a SHIFT, not a population change: the rewrite touches exactly one line — the anchor's line number — and zero prose, and the census population is unchanged (109 sites / 145 anchors before and after). The docs file joining this diff is that re-anchor, not scope creep. Reproduced red locally first, then the bare gate re-run green: "OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read."

Docs-drift assessment (stated, not rewritten, per dispatch): content/docs/kernel/cluster.mdx §6.2 is now INCOMPLETE — it says cross-node metadata invalidation "already works" through the manager's metadata.changed lane alone, which is exactly the account #13331 falsified for the runtime-authoring path (the protocol never reaches the manager, and host-config boots carry no manager at all). The page wants a paragraph on the metadata.mutated protocol lane and the bridge's second attach; its own closing principle there ("the peer re-reads the shared store, which is the source of truth") is the rule the new receipt implements, and its "Target spec (planned)" block already marks MetadataChangedEventPayloadSchema as not wired — corroborating #14180. Same incompleteness class applies to the §5/§7 bridge mentions and content/docs/concepts/metadata-lifecycle.mdx.

Generated by Claude Code


Generated by Claude Code


Generated by Claude Code

…ns out to peer replicas (#13331)
Publisher at the protocol's post-persistence choke point publishes the
mutation's address on the new metadata.mutated cluster channel; peers
converge their ObjectQL registry from their OWN sys_metadata read (write-
through when an active row exists, the delete heal walk when none does),
then replay the event into local onMetadataMutation listeners. The bridge
plugin late-binds attachMetadataMutationPubSub as a second independent
lane at kernel:ready, guarded off the in-process memory driver.
Ruled 2026-09-01 (director batch A, Option A): producer-side fan-out
mirroring the shipped AuthzClusterBridgePlugin shape; the payload is a
signal, never trusted content. Option B (consumer-side self-heal) is not
built.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
…ouble; ledger the new pinned engine doubles
check:objectql-double-limit named the new find double limit-blind — the
bound now applies after the filter, by presence. The engine-double ledger
learned the fanout file's pinned delete/findOne/update doubles via the
gate's own --write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@github-actions

github-actionsBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/metadata-protocol, @objectstack/service-cluster, touching 19 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

5 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/wire-format.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/concepts/metadata-lifecycle.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), ObjectStackProtocolImplementation (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/deployment/validating-metadata.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/kernel/cluster.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/kernel/services-checklist.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class))

3 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v15.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/releases/v16.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts) — pages documenting those are invisible to this run
  • 1 anchor(s) matched too much of the corpus to be a work list: /api/v1/data (route, 35 pages)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 11 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fcpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 1304ec033ce539d92069afd7ac66110cc1951b11 — the merge of head 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec into base e62129153f2aa3d7666eab9c1af2bdb18e7333fc, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 1304ec033ce539d92069afd7ac66110cc1951b11 && git checkout 1304ec033ce539d92069afd7ac66110cc1951b11
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin e62129153f2aa3d7666eab9c1af2bdb18e7333fc 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec && git checkout -B drift-repro e62129153f2aa3d7666eab9c1af2bdb18e7333fc && git merge --no-ff 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec
node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fc

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs e62129153f2aa3d7666eab9c1af2bdb18e7333fc → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 1, 2026
…he fan-out insertion
check-system-context-census red on CI: the metadata.mutated publisher
block inserted above stripReadonlyForInsert moved the context.isSystem
read 1737 -> 1741, and the census page still anchored the old line. The
gate's own --fix performed the shift (one line-number rewrite, zero
prose); bare re-run green: 109 sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
The auto-merge of origin/main kept this branch's pre-#14171 anchor rows
beside the row-21 edit, flunking the census 26 ways (paired stale
anchors / unanchored sites in engine.ts and share-link-service.ts —
pure adjacent-row line rot, zero population change). Resolution per the
tool, not by hand: took main's page wholesale, then re-derived with
check-system-context-census --fix, which rewrote exactly ONE anchor
(row 21, protocol.ts 1737 to 1741 — this branch's publisher-block
shift). Delta vs origin/main is that single line; bare gate green: 109
sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-aiClaude

Copy link
Copy Markdown
Collaborator

Contract review — CONTRACT_REVIEW_TIER — verdict: PASS

Delegated review (dispatching PM seat session_01Q5WBDtaUnoz5XuJ6jk8pQ5 is below tier). Reviewed against the ruling on #13331 (comment 5486837547, director batch A, 「同意。」 on Option A). This is a plain comment, not an approval; the reviewer seat does not flip ready, enqueue, or merge.

Instruments. The diff was measured through the API (get_files / get_diff), never through the shallow checkout's three-dot diff: 9 files, +1199/−45, which sums exactly to the PR's own changed_files/additions/deletions — the file list is complete by that control. File contents were read at the pinned SHAs 9dd022ae (base) and 0cf0a437 (head) via git show/git grep (both objects present locally, verified with cat-file -e). Every zero-reading below carried a positive control on the same instrument that fired non-zero. Locations below are by symbol.


Clause-② limbs, judged from the diff

Content limb: YES — declared honestly and completely. The minted public surface, measured:

  • METADATA_MUTATION_CLUSTER_CHANNEL = 'metadata.mutated' + ClusterMetadataMutationPayload, exported from the @objectstack/metadata-protocol barrel (the only two new exports in packages/metadata-protocol/src/index.ts).
  • attachMetadataMutationPubSub(pubsub, nodeId) / detachMetadataMutationPubSub() as public methods on the already-exported ObjectStackProtocolImplementation.
  • Base-tree zero, with control: git grep at 9dd022ae for metadata.mutated|attachMetadataMutationPubSub|METADATA_MUTATION_CLUSTER_CHANNEL → 0 hits (control on same ref, metadata.changed in metadata-manager.ts → 3). Nothing pre-existing is widened or shadowed; the PR mints exactly what it declares.
  • No undeclared widening found.service-cluster's barrel index.ts is untouched (so the anticipated fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 conflict genuinely cannot arise); the bridge's lane 2 is duck-typed plugin behavior, no new export; emitMetadataMutation's split into notifyMutationListenersLocal is private; the changeset, the pinned-doubles ledger entry, and the census row are process artifacts, not surface.

Path limb: NO — measured, not taken from the author. From the complete API file list: zero paths under packages/spec/**, zero writes to packages/objectql/src/engine.ts. That is both fences, held:

  • engine.ts fence: held (0 files touch it; the authz precedent at attachAuthzInvalidationPubSub was mirrored in shape only — see placement below).
  • packages/spec fence: held, and no contract-first split is owed — see placement.

Is the contract minimal and right-shaped?

Two lanes are correct, not a duplicate. Measured, all at head:

  • Lane 1's only production publisher is MetadataManager.notifyWatchers (metadata-manager.ts, the single non-doc publish(MetadataManager.CLUSTER_CHANNEL…) in the tree; the two other grep hits are JSDoc examples in service-cluster).
  • protocol.ts contains zero code references to the metadata service or notifyWatchers — the only 3 hits are doc comments this PR itself adds. The authoring path (saveMetaItemapplyRegistryWriteThrough) cannot reach lane 1 on any boot shape.
  • The lanes carry different disciplines for different state owners: metadata.changed replays the legacy watch event verbatim (content on the wire) into the metadata SERVICE's caches; metadata.mutated is address-only into the ObjectQL engine registry. Overloading lane 1 with a signal-only receipt would collide with its content-replay contract; and the host-config boot has no manager at all (CORE_FALLBACK_FACTORIES = metadata/cache/queue/i18n — the fallback has no cluster seam, and there is no protocol fallback either, so lane 2 duck-types the real implementation or skips). The author's argument tests out.
  • Wiring is real, not just harness-deep: runtime.ts mounts MetadataClusterBridgePlugin unconditionally, and 'protocol' is registered by exactly one production site (assembleMetadataProtocolctx.registerService('protocol', protocolShim) where the "shim" is the ObjectStackProtocolImplementation instance), reached via ObjectQLPlugin's default registerProtocol = true. The variable-key fallback blind spot that inverted the original card does not apply here.

The payload is genuinely address-only, and the receiver genuinely re-reads.MetadataMutationEvent at head is exactly {type, name, state, organizationId?} — no body field exists on the type; all three emit sites construct only those four fields (the item body goes to the awaited projector, which is a separate mechanism, not the event). On receipt, applyRemoteMetadataMutation consumes the wire only as an ADDRESS: type folded through canonicalMetaType, repo.get(ref, {state:'active'}) against the replica's own store, item taken from current.body (own DB), packageId from resolveOverlayPackageBinding (own DB, the #4636 recovery-caller derivation, verbatim). Wire state is consumed only as a draft-suppression guard. Nothing from the wire is applied. The key-set test pins this at runtime.

Attach/detach shape. Idempotent on the (pubsub, nodeId) pair with a different-pair re-attach detaching first — measured side-by-side identical to MetadataManager.attachClusterPubSub. Loopback via originNode equality. Detach idempotent. The bridge detaches both lanes independently on kernel:shutdown with error isolation (a throwing lane-1 detach doesn't strand lane 2 — pinned). Lane 2 carries isInProcessClusterDriver from birth, matching AuthzClusterBridgePlugin, and lane 1 stays byte-identical (its warn line verbatim — pinned), leaving #14021 unabsorbed as declared.

Scope parity of the receipt. The write-through's env gate and org-scope verdict (hydrateOverlayIntoRegistry, #6602) and the heal walk's org refusal (restoreArtifactRegistryView returns for organizationId !== null, #6780) are inherited, not re-decided — a peer applies exactly what the writer's own kernel would have applied locally. A forged org-scoped object signal fails safe: the peer's own read finds no such row, the heal walk's org gate refuses, no-op.

Failure directions

  • Duplicate delivery: same read, same idempotent registration — pinned.
  • Out-of-order: the DB read decides, not the event name — the draft-discard test pins the sharpest case (a "delete"-shaped signal whose read finds an active row re-registers instead of healing).
  • Lost signal: degrades to the pre-existing bound (stale until boot reload). This matches IPubSub's own contract doc at head verbatim ("no shipped driver exceeds at-most-once… Handlers MUST be idempotent and tolerate loss"); the PR promises exactly one hop of narrowing and nothing more, in the changeset and the attach docblock.
  • No row on the peer: heal walk, org-gated as above.
  • Draft: never published (publisher gate) plus a receiver guard.
  • Failed apply: caught, warned, dropped → fail-closed 404 staleness, never wrong data.

Beyond-minimum: the local listener replay — sound, and in scope

The ruling's clause 1 orders the shipped bridge shape mirrored, and the shipped template itself ships remote replay: MetadataManager.attachClusterPubSub's receipt handler invalidates first, then replays into notifyWatchersLocal (measured at head). The PR's replay is the same shape with the same #5109 invalidate-before-notify ordering (registry first, listeners second — pinned by a test that asserts registry state from inside the listener). Never re-published — the local/publish split plus the docblock's storm reasoning is correct, since the loopback guard only suppresses a node's own messages. Replay targets measured: ObjectQL's hook/action rebind and the i18n authored-translation sync are per-replica in-memory re-syncs from the replica's own reads — precisely the issue's second-order defect (runtime-authored automation being single-node). The consumers with shared-DB side effects (email template, permission-set projection) run through the awaited projector seam on the shipped protocol, which is invoked at the write sites only and is NOT replayed — so no fleet-wide duplication of projection writes. Verdict: not a design fork; it is inside the ruled shape, and it is what makes the fix complete for peers.

Placement: metadata-protocol is the right home

  • The state that goes stale (the engine-registry write-through) and the sys_metadata read the ruled receipt needs both live in the protocol; the engine template was shape-only.
  • Both shipped precedents put channel constants and payloads with their state owners, not in spec — measured: AUTHZ_INVALIDATED_CHANNEL in packages/core/src/security/authz-invalidation-channel.ts; ClusterMetadataChangedPayload in packages/metadata/src/metadata-manager.ts. IPubSub from spec/contracts is the already-shipped generic transport (type-only import; no new dependency edge — metadata-protocol already imports spec).
  • The spec's MetadataChangedEventPayloadSchema (cluster.zod.ts): measured dormant — the only reference outside packages/spec in the whole tree is a prose comment in metadata-manager.ts (control: ClusterCapabilityConfig, same instrument, 4+ consuming packages). Its declared semantics ("compare version with their cached value… out-of-order older versions are ignored") make the wire the thing trusted, the opposite of the ruled re-read receipt; and its version: z.bigint() cannot cross JSON.stringify at all. Filing [finding] spec: MetadataChangedEventPayloadSchema says every metadata persistence layer MUST emit it — zero producers or consumers in-tree, and its bigint version field cannot cross JSON #14180 for the spec lane instead of building on it was correct. No spec-lane contract is wearing local clothes here.

Findings

  1. Non-blocking — concurrent same-row mutations can interleave at the peer. The applier is fire-and-forget per message with no per-key serialization: writer saves v1 then v2 in quick succession; the peer's read for the v1 signal (dispatched first, completed before v2's commit) can — under DB-response jitter — deliver its continuation after the v2 applier has registered, leaving the peer's registry at v1 while sys_metadata holds v2, until the next signal or boot. partitionKey orders delivery, not apply completion. Why non-blocking: the registered body is a genuinely-persisted prior state (never fabricated, never wrong-tenant), the window is one DB-read RTT, any later mutation re-converges, the same interleave class already exists between two racing local writers, and the outcome sits inside the staleness bound the PR explicitly declares (equivalent to a lost signal). If it ever matters in practice, a per-type:name apply queue is a receiver-internal fix that touches no public contract.
  2. Non-blocking — one more silent seam of the metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 family.deleteMetaItem's legacy control-plane branch (code-only rows; no repo path) deletes the row and heals its own registry but never emitted the mutation event even locally — pre-existing and marked deliberate in code ("emits no watch event… pre-existing and deliberate") — so it fans out to no one. Same class as the recovery doors metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 records, but metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 names only rollbackMetaItem/revertCommit. Recommend appending this branch to metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 rather than widening this PR.
  3. Non-blocking — wording. The payload docblock's "at-least-once delivery and duplicates are harmless by construction" states a tolerance, while shipped drivers are at-most-once; a skimming reader could take it for a delivery guarantee. The attach docblock, changeset, and PR body all state the loss bound plainly, so no change is required — noted only so nobody later quotes the payload docblock as a promise.

Also verified

  • The census docs row in this diff is exactly the claimed anchor shift: context?.isSystem in stripReadonlyForInsert measured at protocol.ts:1737 at base and :1741 at head; the docs hunk changes that one number and zero prose. Not scope creep.
  • The write-through caller pin re-opened correctly: measured exactly five this.applyRegistryWriteThrough( sites at head (base: four), the fifth being applyRemoteMetadataMutation; the route-5 case drives the real subscribe path and pins the wire-plural fold.
  • emitMetadataMutation's callers at head are exactly saveMetaItem / runPublishSideEffects / deleteMetaItem — the claimed choke point — and the recovery doors run the write-through without emitting, confirming metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 as a pre-existing gap, not this PR's.
  • CI at head 0cf0a437: 33/33 check runs green (Lint & Repo Gates included). I did not re-run the suites locally; suite verdicts are read from CI and the dev reports, all other claims above are my own measurements.

Verdict: PASS. No blocking findings; no design fork the ruling leaves unsettled. The enqueue-gate decision returns to the dispatching seat.


Generated by Claude Code

Same mechanism as the previous merge: main's #14199 re-anchored 11 rows
of this machine-maintained table while this branch re-anchors row 21,
and a textually clean merge proves nothing about the line numbers.
Resolution per the tool: took main's page wholesale, re-derived with
check-system-context-census --fix on the merged tree — exactly ONE
anchor rewritten (row 21, protocol.ts 1737 to 1741; re-derived, not
carried — main has zero commits on protocol.ts, so the insertion shift
is unchanged). Delta vs origin/main is that single line; bare gate
green (109 sites, 145 anchors). engine-double-contract ledger: zero
main-side commits, gate OK on the merged tree — no regeneration owed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-ai
os-support-ai added this pull request to the merge queueSep 1, 2026
Merged via the queue into main with commit 1403d94Sep 1, 2026
34 checks passed
@os-support-ai
os-support-ai deleted the claude/issue-13331-metadata-registry-cluster-fanout branch September 1, 2026 14:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/xlteststooling

Projects

None yet

2 participants

@os-support-ai@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas - #14183

Merged
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout
Sep 1, 2026
Merged

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas#14183
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout

Conversation

@claude

@claudeclaudeBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes#13331

What this ships

A runtime-authored metadata mutation now reaches every replica's ObjectQL registry, not just the writer's. Measured defect on the shipped 3-replica EE compose (ADR-0018): PUT /api/v1/meta/object/... persisted to the shared sys_metadata (meta reads 200 fleet-wide) while only the writing replica registered the object — /api/v1/data/... answered a hard 404 OBJECT_NOT_FOUND on the other two, indefinitely; 200 concurrent creates through the LB gave 67x201 / 133x404 with a boot-loaded control object at 0 errors.

Mechanism (maintainer-ruled 2026-09-01, director batch A, Option A — verbatim adoption of the escalation's recommendation):

  • Publisher at the producer choke point. The protocol's post-persistence funnel (emitMetadataMutation, the seam onMetadataMutation subscribes — reached by saveMetaItem, runPublishSideEffects, deleteMetaItem) now also publishes the mutation's ADDRESS on a new cluster channel metadata.mutated (METADATA_MUTATION_CLUSTER_CHANNEL, payload ClusterMetadataMutationPayload, both exported from @objectstack/metadata-protocol). Drafts are never published — they enter no registry anywhere.
  • Peers converge from their own DB read. On receipt a replica re-reads the row from its OWN sys_metadata: active row present ⇒ re-run applyRegistryWriteThrough (package binding derived the way the recovery callers derive it); no active row ⇒ run the delete heal walk (restoreArtifactRegistryView). The payload is a signal, never trusted content; duplicates and out-of-order delivery converge to the row's current state by construction. After convergence the event replays into the replica's LOCAL onMetadataMutation listeners (never re-published), so authored hook/action re-binds re-sync on peers.
  • Attach seam + bridge lane.ObjectStackProtocolImplementation.attachMetadataMutationPubSub(pubsub, nodeId) — idempotent on the pair, loopback-suppressed via originNode, mirroring MetadataManager.attachClusterPubSub and the engine's attachAuthzInvalidationPubSub. MetadataClusterBridgePlugin late-binds it at kernel:ready as a second, independent lane: the boot shape that lacks a manager-backed metadata service (TS-config host-config — exactly the shipped EE shape) is the one that needs this lane most. The new lane skips the in-process memory driver from birth (the guard the authz sibling carries).
  • Option B (consumer-side self-heal at assertObjectRegistered) is not built — presented only as a possible stopgap; the maintainer did not order one.

The two fences, both held

Evidence

  • Two-arm, directions declared in the test-file header before running (protocol.cluster-mutation-fanout.test.ts, two replicas over one shared store): Arm B (attached) — the peer converges from its own read; Arm A (control, no bridge) — the identical write leaves the peer empty, so Arm B is the bridge's doing, not a harness artifact. Also pinned: address-only payload (key set equality), draft silence, loopback suppression, duplicate-delivery convergence, re-attach idempotency, detach, delete fan-out, draft-discard keeping the peer's active registration, and listener replay ordered after registry convergence.
  • Committed-tree ablation at 44a2e59 (script with trap restore, absolute paths): the single publisher call deleted — mutation proved on disk (anchor count 0, marker count 1) — turned exactly the 7 declared bus-crossing cases red with 15 staying green (controls held); restore proven by HEAD-blob equality (d7ba0225...) plus empty git diff HEAD plus marker count 0; restored run 22/22 green. Resolution note: these tests import ./protocol.js relative from src — the subject never resolves through dist, so no rebuild leg exists to skip; the on-disk grep is the falsifiable observation for both legs.
  • Suites at final head e460193: metadata-protocol 152 files passed / 2 skipped (2094 tests passed / 10 skipped), service-cluster 6 files / 79 tests passed, tsc --noEmit clean with both edited test files confirmed inside the program via --listFiles (2 hits).
  • Gate union at e460193 (node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands; provenance line names this repo at this commit and the --repo assertion holds): 44 derived; 41 exit 0; 3 NOT MEASURED by their own printed verdicts (check-test-completeness exit 3 — grades a CI turbo log; check:dual-build-cjs-loads exit 3 — needs the full workspace build; check:type-check-debt exit 3 — full re-measure). Two first-pass reds were fixed and re-run to their own OK lines: check:engine-double-contract ("OK — 741 pinned, 134 in the DEBT ledger, 3 exempt"; ledger learned the fanout file's pinned doubles via the gate's own --write) and check:objectql-double-limit ("conformance holds: 299 doubles graded"; the new find double now holds the caller's bound by presence).
  • The write-through caller pin re-opened deliberately: applyRegistryWriteThrough grew its FIFTH caller (the peer applier). The trace, the count pin, and a new route-5 spelling case (a plural delivered over the wire registers under the singular; fold via canonicalMetaType, the complete map) are updated in protocol.object-registry-write-through-spelling.test.ts.

Notes for contract review

CI rework — census re-anchor (cdbe5f0)

Lint & Repo Gates red at e460193: check-system-context-census — pure line rot from this PR's own insertion (the publisher block sits above stripReadonlyForInsert, moving its context.isSystem read from protocol.ts:1737 to :1741 while content/docs/permissions/system-context.mdx row 21 still anchored 1737). Repaired with the gate's own --fix and VERIFIED as a SHIFT, not a population change: the rewrite touches exactly one line — the anchor's line number — and zero prose, and the census population is unchanged (109 sites / 145 anchors before and after). The docs file joining this diff is that re-anchor, not scope creep. Reproduced red locally first, then the bare gate re-run green: "OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read."

Docs-drift assessment (stated, not rewritten, per dispatch): content/docs/kernel/cluster.mdx §6.2 is now INCOMPLETE — it says cross-node metadata invalidation "already works" through the manager's metadata.changed lane alone, which is exactly the account #13331 falsified for the runtime-authoring path (the protocol never reaches the manager, and host-config boots carry no manager at all). The page wants a paragraph on the metadata.mutated protocol lane and the bridge's second attach; its own closing principle there ("the peer re-reads the shared store, which is the source of truth") is the rule the new receipt implements, and its "Target spec (planned)" block already marks MetadataChangedEventPayloadSchema as not wired — corroborating #14180. Same incompleteness class applies to the §5/§7 bridge mentions and content/docs/concepts/metadata-lifecycle.mdx.

Generated by Claude Code


Generated by Claude Code


Generated by Claude Code

…ns out to peer replicas (#13331)
Publisher at the protocol's post-persistence choke point publishes the
mutation's address on the new metadata.mutated cluster channel; peers
converge their ObjectQL registry from their OWN sys_metadata read (write-
through when an active row exists, the delete heal walk when none does),
then replay the event into local onMetadataMutation listeners. The bridge
plugin late-binds attachMetadataMutationPubSub as a second independent
lane at kernel:ready, guarded off the in-process memory driver.
Ruled 2026-09-01 (director batch A, Option A): producer-side fan-out
mirroring the shipped AuthzClusterBridgePlugin shape; the payload is a
signal, never trusted content. Option B (consumer-side self-heal) is not
built.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
…ouble; ledger the new pinned engine doubles
check:objectql-double-limit named the new find double limit-blind — the
bound now applies after the filter, by presence. The engine-double ledger
learned the fanout file's pinned delete/findOne/update doubles via the
gate's own --write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@github-actions

github-actionsBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/metadata-protocol, @objectstack/service-cluster, touching 19 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

5 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/wire-format.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/concepts/metadata-lifecycle.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), ObjectStackProtocolImplementation (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/deployment/validating-metadata.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/kernel/cluster.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/kernel/services-checklist.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class))

3 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v15.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/releases/v16.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts) — pages documenting those are invisible to this run
  • 1 anchor(s) matched too much of the corpus to be a work list: /api/v1/data (route, 35 pages)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 11 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fcpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 1304ec033ce539d92069afd7ac66110cc1951b11 — the merge of head 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec into base e62129153f2aa3d7666eab9c1af2bdb18e7333fc, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 1304ec033ce539d92069afd7ac66110cc1951b11 && git checkout 1304ec033ce539d92069afd7ac66110cc1951b11
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin e62129153f2aa3d7666eab9c1af2bdb18e7333fc 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec && git checkout -B drift-repro e62129153f2aa3d7666eab9c1af2bdb18e7333fc && git merge --no-ff 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec
node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fc

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs e62129153f2aa3d7666eab9c1af2bdb18e7333fc → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 1, 2026
…he fan-out insertion
check-system-context-census red on CI: the metadata.mutated publisher
block inserted above stripReadonlyForInsert moved the context.isSystem
read 1737 -> 1741, and the census page still anchored the old line. The
gate's own --fix performed the shift (one line-number rewrite, zero
prose); bare re-run green: 109 sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
The auto-merge of origin/main kept this branch's pre-#14171 anchor rows
beside the row-21 edit, flunking the census 26 ways (paired stale
anchors / unanchored sites in engine.ts and share-link-service.ts —
pure adjacent-row line rot, zero population change). Resolution per the
tool, not by hand: took main's page wholesale, then re-derived with
check-system-context-census --fix, which rewrote exactly ONE anchor
(row 21, protocol.ts 1737 to 1741 — this branch's publisher-block
shift). Delta vs origin/main is that single line; bare gate green: 109
sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-aiClaude

Copy link
Copy Markdown
Collaborator

Contract review — CONTRACT_REVIEW_TIER — verdict: PASS

Delegated review (dispatching PM seat session_01Q5WBDtaUnoz5XuJ6jk8pQ5 is below tier). Reviewed against the ruling on #13331 (comment 5486837547, director batch A, 「同意。」 on Option A). This is a plain comment, not an approval; the reviewer seat does not flip ready, enqueue, or merge.

Instruments. The diff was measured through the API (get_files / get_diff), never through the shallow checkout's three-dot diff: 9 files, +1199/−45, which sums exactly to the PR's own changed_files/additions/deletions — the file list is complete by that control. File contents were read at the pinned SHAs 9dd022ae (base) and 0cf0a437 (head) via git show/git grep (both objects present locally, verified with cat-file -e). Every zero-reading below carried a positive control on the same instrument that fired non-zero. Locations below are by symbol.


Clause-② limbs, judged from the diff

Content limb: YES — declared honestly and completely. The minted public surface, measured:

  • METADATA_MUTATION_CLUSTER_CHANNEL = 'metadata.mutated' + ClusterMetadataMutationPayload, exported from the @objectstack/metadata-protocol barrel (the only two new exports in packages/metadata-protocol/src/index.ts).
  • attachMetadataMutationPubSub(pubsub, nodeId) / detachMetadataMutationPubSub() as public methods on the already-exported ObjectStackProtocolImplementation.
  • Base-tree zero, with control: git grep at 9dd022ae for metadata.mutated|attachMetadataMutationPubSub|METADATA_MUTATION_CLUSTER_CHANNEL → 0 hits (control on same ref, metadata.changed in metadata-manager.ts → 3). Nothing pre-existing is widened or shadowed; the PR mints exactly what it declares.
  • No undeclared widening found.service-cluster's barrel index.ts is untouched (so the anticipated fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 conflict genuinely cannot arise); the bridge's lane 2 is duck-typed plugin behavior, no new export; emitMetadataMutation's split into notifyMutationListenersLocal is private; the changeset, the pinned-doubles ledger entry, and the census row are process artifacts, not surface.

Path limb: NO — measured, not taken from the author. From the complete API file list: zero paths under packages/spec/**, zero writes to packages/objectql/src/engine.ts. That is both fences, held:

  • engine.ts fence: held (0 files touch it; the authz precedent at attachAuthzInvalidationPubSub was mirrored in shape only — see placement below).
  • packages/spec fence: held, and no contract-first split is owed — see placement.

Is the contract minimal and right-shaped?

Two lanes are correct, not a duplicate. Measured, all at head:

  • Lane 1's only production publisher is MetadataManager.notifyWatchers (metadata-manager.ts, the single non-doc publish(MetadataManager.CLUSTER_CHANNEL…) in the tree; the two other grep hits are JSDoc examples in service-cluster).
  • protocol.ts contains zero code references to the metadata service or notifyWatchers — the only 3 hits are doc comments this PR itself adds. The authoring path (saveMetaItemapplyRegistryWriteThrough) cannot reach lane 1 on any boot shape.
  • The lanes carry different disciplines for different state owners: metadata.changed replays the legacy watch event verbatim (content on the wire) into the metadata SERVICE's caches; metadata.mutated is address-only into the ObjectQL engine registry. Overloading lane 1 with a signal-only receipt would collide with its content-replay contract; and the host-config boot has no manager at all (CORE_FALLBACK_FACTORIES = metadata/cache/queue/i18n — the fallback has no cluster seam, and there is no protocol fallback either, so lane 2 duck-types the real implementation or skips). The author's argument tests out.
  • Wiring is real, not just harness-deep: runtime.ts mounts MetadataClusterBridgePlugin unconditionally, and 'protocol' is registered by exactly one production site (assembleMetadataProtocolctx.registerService('protocol', protocolShim) where the "shim" is the ObjectStackProtocolImplementation instance), reached via ObjectQLPlugin's default registerProtocol = true. The variable-key fallback blind spot that inverted the original card does not apply here.

The payload is genuinely address-only, and the receiver genuinely re-reads.MetadataMutationEvent at head is exactly {type, name, state, organizationId?} — no body field exists on the type; all three emit sites construct only those four fields (the item body goes to the awaited projector, which is a separate mechanism, not the event). On receipt, applyRemoteMetadataMutation consumes the wire only as an ADDRESS: type folded through canonicalMetaType, repo.get(ref, {state:'active'}) against the replica's own store, item taken from current.body (own DB), packageId from resolveOverlayPackageBinding (own DB, the #4636 recovery-caller derivation, verbatim). Wire state is consumed only as a draft-suppression guard. Nothing from the wire is applied. The key-set test pins this at runtime.

Attach/detach shape. Idempotent on the (pubsub, nodeId) pair with a different-pair re-attach detaching first — measured side-by-side identical to MetadataManager.attachClusterPubSub. Loopback via originNode equality. Detach idempotent. The bridge detaches both lanes independently on kernel:shutdown with error isolation (a throwing lane-1 detach doesn't strand lane 2 — pinned). Lane 2 carries isInProcessClusterDriver from birth, matching AuthzClusterBridgePlugin, and lane 1 stays byte-identical (its warn line verbatim — pinned), leaving #14021 unabsorbed as declared.

Scope parity of the receipt. The write-through's env gate and org-scope verdict (hydrateOverlayIntoRegistry, #6602) and the heal walk's org refusal (restoreArtifactRegistryView returns for organizationId !== null, #6780) are inherited, not re-decided — a peer applies exactly what the writer's own kernel would have applied locally. A forged org-scoped object signal fails safe: the peer's own read finds no such row, the heal walk's org gate refuses, no-op.

Failure directions

  • Duplicate delivery: same read, same idempotent registration — pinned.
  • Out-of-order: the DB read decides, not the event name — the draft-discard test pins the sharpest case (a "delete"-shaped signal whose read finds an active row re-registers instead of healing).
  • Lost signal: degrades to the pre-existing bound (stale until boot reload). This matches IPubSub's own contract doc at head verbatim ("no shipped driver exceeds at-most-once… Handlers MUST be idempotent and tolerate loss"); the PR promises exactly one hop of narrowing and nothing more, in the changeset and the attach docblock.
  • No row on the peer: heal walk, org-gated as above.
  • Draft: never published (publisher gate) plus a receiver guard.
  • Failed apply: caught, warned, dropped → fail-closed 404 staleness, never wrong data.

Beyond-minimum: the local listener replay — sound, and in scope

The ruling's clause 1 orders the shipped bridge shape mirrored, and the shipped template itself ships remote replay: MetadataManager.attachClusterPubSub's receipt handler invalidates first, then replays into notifyWatchersLocal (measured at head). The PR's replay is the same shape with the same #5109 invalidate-before-notify ordering (registry first, listeners second — pinned by a test that asserts registry state from inside the listener). Never re-published — the local/publish split plus the docblock's storm reasoning is correct, since the loopback guard only suppresses a node's own messages. Replay targets measured: ObjectQL's hook/action rebind and the i18n authored-translation sync are per-replica in-memory re-syncs from the replica's own reads — precisely the issue's second-order defect (runtime-authored automation being single-node). The consumers with shared-DB side effects (email template, permission-set projection) run through the awaited projector seam on the shipped protocol, which is invoked at the write sites only and is NOT replayed — so no fleet-wide duplication of projection writes. Verdict: not a design fork; it is inside the ruled shape, and it is what makes the fix complete for peers.

Placement: metadata-protocol is the right home

  • The state that goes stale (the engine-registry write-through) and the sys_metadata read the ruled receipt needs both live in the protocol; the engine template was shape-only.
  • Both shipped precedents put channel constants and payloads with their state owners, not in spec — measured: AUTHZ_INVALIDATED_CHANNEL in packages/core/src/security/authz-invalidation-channel.ts; ClusterMetadataChangedPayload in packages/metadata/src/metadata-manager.ts. IPubSub from spec/contracts is the already-shipped generic transport (type-only import; no new dependency edge — metadata-protocol already imports spec).
  • The spec's MetadataChangedEventPayloadSchema (cluster.zod.ts): measured dormant — the only reference outside packages/spec in the whole tree is a prose comment in metadata-manager.ts (control: ClusterCapabilityConfig, same instrument, 4+ consuming packages). Its declared semantics ("compare version with their cached value… out-of-order older versions are ignored") make the wire the thing trusted, the opposite of the ruled re-read receipt; and its version: z.bigint() cannot cross JSON.stringify at all. Filing [finding] spec: MetadataChangedEventPayloadSchema says every metadata persistence layer MUST emit it — zero producers or consumers in-tree, and its bigint version field cannot cross JSON #14180 for the spec lane instead of building on it was correct. No spec-lane contract is wearing local clothes here.

Findings

  1. Non-blocking — concurrent same-row mutations can interleave at the peer. The applier is fire-and-forget per message with no per-key serialization: writer saves v1 then v2 in quick succession; the peer's read for the v1 signal (dispatched first, completed before v2's commit) can — under DB-response jitter — deliver its continuation after the v2 applier has registered, leaving the peer's registry at v1 while sys_metadata holds v2, until the next signal or boot. partitionKey orders delivery, not apply completion. Why non-blocking: the registered body is a genuinely-persisted prior state (never fabricated, never wrong-tenant), the window is one DB-read RTT, any later mutation re-converges, the same interleave class already exists between two racing local writers, and the outcome sits inside the staleness bound the PR explicitly declares (equivalent to a lost signal). If it ever matters in practice, a per-type:name apply queue is a receiver-internal fix that touches no public contract.
  2. Non-blocking — one more silent seam of the metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 family.deleteMetaItem's legacy control-plane branch (code-only rows; no repo path) deletes the row and heals its own registry but never emitted the mutation event even locally — pre-existing and marked deliberate in code ("emits no watch event… pre-existing and deliberate") — so it fans out to no one. Same class as the recovery doors metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 records, but metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 names only rollbackMetaItem/revertCommit. Recommend appending this branch to metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 rather than widening this PR.
  3. Non-blocking — wording. The payload docblock's "at-least-once delivery and duplicates are harmless by construction" states a tolerance, while shipped drivers are at-most-once; a skimming reader could take it for a delivery guarantee. The attach docblock, changeset, and PR body all state the loss bound plainly, so no change is required — noted only so nobody later quotes the payload docblock as a promise.

Also verified

  • The census docs row in this diff is exactly the claimed anchor shift: context?.isSystem in stripReadonlyForInsert measured at protocol.ts:1737 at base and :1741 at head; the docs hunk changes that one number and zero prose. Not scope creep.
  • The write-through caller pin re-opened correctly: measured exactly five this.applyRegistryWriteThrough( sites at head (base: four), the fifth being applyRemoteMetadataMutation; the route-5 case drives the real subscribe path and pins the wire-plural fold.
  • emitMetadataMutation's callers at head are exactly saveMetaItem / runPublishSideEffects / deleteMetaItem — the claimed choke point — and the recovery doors run the write-through without emitting, confirming metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 as a pre-existing gap, not this PR's.
  • CI at head 0cf0a437: 33/33 check runs green (Lint & Repo Gates included). I did not re-run the suites locally; suite verdicts are read from CI and the dev reports, all other claims above are my own measurements.

Verdict: PASS. No blocking findings; no design fork the ruling leaves unsettled. The enqueue-gate decision returns to the dispatching seat.


Generated by Claude Code

Same mechanism as the previous merge: main's #14199 re-anchored 11 rows
of this machine-maintained table while this branch re-anchors row 21,
and a textually clean merge proves nothing about the line numbers.
Resolution per the tool: took main's page wholesale, re-derived with
check-system-context-census --fix on the merged tree — exactly ONE
anchor rewritten (row 21, protocol.ts 1737 to 1741; re-derived, not
carried — main has zero commits on protocol.ts, so the insertion shift
is unchanged). Delta vs origin/main is that single line; bare gate
green (109 sites, 145 anchors). engine-double-contract ledger: zero
main-side commits, gate OK on the merged tree — no regeneration owed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-ai
os-support-ai added this pull request to the merge queueSep 1, 2026
Merged via the queue into main with commit 1403d94Sep 1, 2026
34 checks passed
@os-support-ai
os-support-ai deleted the claude/issue-13331-metadata-registry-cluster-fanout branch September 1, 2026 14:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/xlteststooling

Projects

None yet

2 participants

@os-support-ai@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas - #14183

Merged
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout
Sep 1, 2026
Merged

feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas#14183
os-support-ai merged 7 commits into
mainfrom
claude/issue-13331-metadata-registry-cluster-fanout

Conversation

@claude

@claudeclaudeBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes#13331

What this ships

A runtime-authored metadata mutation now reaches every replica's ObjectQL registry, not just the writer's. Measured defect on the shipped 3-replica EE compose (ADR-0018): PUT /api/v1/meta/object/... persisted to the shared sys_metadata (meta reads 200 fleet-wide) while only the writing replica registered the object — /api/v1/data/... answered a hard 404 OBJECT_NOT_FOUND on the other two, indefinitely; 200 concurrent creates through the LB gave 67x201 / 133x404 with a boot-loaded control object at 0 errors.

Mechanism (maintainer-ruled 2026-09-01, director batch A, Option A — verbatim adoption of the escalation's recommendation):

  • Publisher at the producer choke point. The protocol's post-persistence funnel (emitMetadataMutation, the seam onMetadataMutation subscribes — reached by saveMetaItem, runPublishSideEffects, deleteMetaItem) now also publishes the mutation's ADDRESS on a new cluster channel metadata.mutated (METADATA_MUTATION_CLUSTER_CHANNEL, payload ClusterMetadataMutationPayload, both exported from @objectstack/metadata-protocol). Drafts are never published — they enter no registry anywhere.
  • Peers converge from their own DB read. On receipt a replica re-reads the row from its OWN sys_metadata: active row present ⇒ re-run applyRegistryWriteThrough (package binding derived the way the recovery callers derive it); no active row ⇒ run the delete heal walk (restoreArtifactRegistryView). The payload is a signal, never trusted content; duplicates and out-of-order delivery converge to the row's current state by construction. After convergence the event replays into the replica's LOCAL onMetadataMutation listeners (never re-published), so authored hook/action re-binds re-sync on peers.
  • Attach seam + bridge lane.ObjectStackProtocolImplementation.attachMetadataMutationPubSub(pubsub, nodeId) — idempotent on the pair, loopback-suppressed via originNode, mirroring MetadataManager.attachClusterPubSub and the engine's attachAuthzInvalidationPubSub. MetadataClusterBridgePlugin late-binds it at kernel:ready as a second, independent lane: the boot shape that lacks a manager-backed metadata service (TS-config host-config — exactly the shipped EE shape) is the one that needs this lane most. The new lane skips the in-process memory driver from birth (the guard the authz sibling carries).
  • Option B (consumer-side self-heal at assertObjectRegistered) is not built — presented only as a possible stopgap; the maintainer did not order one.

The two fences, both held

Evidence

  • Two-arm, directions declared in the test-file header before running (protocol.cluster-mutation-fanout.test.ts, two replicas over one shared store): Arm B (attached) — the peer converges from its own read; Arm A (control, no bridge) — the identical write leaves the peer empty, so Arm B is the bridge's doing, not a harness artifact. Also pinned: address-only payload (key set equality), draft silence, loopback suppression, duplicate-delivery convergence, re-attach idempotency, detach, delete fan-out, draft-discard keeping the peer's active registration, and listener replay ordered after registry convergence.
  • Committed-tree ablation at 44a2e59 (script with trap restore, absolute paths): the single publisher call deleted — mutation proved on disk (anchor count 0, marker count 1) — turned exactly the 7 declared bus-crossing cases red with 15 staying green (controls held); restore proven by HEAD-blob equality (d7ba0225...) plus empty git diff HEAD plus marker count 0; restored run 22/22 green. Resolution note: these tests import ./protocol.js relative from src — the subject never resolves through dist, so no rebuild leg exists to skip; the on-disk grep is the falsifiable observation for both legs.
  • Suites at final head e460193: metadata-protocol 152 files passed / 2 skipped (2094 tests passed / 10 skipped), service-cluster 6 files / 79 tests passed, tsc --noEmit clean with both edited test files confirmed inside the program via --listFiles (2 hits).
  • Gate union at e460193 (node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands; provenance line names this repo at this commit and the --repo assertion holds): 44 derived; 41 exit 0; 3 NOT MEASURED by their own printed verdicts (check-test-completeness exit 3 — grades a CI turbo log; check:dual-build-cjs-loads exit 3 — needs the full workspace build; check:type-check-debt exit 3 — full re-measure). Two first-pass reds were fixed and re-run to their own OK lines: check:engine-double-contract ("OK — 741 pinned, 134 in the DEBT ledger, 3 exempt"; ledger learned the fanout file's pinned doubles via the gate's own --write) and check:objectql-double-limit ("conformance holds: 299 doubles graded"; the new find double now holds the caller's bound by presence).
  • The write-through caller pin re-opened deliberately: applyRegistryWriteThrough grew its FIFTH caller (the peer applier). The trace, the count pin, and a new route-5 spelling case (a plural delivered over the wire registers under the singular; fold via canonicalMetaType, the complete map) are updated in protocol.object-registry-write-through-spelling.test.ts.

Notes for contract review

CI rework — census re-anchor (cdbe5f0)

Lint & Repo Gates red at e460193: check-system-context-census — pure line rot from this PR's own insertion (the publisher block sits above stripReadonlyForInsert, moving its context.isSystem read from protocol.ts:1737 to :1741 while content/docs/permissions/system-context.mdx row 21 still anchored 1737). Repaired with the gate's own --fix and VERIFIED as a SHIFT, not a population change: the rewrite touches exactly one line — the anchor's line number — and zero prose, and the census population is unchanged (109 sites / 145 anchors before and after). The docs file joining this diff is that re-anchor, not scope creep. Reproduced red locally first, then the bare gate re-run green: "OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read."

Docs-drift assessment (stated, not rewritten, per dispatch): content/docs/kernel/cluster.mdx §6.2 is now INCOMPLETE — it says cross-node metadata invalidation "already works" through the manager's metadata.changed lane alone, which is exactly the account #13331 falsified for the runtime-authoring path (the protocol never reaches the manager, and host-config boots carry no manager at all). The page wants a paragraph on the metadata.mutated protocol lane and the bridge's second attach; its own closing principle there ("the peer re-reads the shared store, which is the source of truth") is the rule the new receipt implements, and its "Target spec (planned)" block already marks MetadataChangedEventPayloadSchema as not wired — corroborating #14180. Same incompleteness class applies to the §5/§7 bridge mentions and content/docs/concepts/metadata-lifecycle.mdx.

Generated by Claude Code


Generated by Claude Code


Generated by Claude Code

…ns out to peer replicas (#13331)
Publisher at the protocol's post-persistence choke point publishes the
mutation's address on the new metadata.mutated cluster channel; peers
converge their ObjectQL registry from their OWN sys_metadata read (write-
through when an active row exists, the delete heal walk when none does),
then replay the event into local onMetadataMutation listeners. The bridge
plugin late-binds attachMetadataMutationPubSub as a second independent
lane at kernel:ready, guarded off the in-process memory driver.
Ruled 2026-09-01 (director batch A, Option A): producer-side fan-out
mirroring the shipped AuthzClusterBridgePlugin shape; the payload is a
signal, never trusted content. Option B (consumer-side self-heal) is not
built.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
…ouble; ledger the new pinned engine doubles
check:objectql-double-limit named the new find double limit-blind — the
bound now applies after the filter, by presence. The engine-double ledger
learned the fanout file's pinned delete/findOne/update doubles via the
gate's own --write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@github-actions

github-actionsBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/metadata-protocol, @objectstack/service-cluster, touching 19 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

5 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/wire-format.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/concepts/metadata-lifecycle.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), ObjectStackProtocolImplementation (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/deployment/validating-metadata.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/kernel/cluster.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class), originNode (symbol, a field of interface ClusterMetadataMutationPayload))
  • content/docs/kernel/services-checklist.mdx(via MetadataClusterBridgePlugin (symbol, a top-level class))

3 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v15.mdx(via /api/v1/meta/* (route, a path literal in ObjectStackProtocolImplementation; a path literal on a changed line))
  • content/docs/releases/v16.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))
  • content/docs/releases/v17.mdx(via ObjectStackProtocolImplementation (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/metadata-protocol/src/index.ts) — pages documenting those are invisible to this run
  • 1 anchor(s) matched too much of the corpus to be a work list: /api/v1/data (route, 35 pages)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 11 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fcpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 1304ec033ce539d92069afd7ac66110cc1951b11 — the merge of head 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec into base e62129153f2aa3d7666eab9c1af2bdb18e7333fc, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 1304ec033ce539d92069afd7ac66110cc1951b11 && git checkout 1304ec033ce539d92069afd7ac66110cc1951b11
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin e62129153f2aa3d7666eab9c1af2bdb18e7333fc 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec && git checkout -B drift-repro e62129153f2aa3d7666eab9c1af2bdb18e7333fc && git merge --no-ff 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec
node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fc

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs e62129153f2aa3d7666eab9c1af2bdb18e7333fc → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 1, 2026
…he fan-out insertion
check-system-context-census red on CI: the metadata.mutated publisher
block inserted above stripReadonlyForInsert moved the context.isSystem
read 1737 -> 1741, and the census page still anchored the old line. The
gate's own --fix performed the shift (one line-number rewrite, zero
prose); bare re-run green: 109 sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
The auto-merge of origin/main kept this branch's pre-#14171 anchor rows
beside the row-21 edit, flunking the census 26 ways (paired stale
anchors / unanchored sites in engine.ts and share-link-service.ts —
pure adjacent-row line rot, zero population change). Resolution per the
tool, not by hand: took main's page wholesale, then re-derived with
check-system-context-census --fix, which rewrote exactly ONE anchor
(row 21, protocol.ts 1737 to 1741 — this branch's publisher-block
shift). Delta vs origin/main is that single line; bare gate green: 109
sites anchored, 145 anchors resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-aiClaude

Copy link
Copy Markdown
Collaborator

Contract review — CONTRACT_REVIEW_TIER — verdict: PASS

Delegated review (dispatching PM seat session_01Q5WBDtaUnoz5XuJ6jk8pQ5 is below tier). Reviewed against the ruling on #13331 (comment 5486837547, director batch A, 「同意。」 on Option A). This is a plain comment, not an approval; the reviewer seat does not flip ready, enqueue, or merge.

Instruments. The diff was measured through the API (get_files / get_diff), never through the shallow checkout's three-dot diff: 9 files, +1199/−45, which sums exactly to the PR's own changed_files/additions/deletions — the file list is complete by that control. File contents were read at the pinned SHAs 9dd022ae (base) and 0cf0a437 (head) via git show/git grep (both objects present locally, verified with cat-file -e). Every zero-reading below carried a positive control on the same instrument that fired non-zero. Locations below are by symbol.


Clause-② limbs, judged from the diff

Content limb: YES — declared honestly and completely. The minted public surface, measured:

  • METADATA_MUTATION_CLUSTER_CHANNEL = 'metadata.mutated' + ClusterMetadataMutationPayload, exported from the @objectstack/metadata-protocol barrel (the only two new exports in packages/metadata-protocol/src/index.ts).
  • attachMetadataMutationPubSub(pubsub, nodeId) / detachMetadataMutationPubSub() as public methods on the already-exported ObjectStackProtocolImplementation.
  • Base-tree zero, with control: git grep at 9dd022ae for metadata.mutated|attachMetadataMutationPubSub|METADATA_MUTATION_CLUSTER_CHANNEL → 0 hits (control on same ref, metadata.changed in metadata-manager.ts → 3). Nothing pre-existing is widened or shadowed; the PR mints exactly what it declares.
  • No undeclared widening found.service-cluster's barrel index.ts is untouched (so the anticipated fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 conflict genuinely cannot arise); the bridge's lane 2 is duck-typed plugin behavior, no new export; emitMetadataMutation's split into notifyMutationListenersLocal is private; the changeset, the pinned-doubles ledger entry, and the census row are process artifacts, not surface.

Path limb: NO — measured, not taken from the author. From the complete API file list: zero paths under packages/spec/**, zero writes to packages/objectql/src/engine.ts. That is both fences, held:

  • engine.ts fence: held (0 files touch it; the authz precedent at attachAuthzInvalidationPubSub was mirrored in shape only — see placement below).
  • packages/spec fence: held, and no contract-first split is owed — see placement.

Is the contract minimal and right-shaped?

Two lanes are correct, not a duplicate. Measured, all at head:

  • Lane 1's only production publisher is MetadataManager.notifyWatchers (metadata-manager.ts, the single non-doc publish(MetadataManager.CLUSTER_CHANNEL…) in the tree; the two other grep hits are JSDoc examples in service-cluster).
  • protocol.ts contains zero code references to the metadata service or notifyWatchers — the only 3 hits are doc comments this PR itself adds. The authoring path (saveMetaItemapplyRegistryWriteThrough) cannot reach lane 1 on any boot shape.
  • The lanes carry different disciplines for different state owners: metadata.changed replays the legacy watch event verbatim (content on the wire) into the metadata SERVICE's caches; metadata.mutated is address-only into the ObjectQL engine registry. Overloading lane 1 with a signal-only receipt would collide with its content-replay contract; and the host-config boot has no manager at all (CORE_FALLBACK_FACTORIES = metadata/cache/queue/i18n — the fallback has no cluster seam, and there is no protocol fallback either, so lane 2 duck-types the real implementation or skips). The author's argument tests out.
  • Wiring is real, not just harness-deep: runtime.ts mounts MetadataClusterBridgePlugin unconditionally, and 'protocol' is registered by exactly one production site (assembleMetadataProtocolctx.registerService('protocol', protocolShim) where the "shim" is the ObjectStackProtocolImplementation instance), reached via ObjectQLPlugin's default registerProtocol = true. The variable-key fallback blind spot that inverted the original card does not apply here.

The payload is genuinely address-only, and the receiver genuinely re-reads.MetadataMutationEvent at head is exactly {type, name, state, organizationId?} — no body field exists on the type; all three emit sites construct only those four fields (the item body goes to the awaited projector, which is a separate mechanism, not the event). On receipt, applyRemoteMetadataMutation consumes the wire only as an ADDRESS: type folded through canonicalMetaType, repo.get(ref, {state:'active'}) against the replica's own store, item taken from current.body (own DB), packageId from resolveOverlayPackageBinding (own DB, the #4636 recovery-caller derivation, verbatim). Wire state is consumed only as a draft-suppression guard. Nothing from the wire is applied. The key-set test pins this at runtime.

Attach/detach shape. Idempotent on the (pubsub, nodeId) pair with a different-pair re-attach detaching first — measured side-by-side identical to MetadataManager.attachClusterPubSub. Loopback via originNode equality. Detach idempotent. The bridge detaches both lanes independently on kernel:shutdown with error isolation (a throwing lane-1 detach doesn't strand lane 2 — pinned). Lane 2 carries isInProcessClusterDriver from birth, matching AuthzClusterBridgePlugin, and lane 1 stays byte-identical (its warn line verbatim — pinned), leaving #14021 unabsorbed as declared.

Scope parity of the receipt. The write-through's env gate and org-scope verdict (hydrateOverlayIntoRegistry, #6602) and the heal walk's org refusal (restoreArtifactRegistryView returns for organizationId !== null, #6780) are inherited, not re-decided — a peer applies exactly what the writer's own kernel would have applied locally. A forged org-scoped object signal fails safe: the peer's own read finds no such row, the heal walk's org gate refuses, no-op.

Failure directions

  • Duplicate delivery: same read, same idempotent registration — pinned.
  • Out-of-order: the DB read decides, not the event name — the draft-discard test pins the sharpest case (a "delete"-shaped signal whose read finds an active row re-registers instead of healing).
  • Lost signal: degrades to the pre-existing bound (stale until boot reload). This matches IPubSub's own contract doc at head verbatim ("no shipped driver exceeds at-most-once… Handlers MUST be idempotent and tolerate loss"); the PR promises exactly one hop of narrowing and nothing more, in the changeset and the attach docblock.
  • No row on the peer: heal walk, org-gated as above.
  • Draft: never published (publisher gate) plus a receiver guard.
  • Failed apply: caught, warned, dropped → fail-closed 404 staleness, never wrong data.

Beyond-minimum: the local listener replay — sound, and in scope

The ruling's clause 1 orders the shipped bridge shape mirrored, and the shipped template itself ships remote replay: MetadataManager.attachClusterPubSub's receipt handler invalidates first, then replays into notifyWatchersLocal (measured at head). The PR's replay is the same shape with the same #5109 invalidate-before-notify ordering (registry first, listeners second — pinned by a test that asserts registry state from inside the listener). Never re-published — the local/publish split plus the docblock's storm reasoning is correct, since the loopback guard only suppresses a node's own messages. Replay targets measured: ObjectQL's hook/action rebind and the i18n authored-translation sync are per-replica in-memory re-syncs from the replica's own reads — precisely the issue's second-order defect (runtime-authored automation being single-node). The consumers with shared-DB side effects (email template, permission-set projection) run through the awaited projector seam on the shipped protocol, which is invoked at the write sites only and is NOT replayed — so no fleet-wide duplication of projection writes. Verdict: not a design fork; it is inside the ruled shape, and it is what makes the fix complete for peers.

Placement: metadata-protocol is the right home

  • The state that goes stale (the engine-registry write-through) and the sys_metadata read the ruled receipt needs both live in the protocol; the engine template was shape-only.
  • Both shipped precedents put channel constants and payloads with their state owners, not in spec — measured: AUTHZ_INVALIDATED_CHANNEL in packages/core/src/security/authz-invalidation-channel.ts; ClusterMetadataChangedPayload in packages/metadata/src/metadata-manager.ts. IPubSub from spec/contracts is the already-shipped generic transport (type-only import; no new dependency edge — metadata-protocol already imports spec).
  • The spec's MetadataChangedEventPayloadSchema (cluster.zod.ts): measured dormant — the only reference outside packages/spec in the whole tree is a prose comment in metadata-manager.ts (control: ClusterCapabilityConfig, same instrument, 4+ consuming packages). Its declared semantics ("compare version with their cached value… out-of-order older versions are ignored") make the wire the thing trusted, the opposite of the ruled re-read receipt; and its version: z.bigint() cannot cross JSON.stringify at all. Filing [finding] spec: MetadataChangedEventPayloadSchema says every metadata persistence layer MUST emit it — zero producers or consumers in-tree, and its bigint version field cannot cross JSON #14180 for the spec lane instead of building on it was correct. No spec-lane contract is wearing local clothes here.

Findings

  1. Non-blocking — concurrent same-row mutations can interleave at the peer. The applier is fire-and-forget per message with no per-key serialization: writer saves v1 then v2 in quick succession; the peer's read for the v1 signal (dispatched first, completed before v2's commit) can — under DB-response jitter — deliver its continuation after the v2 applier has registered, leaving the peer's registry at v1 while sys_metadata holds v2, until the next signal or boot. partitionKey orders delivery, not apply completion. Why non-blocking: the registered body is a genuinely-persisted prior state (never fabricated, never wrong-tenant), the window is one DB-read RTT, any later mutation re-converges, the same interleave class already exists between two racing local writers, and the outcome sits inside the staleness bound the PR explicitly declares (equivalent to a lost signal). If it ever matters in practice, a per-type:name apply queue is a receiver-internal fix that touches no public contract.
  2. Non-blocking — one more silent seam of the metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 family.deleteMetaItem's legacy control-plane branch (code-only rows; no repo path) deletes the row and heals its own registry but never emitted the mutation event even locally — pre-existing and marked deliberate in code ("emits no watch event… pre-existing and deliberate") — so it fans out to no one. Same class as the recovery doors metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 records, but metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 names only rollbackMetaItem/revertCommit. Recommend appending this branch to metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 rather than widening this PR.
  3. Non-blocking — wording. The payload docblock's "at-least-once delivery and duplicates are harmless by construction" states a tolerance, while shipped drivers are at-most-once; a skimming reader could take it for a delivery guarantee. The attach docblock, changeset, and PR body all state the loss bound plainly, so no change is required — noted only so nobody later quotes the payload docblock as a promise.

Also verified

  • The census docs row in this diff is exactly the claimed anchor shift: context?.isSystem in stripReadonlyForInsert measured at protocol.ts:1737 at base and :1741 at head; the docs hunk changes that one number and zero prose. Not scope creep.
  • The write-through caller pin re-opened correctly: measured exactly five this.applyRegistryWriteThrough( sites at head (base: four), the fifth being applyRemoteMetadataMutation; the route-5 case drives the real subscribe path and pins the wire-plural fold.
  • emitMetadataMutation's callers at head are exactly saveMetaItem / runPublishSideEffects / deleteMetaItem — the claimed choke point — and the recovery doors run the write-through without emitting, confirming metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 as a pre-existing gap, not this PR's.
  • CI at head 0cf0a437: 33/33 check runs green (Lint & Repo Gates included). I did not re-run the suites locally; suite verdicts are read from CI and the dev reports, all other claims above are my own measurements.

Verdict: PASS. No blocking findings; no design fork the ruling leaves unsettled. The enqueue-gate decision returns to the dispatching seat.


Generated by Claude Code

Same mechanism as the previous merge: main's #14199 re-anchored 11 rows
of this machine-maintained table while this branch re-anchors row 21,
and a textually clean merge proves nothing about the line numbers.
Resolution per the tool: took main's page wholesale, re-derived with
check-system-context-census --fix on the merged tree — exactly ONE
anchor rewritten (row 21, protocol.ts 1737 to 1741; re-derived, not
carried — main has zero commits on protocol.ts, so the insertion shift
is unchanged). Delta vs origin/main is that single line; bare gate
green (109 sites, 145 anchors). engine-double-contract ledger: zero
main-side commits, gate OK on the merged tree — no regeneration owed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
@os-support-ai
os-support-ai added this pull request to the merge queueSep 1, 2026
Merged via the queue into main with commit 1403d94Sep 1, 2026
34 checks passed
@os-support-ai
os-support-ai deleted the claude/issue-13331-metadata-registry-cluster-fanout branch September 1, 2026 14:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/xlteststooling

Projects

None yet

2 participants

@os-support-ai@claude