Uh oh!
There was an error while loading. Please reload this page.
feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas - #14183
Conversation
…ns out to peer replicas (#13331) Publisher at the protocol's post-persistence choke point publishes the mutation's address on the new metadata.mutated cluster channel; peers converge their ObjectQL registry from their OWN sys_metadata read (write- through when an active row exists, the delete heal walk when none does), then replay the event into local onMetadataMutation listeners. The bridge plugin late-binds attachMetadataMutationPubSub as a second independent lane at kernel:ready, guarded off the in-process memory driver. Ruled 2026-09-01 (director batch A, Option A): producer-side fan-out mirroring the shipped AuthzClusterBridgePlugin shape; the payload is a signal, never trusted content. Option B (consumer-side self-heal) is not built. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
…ouble; ledger the new pinned engine doubles check:objectql-double-limit named the new find double limit-blind — the bound now applies after the filter, by presence. The engine-double ledger learned the fanout file's pinned delete/findOne/update doubles via the gate's own --write. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
📓 Docs Drift CheckThis PR changes 2 package(s): 5 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 3 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 11 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 1304ec033ce539d92069afd7ac66110cc1951b11 && git checkout 1304ec033ce539d92069afd7ac66110cc1951b11
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin e62129153f2aa3d7666eab9c1af2bdb18e7333fc 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec && git checkout -B drift-repro e62129153f2aa3d7666eab9c1af2bdb18e7333fc && git merge --no-ff 7ffeb7269ce7b579b955ac9b1ccb8aad741913ec
node scripts/docs-audit/affected-docs.mjs --json e62129153f2aa3d7666eab9c1af2bdb18e7333fc
|
…he fan-out insertion check-system-context-census red on CI: the metadata.mutated publisher block inserted above stripReadonlyForInsert moved the context.isSystem read 1737 -> 1741, and the census page still anchored the old line. The gate's own --fix performed the shift (one line-number rewrite, zero prose); bare re-run green: 109 sites anchored, 145 anchors resolve. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
…tadata-registry-cluster-fanout
The auto-merge of origin/main kept this branch's pre-#14171 anchor rows beside the row-21 edit, flunking the census 26 ways (paired stale anchors / unanchored sites in engine.ts and share-link-service.ts — pure adjacent-row line rot, zero population change). Resolution per the tool, not by hand: took main's page wholesale, then re-derived with check-system-context-census --fix, which rewrote exactly ONE anchor (row 21, protocol.ts 1737 to 1741 — this branch's publisher-block shift). Delta vs origin/main is that single line; bare gate green: 109 sites anchored, 145 anchors resolve. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
os-support-ai
commented
Sep 1, 2026
Contract review — CONTRACT_REVIEW_TIER — verdict: PASSDelegated review (dispatching PM seat Instruments. The diff was measured through the API ( Clause-② limbs, judged from the diffContent limb: YES — declared honestly and completely. The minted public surface, measured:
Path limb: NO — measured, not taken from the author. From the complete API file list: zero paths under
Is the contract minimal and right-shaped?Two lanes are correct, not a duplicate. Measured, all at head:
The payload is genuinely address-only, and the receiver genuinely re-reads. Attach/detach shape. Idempotent on the Scope parity of the receipt. The write-through's env gate and org-scope verdict ( Failure directions
Beyond-minimum: the local listener replay — sound, and in scopeThe ruling's clause 1 orders the shipped bridge shape mirrored, and the shipped template itself ships remote replay: Placement: |
…tadata-registry-cluster-fanout
Same mechanism as the previous merge: main's #14199 re-anchored 11 rows of this machine-maintained table while this branch re-anchors row 21, and a textually clean merge proves nothing about the line numbers. Resolution per the tool: took main's page wholesale, re-derived with check-system-context-census --fix on the merged tree — exactly ONE anchor rewritten (row 21, protocol.ts 1737 to 1741; re-derived, not carried — main has zero commits on protocol.ts, so the insertion shift is unchanged). Delta vs origin/main is that single line; bare gate green (109 sites, 145 anchors). engine-double-contract ledger: zero main-side commits, gate OK on the merged tree — no regeneration owed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5
Uh oh!
There was an error while loading. Please reload this page.
Fixes#13331
What this ships
A runtime-authored metadata mutation now reaches every replica's ObjectQL registry, not just the writer's. Measured defect on the shipped 3-replica EE compose (ADR-0018):
PUT /api/v1/meta/object/...persisted to the sharedsys_metadata(meta reads 200 fleet-wide) while only the writing replica registered the object —/api/v1/data/...answered a hard 404OBJECT_NOT_FOUNDon the other two, indefinitely; 200 concurrent creates through the LB gave 67x201 / 133x404 with a boot-loaded control object at 0 errors.Mechanism (maintainer-ruled 2026-09-01, director batch A, Option A — verbatim adoption of the escalation's recommendation):
emitMetadataMutation, the seamonMetadataMutationsubscribes — reached bysaveMetaItem,runPublishSideEffects,deleteMetaItem) now also publishes the mutation's ADDRESS on a new cluster channelmetadata.mutated(METADATA_MUTATION_CLUSTER_CHANNEL, payloadClusterMetadataMutationPayload, both exported from@objectstack/metadata-protocol). Drafts are never published — they enter no registry anywhere.sys_metadata: active row present ⇒ re-runapplyRegistryWriteThrough(package binding derived the way the recovery callers derive it); no active row ⇒ run the delete heal walk (restoreArtifactRegistryView). The payload is a signal, never trusted content; duplicates and out-of-order delivery converge to the row's current state by construction. After convergence the event replays into the replica's LOCALonMetadataMutationlisteners (never re-published), so authored hook/action re-binds re-sync on peers.ObjectStackProtocolImplementation.attachMetadataMutationPubSub(pubsub, nodeId)— idempotent on the pair, loopback-suppressed viaoriginNode, mirroringMetadataManager.attachClusterPubSuband the engine'sattachAuthzInvalidationPubSub.MetadataClusterBridgePluginlate-binds it atkernel:readyas a second, independent lane: the boot shape that lacks a manager-backedmetadataservice (TS-config host-config — exactly the shipped EE shape) is the one that needs this lane most. The new lane skips the in-process memory driver from birth (the guard the authz sibling carries).assertObjectRegistered) is not built — presented only as a possible stopgap; the maintainer did not order one.The two fences, both held
packages/objectql/src/engine.ts: untouched (read-only on this card; held by PR Refuse an undeclared field a before-hook writes — the post-hook half of the declared-field door, one envelope on every driver #13864 at dispatch). Measured answer to the state-owner question: the state that goes stale is the peer's registry write-through, whose choke point AND the DB read the ruled receipt needs both live in the protocol — the engine template (attachAuthzInvalidationPubSub) was mirrored in shape, not in location.packages/spec: untouched, and no split is owed. The payload lives in@objectstack/metadata-protocol, matching both shipped precedents (the authz payload in core's channel module;ClusterMetadataChangedPayloadin the metadata package);IPubSubfromspec/contractsis the already-shipped generic transport. The spec's dormantMetadataChangedEventPayloadSchemawas measured at zero producers and zero consumers in-tree, and its declared version-compare protocol is the opposite of the ruled re-read receipt — filed as [finding] spec: MetadataChangedEventPayloadSchema says every metadata persistence layer MUST emit it — zero producers or consumers in-tree, and its bigint version field cannot cross JSON #14180 for the spec lane.Evidence
protocol.cluster-mutation-fanout.test.ts, two replicas over one shared store): Arm B (attached) — the peer converges from its own read; Arm A (control, no bridge) — the identical write leaves the peer empty, so Arm B is the bridge's doing, not a harness artifact. Also pinned: address-only payload (key set equality), draft silence, loopback suppression, duplicate-delivery convergence, re-attach idempotency, detach, delete fan-out, draft-discard keeping the peer's active registration, and listener replay ordered after registry convergence.git diff HEADplus marker count 0; restored run 22/22 green. Resolution note: these tests import./protocol.jsrelative from src — the subject never resolves through dist, so no rebuild leg exists to skip; the on-disk grep is the falsifiable observation for both legs.tsc --noEmitclean with both edited test files confirmed inside the program via--listFiles(2 hits).node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands; provenance line names this repo at this commit and the --repo assertion holds): 44 derived; 41 exit 0; 3 NOT MEASURED by their own printed verdicts (check-test-completeness exit 3 — grades a CI turbo log; check:dual-build-cjs-loads exit 3 — needs the full workspace build; check:type-check-debt exit 3 — full re-measure). Two first-pass reds were fixed and re-run to their own OK lines: check:engine-double-contract ("OK — 741 pinned, 134 in the DEBT ledger, 3 exempt"; ledger learned the fanout file's pinned doubles via the gate's own --write) and check:objectql-double-limit ("conformance holds: 299 doubles graded"; the new find double now holds the caller's bound by presence).applyRegistryWriteThroughgrew its FIFTH caller (the peer applier). The trace, the count pin, and a new route-5 spelling case (a plural delivered over the wire registers under the singular; fold viacanonicalMetaType, the complete map) are updated inprotocol.object-registry-write-through-spelling.test.ts.Notes for contract review
packages/specuntouched, engine untouched).metadata.changed) behavior is byte-identical, warn line included — that line stays TRUE on host-config boots (the metadata SERVICE still has no cluster seam there; what changed is that the data-plane registry no longer depends on it). [finding]MetadataClusterBridgePluginlogs "bridged metadata.changed → cluster.pubsub" on the in-processmemorydriver, which fans out to nobody — its siblingAuthzClusterBridgePluginguards exactly this, and the metadata bridge has no test file at all #14021's lane-1 in-process misreading is deliberately NOT absorbed; the new lane carries the guard from birth.onMetadataMutationlisteners (never re-published) — the same remote-replay shapeMetadataManager.attachClusterPubSubships for watchers, and what makes authored hook/action re-binds converge on peers.rollbackMetaItem/revertCommit) run the write-through but never emitted the mutation event even locally, so they fan out to no one either — pre-existing seam gap, filed as metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 (Blocked-by this card).index.ts, so the anticipated textual conflict with PR fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 does not arise; measured zero file overlap with everything landed onmainsince the branch point (2bff79a).IPubSubcontract): a lost signal degrades to the pre-existing staleness bound — heal at next boot. This channel narrows the window from "until restart" to one network hop; it does not promise more.CI rework — census re-anchor (cdbe5f0)
Lint & Repo Gatesred at e460193: check-system-context-census — pure line rot from this PR's own insertion (the publisher block sits abovestripReadonlyForInsert, moving itscontext.isSystemread from protocol.ts:1737 to :1741 whilecontent/docs/permissions/system-context.mdxrow 21 still anchored 1737). Repaired with the gate's own--fixand VERIFIED as a SHIFT, not a population change: the rewrite touches exactly one line — the anchor's line number — and zero prose, and the census population is unchanged (109 sites / 145 anchors before and after). The docs file joining this diff is that re-anchor, not scope creep. Reproduced red locally first, then the bare gate re-run green: "OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read."Docs-drift assessment (stated, not rewritten, per dispatch):
content/docs/kernel/cluster.mdx§6.2 is now INCOMPLETE — it says cross-node metadata invalidation "already works" through the manager'smetadata.changedlane alone, which is exactly the account #13331 falsified for the runtime-authoring path (the protocol never reaches the manager, and host-config boots carry no manager at all). The page wants a paragraph on themetadata.mutatedprotocol lane and the bridge's second attach; its own closing principle there ("the peer re-reads the shared store, which is the source of truth") is the rule the new receipt implements, and its "Target spec (planned)" block already marksMetadataChangedEventPayloadSchemaas not wired — corroborating #14180. Same incompleteness class applies to the §5/§7 bridge mentions andcontent/docs/concepts/metadata-lifecycle.mdx.Generated by Claude Code
Generated by Claude Code
Generated by Claude Code