Uh oh!
There was an error while loading. Please reload this page.
feat(service-datasource,service-cluster): fan datasource record writes out to peer replicas so a deleted datasource stops draining every replica (#13805) - #14347
Conversation
…s out to peer replicas (#13805) The ObjectQL driver registry had no cluster propagation in either direction: each replica filled it at boot from the shared datasource records and mutated it only for the writes it served, so after DELETE /api/v1/datasources/:name only the serving replica evicted the stuck driver (#13578) and the other N-1 kept draining /api/v1/ready until restart. Ruled shape (2026-09-01): the driver registry adopts the cluster-invalidation family metadata.mutated established -- no second mechanism, no poll loop, no delete-only broadcast. DatasourceAdminService publishes the record's ADDRESS on datasource.mutated after create, update and delete (symmetric); peers converge their live pool from their OWN read of the durable sys_metadata row through the pool seams they already own (build / rebuild-in-place / evict / leave alone). Only IPubSub from spec/contracts crosses the new attach seam; service-datasource takes no cluster dependency and objectql is handed no bus. MetadataClusterBridgePlugin late-binds the seam as a third independent lane, guarded off the in-process memory driver like the other two. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…ow the stray-signal warn assertion Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
📓 Docs Drift CheckThis PR changes 2 package(s): 9 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 2 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 4 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 177460340ecec87f3d9c6271651f88fd10f89ad8 && git checkout 177460340ecec87f3d9c6271651f88fd10f89ad8
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 72adb7fcb6b6f48b656047835ed3852f314d9de1 6be669f67747210071d0d03fda58ef8b7921b537 && git checkout -B drift-repro 72adb7fcb6b6f48b656047835ed3852f314d9de1 && git merge --no-ff 6be669f67747210071d0d03fda58ef8b7921b537
node scripts/docs-audit/affected-docs.mjs --json 72adb7fcb6b6f48b656047835ed3852f314d9de1
|
os-sales
commented
Sep 2, 2026
Landing provenance (PM seat,
Generated by Claude Code |
Uh oh!
There was an error while loading. Please reload this page.
Fixes#13805
What this does
The ObjectQL driver registry had no cluster propagation in either direction: each replica filled it at boot from the shared datasource records and mutated it only for the writes it served. After
DELETE /api/v1/datasources/:nameonly the replica that served the DELETE evicted the stuck driver (#13578's door, unchanged here); the other N-1 kept it — and/api/v1/readykept answering 503 there — until restart. A create through one replica likewise had no pool anywhere else until restart.Ruled shape (maintainer, 2026-09-01, director batch A — quoted verbatim in the card): the driver registry reuses the cluster-invalidation family #13331-A established; datasource writes (create and delete, symmetric) emit the signal through the same bridge shape; each replica converges by re-reading the shared datasource records; no second propagation mechanism. This PR builds exactly that and nothing beside it:
packages/services/service-datasource/src/datasource-admin-service.ts): aftercreateDatasource,updateDatasourceandremoveDatasource,DatasourceAdminServicepublishes the record's ADDRESS —{ originNode, name }— on a new cluster channeldatasource.mutated(DATASOURCE_MUTATION_CLUSTER_CHANNEL, payloadClusterDatasourceMutationPayload). Fire-and-forget; a publish failure never fails the write.migrateCredentialdoes not publish (it leaves the live pool alone by design, on every replica alike). A delete-only broadcast was deliberately not built — the triage constraint on symmetry.datasource-admin-plugin.ts, newconvergePoolseam onDatasourceAdminServiceConfig): on receipt a replica re-reads the durablesys_metadatarow for that name — the store its boot rehydration reads, NOT its per-replica metadata registry (which on the host-config boot has no cluster seam and is exactly the stale read a peer must not converge from) — and drives its pool through the seams it already owns: build what is missing, rebuild in place what changed (reregisterPool, keeping the old pool on failure exactly as the serving replica's update path does, datasource update never rebuilds its driver —already-registeredshort-circuits the reconfigure path, so the OLD pool stays live and the admin UI reports success #13804), evict what is gone (unregisterPool, the datasource DELETE does not evict the stuck driver from the data-engine driver registry — /ready keeps naming a datasource that no longer exists, recoverable only by process restart #13578 door), leave a matching pool untouched. The record each live pool was built from is remembered by the pool seams themselves (livePoolRecords), which is what makes a duplicate delivery a measured no-op and a stray name harmless: only pools this plugin built are ever touched, so a code-defined pool survives a stray signal. This answers the triage's third question — a replayed create is safe because receipt is convergence, not replay; no new idempotency machinery.DatasourceAdminService.attachDatasourceMutationPubSub(pubsub, nodeId)— idempotent on the(pubsub, nodeId)pair,originNodeloopback suppression, per-name ordered convergence, shaped afterObjectStackProtocolImplementation.attachMetadataMutationPubSub(). OnlyIPubSubfrom@objectstack/spec/contractscrosses it:@objectstack/service-datasourcetakes no dependency on the cluster service, and@objectstack/objectqlis handed no bus (the layering line triage drew — measured:service-clusterdepends on core + spec only;service-datasourceon core/spec/types/drivers; neither on the other).MetadataClusterBridgePlugin(packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts): late-binds the seam atkernel:readybeside the metadata-service and protocol lanes, duck-typed on thedatasource-adminservice, guarded off the in-processmemorydriver from birth. A lane rather than a fourth plugin because that is how runtime: TS-config boot registers a 'metadata' service without attachClusterPubSub — cross-node invalidation disabled; new object gives OBJECT_NOT_FOUND on non-writing replicas, never heals #13331-A itself adopted the family (lane 2 was added to this plugin), it needs noRuntimewiring change, and it adds no exported symbol toservice-cluster.Not touched, per the card's constraints: the
/api/v1/meta/datasourcemetadata registry (#13609 / #13405 — different sink; the peer's metadata registry is deliberately NOT written by convergence, pinned) and #13578's eviction primitive itself. Log levels follow the exemplars' sites one-for-one (debug for skips, info for the bridged/attached lines, warn for publish/converge failures, the exemplar's error for attach/detach exceptions); no newerrorsink.Verification
Base:
909a4417(origin/main at claim). Final head:6be669f6. Dependency closure built in the worktree first (pnpm --filter '@objectstack/service-datasource^...' build, VERDICT command-exit 0), then both packages built.pnpm --filter @objectstack/service-datasource typecheckat6be669f6: exit 0, 0error TS;tsc --listFilescounts the new test file once (its tsconfig includes tests).pnpm --filter @objectstack/service-cluster exec tsc --noEmit: the package's single pre-existing error (src/memory/memory.contract.test.ts:26, untouched by this branch, onorigin/mainsince skills(pm): give sweep disjunct ① the drain clauses the finding box already has #14152) and nothing else — its DEBT ledger count holds.6be669f6, both full suites under the shared verify lock (os-verify-lock.shVERDICT command-exit 0):@objectstack/service-cluster7 files / 105 tests passed;@objectstack/service-datasource30 files / 635 tests passed.src/__tests__/datasource-cluster-convergence.test.ts(21 cases): two REALDatasourceAdminServicePluginboots over one sharedsys_metadatastore, own driver registries, joined by a remote-shaped bus. Arm B: DELETE on the writer evicts on the peer; create builds on the peer from the peer's own read; connectivity change rebuilds in place; active:false / active:true. Arm A controls (no attach): the identical DELETE leaves the peer's driver in place, the identical create leaves it pool-less. Payload address-only for all three doors (symmetric); duplicate delivery no-op by instance identity; label-only edit no churn; ordered convergence for a create chased by an update; stray signal touches nothing (code-defined pool survives); peer rebuild failure keeps the old pool; unreadable row is not spent as "gone"; publish failure never fails the write; detach; late-booting replica converges via rehydration; loopback; malformed payload; idempotent attach; no-convergePool host; migrateCredential does not publish.metadata-cluster-bridge-plugin.test.tsgains 8 lane-3 cases: attaches on a cross-process driver and logs the "bridged datasource.mutated" line verbatim, independently of lanes 1 and 2; memory driver skips (in-process guard, nothing above debug); absent / bare service skip; no cluster service skips; throwing attach reported without taking the other lanes down; shutdown detaches lane 3 even when lane 2's detach throws.publishDatasourceMutationcall fromremoveDatasourceviaperl -0piand PROVED it on disk (grep -cof the call: 2 → 1; marker: 0 → 1), ran the convergence file under the lock: exactly the two declared cases red — "Arm B — DELETE on the writer evicts the driver on the PEER" and the symmetric-payload case — 19 green (VERDICT command-exit 1). Restored withgit checkout HEAD -- ABSOLUTE_PATHunder atrap … EXIT INT TERM; restore proved bygit hash-objectequal to theHEAD:blob andgit diff HEADempty. No dist involved: the test imports the package's ownsrcdirectly.node scripts/pm/dispatch-gates.mjs --commands, 42 commands, 35 families), every one run unlocked with the exit captured before any pipe. Green (exit 0): adr-0087-registration, changeset-no-major, ci-filter-parity, comment-mask-adoption, cross-package-test-inputs (both spellings), empty-changeset, keyed-text-bounds, plugin-teardown-shape, shard-attestation, tenant-audit-census, undeclared-dep-imports, docs-audit affected-docs + drift-comment, release-rehearsal-clone self-test, agent-test-spelling, bash32-floor, changeset-gate-self-tests, cli-command-ids, dispatcher-error-vocabulary, doc-authoring, engine-double-contract (the new test's findOne/update/delete doubles route through the metadata-core asserts and were recorded via the gate's own--write, +3 rows, 0 lost), entry-guard, logger-receiver-detach, objectql-double-limit, objectui-changeset, page-declaration-shape, parse-guard, pm-half-states, pnpm-filter-targets, published-files, query-options-erasure, slot-lookup, test-source-alias, type-check-coverage, type-source-resolution, watch-hint-literal, where-matcher, nul-bytes. NOT MEASURED locally (their own exit 3 / prerequisite text, not red):check:type-check-debtandcheck:dual-build-cjs-loadsneed the whole workspace built (the closure build alone took 4 minutes under the lock; the full one does not fit the container's foreground cap),check-test-completenessparses a CI vitest summary,scripts/pm/check-half-states.mjsis the PM patrol (network, timed out). CI runs all four.pnpm exec eslint --no-inline-config --format jsonover the 6 changed TypeScript files → 6 files, 0 errors, 0 warnings. Population evidence from the config itself: the only global ignores areNEVER_LINTED(node_modules/dist/build/.next/.turbo), none of which contain these paths; invariance:eslint.config.mjsstates it never enables type-aware linting (noparserOptions.project, no typed rules) for any file, so this diff cannot move any untouched file's verdict.Contract review (Clause ②)
New exported symbols, from
git diff -U0 origin/main...HEAD | grep export:DATASOURCE_MUTATION_CLUSTER_CHANNEL,ClusterDatasourceMutationPayload(both from@objectstack/service-datasource), plus two public methods on the exportedDatasourceAdminServiceclass (attachDatasourceMutationPubSub/detachDatasourceMutationPubSub) and one optional key on the exportedDatasourceAdminServiceConfig(convergePool).@objectstack/service-clusterexports nothing new (a lane, not a plugin). Changeset:minorfor both packages.Out of scope, filed unassigned: #14339 (docs:
content/docs/kernel/cluster.mdxdescribes the invalidation family asmetadata.changedonly;metadata.mutatedanddatasource.mutatedare undocumented — one docs-only change for the family, not a rider here).Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
Generated by Claude Code
🤖 Generated with Claude Code
Generated by Claude Code