feat(service-datasource,service-cluster): fan datasource record writes out to peer replicas so a deleted datasource stops draining every replica (#13805)

.changeset/datasource-mutation-cluster-fanout.md

@@ -0,0 +1,63 @@
+---
+"@objectstack/service-datasource": minor
+"@objectstack/service-cluster": minor
+---
+
+feat(service-datasource,service-cluster): fan datasource record writes out to peer replicas — a deleted datasource no longer keeps draining `/api/v1/ready` on every replica that did not serve the DELETE (#13805)
+
+Measured on a live 3-replica EE deployment: the ObjectQL DRIVER registry had
+no cluster propagation in either direction. Each replica filled it at boot
+from the shared datasource records and mutated it only for the writes IT
+served, so after `DELETE /api/v1/datasources/:name` only the replica that
+served the DELETE evicted the stuck driver (#13578's door) — the other N-1
+kept it, and `/api/v1/ready` kept answering 503 there, until restart. A
+datasource created through one replica likewise had no pool on any other
+until restart.
+
+Maintainer-ruled design (2026-09-01): the driver registry adopts the same
+cluster-invalidation family `metadata.mutated` (#13331) established — no
+second propagation mechanism, no bespoke poll loop, and no delete-only
+broadcast (that would have made delete more cluster-aware than create, a new
+asymmetry rather than a repair).
+
+- **Symmetric publisher at the three write doors.** `DatasourceAdminService`
+ now publishes the record's ADDRESS on a new cluster channel
+ `datasource.mutated` (`DATASOURCE_MUTATION_CLUSTER_CHANNEL`, payload
+ `ClusterDatasourceMutationPayload` — `{ originNode?, name }`) after
+ `createDatasource`, `updateDatasource` and `removeDatasource`. Fire-and-
+ forget: a publish failure never fails the write it announces.
+ `migrateCredential` does not publish — it leaves the live pool alone by
+ design, on every replica alike.
+- **Peers converge from their own read of the SHARED record.** On receipt a
+ replica re-reads the durable `sys_metadata` row for that name — the same
+ store its boot rehydration reads, not its per-replica metadata registry —
+ and converges its live pool through the seams it already owns: builds what
+ is missing, rebuilds in place what changed (`reregisterPool`, keeping the
+ old pool on failure exactly as the serving replica's update path does),
+ evicts what is gone (`unregisterPool` → the #13578 eviction door), and
+ leaves a matching pool untouched. The payload is a signal, never trusted
+ content, so a duplicate or re-ordered delivery converges to the same pool
+ state by construction — which is what makes a replayed create safe without
+ any new idempotency machinery. A name the replica never pooled is left
+ alone, so a stray signal cannot reach a code-defined pool.
+- **New attach seam, mirrored from the shipped bridges.**
+ `DatasourceAdminService.attachDatasourceMutationPubSub(pubsub, nodeId)` —
+ idempotent on the `(pubsub, nodeId)` pair, loopback suppression via
+ `originNode`, shaped after the protocol's `attachMetadataMutationPubSub()`.
+ Only `IPubSub` from `@objectstack/spec/contracts` crosses it:
+ `@objectstack/service-datasource` takes no dependency on the cluster
+ service, and `@objectstack/objectql` — the registry's owner — is handed no
+ bus. The host wires the receive half through a new optional
+ `DatasourceAdminServiceConfig.convergePool` seam; `DatasourceAdminServicePlugin`
+ supplies it.
+- **`MetadataClusterBridgePlugin` gains a third, independent lane** that
+ late-binds the seam at `kernel:ready` beside the metadata-service and
+ protocol lanes, duck-typed on the `datasource-admin` service. It skips the
+ in-process memory driver (nothing to fan out to), the guard the other lanes
+ carry, so a single-replica boot behaves byte-identically to before.
+
+No shipped driver exceeds at-most-once delivery, so a lost message still
+degrades to the pre-existing bound (the next boot's full rehydration); this
+channel narrows the window from "until every replica restarts" to one network
+hop. The `/api/v1/meta/datasource` metadata registry's own cross-replica
+coherence (#13609) is a different sink and is not touched here.

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.test.ts

@@ -48,17 +48,34 @@ interface HarnessOptions {
  * `'real'` (exposes attachMetadataMutationPubSub).
  */
  protocol?: 'none' | 'bare' | 'real';
+ /**
+ * [#13805] The `datasource-admin` slot: `'none'` (getService throws — the
+ * default, so the #13331 cases above read exactly as they did), `'bare'`
+ * (present, no attachDatasourceMutationPubSub — an older implementation),
+ * or `'real'` (exposes attachDatasourceMutationPubSub).
+ */
+ datasourceAdmin?: 'none' | 'bare' | 'real';
+ /** When true, the datasource-admin seam throws on attach. */
+ datasourceAttachThrows?: boolean;
 }
 
 function makeHarness(opts: HarnessOptions = {}) {
- const { driver = 'redis', metadata = 'fallback', protocol = 'real' } = opts;
+ const {
+ driver = 'redis', metadata = 'fallback', protocol = 'real',
+ datasourceAdmin = 'none', datasourceAttachThrows = false,
+ } = opts;
 
  const logger = { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() };
 
  const detachMetadata = vi.fn();
  const attachMetadata = vi.fn((_pubsub: unknown, _nodeId: string) => detachMetadata);
  const detachMutation = vi.fn();
  const attachMutation = vi.fn((_pubsub: unknown, _nodeId: string) => detachMutation);
+ const detachDatasource = vi.fn();
+ const attachDatasource = vi.fn((_pubsub: unknown, _nodeId: string) => {
+ if (datasourceAttachThrows) throw new Error('datasource attach exploded');
+ return detachDatasource;
+ });
 
  const pubsub = { publish: vi.fn(), subscribe: vi.fn(), close: vi.fn() };
  const cluster =
@@ -74,6 +91,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  protocol === 'none' ? undefined
  : protocol === 'bare' ? { saveMetaItem: vi.fn() }
  : { attachMetadataMutationPubSub: attachMutation };
+ const datasourceAdminService =
+ datasourceAdmin === 'none' ? undefined
+ : datasourceAdmin === 'bare' ? { listDatasources: vi.fn() }
+ : { attachDatasourceMutationPubSub: attachDatasource };
 
  const hooks = new Map<string, Array<() => Promise<void> | void>>();
  const ctx = {
@@ -96,6 +117,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  if (!protocolService) throw new Error('service not found: protocol');
  return protocolService;
  }
+ if (name === 'datasource-admin') {
+ if (!datasourceAdminService) throw new Error('service not found: datasource-admin');
+ return datasourceAdminService;
+ }
  throw new Error(`service not found: ${name}`);
  },
  } as unknown as PluginContext;
@@ -107,6 +132,7 @@ function makeHarness(opts: HarnessOptions = {}) {
  return {
  ctx, logger, fire, pubsub,
  attachMetadata, detachMetadata, attachMutation, detachMutation,
+ attachDatasource, detachDatasource,
  };
 }
 
@@ -280,3 +306,109 @@ describe('[#14021] lane 1 — an in-process bus must not be reported as “bridg
  expect(h.logger.error).not.toHaveBeenCalled();
  });
 });
+
+describe('[#13805] lane 3 — the datasource admin service’s datasource.mutated fan-out', () => {
+ it('⭐ attaches on a cross-process driver and reports it — independently of lanes 1 and 2', async () => {
+ // The shipped EE shape again, one owner over: no manager-backed
+ // metadata slot, no protocol seam, and a real datasource-admin service.
+ // Lane 3 must attach exactly there, with nothing from the other two
+ // lanes taking it down.
+ const h = makeHarness({ driver: 'redis', metadata: 'none', protocol: 'none', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ // Asserted VERBATIM, like lane 1's and lane 2's lines: the wording is
+ // what an operator reads as "datasource fan-out is on".
+ expect(infoLines(h)).toContain(
+ 'MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=node-a)',
+ );
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('all three lanes attach together when every owner exposes its seam', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachMutation).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(warnLines(h)).toEqual([]);
+ });
+
+ it('skips attach on the in-process memory driver — no peers to reach, nothing said above debug', async () => {
+ const h = makeHarness({ driver: 'memory', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ // The guard lanes 1 and 2 carry, from birth: on the memory driver a
+ // single replica's behaviour stays byte-identical to the pre-bridge
+ // one — no subscription, no publisher, no "bridged" claim.
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ expect(
+ debugLines(h).some((l) => l.includes('is in-process') && l.includes('datasource fan-out')),
+ ).toBe(true);
+ });
+
+ it('skips quietly when no datasource-admin service is registered', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'none' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ expect(warnLines(h).some((l) => l.includes('datasource'))).toBe(false);
+ });
+
+ it('skips quietly when the service does not expose the seam', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'bare' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('no cluster service at all skips lane 3 too', async () => {
+ const h = makeHarness({ driver: null, datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ });
+
+ it('a throwing attach is reported and does not take the other lanes down', async () => {
+ const h = makeHarness({
+ driver: 'redis', metadata: 'manager', protocol: 'real',
+ datasourceAdmin: 'real', datasourceAttachThrows: true,
+ });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledTimes(1);
+ expect(h.attachMutation).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalledWith(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ expect.any(Error),
+ );
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ });
+
+ it('kernel:shutdown detaches lane 3, and a throwing lane-2 detach does not strand it', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ h.detachMutation.mockImplementation(() => { throw new Error('detach exploded'); });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+ await h.fire('kernel:shutdown');
+
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalled();
+
+ // Idempotent: a second shutdown does not detach twice.
+ await h.fire('kernel:shutdown');
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ });
+});

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts

@@ -15,8 +15,8 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * state-owner packages only need the `IPubSub` interface, which lives in
  * `@objectstack/spec/contracts`.
  *
- * TWO lanes, late-bound independently at `kernel:ready`, because the state
- * that goes stale lives in two different owners (#13331):
+ * THREE lanes, late-bound independently at `kernel:ready`, because the state
+ * that goes stale lives in three different owners (#13331, #13805):
  *
  * 1. **Metadata service** (`attachClusterPubSub()` — `metadata.changed`):
  * replays watch events into peer `MetadataManager` caches
@@ -34,6 +34,15 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * 67×201 / 133×404). The lanes are independent on purpose: the boot
  * shape that lacks lane 1 (host-config, fallback metadata slot) is
  * exactly the shipped EE shape that needs lane 2.
+ * 3. **Datasource admin service** (`attachDatasourceMutationPubSub()` —
+ * `datasource.mutated`): fans a datasource create / update / delete out
+ * to peers, which converge their ObjectQL DRIVER registry from their OWN
+ * read of the shared datasource record. Lane 2's family, adopted by the
+ * driver registry (#13805, ruled 2026-09-01 — the same bridge shape, a
+ * symmetric signal, no second propagation mechanism): without it a
+ * `DELETE /api/v1/datasources/:name` recovered `/api/v1/ready` on the
+ * one replica that served it, and every other replica kept the stuck
+ * driver until restart.
  *
  * Activates each lane only when the cluster service and that lane's state
  * owner are present and expose the seam. Late binding is achieved via the
@@ -42,7 +51,9 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * Channels: `metadata.changed` — payload shape defined by
  * `ClusterMetadataChangedPayload` in `@objectstack/metadata`;
  * `metadata.mutated` — payload shape defined by
- * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`.
+ * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`;
+ * `datasource.mutated` — payload shape defined by
+ * `ClusterDatasourceMutationPayload` in `@objectstack/service-datasource`.
  *
  * See `content/docs/kernel/cluster.mdx` §5.
  */
@@ -53,6 +64,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
 
  private detach?: () => void;
  private detachMutation?: () => void;
+ private detachDatasource?: () => void;
 
  async init(ctx: PluginContext): Promise<void> {
  ctx.hook('kernel:ready', async () => {
@@ -67,6 +79,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
  }
  this.attachMetadataServiceLane(ctx, cluster);
  this.attachProtocolLane(ctx, cluster);
+ this.attachDatasourceLane(ctx, cluster);
  });
 
  ctx.hook('kernel:shutdown', async () => {
@@ -88,6 +101,15 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  this.detachMutation = undefined;
+ try {
+ this.detachDatasource?.();
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane detach error',
+ err as Error,
+ );
+ }
+ this.detachDatasource = undefined;
  });
  }
 
@@ -218,4 +240,61 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  }
+
+ /**
+ * Lane 3 — the datasource ADMIN SERVICE's `datasource.mutated` fan-out
+ * (#13805): the driver registry adopting the family lane 2 established.
+ *
+ * Duck-typed exactly like lanes 1 and 2 feature-detect their seams: this
+ * package must not depend on `@objectstack/service-datasource`, and
+ * `@objectstack/objectql` — the driver registry's owner — is handed no
+ * bus at all; the admin service publishes on the write doors it already
+ * owns and converges its pools through the seams it already injects.
+ *
+ * Guarded on {@link isInProcessClusterDriver} from birth, like lane 2: the
+ * in-process memory driver fans out to nobody, and on that driver a single
+ * replica's behaviour stays byte-identical to the pre-bridge one.
+ */
+ private attachDatasourceLane(ctx: PluginContext, cluster: IClusterService): void {
+ let admin: unknown;
+ try {
+ admin = ctx.getService<unknown>('datasource-admin');
+ } catch {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: no "datasource-admin" service registered, skipping datasource fan-out',
+ );
+ return;
+ }
+
+ const attach = (admin as { attachDatasourceMutationPubSub?: unknown })
+ .attachDatasourceMutationPubSub;
+ if (typeof attach !== 'function') {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: datasource-admin service does not expose attachDatasourceMutationPubSub(), skipping datasource fan-out',
+ );
+ return;
+ }
+
+ if (isInProcessClusterDriver(cluster.driver)) {
+ ctx.logger.debug(
+ `MetadataClusterBridgePlugin: cluster driver "${cluster.driver}" is in-process; datasource fan-out has no peers to reach, skipping`,
+ );
+ return;
+ }
+
+ try {
+ this.detachDatasource = (attach as (
+ pubsub: IClusterService['pubsub'],
+ nodeId: string,
+ ) => () => void).call(admin, cluster.pubsub, cluster.nodeId);
+ ctx.logger.info(
+ `MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=${cluster.nodeId})`,
+ );
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ err as Error,
+ );
+ }
+ }
 }

.changeset/datasource-mutation-cluster-fanout.md

@@ -0,0 +1,63 @@
+---
+"@objectstack/service-datasource": minor
+"@objectstack/service-cluster": minor
+---
+
+feat(service-datasource,service-cluster): fan datasource record writes out to peer replicas — a deleted datasource no longer keeps draining `/api/v1/ready` on every replica that did not serve the DELETE (#13805)
+
+Measured on a live 3-replica EE deployment: the ObjectQL DRIVER registry had
+no cluster propagation in either direction. Each replica filled it at boot
+from the shared datasource records and mutated it only for the writes IT
+served, so after `DELETE /api/v1/datasources/:name` only the replica that
+served the DELETE evicted the stuck driver (#13578's door) — the other N-1
+kept it, and `/api/v1/ready` kept answering 503 there, until restart. A
+datasource created through one replica likewise had no pool on any other
+until restart.
+
+Maintainer-ruled design (2026-09-01): the driver registry adopts the same
+cluster-invalidation family `metadata.mutated` (#13331) established — no
+second propagation mechanism, no bespoke poll loop, and no delete-only
+broadcast (that would have made delete more cluster-aware than create, a new
+asymmetry rather than a repair).
+
+- **Symmetric publisher at the three write doors.** `DatasourceAdminService`
+ now publishes the record's ADDRESS on a new cluster channel
+ `datasource.mutated` (`DATASOURCE_MUTATION_CLUSTER_CHANNEL`, payload
+ `ClusterDatasourceMutationPayload` — `{ originNode?, name }`) after
+ `createDatasource`, `updateDatasource` and `removeDatasource`. Fire-and-
+ forget: a publish failure never fails the write it announces.
+ `migrateCredential` does not publish — it leaves the live pool alone by
+ design, on every replica alike.
+- **Peers converge from their own read of the SHARED record.** On receipt a
+ replica re-reads the durable `sys_metadata` row for that name — the same
+ store its boot rehydration reads, not its per-replica metadata registry —
+ and converges its live pool through the seams it already owns: builds what
+ is missing, rebuilds in place what changed (`reregisterPool`, keeping the
+ old pool on failure exactly as the serving replica's update path does),
+ evicts what is gone (`unregisterPool` → the #13578 eviction door), and
+ leaves a matching pool untouched. The payload is a signal, never trusted
+ content, so a duplicate or re-ordered delivery converges to the same pool
+ state by construction — which is what makes a replayed create safe without
+ any new idempotency machinery. A name the replica never pooled is left
+ alone, so a stray signal cannot reach a code-defined pool.
+- **New attach seam, mirrored from the shipped bridges.**
+ `DatasourceAdminService.attachDatasourceMutationPubSub(pubsub, nodeId)` —
+ idempotent on the `(pubsub, nodeId)` pair, loopback suppression via
+ `originNode`, shaped after the protocol's `attachMetadataMutationPubSub()`.
+ Only `IPubSub` from `@objectstack/spec/contracts` crosses it:
+ `@objectstack/service-datasource` takes no dependency on the cluster
+ service, and `@objectstack/objectql` — the registry's owner — is handed no
+ bus. The host wires the receive half through a new optional
+ `DatasourceAdminServiceConfig.convergePool` seam; `DatasourceAdminServicePlugin`
+ supplies it.
+- **`MetadataClusterBridgePlugin` gains a third, independent lane** that
+ late-binds the seam at `kernel:ready` beside the metadata-service and
+ protocol lanes, duck-typed on the `datasource-admin` service. It skips the
+ in-process memory driver (nothing to fan out to), the guard the other lanes
+ carry, so a single-replica boot behaves byte-identically to before.
+
+No shipped driver exceeds at-most-once delivery, so a lost message still
+degrades to the pre-existing bound (the next boot's full rehydration); this
+channel narrows the window from "until every replica restarts" to one network
+hop. The `/api/v1/meta/datasource` metadata registry's own cross-replica
+coherence (#13609) is a different sink and is not touched here.

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.test.ts

@@ -48,17 +48,34 @@ interface HarnessOptions {
  * `'real'` (exposes attachMetadataMutationPubSub).
  */
  protocol?: 'none' | 'bare' | 'real';
+ /**
+ * [#13805] The `datasource-admin` slot: `'none'` (getService throws — the
+ * default, so the #13331 cases above read exactly as they did), `'bare'`
+ * (present, no attachDatasourceMutationPubSub — an older implementation),
+ * or `'real'` (exposes attachDatasourceMutationPubSub).
+ */
+ datasourceAdmin?: 'none' | 'bare' | 'real';
+ /** When true, the datasource-admin seam throws on attach. */
+ datasourceAttachThrows?: boolean;
 }
 
 function makeHarness(opts: HarnessOptions = {}) {
- const { driver = 'redis', metadata = 'fallback', protocol = 'real' } = opts;
+ const {
+ driver = 'redis', metadata = 'fallback', protocol = 'real',
+ datasourceAdmin = 'none', datasourceAttachThrows = false,
+ } = opts;
 
  const logger = { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() };
 
  const detachMetadata = vi.fn();
  const attachMetadata = vi.fn((_pubsub: unknown, _nodeId: string) => detachMetadata);
  const detachMutation = vi.fn();
  const attachMutation = vi.fn((_pubsub: unknown, _nodeId: string) => detachMutation);
+ const detachDatasource = vi.fn();
+ const attachDatasource = vi.fn((_pubsub: unknown, _nodeId: string) => {
+ if (datasourceAttachThrows) throw new Error('datasource attach exploded');
+ return detachDatasource;
+ });
 
  const pubsub = { publish: vi.fn(), subscribe: vi.fn(), close: vi.fn() };
  const cluster =
@@ -74,6 +91,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  protocol === 'none' ? undefined
  : protocol === 'bare' ? { saveMetaItem: vi.fn() }
  : { attachMetadataMutationPubSub: attachMutation };
+ const datasourceAdminService =
+ datasourceAdmin === 'none' ? undefined
+ : datasourceAdmin === 'bare' ? { listDatasources: vi.fn() }
+ : { attachDatasourceMutationPubSub: attachDatasource };
 
  const hooks = new Map<string, Array<() => Promise<void> | void>>();
  const ctx = {
@@ -96,6 +117,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  if (!protocolService) throw new Error('service not found: protocol');
  return protocolService;
  }
+ if (name === 'datasource-admin') {
+ if (!datasourceAdminService) throw new Error('service not found: datasource-admin');
+ return datasourceAdminService;
+ }
  throw new Error(`service not found: ${name}`);
  },
  } as unknown as PluginContext;
@@ -107,6 +132,7 @@ function makeHarness(opts: HarnessOptions = {}) {
  return {
  ctx, logger, fire, pubsub,
  attachMetadata, detachMetadata, attachMutation, detachMutation,
+ attachDatasource, detachDatasource,
  };
 }
 
@@ -280,3 +306,109 @@ describe('[#14021] lane 1 — an in-process bus must not be reported as “bridg
  expect(h.logger.error).not.toHaveBeenCalled();
  });
 });
+
+describe('[#13805] lane 3 — the datasource admin service’s datasource.mutated fan-out', () => {
+ it('⭐ attaches on a cross-process driver and reports it — independently of lanes 1 and 2', async () => {
+ // The shipped EE shape again, one owner over: no manager-backed
+ // metadata slot, no protocol seam, and a real datasource-admin service.
+ // Lane 3 must attach exactly there, with nothing from the other two
+ // lanes taking it down.
+ const h = makeHarness({ driver: 'redis', metadata: 'none', protocol: 'none', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ // Asserted VERBATIM, like lane 1's and lane 2's lines: the wording is
+ // what an operator reads as "datasource fan-out is on".
+ expect(infoLines(h)).toContain(
+ 'MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=node-a)',
+ );
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('all three lanes attach together when every owner exposes its seam', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachMutation).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(warnLines(h)).toEqual([]);
+ });
+
+ it('skips attach on the in-process memory driver — no peers to reach, nothing said above debug', async () => {
+ const h = makeHarness({ driver: 'memory', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ // The guard lanes 1 and 2 carry, from birth: on the memory driver a
+ // single replica's behaviour stays byte-identical to the pre-bridge
+ // one — no subscription, no publisher, no "bridged" claim.
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ expect(
+ debugLines(h).some((l) => l.includes('is in-process') && l.includes('datasource fan-out')),
+ ).toBe(true);
+ });
+
+ it('skips quietly when no datasource-admin service is registered', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'none' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ expect(warnLines(h).some((l) => l.includes('datasource'))).toBe(false);
+ });
+
+ it('skips quietly when the service does not expose the seam', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'bare' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('no cluster service at all skips lane 3 too', async () => {
+ const h = makeHarness({ driver: null, datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ });
+
+ it('a throwing attach is reported and does not take the other lanes down', async () => {
+ const h = makeHarness({
+ driver: 'redis', metadata: 'manager', protocol: 'real',
+ datasourceAdmin: 'real', datasourceAttachThrows: true,
+ });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledTimes(1);
+ expect(h.attachMutation).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalledWith(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ expect.any(Error),
+ );
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ });
+
+ it('kernel:shutdown detaches lane 3, and a throwing lane-2 detach does not strand it', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ h.detachMutation.mockImplementation(() => { throw new Error('detach exploded'); });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+ await h.fire('kernel:shutdown');
+
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalled();
+
+ // Idempotent: a second shutdown does not detach twice.
+ await h.fire('kernel:shutdown');
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ });
+});

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts

@@ -15,8 +15,8 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * state-owner packages only need the `IPubSub` interface, which lives in
  * `@objectstack/spec/contracts`.
  *
- * TWO lanes, late-bound independently at `kernel:ready`, because the state
- * that goes stale lives in two different owners (#13331):
+ * THREE lanes, late-bound independently at `kernel:ready`, because the state
+ * that goes stale lives in three different owners (#13331, #13805):
  *
  * 1. **Metadata service** (`attachClusterPubSub()` — `metadata.changed`):
  * replays watch events into peer `MetadataManager` caches
@@ -34,6 +34,15 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * 67×201 / 133×404). The lanes are independent on purpose: the boot
  * shape that lacks lane 1 (host-config, fallback metadata slot) is
  * exactly the shipped EE shape that needs lane 2.
+ * 3. **Datasource admin service** (`attachDatasourceMutationPubSub()` —
+ * `datasource.mutated`): fans a datasource create / update / delete out
+ * to peers, which converge their ObjectQL DRIVER registry from their OWN
+ * read of the shared datasource record. Lane 2's family, adopted by the
+ * driver registry (#13805, ruled 2026-09-01 — the same bridge shape, a
+ * symmetric signal, no second propagation mechanism): without it a
+ * `DELETE /api/v1/datasources/:name` recovered `/api/v1/ready` on the
+ * one replica that served it, and every other replica kept the stuck
+ * driver until restart.
  *
  * Activates each lane only when the cluster service and that lane's state
  * owner are present and expose the seam. Late binding is achieved via the
@@ -42,7 +51,9 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * Channels: `metadata.changed` — payload shape defined by
  * `ClusterMetadataChangedPayload` in `@objectstack/metadata`;
  * `metadata.mutated` — payload shape defined by
- * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`.
+ * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`;
+ * `datasource.mutated` — payload shape defined by
+ * `ClusterDatasourceMutationPayload` in `@objectstack/service-datasource`.
  *
  * See `content/docs/kernel/cluster.mdx` §5.
  */
@@ -53,6 +64,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
 
  private detach?: () => void;
  private detachMutation?: () => void;
+ private detachDatasource?: () => void;
 
  async init(ctx: PluginContext): Promise<void> {
  ctx.hook('kernel:ready', async () => {
@@ -67,6 +79,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
  }
  this.attachMetadataServiceLane(ctx, cluster);
  this.attachProtocolLane(ctx, cluster);
+ this.attachDatasourceLane(ctx, cluster);
  });
 
  ctx.hook('kernel:shutdown', async () => {
@@ -88,6 +101,15 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  this.detachMutation = undefined;
+ try {
+ this.detachDatasource?.();
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane detach error',
+ err as Error,
+ );
+ }
+ this.detachDatasource = undefined;
  });
  }
 
@@ -218,4 +240,61 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  }
+
+ /**
+ * Lane 3 — the datasource ADMIN SERVICE's `datasource.mutated` fan-out
+ * (#13805): the driver registry adopting the family lane 2 established.
+ *
+ * Duck-typed exactly like lanes 1 and 2 feature-detect their seams: this
+ * package must not depend on `@objectstack/service-datasource`, and
+ * `@objectstack/objectql` — the driver registry's owner — is handed no
+ * bus at all; the admin service publishes on the write doors it already
+ * owns and converges its pools through the seams it already injects.
+ *
+ * Guarded on {@link isInProcessClusterDriver} from birth, like lane 2: the
+ * in-process memory driver fans out to nobody, and on that driver a single
+ * replica's behaviour stays byte-identical to the pre-bridge one.
+ */
+ private attachDatasourceLane(ctx: PluginContext, cluster: IClusterService): void {
+ let admin: unknown;
+ try {
+ admin = ctx.getService<unknown>('datasource-admin');
+ } catch {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: no "datasource-admin" service registered, skipping datasource fan-out',
+ );
+ return;
+ }
+
+ const attach = (admin as { attachDatasourceMutationPubSub?: unknown })
+ .attachDatasourceMutationPubSub;
+ if (typeof attach !== 'function') {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: datasource-admin service does not expose attachDatasourceMutationPubSub(), skipping datasource fan-out',
+ );
+ return;
+ }
+
+ if (isInProcessClusterDriver(cluster.driver)) {
+ ctx.logger.debug(
+ `MetadataClusterBridgePlugin: cluster driver "${cluster.driver}" is in-process; datasource fan-out has no peers to reach, skipping`,
+ );
+ return;
+ }
+
+ try {
+ this.detachDatasource = (attach as (
+ pubsub: IClusterService['pubsub'],
+ nodeId: string,
+ ) => () => void).call(admin, cluster.pubsub, cluster.nodeId);
+ ctx.logger.info(
+ `MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=${cluster.nodeId})`,
+ );
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ err as Error,
+ );
+ }
+ }
 }

.changeset/datasource-mutation-cluster-fanout.md

@@ -0,0 +1,63 @@
+---
+"@objectstack/service-datasource": minor
+"@objectstack/service-cluster": minor
+---
+
+feat(service-datasource,service-cluster): fan datasource record writes out to peer replicas — a deleted datasource no longer keeps draining `/api/v1/ready` on every replica that did not serve the DELETE (#13805)
+
+Measured on a live 3-replica EE deployment: the ObjectQL DRIVER registry had
+no cluster propagation in either direction. Each replica filled it at boot
+from the shared datasource records and mutated it only for the writes IT
+served, so after `DELETE /api/v1/datasources/:name` only the replica that
+served the DELETE evicted the stuck driver (#13578's door) — the other N-1
+kept it, and `/api/v1/ready` kept answering 503 there, until restart. A
+datasource created through one replica likewise had no pool on any other
+until restart.
+
+Maintainer-ruled design (2026-09-01): the driver registry adopts the same
+cluster-invalidation family `metadata.mutated` (#13331) established — no
+second propagation mechanism, no bespoke poll loop, and no delete-only
+broadcast (that would have made delete more cluster-aware than create, a new
+asymmetry rather than a repair).
+
+- **Symmetric publisher at the three write doors.** `DatasourceAdminService`
+ now publishes the record's ADDRESS on a new cluster channel
+ `datasource.mutated` (`DATASOURCE_MUTATION_CLUSTER_CHANNEL`, payload
+ `ClusterDatasourceMutationPayload` — `{ originNode?, name }`) after
+ `createDatasource`, `updateDatasource` and `removeDatasource`. Fire-and-
+ forget: a publish failure never fails the write it announces.
+ `migrateCredential` does not publish — it leaves the live pool alone by
+ design, on every replica alike.
+- **Peers converge from their own read of the SHARED record.** On receipt a
+ replica re-reads the durable `sys_metadata` row for that name — the same
+ store its boot rehydration reads, not its per-replica metadata registry —
+ and converges its live pool through the seams it already owns: builds what
+ is missing, rebuilds in place what changed (`reregisterPool`, keeping the
+ old pool on failure exactly as the serving replica's update path does),
+ evicts what is gone (`unregisterPool` → the #13578 eviction door), and
+ leaves a matching pool untouched. The payload is a signal, never trusted
+ content, so a duplicate or re-ordered delivery converges to the same pool
+ state by construction — which is what makes a replayed create safe without
+ any new idempotency machinery. A name the replica never pooled is left
+ alone, so a stray signal cannot reach a code-defined pool.
+- **New attach seam, mirrored from the shipped bridges.**
+ `DatasourceAdminService.attachDatasourceMutationPubSub(pubsub, nodeId)` —
+ idempotent on the `(pubsub, nodeId)` pair, loopback suppression via
+ `originNode`, shaped after the protocol's `attachMetadataMutationPubSub()`.
+ Only `IPubSub` from `@objectstack/spec/contracts` crosses it:
+ `@objectstack/service-datasource` takes no dependency on the cluster
+ service, and `@objectstack/objectql` — the registry's owner — is handed no
+ bus. The host wires the receive half through a new optional
+ `DatasourceAdminServiceConfig.convergePool` seam; `DatasourceAdminServicePlugin`
+ supplies it.
+- **`MetadataClusterBridgePlugin` gains a third, independent lane** that
+ late-binds the seam at `kernel:ready` beside the metadata-service and
+ protocol lanes, duck-typed on the `datasource-admin` service. It skips the
+ in-process memory driver (nothing to fan out to), the guard the other lanes
+ carry, so a single-replica boot behaves byte-identically to before.
+
+No shipped driver exceeds at-most-once delivery, so a lost message still
+degrades to the pre-existing bound (the next boot's full rehydration); this
+channel narrows the window from "until every replica restarts" to one network
+hop. The `/api/v1/meta/datasource` metadata registry's own cross-replica
+coherence (#13609) is a different sink and is not touched here.

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.test.ts

@@ -48,17 +48,34 @@ interface HarnessOptions {
  * `'real'` (exposes attachMetadataMutationPubSub).
  */
  protocol?: 'none' | 'bare' | 'real';
+ /**
+ * [#13805] The `datasource-admin` slot: `'none'` (getService throws — the
+ * default, so the #13331 cases above read exactly as they did), `'bare'`
+ * (present, no attachDatasourceMutationPubSub — an older implementation),
+ * or `'real'` (exposes attachDatasourceMutationPubSub).
+ */
+ datasourceAdmin?: 'none' | 'bare' | 'real';
+ /** When true, the datasource-admin seam throws on attach. */
+ datasourceAttachThrows?: boolean;
 }
 
 function makeHarness(opts: HarnessOptions = {}) {
- const { driver = 'redis', metadata = 'fallback', protocol = 'real' } = opts;
+ const {
+ driver = 'redis', metadata = 'fallback', protocol = 'real',
+ datasourceAdmin = 'none', datasourceAttachThrows = false,
+ } = opts;
 
  const logger = { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() };
 
  const detachMetadata = vi.fn();
  const attachMetadata = vi.fn((_pubsub: unknown, _nodeId: string) => detachMetadata);
  const detachMutation = vi.fn();
  const attachMutation = vi.fn((_pubsub: unknown, _nodeId: string) => detachMutation);
+ const detachDatasource = vi.fn();
+ const attachDatasource = vi.fn((_pubsub: unknown, _nodeId: string) => {
+ if (datasourceAttachThrows) throw new Error('datasource attach exploded');
+ return detachDatasource;
+ });
 
  const pubsub = { publish: vi.fn(), subscribe: vi.fn(), close: vi.fn() };
  const cluster =
@@ -74,6 +91,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  protocol === 'none' ? undefined
  : protocol === 'bare' ? { saveMetaItem: vi.fn() }
  : { attachMetadataMutationPubSub: attachMutation };
+ const datasourceAdminService =
+ datasourceAdmin === 'none' ? undefined
+ : datasourceAdmin === 'bare' ? { listDatasources: vi.fn() }
+ : { attachDatasourceMutationPubSub: attachDatasource };
 
  const hooks = new Map<string, Array<() => Promise<void> | void>>();
  const ctx = {
@@ -96,6 +117,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  if (!protocolService) throw new Error('service not found: protocol');
  return protocolService;
  }
+ if (name === 'datasource-admin') {
+ if (!datasourceAdminService) throw new Error('service not found: datasource-admin');
+ return datasourceAdminService;
+ }
  throw new Error(`service not found: ${name}`);
  },
  } as unknown as PluginContext;
@@ -107,6 +132,7 @@ function makeHarness(opts: HarnessOptions = {}) {
  return {
  ctx, logger, fire, pubsub,
  attachMetadata, detachMetadata, attachMutation, detachMutation,
+ attachDatasource, detachDatasource,
  };
 }
 
@@ -280,3 +306,109 @@ describe('[#14021] lane 1 — an in-process bus must not be reported as “bridg
  expect(h.logger.error).not.toHaveBeenCalled();
  });
 });
+
+describe('[#13805] lane 3 — the datasource admin service’s datasource.mutated fan-out', () => {
+ it('⭐ attaches on a cross-process driver and reports it — independently of lanes 1 and 2', async () => {
+ // The shipped EE shape again, one owner over: no manager-backed
+ // metadata slot, no protocol seam, and a real datasource-admin service.
+ // Lane 3 must attach exactly there, with nothing from the other two
+ // lanes taking it down.
+ const h = makeHarness({ driver: 'redis', metadata: 'none', protocol: 'none', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ // Asserted VERBATIM, like lane 1's and lane 2's lines: the wording is
+ // what an operator reads as "datasource fan-out is on".
+ expect(infoLines(h)).toContain(
+ 'MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=node-a)',
+ );
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('all three lanes attach together when every owner exposes its seam', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachMutation).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(warnLines(h)).toEqual([]);
+ });
+
+ it('skips attach on the in-process memory driver — no peers to reach, nothing said above debug', async () => {
+ const h = makeHarness({ driver: 'memory', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ // The guard lanes 1 and 2 carry, from birth: on the memory driver a
+ // single replica's behaviour stays byte-identical to the pre-bridge
+ // one — no subscription, no publisher, no "bridged" claim.
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ expect(
+ debugLines(h).some((l) => l.includes('is in-process') && l.includes('datasource fan-out')),
+ ).toBe(true);
+ });
+
+ it('skips quietly when no datasource-admin service is registered', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'none' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ expect(warnLines(h).some((l) => l.includes('datasource'))).toBe(false);
+ });
+
+ it('skips quietly when the service does not expose the seam', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'bare' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('no cluster service at all skips lane 3 too', async () => {
+ const h = makeHarness({ driver: null, datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ });
+
+ it('a throwing attach is reported and does not take the other lanes down', async () => {
+ const h = makeHarness({
+ driver: 'redis', metadata: 'manager', protocol: 'real',
+ datasourceAdmin: 'real', datasourceAttachThrows: true,
+ });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledTimes(1);
+ expect(h.attachMutation).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalledWith(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ expect.any(Error),
+ );
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ });
+
+ it('kernel:shutdown detaches lane 3, and a throwing lane-2 detach does not strand it', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ h.detachMutation.mockImplementation(() => { throw new Error('detach exploded'); });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+ await h.fire('kernel:shutdown');
+
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalled();
+
+ // Idempotent: a second shutdown does not detach twice.
+ await h.fire('kernel:shutdown');
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ });
+});

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts

@@ -15,8 +15,8 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * state-owner packages only need the `IPubSub` interface, which lives in
  * `@objectstack/spec/contracts`.
  *
- * TWO lanes, late-bound independently at `kernel:ready`, because the state
- * that goes stale lives in two different owners (#13331):
+ * THREE lanes, late-bound independently at `kernel:ready`, because the state
+ * that goes stale lives in three different owners (#13331, #13805):
  *
  * 1. **Metadata service** (`attachClusterPubSub()` — `metadata.changed`):
  * replays watch events into peer `MetadataManager` caches
@@ -34,6 +34,15 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * 67×201 / 133×404). The lanes are independent on purpose: the boot
  * shape that lacks lane 1 (host-config, fallback metadata slot) is
  * exactly the shipped EE shape that needs lane 2.
+ * 3. **Datasource admin service** (`attachDatasourceMutationPubSub()` —
+ * `datasource.mutated`): fans a datasource create / update / delete out
+ * to peers, which converge their ObjectQL DRIVER registry from their OWN
+ * read of the shared datasource record. Lane 2's family, adopted by the
+ * driver registry (#13805, ruled 2026-09-01 — the same bridge shape, a
+ * symmetric signal, no second propagation mechanism): without it a
+ * `DELETE /api/v1/datasources/:name` recovered `/api/v1/ready` on the
+ * one replica that served it, and every other replica kept the stuck
+ * driver until restart.
  *
  * Activates each lane only when the cluster service and that lane's state
  * owner are present and expose the seam. Late binding is achieved via the
@@ -42,7 +51,9 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * Channels: `metadata.changed` — payload shape defined by
  * `ClusterMetadataChangedPayload` in `@objectstack/metadata`;
  * `metadata.mutated` — payload shape defined by
- * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`.
+ * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`;
+ * `datasource.mutated` — payload shape defined by
+ * `ClusterDatasourceMutationPayload` in `@objectstack/service-datasource`.
  *
  * See `content/docs/kernel/cluster.mdx` §5.
  */
@@ -53,6 +64,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
 
  private detach?: () => void;
  private detachMutation?: () => void;
+ private detachDatasource?: () => void;
 
  async init(ctx: PluginContext): Promise<void> {
  ctx.hook('kernel:ready', async () => {
@@ -67,6 +79,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
  }
  this.attachMetadataServiceLane(ctx, cluster);
  this.attachProtocolLane(ctx, cluster);
+ this.attachDatasourceLane(ctx, cluster);
  });
 
  ctx.hook('kernel:shutdown', async () => {
@@ -88,6 +101,15 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  this.detachMutation = undefined;
+ try {
+ this.detachDatasource?.();
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane detach error',
+ err as Error,
+ );
+ }
+ this.detachDatasource = undefined;
  });
  }
 
@@ -218,4 +240,61 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  }
+
+ /**
+ * Lane 3 — the datasource ADMIN SERVICE's `datasource.mutated` fan-out
+ * (#13805): the driver registry adopting the family lane 2 established.
+ *
+ * Duck-typed exactly like lanes 1 and 2 feature-detect their seams: this
+ * package must not depend on `@objectstack/service-datasource`, and
+ * `@objectstack/objectql` — the driver registry's owner — is handed no
+ * bus at all; the admin service publishes on the write doors it already
+ * owns and converges its pools through the seams it already injects.
+ *
+ * Guarded on {@link isInProcessClusterDriver} from birth, like lane 2: the
+ * in-process memory driver fans out to nobody, and on that driver a single
+ * replica's behaviour stays byte-identical to the pre-bridge one.
+ */
+ private attachDatasourceLane(ctx: PluginContext, cluster: IClusterService): void {
+ let admin: unknown;
+ try {
+ admin = ctx.getService<unknown>('datasource-admin');
+ } catch {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: no "datasource-admin" service registered, skipping datasource fan-out',
+ );
+ return;
+ }
+
+ const attach = (admin as { attachDatasourceMutationPubSub?: unknown })
+ .attachDatasourceMutationPubSub;
+ if (typeof attach !== 'function') {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: datasource-admin service does not expose attachDatasourceMutationPubSub(), skipping datasource fan-out',
+ );
+ return;
+ }
+
+ if (isInProcessClusterDriver(cluster.driver)) {
+ ctx.logger.debug(
+ `MetadataClusterBridgePlugin: cluster driver "${cluster.driver}" is in-process; datasource fan-out has no peers to reach, skipping`,
+ );
+ return;
+ }
+
+ try {
+ this.detachDatasource = (attach as (
+ pubsub: IClusterService['pubsub'],
+ nodeId: string,
+ ) => () => void).call(admin, cluster.pubsub, cluster.nodeId);
+ ctx.logger.info(
+ `MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=${cluster.nodeId})`,
+ );
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ err as Error,
+ );
+ }
+ }
 }

.changeset/datasource-mutation-cluster-fanout.md

@@ -0,0 +1,63 @@
+---
+"@objectstack/service-datasource": minor
+"@objectstack/service-cluster": minor
+---
+
+feat(service-datasource,service-cluster): fan datasource record writes out to peer replicas — a deleted datasource no longer keeps draining `/api/v1/ready` on every replica that did not serve the DELETE (#13805)
+
+Measured on a live 3-replica EE deployment: the ObjectQL DRIVER registry had
+no cluster propagation in either direction. Each replica filled it at boot
+from the shared datasource records and mutated it only for the writes IT
+served, so after `DELETE /api/v1/datasources/:name` only the replica that
+served the DELETE evicted the stuck driver (#13578's door) — the other N-1
+kept it, and `/api/v1/ready` kept answering 503 there, until restart. A
+datasource created through one replica likewise had no pool on any other
+until restart.
+
+Maintainer-ruled design (2026-09-01): the driver registry adopts the same
+cluster-invalidation family `metadata.mutated` (#13331) established — no
+second propagation mechanism, no bespoke poll loop, and no delete-only
+broadcast (that would have made delete more cluster-aware than create, a new
+asymmetry rather than a repair).
+
+- **Symmetric publisher at the three write doors.** `DatasourceAdminService`
+ now publishes the record's ADDRESS on a new cluster channel
+ `datasource.mutated` (`DATASOURCE_MUTATION_CLUSTER_CHANNEL`, payload
+ `ClusterDatasourceMutationPayload` — `{ originNode?, name }`) after
+ `createDatasource`, `updateDatasource` and `removeDatasource`. Fire-and-
+ forget: a publish failure never fails the write it announces.
+ `migrateCredential` does not publish — it leaves the live pool alone by
+ design, on every replica alike.
+- **Peers converge from their own read of the SHARED record.** On receipt a
+ replica re-reads the durable `sys_metadata` row for that name — the same
+ store its boot rehydration reads, not its per-replica metadata registry —
+ and converges its live pool through the seams it already owns: builds what
+ is missing, rebuilds in place what changed (`reregisterPool`, keeping the
+ old pool on failure exactly as the serving replica's update path does),
+ evicts what is gone (`unregisterPool` → the #13578 eviction door), and
+ leaves a matching pool untouched. The payload is a signal, never trusted
+ content, so a duplicate or re-ordered delivery converges to the same pool
+ state by construction — which is what makes a replayed create safe without
+ any new idempotency machinery. A name the replica never pooled is left
+ alone, so a stray signal cannot reach a code-defined pool.
+- **New attach seam, mirrored from the shipped bridges.**
+ `DatasourceAdminService.attachDatasourceMutationPubSub(pubsub, nodeId)` —
+ idempotent on the `(pubsub, nodeId)` pair, loopback suppression via
+ `originNode`, shaped after the protocol's `attachMetadataMutationPubSub()`.
+ Only `IPubSub` from `@objectstack/spec/contracts` crosses it:
+ `@objectstack/service-datasource` takes no dependency on the cluster
+ service, and `@objectstack/objectql` — the registry's owner — is handed no
+ bus. The host wires the receive half through a new optional
+ `DatasourceAdminServiceConfig.convergePool` seam; `DatasourceAdminServicePlugin`
+ supplies it.
+- **`MetadataClusterBridgePlugin` gains a third, independent lane** that
+ late-binds the seam at `kernel:ready` beside the metadata-service and
+ protocol lanes, duck-typed on the `datasource-admin` service. It skips the
+ in-process memory driver (nothing to fan out to), the guard the other lanes
+ carry, so a single-replica boot behaves byte-identically to before.
+
+No shipped driver exceeds at-most-once delivery, so a lost message still
+degrades to the pre-existing bound (the next boot's full rehydration); this
+channel narrows the window from "until every replica restarts" to one network
+hop. The `/api/v1/meta/datasource` metadata registry's own cross-replica
+coherence (#13609) is a different sink and is not touched here.

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.test.ts

@@ -48,17 +48,34 @@ interface HarnessOptions {
  * `'real'` (exposes attachMetadataMutationPubSub).
  */
  protocol?: 'none' | 'bare' | 'real';
+ /**
+ * [#13805] The `datasource-admin` slot: `'none'` (getService throws — the
+ * default, so the #13331 cases above read exactly as they did), `'bare'`
+ * (present, no attachDatasourceMutationPubSub — an older implementation),
+ * or `'real'` (exposes attachDatasourceMutationPubSub).
+ */
+ datasourceAdmin?: 'none' | 'bare' | 'real';
+ /** When true, the datasource-admin seam throws on attach. */
+ datasourceAttachThrows?: boolean;
 }
 
 function makeHarness(opts: HarnessOptions = {}) {
- const { driver = 'redis', metadata = 'fallback', protocol = 'real' } = opts;
+ const {
+ driver = 'redis', metadata = 'fallback', protocol = 'real',
+ datasourceAdmin = 'none', datasourceAttachThrows = false,
+ } = opts;
 
  const logger = { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() };
 
  const detachMetadata = vi.fn();
  const attachMetadata = vi.fn((_pubsub: unknown, _nodeId: string) => detachMetadata);
  const detachMutation = vi.fn();
  const attachMutation = vi.fn((_pubsub: unknown, _nodeId: string) => detachMutation);
+ const detachDatasource = vi.fn();
+ const attachDatasource = vi.fn((_pubsub: unknown, _nodeId: string) => {
+ if (datasourceAttachThrows) throw new Error('datasource attach exploded');
+ return detachDatasource;
+ });
 
  const pubsub = { publish: vi.fn(), subscribe: vi.fn(), close: vi.fn() };
  const cluster =
@@ -74,6 +91,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  protocol === 'none' ? undefined
  : protocol === 'bare' ? { saveMetaItem: vi.fn() }
  : { attachMetadataMutationPubSub: attachMutation };
+ const datasourceAdminService =
+ datasourceAdmin === 'none' ? undefined
+ : datasourceAdmin === 'bare' ? { listDatasources: vi.fn() }
+ : { attachDatasourceMutationPubSub: attachDatasource };
 
  const hooks = new Map<string, Array<() => Promise<void> | void>>();
  const ctx = {
@@ -96,6 +117,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  if (!protocolService) throw new Error('service not found: protocol');
  return protocolService;
  }
+ if (name === 'datasource-admin') {
+ if (!datasourceAdminService) throw new Error('service not found: datasource-admin');
+ return datasourceAdminService;
+ }
  throw new Error(`service not found: ${name}`);
  },
  } as unknown as PluginContext;
@@ -107,6 +132,7 @@ function makeHarness(opts: HarnessOptions = {}) {
  return {
  ctx, logger, fire, pubsub,
  attachMetadata, detachMetadata, attachMutation, detachMutation,
+ attachDatasource, detachDatasource,
  };
 }
 
@@ -280,3 +306,109 @@ describe('[#14021] lane 1 — an in-process bus must not be reported as “bridg
  expect(h.logger.error).not.toHaveBeenCalled();
  });
 });
+
+describe('[#13805] lane 3 — the datasource admin service’s datasource.mutated fan-out', () => {
+ it('⭐ attaches on a cross-process driver and reports it — independently of lanes 1 and 2', async () => {
+ // The shipped EE shape again, one owner over: no manager-backed
+ // metadata slot, no protocol seam, and a real datasource-admin service.
+ // Lane 3 must attach exactly there, with nothing from the other two
+ // lanes taking it down.
+ const h = makeHarness({ driver: 'redis', metadata: 'none', protocol: 'none', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ // Asserted VERBATIM, like lane 1's and lane 2's lines: the wording is
+ // what an operator reads as "datasource fan-out is on".
+ expect(infoLines(h)).toContain(
+ 'MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=node-a)',
+ );
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('all three lanes attach together when every owner exposes its seam', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachMutation).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(warnLines(h)).toEqual([]);
+ });
+
+ it('skips attach on the in-process memory driver — no peers to reach, nothing said above debug', async () => {
+ const h = makeHarness({ driver: 'memory', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ // The guard lanes 1 and 2 carry, from birth: on the memory driver a
+ // single replica's behaviour stays byte-identical to the pre-bridge
+ // one — no subscription, no publisher, no "bridged" claim.
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ expect(
+ debugLines(h).some((l) => l.includes('is in-process') && l.includes('datasource fan-out')),
+ ).toBe(true);
+ });
+
+ it('skips quietly when no datasource-admin service is registered', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'none' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ expect(warnLines(h).some((l) => l.includes('datasource'))).toBe(false);
+ });
+
+ it('skips quietly when the service does not expose the seam', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'bare' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('no cluster service at all skips lane 3 too', async () => {
+ const h = makeHarness({ driver: null, datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ });
+
+ it('a throwing attach is reported and does not take the other lanes down', async () => {
+ const h = makeHarness({
+ driver: 'redis', metadata: 'manager', protocol: 'real',
+ datasourceAdmin: 'real', datasourceAttachThrows: true,
+ });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledTimes(1);
+ expect(h.attachMutation).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalledWith(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ expect.any(Error),
+ );
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ });
+
+ it('kernel:shutdown detaches lane 3, and a throwing lane-2 detach does not strand it', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ h.detachMutation.mockImplementation(() => { throw new Error('detach exploded'); });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+ await h.fire('kernel:shutdown');
+
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalled();
+
+ // Idempotent: a second shutdown does not detach twice.
+ await h.fire('kernel:shutdown');
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ });
+});

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts

@@ -15,8 +15,8 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * state-owner packages only need the `IPubSub` interface, which lives in
  * `@objectstack/spec/contracts`.
  *
- * TWO lanes, late-bound independently at `kernel:ready`, because the state
- * that goes stale lives in two different owners (#13331):
+ * THREE lanes, late-bound independently at `kernel:ready`, because the state
+ * that goes stale lives in three different owners (#13331, #13805):
  *
  * 1. **Metadata service** (`attachClusterPubSub()` — `metadata.changed`):
  * replays watch events into peer `MetadataManager` caches
@@ -34,6 +34,15 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * 67×201 / 133×404). The lanes are independent on purpose: the boot
  * shape that lacks lane 1 (host-config, fallback metadata slot) is
  * exactly the shipped EE shape that needs lane 2.
+ * 3. **Datasource admin service** (`attachDatasourceMutationPubSub()` —
+ * `datasource.mutated`): fans a datasource create / update / delete out
+ * to peers, which converge their ObjectQL DRIVER registry from their OWN
+ * read of the shared datasource record. Lane 2's family, adopted by the
+ * driver registry (#13805, ruled 2026-09-01 — the same bridge shape, a
+ * symmetric signal, no second propagation mechanism): without it a
+ * `DELETE /api/v1/datasources/:name` recovered `/api/v1/ready` on the
+ * one replica that served it, and every other replica kept the stuck
+ * driver until restart.
  *
  * Activates each lane only when the cluster service and that lane's state
  * owner are present and expose the seam. Late binding is achieved via the
@@ -42,7 +51,9 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * Channels: `metadata.changed` — payload shape defined by
  * `ClusterMetadataChangedPayload` in `@objectstack/metadata`;
  * `metadata.mutated` — payload shape defined by
- * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`.
+ * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`;
+ * `datasource.mutated` — payload shape defined by
+ * `ClusterDatasourceMutationPayload` in `@objectstack/service-datasource`.
  *
  * See `content/docs/kernel/cluster.mdx` §5.
  */
@@ -53,6 +64,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
 
  private detach?: () => void;
  private detachMutation?: () => void;
+ private detachDatasource?: () => void;
 
  async init(ctx: PluginContext): Promise<void> {
  ctx.hook('kernel:ready', async () => {
@@ -67,6 +79,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
  }
  this.attachMetadataServiceLane(ctx, cluster);
  this.attachProtocolLane(ctx, cluster);
+ this.attachDatasourceLane(ctx, cluster);
  });
 
  ctx.hook('kernel:shutdown', async () => {
@@ -88,6 +101,15 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  this.detachMutation = undefined;
+ try {
+ this.detachDatasource?.();
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane detach error',
+ err as Error,
+ );
+ }
+ this.detachDatasource = undefined;
  });
  }
 
@@ -218,4 +240,61 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  }
+
+ /**
+ * Lane 3 — the datasource ADMIN SERVICE's `datasource.mutated` fan-out
+ * (#13805): the driver registry adopting the family lane 2 established.
+ *
+ * Duck-typed exactly like lanes 1 and 2 feature-detect their seams: this
+ * package must not depend on `@objectstack/service-datasource`, and
+ * `@objectstack/objectql` — the driver registry's owner — is handed no
+ * bus at all; the admin service publishes on the write doors it already
+ * owns and converges its pools through the seams it already injects.
+ *
+ * Guarded on {@link isInProcessClusterDriver} from birth, like lane 2: the
+ * in-process memory driver fans out to nobody, and on that driver a single
+ * replica's behaviour stays byte-identical to the pre-bridge one.
+ */
+ private attachDatasourceLane(ctx: PluginContext, cluster: IClusterService): void {
+ let admin: unknown;
+ try {
+ admin = ctx.getService<unknown>('datasource-admin');
+ } catch {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: no "datasource-admin" service registered, skipping datasource fan-out',
+ );
+ return;
+ }
+
+ const attach = (admin as { attachDatasourceMutationPubSub?: unknown })
+ .attachDatasourceMutationPubSub;
+ if (typeof attach !== 'function') {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: datasource-admin service does not expose attachDatasourceMutationPubSub(), skipping datasource fan-out',
+ );
+ return;
+ }
+
+ if (isInProcessClusterDriver(cluster.driver)) {
+ ctx.logger.debug(
+ `MetadataClusterBridgePlugin: cluster driver "${cluster.driver}" is in-process; datasource fan-out has no peers to reach, skipping`,
+ );
+ return;
+ }
+
+ try {
+ this.detachDatasource = (attach as (
+ pubsub: IClusterService['pubsub'],
+ nodeId: string,
+ ) => () => void).call(admin, cluster.pubsub, cluster.nodeId);
+ ctx.logger.info(
+ `MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=${cluster.nodeId})`,
+ );
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ err as Error,
+ );
+ }
+ }
 }

.changeset/datasource-mutation-cluster-fanout.md

@@ -0,0 +1,63 @@
+---
+"@objectstack/service-datasource": minor
+"@objectstack/service-cluster": minor
+---
+
+feat(service-datasource,service-cluster): fan datasource record writes out to peer replicas — a deleted datasource no longer keeps draining `/api/v1/ready` on every replica that did not serve the DELETE (#13805)
+
+Measured on a live 3-replica EE deployment: the ObjectQL DRIVER registry had
+no cluster propagation in either direction. Each replica filled it at boot
+from the shared datasource records and mutated it only for the writes IT
+served, so after `DELETE /api/v1/datasources/:name` only the replica that
+served the DELETE evicted the stuck driver (#13578's door) — the other N-1
+kept it, and `/api/v1/ready` kept answering 503 there, until restart. A
+datasource created through one replica likewise had no pool on any other
+until restart.
+
+Maintainer-ruled design (2026-09-01): the driver registry adopts the same
+cluster-invalidation family `metadata.mutated` (#13331) established — no
+second propagation mechanism, no bespoke poll loop, and no delete-only
+broadcast (that would have made delete more cluster-aware than create, a new
+asymmetry rather than a repair).
+
+- **Symmetric publisher at the three write doors.** `DatasourceAdminService`
+ now publishes the record's ADDRESS on a new cluster channel
+ `datasource.mutated` (`DATASOURCE_MUTATION_CLUSTER_CHANNEL`, payload
+ `ClusterDatasourceMutationPayload` — `{ originNode?, name }`) after
+ `createDatasource`, `updateDatasource` and `removeDatasource`. Fire-and-
+ forget: a publish failure never fails the write it announces.
+ `migrateCredential` does not publish — it leaves the live pool alone by
+ design, on every replica alike.
+- **Peers converge from their own read of the SHARED record.** On receipt a
+ replica re-reads the durable `sys_metadata` row for that name — the same
+ store its boot rehydration reads, not its per-replica metadata registry —
+ and converges its live pool through the seams it already owns: builds what
+ is missing, rebuilds in place what changed (`reregisterPool`, keeping the
+ old pool on failure exactly as the serving replica's update path does),
+ evicts what is gone (`unregisterPool` → the #13578 eviction door), and
+ leaves a matching pool untouched. The payload is a signal, never trusted
+ content, so a duplicate or re-ordered delivery converges to the same pool
+ state by construction — which is what makes a replayed create safe without
+ any new idempotency machinery. A name the replica never pooled is left
+ alone, so a stray signal cannot reach a code-defined pool.
+- **New attach seam, mirrored from the shipped bridges.**
+ `DatasourceAdminService.attachDatasourceMutationPubSub(pubsub, nodeId)` —
+ idempotent on the `(pubsub, nodeId)` pair, loopback suppression via
+ `originNode`, shaped after the protocol's `attachMetadataMutationPubSub()`.
+ Only `IPubSub` from `@objectstack/spec/contracts` crosses it:
+ `@objectstack/service-datasource` takes no dependency on the cluster
+ service, and `@objectstack/objectql` — the registry's owner — is handed no
+ bus. The host wires the receive half through a new optional
+ `DatasourceAdminServiceConfig.convergePool` seam; `DatasourceAdminServicePlugin`
+ supplies it.
+- **`MetadataClusterBridgePlugin` gains a third, independent lane** that
+ late-binds the seam at `kernel:ready` beside the metadata-service and
+ protocol lanes, duck-typed on the `datasource-admin` service. It skips the
+ in-process memory driver (nothing to fan out to), the guard the other lanes
+ carry, so a single-replica boot behaves byte-identically to before.
+
+No shipped driver exceeds at-most-once delivery, so a lost message still
+degrades to the pre-existing bound (the next boot's full rehydration); this
+channel narrows the window from "until every replica restarts" to one network
+hop. The `/api/v1/meta/datasource` metadata registry's own cross-replica
+coherence (#13609) is a different sink and is not touched here.

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.test.ts

@@ -48,17 +48,34 @@ interface HarnessOptions {
  * `'real'` (exposes attachMetadataMutationPubSub).
  */
  protocol?: 'none' | 'bare' | 'real';
+ /**
+ * [#13805] The `datasource-admin` slot: `'none'` (getService throws — the
+ * default, so the #13331 cases above read exactly as they did), `'bare'`
+ * (present, no attachDatasourceMutationPubSub — an older implementation),
+ * or `'real'` (exposes attachDatasourceMutationPubSub).
+ */
+ datasourceAdmin?: 'none' | 'bare' | 'real';
+ /** When true, the datasource-admin seam throws on attach. */
+ datasourceAttachThrows?: boolean;
 }
 
 function makeHarness(opts: HarnessOptions = {}) {
- const { driver = 'redis', metadata = 'fallback', protocol = 'real' } = opts;
+ const {
+ driver = 'redis', metadata = 'fallback', protocol = 'real',
+ datasourceAdmin = 'none', datasourceAttachThrows = false,
+ } = opts;
 
  const logger = { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() };
 
  const detachMetadata = vi.fn();
  const attachMetadata = vi.fn((_pubsub: unknown, _nodeId: string) => detachMetadata);
  const detachMutation = vi.fn();
  const attachMutation = vi.fn((_pubsub: unknown, _nodeId: string) => detachMutation);
+ const detachDatasource = vi.fn();
+ const attachDatasource = vi.fn((_pubsub: unknown, _nodeId: string) => {
+ if (datasourceAttachThrows) throw new Error('datasource attach exploded');
+ return detachDatasource;
+ });
 
  const pubsub = { publish: vi.fn(), subscribe: vi.fn(), close: vi.fn() };
  const cluster =
@@ -74,6 +91,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  protocol === 'none' ? undefined
  : protocol === 'bare' ? { saveMetaItem: vi.fn() }
  : { attachMetadataMutationPubSub: attachMutation };
+ const datasourceAdminService =
+ datasourceAdmin === 'none' ? undefined
+ : datasourceAdmin === 'bare' ? { listDatasources: vi.fn() }
+ : { attachDatasourceMutationPubSub: attachDatasource };
 
  const hooks = new Map<string, Array<() => Promise<void> | void>>();
  const ctx = {
@@ -96,6 +117,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  if (!protocolService) throw new Error('service not found: protocol');
  return protocolService;
  }
+ if (name === 'datasource-admin') {
+ if (!datasourceAdminService) throw new Error('service not found: datasource-admin');
+ return datasourceAdminService;
+ }
  throw new Error(`service not found: ${name}`);
  },
  } as unknown as PluginContext;
@@ -107,6 +132,7 @@ function makeHarness(opts: HarnessOptions = {}) {
  return {
  ctx, logger, fire, pubsub,
  attachMetadata, detachMetadata, attachMutation, detachMutation,
+ attachDatasource, detachDatasource,
  };
 }
 
@@ -280,3 +306,109 @@ describe('[#14021] lane 1 — an in-process bus must not be reported as “bridg
  expect(h.logger.error).not.toHaveBeenCalled();
  });
 });
+
+describe('[#13805] lane 3 — the datasource admin service’s datasource.mutated fan-out', () => {
+ it('⭐ attaches on a cross-process driver and reports it — independently of lanes 1 and 2', async () => {
+ // The shipped EE shape again, one owner over: no manager-backed
+ // metadata slot, no protocol seam, and a real datasource-admin service.
+ // Lane 3 must attach exactly there, with nothing from the other two
+ // lanes taking it down.
+ const h = makeHarness({ driver: 'redis', metadata: 'none', protocol: 'none', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ // Asserted VERBATIM, like lane 1's and lane 2's lines: the wording is
+ // what an operator reads as "datasource fan-out is on".
+ expect(infoLines(h)).toContain(
+ 'MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=node-a)',
+ );
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('all three lanes attach together when every owner exposes its seam', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachMutation).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(warnLines(h)).toEqual([]);
+ });
+
+ it('skips attach on the in-process memory driver — no peers to reach, nothing said above debug', async () => {
+ const h = makeHarness({ driver: 'memory', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ // The guard lanes 1 and 2 carry, from birth: on the memory driver a
+ // single replica's behaviour stays byte-identical to the pre-bridge
+ // one — no subscription, no publisher, no "bridged" claim.
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ expect(
+ debugLines(h).some((l) => l.includes('is in-process') && l.includes('datasource fan-out')),
+ ).toBe(true);
+ });
+
+ it('skips quietly when no datasource-admin service is registered', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'none' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ expect(warnLines(h).some((l) => l.includes('datasource'))).toBe(false);
+ });
+
+ it('skips quietly when the service does not expose the seam', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'bare' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('no cluster service at all skips lane 3 too', async () => {
+ const h = makeHarness({ driver: null, datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ });
+
+ it('a throwing attach is reported and does not take the other lanes down', async () => {
+ const h = makeHarness({
+ driver: 'redis', metadata: 'manager', protocol: 'real',
+ datasourceAdmin: 'real', datasourceAttachThrows: true,
+ });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledTimes(1);
+ expect(h.attachMutation).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalledWith(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ expect.any(Error),
+ );
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ });
+
+ it('kernel:shutdown detaches lane 3, and a throwing lane-2 detach does not strand it', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ h.detachMutation.mockImplementation(() => { throw new Error('detach exploded'); });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+ await h.fire('kernel:shutdown');
+
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalled();
+
+ // Idempotent: a second shutdown does not detach twice.
+ await h.fire('kernel:shutdown');
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ });
+});

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts

@@ -15,8 +15,8 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * state-owner packages only need the `IPubSub` interface, which lives in
  * `@objectstack/spec/contracts`.
  *
- * TWO lanes, late-bound independently at `kernel:ready`, because the state
- * that goes stale lives in two different owners (#13331):
+ * THREE lanes, late-bound independently at `kernel:ready`, because the state
+ * that goes stale lives in three different owners (#13331, #13805):
  *
  * 1. **Metadata service** (`attachClusterPubSub()` — `metadata.changed`):
  * replays watch events into peer `MetadataManager` caches
@@ -34,6 +34,15 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * 67×201 / 133×404). The lanes are independent on purpose: the boot
  * shape that lacks lane 1 (host-config, fallback metadata slot) is
  * exactly the shipped EE shape that needs lane 2.
+ * 3. **Datasource admin service** (`attachDatasourceMutationPubSub()` —
+ * `datasource.mutated`): fans a datasource create / update / delete out
+ * to peers, which converge their ObjectQL DRIVER registry from their OWN
+ * read of the shared datasource record. Lane 2's family, adopted by the
+ * driver registry (#13805, ruled 2026-09-01 — the same bridge shape, a
+ * symmetric signal, no second propagation mechanism): without it a
+ * `DELETE /api/v1/datasources/:name` recovered `/api/v1/ready` on the
+ * one replica that served it, and every other replica kept the stuck
+ * driver until restart.
  *
  * Activates each lane only when the cluster service and that lane's state
  * owner are present and expose the seam. Late binding is achieved via the
@@ -42,7 +51,9 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * Channels: `metadata.changed` — payload shape defined by
  * `ClusterMetadataChangedPayload` in `@objectstack/metadata`;
  * `metadata.mutated` — payload shape defined by
- * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`.
+ * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`;
+ * `datasource.mutated` — payload shape defined by
+ * `ClusterDatasourceMutationPayload` in `@objectstack/service-datasource`.
  *
  * See `content/docs/kernel/cluster.mdx` §5.
  */
@@ -53,6 +64,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
 
  private detach?: () => void;
  private detachMutation?: () => void;
+ private detachDatasource?: () => void;
 
  async init(ctx: PluginContext): Promise<void> {
  ctx.hook('kernel:ready', async () => {
@@ -67,6 +79,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
  }
  this.attachMetadataServiceLane(ctx, cluster);
  this.attachProtocolLane(ctx, cluster);
+ this.attachDatasourceLane(ctx, cluster);
  });
 
  ctx.hook('kernel:shutdown', async () => {
@@ -88,6 +101,15 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  this.detachMutation = undefined;
+ try {
+ this.detachDatasource?.();
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane detach error',
+ err as Error,
+ );
+ }
+ this.detachDatasource = undefined;
  });
  }
 
@@ -218,4 +240,61 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  }
+
+ /**
+ * Lane 3 — the datasource ADMIN SERVICE's `datasource.mutated` fan-out
+ * (#13805): the driver registry adopting the family lane 2 established.
+ *
+ * Duck-typed exactly like lanes 1 and 2 feature-detect their seams: this
+ * package must not depend on `@objectstack/service-datasource`, and
+ * `@objectstack/objectql` — the driver registry's owner — is handed no
+ * bus at all; the admin service publishes on the write doors it already
+ * owns and converges its pools through the seams it already injects.
+ *
+ * Guarded on {@link isInProcessClusterDriver} from birth, like lane 2: the
+ * in-process memory driver fans out to nobody, and on that driver a single
+ * replica's behaviour stays byte-identical to the pre-bridge one.
+ */
+ private attachDatasourceLane(ctx: PluginContext, cluster: IClusterService): void {
+ let admin: unknown;
+ try {
+ admin = ctx.getService<unknown>('datasource-admin');
+ } catch {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: no "datasource-admin" service registered, skipping datasource fan-out',
+ );
+ return;
+ }
+
+ const attach = (admin as { attachDatasourceMutationPubSub?: unknown })
+ .attachDatasourceMutationPubSub;
+ if (typeof attach !== 'function') {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: datasource-admin service does not expose attachDatasourceMutationPubSub(), skipping datasource fan-out',
+ );
+ return;
+ }
+
+ if (isInProcessClusterDriver(cluster.driver)) {
+ ctx.logger.debug(
+ `MetadataClusterBridgePlugin: cluster driver "${cluster.driver}" is in-process; datasource fan-out has no peers to reach, skipping`,
+ );
+ return;
+ }
+
+ try {
+ this.detachDatasource = (attach as (
+ pubsub: IClusterService['pubsub'],
+ nodeId: string,
+ ) => () => void).call(admin, cluster.pubsub, cluster.nodeId);
+ ctx.logger.info(
+ `MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=${cluster.nodeId})`,
+ );
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ err as Error,
+ );
+ }
+ }
 }

.changeset/datasource-mutation-cluster-fanout.md

@@ -0,0 +1,63 @@
+---
+"@objectstack/service-datasource": minor
+"@objectstack/service-cluster": minor
+---
+
+feat(service-datasource,service-cluster): fan datasource record writes out to peer replicas — a deleted datasource no longer keeps draining `/api/v1/ready` on every replica that did not serve the DELETE (#13805)
+
+Measured on a live 3-replica EE deployment: the ObjectQL DRIVER registry had
+no cluster propagation in either direction. Each replica filled it at boot
+from the shared datasource records and mutated it only for the writes IT
+served, so after `DELETE /api/v1/datasources/:name` only the replica that
+served the DELETE evicted the stuck driver (#13578's door) — the other N-1
+kept it, and `/api/v1/ready` kept answering 503 there, until restart. A
+datasource created through one replica likewise had no pool on any other
+until restart.
+
+Maintainer-ruled design (2026-09-01): the driver registry adopts the same
+cluster-invalidation family `metadata.mutated` (#13331) established — no
+second propagation mechanism, no bespoke poll loop, and no delete-only
+broadcast (that would have made delete more cluster-aware than create, a new
+asymmetry rather than a repair).
+
+- **Symmetric publisher at the three write doors.** `DatasourceAdminService`
+ now publishes the record's ADDRESS on a new cluster channel
+ `datasource.mutated` (`DATASOURCE_MUTATION_CLUSTER_CHANNEL`, payload
+ `ClusterDatasourceMutationPayload` — `{ originNode?, name }`) after
+ `createDatasource`, `updateDatasource` and `removeDatasource`. Fire-and-
+ forget: a publish failure never fails the write it announces.
+ `migrateCredential` does not publish — it leaves the live pool alone by
+ design, on every replica alike.
+- **Peers converge from their own read of the SHARED record.** On receipt a
+ replica re-reads the durable `sys_metadata` row for that name — the same
+ store its boot rehydration reads, not its per-replica metadata registry —
+ and converges its live pool through the seams it already owns: builds what
+ is missing, rebuilds in place what changed (`reregisterPool`, keeping the
+ old pool on failure exactly as the serving replica's update path does),
+ evicts what is gone (`unregisterPool` → the #13578 eviction door), and
+ leaves a matching pool untouched. The payload is a signal, never trusted
+ content, so a duplicate or re-ordered delivery converges to the same pool
+ state by construction — which is what makes a replayed create safe without
+ any new idempotency machinery. A name the replica never pooled is left
+ alone, so a stray signal cannot reach a code-defined pool.
+- **New attach seam, mirrored from the shipped bridges.**
+ `DatasourceAdminService.attachDatasourceMutationPubSub(pubsub, nodeId)` —
+ idempotent on the `(pubsub, nodeId)` pair, loopback suppression via
+ `originNode`, shaped after the protocol's `attachMetadataMutationPubSub()`.
+ Only `IPubSub` from `@objectstack/spec/contracts` crosses it:
+ `@objectstack/service-datasource` takes no dependency on the cluster
+ service, and `@objectstack/objectql` — the registry's owner — is handed no
+ bus. The host wires the receive half through a new optional
+ `DatasourceAdminServiceConfig.convergePool` seam; `DatasourceAdminServicePlugin`
+ supplies it.
+- **`MetadataClusterBridgePlugin` gains a third, independent lane** that
+ late-binds the seam at `kernel:ready` beside the metadata-service and
+ protocol lanes, duck-typed on the `datasource-admin` service. It skips the
+ in-process memory driver (nothing to fan out to), the guard the other lanes
+ carry, so a single-replica boot behaves byte-identically to before.
+
+No shipped driver exceeds at-most-once delivery, so a lost message still
+degrades to the pre-existing bound (the next boot's full rehydration); this
+channel narrows the window from "until every replica restarts" to one network
+hop. The `/api/v1/meta/datasource` metadata registry's own cross-replica
+coherence (#13609) is a different sink and is not touched here.

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.test.ts

@@ -48,17 +48,34 @@ interface HarnessOptions {
  * `'real'` (exposes attachMetadataMutationPubSub).
  */
  protocol?: 'none' | 'bare' | 'real';
+ /**
+ * [#13805] The `datasource-admin` slot: `'none'` (getService throws — the
+ * default, so the #13331 cases above read exactly as they did), `'bare'`
+ * (present, no attachDatasourceMutationPubSub — an older implementation),
+ * or `'real'` (exposes attachDatasourceMutationPubSub).
+ */
+ datasourceAdmin?: 'none' | 'bare' | 'real';
+ /** When true, the datasource-admin seam throws on attach. */
+ datasourceAttachThrows?: boolean;
 }
 
 function makeHarness(opts: HarnessOptions = {}) {
- const { driver = 'redis', metadata = 'fallback', protocol = 'real' } = opts;
+ const {
+ driver = 'redis', metadata = 'fallback', protocol = 'real',
+ datasourceAdmin = 'none', datasourceAttachThrows = false,
+ } = opts;
 
  const logger = { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() };
 
  const detachMetadata = vi.fn();
  const attachMetadata = vi.fn((_pubsub: unknown, _nodeId: string) => detachMetadata);
  const detachMutation = vi.fn();
  const attachMutation = vi.fn((_pubsub: unknown, _nodeId: string) => detachMutation);
+ const detachDatasource = vi.fn();
+ const attachDatasource = vi.fn((_pubsub: unknown, _nodeId: string) => {
+ if (datasourceAttachThrows) throw new Error('datasource attach exploded');
+ return detachDatasource;
+ });
 
  const pubsub = { publish: vi.fn(), subscribe: vi.fn(), close: vi.fn() };
  const cluster =
@@ -74,6 +91,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  protocol === 'none' ? undefined
  : protocol === 'bare' ? { saveMetaItem: vi.fn() }
  : { attachMetadataMutationPubSub: attachMutation };
+ const datasourceAdminService =
+ datasourceAdmin === 'none' ? undefined
+ : datasourceAdmin === 'bare' ? { listDatasources: vi.fn() }
+ : { attachDatasourceMutationPubSub: attachDatasource };
 
  const hooks = new Map<string, Array<() => Promise<void> | void>>();
  const ctx = {
@@ -96,6 +117,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  if (!protocolService) throw new Error('service not found: protocol');
  return protocolService;
  }
+ if (name === 'datasource-admin') {
+ if (!datasourceAdminService) throw new Error('service not found: datasource-admin');
+ return datasourceAdminService;
+ }
  throw new Error(`service not found: ${name}`);
  },
  } as unknown as PluginContext;
@@ -107,6 +132,7 @@ function makeHarness(opts: HarnessOptions = {}) {
  return {
  ctx, logger, fire, pubsub,
  attachMetadata, detachMetadata, attachMutation, detachMutation,
+ attachDatasource, detachDatasource,
  };
 }
 
@@ -280,3 +306,109 @@ describe('[#14021] lane 1 — an in-process bus must not be reported as “bridg
  expect(h.logger.error).not.toHaveBeenCalled();
  });
 });
+
+describe('[#13805] lane 3 — the datasource admin service’s datasource.mutated fan-out', () => {
+ it('⭐ attaches on a cross-process driver and reports it — independently of lanes 1 and 2', async () => {
+ // The shipped EE shape again, one owner over: no manager-backed
+ // metadata slot, no protocol seam, and a real datasource-admin service.
+ // Lane 3 must attach exactly there, with nothing from the other two
+ // lanes taking it down.
+ const h = makeHarness({ driver: 'redis', metadata: 'none', protocol: 'none', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ // Asserted VERBATIM, like lane 1's and lane 2's lines: the wording is
+ // what an operator reads as "datasource fan-out is on".
+ expect(infoLines(h)).toContain(
+ 'MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=node-a)',
+ );
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('all three lanes attach together when every owner exposes its seam', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachMutation).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(warnLines(h)).toEqual([]);
+ });
+
+ it('skips attach on the in-process memory driver — no peers to reach, nothing said above debug', async () => {
+ const h = makeHarness({ driver: 'memory', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ // The guard lanes 1 and 2 carry, from birth: on the memory driver a
+ // single replica's behaviour stays byte-identical to the pre-bridge
+ // one — no subscription, no publisher, no "bridged" claim.
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ expect(
+ debugLines(h).some((l) => l.includes('is in-process') && l.includes('datasource fan-out')),
+ ).toBe(true);
+ });
+
+ it('skips quietly when no datasource-admin service is registered', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'none' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ expect(warnLines(h).some((l) => l.includes('datasource'))).toBe(false);
+ });
+
+ it('skips quietly when the service does not expose the seam', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'bare' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('no cluster service at all skips lane 3 too', async () => {
+ const h = makeHarness({ driver: null, datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ });
+
+ it('a throwing attach is reported and does not take the other lanes down', async () => {
+ const h = makeHarness({
+ driver: 'redis', metadata: 'manager', protocol: 'real',
+ datasourceAdmin: 'real', datasourceAttachThrows: true,
+ });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledTimes(1);
+ expect(h.attachMutation).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalledWith(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ expect.any(Error),
+ );
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ });
+
+ it('kernel:shutdown detaches lane 3, and a throwing lane-2 detach does not strand it', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ h.detachMutation.mockImplementation(() => { throw new Error('detach exploded'); });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+ await h.fire('kernel:shutdown');
+
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalled();
+
+ // Idempotent: a second shutdown does not detach twice.
+ await h.fire('kernel:shutdown');
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ });
+});

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts

@@ -15,8 +15,8 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * state-owner packages only need the `IPubSub` interface, which lives in
  * `@objectstack/spec/contracts`.
  *
- * TWO lanes, late-bound independently at `kernel:ready`, because the state
- * that goes stale lives in two different owners (#13331):
+ * THREE lanes, late-bound independently at `kernel:ready`, because the state
+ * that goes stale lives in three different owners (#13331, #13805):
  *
  * 1. **Metadata service** (`attachClusterPubSub()` — `metadata.changed`):
  * replays watch events into peer `MetadataManager` caches
@@ -34,6 +34,15 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * 67×201 / 133×404). The lanes are independent on purpose: the boot
  * shape that lacks lane 1 (host-config, fallback metadata slot) is
  * exactly the shipped EE shape that needs lane 2.
+ * 3. **Datasource admin service** (`attachDatasourceMutationPubSub()` —
+ * `datasource.mutated`): fans a datasource create / update / delete out
+ * to peers, which converge their ObjectQL DRIVER registry from their OWN
+ * read of the shared datasource record. Lane 2's family, adopted by the
+ * driver registry (#13805, ruled 2026-09-01 — the same bridge shape, a
+ * symmetric signal, no second propagation mechanism): without it a
+ * `DELETE /api/v1/datasources/:name` recovered `/api/v1/ready` on the
+ * one replica that served it, and every other replica kept the stuck
+ * driver until restart.
  *
  * Activates each lane only when the cluster service and that lane's state
  * owner are present and expose the seam. Late binding is achieved via the
@@ -42,7 +51,9 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * Channels: `metadata.changed` — payload shape defined by
  * `ClusterMetadataChangedPayload` in `@objectstack/metadata`;
  * `metadata.mutated` — payload shape defined by
- * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`.
+ * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`;
+ * `datasource.mutated` — payload shape defined by
+ * `ClusterDatasourceMutationPayload` in `@objectstack/service-datasource`.
  *
  * See `content/docs/kernel/cluster.mdx` §5.
  */
@@ -53,6 +64,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
 
  private detach?: () => void;
  private detachMutation?: () => void;
+ private detachDatasource?: () => void;
 
  async init(ctx: PluginContext): Promise<void> {
  ctx.hook('kernel:ready', async () => {
@@ -67,6 +79,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
  }
  this.attachMetadataServiceLane(ctx, cluster);
  this.attachProtocolLane(ctx, cluster);
+ this.attachDatasourceLane(ctx, cluster);
  });
 
  ctx.hook('kernel:shutdown', async () => {
@@ -88,6 +101,15 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  this.detachMutation = undefined;
+ try {
+ this.detachDatasource?.();
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane detach error',
+ err as Error,
+ );
+ }
+ this.detachDatasource = undefined;
  });
  }
 
@@ -218,4 +240,61 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  }
+
+ /**
+ * Lane 3 — the datasource ADMIN SERVICE's `datasource.mutated` fan-out
+ * (#13805): the driver registry adopting the family lane 2 established.
+ *
+ * Duck-typed exactly like lanes 1 and 2 feature-detect their seams: this
+ * package must not depend on `@objectstack/service-datasource`, and
+ * `@objectstack/objectql` — the driver registry's owner — is handed no
+ * bus at all; the admin service publishes on the write doors it already
+ * owns and converges its pools through the seams it already injects.
+ *
+ * Guarded on {@link isInProcessClusterDriver} from birth, like lane 2: the
+ * in-process memory driver fans out to nobody, and on that driver a single
+ * replica's behaviour stays byte-identical to the pre-bridge one.
+ */
+ private attachDatasourceLane(ctx: PluginContext, cluster: IClusterService): void {
+ let admin: unknown;
+ try {
+ admin = ctx.getService<unknown>('datasource-admin');
+ } catch {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: no "datasource-admin" service registered, skipping datasource fan-out',
+ );
+ return;
+ }
+
+ const attach = (admin as { attachDatasourceMutationPubSub?: unknown })
+ .attachDatasourceMutationPubSub;
+ if (typeof attach !== 'function') {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: datasource-admin service does not expose attachDatasourceMutationPubSub(), skipping datasource fan-out',
+ );
+ return;
+ }
+
+ if (isInProcessClusterDriver(cluster.driver)) {
+ ctx.logger.debug(
+ `MetadataClusterBridgePlugin: cluster driver "${cluster.driver}" is in-process; datasource fan-out has no peers to reach, skipping`,
+ );
+ return;
+ }
+
+ try {
+ this.detachDatasource = (attach as (
+ pubsub: IClusterService['pubsub'],
+ nodeId: string,
+ ) => () => void).call(admin, cluster.pubsub, cluster.nodeId);
+ ctx.logger.info(
+ `MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=${cluster.nodeId})`,
+ );
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ err as Error,
+ );
+ }
+ }
 }

.changeset/datasource-mutation-cluster-fanout.md

@@ -0,0 +1,63 @@
+---
+"@objectstack/service-datasource": minor
+"@objectstack/service-cluster": minor
+---
+
+feat(service-datasource,service-cluster): fan datasource record writes out to peer replicas — a deleted datasource no longer keeps draining `/api/v1/ready` on every replica that did not serve the DELETE (#13805)
+
+Measured on a live 3-replica EE deployment: the ObjectQL DRIVER registry had
+no cluster propagation in either direction. Each replica filled it at boot
+from the shared datasource records and mutated it only for the writes IT
+served, so after `DELETE /api/v1/datasources/:name` only the replica that
+served the DELETE evicted the stuck driver (#13578's door) — the other N-1
+kept it, and `/api/v1/ready` kept answering 503 there, until restart. A
+datasource created through one replica likewise had no pool on any other
+until restart.
+
+Maintainer-ruled design (2026-09-01): the driver registry adopts the same
+cluster-invalidation family `metadata.mutated` (#13331) established — no
+second propagation mechanism, no bespoke poll loop, and no delete-only
+broadcast (that would have made delete more cluster-aware than create, a new
+asymmetry rather than a repair).
+
+- **Symmetric publisher at the three write doors.** `DatasourceAdminService`
+ now publishes the record's ADDRESS on a new cluster channel
+ `datasource.mutated` (`DATASOURCE_MUTATION_CLUSTER_CHANNEL`, payload
+ `ClusterDatasourceMutationPayload` — `{ originNode?, name }`) after
+ `createDatasource`, `updateDatasource` and `removeDatasource`. Fire-and-
+ forget: a publish failure never fails the write it announces.
+ `migrateCredential` does not publish — it leaves the live pool alone by
+ design, on every replica alike.
+- **Peers converge from their own read of the SHARED record.** On receipt a
+ replica re-reads the durable `sys_metadata` row for that name — the same
+ store its boot rehydration reads, not its per-replica metadata registry —
+ and converges its live pool through the seams it already owns: builds what
+ is missing, rebuilds in place what changed (`reregisterPool`, keeping the
+ old pool on failure exactly as the serving replica's update path does),
+ evicts what is gone (`unregisterPool` → the #13578 eviction door), and
+ leaves a matching pool untouched. The payload is a signal, never trusted
+ content, so a duplicate or re-ordered delivery converges to the same pool
+ state by construction — which is what makes a replayed create safe without
+ any new idempotency machinery. A name the replica never pooled is left
+ alone, so a stray signal cannot reach a code-defined pool.
+- **New attach seam, mirrored from the shipped bridges.**
+ `DatasourceAdminService.attachDatasourceMutationPubSub(pubsub, nodeId)` —
+ idempotent on the `(pubsub, nodeId)` pair, loopback suppression via
+ `originNode`, shaped after the protocol's `attachMetadataMutationPubSub()`.
+ Only `IPubSub` from `@objectstack/spec/contracts` crosses it:
+ `@objectstack/service-datasource` takes no dependency on the cluster
+ service, and `@objectstack/objectql` — the registry's owner — is handed no
+ bus. The host wires the receive half through a new optional
+ `DatasourceAdminServiceConfig.convergePool` seam; `DatasourceAdminServicePlugin`
+ supplies it.
+- **`MetadataClusterBridgePlugin` gains a third, independent lane** that
+ late-binds the seam at `kernel:ready` beside the metadata-service and
+ protocol lanes, duck-typed on the `datasource-admin` service. It skips the
+ in-process memory driver (nothing to fan out to), the guard the other lanes
+ carry, so a single-replica boot behaves byte-identically to before.
+
+No shipped driver exceeds at-most-once delivery, so a lost message still
+degrades to the pre-existing bound (the next boot's full rehydration); this
+channel narrows the window from "until every replica restarts" to one network
+hop. The `/api/v1/meta/datasource` metadata registry's own cross-replica
+coherence (#13609) is a different sink and is not touched here.

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.test.ts

@@ -48,17 +48,34 @@ interface HarnessOptions {
  * `'real'` (exposes attachMetadataMutationPubSub).
  */
  protocol?: 'none' | 'bare' | 'real';
+ /**
+ * [#13805] The `datasource-admin` slot: `'none'` (getService throws — the
+ * default, so the #13331 cases above read exactly as they did), `'bare'`
+ * (present, no attachDatasourceMutationPubSub — an older implementation),
+ * or `'real'` (exposes attachDatasourceMutationPubSub).
+ */
+ datasourceAdmin?: 'none' | 'bare' | 'real';
+ /** When true, the datasource-admin seam throws on attach. */
+ datasourceAttachThrows?: boolean;
 }
 
 function makeHarness(opts: HarnessOptions = {}) {
- const { driver = 'redis', metadata = 'fallback', protocol = 'real' } = opts;
+ const {
+ driver = 'redis', metadata = 'fallback', protocol = 'real',
+ datasourceAdmin = 'none', datasourceAttachThrows = false,
+ } = opts;
 
  const logger = { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() };
 
  const detachMetadata = vi.fn();
  const attachMetadata = vi.fn((_pubsub: unknown, _nodeId: string) => detachMetadata);
  const detachMutation = vi.fn();
  const attachMutation = vi.fn((_pubsub: unknown, _nodeId: string) => detachMutation);
+ const detachDatasource = vi.fn();
+ const attachDatasource = vi.fn((_pubsub: unknown, _nodeId: string) => {
+ if (datasourceAttachThrows) throw new Error('datasource attach exploded');
+ return detachDatasource;
+ });
 
  const pubsub = { publish: vi.fn(), subscribe: vi.fn(), close: vi.fn() };
  const cluster =
@@ -74,6 +91,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  protocol === 'none' ? undefined
  : protocol === 'bare' ? { saveMetaItem: vi.fn() }
  : { attachMetadataMutationPubSub: attachMutation };
+ const datasourceAdminService =
+ datasourceAdmin === 'none' ? undefined
+ : datasourceAdmin === 'bare' ? { listDatasources: vi.fn() }
+ : { attachDatasourceMutationPubSub: attachDatasource };
 
  const hooks = new Map<string, Array<() => Promise<void> | void>>();
  const ctx = {
@@ -96,6 +117,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  if (!protocolService) throw new Error('service not found: protocol');
  return protocolService;
  }
+ if (name === 'datasource-admin') {
+ if (!datasourceAdminService) throw new Error('service not found: datasource-admin');
+ return datasourceAdminService;
+ }
  throw new Error(`service not found: ${name}`);
  },
  } as unknown as PluginContext;
@@ -107,6 +132,7 @@ function makeHarness(opts: HarnessOptions = {}) {
  return {
  ctx, logger, fire, pubsub,
  attachMetadata, detachMetadata, attachMutation, detachMutation,
+ attachDatasource, detachDatasource,
  };
 }
 
@@ -280,3 +306,109 @@ describe('[#14021] lane 1 — an in-process bus must not be reported as “bridg
  expect(h.logger.error).not.toHaveBeenCalled();
  });
 });
+
+describe('[#13805] lane 3 — the datasource admin service’s datasource.mutated fan-out', () => {
+ it('⭐ attaches on a cross-process driver and reports it — independently of lanes 1 and 2', async () => {
+ // The shipped EE shape again, one owner over: no manager-backed
+ // metadata slot, no protocol seam, and a real datasource-admin service.
+ // Lane 3 must attach exactly there, with nothing from the other two
+ // lanes taking it down.
+ const h = makeHarness({ driver: 'redis', metadata: 'none', protocol: 'none', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ // Asserted VERBATIM, like lane 1's and lane 2's lines: the wording is
+ // what an operator reads as "datasource fan-out is on".
+ expect(infoLines(h)).toContain(
+ 'MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=node-a)',
+ );
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('all three lanes attach together when every owner exposes its seam', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachMutation).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(warnLines(h)).toEqual([]);
+ });
+
+ it('skips attach on the in-process memory driver — no peers to reach, nothing said above debug', async () => {
+ const h = makeHarness({ driver: 'memory', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ // The guard lanes 1 and 2 carry, from birth: on the memory driver a
+ // single replica's behaviour stays byte-identical to the pre-bridge
+ // one — no subscription, no publisher, no "bridged" claim.
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ expect(
+ debugLines(h).some((l) => l.includes('is in-process') && l.includes('datasource fan-out')),
+ ).toBe(true);
+ });
+
+ it('skips quietly when no datasource-admin service is registered', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'none' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ expect(warnLines(h).some((l) => l.includes('datasource'))).toBe(false);
+ });
+
+ it('skips quietly when the service does not expose the seam', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'bare' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('no cluster service at all skips lane 3 too', async () => {
+ const h = makeHarness({ driver: null, datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ });
+
+ it('a throwing attach is reported and does not take the other lanes down', async () => {
+ const h = makeHarness({
+ driver: 'redis', metadata: 'manager', protocol: 'real',
+ datasourceAdmin: 'real', datasourceAttachThrows: true,
+ });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledTimes(1);
+ expect(h.attachMutation).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalledWith(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ expect.any(Error),
+ );
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ });
+
+ it('kernel:shutdown detaches lane 3, and a throwing lane-2 detach does not strand it', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ h.detachMutation.mockImplementation(() => { throw new Error('detach exploded'); });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+ await h.fire('kernel:shutdown');
+
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalled();
+
+ // Idempotent: a second shutdown does not detach twice.
+ await h.fire('kernel:shutdown');
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ });
+});

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts

@@ -15,8 +15,8 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * state-owner packages only need the `IPubSub` interface, which lives in
  * `@objectstack/spec/contracts`.
  *
- * TWO lanes, late-bound independently at `kernel:ready`, because the state
- * that goes stale lives in two different owners (#13331):
+ * THREE lanes, late-bound independently at `kernel:ready`, because the state
+ * that goes stale lives in three different owners (#13331, #13805):
  *
  * 1. **Metadata service** (`attachClusterPubSub()` — `metadata.changed`):
  * replays watch events into peer `MetadataManager` caches
@@ -34,6 +34,15 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * 67×201 / 133×404). The lanes are independent on purpose: the boot
  * shape that lacks lane 1 (host-config, fallback metadata slot) is
  * exactly the shipped EE shape that needs lane 2.
+ * 3. **Datasource admin service** (`attachDatasourceMutationPubSub()` —
+ * `datasource.mutated`): fans a datasource create / update / delete out
+ * to peers, which converge their ObjectQL DRIVER registry from their OWN
+ * read of the shared datasource record. Lane 2's family, adopted by the
+ * driver registry (#13805, ruled 2026-09-01 — the same bridge shape, a
+ * symmetric signal, no second propagation mechanism): without it a
+ * `DELETE /api/v1/datasources/:name` recovered `/api/v1/ready` on the
+ * one replica that served it, and every other replica kept the stuck
+ * driver until restart.
  *
  * Activates each lane only when the cluster service and that lane's state
  * owner are present and expose the seam. Late binding is achieved via the
@@ -42,7 +51,9 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * Channels: `metadata.changed` — payload shape defined by
  * `ClusterMetadataChangedPayload` in `@objectstack/metadata`;
  * `metadata.mutated` — payload shape defined by
- * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`.
+ * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`;
+ * `datasource.mutated` — payload shape defined by
+ * `ClusterDatasourceMutationPayload` in `@objectstack/service-datasource`.
  *
  * See `content/docs/kernel/cluster.mdx` §5.
  */
@@ -53,6 +64,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
 
  private detach?: () => void;
  private detachMutation?: () => void;
+ private detachDatasource?: () => void;
 
  async init(ctx: PluginContext): Promise<void> {
  ctx.hook('kernel:ready', async () => {
@@ -67,6 +79,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
  }
  this.attachMetadataServiceLane(ctx, cluster);
  this.attachProtocolLane(ctx, cluster);
+ this.attachDatasourceLane(ctx, cluster);
  });
 
  ctx.hook('kernel:shutdown', async () => {
@@ -88,6 +101,15 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  this.detachMutation = undefined;
+ try {
+ this.detachDatasource?.();
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane detach error',
+ err as Error,
+ );
+ }
+ this.detachDatasource = undefined;
  });
  }
 
@@ -218,4 +240,61 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  }
+
+ /**
+ * Lane 3 — the datasource ADMIN SERVICE's `datasource.mutated` fan-out
+ * (#13805): the driver registry adopting the family lane 2 established.
+ *
+ * Duck-typed exactly like lanes 1 and 2 feature-detect their seams: this
+ * package must not depend on `@objectstack/service-datasource`, and
+ * `@objectstack/objectql` — the driver registry's owner — is handed no
+ * bus at all; the admin service publishes on the write doors it already
+ * owns and converges its pools through the seams it already injects.
+ *
+ * Guarded on {@link isInProcessClusterDriver} from birth, like lane 2: the
+ * in-process memory driver fans out to nobody, and on that driver a single
+ * replica's behaviour stays byte-identical to the pre-bridge one.
+ */
+ private attachDatasourceLane(ctx: PluginContext, cluster: IClusterService): void {
+ let admin: unknown;
+ try {
+ admin = ctx.getService<unknown>('datasource-admin');
+ } catch {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: no "datasource-admin" service registered, skipping datasource fan-out',
+ );
+ return;
+ }
+
+ const attach = (admin as { attachDatasourceMutationPubSub?: unknown })
+ .attachDatasourceMutationPubSub;
+ if (typeof attach !== 'function') {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: datasource-admin service does not expose attachDatasourceMutationPubSub(), skipping datasource fan-out',
+ );
+ return;
+ }
+
+ if (isInProcessClusterDriver(cluster.driver)) {
+ ctx.logger.debug(
+ `MetadataClusterBridgePlugin: cluster driver "${cluster.driver}" is in-process; datasource fan-out has no peers to reach, skipping`,
+ );
+ return;
+ }
+
+ try {
+ this.detachDatasource = (attach as (
+ pubsub: IClusterService['pubsub'],
+ nodeId: string,
+ ) => () => void).call(admin, cluster.pubsub, cluster.nodeId);
+ ctx.logger.info(
+ `MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=${cluster.nodeId})`,
+ );
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ err as Error,
+ );
+ }
+ }
 }

.changeset/datasource-mutation-cluster-fanout.md

@@ -0,0 +1,63 @@
+---
+"@objectstack/service-datasource": minor
+"@objectstack/service-cluster": minor
+---
+
+feat(service-datasource,service-cluster): fan datasource record writes out to peer replicas — a deleted datasource no longer keeps draining `/api/v1/ready` on every replica that did not serve the DELETE (#13805)
+
+Measured on a live 3-replica EE deployment: the ObjectQL DRIVER registry had
+no cluster propagation in either direction. Each replica filled it at boot
+from the shared datasource records and mutated it only for the writes IT
+served, so after `DELETE /api/v1/datasources/:name` only the replica that
+served the DELETE evicted the stuck driver (#13578's door) — the other N-1
+kept it, and `/api/v1/ready` kept answering 503 there, until restart. A
+datasource created through one replica likewise had no pool on any other
+until restart.
+
+Maintainer-ruled design (2026-09-01): the driver registry adopts the same
+cluster-invalidation family `metadata.mutated` (#13331) established — no
+second propagation mechanism, no bespoke poll loop, and no delete-only
+broadcast (that would have made delete more cluster-aware than create, a new
+asymmetry rather than a repair).
+
+- **Symmetric publisher at the three write doors.** `DatasourceAdminService`
+ now publishes the record's ADDRESS on a new cluster channel
+ `datasource.mutated` (`DATASOURCE_MUTATION_CLUSTER_CHANNEL`, payload
+ `ClusterDatasourceMutationPayload` — `{ originNode?, name }`) after
+ `createDatasource`, `updateDatasource` and `removeDatasource`. Fire-and-
+ forget: a publish failure never fails the write it announces.
+ `migrateCredential` does not publish — it leaves the live pool alone by
+ design, on every replica alike.
+- **Peers converge from their own read of the SHARED record.** On receipt a
+ replica re-reads the durable `sys_metadata` row for that name — the same
+ store its boot rehydration reads, not its per-replica metadata registry —
+ and converges its live pool through the seams it already owns: builds what
+ is missing, rebuilds in place what changed (`reregisterPool`, keeping the
+ old pool on failure exactly as the serving replica's update path does),
+ evicts what is gone (`unregisterPool` → the #13578 eviction door), and
+ leaves a matching pool untouched. The payload is a signal, never trusted
+ content, so a duplicate or re-ordered delivery converges to the same pool
+ state by construction — which is what makes a replayed create safe without
+ any new idempotency machinery. A name the replica never pooled is left
+ alone, so a stray signal cannot reach a code-defined pool.
+- **New attach seam, mirrored from the shipped bridges.**
+ `DatasourceAdminService.attachDatasourceMutationPubSub(pubsub, nodeId)` —
+ idempotent on the `(pubsub, nodeId)` pair, loopback suppression via
+ `originNode`, shaped after the protocol's `attachMetadataMutationPubSub()`.
+ Only `IPubSub` from `@objectstack/spec/contracts` crosses it:
+ `@objectstack/service-datasource` takes no dependency on the cluster
+ service, and `@objectstack/objectql` — the registry's owner — is handed no
+ bus. The host wires the receive half through a new optional
+ `DatasourceAdminServiceConfig.convergePool` seam; `DatasourceAdminServicePlugin`
+ supplies it.
+- **`MetadataClusterBridgePlugin` gains a third, independent lane** that
+ late-binds the seam at `kernel:ready` beside the metadata-service and
+ protocol lanes, duck-typed on the `datasource-admin` service. It skips the
+ in-process memory driver (nothing to fan out to), the guard the other lanes
+ carry, so a single-replica boot behaves byte-identically to before.
+
+No shipped driver exceeds at-most-once delivery, so a lost message still
+degrades to the pre-existing bound (the next boot's full rehydration); this
+channel narrows the window from "until every replica restarts" to one network
+hop. The `/api/v1/meta/datasource` metadata registry's own cross-replica
+coherence (#13609) is a different sink and is not touched here.

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.test.ts

@@ -48,17 +48,34 @@ interface HarnessOptions {
  * `'real'` (exposes attachMetadataMutationPubSub).
  */
  protocol?: 'none' | 'bare' | 'real';
+ /**
+ * [#13805] The `datasource-admin` slot: `'none'` (getService throws — the
+ * default, so the #13331 cases above read exactly as they did), `'bare'`
+ * (present, no attachDatasourceMutationPubSub — an older implementation),
+ * or `'real'` (exposes attachDatasourceMutationPubSub).
+ */
+ datasourceAdmin?: 'none' | 'bare' | 'real';
+ /** When true, the datasource-admin seam throws on attach. */
+ datasourceAttachThrows?: boolean;
 }
 
 function makeHarness(opts: HarnessOptions = {}) {
- const { driver = 'redis', metadata = 'fallback', protocol = 'real' } = opts;
+ const {
+ driver = 'redis', metadata = 'fallback', protocol = 'real',
+ datasourceAdmin = 'none', datasourceAttachThrows = false,
+ } = opts;
 
  const logger = { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() };
 
  const detachMetadata = vi.fn();
  const attachMetadata = vi.fn((_pubsub: unknown, _nodeId: string) => detachMetadata);
  const detachMutation = vi.fn();
  const attachMutation = vi.fn((_pubsub: unknown, _nodeId: string) => detachMutation);
+ const detachDatasource = vi.fn();
+ const attachDatasource = vi.fn((_pubsub: unknown, _nodeId: string) => {
+ if (datasourceAttachThrows) throw new Error('datasource attach exploded');
+ return detachDatasource;
+ });
 
  const pubsub = { publish: vi.fn(), subscribe: vi.fn(), close: vi.fn() };
  const cluster =
@@ -74,6 +91,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  protocol === 'none' ? undefined
  : protocol === 'bare' ? { saveMetaItem: vi.fn() }
  : { attachMetadataMutationPubSub: attachMutation };
+ const datasourceAdminService =
+ datasourceAdmin === 'none' ? undefined
+ : datasourceAdmin === 'bare' ? { listDatasources: vi.fn() }
+ : { attachDatasourceMutationPubSub: attachDatasource };
 
  const hooks = new Map<string, Array<() => Promise<void> | void>>();
  const ctx = {
@@ -96,6 +117,10 @@ function makeHarness(opts: HarnessOptions = {}) {
  if (!protocolService) throw new Error('service not found: protocol');
  return protocolService;
  }
+ if (name === 'datasource-admin') {
+ if (!datasourceAdminService) throw new Error('service not found: datasource-admin');
+ return datasourceAdminService;
+ }
  throw new Error(`service not found: ${name}`);
  },
  } as unknown as PluginContext;
@@ -107,6 +132,7 @@ function makeHarness(opts: HarnessOptions = {}) {
  return {
  ctx, logger, fire, pubsub,
  attachMetadata, detachMetadata, attachMutation, detachMutation,
+ attachDatasource, detachDatasource,
  };
 }
 
@@ -280,3 +306,109 @@ describe('[#14021] lane 1 — an in-process bus must not be reported as “bridg
  expect(h.logger.error).not.toHaveBeenCalled();
  });
 });
+
+describe('[#13805] lane 3 — the datasource admin service’s datasource.mutated fan-out', () => {
+ it('⭐ attaches on a cross-process driver and reports it — independently of lanes 1 and 2', async () => {
+ // The shipped EE shape again, one owner over: no manager-backed
+ // metadata slot, no protocol seam, and a real datasource-admin service.
+ // Lane 3 must attach exactly there, with nothing from the other two
+ // lanes taking it down.
+ const h = makeHarness({ driver: 'redis', metadata: 'none', protocol: 'none', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ // Asserted VERBATIM, like lane 1's and lane 2's lines: the wording is
+ // what an operator reads as "datasource fan-out is on".
+ expect(infoLines(h)).toContain(
+ 'MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=node-a)',
+ );
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('all three lanes attach together when every owner exposes its seam', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachMutation).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(h.attachDatasource).toHaveBeenCalledWith(h.pubsub, 'node-a');
+ expect(warnLines(h)).toEqual([]);
+ });
+
+ it('skips attach on the in-process memory driver — no peers to reach, nothing said above debug', async () => {
+ const h = makeHarness({ driver: 'memory', datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ // The guard lanes 1 and 2 carry, from birth: on the memory driver a
+ // single replica's behaviour stays byte-identical to the pre-bridge
+ // one — no subscription, no publisher, no "bridged" claim.
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ expect(
+ debugLines(h).some((l) => l.includes('is in-process') && l.includes('datasource fan-out')),
+ ).toBe(true);
+ });
+
+ it('skips quietly when no datasource-admin service is registered', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'none' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ expect(warnLines(h).some((l) => l.includes('datasource'))).toBe(false);
+ });
+
+ it('skips quietly when the service does not expose the seam', async () => {
+ const h = makeHarness({ driver: 'redis', datasourceAdmin: 'bare' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ expect(h.logger.error).not.toHaveBeenCalled();
+ });
+
+ it('no cluster service at all skips lane 3 too', async () => {
+ const h = makeHarness({ driver: null, datasourceAdmin: 'real' });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachDatasource).not.toHaveBeenCalled();
+ });
+
+ it('a throwing attach is reported and does not take the other lanes down', async () => {
+ const h = makeHarness({
+ driver: 'redis', metadata: 'manager', protocol: 'real',
+ datasourceAdmin: 'real', datasourceAttachThrows: true,
+ });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+
+ expect(h.attachMetadata).toHaveBeenCalledTimes(1);
+ expect(h.attachMutation).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalledWith(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ expect.any(Error),
+ );
+ expect(infoLines(h).some((l) => l.includes('datasource.mutated'))).toBe(false);
+ });
+
+ it('kernel:shutdown detaches lane 3, and a throwing lane-2 detach does not strand it', async () => {
+ const h = makeHarness({ driver: 'redis', metadata: 'manager', protocol: 'real', datasourceAdmin: 'real' });
+ h.detachMutation.mockImplementation(() => { throw new Error('detach exploded'); });
+ await new MetadataClusterBridgePlugin().init(h.ctx);
+ await h.fire('kernel:ready');
+ await h.fire('kernel:shutdown');
+
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ expect(h.logger.error).toHaveBeenCalled();
+
+ // Idempotent: a second shutdown does not detach twice.
+ await h.fire('kernel:shutdown');
+ expect(h.detachDatasource).toHaveBeenCalledTimes(1);
+ });
+});

packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts

@@ -15,8 +15,8 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * state-owner packages only need the `IPubSub` interface, which lives in
  * `@objectstack/spec/contracts`.
  *
- * TWO lanes, late-bound independently at `kernel:ready`, because the state
- * that goes stale lives in two different owners (#13331):
+ * THREE lanes, late-bound independently at `kernel:ready`, because the state
+ * that goes stale lives in three different owners (#13331, #13805):
  *
  * 1. **Metadata service** (`attachClusterPubSub()` — `metadata.changed`):
  * replays watch events into peer `MetadataManager` caches
@@ -34,6 +34,15 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * 67×201 / 133×404). The lanes are independent on purpose: the boot
  * shape that lacks lane 1 (host-config, fallback metadata slot) is
  * exactly the shipped EE shape that needs lane 2.
+ * 3. **Datasource admin service** (`attachDatasourceMutationPubSub()` —
+ * `datasource.mutated`): fans a datasource create / update / delete out
+ * to peers, which converge their ObjectQL DRIVER registry from their OWN
+ * read of the shared datasource record. Lane 2's family, adopted by the
+ * driver registry (#13805, ruled 2026-09-01 — the same bridge shape, a
+ * symmetric signal, no second propagation mechanism): without it a
+ * `DELETE /api/v1/datasources/:name` recovered `/api/v1/ready` on the
+ * one replica that served it, and every other replica kept the stuck
+ * driver until restart.
  *
  * Activates each lane only when the cluster service and that lane's state
  * owner are present and expose the seam. Late binding is achieved via the
@@ -42,7 +51,9 @@ import { isInProcessClusterDriver } from './split-brain-guard.js';
  * Channels: `metadata.changed` — payload shape defined by
  * `ClusterMetadataChangedPayload` in `@objectstack/metadata`;
  * `metadata.mutated` — payload shape defined by
- * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`.
+ * `ClusterMetadataMutationPayload` in `@objectstack/metadata-protocol`;
+ * `datasource.mutated` — payload shape defined by
+ * `ClusterDatasourceMutationPayload` in `@objectstack/service-datasource`.
  *
  * See `content/docs/kernel/cluster.mdx` §5.
  */
@@ -53,6 +64,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
 
  private detach?: () => void;
  private detachMutation?: () => void;
+ private detachDatasource?: () => void;
 
  async init(ctx: PluginContext): Promise<void> {
  ctx.hook('kernel:ready', async () => {
@@ -67,6 +79,7 @@ export class MetadataClusterBridgePlugin implements Plugin {
  }
  this.attachMetadataServiceLane(ctx, cluster);
  this.attachProtocolLane(ctx, cluster);
+ this.attachDatasourceLane(ctx, cluster);
  });
 
  ctx.hook('kernel:shutdown', async () => {
@@ -88,6 +101,15 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  this.detachMutation = undefined;
+ try {
+ this.detachDatasource?.();
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane detach error',
+ err as Error,
+ );
+ }
+ this.detachDatasource = undefined;
  });
  }
 
@@ -218,4 +240,61 @@ export class MetadataClusterBridgePlugin implements Plugin {
  );
  }
  }
+
+ /**
+ * Lane 3 — the datasource ADMIN SERVICE's `datasource.mutated` fan-out
+ * (#13805): the driver registry adopting the family lane 2 established.
+ *
+ * Duck-typed exactly like lanes 1 and 2 feature-detect their seams: this
+ * package must not depend on `@objectstack/service-datasource`, and
+ * `@objectstack/objectql` — the driver registry's owner — is handed no
+ * bus at all; the admin service publishes on the write doors it already
+ * owns and converges its pools through the seams it already injects.
+ *
+ * Guarded on {@link isInProcessClusterDriver} from birth, like lane 2: the
+ * in-process memory driver fans out to nobody, and on that driver a single
+ * replica's behaviour stays byte-identical to the pre-bridge one.
+ */
+ private attachDatasourceLane(ctx: PluginContext, cluster: IClusterService): void {
+ let admin: unknown;
+ try {
+ admin = ctx.getService<unknown>('datasource-admin');
+ } catch {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: no "datasource-admin" service registered, skipping datasource fan-out',
+ );
+ return;
+ }
+
+ const attach = (admin as { attachDatasourceMutationPubSub?: unknown })
+ .attachDatasourceMutationPubSub;
+ if (typeof attach !== 'function') {
+ ctx.logger.debug(
+ 'MetadataClusterBridgePlugin: datasource-admin service does not expose attachDatasourceMutationPubSub(), skipping datasource fan-out',
+ );
+ return;
+ }
+
+ if (isInProcessClusterDriver(cluster.driver)) {
+ ctx.logger.debug(
+ `MetadataClusterBridgePlugin: cluster driver "${cluster.driver}" is in-process; datasource fan-out has no peers to reach, skipping`,
+ );
+ return;
+ }
+
+ try {
+ this.detachDatasource = (attach as (
+ pubsub: IClusterService['pubsub'],
+ nodeId: string,
+ ) => () => void).call(admin, cluster.pubsub, cluster.nodeId);
+ ctx.logger.info(
+ `MetadataClusterBridgePlugin: bridged datasource.mutated → cluster.pubsub (node=${cluster.nodeId})`,
+ );
+ } catch (err) {
+ ctx.logger.error(
+ 'MetadataClusterBridgePlugin: datasource-lane attach failed',
+ err as Error,
+ );
+ }
+ }
 }