Surface a licensed max_nodes oversell to operators as telemetry (visibility only)

.changeset/lucky-donkeys-vanish.md

@@ -0,0 +1,35 @@
+---
+"@objectstack/observability": minor
+"@objectstack/cli": minor
+---
+
+Surface a licensed `max_nodes` oversell to operators as telemetry
+
+`os serve` already warned loudly at boot when `OS_CLUSTER_REPLICAS` declared more
+nodes than the licence gate admits, but that warning existed only in one
+process's startup output: an operator who scaled past their cap three weeks ago
+had no way to ask the question today, and no way to alert on it. The same
+advisory verdict is now also published through the deployment's configured
+metrics backend, so it reaches the place operators already look.
+
+Three names join `SEMCONV` in `@objectstack/observability`, emitted once per boot
+by `os serve` when a remote cluster driver is configured, each labelled with the
+gate's own verdict vocabulary (`admitted` / `capped` / `refused`):
+
+- `cluster_declared_nodes` (gauge) — the replica count the operator **declared**;
+- `cluster_admitted_nodes` (gauge) — how many of them the licence **admits**;
+- `cluster_node_cap_verdicts_total` (counter) — one increment per process boot
+ that consulted the gate, so an alert stays writable after a one-shot gauge has
+ aged out of a push-based backend.
+
+**Visibility only — the cap remains advisory and nothing is refused.** The gate is
+consulted once per process at boot, every replica computes the same verdict, and
+none can know whether it is one of the admitted ones, so all of them still join.
+The names say so on purpose: this process has no cluster membership view at all,
+so a series called `cluster_nodes` or `cluster_active_nodes` would be a false
+statement dressed as telemetry. Nothing here counts peers, and no accept/reject
+behaviour changed.
+
+Absence is meaningful rather than an instrumentation gap: a single-node
+deployment never consults the gate and emits nothing, and an emission also needs
+a metrics backend configured via `OS_OBS_EXPORTER`.

packages/cli/src/commands/serve-multi-node-cap-advisory.pin.test.ts

@@ -214,3 +214,82 @@ describe('the shape assertions ignore a comment that quotes the old call (#10514
  expect(maskComments(regressed)).not.toMatch(/checkMultiNodeAllowed\(\s*[^)\s]/);
  });
 });
+
+/**
+ * THE SECOND PIN: serve's declared-count normalization still matches the gate's
+ * own, byte for byte modulo comments and whitespace.
+ *
+ * The telemetry reading (#12667) has to report the count the operator DECLARED,
+ * and the resolved verdict cannot give it back: `admitted` is `min(cap,
+ * wanted)`, so `{admitted: 3, refused: 0}` is produced BOTH by "declared 3 under
+ * a cap of 5" and by "declared nothing under a cap of 3". The declaration is
+ * only knowable from `OS_CLUSTER_REPLICAS`, so `serve.ts` normalizes that value
+ * itself — and a normalization that disagrees with the gate's would publish a
+ * declaration the gate never saw (a `0` or a `2.7` the gate had already thrown
+ * away as "not declared").
+ *
+ * Both sides are read from the file that OWNS each, for the same reason the
+ * shape pin above is: an expected rule re-typed here would just relocate the
+ * divergence into this file, where it would be equally silent.
+ */
+
+/**
+ * The brace-matched body of a top-level `function <name>(…) … { … }`, with
+ * comments blanked and whitespace collapsed, so two implementations can be
+ * compared on what they DO.
+ *
+ * The first `{` after the declaration is taken as the body opener — true for
+ * both functions compared below (neither has an object type or a destructured
+ * parameter in its signature); a signature that grows one would need the scan
+ * to skip the parameter list first, and would fail loudly here rather than
+ * quietly compare the wrong span.
+ */
+function functionBody(source: string, name: string): string {
+ const masked = maskComments(source);
+ const at = masked.indexOf(`function ${name}(`);
+ expect(at, `function ${name} not found — did it move or get renamed?`).toBeGreaterThan(-1);
+
+ const open = masked.indexOf('{', at);
+ expect(open, `function ${name} has no body brace`).toBeGreaterThan(-1);
+
+ let depth = 1;
+ let i = open + 1;
+ for (; i < masked.length && depth > 0; i++) {
+ if (masked[i] === '{') depth++;
+ else if (masked[i] === '}') depth--;
+ }
+ expect(depth, `function ${name} body is unbalanced`).toBe(0);
+
+ return masked.slice(open + 1, i - 1).replace(/\s+/g, ' ').trim();
+}
+
+describe('os serve ↔ multi-node gate: the declared-count rule', () => {
+ it("serve's `normalizeDeclaredNodeCount` still mirrors the gate's `normalizeCount`", () => {
+ const producer = functionBody(GATE_SOURCE, 'normalizeCount');
+ const consumer = functionBody(SERVE_SOURCE, 'normalizeDeclaredNodeCount');
+
+ // Guard the extractor: two empty bodies would agree vacuously.
+ expect(producer).toContain('Number.isFinite');
+ expect(producer).toContain('Math.floor');
+ expect(producer.length).toBeGreaterThan(40);
+
+ expect(
+ consumer,
+ 'packages/services/service-cluster/src/multi-node-gate.ts changed how it decides '
+ + 'whether a requested node count counts as DECLARED. serve.ts mirrors that rule by '
+ + 'hand (no static dependency) so its operator telemetry reports the same declaration '
+ + 'the gate saw — update `normalizeDeclaredNodeCount` in '
+ + 'packages/cli/src/commands/serve.ts to match.',
+ ).toEqual(producer);
+ });
+
+ it('the telemetry reading is fed from the DECLARED env var, not from a count of anything', () => {
+ // The whole card turns on this: `OS_CLUSTER_REPLICAS` is what the operator
+ // wrote, identical in every replica. There is no membership count to read
+ // instead, and a future edit that reached for one would be publishing a
+ // number this process cannot know.
+ expect(MASKED_SERVE_SOURCE).toMatch(
+ /describeMultiNodeCapTelemetry\(\s*verdict\s*,\s*Number\(process\.env\.OS_CLUSTER_REPLICAS\)\s*\)/,
+ );
+ });
+});

packages/cli/src/commands/serve-multi-node-cap-telemetry.test.ts

@@ -0,0 +1,219 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * What an OPERATOR sees when `os serve` publishes the multi-node licence
+ * reading as telemetry (#12667 — maintainer ruling 2026-08-27, verbatim
+ * 「其他接受」, adopting option C on the `max_nodes` decision: make a licensed
+ * oversell VISIBLE; the atomic slot-claim enforcement mechanism is deliberately
+ * not built).
+ *
+ * These assertions are about the published SURFACE — the exact series, values
+ * and labels a dashboard receives — not about a value having been computed.
+ * The distinction matters here more than usual: the deliverable of this card is
+ * a reading that is honest about what the process can and cannot know, so a
+ * test that only checked "a number came out" would pass over every regression
+ * worth catching.
+ *
+ * ⚠️ The three facts that make this visibility and not enforcement, re-measured
+ * on the tree this landed against:
+ *
+ * - the gate is consulted ONCE PER PROCESS at boot (`serve.ts`);
+ * - there is NO cluster membership view — `generateNodeId` is random per
+ * process and there is no join/leave registry (`cluster.ts`);
+ * - `OS_CLUSTER_REPLICAS` is an operator-DECLARED count, identical in every
+ * replica (`split-brain-guard.ts`).
+ *
+ * So nothing in the process knows how many peers exist, and the surface must
+ * not read as though it does. The "honest naming" block at the bottom pins
+ * that, because it is the regression most likely to arrive later as a helpful
+ * wording change.
+ */
+
+import { describe, it, expect } from 'vitest';
+import {
+ describeMultiNodeCapTelemetry,
+ type MultiNodeCapMetric,
+ type MultiNodeGateVerdict,
+} from './serve.js';
+
+/**
+ * The four verdicts the producer can hand this consumer, spelled the way
+ * `checkMultiNodeAllowed` builds them (`multi-node-gate.ts`). Same fixtures as
+ * `serve-multi-node-cap-advisory.test.ts`, deliberately: the two reaches of one
+ * advisory must be tested against one set of inputs, or they can drift into
+ * telling an operator two different stories about the same boot.
+ */
+const VERDICTS = {
+ /** No gate registered, or an allowing gate that declared no cap. */
+ uncapped: { allowed: true, refused: 0, capped: false },
+ /** A cap exists and the declared topology fits inside it. */
+ withinCap: { allowed: true, admitted: 3, refused: 0, capped: false },
+ /** The licensed-overflow case: 5 declared, 3 paid for. */
+ overflow: { allowed: true, admitted: 3, refused: 2, capped: true },
+ /** Unlicensed: the whole cluster is denied. `capped` stays false by design. */
+ denied: { allowed: false, reason: 'no clustering entitlement', admitted: 0, refused: 5, capped: false },
+} satisfies Record<string, MultiNodeGateVerdict>;
+
+/** `NaN` is what `Number(process.env.OS_CLUSTER_REPLICAS)` yields when unset. */
+const NOT_DECLARED = Number(undefined);
+
+describe('describeMultiNodeCapTelemetry — the series an operator receives', () => {
+ it('a licensed overflow publishes the declared count, the admitted count and the boot event', () => {
+ expect(describeMultiNodeCapTelemetry(VERDICTS.overflow, 5)).toEqual<MultiNodeCapMetric[]>([
+ { name: 'cluster_node_cap_verdicts_total', kind: 'counter', value: 1, labels: { verdict: 'capped' } },
+ { name: 'cluster_declared_nodes', kind: 'gauge', value: 5, labels: { verdict: 'capped' } },
+ { name: 'cluster_admitted_nodes', kind: 'gauge', value: 3, labels: { verdict: 'capped' } },
+ ]);
+ });
+
+ it('a topology that fits reads `admitted`, with declared and admitted agreeing', () => {
+ expect(describeMultiNodeCapTelemetry(VERDICTS.withinCap, 3)).toEqual<MultiNodeCapMetric[]>([
+ { name: 'cluster_node_cap_verdicts_total', kind: 'counter', value: 1, labels: { verdict: 'admitted' } },
+ { name: 'cluster_declared_nodes', kind: 'gauge', value: 3, labels: { verdict: 'admitted' } },
+ { name: 'cluster_admitted_nodes', kind: 'gauge', value: 3, labels: { verdict: 'admitted' } },
+ ]);
+ });
+
+ it('an outright denial reads `refused`, NOT `capped` — the two are different facts', () => {
+ const samples = describeMultiNodeCapTelemetry(VERDICTS.denied, 5);
+ expect(samples.every((s) => s.labels.verdict === 'refused')).toBe(true);
+ expect(samples).toEqual<MultiNodeCapMetric[]>([
+ { name: 'cluster_node_cap_verdicts_total', kind: 'counter', value: 1, labels: { verdict: 'refused' } },
+ { name: 'cluster_declared_nodes', kind: 'gauge', value: 5, labels: { verdict: 'refused' } },
+ { name: 'cluster_admitted_nodes', kind: 'gauge', value: 0, labels: { verdict: 'refused' } },
+ ]);
+ });
+
+ it('an uncapped gate publishes NO admitted series — a number there would invent a limit', () => {
+ const samples = describeMultiNodeCapTelemetry(VERDICTS.uncapped, 4);
+ expect(samples.map((s) => s.name)).toEqual([
+ 'cluster_node_cap_verdicts_total',
+ 'cluster_declared_nodes',
+ ]);
+ // Specifically NOT `cluster_admitted_nodes 0`, which would read as "your
+ // licence admits zero nodes" on a deployment with no cap at all.
+ expect(samples.find((s) => s.name === 'cluster_admitted_nodes')).toBeUndefined();
+ });
+
+ it('publishes NO declared series when nothing was declared — `0` would be a declaration of zero', () => {
+ const samples = describeMultiNodeCapTelemetry(VERDICTS.withinCap, NOT_DECLARED);
+ expect(samples.map((s) => s.name)).toEqual([
+ 'cluster_node_cap_verdicts_total',
+ 'cluster_admitted_nodes',
+ ]);
+ expect(samples.find((s) => s.name === 'cluster_declared_nodes')).toBeUndefined();
+ });
+
+ it('the boot event is emitted for every verdict, so a silent series is a CONFIGURATION answer', () => {
+ // An operator reading a dashboard has to be able to tell "gate consulted,
+ // everything fine" from "nothing here is instrumented". The counter is
+ // present in all four cases; absence therefore means the gate was never
+ // consulted (single-node boot) or no metrics backend is configured.
+ for (const verdict of Object.values(VERDICTS)) {
+ const counters = describeMultiNodeCapTelemetry(verdict, 5)
+ .filter((s) => s.kind === 'counter');
+ expect(counters).toHaveLength(1);
+ expect(counters[0]!.name).toBe('cluster_node_cap_verdicts_total');
+ expect(counters[0]!.value).toBe(1);
+ }
+ });
+});
+
+describe('the declared count is normalized exactly as the gate normalizes its own input', () => {
+ // `checkMultiNodeAllowed` treats meaningless values (unset, zero, negative,
+ // non-finite) as "not declared" and floors a fractional one. The reading has
+ // to agree, or the surface reports a declaration the gate never saw.
+ const cases: Array<[number, number | undefined]> = [
+ [NOT_DECLARED, undefined],
+ [0, undefined],
+ [-1, undefined],
+ [Number.POSITIVE_INFINITY, undefined],
+ [2.7, 2],
+ [3, 3],
+ ];
+
+ for (const [input, expected] of cases) {
+ it(`OS_CLUSTER_REPLICAS=${String(input)} → ${expected === undefined ? 'no declared series' : `declared ${expected}`}`, () => {
+ const declared = describeMultiNodeCapTelemetry(VERDICTS.uncapped, input)
+ .find((s) => s.name === 'cluster_declared_nodes');
+ expect(declared?.value).toBe(expected);
+ });
+ }
+});
+
+describe('⚠️ the surface never claims observed membership', () => {
+ /**
+ * THE assertion this card exists to protect. Every fact the process holds at
+ * this moment is a DECLARATION or a LICENCE verdict; it has no membership
+ * view whatsoever. A later "helpful" rename — `cluster_nodes`,
+ * `cluster_active_nodes`, a label `state="running"` — would turn a true
+ * reading into a false one while every other test here stayed green, because
+ * the numbers would not change at all. Only the words would.
+ */
+ //
+ // ⚠️ The segment anchors are `[^a-z0-9]`, NOT `\b`. This regex was first
+ // written with `\b` and the vacuity proof at the bottom caught it
+ // immediately: `_` is a WORD character, so `\bactive\b` does not match
+ // inside `cluster_active_nodes` — the exact rename this guard exists to
+ // reject would have sailed through while all three sweeps below reported
+ // green. Metric names are snake_case, so the separator has to be treated as
+ // a boundary explicitly.
+ const MEMBERSHIP_CLAIMS =
+ /(?:^|[^a-z0-9])(?:running|active|live|alive|online|healthy|current|observed|actual|joined|members?|membership|peers?|connected|up)(?:[^a-z0-9]|$)/i;
+
+ const ALL_SAMPLES = Object.values(VERDICTS).flatMap((v) => [
+ ...describeMultiNodeCapTelemetry(v, 5),
+ ...describeMultiNodeCapTelemetry(v, NOT_DECLARED),
+ ]);
+
+ it('guards itself: the sweep actually has samples to inspect', () => {
+ expect(ALL_SAMPLES.length).toBeGreaterThan(8);
+ });
+
+ it('no metric NAME claims an observed count', () => {
+ for (const sample of ALL_SAMPLES) {
+ expect(
+ sample.name,
+ `"${sample.name}" reads as a count of what is RUNNING. This process has no `
+ + 'cluster membership view — nodeId is random per process and there is no '
+ + 'join/leave registry — so such a series would be false. Name it for what '
+ + 'is known: what the operator DECLARED, and what the licence ADMITS.',
+ ).not.toMatch(MEMBERSHIP_CLAIMS);
+ }
+ });
+
+ it('no LABEL name or value claims an observed count', () => {
+ for (const sample of ALL_SAMPLES) {
+ for (const [key, value] of Object.entries(sample.labels)) {
+ expect(key).not.toMatch(MEMBERSHIP_CLAIMS);
+ expect(value).not.toMatch(MEMBERSHIP_CLAIMS);
+ }
+ }
+ });
+
+ it('the names that ARE published say declared / admitted, and use the gate\'s own vocabulary', () => {
+ const names = new Set(ALL_SAMPLES.map((s) => s.name));
+ expect(names).toEqual(new Set([
+ 'cluster_node_cap_verdicts_total',
+ 'cluster_declared_nodes',
+ 'cluster_admitted_nodes',
+ ]));
+
+ // The verdict label is the vocabulary #8367 / PR #8503 landed — not a
+ // second one invented for the display.
+ const words = new Set(ALL_SAMPLES.map((s) => s.labels.verdict));
+ expect(words).toEqual(new Set(['admitted', 'capped', 'refused']));
+ });
+
+ it('vacuity proof: the sweep DOES reject a membership-flavoured rename', () => {
+ // Without this, a regex that silently stopped matching would leave the
+ // three tests above green over exactly the rename they exist to catch.
+ expect('cluster_active_nodes').toMatch(MEMBERSHIP_CLAIMS);
+ expect('cluster_nodes_running').toMatch(MEMBERSHIP_CLAIMS);
+ expect('cluster_live_members').toMatch(MEMBERSHIP_CLAIMS);
+ expect('members').toMatch(MEMBERSHIP_CLAIMS);
+ // ...and does not reject the honest ones.
+ expect('cluster_declared_nodes').not.toMatch(MEMBERSHIP_CLAIMS);
+ expect('cluster_admitted_nodes').not.toMatch(MEMBERSHIP_CLAIMS);
+ });
+});