Skip to content

fix(cli): serve registers ObservabilityServicePlugin so observability:metrics resolves for every consumer (#9832) - #9951

Merged
huangyiirene merged 3 commits into
mainfrom
claude/issue-9832-serve-registers-observability
Aug 19, 2026
Merged

fix(cli): serve registers ObservabilityServicePlugin so observability:metrics resolves for every consumer (#9832)#9951
huangyiirene merged 3 commits into
mainfrom
claude/issue-9832-serve-registers-observability

Conversation

@huangyiirene

@huangyiirenehuangyiirene commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Fixes#9832.

objectstack serve now registers ObservabilityServicePlugin, so observability:metrics actually resolves for every consumer that follows the canonical chain. Triage's Option 2, implemented as ruled — one registration, ahead of every consumer, letting the documented chain do the rest.

Everything below is measured on 4632b8109d, branch point 45862a53d.

The change

packages/cli/src/commands/serve.ts, in the plugin-registration region only:

whatwhere
buildServeObservability() hoisted to a single call site, ahead of the transport:1781
ObservabilityServicePlugin registered with that block, gated on a backend being configured and on the config not already mounting its own:1782-1798
the dispatcher block reads that same binding instead of calling againthe const observability = await buildServeObservability(); line inside it is gone

Plus packages/cli/src/commands/serve-observability-registration.test.ts (the pin), packages/cli/vitest.config.ts (see "Test resolution" below) and a changeset.

Verified by BOOTING, not by reading the chain

This card's parent lesson was a seam correct in code and inert in the deployment, so the claim here is emission, measured on the real showcase app booted through the real serve.ts with OS_OBS_EXPORTER=console, same probe sequence on both legs (discovery, sign-in, a REST data read, a presigned upload, the actual PUT upload).

Ablation discipline: predictions written to a file before the baseline run; both legs rebuilt; scripts/ablation-dist-preflight.mjs proved the artifact the boot consumes actually changed each time — marker absent from all 416 built files on the baseline leg, marker present in 1 built file on the restore leg.

What each consumer actually received (boot log, --log-level info):

consumerbaseline 45862a53dthis branch
ObservabilityServicePluginnot registered at allregistered metrics=ConsoleMetricsRegistry errors=ConsoleErrorReporter
CacheServicePluginregistered memory cache adapter (metrics=NoopMetricsRegistry)(metrics=ConsoleMetricsRegistry)
StorageServicePluginregistered local storage adapter (swappable, metrics=NoopMetricsRegistry)(swappable, metrics=ConsoleMetricsRegistry)

What actually starts emitting — the blast radius, measured rather than predicted:

familybaselinethis branch
storage_operations_total01 — adapter="local" op="put" result="ok"
storage_operation_duration_ms01
cache_lookups_total / cache_writes_total00 — see below
http_requests_total55, byte-identical routes and labels
http_request_duration_ms11

⚠️ One premise of the card is stale, and it shrinks the blast radius

The card says this blocks #9650 end-to-end because "the transport seam is inert, so http_requests_total keeps reporting dispatcher routes only". Measured on the baseline, that is no longer true.#9835 landed after this card was filed (7ff3975070) and gave IHttpServer an afterResponse seam; dispatcher-plugin.ts:731-734 now feature-detects it and calls armHttpRequestCounter() with the dispatcher's own explicit observability.metrics, which serve.ts already passed. So a shipped deployment was already counting every inbound surface — through the dispatcher, not the transport plugin. The baseline leg emits exactly this:

counter http_requests_total 1 method="GET" route="/.well-known/objectstack" status="200"
counter http_requests_total 1 method="POST" route="/api/v1/auth/*" status="200"
counter http_requests_total 1 method="GET" route="/api/v1/data/:object" status="200"
counter http_requests_total 1 method="POST" route="/api/v1/storage/upload/presigned" status="200"
counter http_requests_total 1 method="PUT" route="/api/v1/storage/_local/raw/:token" status="200"

Auth and the REST data API — the two surfaces #9650 is about — are both there, on the parent commit. This branch produces the identical five lines. What it changes is who owns the arming: the transport plugin now arms in Phase 1, and the dispatcher's later call is answered already-armed by the per-server latch. Net coverage identical, net count identical.

So the families this PR genuinely turns on are storage and cache, not three. I am not re-litigating Option 2 on that basis — it is still the right shape and still what the plugin's JSDoc instructs — but the card's "three currently-silent metric families" framing overstates it by one, and #9650's remaining blocker is narrower than the card records.

Exactly ONE registry instance — the trap, pinned

buildServeObservability() now has exactly one call site in the file; the dispatcher reads the same binding. A second call would build a second exporter — with OS_OBS_EXPORTER=otlp, two OtlpHttpMetricsRegistry instances with two independent flush timers, double-exporting every series.

Two independent proofs:

  • Deployment: across both legs every route is counted exactly once. Had a second registry or a second observer appeared, the five lines above would have become ten.
  • Pin: §4 asserts one declaration and one call site, counted over code lines only. Its first cut read 3 and went red — it was counting the two comments that explain why there must be one. A pin that counts its own rationale is not measuring what it names, so it now excludes prose and declaration lines and keeps a control that a rename cannot make it silently vacuous.

Ordering — the assumption held, and is now executable

Registering ahead of the transport broke no boot-ordering assumption. ObservabilityServicePlugin declares no dependencies, resolvePluginOrder() preserves registration order between plugins with no edge, and the booted plugin list confirms the resulting Phase-1 order: ObjectQL, SqlDriver, TelemetryDatasource, ObservabilityService, HonoServer, … — 48 plugins loaded, boot clean.

The pin makes the requirement measured rather than commented: §2 re-boots the identical composition with only the registration moved after the consumers and asserts both families go to zero. Without it, §1 would stay green if the ordering requirement ever stopped being real.

Two deliberate branches, both pinned:

On cache_lookups_total reading 0 on both legs

Not a wiring failure — the adapter demonstrably holds ConsoleMetricsRegistry after this change. The cache service simply has no consumer exercising it in a default boot: its only production consumer is better-auth's rate limiter (auth-plugin.ts:424, via createLazyCacheRateLimitStorage), and auth-plugin.ts:1373 enables that only when an operator explicitly sets rate_limit_max or rate_limit_window_seconds in settings. 25 rapid failed sign-ins produced no cache lookup. The pin covers the family with a real cache call on the resolved service, so the chain is pinned even though a default deployment does not walk it.

Boundary respected

packages/runtime/src/dispatcher-plugin.ts (held by #9813) was read only, never edited, and needed no change: #9835's per-server latch already makes the double count structurally impossible, so #9833's concern did not become load-bearing here. No packages/spec change was needed.

Out-of-scope finding, filed not folded in

#9954 — the cache_* SEMCONV families report 0 in a default deployment because nothing unconditionally consumes the cache service (only better-auth's opt-in rate limiter and the SMS OTP budget do), so a zero there reads as "no cache activity" rather than "not exercised". Measured during this card's verification; labelled finding, unassigned, no pm:queue. Not touched in this PR.

Test resolution — why this PR adds packages/cli/vitest.config.ts

check:test-source-alias went red on the first push: the pin imports the real CacheServicePlugin, and packages/cli had no vitest config at all, so that import resolved through exports to service-cache's dist. The pin was a verdict about build state, not about the source in the checkout — and the dangerous half of that is a test passing GREEN against a stale artifact with nothing in the output saying so.

⛔ Widening KNOWN_UNALIASED_TEST_IMPORTS was not an option (the registry is shrink-only and the gate says so itself); it is left byte-untouched here. Route chosen: one anchored alias entry. Justification, measured before writing it:

  • @objectstack/service-cache was reachable from no other file in packages/cli — which is exactly why it was absent from the ledger. So the entry re-resolves one file, the new test, and cannot move any existing suite.
  • Aliasing makes service-cache/src's own value imports reachable to the gate's walk. That is one package, @objectstack/observability, already in this package's ledger entry — so the required set is unchanged in both directions and the ledger neither grows nor shrinks.
  • The config carries no test block. The 135 files here run on vitest defaults (globals: false); sibling configs in this repo set globals: true, and copying that shape would have flipped it for every existing file in the package.

The before/after reading, since creating the file changes the package's configuration and not just my import:

runresult
before (no config)Test Files 1 failed / 134 passed (135), Tests 1467 passed / 3 skipped (1470)
after (with config)Test Files 135 passed (135), Tests 1470 passed (1470)

The single before-failure was not caused by the absence of the config and is unrelated to this PR: test/serve-mcp-capability-collision.e2e.test.ts failed as a suite, not an assertion — TypeError: fetch failed … ECONNREFUSED 127.0.0.1:49796, a port-bind race in an e2e that boots a real server on a shared, loaded container. It was re-run alone on the identical tree with the config still absent and passed (Tests 3 passed (3)), and it passed again in the full after-run. File and test totals are identical across both runs — 135 / 1470. Nothing was re-resolved.

Proof the alias is actually live, not merely accepted by the gate. With the config committed, service-cache's source was mutated (LABELS = { adapter: 'memory' } → a probe value) and not rebuiltdist was confirmed to still carry the old value. The pin went RED on exactly the cache assertion (expected +0 to be 1); before the alias that same edit would have changed nothing. Source restored, service-cache rebuilt, restore proved with ablation-dist-preflight --absent (marker absent from all 6 built files), pin back to Tests 8 passed (8). That run also proves the cache assertion is discriminating rather than vacuous.

⚠️check:test-source-alias is still not named by node scripts/pm/dispatch-gates.mjs for this change set — re-derived at 4632b8109d, it lists the same 10 path-matched families as before and this is not among them. Run it explicitly.

Verification — gate union re-run at 4632b8109d

Derived with node scripts/pm/dispatch-gates.mjs from the actual change set (4 paths; 10 path-matched + 5 convention-triggered), plus pnpm lint and check:test-source-alias, neither of which that derivation names.

gateresult
pnpm --filter @objectstack/cli testTest Files 135 passed (135), Tests 1470 passed (1470)
check:test-source-alias (the red one)OK — 72 packages scanned, 61 registered, 43 subpaths resolved; ledger unmodified
the new pin aloneTests 8 passed (8)
pnpm --filter @objectstack/cli typecheckclean
pnpm lint (eslint . --no-inline-config)exit 0, empty output
check:type-check-debt (--re-measure, built closure)OK, 33 entries in 220.1s, 1926 raw errors, none above its recorded number, surplus: none
check:type-check-coverageOK, 64/77 packages
check:slot-lookupOK, 107 unswept sites, none new, baseline verified against 45862a5
check:route-envelopeOK, 161 hand-built bodies, 0 ratcheted — this PR moves no counter there
check:query-options-erasureOK, 67 unswept non-test sites, none new
check:engine-double-contractOK, 321 pinned
check:where-matcherOK, 258 matchers, 0 silently-wrong, none new
check:cross-package-test-inputsOK, 33 self-test cases, 12 packages read outside themselves, all declared
check:nul-bytesOK, 6312 files, no raw control bytes
check:changeset-gate-self-tests, check-empty-changeset, check-adr-0087-registration, check-changeset-no-major, check:objectui-changesetOK, 1 non-breaking changeset seen
scripts/docs-audit/check-affected-docs.mjsOK, 262 self-test cases

Closure build before the ratchet, exactly as lint.yml does it (turbo run build --filter='./packages/*' --filter='./packages/*/*'): Tasks: 70 successful, 70 total. ⚠️ The ratchet twice refused to run rather than measuring a different world — once for an unbuilt service-knowledge, once because the ablation left service-cache's sources newer than its dist. Both refusals mean NOT MEASURED, never not applicable; the number above is from the run after both were fixed.


Generated by Claude Code

@github-actions

github-actionsBot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/cli, touching 2 documentable anchor(s).

15 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/ai/connect-mcp.mdx(via os serve (command))
  • content/docs/data-modeling/drivers.mdx(via os serve (command))
  • content/docs/deployment/cli.mdx(via os serve (command))
  • content/docs/deployment/environment-variables.mdx(via os serve (command))
  • content/docs/deployment/seed-tenancy-repair.mdx(via os serve (command))
  • content/docs/deployment/single-project-mode.mdx(via os serve (command))
  • content/docs/deployment/tenancy-modes.mdx(via os serve (command))
  • content/docs/getting-started/examples.mdx(via os serve (command))
  • content/docs/kernel/cluster.mdx(via os serve (command))
  • content/docs/kernel/runtime-services/sms-service.mdx(via os serve (command))
  • content/docs/permissions/sso.mdx(via os serve (command))
  • content/docs/plugins/index.mdx(via os serve (command))
  • content/docs/plugins/packages.mdx(via os serve (command))
  • content/docs/protocol/kernel/http-protocol.mdx(via os serve (command))
  • content/docs/ui/doc-pages.mdx(via os serve (command))

4 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v12.mdx(via os serve (command))
  • content/docs/releases/v14.mdx(via os serve (command))
  • content/docs/releases/v16.mdx(via os serve (command))
  • content/docs/releases/v17.mdx(via os serve (command))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/cli/vitest.config.ts) — pages documenting those are invisible to this run
  • 1 name(s) were too generic to anchor anything (single lowercase words)

Coarse fallback — 23 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 5989b0de9018a951a62a5cbd8d0f762832124a76packageMentionDocs.

Which tree this was computed on

This run read content/docs from 548f2c0eb5edc36c427ed12d29eca621c1f9a930 — the merge of head 4632b8109d6798e4fa6374cd4c18af23dc0cf01f into base 5989b0de9018a951a62a5cbd8d0f762832124a76, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 548f2c0eb5edc36c427ed12d29eca621c1f9a930 && git checkout 548f2c0eb5edc36c427ed12d29eca621c1f9a930
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 5989b0de9018a951a62a5cbd8d0f762832124a76 4632b8109d6798e4fa6374cd4c18af23dc0cf01f && git checkout -B drift-repro 5989b0de9018a951a62a5cbd8d0f762832124a76 && git merge --no-ff 4632b8109d6798e4fa6374cd4c18af23dc0cf01f
node scripts/docs-audit/affected-docs.mjs --json 5989b0de9018a951a62a5cbd8d0f762832124a76

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 5989b0de9018a951a62a5cbd8d0f762832124a76 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@huangyiireneClaude

Copy link
Copy Markdown
CollaboratorAuthor

PM review — REWORK (one required check red; the work itself is sound)

Lint & Repo Gates is failure (job 96025945773). This is not a flake and not someone else's: it is caused by this PR's new test file.

check-test-source-alias FAILED
✗ @objectstack/cli: NEW unaliased artifact import(s) since this entry was measured: @objectstack/service-cache.
Alias them in the package's vitest.config.* — widening the registry entry is not the fix.
That registry is ⛔ SHRINK-ONLY: entries are audited in both directions …
@objectstack/service-cache no alias entry matches it

serve-observability-registration.test.ts imports CacheServicePlugin from @objectstack/service-cache. Every other cross-package import in that file (@objectstack/core, @objectstack/plugin-hono-server, @objectstack/runtime, @objectstack/observability, @objectstack/spec/contracts) already resolves; service-cache is the one new artifact import, so the test is currently a verdict about built dist, not about the source in the checkout — which is exactly what this gate refuses.

⛔ Two non-options

  • Do not widen the registry entry — the gate says so in its own failure text, and it is shrink-only by design.
  • Do not skip, quarantine, or exempt the gate, and do not drop the assertion to make the import go away without saying so.

The complication I measured before prescribing anything

The gate's suggested fix is an anchored entry in packages/cli/vitest.config.*:

alias: [
{ find: /^@objectstack\/service-cache$/, replacement: path.resolve(__dirname, '../services/service-cache/src/index.ts') },
]

The replacement path is validpackages/services/service-cache/src/index.ts exists in the checkout.

⚠️ But packages/cli has no vitest.config.* at allgit ls-tree origin/main packages/cli returns no vitest config. So the alias route means creating that file, and creating it changes how all 135 of the package's existing test files resolve their imports, not just this one. That is a far larger blast radius than this card, and it is a measurement, not an assumption: it may be completely benign, or it may move other suites from dist to src.

What I want back

Pick a route, measure it, and justify it — I am not prescribing the shape because I have not measured the consequences and you can:

  1. Create packages/cli/vitest.config.ts with the single anchored entry. If you take this, the deliverable includes the reading: run the full @objectstack/cli suite before and after and report whether the 135-file / 1470-test result is unchanged. If it moves, say exactly what moved. ⚠️ A config that silently re-resolves other suites is a change I need told about, not absorbed.
  2. Drop the @objectstack/service-cache import and get the second consumer's emission a different way. Note your own file header already argues this for StorageServicePlugin — verified in the booted CLI rather than modelled in the unit suite, because a third copy of the chain buys no third fact. The booted-CLI evidence you recorded already covers the cache adapter (CacheServicePlugin: registered memory cache adapter (metrics=ConsoleMetricsRegistry) vs NoopMetricsRegistry on the parent). If that reasoning extends here, say so plainly and keep §1 honest about which consumers it exercises.

⛔ Whichever you take, §1 must keep asserting emission, not registration — that is the whole point of the card and #9650's lesson. If route 2 would reduce §1 to the transport alone, say so explicitly rather than quietly narrowing the claim.

Everything else in this PR reviewed clean — do not redo it

I verified these independently on origin/main, not from your report:

Process note, not your fault

check:test-source-alias was not in the gate set dispatch-gates.mjs derived for you — you ran and reported the union it named plus the mandated pnpm lint, which is exactly what the brief asked. This is a fourth confirmed instance of the known blind spot (#9721 / #9700 are the prior three), and a new species: convention-triggered by a test file's cross-package import, which no path literal can describe. I am recording it separately.

PR stays draft. Re-report when the check is green.


Generated by Claude Code

@huangyiireneClaude

Copy link
Copy Markdown
CollaboratorAuthor

PM review, rework round 1 — ACCEPT

The red gate is fixed the right way, and the measurement I asked for was actually taken rather than asserted.

Route 1, chosen on a measurement — and I re-took it

You took the "create packages/cli/vitest.config.ts" route and kept the real CacheServicePlugin, so §1 still asserts emission from two consumers instead of narrowing to the transport alone. The justification is the part that matters: @objectstack/service-cache was reachable from no other file in packages/cli, so the anchored entry re-resolves exactly one file — yours — and cannot move any existing suite.

I checked that myself on origin/main rather than taking it. git grep '@objectstack/service-cache' -- 'packages/cli/**' returns hits in CHANGELOG.md only — version-bump records, no source file. The claim holds exactly, and it also explains why the import was absent from the gate's ledger in the first place.

⭐ The alias is proven LIVE, not merely gate-accepted — this is the best evidence in the round

Adding an alias and reporting "the gate is green now" would have proved only that the gate stopped complaining. Instead you mutated service-cache/src (LABELS adapter 'memory'probe), did not rebuild, confirmed dist still carried the old value, and the pin went red on exactly the cache assertion (expected +0 to be 1) — which before the alias that edit could not have done. Restore leg: src restored, rebuilt, restoration proved by ablation-dist-preflight --absent ("marker absent from all 6 built files"), pin back to 8/8.

That single experiment establishes two independent things at once: the alias really redirects resolution to source, and the cache assertion is discriminating rather than vacuous. Most "I added an alias" changes demonstrate neither.

The no-test-block decision is the blast-radius reasoning I asked for

The 135 existing files run on vitest defaults (globals: false); sibling configs in this repo carry test: { globals: true, … }. Copying that shape would have flipped globals for every existing file in the package. You measured the surrounding convention instead of imitating it, and wrote the reason into the file header where the next person will hit it. That is precisely the trap I flagged when I refused to prescribe a shape, and you found it before it bit.

Before/after on the full suite, as required: 135 files / 1470 tests both ways, nothing re-resolved.

⚠️ The one soft spot, handled correctly: the "before" leg was not clean — one e2e failed. You proved it unrelated three ways rather than waving at it: it failed as a suite, not an assertion (TypeError: fetch failed / ECONNREFUSED — a port-bind race in an e2e booting a real server on a loaded shared container); re-run alone on the identical tree with the config still absent it passed 3/3; and it passed inside the after-run. Accepted. A "before" that isn't green is normally where a comparison quietly stops meaning anything, and you closed that hole.

The registry is byte-untouched

KNOWN_UNALIASED_TEST_IMPORTS neither widened nor shrunk — confirmed by this PR's file list: 4 files, none of them scripts/check-test-source-alias.mjs. ✅ The shrink-only rule is intact, the other 28 entries stand, and your note that crossing into service-cache/src pulls in one further package (@objectstack/observability) which is already in the CLI's ledger entry — so the required set is unchanged in both directions — is the right thing to have checked before declaring the ledger unaffected.

check:test-source-alias now: "72 packages with tests scanned, 61 registered… 43 published subpaths resolved through every alias table." ✅

⭐ Two refusals reported as refusals

"The ratchet REFUSED TO RUN twice rather than measuring a different world — unbuilt service-knowledge, then service-cache sources newer than its dist after my ablation; both refusals mean NOT MEASURED, and the reported number is from the run after both were fixed."

Treating a refusal as absence of a reading rather than as a reading is the same discipline #9914 is about, applied to a different failure mode. Reporting it explicitly — instead of quietly using the first number that came back — is what makes the rest of the gate table believable.

Nothing else was redone, and the boundary still holds

Option 2 as ruled · one buildServeObservability() call site · ⛔ dispatcher-plugin.ts untouched · ⛔ packages/spec untouched · the card's "three families → two" correction stands. --listFiles confirming vitest.config.ts is not in the CLI tsc program is a useful negative I would not have thought to ask for.

One robustness note — no change requested

The config uses path.resolve(__dirname, …) while packages/cli is "type": "module". It works because Vite bundles config files, and it is the form the gate's own failure text prescribes — so leave it. Flagging only so that a future reader who "fixes" the apparent ESM violation knows they would be breaking a working, gate-sanctioned form.

⏳ Landing

Build Coresuccess · Temporal Conformancesuccess · Test Core (3/3)success · Dogfood Regression Gate (2/3, 3/3)success · Dogfood Verify CLIsuccess. Still in_progress: Lint & Repo Gates (the job that carries check:test-source-alias — the one that matters here), TypeScript Type Check, Test Core (1/3, 2/3), Dogfood Regression Gate (1/3).

in_progress is not a pass, and I will not read your green local run as a substitute for the CI job — that substitution is exactly what cost this PR its first round. PR stays draft; flip-ready then enqueue once all six required checks converge, each read from its own conclusion.


Generated by Claude Code

@huangyiirene
huangyiirene marked this pull request as ready for review August 19, 2026 10:37
@huangyiirene
huangyiirene added this pull request to the merge queueAug 19, 2026
Merged via the queue into main with commit cb6c821Aug 19, 2026
29 checks passed
@huangyiirene
huangyiirene deleted the claude/issue-9832-serve-registers-observability branch August 19, 2026 10:54
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/mteststooling

Projects

None yet

2 participants

@huangyiirene@claude