Skip to content

[finding] getMetaDiagnostics swallows the 503 that #5532 raised — an unreadable metadata store is published as "0 problems", and the type vanishes from the Studio tile counts #8855

Description

@hotlong

Filed by the domain:metadata PM seat off evidence surfaced during #6504 (list-shaped reads whose count is published as a whole-corpus fact). It is deliberately not folded into #6504 — see "Why this is its own card" — and it is filed ungraded: the type field and the domain:* routing are triage's to produce.

Everything below is read from origin/main. Nothing here has been executed; the reproduction is written out but not run, and that is stated where it matters.

What it publishes

packages/metadata-protocol/src/protocol.ts:4637getMetaDiagnostics() returns four numeric/aggregate facts about the metadata corpus:

total // count of entries failing spec validation
scannedTypes // = targetTypes.length
scannedItems // incremented once per item actually walked
stats // keyed by type name -> { count, locked, packages }; count = items.length

Its own doc names the consumer, so the audience is not inferred:

Per-type aggregate stats — count of items and the list of packages contributing to each type. Computed in the same sweep so the Studio directory page can render tile counts and a package filter in one round-trip.

The REST door is packages/rest/src/rest-server.ts:4759GET {metaPath}/diagnostics — which returns the object verbatim (res.json(result)). No shaping, no degradation flag.

The swallow

The per-type read is the only thing between that payload and the store:

for(consttoftargetTypes){letlisted: any;try{listed=awaitthis.getMetaItems({type: t,organizationId: ,packageId: }asany);}catch{// Type not listable in this kernel scope — skip.continue;}stats[t]={count: items.length,locked: lockedCount,packages: […pkgSet].sort()};}

The comment names a benign reason, and that reason is real. The catch is untyped and takes everything else with it — including the one error the callee exists to raise.

getMetaItems is protocol.ts:5129. It calls rethrowUnlessMetadataStoreUnprovisioned at :5310 and :5369, and that guard (protocol.ts:4760) throws metadataStoreUnavailableError — a 503 — for every read failure that is not "sys_metadata has not been provisioned yet". Its doc states the rule this card is about, ~130 lines below the swallow, in the same file:

ADR-0110 D3 is the rule: a miss and an outage are different facts with opposite meanings, and a consumer must never read one as the other.

and lists, among the three defects #5532 removed:

getMetaItems turned it into items: [] — an outage answered as "this environment declares none of these"

#5532 raised that 503 out of getMetaItems precisely so an outage would stop looking like emptiness. getMetaDiagnostics catches it back into emptiness one layer up, and then publishes the emptiness as a number.

Why the worst value is the benign one

Three dispositions, none of which the caller can tell apart from a healthy answer:

  1. Per type — an unreadable type produces no stats[t] key at all. Not zero: absent. The payload is byte-shaped like an environment that declares none of that type.
  2. Corpus-widetotal counts entries that failed validation, and skipped types contribute none. A store that is entirely unreachable therefore answers total: 0a diagnostics endpoint whose whole job is to report problems reports zero problems when it cannot read anything. Green is the failure mode.
  3. scannedItems under-counts silently, with no companion field saying by how much.

scannedTypes is the field that would have caught it, and it is computed from the intent

consttargetTypes=request.type ? [request.type] : DEFAULT_METADATA_TYPE_REGISTRY.filter().map();return{ entries,total: entries.length,scannedTypes: targetTypes.length, scannedItems, stats };

targetTypes is fixed before the loop and never decremented on continue. So the response asserts "I scanned N types" while having scanned fewer, and the discrepancy is not derivable by the caller either — Object.keys(stats).length is only a lower bound, since a type that reads fine but holds no items also produces a stats entry (count: 0). There is no field in the payload from which a client can compute "some of this is missing".

What a fix must not do

⛔ Not "delete the catch". The benign reason in the comment is genuine — a kernel scope where a type is not listable must not 500 the whole diagnostics sweep. The discrimination already exists in this file and is the platform's single answer to this question: classify by error type through the same isMissingTableError / rethrowUnlessMetadataStoreUnprovisioned path that DatabaseLoader (#5108) and SysMetadataRepository (#4867) ask. Whether the outage case should then propagate the 503 or be surfaced as a per-type degradation marker in the payload is the actual design call on this card, and it is a read-surface change to metadata-protocol — which is why it is not consumer-sweep work.

Why this is its own card, not part of #6504

#6504 swept direct consumers of a list()-shaped read that publish its length as a corpus fact. getMetaDiagnostics is two hops away — it consumes getMetaItems, not list() — and its cleanest fix reads the verdict off the metadata service rather than hardening a call site. Folding it into #6504 would have widened that card after the fact instead of completing it, and would have pre-empted this routing. #6504's dev surfaced the evidence and correctly declined to file it from inside the card.

The relationship to #6504 is a shared shape, not a shared fix: a count taken from a read that can silently return less than the whole, published as though it were the whole.

Reproduction — written, NOT run

Stub an engine whose sys_metadata read throws a non-isMissingTableError driver error, then:

GET /api/v1/meta/diagnostics

Expected under the defect: 200 with total: 0, scannedItems: 0, stats: {}, and scannedTypes still reporting the full registry count (undecremented). Whoever takes this should measure it before writing anything — I have read the path, not executed it, and the direction should be predicted before the run.

Adjacent observation — UNVERIFIED, do not act on it from this card

The REST handler forwards type, severity and packageId but no organizationId, while getMetaDiagnostics accepts one and forwards it to getMetaItems. Whether an undefined organizationId widens this read across tenants was not measured, and this card asserts nothing about it. It is recorded here only so it is not lost; it belongs with the tenant-scoping thread (#8805, and the audit-read scoping that landed for #8747), and needs its own measurement before it is anybody's card.

Triage note (input, not a claim)

Fix lands in packages/metadata-protocol/src/protocol.ts (with packages/rest in the consumption radius if the payload gains a degradation marker) — reading as domain:metadata, but the domain:* label is triage's to produce and is deliberately left off. Under the mechanical boundary test this looks like Bug rather than Feature: the acceptance set does not widen, and the change restores a declared contract (ADR-0110 D3) that this file states in its own prose.

⚠️ Editorial note: this body was posted, read back, and re-posted once — the first write had two angle-bracket spans silently eaten by the sanitizer (a generic type in the field list, and a placeholder in the reproduction). Both are rewritten above without angle brackets. Nothing else changed.

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions