diff --git a/.changeset/http-request-errors-total-retired.md b/.changeset/http-request-errors-total-retired.md new file mode 100644 index 0000000000..210fcaf471 --- /dev/null +++ b/.changeset/http-request-errors-total-retired.md @@ -0,0 +1,60 @@ +--- +"@objectstack/observability": minor +"@objectstack/runtime": minor +"@objectstack/spec": minor +--- + +fix(observability): **BREAKING** — `http_request_errors_total` is retired (ADR-0049 enforce-or-remove, #9834) + +**⛔ If you have a Grafana panel, an alert rule or a recording rule keyed on +`http_request_errors_total`, it will read a FLAT ZERO after this upgrade.** That +zero is the removal, not a healthy server, and it is the one way this change can +hurt you — nothing throws, nothing warns, the series simply stops receiving +samples. Rewrite the query before you deploy. + +Maintainer ruling 2026-08-20: **RETIRE**. The name was declared in `SEMCONV` as +part of a stable namespace *"so hosts can wire alerts/dashboards against it"*, +but the only emitter was `@objectstack/runtime`'s `instrumentRouteHandler`, +applied only by the dispatcher's own route Proxy — so the series never saw +auth's `getRawApp()` mount, the REST data API via `RouteManager`, or any other +inbound surface. Its two siblings in the same HTTP family moved to the +`IHttpServer.afterResponse` transport seam (`http_requests_total`, #9650/#9835; +`http_request_duration_ms`, #9834/#10004) and this one could not follow: +`HttpResponseObservation` carries `{method, routePattern, status, elapsedMs}` +and **no throw signal of any kind**, so every transport-side shape would have +counted a *different* population rather than the same one more widely. + +Migration (FROM → TO): + +| Wrote | Write instead | +|---|---| +| `rate(http_request_errors_total[5m])` in a panel or alert | `rate(http_requests_total{status=~"5.."}[5m])` — emitted by the transport, so it covers every inbound surface instead of the dispatcher's routes only | +| `sum by (route) (http_request_errors_total)` | `sum by (route) (http_requests_total{status=~"5.."})` | +| `SEMCONV.httpRequestErrorsTotal` / `RUNTIME_METRICS.httpRequestErrorsTotal` in host code | Delete the read. Both members are gone; `tsc` reports the missing property at the read site. | + +One-line fix: replace the metric name with `http_requests_total{status=~"5.."}`. + + + +**The replacement is wider, not merely different.** The retired counter was +divergent from a 5xx rate in *both* directions, measured: the dispatcher answers +its own errors through `errorResponseBase`, which sets a status and does **not** +re-throw — so the counter **missed** those — while its `catch` incremented +unconditionally, so a **thrown 4xx WAS counted** as an error. And +`http_requests_total` already carries a `status` label, so a status-class error +counter was fully derivable from data the transport already publishes. Prove the +new query wider rather than merely non-empty: make an auth route or a REST +data-API route answer 5xx and confirm it moves, where the retired counter would +not have moved at all. + +**If what you were actually alerting on was "a handler threw rather than +returning an error envelope"** — the one signal this counter uniquely carried — +that is the `errorReporter`, not a metric. Wire an `ErrorReporter` adapter +(Sentry / Datadog / your own); it still fires on every 5xx throw and is +untouched by this change. + +What is NOT removed: `http_requests_total`, `http_request_duration_ms`, +request-id propagation, the 5xx error reporter, and the +`res.__obsRecordedError` side channel that carries a swallowed error to it. The +dispatcher still instruments every route it mounts; it just no longer publishes +a fourth series whose name promised more coverage than it had. diff --git a/content/docs/deployment/production-readiness.mdx b/content/docs/deployment/production-readiness.mdx index f25882d15c..c6248b9673 100644 --- a/content/docs/deployment/production-readiness.mdx +++ b/content/docs/deployment/production-readiness.mdx @@ -55,8 +55,7 @@ That's it — every route the dispatcher mounts now: 1. Sends conservative security headers. 2. Echoes `X-Request-Id` (honored from caller, or freshly minted). -3. Emits the `http_request_errors_total` counter on a thrown handler. -4. Reports 5xx exceptions to Sentry (or your reporter) with the +3. Reports 5xx exceptions to Sentry (or your reporter) with the request id attached. The `http_requests_total{method,route,status}` counter and the @@ -73,6 +72,15 @@ is the registered pattern (`/api/v1/data/:id`), never the concrete path. (middleware chain and body parse included), not the handler alone — so p95 is the latency a caller experiences minus the network. +**`http_request_errors_total` was retired in 17.2.0** (#9834). Nothing declares +or emits it any more, so a dashboard keyed on that name reads a flat zero — +that is the removal, not a healthy server. **Alert on +`http_requests_total{status=~"5.."}` instead**: the transport emits it for every +inbound surface, whereas the retired counter only ever saw the dispatcher's own +routes and only when a handler *threw*, missing every error answered politely +with a status. Unhandled exceptions specifically still reach the +`errorReporter`, which is the signal that counter was mistaken for. + **A transport that does not implement the seam reports no HTTP metrics**: zero on `http_requests_total` there means "not instrumented", never "no traffic". The shipped Hono adapter implements it; verify a custom adapter with diff --git a/docs/OBSERVABILITY.md b/docs/OBSERVABILITY.md index a5121b0d09..84ef560da4 100644 --- a/docs/OBSERVABILITY.md +++ b/docs/OBSERVABILITY.md @@ -9,7 +9,6 @@ `createDispatcherPlugin` automatically instruments every route it mounts with: - **Request id** propagation: honors incoming `X-Request-Id` (or mints `req_`); echoes on the response. -- **`http_request_errors_total{method,route}`** counter (incremented on thrown errors). - **Error reporting** for 5xx (handler-thrown or via `errorResponseBase` side channel). **`http_requests_total{method,route,status}`** (1 per request) and @@ -75,9 +74,23 @@ import { RUNTIME_METRICS } from '@objectstack/runtime'; RUNTIME_METRICS.httpRequestsTotal // 'http_requests_total' RUNTIME_METRICS.httpRequestDurationMs // 'http_request_duration_ms' -RUNTIME_METRICS.httpRequestErrorsTotal // 'http_request_errors_total' ``` +> **⛔ `http_request_errors_total` was RETIRED in 17.2.0 (#9834).** It is no +> longer declared in `SEMCONV`/`RUNTIME_METRICS` and nothing emits it — a panel +> or alert keyed on that name reads flat zero, which is the removal, not an +> outage. +> +> **Read the 5xx rate from `http_requests_total{status=~"5.."}` instead**, which +> the transport emits for *every* inbound surface. The retired counter was +> declared as a server-wide error signal but incremented only from the +> dispatcher's per-route wrapper, and only when a handler **threw** — so it +> missed auth, the REST data API and every error a handler answered politely +> through `errorResponseBase`, while counting thrown 4xx as errors. The +> replacement query is both wider and better defined. If what you actually want +> is the *unhandled-exception* rate rather than the 5xx rate, that signal is the +> `errorReporter` (below), which still fires on every 5xx throw. + ### `cache_*` — a flat zero means "no configured consumer" The `@objectstack/service-cache` adapters emit `cache_lookups_total`, @@ -411,7 +424,8 @@ countServerTiming('db', queryMs, 'queries'); // → db;dur=;desc=" queri - [ ] Verified 4xx does **not** flood the APM. - [ ] Log records include `requestId` field; cross-checked one against the response `X-Request-Id` header. -- [ ] Alerts wired: error rate, p95 latency per route. +- [ ] Alerts wired: error rate (`http_requests_total{status=~"5.."}` — **not** + the retired `http_request_errors_total`, #9834), p95 latency per route. - [ ] (Optional) `Server-Timing` verified in DevTools with global mode (`serverTiming: true` / `OS_PERF_TIMING=1`) on, confirmed **absent** for a normal request, confirmed the per-request `X-OS-Debug-Timing: 1` header diff --git a/docs/protocol-upgrade-guide.md b/docs/protocol-upgrade-guide.md index c988e6f7b4..3ad5bd8a1e 100644 --- a/docs/protocol-upgrade-guide.md +++ b/docs/protocol-upgrade-guide.md @@ -469,6 +469,9 @@ No mechanical rewrite exists, in either direction. The refused values carry no r This is a RUNTIME registration API, not stored metadata, so — like `hook-context-session-roles-retired` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and the ledger entry is the notification channel. One metadata surface reaches it INDIRECTLY and is the reason this is not purely a code-side note: a `record-change` flow's start node forwards `config.objectName` verbatim into `registerHook` (`RecordChangeTrigger.start`), so a flow authored with a blank `objectName` used to bind a trigger to EVERY object in the tenant. It now fails to bind instead, loudly — the automation engine's per-flow bind guard warns and the `kernel:bootstrapped` binding audit re-reports it — which is the correct end state, but it is an observable change for that flow. #6573, #4281, #4001, #5928, ADR-0078. - Done when: No `registerHook` call site passes an empty `object` target, and none passes an `excludeObjects` list covering every name in its `object` list. Every `record-change` flow start node declares a non-blank `config.objectName`, or omits the key if the flow is genuinely meant to fire on every object. Boot completes with no "[ObjectQL] Hook ... declares an empty `object` target" throw and no "[record-change] ... not bound" warning naming a flow you expect to fire. +- **`http-request-errors-total-retired`** — `observability.SEMCONV.httpRequestErrorsTotal (the published metric name http_request_errors_total{method,route}, and its emission from the runtime dispatcher's per-route wrapper)` → the 5xx rate is `http_requests_total{status=~"5.."}` — the TRANSPORT emits that family through the `IHttpServer.afterResponse` seam, so it covers every inbound surface; unhandled-exception rate specifically, which is the one thing the retired counter uniquely reported, is the `errorReporter` (Sentry / Datadog / your adapter), which still fires on every 5xx throw + - Why not automatic: ADR-0049 enforce-or-remove, on a DECLARED-not-enforced metric name. `SEMCONV` published `http_request_errors_total` as part of a stable namespace declared "so hosts can wire alerts/dashboards against it", but the only emitter was `@objectstack/runtime`'s `instrumentRouteHandler`, applied only by the dispatcher's own route Proxy — so the series never saw auth's `getRawApp()` mount, the REST data API via `RouteManager`, or any other inbound surface. Its two siblings in the same family were moved to the transport seam (#9650/#9835 for the counter, #9834/#10004 for the histogram) and this one could not follow: `HttpResponseObservation` carries `{method, routePattern, status, elapsedMs}` and NO throw signal of any kind, so every transport-side shape would have counted a DIFFERENT population rather than the same one more widely. The divergence was measured in both directions — the dispatcher answers its own errors through `errorResponseBase`, which sets a status and does not re-throw, so the old counter MISSED those, while its `catch` incremented unconditionally, so a thrown 4xx WAS counted as an error. And `http_requests_total` already carries a `status` label, so a status-class error counter would be fully derivable from data the transport already publishes. Maintainer ruling 2026-08-20 (option C of four presented, over B "move it to the transport as a status class" and D "keep it dispatcher-scoped and rename it"): RETIRE. A metric NAME is a RESPONSE surface, not authorable metadata — no stack, example or template carries it, so there is no source for a D2 conversion to rewrite and no schema to tombstone; a host names the series in its own dashboard or alert file, outside this repo. That is exactly why this entry exists: for an operator whose Grafana keys on the string, the ledger is the only notification channel there is. Same disposition, and the same reason, as `runtime-httpserver-wrapper-retired` (#5122) and `enhanced-api-error-field-errors-renamed` (#3977). ADR-0049 / ADR-0087, #9834. + - Done when: No dashboard, alert rule or exporter config names `http_request_errors_total`: the series stops receiving samples the moment 17.2.0 is deployed, so a panel keyed on it draws a flat zero that reads as a healthy server rather than as a removed metric — the one failure mode this retirement can produce, and the reason the changeset announces it loudly. A 5xx-rate panel or alert is rewritten to `http_requests_total{status=~"5.."}` and then PROVEN wider, not merely non-empty: make an auth route or a REST data-API route answer 5xx and confirm the new query moves, where the retired counter would not have moved at all. If the signal you were actually alerting on was "a handler threw rather than returning an error envelope", that is the `errorReporter`, not a counter — wire an APM adapter and assert one synthetic 5xx throw arrives. In code, `SEMCONV.httpRequestErrorsTotal` and `RUNTIME_METRICS.httpRequestErrorsTotal` no longer resolve (tsc reports TS2339 at any surviving read) and no `metrics.counter` call names the string. - **`http-server-runtime-vocabulary-retired`** — `system.serverEvent / system.serverEventType / system.serverCapabilities / system.serverStatus (the lifecycle-event, capability-report and status vocabulary of system/http-server.zod.ts — 4 defs, 8 exported names)` → (removed — there is no replacement key, because there was never a key. Server lifecycle is the transport plugin's own start/stop seam; per-request and per-server observability is `system/metrics.zod.ts` and `system/logging.zod.ts` (plus `OS_SERVER_TIMING` for timings), and liveness is the `/health` endpoint. What a transport plugin can DO it states by implementing the kernel plugin contract — the seams it registers are the capability statement, and a self-described capability record can only disagree with them. Server-level configuration that IS authorable lives on `defineStack({ server })` / `StackServerConfigSchema`, which is unaffected) - Why not automatic: The second and final ADR-0049 pass over `system/http-server.zod.ts`. #4938 removed the CONFIG half (`HttpServerConfigSchema`, nine keys, zero readers, zero authoring entry); this removes the RUNTIME half — a 7-member lifecycle event union with a timestamped envelope, an eight-boolean capability report, and a five-state status record with connection and request counters. Nothing ever emitted, consumed or parsed any of them. This card was HELD for four days rather than queued, on a specific and legitimate doubt: a response/capability vocabulary can be a REFERENCE surface for host implementers, so "zero consumers in this repo" is weaker evidence for one of those than for an authorable key (the CSS-variable rebuttal). The hold was lifted by measuring the reference reader itself rather than by re-running the same grep: `plugin-hono-server`, the one in-tree host implementation, neither implements nor reports any of the three — it names no capability record, no status shape and no event union, and what it registers is routes and middleware through the kernel plugin contract. A declaration-site grep put every declaration in this one file, a quoted-name sweep across objectstack and objectui found no reader outside it, and the control passed in the SAME run: `MiddlewareConfig`, declared twelve lines away, resolves to `packages/runtime/src/middleware.ts`. So the sweep could see a reader in this file when there was one. With no carrier key there is nothing to tombstone, and with no author there is no source or `sys_metadata` row for a D2 conversion to rewrite: RETIRED_DEFS_BY_MAJOR plus this entry are the declaration — route 3, the same shape as #4938 in this very file, #4834, #4988 and #5055. If host-implementer conformance becomes a real requirement it returns through the ENFORCE route: an adapter contract with a checker behind it, vocabulary second. ADR-0049, #5295. - Done when: No source imports `ServerEvent`, `ServerEventType`, `ServerEventSchema`, `ServerCapabilities`, `ServerCapabilitiesSchema`, `ServerCapabilitiesParsed`, `ServerStatus` or `ServerStatusSchema` from `@objectstack/spec/system` — a grep over consumer code resolves none of them, and `tsc` reports TS2724/TS2305 on any that survives. The route-registration half of the same module still resolves (`RouteHandlerMetadataSchema`, `MiddlewareType`, `MiddlewareConfigSchema`, `MiddlewareConfig`), and `StackServerConfigSchema` — the one authorable server surface — is untouched: a stack declaring `server: { trustProxy, security }` parses exactly as it did in 16.x. diff --git a/packages/observability/src/__tests__/semconv.test.ts b/packages/observability/src/__tests__/semconv.test.ts new file mode 100644 index 0000000000..9c71af5254 --- /dev/null +++ b/packages/observability/src/__tests__/semconv.test.ts @@ -0,0 +1,38 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; +import { SEMCONV, RUNTIME_METRICS } from '../semconv.js'; + +describe('SEMCONV', () => { + it('declares the two HTTP families the transport seam emits', () => { + expect(SEMCONV.httpRequestsTotal).toBe('http_requests_total'); + expect(SEMCONV.httpRequestDurationMs).toBe('http_request_duration_ms'); + }); + + /** + * The retirement pin for `http_request_errors_total` (#9834, ADR-0049 + * enforce-or-remove, maintainer ruling 2026-08-20). + * + * It is asserted HERE, on the declaration, rather than only on the emission + * site in `@objectstack/runtime`: `SEMCONV` is the published namespace the + * docs point hosts at "so hosts can wire alerts/dashboards against it", so + * the name being back in this object is itself the regression — a host + * reading the constant would start naming a series nothing writes, which is + * the declared-not-enforced shape the retirement removed. + * + * Both directions are checked on purpose. The key check catches a re-add + * under the old member name; the VALUE check catches a re-add under a new + * member name that resurrects the same wire string, which is what an + * external dashboard actually keys on. + */ + it('does not declare the retired http_request_errors_total name (#9834)', () => { + expect(SEMCONV as Record).not.toHaveProperty( + 'httpRequestErrorsTotal', + ); + expect(Object.values(SEMCONV) as string[]).not.toContain('http_request_errors_total'); + expect(RUNTIME_METRICS as Record).not.toHaveProperty( + 'httpRequestErrorsTotal', + ); + expect(Object.values(RUNTIME_METRICS) as string[]).not.toContain('http_request_errors_total'); + }); +}); diff --git a/packages/observability/src/semconv.ts b/packages/observability/src/semconv.ts index f1e1c2910e..31e15edae4 100644 --- a/packages/observability/src/semconv.ts +++ b/packages/observability/src/semconv.ts @@ -34,15 +34,20 @@ export const SEMCONV = { * body parse included — not the handler's share of it. */ httpRequestDurationMs: 'http_request_duration_ms', - /** - * Counter, labels: `method`, `route`. Incremented when an in-flight - * handler throws after the response is sent. Emitted by - * `@objectstack/runtime`'s `instrumentRouteHandler`, and NOT movable to - * the seam above as-is: the observation carries a status but no throw - * signal, so a transport-side emitter would count a different population - * (#9834 records the fork). - */ - httpRequestErrorsTotal: 'http_request_errors_total', + // ⛔ RETIRED — `http_request_errors_total` was removed in + // `@objectstack/observability` 17.2.0 (#9834, ADR-0049 enforce-or-remove). + // ⛔ Do not re-add the name. It was DECLARED here as a stable server-wide + // signal and EMITTED only from `@objectstack/runtime`'s per-route wrapper, + // on a THROWN handler — so it never saw auth's `getRawApp()` mount, the + // REST data API, or any error a handler answered politely through + // `errorResponseBase` (which sets a status and does not re-throw). No + // transport-side emitter could preserve that population either: the + // `IHttpServer.afterResponse` observation carries `{method, routePattern, + // status, elapsedMs}` and no throw signal at all. + // ⇒ Read the 5xx rate from `http_requests_total{status=~"5.."}` instead. + // The transport emits that family through the seam, so it covers every + // inbound surface (#9650 / #9835 / #10004) and carries the status label + // this counter only stood in for. Maintainer ruling 2026-08-20. // ── Storage — emitted by `@objectstack/service-storage` adapters ── /** Counter, labels: `adapter` (`local`|`s3`|…), `op` (`get`|`put`|`delete`|`head`), `result` (`ok`|`error`). */ @@ -117,5 +122,8 @@ export const SEMCONV = { export const RUNTIME_METRICS = { httpRequestsTotal: SEMCONV.httpRequestsTotal, httpRequestDurationMs: SEMCONV.httpRequestDurationMs, - httpRequestErrorsTotal: SEMCONV.httpRequestErrorsTotal, + // `httpRequestErrorsTotal` retired with its SEMCONV declaration above + // (#9834). The alias is not a compatibility window of its own: there is no + // emitter left to read, so keeping the name here would hand callers a + // string nothing ever writes. } as const; diff --git a/packages/runtime/README.md b/packages/runtime/README.md index 7550f91469..ef10e371ec 100644 --- a/packages/runtime/README.md +++ b/packages/runtime/README.md @@ -613,9 +613,10 @@ if (!decision.allowed) reply.code(429).send({ retryAfterMs: decision.retryAfterM ### Observability (opt-in adapters) `createDispatcherPlugin` instruments every route with request-id propagation, -`http_requests_total{method,route,status}`, `http_request_duration_ms`, -`http_request_errors_total`, and 5xx error reporting. Plug your own -`MetricsRegistry` (Prometheus / OTel) and `ErrorReporter` (Sentry / Datadog). +`http_requests_total{method,route,status}`, `http_request_duration_ms`, and 5xx +error reporting. Plug your own `MetricsRegistry` (Prometheus / OTel) and +`ErrorReporter` (Sentry / Datadog). (`http_request_errors_total` was retired in +17.2.0, #9834 — read the 5xx rate from `http_requests_total{status=~"5.."}`.) Adapter recipes + go-live checklist in [`docs/OBSERVABILITY.md`](../../docs/OBSERVABILITY.md). diff --git a/packages/runtime/src/dispatcher-plugin.ts b/packages/runtime/src/dispatcher-plugin.ts index 2b9b90f4db..fa7f5ccaa5 100644 --- a/packages/runtime/src/dispatcher-plugin.ts +++ b/packages/runtime/src/dispatcher-plugin.ts @@ -82,10 +82,15 @@ export interface DispatcherPluginConfig { * Observability wiring. All fields optional; defaults are noop * (zero overhead, no behavior change). * - * - `metrics`: registry receiving `http_requests_total`, - * `http_request_duration_ms`, `http_request_errors_total` for - * every route this plugin mounts. Plug in `prom-client` / + * - `metrics`: registry receiving `http_requests_total` and + * `http_request_duration_ms`. Both are emitted by the TRANSPORT + * through the `IHttpServer.afterResponse` seam when it offers one, so + * they cover every inbound request on the server rather than only the + * routes this plugin mounts; on a transport without the seam this + * plugin emits them for its own routes instead. Plug in `prom-client` / * `@opentelemetry/api-metrics` / your own adapter. + * (`http_request_errors_total` was retired by #9834 — read the 5xx rate + * from `http_requests_total{status=~"5.."}`.) * * - `errorReporter`: invoked on 5xx responses with the thrown * error and `{ requestId, method, route }`. Plug in Sentry / diff --git a/packages/runtime/src/observability/instrument.test.ts b/packages/runtime/src/observability/instrument.test.ts index a55ac39142..afd8d21702 100644 --- a/packages/runtime/src/observability/instrument.test.ts +++ b/packages/runtime/src/observability/instrument.test.ts @@ -55,6 +55,19 @@ function makeClock() { }; } +/** + * The retired name (#9834, ADR-0049 enforce-or-remove). Kept as a LITERAL + * rather than a `RUNTIME_METRICS` member on purpose: the constant is gone, so + * a member read would not compile, and the thing worth pinning is that nothing + * writes this SERIES — which is what an external dashboard keyed on the string + * would look for. Any sample under this name is the retirement coming undone. + */ +const RETIRED_ERROR_COUNTER = 'http_request_errors_total'; + +function retiredErrorCounterSamples(m: InMemoryMetricsRegistry) { + return m.samples.filter((s) => s.name === RETIRED_ERROR_COUNTER); +} + describe('instrumentRouteHandler', () => { let metrics: InMemoryMetricsRegistry; let errorReporter: InMemoryErrorReporter; @@ -146,7 +159,7 @@ describe('instrumentRouteHandler', () => { ).toEqual([15]); }); - it('emitHttpRequestsTotal: false suppresses ONLY the request counter — histogram, error counter, reporter and request-id stay on (#9835)', async () => { + it('emitHttpRequestsTotal: false suppresses ONLY the request counter — histogram, reporter and request-id stay on (#9835)', async () => { // The dispatcher passes this when the transport implements the // `IHttpServer.afterResponse` seam, which then owns the counter // (#9833's duplicate). Everything else the wrapper emits is NOT @@ -173,19 +186,20 @@ describe('instrumentRouteHandler', () => { ).toEqual([15]); expect(res.headers['X-Request-Id']).toBeTruthy(); - // The error counter is likewise ungated. + // A throw under the gate: the REPORTER is what survives it. The + // error counter used to be asserted here; #9834 retired it, so the + // assertion is now that nothing writes that series. const throwing = instrumentRouteHandler( 'POST', '/boom', async () => { throw new Error('kaboom'); }, - { metrics, emitHttpRequestsTotal: false }, + { metrics, errorReporter, emitHttpRequestsTotal: false }, ); await expect(throwing({ headers: {} }, makeRes())).rejects.toThrow('kaboom'); - expect( - metrics.totalCounter('http_request_errors_total', { route: '/boom' }), - ).toBe(1); + expect(errorReporter.captured).toHaveLength(1); + expect(retiredErrorCounterSamples(metrics)).toEqual([]); expect( metrics.totalCounter('http_requests_total', { route: '/boom' }), ).toBe(0); @@ -204,7 +218,7 @@ describe('instrumentRouteHandler', () => { ).toBe(1); }); - it('emitHttpRequestDurationMs: false suppresses ONLY the histogram — counter, error counter, reporter and request-id stay on (#9834)', async () => { + it('emitHttpRequestDurationMs: false suppresses ONLY the histogram — counter, reporter and request-id stay on (#9834)', async () => { // The dispatcher passes this once the transport's afterResponse // seam has the duration histogram armed on it. The counter is // gated by its OWN flag, so a transport that owns one family and @@ -239,12 +253,10 @@ describe('instrumentRouteHandler', () => { { metrics, errorReporter, emitHttpRequestDurationMs: false }, ); await expect(throwing({ headers: {} }, makeRes())).rejects.toThrow('kaboom'); - // The error counter has NO transport-side emitter to defer to — - // the observation carries no throw signal — so it is ungated on - // every transport (#9834 reported that fork rather than moving it). - expect( - metrics.totalCounter('http_request_errors_total', { route: '/boom' }), - ).toBe(1); + // The error counter had NO transport-side emitter to defer to — the + // observation carries no throw signal — and #9834's fork was ruled + // RETIRE rather than move, so no emitter is left on any transport. + expect(retiredErrorCounterSamples(metrics)).toEqual([]); expect(errorReporter.captured).toHaveLength(1); }); @@ -264,7 +276,7 @@ describe('instrumentRouteHandler', () => { ).toEqual([4]); }); - it('the two gates are INDEPENDENT — both off leaves request-id, the error counter and the reporter', async () => { + it('the two gates are INDEPENDENT — both off leaves request-id and the reporter', async () => { const clock = makeClock(); const wrapped = instrumentRouteHandler( 'POST', @@ -287,9 +299,7 @@ describe('instrumentRouteHandler', () => { expect( metrics.histogramValues('http_request_duration_ms', { route: '/both-off' }), ).toEqual([]); - expect( - metrics.totalCounter('http_request_errors_total', { route: '/both-off' }), - ).toBe(1); + expect(retiredErrorCounterSamples(metrics)).toEqual([]); expect(res.headers['X-Request-Id']).toBeTruthy(); expect(errorReporter.captured).toHaveLength(1); }); @@ -311,7 +321,7 @@ describe('instrumentRouteHandler', () => { ).toBe(1); }); - it('records errors counter and 5xx status on thrown errors', async () => { + it('records 5xx status on thrown errors — and emits NO error counter (#9834 retired it)', async () => { const wrapped = instrumentRouteHandler( 'POST', '/boom', @@ -321,15 +331,40 @@ describe('instrumentRouteHandler', () => { { metrics, errorReporter }, ); await expect(wrapped({ headers: {} }, makeRes())).rejects.toThrow('kaboom'); + // The tombstone, on the emission path that used to write it: the + // wrapper's own default composition, no gate flags set. This is the + // assertion that fails if the emission is ever restored. + expect(retiredErrorCounterSamples(metrics)).toEqual([]); + // What replaced it as the 5xx signal — same throw, on the family the + // TRANSPORT emits for every inbound surface. expect( - metrics.totalCounter('http_request_errors_total', { - method: 'POST', - route: '/boom', - }), + metrics.totalCounter('http_requests_total', { status: '500' }), ).toBe(1); + expect(errorReporter.captured).toHaveLength(1); + }); + + it('a THROWN 4xx writes no error counter either — the retired series had no status filter (#9834)', async () => { + // Recorded because it is the half of the old population a status-class + // replacement would have dropped: the retired counter incremented + // unconditionally in the `catch`, so a thrown 400 WAS counted by it. + // Nothing counts it now; `http_requests_total{status="400"}` carries + // the request, and the reporter deliberately stays out of 4xx. + const wrapped = instrumentRouteHandler( + 'POST', + '/validate', + async () => { + const err: any = new Error('bad input'); + err.statusCode = 400; + throw err; + }, + { metrics, errorReporter }, + ); + await expect(wrapped({ headers: {} }, makeRes())).rejects.toThrow('bad input'); + expect(retiredErrorCounterSamples(metrics)).toEqual([]); expect( - metrics.totalCounter('http_requests_total', { status: '500' }), + metrics.totalCounter('http_requests_total', { status: '400' }), ).toBe(1); + expect(errorReporter.captured).toHaveLength(0); }); it('uses err.statusCode when present (e.g. 400 from validation)', async () => { diff --git a/packages/runtime/src/observability/instrument.ts b/packages/runtime/src/observability/instrument.ts index 9ff57418ae..d472fa0a11 100644 --- a/packages/runtime/src/observability/instrument.ts +++ b/packages/runtime/src/observability/instrument.ts @@ -61,9 +61,9 @@ export interface InstrumentOptions { * (the docs' p95 guidance wants the request's latency, not one layer's * share of it), but it is a change a dashboard can see. * - * NOT gated by either flag, on any transport: request-id echo, the error - * counter and the error reporter. The transport seam emits none of them — - * the observation carries no throw signal at all — so they always stay on. + * NOT gated by either flag, on any transport: request-id echo and the error + * reporter. The transport seam emits neither — the observation carries no + * throw signal at all — so they always stay on. */ emitHttpRequestDurationMs?: boolean; } @@ -80,8 +80,10 @@ export interface InstrumentOptions { * transport owns that family (see * {@link InstrumentOptions.emitHttpRequestsTotal} and * {@link InstrumentOptions.emitHttpRequestDurationMs}). - * 5. On thrown errors, emit `http_request_errors_total` and call - * `errorReporter.captureException` for 5xx. + * 5. On thrown errors, record the status (`err.statusCode ?? 500`, which + * reaches `http_requests_total{status}`) and call + * `errorReporter.captureException` for 5xx. No error COUNTER is emitted: + * `http_request_errors_total` was retired by #9834. * 6. When the handler catches its own error and calls * `errorResponseBase` (which leaves a side-channel * `res.__obsRecordedError`), still call the error reporter. @@ -138,7 +140,12 @@ export function instrumentRouteHandler( } catch (err: any) { threw = true; status = err?.statusCode ?? 500; - metrics.counter(RUNTIME_METRICS.httpRequestErrorsTotal, { method, route }); + // `http_request_errors_total` was emitted here until #9834 retired + // it (ADR-0049 enforce-or-remove, maintainer ruling 2026-08-20). + // ⛔ Do not re-add it. The throw is still fully reported: `status` + // lands on `http_requests_total{status}` in the `finally` below — + // which the TRANSPORT emits for every inbound surface — and 5xx + // still reaches the error reporter two lines down. if (status >= 500) { safeReport(errorReporter, err, { requestId, method, route }); } diff --git a/packages/runtime/src/observability/metrics.test.ts b/packages/runtime/src/observability/metrics.test.ts index b798fc9a2e..9384d284b0 100644 --- a/packages/runtime/src/observability/metrics.test.ts +++ b/packages/runtime/src/observability/metrics.test.ts @@ -82,6 +82,18 @@ describe('RUNTIME_METRICS', () => { it('exposes the canonical metric names', () => { expect(RUNTIME_METRICS.httpRequestsTotal).toBe('http_requests_total'); expect(RUNTIME_METRICS.httpRequestDurationMs).toBe('http_request_duration_ms'); - expect(RUNTIME_METRICS.httpRequestErrorsTotal).toBe('http_request_errors_total'); + }); + + it('no longer carries the retired http_request_errors_total name (#9834)', () => { + // The negative pin for the retirement, on the CONSTANT rather than on an + // emission: `RUNTIME_METRICS` is the published lookup a host reads to + // name its series, so a member reappearing here would re-publish the + // name even before anything wrote a sample. Asserted through a cast + // because the member is gone from the type — `tsc` is the other half of + // this pin and would reject a direct read. + expect(RUNTIME_METRICS as Record).not.toHaveProperty( + 'httpRequestErrorsTotal', + ); + expect(Object.values(RUNTIME_METRICS) as string[]).not.toContain('http_request_errors_total'); }); }); diff --git a/packages/spec/spec-changes.json b/packages/spec/spec-changes.json index 0a6c3cb327..abcbfbcfd7 100644 --- a/packages/spec/spec-changes.json +++ b/packages/spec/spec-changes.json @@ -796,6 +796,13 @@ "toMajor": 17, "rationale": "#4281 ruled that an empty hook target is not \"no target\" and closed the shape at the two METADATA doors — `HookSchema.object`'s refine and `hook-binder.ts`'s `normalizeObjects`. `engine.registerHook`, the CODE door, goes through neither, so all three spellings still registered, each producing a defect the author did not write: `''` is FALSY, so the allow face was skipped entirely and the entry became a GLOBAL hook (#4281's headline failure mode — blank intent taking the broadest possible blast radius); `[]` and `['']` are truthy but admit no object name, so the entry could never fire. #5928 then added the `excludeObjects` face, which brought a fourth shape reached by arithmetic rather than by one bad name: an `object` list every member of which is also excluded admits nothing, so that entry can never fire either. All four are ADR-0078 silently-inert declarations, and all four are now refused at REGISTRATION.\n\nNo mechanical rewrite exists, in either direction. The refused values carry no recoverable intent — `object: ''` could have meant `'*'` (what it actually did) or a specific object name the author forgot to fill in, and those are opposite registrations; choosing between them is a judgment the chain cannot make. Nor could the MATCHING read be changed instead: teaching the matcher that `''` is an unmatchable name would silently convert a hook firing on every object into one firing on none — the same class of defect pointing the other way, which is why #5928 declined to do it in passing.\n\nThis is a RUNTIME registration API, not stored metadata, so — like `hook-context-session-roles-retired` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and the ledger entry is the notification channel. One metadata surface reaches it INDIRECTLY and is the reason this is not purely a code-side note: a `record-change` flow's start node forwards `config.objectName` verbatim into `registerHook` (`RecordChangeTrigger.start`), so a flow authored with a blank `objectName` used to bind a trigger to EVERY object in the tenant. It now fails to bind instead, loudly — the automation engine's per-flow bind guard warns and the `kernel:bootstrapped` binding audit re-reports it — which is the correct end state, but it is an observable change for that flow. #6573, #4281, #4001, #5928, ADR-0078." }, + { + "surface": "observability.SEMCONV.httpRequestErrorsTotal (the published metric name http_request_errors_total{method,route}, and its emission from the runtime dispatcher's per-route wrapper)", + "replacement": "the 5xx rate is `http_requests_total{status=~\"5..\"}` — the TRANSPORT emits that family through the `IHttpServer.afterResponse` seam, so it covers every inbound surface; unhandled-exception rate specifically, which is the one thing the retired counter uniquely reported, is the `errorReporter` (Sentry / Datadog / your adapter), which still fires on every 5xx throw", + "migrationId": "http-request-errors-total-retired", + "toMajor": 17, + "rationale": "ADR-0049 enforce-or-remove, on a DECLARED-not-enforced metric name. `SEMCONV` published `http_request_errors_total` as part of a stable namespace declared \"so hosts can wire alerts/dashboards against it\", but the only emitter was `@objectstack/runtime`'s `instrumentRouteHandler`, applied only by the dispatcher's own route Proxy — so the series never saw auth's `getRawApp()` mount, the REST data API via `RouteManager`, or any other inbound surface. Its two siblings in the same family were moved to the transport seam (#9650/#9835 for the counter, #9834/#10004 for the histogram) and this one could not follow: `HttpResponseObservation` carries `{method, routePattern, status, elapsedMs}` and NO throw signal of any kind, so every transport-side shape would have counted a DIFFERENT population rather than the same one more widely. The divergence was measured in both directions — the dispatcher answers its own errors through `errorResponseBase`, which sets a status and does not re-throw, so the old counter MISSED those, while its `catch` incremented unconditionally, so a thrown 4xx WAS counted as an error. And `http_requests_total` already carries a `status` label, so a status-class error counter would be fully derivable from data the transport already publishes. Maintainer ruling 2026-08-20 (option C of four presented, over B \"move it to the transport as a status class\" and D \"keep it dispatcher-scoped and rename it\"): RETIRE. A metric NAME is a RESPONSE surface, not authorable metadata — no stack, example or template carries it, so there is no source for a D2 conversion to rewrite and no schema to tombstone; a host names the series in its own dashboard or alert file, outside this repo. That is exactly why this entry exists: for an operator whose Grafana keys on the string, the ledger is the only notification channel there is. Same disposition, and the same reason, as `runtime-httpserver-wrapper-retired` (#5122) and `enhanced-api-error-field-errors-renamed` (#3977). ADR-0049 / ADR-0087, #9834." + }, { "surface": "system.serverEvent / system.serverEventType / system.serverCapabilities / system.serverStatus (the lifecycle-event, capability-report and status vocabulary of system/http-server.zod.ts — 4 defs, 8 exported names)", "replacement": "(removed — there is no replacement key, because there was never a key. Server lifecycle is the transport plugin's own start/stop seam; per-request and per-server observability is `system/metrics.zod.ts` and `system/logging.zod.ts` (plus `OS_SERVER_TIMING` for timings), and liveness is the `/health` endpoint. What a transport plugin can DO it states by implementing the kernel plugin contract — the seams it registers are the capability statement, and a self-described capability record can only disagree with them. Server-level configuration that IS authorable lives on `defineStack({ server })` / `StackServerConfigSchema`, which is unaffected)", @@ -1860,6 +1867,13 @@ "toMajor": 17, "rationale": "#4281 ruled that an empty hook target is not \"no target\" and closed the shape at the two METADATA doors — `HookSchema.object`'s refine and `hook-binder.ts`'s `normalizeObjects`. `engine.registerHook`, the CODE door, goes through neither, so all three spellings still registered, each producing a defect the author did not write: `''` is FALSY, so the allow face was skipped entirely and the entry became a GLOBAL hook (#4281's headline failure mode — blank intent taking the broadest possible blast radius); `[]` and `['']` are truthy but admit no object name, so the entry could never fire. #5928 then added the `excludeObjects` face, which brought a fourth shape reached by arithmetic rather than by one bad name: an `object` list every member of which is also excluded admits nothing, so that entry can never fire either. All four are ADR-0078 silently-inert declarations, and all four are now refused at REGISTRATION.\n\nNo mechanical rewrite exists, in either direction. The refused values carry no recoverable intent — `object: ''` could have meant `'*'` (what it actually did) or a specific object name the author forgot to fill in, and those are opposite registrations; choosing between them is a judgment the chain cannot make. Nor could the MATCHING read be changed instead: teaching the matcher that `''` is an unmatchable name would silently convert a hook firing on every object into one firing on none — the same class of defect pointing the other way, which is why #5928 declined to do it in passing.\n\nThis is a RUNTIME registration API, not stored metadata, so — like `hook-context-session-roles-retired` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and the ledger entry is the notification channel. One metadata surface reaches it INDIRECTLY and is the reason this is not purely a code-side note: a `record-change` flow's start node forwards `config.objectName` verbatim into `registerHook` (`RecordChangeTrigger.start`), so a flow authored with a blank `objectName` used to bind a trigger to EVERY object in the tenant. It now fails to bind instead, loudly — the automation engine's per-flow bind guard warns and the `kernel:bootstrapped` binding audit re-reports it — which is the correct end state, but it is an observable change for that flow. #6573, #4281, #4001, #5928, ADR-0078." }, + { + "surface": "observability.SEMCONV.httpRequestErrorsTotal (the published metric name http_request_errors_total{method,route}, and its emission from the runtime dispatcher's per-route wrapper)", + "replacement": "the 5xx rate is `http_requests_total{status=~\"5..\"}` — the TRANSPORT emits that family through the `IHttpServer.afterResponse` seam, so it covers every inbound surface; unhandled-exception rate specifically, which is the one thing the retired counter uniquely reported, is the `errorReporter` (Sentry / Datadog / your adapter), which still fires on every 5xx throw", + "migrationId": "http-request-errors-total-retired", + "toMajor": 17, + "rationale": "ADR-0049 enforce-or-remove, on a DECLARED-not-enforced metric name. `SEMCONV` published `http_request_errors_total` as part of a stable namespace declared \"so hosts can wire alerts/dashboards against it\", but the only emitter was `@objectstack/runtime`'s `instrumentRouteHandler`, applied only by the dispatcher's own route Proxy — so the series never saw auth's `getRawApp()` mount, the REST data API via `RouteManager`, or any other inbound surface. Its two siblings in the same family were moved to the transport seam (#9650/#9835 for the counter, #9834/#10004 for the histogram) and this one could not follow: `HttpResponseObservation` carries `{method, routePattern, status, elapsedMs}` and NO throw signal of any kind, so every transport-side shape would have counted a DIFFERENT population rather than the same one more widely. The divergence was measured in both directions — the dispatcher answers its own errors through `errorResponseBase`, which sets a status and does not re-throw, so the old counter MISSED those, while its `catch` incremented unconditionally, so a thrown 4xx WAS counted as an error. And `http_requests_total` already carries a `status` label, so a status-class error counter would be fully derivable from data the transport already publishes. Maintainer ruling 2026-08-20 (option C of four presented, over B \"move it to the transport as a status class\" and D \"keep it dispatcher-scoped and rename it\"): RETIRE. A metric NAME is a RESPONSE surface, not authorable metadata — no stack, example or template carries it, so there is no source for a D2 conversion to rewrite and no schema to tombstone; a host names the series in its own dashboard or alert file, outside this repo. That is exactly why this entry exists: for an operator whose Grafana keys on the string, the ledger is the only notification channel there is. Same disposition, and the same reason, as `runtime-httpserver-wrapper-retired` (#5122) and `enhanced-api-error-field-errors-renamed` (#3977). ADR-0049 / ADR-0087, #9834." + }, { "surface": "system.serverEvent / system.serverEventType / system.serverCapabilities / system.serverStatus (the lifecycle-event, capability-report and status vocabulary of system/http-server.zod.ts — 4 defs, 8 exported names)", "replacement": "(removed — there is no replacement key, because there was never a key. Server lifecycle is the transport plugin's own start/stop seam; per-request and per-server observability is `system/metrics.zod.ts` and `system/logging.zod.ts` (plus `OS_SERVER_TIMING` for timings), and liveness is the `/health` endpoint. What a transport plugin can DO it states by implementing the kernel plugin contract — the seams it registers are the capability statement, and a self-described capability record can only disagree with them. Server-level configuration that IS authorable lives on `defineStack({ server })` / `StackServerConfigSchema`, which is unaffected)", diff --git a/packages/spec/src/migrations/entries/semantic/17.http-request-errors-total-retired.ts b/packages/spec/src/migrations/entries/semantic/17.http-request-errors-total-retired.ts new file mode 100644 index 0000000000..06b4b0e88a --- /dev/null +++ b/packages/spec/src/migrations/entries/semantic/17.http-request-errors-total-retired.ts @@ -0,0 +1,57 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import type { SemanticMigration } from '../../types.js'; + +export const entry: SemanticMigration = { + id: 'http-request-errors-total-retired', + surface: + 'observability.SEMCONV.httpRequestErrorsTotal (the published metric name ' + + 'http_request_errors_total{method,route}, and its emission from the runtime ' + + "dispatcher's per-route wrapper)", + replacement: + 'the 5xx rate is `http_requests_total{status=~"5.."}` — the TRANSPORT emits that family ' + + 'through the `IHttpServer.afterResponse` seam, so it covers every inbound surface; ' + + 'unhandled-exception rate specifically, which is the one thing the retired counter ' + + 'uniquely reported, is the `errorReporter` (Sentry / Datadog / your adapter), which ' + + 'still fires on every 5xx throw', + reason: + 'ADR-0049 enforce-or-remove, on a DECLARED-not-enforced metric name. `SEMCONV` ' + + 'published `http_request_errors_total` as part of a stable namespace declared "so ' + + 'hosts can wire alerts/dashboards against it", but the only emitter was ' + + '`@objectstack/runtime`\'s `instrumentRouteHandler`, applied only by the dispatcher\'s ' + + 'own route Proxy — so the series never saw auth\'s `getRawApp()` mount, the REST data ' + + 'API via `RouteManager`, or any other inbound surface. Its two siblings in the same ' + + 'family were moved to the transport seam (#9650/#9835 for the counter, #9834/#10004 ' + + 'for the histogram) and this one could not follow: `HttpResponseObservation` carries ' + + '`{method, routePattern, status, elapsedMs}` and NO throw signal of any kind, so every ' + + 'transport-side shape would have counted a DIFFERENT population rather than the same ' + + 'one more widely. The divergence was measured in both directions — the dispatcher ' + + 'answers its own errors through `errorResponseBase`, which sets a status and does not ' + + 're-throw, so the old counter MISSED those, while its `catch` incremented ' + + 'unconditionally, so a thrown 4xx WAS counted as an error. And `http_requests_total` ' + + 'already carries a `status` label, so a status-class error counter would be fully ' + + 'derivable from data the transport already publishes. Maintainer ruling 2026-08-20 ' + + '(option C of four presented, over B "move it to the transport as a status class" and ' + + 'D "keep it dispatcher-scoped and rename it"): RETIRE. A metric NAME is a RESPONSE ' + + 'surface, not authorable metadata — no stack, example or template carries it, so there ' + + 'is no source for a D2 conversion to rewrite and no schema to tombstone; a host names ' + + 'the series in its own dashboard or alert file, outside this repo. That is exactly why ' + + 'this entry exists: for an operator whose Grafana keys on the string, the ledger is the ' + + 'only notification channel there is. Same disposition, and the same reason, as ' + + '`runtime-httpserver-wrapper-retired` (#5122) and `enhanced-api-error-field-errors-renamed` ' + + '(#3977). ADR-0049 / ADR-0087, #9834.', + acceptanceCriteria: + 'No dashboard, alert rule or exporter config names `http_request_errors_total`: the ' + + 'series stops receiving samples the moment 17.2.0 is deployed, so a panel keyed on it ' + + 'draws a flat zero that reads as a healthy server rather than as a removed metric — ' + + 'the one failure mode this retirement can produce, and the reason the changeset ' + + 'announces it loudly. A 5xx-rate panel or alert is rewritten to ' + + '`http_requests_total{status=~"5.."}` and then PROVEN wider, not merely non-empty: ' + + 'make an auth route or a REST data-API route answer 5xx and confirm the new query ' + + 'moves, where the retired counter would not have moved at all. If the signal you were ' + + 'actually alerting on was "a handler threw rather than returning an error envelope", ' + + 'that is the `errorReporter`, not a counter — wire an APM adapter and assert one ' + + 'synthetic 5xx throw arrives. In code, `SEMCONV.httpRequestErrorsTotal` and ' + + '`RUNTIME_METRICS.httpRequestErrorsTotal` no longer resolve (tsc reports TS2339 at any ' + + 'surviving read) and no `metrics.counter` call names the string.', +}; diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index 5ff2ab96a0..d55e82a419 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -3469,6 +3469,59 @@ const step17: MigrationStep = { + '"[ObjectQL] Hook ... declares an empty `object` target" throw and no ' + '"[record-change] ... not bound" warning naming a flow you expect to fire.', }, + { + id: 'http-request-errors-total-retired', + surface: + 'observability.SEMCONV.httpRequestErrorsTotal (the published metric name ' + + 'http_request_errors_total{method,route}, and its emission from the runtime ' + + "dispatcher's per-route wrapper)", + replacement: + 'the 5xx rate is `http_requests_total{status=~"5.."}` — the TRANSPORT emits that family ' + + 'through the `IHttpServer.afterResponse` seam, so it covers every inbound surface; ' + + 'unhandled-exception rate specifically, which is the one thing the retired counter ' + + 'uniquely reported, is the `errorReporter` (Sentry / Datadog / your adapter), which ' + + 'still fires on every 5xx throw', + reason: + 'ADR-0049 enforce-or-remove, on a DECLARED-not-enforced metric name. `SEMCONV` ' + + 'published `http_request_errors_total` as part of a stable namespace declared "so ' + + 'hosts can wire alerts/dashboards against it", but the only emitter was ' + + '`@objectstack/runtime`\'s `instrumentRouteHandler`, applied only by the dispatcher\'s ' + + 'own route Proxy — so the series never saw auth\'s `getRawApp()` mount, the REST data ' + + 'API via `RouteManager`, or any other inbound surface. Its two siblings in the same ' + + 'family were moved to the transport seam (#9650/#9835 for the counter, #9834/#10004 ' + + 'for the histogram) and this one could not follow: `HttpResponseObservation` carries ' + + '`{method, routePattern, status, elapsedMs}` and NO throw signal of any kind, so every ' + + 'transport-side shape would have counted a DIFFERENT population rather than the same ' + + 'one more widely. The divergence was measured in both directions — the dispatcher ' + + 'answers its own errors through `errorResponseBase`, which sets a status and does not ' + + 're-throw, so the old counter MISSED those, while its `catch` incremented ' + + 'unconditionally, so a thrown 4xx WAS counted as an error. And `http_requests_total` ' + + 'already carries a `status` label, so a status-class error counter would be fully ' + + 'derivable from data the transport already publishes. Maintainer ruling 2026-08-20 ' + + '(option C of four presented, over B "move it to the transport as a status class" and ' + + 'D "keep it dispatcher-scoped and rename it"): RETIRE. A metric NAME is a RESPONSE ' + + 'surface, not authorable metadata — no stack, example or template carries it, so there ' + + 'is no source for a D2 conversion to rewrite and no schema to tombstone; a host names ' + + 'the series in its own dashboard or alert file, outside this repo. That is exactly why ' + + 'this entry exists: for an operator whose Grafana keys on the string, the ledger is the ' + + 'only notification channel there is. Same disposition, and the same reason, as ' + + '`runtime-httpserver-wrapper-retired` (#5122) and `enhanced-api-error-field-errors-renamed` ' + + '(#3977). ADR-0049 / ADR-0087, #9834.', + acceptanceCriteria: + 'No dashboard, alert rule or exporter config names `http_request_errors_total`: the ' + + 'series stops receiving samples the moment 17.2.0 is deployed, so a panel keyed on it ' + + 'draws a flat zero that reads as a healthy server rather than as a removed metric — ' + + 'the one failure mode this retirement can produce, and the reason the changeset ' + + 'announces it loudly. A 5xx-rate panel or alert is rewritten to ' + + '`http_requests_total{status=~"5.."}` and then PROVEN wider, not merely non-empty: ' + + 'make an auth route or a REST data-API route answer 5xx and confirm the new query ' + + 'moves, where the retired counter would not have moved at all. If the signal you were ' + + 'actually alerting on was "a handler threw rather than returning an error envelope", ' + + 'that is the `errorReporter`, not a counter — wire an APM adapter and assert one ' + + 'synthetic 5xx throw arrives. In code, `SEMCONV.httpRequestErrorsTotal` and ' + + '`RUNTIME_METRICS.httpRequestErrorsTotal` no longer resolve (tsc reports TS2339 at any ' + + 'surviving read) and no `metrics.counter` call names the string.', + }, { id: 'http-server-runtime-vocabulary-retired', surface: