fix(hono-server): emit http_requests_total from the transport so every inbound mount is counted (#9650)

.changeset/http-requests-total-transport-seam.md

@@ -0,0 +1,55 @@
+---
+"@objectstack/plugin-hono-server": minor
+---
+
+fix(hono-server): `http_requests_total` is emitted by the transport, so every inbound mount is counted (#9650)
+
+The counter had exactly one emitter: a `Proxy` the runtime dispatcher built
+over its **own** `IHttpServer` handle. It therefore saw only the routes the
+dispatcher itself registered — and nothing else on the same server.
+
+Measured, that left at least **14** inbound surfaces uncounted, in two
+structurally different classes:
+
+- plugins that mount through `getRawApp()` — auth (`/api/v1/auth/*`),
+ metadata HMR, cloud-connection, marketplace, runtime-config, trigger-api,
+ webhooks, approvals, the console SPA. These bypass `IHttpServer` entirely,
+ so **no** wrapper at that level can ever reach them.
+- plugins that resolve `http.server` themselves and mount through the verb
+ methods — the REST data API via `RouteManager`, storage, i18n, settings,
+ datasource admin.
+
+The two highest-traffic surfaces an operator actually cares about, auth and
+the REST data API, were both in that set. The documented guidance is to alert
+on the 5xx rate derived from this counter, so a deployment could be melting
+down on `/api/v1/*` with the counter flat.
+
+The counter is now emitted from the Hono adapter itself, as a raw-app
+middleware installed at the end of `HonoServerPlugin.init()` beside
+`installMiddlewareSeam()` — the one layer every inbound request converges on,
+whatever registered the handler.
+
+**The route label is the matched PATTERN, never the concrete path.**
+`/api/v1/data/:id`, not one series per record id; `/api/v1/auth/*`, not one
+per sign-in endpoint. Cardinality has to stay bounded or the counter is
+unusable for the alerting it exists for, and the label is unfixable in place
+once dashboards are wired against the first shipped one.
+
+**Wiring.** `HonoServerPlugin` takes a new `observability.metrics` option and
+otherwise follows the canonical chain `ObservabilityServicePlugin` documents:
+explicit option, then the `observability:metrics` service, then **nothing** —
+with no backend configured no middleware is installed at all, so an
+unconfigured deployment pays no per-request cost.
+
+**Two consequences, stated rather than left to be discovered:**
+
+- **A transport that does not implement this seam reports no HTTP metrics.**
+ The seam is Hono's. Another `IHttpServer` implementation emits nothing until
+ it grows its own, and a zero there means "not instrumented", never "no
+ traffic". A response-observing hook on the `IHttpServer` contract is the
+ transport-agnostic successor and is filed separately.
+- **A request the `use()` chain refuses is still counted** — the seam is
+ installed before the middleware seam, so the inbound rate limiter's `429`
+ appears with `status="429"`. A preflight `OPTIONS` that the transport's own
+ CORS built-in answers is **not** counted: it short-circuits earlier and
+ never reaches a route.

packages/plugins/plugin-hono-server/src/adapter.ts

@@ -11,8 +11,9 @@ import {
 } from '@objectstack/core';
 import type { Logger } from '@objectstack/spec/contracts';
 import type { Context } from 'hono';
-import { currentPerfTiming } from '@objectstack/observability';
+import { currentPerfTiming, RUNTIME_METRICS, type MetricsRegistry } from '@objectstack/observability';
 import { Hono } from 'hono';
+import { routePath } from 'hono/route';
 import { serve } from '@hono/node-server';
 import { serveStatic } from '@hono/node-server/serve-static';
 import { matchesRoutePattern } from './route-pattern';
@@ -227,6 +228,8 @@ export class HonoHttpServer implements IHttpServer {
  private middlewares: Array<{ path?: string; handler: Middleware }> = [];
  /** Whether the Hono middleware that runs {@link middlewares} is mounted. */
  private middlewareSeamInstalled = false;
+ /** Whether the `http_requests_total` middleware is mounted. See {@link installHttpMetricsSeam}. */
+ private httpMetricsSeamInstalled = false;
  /**
  * The LAST-RESORT handler installed by {@link setFallbackHandler}, or
  * `undefined` when no consumer installed one. Exactly one — installing
@@ -944,6 +947,91 @@ export class HonoHttpServer implements IHttpServer {
  });
  }
 
+ /**
+ * Mount the single Hono middleware that emits
+ * `http_requests_total{method,route,status}` for EVERY inbound request on
+ * this transport. Idempotent.
+ *
+ * ## Why the counter lives here and not one layer up (#9650)
+ *
+ * The counter used to be emitted by a wrapper the runtime dispatcher built
+ * over its own `IHttpServer` handle, so it saw only the routes the
+ * dispatcher itself registered. Everything else on the same server was
+ * invisible to it — measured, at least 14 inbound surfaces in two classes:
+ *
+ * - plugins that mount through {@link getRawApp} (auth, metadata HMR,
+ * cloud-connection, marketplace, runtime-config, trigger-api, webhooks,
+ * approvals, the console SPA). These bypass `IHttpServer` entirely, so
+ * NO wrapper at that level can ever reach them.
+ * - plugins that resolve `http.server` themselves and mount through the
+ * verb methods (the REST data API via `RouteManager`, storage, i18n,
+ * settings, datasource admin).
+ *
+ * An operator following the documented guidance ("alert on the 5xx rate
+ * from `http_requests_total`") was therefore watching a counter that could
+ * stay flat while `/api/v1/*` melted down. The transport is the one layer
+ * every inbound request already converges on, whatever registered the
+ * handler, which is why the counter is emitted from here.
+ *
+ * ## Route label is the PATTERN, never the concrete path
+ *
+ * `/api/v1/auth/*`, not `/api/v1/auth/sign-in/email`; `/api/v1/:object/:id`,
+ * not one series per record id. Cardinality has to stay bounded or the
+ * counter is unusable for exactly the alerting it exists for — and the
+ * label is unfixable in place once dashboards are wired against the first
+ * shipped one.
+ *
+ * ## Two consequences, stated so they are not discovered
+ *
+ * - **A transport that does not install this seam reports no HTTP
+ * metrics.** The seam is Hono's; another `IHttpServer` implementation
+ * emits nothing until it grows its own. Zero is "not instrumented",
+ * never "no traffic".
+ * - **WHERE it is mounted decides what is counted.** `HonoServerPlugin`
+ * installs it immediately BEFORE {@link installMiddlewareSeam}, so a
+ * request a `use()` middleware short-circuits (the inbound rate
+ * limiter's 429) is still counted — a refused request is exactly the
+ * one an operator is alerting on. It sits AFTER the transport's own
+ * CORS built-in, so a preflight `OPTIONS` that CORS answers itself
+ * never reaches a route and is not counted.
+ *
+ * @param metrics the host's registry. Resolve it with the canonical chain
+ * (explicit option → `observability:metrics` service → none); pass
+ * nothing at all rather than a no-op, so an unconfigured deployment pays
+ * no per-request cost.
+ */
+ installHttpMetricsSeam(metrics: MetricsRegistry): void {
+ if (this.httpMetricsSeamInstalled) return;
+ this.httpMetricsSeamInstalled = true;
+
+ this.app.use('*', async (c, next) => {
+ // Default 500: if `next()` rejects, Hono's error path renders the
+ // 500 and `c.res` is not yet set — reading it would synthesize a
+ // response and change what the caller receives.
+ let status = 500;
+ try {
+ await next();
+ status = c.res.status;
+ } finally {
+ try {
+ metrics.counter(RUNTIME_METRICS.httpRequestsTotal, {
+ method: c.req.method,
+ // `routePath(c)` is the non-deprecated spelling of
+ // `c.req.routePath` and returns the same matched
+ // PATTERN — read after `next()`, so it is the route
+ // that actually answered rather than this middleware's
+ // own `*`. Empty only when nothing matched at all.
+ route: routePath(c) || 'unmatched',
+ status: String(status),
+ });
+ } catch {
+ // A metrics backend must never be able to break a
+ // response. Same discipline as the error reporter.
+ }
+ }
+ });
+ }
+
  /**
  * Mount a sub-application or router
  */

packages/plugins/plugin-hono-server/src/hono-plugin.ts

@@ -21,7 +21,9 @@ import {
  runWithPerfDisclosure,
  allowPerfDisclosure,
  isPerfDisclosurePrincipal,
+ OBSERVABILITY_METRICS_SERVICE,
  type PerfDisclosureGate,
+ type MetricsRegistry,
 } from '@objectstack/observability';
 
 export interface StaticMount {
@@ -90,6 +92,27 @@ export interface HonoPluginOptions {
  * @default undefined
  */
  serverTiming?: boolean;
+
+ /**
+ * Observability backends for the transport's own signals.
+ *
+ * `metrics` receives `http_requests_total{method,route,status}` for every
+ * inbound request on this server — see
+ * {@link HonoHttpServer.installHttpMetricsSeam} for why the counter is
+ * emitted at the transport rather than one layer up (#9650).
+ *
+ * Resolution chain, the canonical one (`ObservabilityServicePlugin`):
+ *
+ * 1. this option — explicit wiring, and the escape hatch tests use;
+ * 2. the `observability:metrics` service, when the host registered one
+ * BEFORE this plugin;
+ * 3. neither — then **no middleware is installed at all**, so an
+ * unconfigured deployment pays no per-request cost. Not a disabled
+ * counter; no counter.
+ */
+ observability?: {
+ metrics?: MetricsRegistry;
+ };
 }
 
 /**
@@ -402,9 +425,56 @@ export class HonoServerPlugin implements Plugin {
  // moment at which a gate can still precede all of them — and placing it
  // here means a consumer's `use()` no longer has to win a race with route
  // registration. It can register whenever it has the facts.
+
+ // ─── `http_requests_total` seam (#9650) ───────────────────────────────
+ // Mounted immediately BEFORE the middleware seam, and that order is
+ // load-bearing in both directions:
+ //
+ // - before it, so a request the `use()` chain short-circuits (the
+ // dispatcher's inbound rate limiter answering 429) is still counted.
+ // A refused request is precisely the one an operator alerts on;
+ // counting only what got through would rebuild, one layer in, the
+ // blind spot this seam exists to close.
+ // - still at the END of `init()`, i.e. before any route exists, since
+ // every route in the platform is mounted in some plugin's `start()`
+ // and Hono composes the handlers that MATCHED in registration
+ // order. A middleware Hono learns about after a route runs after
+ // that route's handler — which is why installing this from a later
+ // phase (or from `kernel:bootstrapped`) observes nothing at all.
+ const metrics = this.resolveMetrics(ctx);
+ if (metrics) {
+ this.server.installHttpMetricsSeam(metrics);
+ ctx.logger.debug('HTTP request counter armed', {
+ metric: 'http_requests_total',
+ labels: 'method, route (pattern), status',
+ });
+ }
+
  this.server.installMiddlewareSeam();
  }
 
+ /**
+ * The canonical metrics-resolution chain, as documented on
+ * `ObservabilityServicePlugin`: explicit option, then the
+ * `observability:metrics` service, then nothing.
+ *
+ * Returns `undefined` rather than a no-op registry so the caller can skip
+ * installing the middleware entirely when no backend is configured.
+ */
+ private resolveMetrics(ctx: PluginContext): MetricsRegistry | undefined {
+ const explicit = this.options.observability?.metrics;
+ if (explicit) return explicit;
+ try {
+ const fromService = ctx.getService<MetricsRegistry | undefined>(
+ OBSERVABILITY_METRICS_SERVICE,
+ );
+ if (fromService) return fromService;
+ } catch {
+ // No host registry registered — the counter is simply absent.
+ }
+ return undefined;
+ }
+
  /**
  * Start phase - Configure static files and start listening
  */