fix(service-messaging): store the delivery signature, not the signing secret (#7722)

.changeset/webhook-delivery-signature-not-secret.md

@@ -0,0 +1,44 @@
+---
+"@objectstack/service-messaging": patch
+"@objectstack/plugin-webhooks": patch
+---
+
+fix(service-messaging): stop persisting webhook HMAC signing secrets on every delivery row (#7722)
+
+`sys_http_delivery` carried the caller's `signingSecret` verbatim, once per
+delivery attempt, in a plain `signing_secret` column — and that table is
+readable over the ordinary data API (`GET /api/v1/data/sys_http_delivery`).
+Anyone who could read deliveries recovered the shared key that authenticates
+ObjectStack to the receiver, for **every** subscriber at once. The signature is
+the receiver's only proof of origin, so the blast radius reaches outside the
+deployment: a leaked key mints payloads the receiver accepts as genuine, and
+rotating it means re-coordinating with every receiver operator.
+
+**The row now carries the signature, not the key.** A delivery's body is
+decided at enqueue and replayed byte-for-byte by every retry and by
+`redeliver()` — so the HMAC has exactly one correct value for the row's whole
+life. `enqueue()` computes it once from the producer's secret and stores only
+the result (`signature`, `sha256=<hex>`); the secret is consumed and dropped.
+The stored value is what the receiver is handed on the wire anyway and is
+one-way in the key, so reading a delivery row tells you what was sent, not how
+to forge something else.
+
+Signing behaviour on the wire is unchanged: `X-Objectstack-Signature` still
+carries `sha256=HMAC-SHA256(raw body, secret)` and verifies against the
+subscriber's secret exactly as before — now pinned by tests that recompute the
+HMAC over the delivered body rather than asserting a header is merely present,
+and by an at-rest guard that byte-scans every column of a real delivery table
+after a real delivery.
+
+Producers are unaffected: `enqueueHttp({ …, signingSecret })` keeps its shape
+for both callers (webhook fan-out and the Flow `http` node), and the fix sits at
+the outbox, so both stop writing cleartext.
+
+**Upgrading.** The `signing_secret` column is no longer declared, so an existing
+database keeps it as an unmapped column holding the old cleartext until it is
+dropped: run `os migrate plan` and apply the reported `drop_column` op (it is
+classified destructive, so it is never applied unattended). Until then those
+rows also age out on the table's existing 30-day telemetry retention. Rotate any
+signing secret that was exposed. Code reading `HttpDelivery.signingSecret` off a
+row should read `signature` instead — the secret is not available there by
+design.

content/docs/automation/webhooks.mdx

@@ -122,9 +122,16 @@ node executor and webhook fan-out, so both inherit retry / idempotency /
 dead-letter from one substrate. Webhook deliveries are the rows with
 `source = 'webhook'`. It generalises the old design's per-webhook table:
 `webhook_id` → `ref_id`, `event_id` → `dedup_key`, `event_type` → `label`,
-`secret` → `signing_secret`. Rows are managed by the platform and not directly
+`secret` → `signature`. Rows are managed by the platform and not directly
 writable.
 
+The row carries the **signature**, never the signing secret. The body is fixed
+at enqueue and replayed byte-for-byte by retries and redelivery, so the HMAC has
+exactly one correct value: the outbox computes it once at enqueue from the
+subscriber's `secret` and stores only the result — the same `sha256=<hex>` the
+receiver is handed on the wire. Reading a delivery row tells you what was sent,
+not how to forge it.
+
 | Field | Type | Notes |
 |-------------------|----------|-----------------------------------------------------------------------------|
 | `id` | text | Primary key. Also doubles as the receiver-side idempotency key. |
@@ -135,7 +142,7 @@ writable.
 | `url` | text | Target URL, snapshotted at enqueue so config edits do not rewrite live rows. |
 | `method` | text | HTTP method. |
 | `headers_json` | textarea | Custom headers, serialised. |
-| `signing_secret` | text | HMAC secret used for the signature header (see §6).  |
+| `signature` | text | The `X-Objectstack-Signature` value sent with this delivery, `sha256=<hex>` — computed at enqueue; the secret is not stored (see §6). |
 | `timeout_ms` | number | Per-attempt timeout. |
 | `payload_json` | textarea | The full payload that will be POSTed. |
 | `partition_key` | number | `hash(ref_id) mod partitionCount`, precomputed for cheap `WHERE`. |
@@ -432,6 +439,14 @@ Where the hex value is `HMAC-SHA256( secret, raw_request_body )`. There is no
 timestamp, no `{t}.{body}` concatenation, no version (`v1`) prefix, and no
 replay window — the signature covers the body bytes only.
 
+The signature is computed **once, at enqueue**, and stored on the delivery row
+(`signature`); the secret is not. The body is decided at enqueue and every
+retry replays it byte-for-byte, so one HMAC is correct for every attempt — and
+a delivery row read over the data API exposes the signature that was sent, not
+the key that can mint new ones. Rotating a webhook's `secret` changes what
+*subsequent* deliveries are signed with; rows already enqueued keep the
+signature that matches the body they carry.
+
 ### 6.2 Receiver-side verification
 
 Receivers MUST:

packages/plugins/plugin-webhooks/package.json

@@ -29,6 +29,7 @@
  "@objectstack/spec": "workspace:*"
  },
  "devDependencies": {
+ "@objectstack/metadata-core": "workspace:*",
  "@types/node": "^26.1.2",
  "typescript": "^6.0.3",
  "vitest": "^4.1.10"

packages/plugins/plugin-webhooks/src/webhook-signing-secret.test.ts

@@ -0,0 +1,146 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * #7722 — the webhook chain, end to end: a subscriber's signing secret reaches
+ * the receiver as a verifiable signature and reaches the delivery row as
+ * nothing at all.
+ *
+ * The enqueuer still hands `signingSecret` to the outbox (it belongs to the
+ * subscriber and the outbox needs it for exactly one HMAC); what changed is that
+ * the outbox consumes it instead of copying it onto the attempt row. This test
+ * drives the REAL classes on both sides of that hand-off — `AutoEnqueuer`,
+ * `MemoryHttpOutbox` (same enqueue path as the SQL outbox), `HttpDispatcher` —
+ * from a `data.record.created` event, because a unit test of either half alone
+ * cannot see a secret that leaks between them.
+ *
+ * The at-rest scan against a real SQL table lives next to the outbox itself:
+ * `service-messaging/src/http-signature-at-rest.integration.test.ts`.
+ */
+
+import { randomUUID, createHmac } from 'node:crypto';
+import { describe, expect, it } from 'vitest';
+import { DataEventSchema } from '@objectstack/spec/api';
+import type {
+ IDataEngine,
+ IRealtimeService,
+ RealtimeEventHandler,
+ RealtimeEventPayload,
+} from '@objectstack/spec/contracts';
+import { assertEngineDeleteDispatch, assertEngineUpdateDispatch } from '@objectstack/metadata-core';
+import { MemoryHttpOutbox, HttpDispatcher, type FetchImpl } from '@objectstack/service-messaging';
+import { AutoEnqueuer } from './auto-enqueuer.js';
+
+const SECRET = 'whsec_7722_subscriber_key';
+
+class FakeRealtime implements IRealtimeService {
+ private subs = new Map<string, { handler: RealtimeEventHandler; opts?: any }>();
+ private n = 0;
+ async publish(event: RealtimeEventPayload): Promise<void> {
+ for (const sub of this.subs.values()) {
+ const o = sub.opts ?? {};
+ if (o.object && event.object !== o.object) continue;
+ await sub.handler(event);
+ }
+ }
+ async subscribe(_channel: string, handler: any, opts?: any): Promise<string> {
+ const id = `s-${++this.n}`;
+ this.subs.set(id, { handler, opts });
+ return id;
+ }
+ async unsubscribe(id: string): Promise<void> {
+ this.subs.delete(id);
+ }
+}
+
+/**
+ * Just enough engine to serve the `sys_webhook` subscription cache.
+ *
+ * The write verbs open with the PRODUCER's own dispatch predicates
+ * (`assertEngineUpdateDispatch` / `assertEngineDeleteDispatch` from
+ * `@objectstack/metadata-core`) rather than accepting anything, because a
+ * double looser than the real engine turns a green suite into no suite at all
+ * on exactly the call shapes `ObjectQL` refuses (#4434) — the failure this
+ * file's own subject is a cousin of. `check:engine-double-contract` pins it.
+ */
+function makeEngine(rows: any[]): IDataEngine {
+ return {
+ async find() { return rows; },
+ async findOne() { return rows[0] ?? null; },
+ async insert(_n: string, d: any) { return d; },
+ async update(_n: string, data: any, options?: any) {
+ assertEngineUpdateDispatch(data, options);
+ return { affected: 0 };
+ },
+ async delete(_n: string, options?: any) {
+ assertEngineDeleteDispatch(options);
+ return { affected: 0 };
+ },
+ async count() { return rows.length; },
+ async aggregate() { return []; },
+ } as unknown as IDataEngine;
+}
+
+function recordEvent(object: string, record: any): RealtimeEventPayload {
+ const payload = DataEventSchema.parse({
+ id: randomUUID(),
+ type: 'data.record.created',
+ object,
+ recordId: String(record.id),
+ after: record,
+ timestamp: '2026-08-11T00:00:00.000Z',
+ });
+ return { type: payload.type, object, payload: { ...payload }, timestamp: payload.timestamp };
+}
+
+function makeFetch() {
+ const calls: Array<{ headers: Record<string, string>; body: string }> = [];
+ const impl: FetchImpl = async (_url, init) => {
+ calls.push({ headers: init.headers, body: init.body });
+ return { ok: true, status: 200, async text() { return 'ok'; } };
+ };
+ return { impl, calls };
+}
+
+describe('webhook signing secret (#7722)', () => {
+ it('delivers a verifiable signature without persisting the subscriber secret', async () => {
+ const engine = makeEngine([
+ {
+ id: 'wh-1',
+ name: 'crm_hook',
+ active: true,
+ object_name: 'contact',
+ triggers: ['create'],
+ url: 'https://receiver.example/hook',
+ method: 'POST',
+ definition_json: JSON.stringify({ secret: SECRET, headers: { 'X-Team': 'crm' } }),
+ },
+ ]);
+ const realtime = new FakeRealtime();
+ const outbox = new MemoryHttpOutbox();
+ const enqueuer = new AutoEnqueuer(engine, realtime, (input) => outbox.enqueue(input));
+ await enqueuer.start();
+
+ await realtime.publish(recordEvent('contact', { id: 'c1', name: 'Ada', email: 'ada@example.com' }));
+ // The enqueue is deliberately fire-and-forget on the hot path.
+ await new Promise((r) => setTimeout(r, 0));
+
+ const { impl, calls } = makeFetch();
+ await new HttpDispatcher({ nodeId: 'n1', outbox, fetchImpl: impl, partitionCount: 1 }).tick();
+ await enqueuer.stop();
+
+ // ── The receiver's check: recompute HMAC-SHA256 over the raw body ──
+ expect(calls).toHaveLength(1);
+ const expected = createHmac('sha256', SECRET).update(calls[0].body).digest('hex');
+ expect(calls[0].headers['X-Objectstack-Signature']).toBe(`sha256=${expected}`);
+ // The signed body is the real event, and the subscriber's own headers
+ // still ride along — a blank delivery would satisfy the HMAC too.
+ expect(JSON.parse(calls[0].body)).toMatchObject({ object: 'contact', recordId: 'c1', action: 'created' });
+ expect(calls[0].headers['X-Team']).toBe('crm');
+
+ // ── …and no attempt row carries the key that produced it ──
+ const rows = await outbox.list();
+ expect(rows).toHaveLength(1);
+ expect(JSON.stringify(rows)).not.toContain(SECRET);
+ expect(rows[0].signature).toBe(`sha256=${expected}`);
+ });
+});

packages/services/service-messaging/package.json

@@ -25,7 +25,9 @@
  "@objectstack/types": "workspace:*"
  },
  "devDependencies": {
+ "@objectstack/driver-sql": "workspace:*",
  "@objectstack/metadata-core": "workspace:*",
+ "@objectstack/objectql": "workspace:*",
  "@types/node": "^26.1.2",
  "typescript": "^6.0.3",
  "vitest": "^4.1.10"

packages/services/service-messaging/src/http-outbox.test.ts

@@ -1,6 +1,7 @@
 // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
 
 import { describe, it, expect } from 'vitest';
+import { createHmac } from 'node:crypto';
 import { MemoryHttpOutbox } from './memory-http-outbox.js';
 import { HttpDispatcher } from './http-dispatcher.js';
 import type { FetchImpl } from './http-sender.js';
@@ -120,6 +121,28 @@ describe('HttpDispatcher', () => {
 
  await dispatcher.tick();
 
- expect(calls[0].headers['X-Objectstack-Signature']).toMatch(/^sha256=/);
+ // Verify the way a receiver does: recompute HMAC-SHA256 over the RAW
+ // body that arrived. `toMatch(/^sha256=/)` only proved a header was
+ // present — it would pass on a signature computed over other bytes.
+ const expected = createHmac('sha256', 's3cr3t').update(calls[0].body).digest('hex');
+ expect(calls[0].headers['X-Objectstack-Signature']).toBe(`sha256=${expected}`);
+ });
+
+ // [#7722] The secret is consumed by enqueue, never stored: the row the
+ // dispatcher works from carries the signature only.
+ it('keeps the signing secret off the delivery row', async () => {
+ const outbox = new MemoryHttpOutbox();
+ await outbox.enqueue({
+ source: 'webhook',
+ refId: 'w1',
+ dedupKey: 'e1',
+ url: 'https://x',
+ signingSecret: 's3cr3t',
+ payload: { a: 1 },
+ });
+
+ const [row] = await outbox.list();
+ expect(row.signature).toMatch(/^sha256=[0-9a-f]{64}$/);
+ expect(JSON.stringify(row)).not.toContain('s3cr3t');
  });
 });

packages/services/service-messaging/src/http-outbox.ts

@@ -9,6 +9,9 @@
  * idempotency — with retry / backoff / dead-letter handled by the shared
  * {@link HttpDispatcher}.
  *
+ * Rows are signed, never keyed: the HMAC signature is computed once at enqueue
+ * and the signing secret is discarded (#7722) — see {@link HttpDelivery.signature}.
+ *
  * It generalises the original `plugin-webhooks` outbox so two callers share one
  * reliable substrate:
  * - the Flow `http` node executor (`@objectstack/service-automation`), and
@@ -54,8 +57,22 @@ export interface HttpDelivery {
  method?: string;
  /** Custom headers. */
  headers?: Record<string, string>;
- /** HMAC-SHA256 secret. If present, an `X-Objectstack-Signature` header is added. */
- signingSecret?: string;
+ /**
+ * Pre-computed `X-Objectstack-Signature` value (`sha256=<hex>`), or absent
+ * for an unsigned delivery.
+ *
+ * **A delivery row never holds the signing secret** (#7722). The body is
+ * fixed at enqueue and never rewritten — retries and `redeliver()` replay
+ * the same bytes — so the HMAC has exactly one correct value, and the outbox
+ * computes it once, at enqueue, from {@link EnqueueHttpInput.signingSecret}.
+ * What lands on the row is the signature: a one-way function of
+ * (body, secret) that the receiver is handed on the wire anyway, so it
+ * authenticates this payload without being usable to forge another. The
+ * secret stays with the subscriber that owns it instead of being copied onto
+ * every attempt, where it used to sit in cleartext for anyone who could read
+ * `sys_http_delivery`.
+ */
+ signature?: string;
  /** Per-request timeout in ms. */
  timeoutMs?: number;
  /** JSON-serialisable body. */
@@ -92,6 +109,13 @@ export interface EnqueueHttpInput {
  url: string;
  method?: string;
  headers?: Record<string, string>;
+ /**
+ * HMAC-SHA256 secret. **Consumed, not stored** (#7722): `enqueue()` signs the
+ * body with it and persists only the resulting
+ * {@link HttpDelivery.signature}. It exists on this input — and nowhere on
+ * the delivery row — so a producer can keep owning its secret (a webhook
+ * subscriber's, a flow node's) without every attempt taking a cleartext copy.
+ */
  signingSecret?: string;
  timeoutMs?: number;
  payload: unknown;
@@ -177,8 +201,9 @@ export interface IHttpOutbox {
 
  /**
  * Reset a terminal row (`success` / `failed` / `dead`) back to `pending` so
- * the dispatcher re-sends it. Resets `attempts=0`; URL / payload / secret are
- * NOT touched (byte-for-byte replay). Throws {@link HttpRedeliverError}.
+ * the dispatcher re-sends it. Resets `attempts=0`; URL / payload / signature
+ * are NOT touched (byte-for-byte replay — the same body carries the same
+ * signature). Throws {@link HttpRedeliverError}.
  */
  redeliver(id: string): Promise<HttpDelivery>;
 }