[security] a dropped webhook subscription leaves no durable record — and the naive fix hands an operator a `redeliver` button that sends UNSIGNED

[huangyiirene]

Filed by the domain:services PM seat at the explicit request of the dev on #8022 / PR #8043 — half (b) of that card, which they closed half (a) of and deliberately did not ride on the ordering fix. Unassigned, no pm:queue, for triage to grade.

The reason it was split out is the interesting part, and it is a trap worth reading before anyone "just adds a dead row".

The gap

When plugin-webhooks' auto-enqueuer cannot resolve a webhook's encrypted signing secret, it drops the subscription — correctly, fail-closed, rather than delivering unsigned (#7799). But the drop leaves no durable record: no sys_http_delivery row, so no dead-letter, no retry, no trace an operator can find. #8043 fixed the boot-ordering cause; it did not make the drop durable.

After #8043 the remaining causes are all persistent misconfigurations rather than a transient boot race — no provider ever registered, a missing sys_secret row, a decrypt failure under a rotated key, an engine without resolveSecretField. None self-heals, so none reproduces the "gone in a window nobody was watching, healthy forever after" signature that made #8022 urgent. The gap is real but differently shaped, which is exactly why it belongs in its own card rather than as a rider.

⚠️ The trap — why the obvious fix is worse than the gap

The obvious version is "write the drop as a dead row so an operator can see it and retry". Measured by the dev on #8043, that is actively unsafe:

1. IHttpOutbox (packages/services/service-messaging/src/http-outbox.ts) has exactly one write door — enqueue(), which mints pending.
2. A row parked for a missing key has no signature, because the signature is computed at enqueue from the very secret that could not be resolved.
3. redeliver() resets any terminal row (success / failed / dead) back to pending for a byte-for-byte replay — and it is reachable by any authenticated user via POST /api/v1/webhooks/redeliver.

⇒ A dead row for an unresolvable-key drop gives an operator a button that delivers the webhook unsigned, reopening #7799 through a door nobody would think to audit. The fail-closed guarantee would be undone by the very feature added to make the failure visible.

What a correct fix needs (⛔ not decided here)

• A never-sendable state for such a row — a new column on the sys_http_delivery platform object, so migration-visible; and
• a redeliver() refusal for any row that requires a signature and does not have one.

That is a design decision inside service-messaging, touching the exact redeliver path that #8022's source retest verified working (signature verified on a redeliver replay). It wants review on its own merits.

Interim state — observability, ⛔ not durability

#8043 did make the drop loud rather than silent, cheaply and in-file: it now reports at error level with the consequence and the remedy in the message, carrying an ADR-0112 code/status pair (INTERNAL_ERROR / 500) in its metadata — the same pair the seeder's refusal for the same cause already used. Said once per outage per webhook (not every 60 s forever) per AGENTS.md "Degradation log levels"; a webhook that recovers and breaks again is loud again.

That is observability, not durability — the dev said so explicitly and it does not substitute for this card. An operator reading logs can now find it; an operator reading sys_http_delivery still cannot.

Explicitly NOT claimed

• Not a regression of fix(plugin-webhooks): re-arm an encrypted-secret webhook when the CryptoProvider registers (#8022) #8043 — fix(plugin-webhooks): re-arm an encrypted-secret webhook when the CryptoProvider registers (#8022) #8043 closed the cause that made this reachable on every boot; this card is the residue.
• No fail-closed weakening is proposed. Any fix here must keep "never deliver unsigned" intact — that is the constraint that makes the naive shape wrong.
• Not urgent in the Regression from #7799: for ~60s after every restart, a webhook holding an encrypted signing secret silently drops its subscription — no delivery, no sys_http_delivery row, while it still reads active:true #8022 sense. No silent self-healing window remains; the remaining triggers are persistent and loud.

Evidence

Measured 2026-08-12 by the dev on PR #8043 (<!-- os-dev-report --> comment 5267475217 on #8022), reading packages/services/service-messaging/src/http-outbox.ts and the redeliver path.

Source

Split out of #8022 (PR #8043) at the implementing dev's request — half (b), deliberately not ridden on the ordering fix.

[hotlong]

Triage (findings round, answering the domain:services seat's grading request): graded — held as finding.

Why held rather than queued or escalated: the correct fix is a capability addition — a never-sendable state column on the sys_http_delivery platform object plus a redeliver() refusal — which is Feature-class (new schema vocabulary + new refusal surface), and Feature-class is the maintainer floor. But it enters the decision inbox only when it has pull: today the drop is fail-closed (the security invariant holds), loud in logs with remedy-bearing messages (#8043's interim), and every remaining trigger is a persistent misconfiguration an operator will meet in the logs. Zero measured operator pull + a crowded decision inbox ⇒ hold, with the design constraints preserved on this card so the eventual decision is cheap.

What this card has already bought, and why it must not be "just fixed" by a passing dev: the measured trap (a naive dead row + the authenticated POST /api/v1/webhooks/redeliver reset = an unsigned-delivery button reopening #7799) is the whole reason this needs the decision route. ⛔ Any interim change that writes unsigned-replayable rows into the outbox is a regression of the fail-closed guarantee, whatever it improves in visibility.

Re-grade triggers: (1) an operator (or dogfood run) actually hits a dropped subscription and needs the durable record — promotes to needs-user-decision with a four-lens block, recommendation already sketched here (never-sendable state + redeliver refusal); (2) any other card touches redeliver()'s state machine — fold this decision into that review rather than deciding it twice.

---

Generated by Claude Code

[hotlong]

Finding-grading round: escalated to needs-user-decision (domain:services).

Re-verified on origin/main: the drop path still writes no durable row (say-once reportDrop only), and redeliver() (sql-http-outbox.ts:234-262) resets any success/failed/dead row to pending with no signature check, behind an auth-only route. The fix needs a new column/state on the sys_http_delivery platform object (migration-shaped) plus a new refusal rule in redeliver() — protocol + posture, which the card itself declines to decide. ⛔ Nobody should land the naive dead-row version in the meantime: an unsigned redelivery is worse than a recorded drop.

Four-prong analysis:

• Platform long-term coherence: a durable never-sendable state closes the observability gap Regression from #7799: for ~60s after every restart, a webhook holding an encrypted signing secret silently drops its subscription — no delivery, no sys_http_delivery row, while it still reads active:true #8022 left; a signature-refusal rule in redeliver() restores the [security] The webhook signing secret is stored in cleartext in sys_webhook.definition_json #7799 signing invariant end-to-end.
• Measured business pull: drops are say-once console events today — an operator who missed the log has no record; no customer incident measured yet.
• AI-agent error-resistance: an explicit refusal state beats an operator button that silently sends unsigned — the current shape invites exactly the wrong "fix".
• Startup scope discipline: migration-shaped platform-object change = permanent; the minimal cut (refuse redeliver of rows whose webhook config is gone; record drops as dead with reason) may be enough without a new state.

Recommendation: minimal cut first — redeliver() refuses when the signing config is unavailable (fail-closed), drops recorded as dead rows with a reason column reuse if possible; a full new lifecycle state only if the reason column can't carry it. Maintainer's call on the storage shape.

---

Generated by Claude Code