Regression from #7799: for ~60s after every restart, a webhook holding an encrypted signing secret silently drops its subscription — no delivery, no `sys_http_delivery` row, while it still reads `active:true`

[baozhoutao]

Found by a retest of integration-system.webhook-lifecycle (QA run #7690) against origin/main @ b602d536, rebuilt. The fix under retest — #7722/#7798 (delivery rows store the signature, not the secret) and #7799/#7901 (encrypt sys_webhook.signing_secret) — is otherwise verified correct. This card is about a new failure the second half introduced.

Impact

For roughly 60 seconds after every server start, a webhook that holds an encrypted signing secret is not subscribed. A record change in that window produces:

• no delivery to the receiver, and
• no sys_http_delivery row at all — so there is no dead-letter, no retry, no durable trace that anything was missed,

while GET /api/v1/data/sys_webhook/<id> still reads active: true, i.e. it looks armed in Setup. The event is simply gone. It self-heals at the next periodic cache refresh (60 s), so it is invisible to anyone not looking in that window.

The affected population is exactly the one #7799 created: webhooks with an encrypted signing_secret. Before #7799 no secret needed decrypting at cache-build time, so this window did not exist.

Reproduction (2/2 — 11:29 and 11:33)

1. Give an active webhook a signing secret:
   PATCH /api/v1/data/sys_webhook/<id> {"active":true,"url":"<receiver>","signing_secret":"<literal>"}
   The engine stores secret:sec_<id> and encrypts into sys_secret.
2. Restart the server on the same DB.
3. Within ~60 s of boot, mutate a record the webhook subscribes to.

Expected: one receiver hit and one sys_http_delivery row.
Actual: no receiver hit, and no sys_http_delivery row.

After the 60 s cache refresh, the same mutation delivers correctly and its signature verifies.

Root cause — boot ordering, not the fail-closed choice

From the server log of the 11:33 run:

11:33:35.145 WARN [webhook-auto-enqueuer] webhook 'showcase_task_changed' holds an encrypted
 signing secret that could not be decrypted — the subscription is DROPPED rather
 than delivered unsigned (#7799)
 err: "Cannot resolve secret: no CryptoProvider is registered (fail-closed)."
11:33:36.4xx INFO CryptoProvider configured for secret fields

The webhook-auto-enqueuer builds its subscription cache ~1.3–2 s before the CryptoProvider is registered. It cannot decrypt, and — correctly — refuses to deliver unsigned, so it drops the subscription. Nothing re-arms it until the periodic 60 s refresh (cache refreshed {objects:1,rows:1} observed at 11:29:45).

The fail-closed decision is right; the ordering is the bug. Dropping unsigned deliveries is the correct choice — the problem is that the enqueuer's first cache build races the crypto registration, and that the drop leaves no durable record.

Suggested shape (not prescriptive)

Either make the first cache build wait on CryptoProvider registration (or rebuild immediately on the registration event) instead of leaving a 60 s hole, or — if a drop is ever legitimate — write a sys_http_delivery row in a failed/deferred state so the miss is durable and retryable rather than silent. The silence is arguably the worse half: a fail-closed drop that leaves a trace is an operational event; one that leaves nothing is data loss.

What is verified working (so this card is narrowly scoped)

The retest confirms the secret fix itself is sound, and none of this should be reverted:

• The secret is genuinely gone from the delivery table. sys_http_delivery has no secret column; it now carries signature instead. A byte-level full-table/full-column scan of both live SQLite files (data.db: 88 tables / 1395 columns / 4227 cells; data.telemetry.db: 9 tables / 186 columns / 3689 cells) plus a raw grep -a -F over data.db, -wal and -shm found 0 occurrences of the secret literal — against a positive control (an unrelated literal returned 24 hits, the sec_ handle 2 hits) proving the scanner reads live WAL data.
• Signing still works everywhere it must: recomputed HMAC-SHA256(raw body, secret) matched x-objectstack-signature on all three per-record variants, on a 4th-attempt retry after receiver downtime, on a delivery after a full restart where the secret exists only as ciphertext, and on a redeliver replay.
• Steady-state delivery, the retry ladder (pending → attempts 4 → success), trigger-enum enforcement and the customized-row-survives-reseed behaviour all pass.

Source

Retest of integration-system.webhook-lifecycle at b602d536. The two fixes under retest do not interfere with each other; this defect belongs entirely to the #7799 half.

[huangyiirene]

Claim: PM loop round 8 — regression repair, dispatched immediately without waiting for a grading cycle
Session: session_01JVfWSQN9RDDGwdVSmMc29x
Branch: claude/issue-8022-webhook-cache-crypto-ordering
Worktree: cloud session's own clone (mode:cloud)
Domain: domain:services
File surface: packages/plugins/plugin-webhooks/src/auto-enqueuer.ts (cache build + the #7799 drop arm) and the plugin's boot/registration wiring. sys_http_delivery / sql-http-outbox.ts are in scope only if the chosen shape writes a durable row for the drop.
Container & model: M (small surface, but the shape is a boot-ordering decision with a durability half), mode:cloud, model: opus

Why this seat dispatched it before triage graded it — stated so it is not read as self-grading

⛔ This is not a finding promoted by this seat, and it is not a domain:* write — grading and routing remain the triage seat's single channel, and neither was touched. The card arrived already bug + domain:services; the only label this seat added is the pm:dispatched half of its own claim.

The basis for acting now rather than waiting:

1. This lane caused it. [security] The webhook signing secret is stored in cleartext in sys_webhook.definition_json #7799 → PR fix(webhooks): stop storing the subscriber signing secret in cleartext (#7799) #7901 is this seat's round-4 card, reviewed and accepted by this seat. The affected population is, in the filer's words, "exactly the one [security] The webhook signing secret is stored in cleartext in sys_webhook.definition_json #7799 created".
2. It is silent data loss, live in main. Not a wrong answer — an event that vanishes: no delivery, no sys_http_delivery row at all, so no dead-letter, no retry, no trace, while active: true still reads armed in Setup. It self-heals at the 60 s refresh, so nobody outside that window ever sees it.
3. The skill's escalation bar puts restore-invariant repairs in the act-immediately class explicitly: "queue it, dispatch it, report it. Asking 'may I restore the invariant?' is the anti-pattern."

If the triage seat judges this seat overstepped, say so and I will hand the card back — but I am not leaving a silent-drop regression from my own merge sitting for a grading cycle.

Premise (measured by the filer, treated as given — 2/2 reproduced)

The webhook-auto-enqueuer builds its subscription cache ~1.3–2 s before the CryptoProvider is registered:

11:33:35.145 WARN [webhook-auto-enqueuer] ... the subscription is DROPPED rather than
 delivered unsigned (#7799) err: "Cannot resolve secret: no CryptoProvider
 is registered (fail-closed)."
11:33:36.4xx INFO CryptoProvider configured for secret fields

The fail-closed choice is correct and stays. ⛔ The dev does not get to "fix" this by delivering unsigned or by falling back to the cleartext path — that would undo #7799 and re-open the hole it closed. The bug is the ordering, plus the silence.

Two halves, and the second is arguably the worse one

• Ordering — the first cache build must not race crypto registration: wait on it, or rebuild on the registration event. Closing this removes the 60 s hole.
• Durability — a fail-closed drop that leaves no trace is data loss; one that leaves a sys_http_delivery row in a failed/deferred state is an operational event. ⚠️ Even with the ordering fixed, any future path that drops a subscription should be durable. The dev should say explicitly whether the shape it lands covers only the first half or both, and why.

Serial constraint — declared, because it is real

⚠️ #7986 is pm:queue and unclaimed in the same package (plugin-webhooks: webhook headers still cleartext in two blobs — the #7799 sibling PR #7901 did not close). It is deliberately NOT in this batch: same package, and its likely surface (bootstrap-declared-webhooks.ts, sys-webhook.object.ts) sits next to this card's. Serialized to a later round per the same-file/same-package rule. Recording it here so the next selection round does not read the two as independent.

Also verified as not overlapping: nothing else is in flight in this lane (round 7 closed, PR #7998 merged).

What must NOT regress

The retest that found this also verified the #7799 fix is sound — byte-level full-table scans of both live SQLite files plus raw grep -a -F over data.db/-wal/-shm found 0 occurrences of the secret literal, against a positive control proving the scanner reads live WAL data; HMAC verification passed on all per-record variants, on a 4th-attempt retry, after a full restart with the secret existing only as ciphertext, and on a redeliver replay. ⛔ None of that may be given back to fix this.

---

Generated by Claude Code

[huangyiirene]

PR #8043 (draft, Fixes #8022) · branch claude/issue-8022-webhook-cache-crypto-ordering

Your framing holds — and it is stronger than the log shows

The premise reproduces, and the root cause is where the log says. One correction, in your favour: it is not a ~1.3–2s race, it is a deterministic ordering. Plugins run inside kernel:ready, which runtime.start() completes (packages/core/src/kernel.ts Phase 3). Every host calls engine.setCryptoProvider(...) after await runtime.start() returns, unconditionally — packages/cli/src/commands/serve.ts:2752-2766, and packages/verify/src/harness.ts:561-563 has the same shape. So AutoEnqueuer's first cache build does not merely usually lose the race; on every host, every boot, it always precedes the capability. The two timestamps you captured are the size of the gap, not the reason there is one.

That matters for the fix: a shape that only reorders the CLI's composition root would leave apps/cloud, packages/verify and every embedder with the same window.

Half (a) — ordering. Closed.

ObjectQL.onCryptoProviderChange(listener) reports the registration, fired after the provider is in place. AutoEnqueuer subscribes before its first build and rebuilds the cache when it fires.

Why this one, over the two alternatives:

• Move the wiring earlier in serve.ts — fixes one host. The engine service is not resolvable before plugins init anyway, so it would end up as hook-ordering-by-registration-order: fragile, and silently wrong the first time someone reorders kernel.use(...).
• Short-backoff retry after a drop — shrinks the hole, never closes it, and turns a deterministic property into a timing one that a test can only assert by waiting.

The engine is the only party that knows the moment the provider arrives, so the notification belongs there. Feature-detected on the plugin side exactly as resolveSecretField already was — plugin-webhooks still takes no dependency on @objectstack/objectql, and an engine without the channel keeps today's behaviour with the periodic refresh as backstop.

One deliberate detail: the re-arm does not call refresh() directly. refresh() coalesces onto an in-flight build, and the build most likely running at that instant is the pre-registration one — joining it would report success having re-armed nothing. It waits for whatever is in flight, then reads again.

Half (b) — durability. Not closed. Needs its own card — please file it.

Not deferred for cost. There is a trap in it that I do not think should be resolved as a rider on this fix:

IHttpOutbox (packages/services/service-messaging/src/http-outbox.ts) has exactly one write door — enqueue(), which mints pending. A row parked for a missing key has no signature, because the signature is computed at enqueue from the secret we could not resolve. And redeliver() resets any terminal row (success/failed/dead) back to pending for a byte-for-byte replay — which is reachable by any authenticated user through POST /api/v1/webhooks/redeliver. So the naive version of (b) — write the drop as a dead row so an operator "can see and retry it" — hands that operator a button that delivers the webhook unsigned. That re-opens #7799 through a door nobody would look at.

Doing it properly needs a never-sendable state: a new column on the sys_http_delivery platform object (migration-visible), plus a redeliver() refusal for a row that requires a signature and has none. That is a design decision in service-messaging touching the exact redeliver path §5 verified working — it wants review on its own merits, not to be smuggled in behind an ordering fix.

On reachability after (a) — I am not claiming a drop is unreachable, and I want to be precise about what did change. Four causes remain: no provider ever registered, sys_secret row missing, decrypt failure under a rotated key, engine without resolveSecretField. But every one of them is now a persistent misconfiguration rather than a transient boot race — none self-heals, so none produces the "gone in a window nobody was watching, healthy forever after" signature that made this card. The residue is a real gap; it is a different-shaped one.

What I did close, cheaply and in-file: the drop is no longer quiet. It reports at error with the consequence and the fix in the message (AGENTS.md "Degradation log levels" — active: true reads armed while events vanish is exactly its judgment question answering yes), and carries an ADR-0112 code/status pair (INTERNAL_ERROR/500) in its metadata — the same pair the seeder's refusal for the same cause already carried. Said once per outage per webhook, not every 60s forever, per the same section; a webhook that recovers and breaks again is loud again. That is observability, not durability — it does not substitute for (b).

Reverse-verification (§6.3)

Same worktree, source files reverted to origin/main and rebuilt, test file unchanged:

	reading
origin/main	2 failed / 10 passed (12) — AssertionError: expected [] to have a length of 1 but got +0
after the fix	12 passed (12)

Full package suite after: 47/47. packages/objectql: 3356/3356 (189 files).

The window test runs with refreshIntervalMs: 0, so it cannot pass by waiting — the only thing that can re-arm the cache is the registration itself. It asserts on the durable record (the sys_http_delivery row and the receiver's recomputed HMAC), never on cache internals or call counts; the pre-registration half asserts outbox.list() is empty, which is the symptom you measured.

One test I wrote and then deleted, stated because it is the kind of thing worth knowing: a case pinning that the re-arm does not coalesce onto the in-flight pre-registration build. It passed on main too. Every harness that can inject the registration at that instant also moves the secret resolution to after it, so the naive () => this.refresh() satisfies it as well. A case that cannot separate the two revisions is a false green, so it is gone rather than kept as coverage it does not provide; the guard stays, with the reasoning where it lives. (Commit 55ba74d6.)

§5 — nothing given back

The fail-closed drop is untouched: no unsigned delivery, no cleartext fallback, no weakened refusal. Verified rather than asserted:

• The pre-existing at-rest suite (byte-scan of the stored sys_webhook row and the sys_secret store, the wire-form pin, the seeder/sweep/rotation cases) is green, unmodified.
• My own tests re-assert the three §5 properties on the fixed path: the re-armed delivery is signed with a key that exists only as ciphertext after a restart, createHmac('sha256', SECRET) over the raw body matches X-Objectstack-Signature byte for byte, the delivery row carries signature and JSON.stringify(rows) does not contain the secret.
• A third test pins the boundary directly: while the key stays unresolvable, calls is 0 and outbox.list() is 0 — it refuses, it does not downgrade.

The only behavioural change to the drop itself is its log level and metadata.

§5.5 ratchets

Both apply-checked, though neither trigger fired — I added no new test file (the cases live in the existing webhook-secret-at-rest.test.ts, which already imports @objectstack/objectql, @objectstack/spec and @objectstack/service-messaging) and no new workspace dependency:

• check:test-source-alias — OK, 72 packages scanned, 63 registered. No registry widening.
• check:type-check-debt — OK, 35 entries re-measured, 1975 raw errors, none above its recorded number. No entry raised. (It reports 9 pre-existing surplus entries that could be lowered — none in objectql or plugin-webhooks, all untouched by this diff.)
• check:engine-double-contract — OK, 166 pinned / 133 ledger / 2 exempt. I added no engine fake: the tests drive the real ObjectQL with a driver double, so the engine's own dispatch contract runs above it.
• Also green: check-durability-degradation-log-level (24 seams, all loud), check-error-code-casing (ADR-0112), check-changeset-fixed, check-empty-changeset, check-changeset-no-major, tsc --noEmit on plugin-webhooks, eslint on all four changed files.

Files, CI, and what I did not do

packages/objectql/src/engine.ts (+listener set, onCryptoProviderChange, fire in setCryptoProvider) · packages/plugins/plugin-webhooks/src/webhook-secret.ts (feature-detect seam) · .../auto-enqueuer.ts (subscribe-before-build, re-arm, reportDrop) · .../webhook-secret-at-rest.test.ts · one changeset. No packages/spec change, so §6.7's gen:schema/gen:docs does not apply. Nothing under content/docs/releases/**, docs/adr/**, .claude/skills/**, skills/**.

CI is running on the draft; I will drive it to green and report if anything reddens.

Deliberately not done: half (b), argued above — please file it. And I did not touch #7986 (headers still cleartext), per your serial constraint.

---

Generated by Claude Code

[huangyiirene]

CI state for the report above: #8043 is fully green — 25 checks, 23 success and 2 skipped (Console Pin Gate, Build Docs, both conditional and not reached by this diff). Test Core (3 shards), TypeScript Type Check, ESLint, Build Core, Dogfood Regression Gate (3 shards), Dogfood Verify CLI, Temporal Conformance (live PG + MySQL) and Check Changeset all passed. No follow-up patch round needed.

The one thing still outstanding from my report is half (b) — the durable sys_http_delivery record for a dropped subscription — which I recommend as its own card for the redeliver-replays-unsigned reason set out above.

---

Generated by Claude Code

[huangyiirene]

PM 复核:ACCEPT —— PR #8043 转 ready + 已挂 auto-merge

亲核 25 个 check:23 success + 2 skipped(Console Pin Gate、Build Docs,条件式、本 diff 未触及),含 ESLint ✅(13:32:56Z)与 TypeScript Type Check ✅(13:41:50Z)两个必查 job 的 conclusion。

证伪背书第十九笔 —— 把我的框定纠正得更强,而不是更弱

我的派发令(与卡片)都把它写成 ~1.3–2 s 的竞态。实测是确定性时序:插件跑在 kernel:ready 内、由 runtime.start() 完成(kernel.ts Phase 3),而每个 host 都在 await runtime.start() 返回之后才调 engine.setCryptoProvider(...)(serve.ts:2752-2766;verify/src/harness.ts:561-563 同形)⇒ 首次缓存构建不是通常输掉竞态,而是在每个 host、每次启动上必然先于能力到位。那两个时间戳量的是缺口大小,不是缺口成因。

这条纠正直接决定修法边界:只调 CLI 组合根的顺序,会把 apps/cloud、packages/verify 与每个嵌入方原样留在同一个窗口里。两个替代方案他都实测否掉并给了理由(提前接线只修一个 host,且引擎服务在插件 init 前不可解析,会退化成按注册顺序排 hook —— 有人重排 kernel.use(...) 时静默出错;短退避重试只缩小不关闭,还把确定性性质换成只能靠等待断言的时序性质)。⇒ 只有引擎知道 provider 到位那一刻,通知必须由引擎发出。

半边 (a) 关闭,且有一个我不会想到的细节

ObjectQL.onCryptoProviderChange + 订阅先于首次构建 + 注册时重建。re-arm 不直接调 refresh() —— refresh() 会合并到在飞构建上,而那一刻在飞的多半正是注册前那次,汇入它会「报告成功却什么都没重新武装」。改为等在飞那次结束后重新读一遍。这是那种测试不会抓、只有读懂并发语义才会避开的坑。

半边 (b) 不关闭 —— 理由比关闭它更有价值,已按其请求立卡 #8069

他没有以成本为由推迟,而是指出朴素做法主动不安全:IHttpOutbox 只有一扇写门 enqueue()(铸 pending);为缺失密钥停放的行没有签名(签名在 enqueue 时用那把解析不出来的密钥算);而 redeliver() 会把任何终态行重置回 pending 做逐字节重放,且经 POST /api/v1/webhooks/redeliver 对任何已认证用户可达。⇒ 把丢弃写成 dead 行,等于给操作员一个「无签名投递」的按钮,从一扇没人会去审计的门重新打开 #7799。 正解需要一个永不可发状态(sys_http_delivery 新列,迁移可见)+ redeliver() 对「需签名而无签名」的行拒收 —— 那是 service-messaging 的设计决定,且触及 #8022 源头 retest 验证过的 redeliver 路径,该独立评审。已立 #8069(先 create 拿真号再写引用,纪律⑰)。

他对残余可达性的表述同样准确、不夸大:(a) 之后仍有四种成因(从未注册 provider / sys_secret 行缺失 / 轮换密钥下解密失败 / 引擎无 resolveSecretField),但每一种都是持久错配而非瞬时开机竞态 —— 都不自愈,所以都不再产生「在没人看的窗口里消失、之后永远健康」那个使本卡紧急的签名。缺口是真的,形状不同了。

他就地便宜关掉的那一半:丢弃不再安静 —— error 级 + 消息里带后果与修法,元数据带 ADR-0112 的 code/status(INTERNAL_ERROR/500,与 seeder 对同一成因的拒绝同一对),且按 AGENTS.md「Degradation log levels」每次故障每个 webhook 只说一次而非每 60s 永远刷。他明说这是可观测性不是持久性,不替代 (b) —— 判读正确。

反向验证与「删掉一条假绿测试」

同一 worktree、源文件回退到 origin/main 重建、测试文件不变:main 上 2 failed / 10 passed(expected [] to have a length of 1 but got +0),修后 12/12;包内 47/47,objectql 3356/3356。窗口测试跑在 refreshIntervalMs: 0 上 ⇒ 不可能靠等待通过,唯一能重新武装缓存的就是注册本身;断言落在durable 记录(sys_http_delivery 行 + 接收方重算的 HMAC),⛔ 不碰缓存内部与调用计数。

⭐ 他写了一条 pin 又删掉,并主动说明:那条「re-arm 不会 coalesce 到注册前构建」的用例在 main 上也通过 —— 任何能在那一刻注入注册的 harness,也把密钥解析移到了注册之后。分不开两个版本的用例是假绿,删掉而不是留作它并不提供的覆盖率(commit 55ba74d6),理由留在代码旁。这是本班第一次有人主动删除自己写的绿测试并说清为什么,值得记进座位贴。

§5 未回退,且是被重新断言而非声称

既有 at-rest 套件(存储行字节扫描、sys_secret 存储、线格式 pin、seeder/sweep/rotation)绿且未改;他自己的测试在修复路径上重新断言三条性质 —— 重新武装后的投递用重启后仅以密文存在的密钥签名、createHmac('sha256', SECRET) 对原始 body 与 X-Objectstack-Signature 逐字节相符、投递行带 signature 且 JSON.stringify(rows) 不含密钥;第三条测试直接钉边界:密钥仍不可解析时 calls 为 0、outbox.list() 为 0 —— 拒绝,不降级。

⚠️ 跨席:第二笔 packages/objectql 突破,已声明,窗开

本 PR 在 packages/objectql/src/engine.ts 增 onCryptoProviderChange + listener 集合 + 在 setCryptoProvider 内触发 ⇒ domain:engine-core(#6019)的文件面。已按纪律①在合并前发三判据声明(#6019 评论 5268056497),并在其中一并重提第一笔(PR #7901 的 resolveSecretField,声明 5261853359,至今无回复)与本席对 maskSecretFields 描述不完整的公开更正。⛔ 若该席否决任一笔 ⇒ 走补丁轮,不辩护。

纪律㉓ 生效验证

本席这次在派发令里点名了两族 ratchet,dev 逐条给了读数:check:test-source-alias OK(72 扫描/63 注册,⛔ 无注册表放宽)、check:type-check-debt OK(35 条重测,无一高于记录值,⛔ 未抬高),另附 check:engine-double-contract OK 与 check-durability-degradation-log-level(24 seams 全 loud)。两族触发条件均未命中(未新增测试文件、未新增 workspace 依赖)—— 但读数照给,这正是㉓要的形状:点名之后不再靠运气。

---

Generated by Claude Code