docs(rfc): hooks framework design

doc/rfc/hook-framework.md

@@ -0,0 +1,123 @@
+# Hooks Framework
+
+Fire-and-forget side effects for pipeline lifecycle events: one shared event contract, a durable hook topic per domain, pluggable hooks.
+
+## Problem
+
+The pipelines emit lifecycle transitions — a request lands or fails, a batch merges, a build finishes — but nothing can react outside pipeline state: no warehouse export, no PR comments or closes on merge events, no notifications or audit trails. The log topic is not this seam: SubmitQueue request statuses only, consumed solely to build gateway read models.
+
+Two requirements: side effects must never stall or fail the pipeline, and "fire and forget" must not mean lossy — a merge-failure comment that silently never posts is a support ticket.
+
+## Proposal
+
+When a controller performs a transition, it also publishes a **hook event** to a durable `hook` topic. A thin per-domain dispatcher stage consumes it and hands each event to the **hooks** the host wired — no-op by default, real integrations as they arrive.
+
+```
+pipeline controller dispatcher stage (per domain)
+state write → hook publish → downstream ──▶ [hook topic] ──▶ decode → validate → hook.Handle
+ │ ├─ noop (default)
+ │ retries exhausted └─ composite ─▶ warehouse, code host, …
+ ▼
+ [hook_dlq] ──▶ log full event + page; manual republish
+```
+
+One event shape for every domain — the CloudEvents core plus a version ordinal:
+
+| Field | What it is | Why it is on the envelope |
+|---|---|---|
+| `id` | Opaque occurrence identity, publisher-minted (`source/type/subject-id/version`) | Queue dedupe key and hook idempotency key |
+| `source` | Producing domain (`submitqueue`) | Keeps multi-domain sinks unambiguous |
+| `type` | What happened, one dotted open string (`batch.failed`) | The single filter dimension |
+| `timestamp_ms` | Occurrence time, ms since epoch | Uniform time axis |
+| `version` | Subject's ordinal at the transition; 0 = not a versioned write | Staleness detection under at-least-once — wall clocks can't |
+| `payload` | Per-type facts, always including the subject's id | Everything else; open, additive vocabulary |
+
+Delivery promise:
+
+- At least once, deduped by `id` within the queue's retention window; hooks are idempotent by `id`.
+- The hook publish rides the delivery that causes the transition — state write, hook publish, downstream publishes, ack — so a crash replays everything. No outbox.
+- A failed publish fails the stage (the hook topic shares the pipeline's queue backend); loss is never silent.
+- Past the retry budget, the DLQ logs the complete event and pages; manual republish recovers it.
+- Ordering per subject only. Hook outcomes never write pipeline state.
+
+## Decisions
+
+### Identity and replay
+
+- `id` = source / type / subject id / post-transition version; the causal message id stands in when unversioned, plus an ordinal for multiple same-typed events per cause. Components are separator-free. Consumers never parse it.
+- Partition key = subject id; per-subject order only.
+- Replay finding the target state already written → republish (idempotent). Beyond it → superseded; the event may be lost. Named, accepted gap.
+- RPC-caused transitions are at-most-once; needing the guarantee means publishing from the first queue-driven stage.
+- Opt-in per host via topic-key registration; a registered host never skips, so off and loss are distinguishable.
+
+### Contract
+
+- `api/base/hook/`: no owning domain, so the message-queue location rule extends — platform-owned contracts live under `api/base/`.
+- Envelope = only fields every consumer keys on uniformly; subject, queue, and error are occurrence facts → payload. `source`/`type` are strings, not enums, for additive evolution.
+- Payload (`Struct`): shaped per type, add-only, documented by its domain; must carry the subject's id and transient facts (merge step outcomes, build failure detail) — the event is their only durable record. Never entity snapshots; hooks resolve entities from stores.
+
+### Hooks and dispatch
+
+- Extension at `platform/extension/hook/`, singleton shape (counter precedent), wired once per host; no per-queue factory.
+- Hook contract: at-least-once, idempotent by `id`, plain errors, never writes pipeline state; ignore an event by returning nil (no filter API).
+- Ships `noop` (default) and `composite` (runs all children, joins failures, names failing children). A cross-domain sink is the same impl wired into each domain.
+- Dispatcher: decode, validate (`id`/`source`/`type` non-empty), invoke. Malformed events dead-letter, never silently acked; hook errors retry then dead-letter, with errs classifiers fast-pathing permanent failures.
+- DLQ reconciler: log the full event with its failure attribution, page (new metric — the log DLQ only warns), then ack. Manual republish recovers; pipeline state is never touched.
+- Per-hook retry isolation later: consumer groups on the same `hook` topic key, once the registry supports multiple groups per key and rejection becomes group-local (today it moves the shared row). Until then the composite's shared budget is accepted.
+
+## Example
+
+```proto
+syntax = "proto3";
+
+package uber.base.hook;
+
+import "google/protobuf/struct.proto";
+
+import "api/base/messagequeue/proto/messagequeue.proto";
+
+// HookEvent is one fire-and-forget lifecycle event. Every domain publishes
+// this same shape to its own hook topic; hook implementations consume it.
+message HookEvent {
+ option (uber.base.messagequeue.topic_keys) = "hook";
+
+ string id = 1; // Opaque occurrence identity; queue message id and hook idempotency key.
+ string source = 2; // Producing domain: "submitqueue", "stovepipe", ...
+ string type = 3; // What happened: "request.landed", "batch.failed", ...
+ int64 timestamp_ms = 4; // Occurrence time, ms since the Unix epoch.
+ int32 version = 5; // Subject's version at the transition; 0 when not tied to a state write.
+ google.protobuf.Struct payload = 6; // Publisher-defined facts, including the subject's id; never a snapshot.
+}
+```
+
+A failed batch, carrying merge-result facts persisted nowhere else (protojson: int64 as string, empty fields omitted):
+
+```json
+{
+ "id": "submitqueue/batch.failed/batch-778/4",
+ "source": "submitqueue",
+ "type": "batch.failed",
+ "timestamp_ms": "1722800012345",
+ "version": 4,
+ "payload": {
+ "batch_id": "batch-778",
+ "queue": "go-monorepo",
+ "error": "merge conflict",
+ "failed_step": "sq-12346",
+ "conflict_paths": ["foo/bar.go"]
+ }
+}
+```
+
+## Rejected
+
+- **A contract per domain.** N schemas, N hook shapes, N warehouse tables; one envelope absorbs differences additively.
+- **Inline hook calls.** Couples pipeline latency to integrations; a crash between write and call silently drops the notification.
+- **A second consumer group on the log topic.** Request statuses only; no path to batch, build, merge, or other domains.
+- **Enums for source/type.** protojson rejects unknown enum values; every addition would break consumers.
+- **Subject, queue, or error on the envelope.** Occurrence facts; they live in the payload. No major event platform carries a top-level error.
+- **Entity snapshots as payload.** Stale on redelivery; competes with the store; drags domain schemas into the shared contract.
+- **Typed per-event payloads (`oneof`).** Every new type becomes a wire-contract change.
+- **A filter/subscription API.** Returning nil costs nothing; routing can be a wiring decorator later.
+- **Best-effort publishing.** Silent loss; failing the stage is safe because dedupe makes the retry idempotent.
+- **A transactional outbox.** The publish rides the triggering delivery before ack; a crash replays both. RPC-edge transitions are scoped out instead.

doc/rfc/index.md

@@ -9,6 +9,7 @@ Design documents and technical proposals, grouped by scope. Shared/cross-cutting
 - [Consumer Gate](consumer-gate.md) - Stopping and starting individual queue controllers at runtime via consumer middleware: parked deliveries held in-flight with visibility extension, gate state as a separate extension with a file-based first implementation shared by tests and operators
 - [Consumer Hold](consumer-hold.md) - Fourth delivery outcome letting a controller postpone its delivery: the message becomes a partition barrier that pauses consumption for a chosen delay, redelivers in order, and does not count as a failure toward dead-lettering
 - [Change URIs](change-uri.md) - Identity of a code change: `scheme://{host[:port]}/{path}` per provider (GitHub PR, Phabricator Diff, git ref/commit) and canonical-form rules
+- [Hooks Framework](hook-framework.md) - Fire-and-forget side effects off pipeline lifecycle events: one shared `HookEvent` contract (`api/base/hook/`) published to a durable per-domain hook topic, dispatched by a per-domain stage to a pluggable hook extension (`platform/extension/hook/`) for integrations like warehouse export and code-review notifications
 
 ## SubmitQueue