docs(audit): 2026-08-14 incident forensics and live index restore evidence (#316, #231)

docs/audit/live-drift-forensics-2026-08.md

@@ -33,14 +33,38 @@ pinned issue (provider-backed — not run from the authoring session), and add
 
 ## Phase 1 — Read-only forensics
 
-_Not yet run. Requires an approved read-only production window._
+_Partially run 2026-08-14 in an owner-authorized incident window; 1.2 and the audit-history
+pairing remain pending._
 
 ### 1.1 Migration-history fingerprint
 
-_Pending._ Record every `statements IS NULL` version with its name, and state explicitly whether
-`20260705180000` carries that signal — then pair it with audit history to distinguish a
-mark-applied/repair history from indexes that were created and later dropped. Do not close `#248`
-on the fingerprint alone.
+_2026-08-14 (owner-authorized Supabase connector session, incident-driven partial run)._
+Full `schema_migrations` fingerprint captured. Decisive rows:
+
+- `20260705180000 reconcile_search_health_indexes` — `no_statements = false`, **stmt_count 14**.
+ It does **not** carry the mark-applied signal: its DDL was recorded as executed.
+- `20260804110240 restore_rag_search_health_indexes` (the guard) — applied with its statement on
+ 2026-08-04. Note (per PR #1960 review): that guard validates four **other** indexes and never
+ checks this pair, so its application gives **no** existence bound for
+ `documents_title_trgm_idx` / `document_chunks_content_trgm_idx`.
+- Rows with the mark-applied signal (`statements IS NULL` or empty): the 2026-07-01…07-02 cluster
+ (`fix_chunks_hybrid_perf_and_ambiguity`, `fix_remaining_hybrid_perf_and_ambiguity`,
+ `schema_health_hybrid_execution_smoke`, `drop_dead_drifted_hybrid_variants`,
+ `clinical_query_term_trgm_correction`, `commit_generation_preserve_legacy_artifacts`,
+ `add_claim_ingestion_jobs_comment`, `drop_redundant_indexes`, `rag_retrieval_logs_retention`,
+ `storage_cleanup_jobs_document_fk`, `fix_reset_document_index_duplicate`,
+ `documents_owner_covering_index`, `fix_invoke_agent_url_to_guc`,
+ `promote_index_generation_id_columns`) and the 2026-07-12 reconciliation batch
+ (`reconcile_ingestion_index_shapes` … `add_legacy_index_health_batch_repair`, stmt_count 0).
+
+**Conclusion for the two retrieval-critical indexes:** their creation was recorded as executed on
+2026-07-05 (`20260705180000`, 14 statements), and both were reported missing by the live-drift
+runs of 2026-08-02 (Actions 30763871562) and 2026-08-09 (31330856982), with the weekly check red
+since 2026-07-26 — so the drop happened **between 2026-07-05 and 2026-08-02** (likely by
+2026-07-26). No app/worker/edge-function code issues `DROP INDEX` (repo grep, this session), so a
+manual/dashboard action — e.g. an accepted "unused index" advisor suggestion — is the leading
+**inference, not an established attribution**; pairing with the dashboard audit/query history for
+that window remains **pending** (owner action). `#248` stays open.
 
 ### 1.2 RPC divergence dossier
 
@@ -50,10 +74,16 @@ Protected RAG surface: an ambiguous diff is recorded as UNCLASSIFIED and escalat
 
 ### 1.3 Index inventory, sizing, and EXPLAIN baselines
 
-_Pending._ Owning-table `pg_relation_size` for the 21 missing and 2 unexpected indexes, plus
-`EXPLAIN (ANALYZE, BUFFERS)` baselines for the `documents` title ILIKE query, the `document_chunks`
-content search, and the `rag_retrieval_logs` miss scan. These are the before-measurements for
-Phases 4 and 5.
+_2026-08-14 (partial — retrieval-critical scope only)._ Live inventory of the ten
+`20260705180000` indexes: **exactly two missing** — `documents_title_trgm_idx` and
+`document_chunks_content_trgm_idx`; the other eight present (labels/summaries trgm, table-facts,
+index-units, pages, sections, both `rag_retrieval_logs` indexes). Owning tables at repair time:
+`document_chunks` 1562 MB / 70,120 live rows; `documents` 18 MB / 3,301 rows.
+
+Before-measurements came from live production probes rather than raw EXPLAIN (the incident was
+end-to-end visible): `/api/search` semantic query 2026-08-14 → total 37.7 s,
+`supabase_rpc_latency_ms` **31,610**; a second semantic probe 29.9 s / 21,757. The remaining
+missing-index sizing and the `rag_retrieval_logs` miss-scan baseline are **pending**.
 
 ## Phase 2 — Staging parity rehearsal
 
@@ -73,17 +103,49 @@ behaviour-changing deploy.
 
 ## Phase 4 — Index restoration
 
-_Not yet run. Requires an approved off-peak production window._
-
-_Pending._ PITR restore point, per-index `CREATE INDEX CONCURRENTLY` result with its
-`indisvalid`/`indisready` verification, disposition of the 2 unexpected live indexes with reasons,
-the guard migrations landed, and the green live-drift dispatch output.
+_2026-08-14 (partial, incident-driven: the two retrieval-critical indexes only, owner-approved
+"i authorise" in-session)._ Executed via the owner-authorized Supabase connector:
+
+- `create index concurrently if not exists documents_title_trgm_idx …` — definition verbatim from
+ `20260705180000`. Result: `indisvalid = true`, `indisready = true`, 648 kB.
+- `create index concurrently if not exists document_chunks_content_trgm_idx …` — same source.
+ Result: `indisvalid = true`, `indisready = true`, 68 MB.
+- `ANALYZE public.documents; ANALYZE public.document_chunks;` after both builds.
+- Canonical-shape validation (per PR #1960 review — `IF NOT EXISTS` could otherwise no-op on a
+ same-named index; here the prior inventory proved both absent, and post-build `pg_indexes`
+ returns the canonical normalized definitions verbatim):
+ `CREATE INDEX document_chunks_content_trgm_idx ON public.document_chunks USING gin (lower(((COALESCE(section_heading, ''::text) || ' '::text) || COALESCE(content, ''::text))) gin_trgm_ops)` and
+ `CREATE INDEX documents_title_trgm_idx ON public.documents USING gin (lower(((COALESCE(title, ''::text) || ' '::text) || COALESCE(file_name, ''::text))) gin_trgm_ops)` —
+ both matching `20260705180000` / `schema.sql`.
+
+Deviation from the phase template, recorded honestly: no PITR restore point was captured first —
+the operation was additive index creation with a one-statement rollback
+(`drop index concurrently`), no data-loss surface. No migration was added in the incident window:
+the definitions are already codified in `20260705180000` + `schema.sql`, and this was the
+documented operator prebuild for a drifted hosted target. **Outstanding phase debt (PR #1960
+review):** plan phase 4.4 still requires a fail-fast reconcile/guard migration for this repaired
+pair (the `20260804110240` pattern names four other indexes only), so a later replay cannot
+silently proceed if either index disappears again — queued as follow-up work for the full Phase 4
+batch, deliberately not bundled into this docs-only PR because migrations are an operational-risk
+surface with their own replay gates. The other 19 drift findings, the 2 unexpected live indexes,
+and the green live-drift dispatch also remain **pending** for the full phase.
 
 ## Phase 5 — Measure and close the loop
 
-_Not yet run. Requires a read-only production window (plus eval approval only if Phase 3 changed
-behaviour)._
+_Partially run 2026-08-14 (incident scope); full close-out still requires the remaining phases._
+
+_2026-08-14 (partial)._ Before/after production probes (identical endpoint and query style):
+
+| Measurement | Before | After restore + ANALYZE |
+| ------------------------------------------------------------------- | -------------------- | ----------------------- |
+| Semantic query, text fast path — total / `supabase_rpc_latency_ms` | 37.7 s / 31,610 | 4.8 s / **1,535** |
+| Semantic query, hybrid strategy — total / `supabase_rpc_latency_ms` | 29.9 s / 21,757 | 17.2 s / 8,519 |
+| `match_document_chunks_text_v2` single call | (dominated the 31 s) | 14 ms |
 
-_Pending._ Before/after `EXPLAIN` table against the Phase 1.3 baselines showing plan flips and
-timings, the evidence-backed verdict on ledger `#231`'s 25 s fast-route budget, and the
-`check:production-readiness` output.
+**#231 verdict from this evidence:** the 25 s fast-route budget was being consumed by retrieval
+itself while the two trigram indexes were missing — pre-generation latency was the binding cause
+of semantic-query source-only fallbacks in this window (README §A1 ladder rung 2, now measured).
+The A1/S1 packet must re-verify `generation_quality_gate:*` dominance on healthy latency before
+choosing any code mitigation. Residual: hybrid fan-out still costs ~8.5 s worst-observed — owned
+by the remaining remediation phases, not a route-budget change (`#231`'s stop condition stands).
+`check:production-readiness` on the final state is **pending**.

docs/branch-review-records/1eb170403cacbc03de02adf5632f19f46b5c09228a76c7dcb950557c88123401.record.md

@@ -0,0 +1 @@
+| 2026-08-14 | claude/live-drift-routing-lnhvja | 61b53680ad49196a392dea91e1a4a7345d88c522 | live-drift workflow failure routing + post-migration trigger (#316 phase 0) | PR #1939 open | check:github-actions pass; verify:pr-local failed:(none); test:ci-workflows pass |

docs/branch-review-records/71e026951e34c888a933c97c6f0f48f3825198aad8e34f32ecc8d6381b2ce5a2.record.md

@@ -0,0 +1 @@
+| 2026-08-14 | claude/live-drift-forensics-lnhvja | b9485d897dbec528b9141d3b93bbc7a067bfd010 | incident forensics + live index restore evidence (#316/#231) | PR #1960 open | verify:pr-local failed:(none) incl check:ledger-write-discipline |

docs/outstanding-issues-inbox/19fb70c0-1fd4-43a8-a4f4-efa9c31d16b1.json

@@ -0,0 +1,11 @@
+{
+ "version": 1,
+ "id": "19fb70c0-1fd4-43a8-a4f4-efa9c31d16b1",
+ "createdOn": "2026-08-14",
+ "action": "update",
+ "payload": {
+ "id": "#231",
+ "detail": "2026-08-14: rung-2 evidence measured live - supabase_rpc_latency_ms 31610 on a semantic query (route budget 25000 starved generation), caused by the #316 dropped trigram indexes; after restore 1535 (text) / 8519 (hybrid). Pre-generation latency was the binding residual cause for semantic-query fallbacks; S1 (A1 phase 2) should re-verify generation-quality-gate dominance on healthy latency before choosing a code mitigation rung.",
+ "source": "session 2026-08-14 live incident probes"
+ }
+}

docs/outstanding-issues-inbox/606b1573-6217-457a-9183-7f8e550d3094.json

@@ -0,0 +1,10 @@
+{
+ "version": 1,
+ "id": "606b1573-6217-457a-9183-7f8e550d3094",
+ "createdOn": "2026-08-14",
+ "action": "cancel",
+ "payload": {
+ "requestId": "7f5e6922-8d0b-424a-b359-12bed3a4e315",
+ "reason": "superseded: merged verbatim into the combined #231 update queued in this same PR alongside the 2026-08-14 incident rung-2 evidence"
+ }
+}

docs/outstanding-issues-inbox/81845ded-27a8-43d8-9a56-d85a5515935b.json

@@ -0,0 +1,11 @@
+{
+ "version": 1,
+ "id": "81845ded-27a8-43d8-9a56-d85a5515935b",
+ "createdOn": "2026-08-14",
+ "action": "update",
+ "payload": {
+ "id": "#231",
+ "detail": "PARTIAL 2026-08-12: This PR fixes the clinically consequential stale-fallback path: every answer whose routing or degraded reason contains generation_fallback is excluded from rag_response_cache. Offline evidence: 96 focused answer-route tests and 574 RAG fixture/contract tests passed. Approved live baseline/final canaries preserved 36/36 document and content recall at 1.0 with zero per-case reciprocal-rank regressions; the final 44-case answer gate had zero citation or numeric-grounding failures. A budget extension was tested and rejected: four cache-bypassed 'Lithium dosing?' probes remained grounded, cited safe extractive fallbacks at 35-40 second candidate budgets; the decisive 40-second probe completed generation in 25.272 seconds and 27.237 seconds total with route_deadline_exceeded=false, but failed generation quality. Therefore OPENAI_ANSWER_TIMEOUT_MS and the route budget are not the current residual binding cause. INSTRUMENT NOW EXISTS 2026-08-14: the \"Next: instrument\" half of this row is done. Commit a3bc4da adds scripts/probe-generation-quality.ts — one cache-bypassed live answer reporting the structured generation_quality_gate_reasons, provider-backed, refusing demo mode, never caching or logging the probe. The same commit adjudicates PR #1861: superseded for phase 1, close recommended, with the numeric-retry half deferred to phase 2 pending probe evidence. So do not review #1861 as though it were the live fix, and do not re-implement the probe. Next: run scripts/probe-generation-quality.ts in an environment that has OPENAI and Supabase credentials — it is blocked in offline containers, which is why it has not been run yet — then make a separate bounded output-quality fix with an offline fixture and live canary. Stop: do not increase route/provider timeouts or cache any generation fallback. INCIDENT ADDENDUM 2026-08-14 (later the same day): rung-2 evidence was then measured live - supabase_rpc_latency_ms 31610 on a semantic query (route budget 25000 starved generation), caused by the #316 dropped trigram indexes; after their owner-approved restore, 1535 (text fast path) / 8519 (hybrid). Pre-generation latency was the binding residual cause of semantic-query source-only fallbacks in that window; evidence in docs/audit/live-drift-forensics-2026-08.md. S1 (A1 phase 2) must re-verify generation_quality_gate:* dominance on healthy latency (run the probe with node --env-file=.env.local, which the probe does not load itself) before choosing a code mitigation rung. The route-budget stop condition stands unchanged.",
+ "source": "sessions 2026-08-14: instrument adjudication + live incident probes (owner-authorized Supabase connector)"
+ }
+}

docs/outstanding-issues-inbox/bbf21714-0ef9-4c2f-942f-1b8d7e328ac8.json

@@ -0,0 +1,11 @@
+{
+ "version": 1,
+ "id": "bbf21714-0ef9-4c2f-942f-1b8d7e328ac8",
+ "createdOn": "2026-08-14",
+ "action": "update",
+ "payload": {
+ "id": "#316",
+ "detail": "2026-08-14 incident session: two of the 21 missing indexes (documents_title_trgm_idx, document_chunks_content_trgm_idx) were the retrieval-critical pair; restored live via owner-approved CREATE INDEX CONCURRENTLY + ANALYZE. Before/after supabase_rpc_latency_ms 31610 -> 1535 (text) / 8519 (hybrid). Forensics: indexes existed 2026-08-04 (guard 20260804110240 passed) and were dropped by 2026-08-09; no app code drops indexes - owner to check dashboard audit for manual/advisor DROP INDEX. Evidence: docs/audit/live-drift-forensics-2026-08.md. Remaining drift findings + 10 diverged match_* RPC bodies stay with docs/database-remediation-plan.md. Drift routing: PR #1939.",
+ "source": "session 2026-08-14 live incident (owner-authorized Supabase connector)"
+ }
+}

docs/outstanding-issues-inbox/d52dbc3d-6759-4f6e-9089-24059830ed5a.json

@@ -0,0 +1,10 @@
+{
+ "version": 1,
+ "id": "d52dbc3d-6759-4f6e-9089-24059830ed5a",
+ "createdOn": "2026-08-14",
+ "action": "cancel",
+ "payload": {
+ "requestId": "19fb70c0-1fd4-43a8-a4f4-efa9c31d16b1",
+ "reason": "superseded: merged into the combined #231 update queued in this same PR"
+ }
+}