diff --git a/docs/branch-review-records/aedb200d3e2ca20b411be7be48c1e744356ce8168ee76751786d1ec8eccb476d.record.md b/docs/branch-review-records/aedb200d3e2ca20b411be7be48c1e744356ce8168ee76751786d1ec8eccb476d.record.md new file mode 100644 index 0000000000..e2e4209db8 --- /dev/null +++ b/docs/branch-review-records/aedb200d3e2ca20b411be7be48c1e744356ce8168ee76751786d1ec8eccb476d.record.md @@ -0,0 +1 @@ +| 2026-08-18 | claude/db-remediation-board-phase3-2026-08-18 | d78029abf911dfb9e728f3c64a7dc166e2e6cea2 | docs/database-remediation-coordination.md board update after #2104/#2106 (#316) | coordinator self-review: docs-only, verified against main 511d22f4d and open PR #2111 | prettier --check pass; docs:check-links 1923 refs resolve | diff --git a/docs/database-remediation-coordination.md b/docs/database-remediation-coordination.md index 0e0b7347ff..4e012496e0 100644 --- a/docs/database-remediation-coordination.md +++ b/docs/database-remediation-coordination.md @@ -79,38 +79,45 @@ _Board re-baselined 2026-08-17 from `main` `f5b093291`; **updated 2026-08-18 fro forensics file, and the PR list — worker self-reports were checked against merged content, not trusted as-is._ -| Phase | Status | PR / evidence | Notes | -| ----------------------------------------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Prereq: docs on `main` | ☑ done | #1906 (plan/playbook), #2027 (traps), #2044 (this board, `e663203ce`) | — | -| Prereq: `SUPABASE_ACCESS_TOKEN` secret (`#183`) | ☐ pending — not blocking | repo-level secret names checked 2026-08-17: not present | Owner chose the Supabase MCP connector for all windows (2026-08-17). Token gates CLI repair paths only. | -| 0 — Enablement | ☑ COMPLETE 2026-08-14 | #1938, #1939, #1951 (proof run `31813064485`, auto-issue #1963), #1978 | Issue #1963 stays open until Phases 3–4 land; now that Phase 6.1 is on `main` repo-side, weekly live-drift will _additionally_ report `schema_drift_snapshot()` as repo-ahead plus an info line "migration-history probe not present" — expected, not a regression. | -| 1.1 — History fingerprint | ☑ done (owner action open) | #1960 §1.1 | `20260705180000` NOT mark-applied (14 stmts). Drop window 2026-07-05→08-02; Phase 2 narrowed it: staging **kept** the four indexes production lost, so an out-of-band drop on production is the leading explanation. Dashboard audit-history pairing = owner. `#248` open. | -| 1.2 — RPC divergence dossier | ☑ COMPLETE 2026-08-18 | **#2087** merged `173ea9f28` → forensics §1.2; `#316` row updated by reconcile #2098 (`d113396b0`) | Read-only connector window, 4 SELECTs, zero writes. **All 10 mismatches are attribute-only `SET work_mem`** present on live, absent from `schema.sql`; strip it → manifest hash 10/10. 4 mirror-stale (64 MB = `20260724000000`), 6 live-ahead attribute-only (128 MB ×4 hybrids; live-only 64 MB on `chunks_text_v2` and on `table_facts_text` after its `20260724120000` recreate). Zero body divergences, zero repo-ahead, zero UNCLASSIFIED. | -| 1.3 — Inventory / EXPLAIN | ◐ whole-schema done; sizing pending | #1970 §1.3 | 20 absent, 2 orphaned, zero invalid. Remaining index sizing + `rag_retrieval_logs` miss-scan baseline pending — fold into Phase 5. | -| 2 — Staging parity (`#056`) | ☑ COMPLETE 2026-08-18 — re-measure owed | **#2093** merged `e546840a1` → forensics §2.0–2.4; `#056` row updated by reconcile #2098 | Gap was **28**, not 26. Staging `ikoiolksxqxfxgiyqpnu` at 194/194 parity, zero `statements IS NULL`, all 28 rows md5-identical, corpus untouched, production never targeted; `check:drift` can now target staging (`scripts/check-drift.ts` fix). **`check:drift` vs staging RED, 19 items** — the migration chain does not reproduce `schema.sql`: (a) 7 `work_mem` fn mismatches (8th absent for exactly 1.2's recreate reason); (b) **8 objects `schema.sql` declares that no migration creates** (five `document_embedding_fields` indexes, `documents_status_idx`, `documents_updated_at` / `ingestion_jobs_updated_at` triggers); (c) 3 table column-set mismatches (`document_chunks`, `rag_visual_eval_cases`, `rag_visual_eval_runs`); (d) 1 index-def mismatch `document_chunks_content_trgm_idx` (manifest `8499c3d3…` vs staging `c3db2960…` — one of the two restored on production 2026-08-14; confirm which definition production actually carries). §2.4: staging KEPT the indexes production lost; four duplicate migration version pairs (`20260717161000`/`173000`/`20260719053532`/`053533` vs `20260719055541`/`055555`/`055609`/`055623`) — a plain `db push --include-all` applies older bodies over newer; `20260725000000` points staging's `app.ingestion_worker_base_url` at production — inert, **never seed `cron_ingestion_jwt` / `indexing_v3_agent_secret` / `ingestion_webhook_secret` into staging's vault**. **Owed:** re-measure at `main` ≥ 195 migrations (apply `20260818090000` to staging, re-run `check:drift`). | -| 3 — Reconciliation (reframed) | ☐ NEXT — repo-side, no canary | needs owner decisions D1/D2 below | Fable. Not a production deploy: (a) add `SET work_mem` to the 10 `schema.sql` definitions + one migration `ALTER FUNCTION … SET work_mem = ` per function, ordered after every recreate (idempotent on prod); (b) codify the 8 never-created objects; (c) triage the 3 column-set + 1 index-def staging mismatches; regenerate `drift-manifest.json` (Docker); `RAG impact: no retrieval behaviour change — codifying already-live attribute`; governance preflight. Applying the migration to production is a trivial window (state already matches); the staging apply proves it first. | -| 4 — Index restoration | ◐ PARTIAL (incident) | #1960 §Phase 4 | 2 trgm restored; **20** still missing, 2 unexpected undispositioned; guard migration for the repaired pair owed (4.4); Phase 6.3 named 8 monitor-candidates for `required_indexes` — decide in 4.4. Blocked on Phase 3 landing. | -| 5 — Measure | ◐ PARTIAL (incident) | #1960 §Phase 5 | Text fast path 31,610 → 1,535 ms; hybrid 8,519 ms residual. Full EXPLAIN set, `work_mem` before/after (only if D1 changes values), `check:production-readiness` pending. | -| 6 — Future-proofing | ◐ REPO-SIDE COMPLETE 2026-08-18 | **#2058** merged `9c660af1f` (14 paths verified on `main`); follow-up #2090 merged `79e5d2a68` | 6.1 migration `20260818090000_schema_drift_snapshot_history_probe.sql` built + mirrored + manifest regenerated, **NOT deployed** — needs a production migration window after Phase 4. 6.2 contract + `supabase/drift-allowlist.json` (5 seeded superseded guards; ~18 §1.1 versions deliberately NOT allowlisted → first post-deploy run reports them; fix = fail-fast guard migrations, never bare allowlisting). 6.3 `tests/search-health-index-coverage.test.ts` + unmonitored list (44 entries, 8 monitor-candidates). New P2 row queued (inbox `d6ce8a1d`). Stale branch `claude/database-drift-allowlist-48839e` (tip `3fe1deafe`) was recreated after merge — redundant, owner may delete. | -| 7 — Deferred | ☐ not started | — | Each its own session, per playbook. | - -**Owner decisions outstanding (2026-08-18):** - -- **D1 — `work_mem` values.** Keep 128 MB on `chunks_hybrid` / `embedding_fields_hybrid` / - `index_units_hybrid` / `index_units_hybrid_v2` and codify as-is (no hosted value change), or - standardise to 64 MB (hosted change; Phase 5 must measure before/after). Recommendation: - codify-as-live; provenance of the 128 MB / `_v2` settings pairs with the §1.1 dashboard - audit-history owner action. -- **D2 — canary exemption.** `SET work_mem` is planner memory: latency only; only exactly-tied rows - could reorder. Recommendation: codify-as-live without an eval-canary; any hosted value change is - verified by Phase 5 EXPLAIN, not a canary. The RAG-surface flag and `RAG impact:` line still apply. -- **D3 — 6.1 deploy window** timing (after Phase 4). - -**Ledger state note (2026-08-18, later):** reconcile PR #2098 (`d113396b0`) applied 16 requests -including `1591ee4a` (`#316` Phase 1.2), `d958d671` (`#056` Phase 2) and `d6ce8a1d` (Phase 6 row); -inbox is empty on `main`. `#183` rider: the Supabase CLI is now authenticated and the repo linked -to staging (owner, 2026-08-18) — queue that `#183` update on the next product PR (Phase 3), not a -ledger-only branch. Never merge `main` into a PR carrying a reconciliation; re-cut from `main`. +| Phase | Status | PR / evidence | Notes | +| ----------------------------------------------- | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Prereq: docs on `main` | ☑ done | #1906 (plan/playbook), #2027 (traps), #2044 (this board, `e663203ce`) | — | +| Prereq: `SUPABASE_ACCESS_TOKEN` secret (`#183`) | ☐ pending — not blocking | repo-level secret names checked 2026-08-17: not present | Owner chose the Supabase MCP connector for all windows (2026-08-17). Token gates CLI repair paths only. | +| 0 — Enablement | ☑ COMPLETE 2026-08-14 | #1938, #1939, #1951 (proof run `31813064485`, auto-issue #1963), #1978 | Issue #1963 stays open until Phases 3–4 land; now that Phase 6.1 is on `main` repo-side, weekly live-drift will _additionally_ report `schema_drift_snapshot()` as repo-ahead plus an info line "migration-history probe not present" — expected, not a regression. | +| 1.1 — History fingerprint | ☑ done (owner action open) | #1960 §1.1 | `20260705180000` NOT mark-applied (14 stmts). Drop window 2026-07-05→08-02; Phase 2 narrowed it: staging **kept** the four indexes production lost, so an out-of-band drop on production is the leading explanation. Dashboard audit-history pairing = owner. `#248` open. | +| 1.2 — RPC divergence dossier | ☑ COMPLETE 2026-08-18 | **#2087** merged `173ea9f28` → forensics §1.2; `#316` row updated by reconcile #2098 (`d113396b0`) | Read-only connector window, 4 SELECTs, zero writes. **All 10 mismatches are attribute-only `SET work_mem`** present on live, absent from `schema.sql`; strip it → manifest hash 10/10. 4 mirror-stale (64 MB = `20260724000000`), 6 live-ahead attribute-only (128 MB ×4 hybrids; live-only 64 MB on `chunks_text_v2` and on `table_facts_text` after its `20260724120000` recreate). Zero body divergences, zero repo-ahead, zero UNCLASSIFIED. | +| 1.3 — Inventory / EXPLAIN | ◐ whole-schema done; sizing pending | #1970 §1.3 | 20 absent, 2 orphaned, zero invalid. Remaining index sizing + `rag_retrieval_logs` miss-scan baseline pending — fold into Phase 5. | +| 2 — Staging parity (`#056`) | ☑ COMPLETE 2026-08-18 (re-measured) | **#2093** merged `e546840a1` (§2.0–2.4); **#2104** merged `f19cf8f60` (§2.5 re-measure); `#056` update `5ed2f873` pending reconcile | Gap was 28. Staging `ikoiolksxqxfxgiyqpnu` replayed to parity, corpus untouched, production never targeted. `check:drift` vs staging RED 19 at base `ed43a64f2`; **re-measured at current `main` after applying `20260818090000`: still exactly 19** (same keys, same hashes). Snapshot v2 `migration_history` block runs end-to-end on staging (probe `ok`, 0 rows). Findings: (a) 7 `work_mem` fn mismatches; (b) 8 objects `schema.sql` declares that no migration creates; (c) 3 column-set mismatches; (d) `document_chunks_content_trgm_idx` def mismatch. All four dispositioned by Phase 3. §2.4 traps: staging KEPT the indexes production lost; four duplicate migration version pairs; `20260725000000` points staging's worker GUC at production — **never seed `cron_ingestion_jwt` / `indexing_v3_agent_secret` / `ingestion_webhook_secret` into staging's vault**; **never run `check:drift --prune-stale` against staging** (the five seeded `migration_history` allowlist entries are production's and read stale there — warning only). | +| 3 — Reconciliation (reframed) | ☑ COMPLETE 2026-08-18 — repo-side + staging-proven; **production window owed** | **#2106** merged `72aa18865` (migrations `20260818110000` work_mem ×10, `111000` eight schema-only objects, `112000` chain-stale columns; forensics §3.1–3.6); **#2111** open (tip `8b33c4dbf`, BEHIND, merge-tree clean — staging proof + `20260818113000` forward-codify of three hybrid RPC bodies the chain never received; `#316` final + `#183` rider + 2 captures + 3 cancels queued) | D1 codify-as-live, D2 exemption granted. Manifest `def_hash` == live production hash 10/10 — the ten RPC "divergences" are closed as a mirror gap. §3.3: `token_estimate` was chain-stale (added by `112000`, no-op on prod); `rag_visual_eval_*` id defaults rebound to `pg_catalog.gen_random_uuid()` (no-op on prod); (d) production's restored `document_chunks_content_trgm_idx` (`coalesce(content,'')` form, `8499c3d3…`) IS canonical — staging carries the 2026-06-06 form; repair = Phase 4.4 guard migration + staging drop-and-recreate. Staging at 199 migrations, single drift residual = that index. **New finding:** `20260712000000`'s owner step never landed for three hybrid RPCs — chain-stale, forward-codified by `113000` (not a production hole). Tooling: `scripts/check-drift.ts:192` clips column diffs to 240 chars (never names the column) — queued P3; PR template lacks `RAG impact:` guidance — queued P3. | +| 4 — Index restoration | ◐ PARTIAL (incident) — **NEXT after the production window** | #1960 §Phase 4 | 2 trgm restored 2026-08-14; **20** still missing, 2 unexpected undispositioned; **4.4 guard migration for the restored trgm pair owed** (`20260804110240` pattern) plus staging drop-and-recreate of `document_chunks_content_trgm_idx` (2026-06-06 form → canonical); decide the 8 Phase 6.3 monitor-candidates via a `required_indexes` migration. Opus, off-peak prod window. | +| 5 — Measure | ◐ PARTIAL (incident) | #1960 §Phase 5 | Text fast path 31,610 → 1,535 ms; hybrid 8,519 ms residual. Full EXPLAIN set, `work_mem` before/after (only if D1 changes values), `check:production-readiness` pending. | +| 6 — Future-proofing | ◐ REPO-SIDE COMPLETE 2026-08-18 | **#2058** merged `9c660af1f` (14 paths verified on `main`); follow-up #2090 merged `79e5d2a68` | 6.1 migration `20260818090000_schema_drift_snapshot_history_probe.sql` built + mirrored + manifest regenerated, **NOT deployed on production** (D3: bundled into the Phase 3 production window below); deployed to staging by #2104 and proven end-to-end there. 6.2 contract + `supabase/drift-allowlist.json` (5 seeded superseded guards; ~18 §1.1 versions deliberately NOT allowlisted → first post-deploy run reports them; fix = fail-fast guard migrations, never bare allowlisting). 6.3 `tests/search-health-index-coverage.test.ts` + unmonitored list (44 entries, 8 monitor-candidates). New P2 row queued (inbox `d6ce8a1d`). Stale branch `claude/database-drift-allowlist-48839e` (tip `3fe1deafe`) was recreated after merge — redundant, owner may delete. | +| 7 — Deferred | ☐ not started | — | Each its own session, per playbook. | + +**Owner decisions (2026-08-18):** D1 = codify-as-live (128 MB on the four hybrids, 64 MB on the +other six) — DECIDED. D2 = `work_mem` eval-canary exemption — GRANTED. D3 = 6.1 deploy bundled into +the Phase 3 production window — DECIDED. + +**Production window owed (one window, `supabase db push` — real execution, NOT +`migration repair --status applied`; none of these ships a validation guard so none is eligible +for history repair):** + +| Migration | Effect on production | +| -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `20260818090000_schema_drift_snapshot_history_probe` | **real change** — `schema_drift_snapshot()` v2 (Phase 6.1); first live `migration_history` report expected to list the ~18 unallowlisted §1.1 versions | +| `20260818110000_codify_live_rpc_work_mem` | no-op — ten values already match | +| `20260818111000_codify_schema_only_indexes_and_triggers` | no-op — all eight objects exist | +| `20260818112000_reconcile_chain_stale_table_columns` | no-op — column and both defaults already as declared | +| `20260818113000_forward_codify_hybrid_owner_matches_bodies` (after #2111 merges) | no-op — live already carries the bodies | + +No canary (D2), no index build. Prove with a live-drift dispatch afterwards: function mismatches +must be zero; `missing_live` indexes stay 20 (Phase 4's job); `migration_history` findings appear +for the first time (Phase 6.2's job — guard migrations, never bare allowlisting). + +**Ledger state note (2026-08-18, evening):** pending inbox on `main`: 7 (incl. `#056` `5ed2f873`) +plus #2111's four (`#316` final, `#183` rider, two P3 captures) and its three cancels. One serialized +`issues:reconcile` from a fresh base **after #2111 merges**, not before. Never merge `main` into a +PR carrying a reconciliation; re-cut from `main`. ## Coordination rules