rag.ts decomposition — part 1: quote verification, source block, numeric verification, context selection (conflict-resolved merge)

.claude/hooks/session-start.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+# SessionStart hook for Claude Code on the web.
+# The app is engine-strict on Node 24.x / npm 11.x, but web containers ship an
+# older Node on PATH, so nothing installs or runs until Node 24 is present.
+# Installs Node 24 into $HOME/.node24 (cached with the container), exposes it
+# via $CLAUDE_ENV_FILE, and installs npm dependencies.
+set -euo pipefail
+
+if [ "${CLAUDE_CODE_REMOTE:-}" != "true" ]; then
+ exit 0
+fi
+
+NODE_VERSION="24.13.0"
+NODE_HOME="$HOME/.node24"
+NODE_BIN="$NODE_HOME/node-v${NODE_VERSION}-linux-x64/bin"
+
+current_major="$(node -v 2>/dev/null | sed -E 's/^v([0-9]+).*/\1/' || echo 0)"
+if [ "$current_major" != "24" ] && [ ! -x "$NODE_BIN/node" ]; then
+ echo "[session-start] Installing Node ${NODE_VERSION} (found v${current_major:-none})"
+ mkdir -p "$NODE_HOME"
+ curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.xz" \
+ | tar -xJ -C "$NODE_HOME"
+fi
+
+if [ -x "$NODE_BIN/node" ]; then
+ export PATH="$NODE_BIN:$PATH"
+ echo "export PATH=\"$NODE_BIN:\$PATH\"" >> "$CLAUDE_ENV_FILE"
+fi
+
+echo "[session-start] Using node $(node -v) / npm $(npm -v)"
+
+cd "$CLAUDE_PROJECT_DIR"
+# npm ci keeps the lockfile untouched (npm install rewrites peer/optional
+# metadata and dirties the worktree); skip entirely when the cached container
+# already has node_modules.
+if [ ! -d node_modules ]; then
+ npm ci --no-audit --no-fund
+ echo "[session-start] Dependencies installed"
+else
+ echo "[session-start] node_modules already present, skipping install"
+fi

.claude/settings.json

@@ -0,0 +1,14 @@
+{
+"hooks": {
+"SessionStart": [
+ {
+"hooks": [
+ {
+"type": "command",
+"command": "$CLAUDE_PROJECT_DIR/.claude/hooks/session-start.sh"
+ }
+ ]
+ }
+ ]
+ }
+}

.prettierignore

@@ -16,3 +16,4 @@ scratch/
 # Generated by `supabase gen types`; keep the generator's formatting so
 # regeneration stays churn-free.
 src/lib/supabase/database.types.ts
+supabase/drift-manifest.json

data/differentials-snapshot.json

@@ -27730,13 +27730,6 @@
  }
  ],
  "presets": [
- {
- "id": "scenario-presets",
- "query": "# Scenario Presets",
- "signals": [],
- "entryIds": [],
- "presentationSlugs": []
- },
  {
  "id": "1-older-adult-acute-confusion",
  "query": "older adult acute confusion",
@@ -27788,14 +27781,6 @@
  }
  ],
  "redFlagFlows": [
- {
- "id": "red-flag-flows",
- "title": "# Red Flag Flows",
- "entryId": "",
- "presentationSlug": "red-flag-flows",
- "bedsideQuestions": "",
- "keyRedFlags": ""
- },
  {
  "id": "1-suicide",
  "title": "1. Suicide",
@@ -27873,15 +27858,7 @@
  "medicine": ["medication", "iatrogenic", "toxicity"],
  "qt": ["qtc", "arrhythmia", "toxicity"],
  "catatonia": ["mutism", "shutdown", "stupor"],
- "field": ["weight"],
- "presentation": ["2.6"],
- "clinicalhinge": ["2.2"],
- "mustnotmiss": ["2.0"],
- "immediateactions": ["1.7"],
- "mimics": ["1.5"],
- "investigations": ["1.2"],
- "tags": ["1.1"],
- "optiontext": ["1.0"]
+ "field": ["weight"]
  },
  "governance": {
  "version": "v10",

docs/archive/operator-decisions-2026-07-06.md

@@ -0,0 +1,27 @@
+# Operator decisions — 2026-07-06
+
+Approvals granted during the repository-review follow-up session. **No live actions were taken from this checklist** — the authoring environment had no authenticated Supabase connection. Any live-connected session (or the operator) can execute these without re-asking.
+
+## Pending live migrations — APPROVED to apply
+
+**Decision:** The operator explicitly approved applying both pending migrations to the live `Clinical KB Database` project (`sjrfecxgysukkwxsowpy`).
+
+1. **M13** `supabase/migrations/20260702000000_commit_generation_preserve_legacy_artifacts.sql` — prerequisite for reindex commits safely purging legacy NULL-generation rows.
+2. `supabase/migrations/20260703030000_reconcile_storage_cleanup_jobs_indexes.sql` — drops the legacy auto-named indexes and (re)creates the intended named/partial indexes to match `supabase/schema.sql`. (This is the migration the debt log marked "apply to live only with explicit approval" — that approval is now recorded here.)
+
+**Post-apply verification (required):**
+
+```bash
+npm run check:m13-migration
+npm run reindex:health
+npm run check:indexing
+npm run check:supabase-project
+```
+
+## Edge function deploy — APPROVED
+
+**Decision:** Deploy `supabase/functions/indexing-v3-agent` so the JSONB status-RPC parsing is live (follow-up noted in `docs/process-hardening.md` "Live database drift reconciliation (2026-07-05)").
+
+## Context
+
+Approval came alongside the operator green-lighting the two structural efforts (finish the ClinicalDashboard admin cutover; decompose `src/lib/rag.ts`) tracked in `docs/process-hardening.md`.

docs/codebase-index.md

@@ -250,6 +250,16 @@ Golden retrieval fixture: `scripts/fixtures/rag-retrieval-golden.json`
 - Registry modes: services, forms, medications, differentials
 - Demo mode: synthetic data when Supabase unavailable (`demo-data.ts`, `isDemoMode()` in `env.ts`)
 
+### Global search composer placement rules
+
+One shared composer (`master-search-header.tsx`) serves every mode. Placement:
+
+- **Mode homes** (`/services`, `/forms`, `/favourites`, `/differentials`, `/applications`, and dashboard homes): inline in the hero via the `mode-home-composer-slot` portal, on phone and tablet+ alike.
+- **Result and detail views**: fixed bottom dock on phone (compact variant on submitted searches), sticky top from `sm` up.
+- **Results routing**: standalone routes own their submitted searches via `?q=…&run=1` (`/services` → `ServicesNavigatorPage`, `/forms` → `FormsSearchResultsPage`, `/differentials` → `DifferentialsHome` results view, `/favourites` filters the command library in place). Answer, Documents, and Prescribing submitted searches render inside `ClinicalDashboard` — intentional, since they need retrieval/answer state. `/?mode=favourites` redirects to `/favourites`; `/?mode=differentials` redirects to `/differentials`.
+- **Intentionally composer-free routes**: `/differentials/presentations/*` (comparison workflow owns its chrome), `/documents/[id]` viewer (has its own in-document ask composer), `/documents/source/*` (document flow owns mobile chrome). Do not re-flag these in search-consistency audits.
+- **Local filter fields** (sidebar "Search chats", document drawer "Find a document"/"Find a source PDF") are scoped filters, not global search; they share the `fieldControlWithIcon`/`fieldIcon` primitives.
+
 ---
 
 ## Key config files

docs/database-drift-detection.md

@@ -0,0 +1,134 @@
+# Database drift detection (`npm run check:drift`)
+
+Last updated: 2026-07-07
+
+This repo's worst operational incidents were live-vs-repo schema drift: hybrid
+retrieval RPCs silently broken on live for an unknown period, and migrations
+recorded as applied whose objects were absent. `search_schema_health()` guards
+a curated subset (signatures, 22 required indexes, execution smoke).
+`check:drift` generalizes that into a full-inventory comparison of **every**
+application-owned object against `supabase/schema.sql`.
+
+## How it works
+
+Three committed artifacts:
+
+| Artifact | Role |
+| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `supabase/migrations/20260706200000_schema_drift_snapshot.sql` | `public.schema_drift_snapshot()` — service-role-only RPC returning the normalized live inventory (also declared in `supabase/schema.sql`; a test enforces byte parity). |
+| `supabase/drift-manifest.json` | The expected state: the same snapshot captured from a **from-scratch replay of `supabase/schema.sql`** into a disposable `supabase/postgres` Docker container (`npm run drift:manifest`). Embeds the sha256 of the schema.sql it came from. |
+| `supabase/drift-allowlist.json` | Known, documented divergence (each entry has a `reason`). Reported as warnings; anything not listed fails the check. |
+
+`npm run check:drift` (needs live service-role env) verifies the project ref,
+fails fast if the manifest is stale, calls the RPC, diffs, applies the
+allowlist, and exits 1 on unallowlisted divergence. The offline half runs in
+`tests/drift-detection.test.ts` under `verify:cheap`: manifest freshness
+(sha256), migration↔schema.sql parity for the snapshot function, allowlist
+hygiene, and unit tests of the comparison engine.
+
+Inventory coverage: functions (comment/whitespace-stripped `pg_get_functiondef`
+md5 + sorted ACLs), indexes (normalized `pg_get_indexdef`), RLS policies
+(permissive/roles/cmd/qual/with_check), table shapes (columns sorted by name,
+RLS flags, reloptions, ACLs), constraints, triggers, views, extensions, and
+storage bucket rows + storage.objects policies.
+
+### Noise sources handled by design
+
+- **Whitespace/comments in function bodies** — `prosrc` is stored verbatim, so
+ migration text vs schema.sql text differ trivially; both are stripped before
+ hashing (the same trick `20260701140631` used to validate byte-equivalence).
+- **Rendering search_path** — the snapshot pins `search_path = ''` so
+ `pg_get_expr`/`pg_get_indexdef`/policy quals render fully qualified and
+ identically on live and replay.
+- **Column ordinal drift** — live tables grew via `ALTER TABLE ADD COLUMN`;
+ columns compare sorted by name, not `attnum`.
+- **ACL append order** — aclitem arrays are sorted.
+- **Duplicate migration-history versions** — history is _not_ compared at all;
+ the check compares actual object state (history presence proved unreliable:
+ see `20260703030000` below).
+- **Platform-provisioned extensions** (pg_net, pgsodium, pgmq, …) — extra live
+ extensions are informational; missing schema.sql-declared ones fail.
+- **Legacy index names** — `alias` allowlist entries assert the live database
+ carries the _identical_ name-stripped index definition under a legacy name
+ (the machine-checked version of `search_schema_health()`'s `index_aliases`).
+
+### Workflow
+
+- Change `supabase/schema.sql` → run `npm run drift:manifest` (Docker) in the
+ same PR. The freshness test fails otherwise. This also continuously proves
+ schema.sql replays from scratch — which it did **not** before 2026-07-07
+ (`document_index_units` was declared after its first validating reference).
+- Live drifts (check:drift red) → either codify live state (migration +
+ schema.sql + manifest regen) or fix live **through an approved migration**.
+ Never raw SQL against live; that is how this incident class started.
+- New known-divergence → allowlist entry with a reason and a backlog line here.
+- After the pending-migration backlog lands, delete the matching allowlist
+ entries; check:drift reports stale entries so they cannot silently linger.
+
+## 2026-07-07 baseline audit (three-way: live vs schema.sql vs migration chain)
+
+Both repo lineages were replayed into scratch containers and compared with the
+live inventory. 166 divergent keys, fully classified:
+
+- **Reconciled in this PR (schema.sql/migrations only, live untouched):**
+ replay-order fix; `20260707000000_codify_live_observed_drift.sql` codifying
+ 15 live-only columns (`document_images` ×7, `document_index_quality` ×6,
+ `ingestion_job_stages` ×2 — worker-written, branch DBs broke without them),
+ 3 `content_not_blank` NOT VALID checks, autovacuum reloptions on 5 RAG
+ tables, `content_hash` nullability alignment, 4 live-only functions
+ (`set_owner_id_from_auth_uid` + rag_queries/misses triggers,
+ `purge_expired_rag_queries`, `correct_clinical_query_terms`,
+ `invoke_ingestion_worker`) and 2 ACL tightenings; schema.sql function/policy
+ text realigned to the migration-chain truth for `analyze_rag_tables`,
+ `claim_indexing_v3_agent_jobs`, `is_committed_artifact_generation(uuid,jsonb)`,
+ `match_document_memory_cards_hybrid`, and 6 owner-read policies (operand
+ order only).
+- **Allowlisted (124 entries)** — see `supabase/drift-allowlist.json`; backlog
+ below.
+
+## Reconciliation backlog
+
+Ordered; each item removes allowlist entries when it lands. Items touching the
+live project need explicit operator approval.
+
+1. **Apply the pending migrations** (`supabase db push` after review):
+ `20260705210000` (owner-sentinel rewrite — 8 live function bodies stale),
+ `20260706010000` (search_schema_health M13 guard), `20260706130000`
+ (embedding-fields-text sentinel), `20260706200000` (drift snapshot RPC —
+ required before check:drift can run at all), `20260707000000` (codification
+ wave; no-op on live by construction).
+2. **`20260703030000` is recorded as applied but its effects are absent on
+ live** (storage_cleanup_jobs still has the legacy index names and the
+ non-partial status index). Recorded-but-ineffective is the exact original
+ incident class, recurring. Re-apply its idempotent statements under a new
+ version with approval; do not trust history presence.
+3. **Codify the remaining live-only functions**: `get_visual_evidence_cards`,
+ `repair_enrichment_quality_batch`, `run_all_visual_eval_cases`,
+ `run_visual_eval_case` (same pattern as `20260707000000`).
+4. **Authenticated-grant posture decision**: live revoked the authenticated
+ Data API grants on 17 tables (fail-closed hardening; the owner-read RLS
+ policies are currently dead on live) while schema.sql still declares them.
+ Either codify the revokes (schema.sql + tests + migration) or restore the
+ grants live.
+5. **PUBLIC-execute revokes**: 4 security-invoker functions retain default
+ PUBLIC execute on live (`detect_legacy_ivfflat_indexes`,
+ `document_summary_text`, `search_document_chunks`,
+ `set_document_embedding_field_content_hash`).
+6. **`document_label_metadata` direction**: schema.sql is AHEAD (hidden-label
+ filtering added without a migration). Ship the migration or revert.
+7. **Index estate**: rename 10 legacy-named live indexes to schema.sql names;
+ decide the 24 schema.sql-declared indexes absent on live (recreate vs
+ remove — includes `documents_search_idx`, `document_chunks_anchor_idx`,
+ `documents_owner_content_hash_unique_idx`); drop ~45 live-only duplicate
+ indexes after `pg_stat_user_indexes` scan verification; reshape 3
+ (`import_batches_status_created_idx`, `ingestion_jobs_document_status_idx`,
+ `ingestion_jobs_status_next_run_idx`).
+8. **Constraints**: add `ingestion_job_stages_job_id_fkey` to live; align the
+ `rag_visual_eval_*` document FK definitions.
+9. **`invoke_ingestion_worker`** hardcodes the project URL — migrate to the
+ GUC pattern (`20260702160000` precedent).
+10. **Migration-chain fidelity** (affects Supabase Preview/branches, not
+ live): 13 keys where the chain diverges from schema.sql — buckets are only
+ created by schema.sql, `documents`/`ingestion_jobs` updated_at trigger
+ variants, post-legacy-drop embedding-fields index set,
+ `document_chunks_content_trgm_idx` shape, `rag_visual_eval_*` shapes.