Retrieval correctness: repair #325 golden regression, corpus-grounded finding #11 fix, RC9 confidence hardening

[BigSimmo]

Retrieval correctness workstream

Branch: claude/retrieval-correctness. Every behavior change below was gated live against Clinical KB Database (sjrfecxgysukkwxsowpy, 2065 public docs) with RAG_EVAL_OWNER_ID=00000000-0000-0000-0000-000000000000 (the public-owner sentinel — the corpus went all-public on 2026-07-06, so the old real-UUID eval owner scopes to zero documents; runbook updated expectation).

Commit 1 — repair PR #325 golden-eval regression (task 0, unplanned)

PR #325 merged with its golden eval unrun (documented eval debt). Settling that debt found main at 31/34. Isolated against the same live corpus (pre-#325 code passes all three):

• Weak-match OR-augmentation (default on) buried opioid-withdrawal-doses (docRecall@5 → 0.00): OR recall is append-only at the RPC merge but not after re-ranking. → RAG_TEXT_WEAK_OR_RELAXATION now defaults off (opt-in experiment flag; re-enable only behind a fresh full golden run).
• selectRetrievalEvidence pre-clamp tiebreak buried alcohol-ciwa-threshold + clozapine-cbc-abbreviation-threshold: "tie-only by construction" still changes which saturated candidate wins. → removed; the eval-proven pre-clamp deep tiebreak lives in rankClinicalResults below the engineered rankingTieBreakScore (PR fix(rag): deterministic ranking tiebreak + alias cache hardening (Phase 1) #218) and stays. Regression-guard unit test replaces Universal search: fix core RAG search defects and federate medications/services/forms/differentials/tools into one cross-entity search #325's test.

Golden after fix: 34/34, top_k_hit_rate=1, failed_cases=0.

Commit 2 — finding #11: corpus-grounded relevance (task 2)

The unsupported soft tail is now decided by the corpus, not the LLM classifier lottery:

• New corpus_topic_term_stats RPC (migration 20260707100000, applied live; read-only, additive, service_role-only, scoped exactly like retrieval: retrieval_owner_matches + status='indexed' + committed generation).
• src/lib/corpus-grounding.ts: title-topic anchor under a 5% genericity ceiling (measured live: "management" 18.2% / "guideline" 19.6% of titles = scaffolding; real topics ≤3%: assessment 3.0%, disorder 1.6%, bipolar/anorexia <0.1%) + chunk-absence as the invented-term signal. In-corpus bare topic → deterministic broad_summary reclassification; corpus-absent token → LLM skipped, deterministic soft-tail refusal (trigram-correction escape hatch preserved); inconclusive/DB-error/missing-RPC → legacy behaviour (fail-open).
• Scoped strictly to the soft-tail branch — the pattern-guarded unsupported eval cases (DKA/pneumonia/SSRI/consumer noise) can't reach it by construction.
• Eval controls added: golden bare-topic-bipolar / bare-topic-anorexia (must answer, rank-1 live) and ragEvalCasesunsupported-invented-florbizone / unsupported-invented-quxbyria (must refuse).

Live determinism probe (4 runs each, identical outcomes):

florbizone syndrome management -> unsupported_short_circuit / 0 results / out_of_corpus
quxbyria disorder treatment -> unsupported_short_circuit / 0 results / out_of_corpus
bipolar disorder -> text_fast_path / broad_summary / in_corpus_topic / doc rank 1
anorexia management -> text_fast_path / broad_summary / in_corpus_topic / doc rank 1

Commit 3 — RC9: fabricated similarity vs answer confidence (task 3)

Full consumer audit (recorded in docs/rag-hybrid-findings-and-todo.md item 21): the headline least(0.95, 0.56 + text_rank*0.39) proxy no longer exists — match_document_chunks_text already returns similarity=0 with hybrid capped at 0.5. The three remaining app-side fabricators (document-lookup fast path, memory-card chunks, table-fact signal matches) are tagged synthetic_text; their 0.58-floor values are deliberately load-bearing in route/coverage gates (always paired with structural checks), so re-gating those on native signals stays deferred to the telemetry-backed recalibration (item 21).

Fixed the one provable defect: deriveConfidence let a fabricated 0.82+ (memory-card hybrid ≤0.89, document-lookup ≤0.94) mint a "high" clinician-facing confidence label from purely lexical evidence. "high" now requires a genuine-cosine citation; synthetic-origin evidence caps at "medium". Strictly tightening; ordering/routing/coverage untouched.

Tasks 1 & 4 — already on main (verify-only)

• Classifier-verdict memoization (task 1): shipped in PR Universal search: fix core RAG search defects and federate medications/services/forms/differentials/tools into one cross-entity search #325 (classifierVerdictMemo, NFKC-normalized keys, in-flight dedup, errors-not-memoized, 5 unit tests). Verified complete; corpus grounding now supersedes it for the soft tail and the memo remains for inconclusive verdicts.
• Pre-clamp tiebreak (task 4): rankClinicalResults has carried the pre-clamp deep tiebreak below rankingTieBreakScore since PR fix(rag): deterministic ranking tiebreak + alias cache hardening (Phase 1) #218 (eval-gated). The additional selection-layer copy from Universal search: fix core RAG search defects and federate medications/services/forms/differentials/tools into one cross-entity search #325 was the regression removed in commit 1.

Task 5 — deferred with design note

Seeding synonymGroups/domainAliasGroups/medicationAliasGroups into rag_aliases is not a data chore: the groups live in the synchronous analyzer (demo/test-safe) while rag_aliases rows flow through the variant + short-circuit-guard mechanism — seeding both double-expands and changes behaviour corpus-wide. Documented in findings item 10; needs its own eval-gated change.

Eval summaries (live)

Golden retrieval eval — baseline on main (before commit 1)

cases=34 top_k_hit_rate=1 failed_cases=3
FAIL opioid-withdrawal-doses: missing expected document(s) in top 5: Opioid use disorder
FAIL alcohol-ciwa-threshold: missing expected document(s) in top 5: Alcohol withdrawal; missing expected content term(s)
FAIL clozapine-cbc-abbreviation-threshold: missing expected content term(s) in top 5: withhold OR cease OR stop OR red

Golden retrieval eval — after commit 1

cases=34 top_k_hit_rate=1 median_latency_ms=2948 failed_cases=0

Golden retrieval eval — after commits 2+3 (36 cases incl. new bare-topic controls)

Running 36 golden retrieval case(s). mode=quality
cases=36 top_k_hit_rate=1 median_latency_ms=2768
retrieval_strategy_counts={"text_fast_path":25,"document_lookup_fast_path":1,"hybrid":10}
failed_cases=0

eval:quality --rag-only — baseline (after commit 1, before commits 2+3)

Grounded supported rate 0.9333
Unsupported correct rate 1
Citation failure rate 0.0476 (pre-existing: live corpus-sibling + local-latency issues)
Numeric grounding failure 0.0238 (pre-existing)

eval:quality --rag-only — final head (44 cases incl. the 2 new invented-term controls)

Grounded supported rate 0.9333 (baseline 0.9333 — unchanged)
Unsupported correct rate 1 (hard constraint held, WITH the new invented controls)
Citation failure rate 0.0455 (baseline 0.0476 — same 2 pre-existing absolute failures)
Numeric grounding failure 0.0227 (baseline 0.0238 — same 1 pre-existing absolute failure)

Both new invented-term controls (unsupported-invented-florbizone, unsupported-invented-quxbyria) pass (deterministic refusal via out_of_corpus).

Case-level deltas vs baseline are pre-existing live variance on cases outside the touched paths: unsupported-close-title-noise (route flake in baseline) now passes; assessment-documentation / illegal-substances flipped to "unexpected route strong" — both classify document_lookup at 0.73/0.78 confidence with needsClassifierFallback=false (probed), so they never reach the classifier fallback, soft tail, or corpus grounding. The recurring latency over 20000ms failures are local-machine→remote-DB latency (prod is co-located; same set failed at baseline).

verify:cheap (rebased head)

check:runtime + sitemap:check + lint (0 errors, 5 pre-existing warnings)
+ typecheck + unit tests: 130 files passed, 1203 tests passed

Notes

• .env.local recovery + key rotation: the live Supabase keys were rotated twice (Jul 5 legacy-disable, Jul 6 rotated_20260706); the current secret was recovered via the authenticated Supabase CLI --reveal and never pasted into chat.
• The corpus_topic_term_stats migration was applied to live via the MCP migration mechanism with content identical to the committed file (additive, read-only, service_role-only; verified against probe terms before wiring).
• Known pre-existing suite flake: the first test of heavy module-graph files (universal-search.test.ts) sat at the 15s global timeout edge under load; bumped that first test (and the new corpus-grounding first test) to 60s.

🤖 Generated with Claude Code

[supabase]

Updates to Preview Branch (claude/retrieval-correctness) ↗︎

Deployments	Status	Updated
Database	✅	Tue, 07 Jul 2026 05:37:27 UTC
Services	✅	Tue, 07 Jul 2026 05:37:27 UTC
APIs	✅	Tue, 07 Jul 2026 05:37:27 UTC

Tasks are run on every commit but only new migration files are pushed.
Close and reopen this PR if you want to apply changes from existing seed or migration files.

Tasks	Status	Updated
Configurations	✅	Tue, 07 Jul 2026 05:37:29 UTC
Migrations	⚠️	Tue, 07 Jul 2026 05:37:32 UTC
Seeding	✅	Tue, 07 Jul 2026 05:37:33 UTC
Edge Functions	✅	Tue, 07 Jul 2026 05:37:34 UTC

---

⚠️ Warning — Applied out-of-order migrations: [supabase/migrations/20260706200000_schema_drift_snapshot.sql supabase/migrations/20260707000000_codify_live_observed_drift.sql]

---

View logs for this Workflow Run ↗︎.
Learn more about Supabase for Git ↗︎.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.