Retrieval correctness: repair #325 golden regression, corpus-grounded finding #11 fix, RC9 confidence hardening

docs/process-hardening.md

@@ -155,6 +155,7 @@ All approved render-surface modules are extracted. `ClinicalDashboard.tsx` went
 - **Root cause (confirmed live):** the soft-tail branch of `shouldShortCircuitUnsupportedSearch` (`src/lib/rag.ts`) fires on `analysis.queryClass === "unsupported_or_general"`, and that class is set by `analyzeQueryWithClassifierFallback`, which calls a **generative LLM classifier** (6s timeout, `reasoningEffort:"low"`, uncached) for low-confidence queries. That call is nondeterministic: it reclassifies "bipolar disorder" to a supported class on some runs (→ answered) and declines/times-out on others (→ short-circuited to 0). Reproduced with a same-process ×N probe.
 - **Why a clean Phase-1 fix does NOT exist:** the deterministic analyzer carries **no signal** distinguishing in-corpus topics from out-of-corpus (bipolar, anorexia, gout, DKA all produce identical `unsupported_or_general`, confidence ≈ 0.40, `canonicalTerms` = query tokens). Only the corpus can tell them apart. Removing the soft-tail short-circuit fixes the valid topics but **regresses `unsupported_correct_rate` 1.0 → 0.79** on `eval:quality --rag-only`: a lexical distinctive-term relevance gate in `chooseAnswerRoute` either over-refuses legitimate semantic/vector matches (whose exact term is not in the retrieved text — e.g. the `rag-routing.test.ts` "admission" fixtures) or under-refuses invented terms ("florbizone syndrome") that score strongly on generic scaffolding words ("syndrome"/"management"). The corpus is broad (gout 13 / crohn 49 / appendicitis 30 / angioplasty 32 / bipolar 719 chunks), so almost no common medical term is truly absent — grounded low-confidence answers, not fabrication, are the realistic worst case for real terms; only genuinely invented terms should refuse.
 - **Decision (2026-07-03):** the risky change (soft-tail removal + relevance gate + invented-term eval controls) was **reverted**; only a safe, independent hardening was kept — `fetchEnabledRagAliases` no longer caches `[]` on a transient `rag_aliases` read error (which would suppress alias expansion for the whole TTL). Finding #11 is **re-scoped into RAG optimisation Phase 2** ("fit retrieval to content"), where it should be fixed with corpus-grounded relevance — IDF/corpus-frequency weighting of query terms and/or the semantic relevance model, plus the data-driven query-understanding vocabulary (RC6) so the deterministic classifier recognises in-corpus topics up front. Any gate must keep `eval:quality --rag-only` `unsupported_correct_rate` at 1.0 (add invented-term controls like "florbizone syndrome management" / "quxbyria disorder treatment" once it can pass them) while letting valid bare topics answer. A cheaper interim option worth measuring first: **classifier-verdict memoization** to at least make the current behaviour deterministic per query.
+- **RESOLVED (2026-07-07, `claude/retrieval-correctness`):** the Phase-2 corpus-grounded fix shipped. `corpus_topic_term_stats` (migration `20260707100000`, applied live) reports per-term title-topic membership (with a measured 5% genericity ceiling — "management"/"guideline" headline ~18–20% of titles and are scaffolding; real topics ≤3%) and chunk-level presence, scoped with the exact retrieval `owner_filter`. `src/lib/corpus-grounding.ts` + the `analyzeQueryWithClassifierFallback` hook classify only the soft-tail branch (pattern-guarded refusals and higher-confidence classes untouched): in-corpus bare topics deterministically reclassify to `broad_summary` and answer; corpus-absent queries skip the LLM and refuse deterministically (trigram correction still runs); inconclusive keeps memoized-LLM behaviour; DB errors fail open. Verified live: "bipolar disorder" / "anorexia management" answer 4/4 runs with the right document at rank 1 (new golden cases `bare-topic-bipolar` / `bare-topic-anorexia`); "florbizone syndrome management" / "quxbyria disorder treatment" refuse 4/4 runs (new `ragEvalCases` controls); golden eval 36/36.
 
 ## Retrieval changes must pass the golden eval before merge (2026-07-03)
 
@@ -188,6 +189,7 @@ All approved render-surface modules are extracted. `ClinicalDashboard.tsx` went
 
 - **Shipped on `claude/universal-search-algorithm-ryrps7`:** finding #11 interim fix (classifier-verdict memoization, 15-min TTL, errors not memoized — the "cheaper interim option" from the 2026-07-03 entry above), pre-clamp tiebreak completion in `selectRetrievalEvidence`, weak-match OR-augmentation (kill switch `RAG_TEXT_WEAK_OR_RELAXATION=false`), `similarity_origin` telemetry, shared `catalog-search` primitives replacing the four per-domain rankers, forms mode-kind honesty, tools dataset dedupe, `/api/search/universal` federated endpoint, and the cross-entity typeahead in `UniversalSearchCommandSurface`.
 - **Eval debt (blocking merge, not development):** `npm run eval:retrieval:quality` (23/23) and `eval:quality --rag-only` (`unsupported_correct_rate` 1.0) could NOT be run in the authoring environment (no live keys) — they MUST be run before merge per the standing gate above, with special attention to the weak-match OR-augmentation (flag off restores relax-on-empty exactly) and the retrieval-selection tiebreak (tie-only by construction).
+- **Eval debt settled 2026-07-07 — both flagged changes had regressed the golden eval (31/34 on main).** Isolated case-by-case against the same live corpus (pre-#325 code passed all 3): the weak-match OR-augmentation buried `opioid-withdrawal-doses` (docRecall@5 → 0; OR recall is append-only at the RPC merge but NOT after re-ranking), and the `selectRetrievalEvidence` pre-clamp tiebreak buried `alcohol-ciwa-threshold` + `clozapine-cbc-abbreviation-threshold` (boost-stacking magnitude re-ordered saturated top-5 sets). Fixed on `claude/retrieval-correctness`: `RAG_TEXT_WEAK_OR_RELAXATION` now defaults to `false` (opt-in experiment flag; re-enable only behind a fresh full golden run) and the selection-layer tiebreak is removed — the pre-clamp deep tiebreak lives in `rankClinicalResults` (below the engineered `rankingTieBreakScore`), which is the only place it is eval-proven. Lesson reinforced: "tie-only by construction" still changes which tied candidate wins; ordering changes inside saturated score regions are behavior changes and need the golden gate.
 - **UI verification run:** new `tests/ui-universal-search.spec.ts` (grouped typeahead renders, item selection navigates, Enter still runs the mode search — universal endpoint mocked), full `ui-tools`/`ui-tools-task-directory` (40/40) and `ui-smoke`/`ui-overlap` suites against a live dev server in demo mode, plus a live curl of `/api/search/universal` (grouped payload, domain filter, 400 on short query).
 - **Known limitation:** the typeahead spec mocks the universal endpoint; an end-to-end spec against live seeded registries needs the owner-auth Playwright project (E2E_USER_* keys).
 

docs/rag-hybrid-findings-and-todo.md

@@ -92,9 +92,10 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows
  Note: this only affects the **panel**; the answer-retrieval path (`searchChunksWithTelemetry`) has
  no per-doc cap and doesn't need one — the comparison gate already enforces ≥2 distinct docs, and
  single-topic queries _should_ be able to draw multiple chunks from the best document.
- - ⏳ **Synthetic text similarity (RC9)** `least(0.95, 0.56 + text_rank*0.39)` still feeds coverage
- gates that assume a real cosine — gate text-only paths on `text_rank`/`rrf` instead. (Cleanest
- remaining ranking-correctness item.)
+ - 🔧 **Synthetic text similarity (RC9)** — superseded; see item 21 (2026-07-07 audit): the SQL
+ formula is gone (`match_document_chunks_text` returns similarity 0), the three remaining
+ app-side fabricators are tagged, and the fabricated-"high"-confidence defect is fixed;
+ only the telemetry-gated threshold recalibration remains.
  - ⏳ **Source-strength as a filter not just a penalty (RC8)**; **threshold floors (RC5)**;
  **rerank trigger (RC10)** — see item 4's note (marginal without a chunk-level eval metric).
  - ⏳ **Differentials flowchart-action boost (dropped in the PR #120 merge).** The codex/RAG_FIX
@@ -199,19 +200,34 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows
  `typoCorrected` flag; only fires for would-be-unsupported queries so no hot-path cost). Rescues
  typo queries whose corrected form is a _supported_ class (e.g. a typo'd clozapine/dose query
  → table_threshold). Golden 23/23 unchanged, 682 tests pass.
- - ⚠️ **Pre-existing bug surfaced (NEW, finding #11):** unsupported-classified queries retrieve
- **nondeterministically** — the _same_ query in the _same_ process alternates
- `unsupported_short_circuit` (0 results) vs `text_fast_path`/`hybrid` (real results), e.g.
- "anorexia management" (no typo). Classification is pure and all caches honour `skipCache`, so the
- variance is elsewhere in the unsupported-query path (candidate: alias fetch/expansion or an async
- step) — needs runtime instrumentation to pin. It masks the benefit above (a typo query whose
- corrected form is ALSO borderline-unsupported, like "schizophrenai management", inherits the
- flakiness). Confined to unsupported queries (golden set never hits it), so it never affected the
- committed metrics. High-priority to fix — it means some valid clinical topics ("bipolar disorder",
- "anorexia management") intermittently return nothing.
+ - ✅ **Finding #11 FIXED (2026-07-07) — corpus-grounded relevance.** Root cause was the
+ nondeterministic LLM classifier deciding the unsupported soft tail (see
+ docs/process-hardening.md 2026-07-03 entry). Two-part fix: PR #325's classifier-verdict
+ memoization (interim determinism per query per 15-min TTL), then the Phase-2 fix on
+ `claude/retrieval-correctness`: `corpus_topic_term_stats` (migration `20260707100000`,
+ applied live) + `src/lib/corpus-grounding.ts` classify soft-tail queries against the
+ corpus's own topic vocabulary (title-tsvector matches under a 5% genericity ceiling;
+ chunk-absence = invented term) BEFORE any LLM call. In-corpus bare topics ("bipolar
+ disorder", "anorexia management") deterministically reclassify to `broad_summary` and
+ answer (verified live: 4/4 identical runs, docs at rank 1); corpus-absent queries
+ ("florbizone syndrome management", "quxbyria disorder treatment") skip the LLM and refuse
+ deterministically, with the trigram-correction escape hatch preserved for typos.
+ Invented-term controls added to `ragEvalCases`; bare-topic golden cases added
+ (`bare-topic-bipolar`, `bare-topic-anorexia`). Inconclusive verdicts (e.g. "gout
+ management" — chunk-present but no title topic) keep the legacy memoized-LLM behaviour.
  - ⏳ Still hard-coded (lower priority now the trigram path exists): moving `synonymGroups` /
  `domainAliasGroups` / `medicationAliasGroups` into `rag_aliases`; generalising the special-case
- rewrites off `RagQueryClass`.
+ rewrites off `RagQueryClass`. **Design constraint (2026-07-07):** this is NOT a plain seed
+ migration. The groups are consumed inside the synchronous deterministic analyzer
+ (`analyzeClinicalQuery`), which must keep working in demo mode and in unit tests without a
+ DB, while `rag_aliases` rows flow through a different mechanism (`fetchEnabledRagAliases` →
+ retrieval query variants + the unsupported-short-circuit alias guard). Seeding the same
+ groups into `rag_aliases` while the in-code groups remain would double-expand variants and
+ change short-circuit behaviour corpus-wide — a behavior change needing its own golden +
+ rag-only eval run, not a data chore. Deferred from the 2026-07-07 retrieval-correctness
+ branch for that reason; do it as a dedicated eval-gated change (either an async analyzer
+ vocabulary refactor, or DB-only expansion with the in-code groups retired from the variant
+ path in the same change).
 
 ## P2 — offline/fallback remainder (Workstream F)
 
@@ -261,12 +277,29 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows
  `review_due`/`unverified` source outranks a lower-relevance `current`/`reviewed` one. The
  manual golden-eval checklist remains the live backstop; no further action.
 21. 🔶 **Recalibrate gates for synthetic text-only similarity (RC9 residual) — DATA NOW
- FLOWING (2026-07-06).** `synthetic_similarity_count` and `text_or_relaxation_used` are now
- persisted into `rag_retrieval_logs.metadata` (they were computed but dropped by the
- telemetry whitelist in /api/search). Once ~2 weeks of live rows exist, recalibrate
- `evaluateEvidenceCoverageGate` / text-fast-path thresholds against real cosine
- distributions: query `metadata->>'synthetic_similarity_count'` joined to `is_miss` to see
- how often synthetic scores cross the 0.58/0.62 gates on misses vs hits.
+ FLOWING (2026-07-06); audited 2026-07-07, scope reduced.** `synthetic_similarity_count` and
+ `text_or_relaxation_used` are now persisted into `rag_retrieval_logs.metadata` (they were
+ computed but dropped by the telemetry whitelist in /api/search). Once ~2 weeks of live rows
+ exist, recalibrate `evaluateEvidenceCoverageGate` / text-fast-path thresholds against real
+ cosine distributions: query `metadata->>'synthetic_similarity_count'` joined to `is_miss` to
+ see how often synthetic scores cross the 0.58/0.62 gates on misses vs hits. Full consumer
+ audit on `claude/retrieval-correctness`: the headline `least(0.95, 0.56 + text_rank*0.39)`
+ proxy NO LONGER EXISTS — `match_document_chunks_text` already returns `similarity = 0` with
+ hybrid capped at 0.5 and the lexical signal isolated in `lexical_score` (codified in
+ schema.sql with the "do not fabricate" comment). What remains synthetic are three app-side
+ fabricators, all tagged `similarity_origin: "synthetic_text"`: the document-lookup fast path
+ (0.58 + documentScore, hybrid ≤ 0.94), memory-card chunk loader (0.58 + confidence·0.28,
+ hybrid ≤ 0.89), and table-fact signal matches. Their consumers:
+ `evaluateEvidenceCoverageGate` / `shouldReturnTextFastPath` / `chooseAnswerRoute` /
+ `shouldUseExtractiveAnswer` (thresholds 0.32–0.76 — the fabricated 0.58 floor is
+ deliberately load-bearing there, always paired with structural checks like
+ `directTitleSupport`; re-gating them on native signals is the deferred recalibration and
+ must not be attempted without the telemetry distributions), `buildRetrievalDiagnostics`
+ (topScore < 0.5 weak gate — floor also load-bearing), and `deriveConfidence`. **Fixed
+ (2026-07-07):** `deriveConfidence` no longer lets a fabricated 0.82+ mint a "high"
+ answer-confidence label — "high" requires a genuine-cosine citation; synthetic-origin
+ evidence caps at "medium" (strictly tightening, ordering/routing untouched, unit-tested in
+ tests/rag-score.test.ts).
 22. ⏳ **Registry-to-corpus embedding (universal search Phase 5).** Medications/services/forms/
  differentials are federated into `/api/search/universal` but are not retrieval-corpus
  entities, so Answer mode cannot cite them. Concrete implementation spec (in order):

docs/retrieval-quality-runbook.md

@@ -50,6 +50,12 @@ The command requires the same live-eval environment as the existing RAG eval scr
 - `OPENAI_API_KEY`
 - `RAG_EVAL_OWNER_ID`, `LOCAL_NO_AUTH_OWNER_ID`, or `RAG_EVAL_OWNER_EMAIL`
 
+Since the 2026-07-06 public promotion the live corpus is entirely `owner_id = NULL`, so set
+`RAG_EVAL_OWNER_ID=00000000-0000-0000-0000-000000000000` (the public-owner sentinel —
+`retrieval_owner_matches` maps it to NULL-owner rows, mirroring anonymous production search). A
+real owner UUID now scopes retrieval to zero documents and fails every case; leaving it unset
+throws the owner-scope guard.
+
 Optional cost fields:
 
 - `RAG_EVAL_INPUT_USD_PER_MILLION`

scripts/fixtures/rag-retrieval-golden.json

@@ -327,5 +327,23 @@
  "topK": 8,
  "expectTableEvidence": false,
  "forceEmbedding": true
+ },
+ {
+ "id": "bare-topic-bipolar",
+ "query": "bipolar disorder",
+ "expectedQueryClass": "broad_summary",
+ "expectedDocumentSubstrings": ["Bipolar"],
+ "expectedContentTerms": [["bipolar", "mania", "manic", "mood"]],
+ "topK": 8,
+ "expectTableEvidence": false
+ },
+ {
+ "id": "bare-topic-anorexia",
+ "query": "anorexia management",
+ "expectedQueryClass": "broad_summary",
+ "expectedDocumentSubstrings": ["Anorexia"],
+ "expectedContentTerms": [["anorexia", "eating", "weight"]],
+ "topK": 8,
+ "expectTableEvidence": false
  }
 ]