From d927ce10c3b3bda563b4e7c841915ee06f49425f Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Tue, 7 Jul 2026 02:29:52 +0800 Subject: [PATCH 1/6] fix(retrieval): repair PR #325 golden-eval regression (31/34 -> 34/34) PR #325 merged with its golden retrieval eval unrun (no live keys in the authoring environment). Run live against the same corpus, pre-#325 code passes all 34 cases while main fails 3; both flagged changes regressed: - Weak-match OR-augmentation (default on) buried opioid-withdrawal-doses (docRecall@5 1.0 -> 0.0): OR recall is append-only at the RPC merge but not after re-ranking, so generic OR matches displace the expected document. RAG_TEXT_WEAK_OR_RELAXATION now defaults to false and is an opt-in experiment flag gated on a fresh full golden run. - The selectRetrievalEvidence pre-clamp tiebreak buried alcohol-ciwa-threshold and clozapine-cbc-abbreviation-threshold: ordering saturated ties by boost-stacking magnitude re-picks which tied candidate wins. Removed; the eval-proven pre-clamp deep tiebreak lives in rankClinicalResults below the engineered rankingTieBreakScore (PR #218) and stays. Golden retrieval eval after fix: 34/34, top_k_hit_rate=1, failed_cases=0. Co-Authored-By: Claude Fable 5 --- docs/process-hardening.md | 1 + src/lib/env.ts | 8 +++++--- src/lib/retrieval-selection.ts | 6 ------ src/lib/types.ts | 3 --- tests/retrieval-selection.test.ts | 15 +++++++++------ 5 files changed, 15 insertions(+), 18 deletions(-) diff --git a/docs/process-hardening.md b/docs/process-hardening.md index 053ed64a45..b1ffb080fe 100644 --- a/docs/process-hardening.md +++ b/docs/process-hardening.md @@ -181,6 +181,7 @@ All approved render-surface modules are extracted. `ClinicalDashboard.tsx` went - **Shipped on `claude/universal-search-algorithm-ryrps7`:** finding #11 interim fix (classifier-verdict memoization, 15-min TTL, errors not memoized — the "cheaper interim option" from the 2026-07-03 entry above), pre-clamp tiebreak completion in `selectRetrievalEvidence`, weak-match OR-augmentation (kill switch `RAG_TEXT_WEAK_OR_RELAXATION=false`), `similarity_origin` telemetry, shared `catalog-search` primitives replacing the four per-domain rankers, forms mode-kind honesty, tools dataset dedupe, `/api/search/universal` federated endpoint, and the cross-entity typeahead in `UniversalSearchCommandSurface`. - **Eval debt (blocking merge, not development):** `npm run eval:retrieval:quality` (23/23) and `eval:quality --rag-only` (`unsupported_correct_rate` 1.0) could NOT be run in the authoring environment (no live keys) — they MUST be run before merge per the standing gate above, with special attention to the weak-match OR-augmentation (flag off restores relax-on-empty exactly) and the retrieval-selection tiebreak (tie-only by construction). +- **Eval debt settled 2026-07-07 — both flagged changes had regressed the golden eval (31/34 on main).** Isolated case-by-case against the same live corpus (pre-#325 code passed all 3): the weak-match OR-augmentation buried `opioid-withdrawal-doses` (docRecall@5 → 0; OR recall is append-only at the RPC merge but NOT after re-ranking), and the `selectRetrievalEvidence` pre-clamp tiebreak buried `alcohol-ciwa-threshold` + `clozapine-cbc-abbreviation-threshold` (boost-stacking magnitude re-ordered saturated top-5 sets). Fixed on `claude/retrieval-correctness`: `RAG_TEXT_WEAK_OR_RELAXATION` now defaults to `false` (opt-in experiment flag; re-enable only behind a fresh full golden run) and the selection-layer tiebreak is removed — the pre-clamp deep tiebreak lives in `rankClinicalResults` (below the engineered `rankingTieBreakScore`), which is the only place it is eval-proven. Lesson reinforced: "tie-only by construction" still changes which tied candidate wins; ordering changes inside saturated score regions are behavior changes and need the golden gate. - **UI verification run:** new `tests/ui-universal-search.spec.ts` (grouped typeahead renders, item selection navigates, Enter still runs the mode search — universal endpoint mocked), full `ui-tools`/`ui-tools-task-directory` (40/40) and `ui-smoke`/`ui-overlap` suites against a live dev server in demo mode, plus a live curl of `/api/search/universal` (grouped payload, domain filter, 400 on short query). - **Known limitation:** the typeahead spec mocks the universal endpoint; an end-to-end spec against live seeded registries needs the owner-auth Playwright project (E2E_USER_* keys). diff --git a/src/lib/env.ts b/src/lib/env.ts index 67db76f222..19fcfcab43 100644 --- a/src/lib/env.ts +++ b/src/lib/env.ts @@ -76,11 +76,13 @@ const envSchema = z.object({ RAG_RANKING_CONFIG: z.string().optional(), // P8b extension: when strict-AND text retrieval returns weak-but-nonzero matches (sparse // result set or negligible top text_rank), append OR-relaxed recall behind the strict - // matches. Kill switch for the golden retrieval eval: set false to restore - // relax-only-on-empty behaviour without a code change. + // matches. Default OFF: with it on, the golden retrieval eval measured OR-noise displacing + // the expected document out of top-5 (opioid-withdrawal-doses docRecall@5 1.0 -> 0.0) — + // "append-only" at the RPC merge is not append-only after re-ranking. Opt-in experiment + // flag only; re-enable solely behind a fresh 34/34 golden run. RAG_TEXT_WEAK_OR_RELAXATION: z .enum(["true", "false"]) - .default("true") + .default("false") .transform((value) => value === "true"), RAG_ANSWER_CACHE_TTL_MS: z.coerce.number().int().nonnegative().default(300000), RAG_ANSWER_CACHE_SIZE: z.coerce.number().int().nonnegative().default(100), diff --git a/src/lib/retrieval-selection.ts b/src/lib/retrieval-selection.ts index 870129f762..eaf7bef498 100644 --- a/src/lib/retrieval-selection.ts +++ b/src/lib/retrieval-selection.ts @@ -449,7 +449,6 @@ export function buildRetrievalCandidates( lexicalScore: 0, semanticScore: result.similarity, rerankScore: result.score_explanation?.finalScore ?? result.hybrid_score, - preClampScore: result.score_explanation?.preClampFinalScore, matchedSignals: [], sourceHref: documentCitationHref(citationFromResult(result)), }; @@ -558,11 +557,6 @@ export function selectRetrievalEvidence(args: { if ((right.lexicalScore ?? 0) !== (left.lexicalScore ?? 0)) return (right.lexicalScore ?? 0) - (left.lexicalScore ?? 0); if ((right.rerankScore ?? 0) !== (left.rerankScore ?? 0)) return (right.rerankScore ?? 0) - (left.rerankScore ?? 0); - // rerankScore is the clamped finalScore, which saturates at 1.0 for heavily-boosted results; - // the pre-clamp sum still discriminates within that saturated region. Tie-only by - // construction — every primary signal above has already tied when this fires. - if ((right.preClampScore ?? 0) !== (left.preClampScore ?? 0)) - return (right.preClampScore ?? 0) - (left.preClampScore ?? 0); return left.chunkId.localeCompare(right.chunkId); }); const selectedCandidates: RetrievalCandidate[] = []; diff --git a/src/lib/types.ts b/src/lib/types.ts index 7202600e35..d5f759cedc 100644 --- a/src/lib/types.ts +++ b/src/lib/types.ts @@ -173,9 +173,6 @@ export type RetrievalCandidate = { lexicalScore?: number; semanticScore?: number; rerankScore?: number; - // Deep tiebreak only (see SearchScoreExplanation.preClampFinalScore): discriminates - // candidates whose clamped scores all saturate at 1.0. Never a primary ordering signal. - preClampScore?: number; matchedSignals: string[]; sourceHref?: string; }; diff --git a/tests/retrieval-selection.test.ts b/tests/retrieval-selection.test.ts index 990bc6ed35..380454ec41 100644 --- a/tests/retrieval-selection.test.ts +++ b/tests/retrieval-selection.test.ts @@ -628,7 +628,7 @@ describe("retrieval source selection", () => { }); }); -describe("saturated-score tie-breaking (pre-clamp)", () => { +describe("saturated-score tie-breaking", () => { function saturatedExplanation(preClampFinalScore: number): NonNullable { return { vectorScore: 0.9, @@ -651,10 +651,13 @@ describe("saturated-score tie-breaking (pre-clamp)", () => { }; } - it("orders fully-tied saturated candidates by pre-clamp score, not chunk id", () => { - // Both results are identical on every primary signal (score, lexical, rerank all tie at the - // 1.0 clamp). Without the pre-clamp tiebreak, ordering would fall through to - // chunkId.localeCompare and pick "chunk-a" first; the higher pre-clamp sum must win instead. + it("orders fully-tied saturated candidates by stable chunk id, ignoring pre-clamp boost magnitude", () => { + // Regression guard for the PR #325 golden-eval regression: tie-breaking selection by the + // pre-clamp boost sum re-ordered saturated top-5 sets by boost-stacking magnitude and buried + // golden documents (alcohol-ciwa-threshold and clozapine-cbc-abbreviation-threshold + // docRecall@5 1.0 -> 0.0, measured live 2026-07-07). The pre-clamp deep tiebreak belongs in + // rankClinicalResults (below the engineered rankingTieBreakScore); at the selection layer, + // fully-tied candidates must keep the stable chunk-id order that rankClinicalResults produced. const higherPreClamp = source({ id: "chunk-b", hybrid_score: 1, @@ -676,6 +679,6 @@ describe("saturated-score tie-breaking (pre-clamp)", () => { maxResultsPerDocument: 2, }); - expect(selection.results.map((item) => item.id)).toEqual(["chunk-b", "chunk-a"]); + expect(selection.results.map((item) => item.id)).toEqual(["chunk-a", "chunk-b"]); }); }); From 830671181b6c20af2ffc33c7bc17be04e1bfa4ba Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Tue, 7 Jul 2026 03:27:10 +0800 Subject: [PATCH 2/6] feat(rag): corpus-grounded relevance for the unsupported soft tail (finding #11) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Valid bare topics ("bipolar disorder", "anorexia management") intermittently returned 0 results because the nondeterministic LLM classifier decided the unsupported soft tail. The corpus now decides first, deterministically: - New corpus_topic_term_stats RPC (migration 20260707100000, applied live; service_role-only, read-only) reports per query term: title-topic membership (documents_title_search_idx), chunk-level presence, and scoped corpus size — scoped exactly like retrieval (retrieval_owner_matches + status='indexed' + committed generation). - src/lib/corpus-grounding.ts classifies soft-tail queries: a title-topic anchor under a 5% genericity ceiling (measured live: "management"/"guideline" headline ~18-20% of titles; real topics <=3%) with no corpus-absent token => deterministic reclassification to broad_summary (mirrors an accepted classifier verdict, minus the coin flip); any corpus-absent token => skip the LLM so the soft-tail refusal is deterministic (trigram correction preserved); inconclusive / DB error / unapplied migration => legacy memoized-LLM path. - Scoped strictly to the soft-tail branch: pattern-guarded refusals (DKA, pneumonia, SSRI, consumer noise) and every higher-confidence class are untouched by construction. - Eval controls: golden cases bare-topic-bipolar / bare-topic-anorexia (rank-1 live) and invented-term ragEvalCases unsupported-invented-florbizone / unsupported-invented-quxbyria. Verified live: 4/4 identical runs per query (answer vs refuse), golden retrieval eval 36/36 (failed_cases=0), corpus_grounding telemetry recorded. Co-Authored-By: Claude Fable 5 --- docs/process-hardening.md | 1 + docs/rag-hybrid-findings-and-todo.md | 25 +- scripts/fixtures/rag-retrieval-golden.json | 18 ++ src/lib/corpus-grounding.ts | 152 +++++++++ src/lib/rag-eval-cases.ts | 34 ++ src/lib/rag.ts | 111 ++++++- src/lib/supabase/database.types.ts | 10 + src/lib/types.ts | 7 + ...20260707100000_corpus_topic_term_stats.sql | 72 +++++ supabase/schema.sql | 62 ++++ tests/corpus-grounding.test.ts | 295 ++++++++++++++++++ tests/supabase-schema.test.ts | 17 + tests/universal-search.test.ts | 5 +- 13 files changed, 788 insertions(+), 21 deletions(-) create mode 100644 src/lib/corpus-grounding.ts create mode 100644 supabase/migrations/20260707100000_corpus_topic_term_stats.sql create mode 100644 tests/corpus-grounding.test.ts diff --git a/docs/process-hardening.md b/docs/process-hardening.md index b1ffb080fe..dc4f3eedb1 100644 --- a/docs/process-hardening.md +++ b/docs/process-hardening.md @@ -148,6 +148,7 @@ All approved render-surface modules are extracted. `ClinicalDashboard.tsx` went - **Root cause (confirmed live):** the soft-tail branch of `shouldShortCircuitUnsupportedSearch` (`src/lib/rag.ts`) fires on `analysis.queryClass === "unsupported_or_general"`, and that class is set by `analyzeQueryWithClassifierFallback`, which calls a **generative LLM classifier** (6s timeout, `reasoningEffort:"low"`, uncached) for low-confidence queries. That call is nondeterministic: it reclassifies "bipolar disorder" to a supported class on some runs (→ answered) and declines/times-out on others (→ short-circuited to 0). Reproduced with a same-process ×N probe. - **Why a clean Phase-1 fix does NOT exist:** the deterministic analyzer carries **no signal** distinguishing in-corpus topics from out-of-corpus (bipolar, anorexia, gout, DKA all produce identical `unsupported_or_general`, confidence ≈ 0.40, `canonicalTerms` = query tokens). Only the corpus can tell them apart. Removing the soft-tail short-circuit fixes the valid topics but **regresses `unsupported_correct_rate` 1.0 → 0.79** on `eval:quality --rag-only`: a lexical distinctive-term relevance gate in `chooseAnswerRoute` either over-refuses legitimate semantic/vector matches (whose exact term is not in the retrieved text — e.g. the `rag-routing.test.ts` "admission" fixtures) or under-refuses invented terms ("florbizone syndrome") that score strongly on generic scaffolding words ("syndrome"/"management"). The corpus is broad (gout 13 / crohn 49 / appendicitis 30 / angioplasty 32 / bipolar 719 chunks), so almost no common medical term is truly absent — grounded low-confidence answers, not fabrication, are the realistic worst case for real terms; only genuinely invented terms should refuse. - **Decision (2026-07-03):** the risky change (soft-tail removal + relevance gate + invented-term eval controls) was **reverted**; only a safe, independent hardening was kept — `fetchEnabledRagAliases` no longer caches `[]` on a transient `rag_aliases` read error (which would suppress alias expansion for the whole TTL). Finding #11 is **re-scoped into RAG optimisation Phase 2** ("fit retrieval to content"), where it should be fixed with corpus-grounded relevance — IDF/corpus-frequency weighting of query terms and/or the semantic relevance model, plus the data-driven query-understanding vocabulary (RC6) so the deterministic classifier recognises in-corpus topics up front. Any gate must keep `eval:quality --rag-only` `unsupported_correct_rate` at 1.0 (add invented-term controls like "florbizone syndrome management" / "quxbyria disorder treatment" once it can pass them) while letting valid bare topics answer. A cheaper interim option worth measuring first: **classifier-verdict memoization** to at least make the current behaviour deterministic per query. +- **RESOLVED (2026-07-07, `claude/retrieval-correctness`):** the Phase-2 corpus-grounded fix shipped. `corpus_topic_term_stats` (migration `20260707100000`, applied live) reports per-term title-topic membership (with a measured 5% genericity ceiling — "management"/"guideline" headline ~18–20% of titles and are scaffolding; real topics ≤3%) and chunk-level presence, scoped with the exact retrieval `owner_filter`. `src/lib/corpus-grounding.ts` + the `analyzeQueryWithClassifierFallback` hook classify only the soft-tail branch (pattern-guarded refusals and higher-confidence classes untouched): in-corpus bare topics deterministically reclassify to `broad_summary` and answer; corpus-absent queries skip the LLM and refuse deterministically (trigram correction still runs); inconclusive keeps memoized-LLM behaviour; DB errors fail open. Verified live: "bipolar disorder" / "anorexia management" answer 4/4 runs with the right document at rank 1 (new golden cases `bare-topic-bipolar` / `bare-topic-anorexia`); "florbizone syndrome management" / "quxbyria disorder treatment" refuse 4/4 runs (new `ragEvalCases` controls); golden eval 36/36. ## Retrieval changes must pass the golden eval before merge (2026-07-03) diff --git a/docs/rag-hybrid-findings-and-todo.md b/docs/rag-hybrid-findings-and-todo.md index 0794142ef0..358083b156 100644 --- a/docs/rag-hybrid-findings-and-todo.md +++ b/docs/rag-hybrid-findings-and-todo.md @@ -199,16 +199,21 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows `typoCorrected` flag; only fires for would-be-unsupported queries so no hot-path cost). Rescues typo queries whose corrected form is a _supported_ class (e.g. a typo'd clozapine/dose query → table_threshold). Golden 23/23 unchanged, 682 tests pass. - - ⚠️ **Pre-existing bug surfaced (NEW, finding #11):** unsupported-classified queries retrieve - **nondeterministically** — the _same_ query in the _same_ process alternates - `unsupported_short_circuit` (0 results) vs `text_fast_path`/`hybrid` (real results), e.g. - "anorexia management" (no typo). Classification is pure and all caches honour `skipCache`, so the - variance is elsewhere in the unsupported-query path (candidate: alias fetch/expansion or an async - step) — needs runtime instrumentation to pin. It masks the benefit above (a typo query whose - corrected form is ALSO borderline-unsupported, like "schizophrenai management", inherits the - flakiness). Confined to unsupported queries (golden set never hits it), so it never affected the - committed metrics. High-priority to fix — it means some valid clinical topics ("bipolar disorder", - "anorexia management") intermittently return nothing. + - ✅ **Finding #11 FIXED (2026-07-07) — corpus-grounded relevance.** Root cause was the + nondeterministic LLM classifier deciding the unsupported soft tail (see + docs/process-hardening.md 2026-07-03 entry). Two-part fix: PR #325's classifier-verdict + memoization (interim determinism per query per 15-min TTL), then the Phase-2 fix on + `claude/retrieval-correctness`: `corpus_topic_term_stats` (migration `20260707100000`, + applied live) + `src/lib/corpus-grounding.ts` classify soft-tail queries against the + corpus's own topic vocabulary (title-tsvector matches under a 5% genericity ceiling; + chunk-absence = invented term) BEFORE any LLM call. In-corpus bare topics ("bipolar + disorder", "anorexia management") deterministically reclassify to `broad_summary` and + answer (verified live: 4/4 identical runs, docs at rank 1); corpus-absent queries + ("florbizone syndrome management", "quxbyria disorder treatment") skip the LLM and refuse + deterministically, with the trigram-correction escape hatch preserved for typos. + Invented-term controls added to `ragEvalCases`; bare-topic golden cases added + (`bare-topic-bipolar`, `bare-topic-anorexia`). Inconclusive verdicts (e.g. "gout + management" — chunk-present but no title topic) keep the legacy memoized-LLM behaviour. - ⏳ Still hard-coded (lower priority now the trigram path exists): moving `synonymGroups` / `domainAliasGroups` / `medicationAliasGroups` into `rag_aliases`; generalising the special-case rewrites off `RagQueryClass`. diff --git a/scripts/fixtures/rag-retrieval-golden.json b/scripts/fixtures/rag-retrieval-golden.json index 34b5d4f368..cc9c809fbb 100644 --- a/scripts/fixtures/rag-retrieval-golden.json +++ b/scripts/fixtures/rag-retrieval-golden.json @@ -327,5 +327,23 @@ "topK": 8, "expectTableEvidence": false, "forceEmbedding": true + }, + { + "id": "bare-topic-bipolar", + "query": "bipolar disorder", + "expectedQueryClass": "broad_summary", + "expectedDocumentSubstrings": ["Bipolar"], + "expectedContentTerms": [["bipolar", "mania", "manic", "mood"]], + "topK": 8, + "expectTableEvidence": false + }, + { + "id": "bare-topic-anorexia", + "query": "anorexia management", + "expectedQueryClass": "broad_summary", + "expectedDocumentSubstrings": ["Anorexia"], + "expectedContentTerms": [["anorexia", "eating", "weight"]], + "topK": 8, + "expectTableEvidence": false } ] diff --git a/src/lib/corpus-grounding.ts b/src/lib/corpus-grounding.ts new file mode 100644 index 0000000000..25e2a6e5d0 --- /dev/null +++ b/src/lib/corpus-grounding.ts @@ -0,0 +1,152 @@ +import type { createAdminClient } from "@/lib/supabase/admin"; +import type { CorpusGroundingVerdict } from "@/lib/types"; +import { normalizedClinicalSearchTokens } from "@/lib/clinical-search"; + +// Finding #11 (corpus-grounded relevance): the deterministic query analyzer cannot tell an +// in-corpus bare topic ("bipolar disorder") from an invented one ("florbizone syndrome +// management") — both land in the unsupported soft tail with identical confidence, and the LLM +// classifier fallback answers or refuses nondeterministically. Only the corpus can separate +// them, so this module classifies the query's content tokens against corpus statistics served +// by the `corpus_topic_term_stats` RPC (scoped exactly like retrieval): +// +// * a token is ABSENT when no committed chunk has ever seen it (and no title matches) — the +// signature of invented/unknown terms. Any absent token => "out_of_corpus". Typos are also +// absent, and stay rescuable: the caller skips the LLM but the downstream short-circuit +// still runs trigram correction before giving up. +// * a token is a TOPIC ANCHOR when it matches at least one indexed document title AND its +// title share stays under the genericity ceiling. Title words that headline a large share +// of the corpus ("management" ~18%, "guideline" ~20% of titles, measured live 2026-07-07) +// are scaffolding, not topics; real topics measure far lower (assessment 3.0%, disorder +// 1.6%, bipolar/anorexia <0.1%). No absent tokens + at least one anchor => "in_corpus_topic". +// * anything else => "inconclusive", which callers must treat as "behave exactly as before" +// (LLM classifier fallback + memoization). DB errors, demo mode, and a missing RPC also +// fail open to "inconclusive" so this can never take retrieval down. +export type { CorpusGroundingVerdict }; + +export type CorpusGroundingResult = { + verdict: CorpusGroundingVerdict; + anchorTerms: string[]; + absentTerms: string[]; +}; + +export type CorpusTopicTermStats = { + term: string; + has_ts_signal: boolean; + title_doc_count: number; + chunk_present: boolean; + total_doc_count: number; +}; + +// Title-share above this is corpus scaffolding ("management" 18.2%, "guideline" 19.6% measured +// live), well clear of the largest measured real topic share ("assessment" 3.0%). +const topicGenericityCeiling = 0.05; +const maxGroundingTerms = 8; +const termStatsCacheTtlMs = 10 * 60 * 1000; +const termStatsCacheMaxEntries = 1024; + +const termStatsCache = new Map(); + +export function resetCorpusGroundingCacheForTests() { + termStatsCache.clear(); +} + +function cacheKey(ownerScopeKey: string, term: string) { + return `${ownerScopeKey}|${term}`; +} + +function readCachedStats(ownerScopeKey: string, term: string): CorpusTopicTermStats | null { + const cached = termStatsCache.get(cacheKey(ownerScopeKey, term)); + if (!cached) return null; + if (cached.expiresAt <= Date.now()) { + termStatsCache.delete(cacheKey(ownerScopeKey, term)); + return null; + } + return cached.stats; +} + +function storeCachedStats(ownerScopeKey: string, stats: CorpusTopicTermStats) { + if (termStatsCache.size >= termStatsCacheMaxEntries) { + const oldestKey = termStatsCache.keys().next().value; + if (oldestKey !== undefined) termStatsCache.delete(oldestKey); + } + termStatsCache.set(cacheKey(ownerScopeKey, stats.term), { + expiresAt: Date.now() + termStatsCacheTtlMs, + stats, + }); +} + +export function corpusGroundingTerms(query: string): string[] { + const seen = new Set(); + const terms: string[] = []; + for (const token of normalizedClinicalSearchTokens(query)) { + if (/^\d+$/.test(token)) continue; + if (seen.has(token)) continue; + seen.add(token); + terms.push(token); + if (terms.length >= maxGroundingTerms) break; + } + return terms; +} + +export function classifyCorpusGroundingFromStats(stats: CorpusTopicTermStats[]): CorpusGroundingResult { + // Stopword-ish tokens stem to an empty tsquery and carry no corpus signal either way. + const signals = stats.filter((entry) => entry.has_ts_signal); + if (signals.length === 0) return { verdict: "inconclusive", anchorTerms: [], absentTerms: [] }; + + const totalDocs = signals[0]?.total_doc_count ?? 0; + if (totalDocs <= 0) return { verdict: "inconclusive", anchorTerms: [], absentTerms: [] }; + + const absentTerms = signals + .filter((entry) => !entry.chunk_present && entry.title_doc_count === 0) + .map((entry) => entry.term); + const anchorTerms = signals + .filter((entry) => entry.title_doc_count >= 1 && entry.title_doc_count / totalDocs <= topicGenericityCeiling) + .map((entry) => entry.term); + + if (absentTerms.length > 0) return { verdict: "out_of_corpus", anchorTerms, absentTerms }; + if (anchorTerms.length > 0) return { verdict: "in_corpus_topic", anchorTerms, absentTerms }; + return { verdict: "inconclusive", anchorTerms, absentTerms }; +} + +export async function classifyCorpusGrounding(args: { + supabase: ReturnType; + query: string; + // The exact owner_filter retrieval will use (null = unscoped, zero-UUID = public docs only). + ownerFilter: string | null; +}): Promise { + const terms = corpusGroundingTerms(args.query); + if (terms.length === 0) return { verdict: "inconclusive", anchorTerms: [], absentTerms: [] }; + + const ownerScopeKey = args.ownerFilter ?? "unscoped"; + const stats: CorpusTopicTermStats[] = []; + const missing: string[] = []; + for (const term of terms) { + const cached = readCachedStats(ownerScopeKey, term); + if (cached) stats.push(cached); + else missing.push(term); + } + + if (missing.length > 0) { + try { + const { data, error } = await args.supabase.rpc("corpus_topic_term_stats", { + terms: missing, + owner_filter: args.ownerFilter, + }); + if (error) throw error; + const rows = (data ?? []) as CorpusTopicTermStats[]; + // A term the RPC did not echo back got dropped SQL-side (blank after trim); treat the + // whole classification as inconclusive rather than guessing. + if (rows.length !== missing.length) return { verdict: "inconclusive", anchorTerms: [], absentTerms: [] }; + for (const row of rows) { + storeCachedStats(ownerScopeKey, row); + stats.push(row); + } + } catch { + // Fail open: missing RPC (migration not applied), transient DB error, demo mode — the + // caller keeps today's behaviour (LLM classifier fallback + soft-tail short-circuit). + return { verdict: "inconclusive", anchorTerms: [], absentTerms: [] }; + } + } + + return classifyCorpusGroundingFromStats(stats); +} diff --git a/src/lib/rag-eval-cases.ts b/src/lib/rag-eval-cases.ts index a2f612a459..d1d061d44d 100644 --- a/src/lib/rag-eval-cases.ts +++ b/src/lib/rag-eval-cases.ts @@ -1197,6 +1197,40 @@ export const ragEvalCases: RagEvalCase[] = [ minCitations: 0, latencyTargetMs: 2000, }, + // Finding #11 invented-term controls (docs/process-hardening.md): bare topic-shaped queries + // built from terms the corpus has NEVER seen. Corpus grounding must classify them + // out_of_corpus (the invented head noun is chunk-absent) and refuse deterministically — + // without the LLM classifier lottery that used to decide these. The scaffolding words + // ("syndrome", "disorder", "treatment", "management") are all corpus-present, so these also + // prove that generic-word presence alone cannot rescue an invented topic. + { + id: "unsupported-invented-florbizone", + question: "florbizone syndrome management", + category: "unsupported", + suite: "false_positive", + relevanceGrade: "unsupported", + expectedQueryClass: "unsupported_or_general", + falsePositiveControl: true, + supported: false, + expectedFiles: [], + allowedRoutes: ["unsupported"], + minCitations: 0, + latencyTargetMs: 4000, + }, + { + id: "unsupported-invented-quxbyria", + question: "quxbyria disorder treatment", + category: "unsupported", + suite: "false_positive", + relevanceGrade: "unsupported", + expectedQueryClass: "unsupported_or_general", + falsePositiveControl: true, + supported: false, + expectedFiles: [], + allowedRoutes: ["unsupported"], + minCitations: 0, + latencyTargetMs: 4000, + }, ]; export function selectRagEvalCases(args: { limit?: number; question?: string }) { diff --git a/src/lib/rag.ts b/src/lib/rag.ts index ccfdf32f0f..fdfea34fab 100644 --- a/src/lib/rag.ts +++ b/src/lib/rag.ts @@ -1,5 +1,6 @@ import { createAdminClient } from "@/lib/supabase/admin"; import { requireOwnerScope, retrievalOwnerFilter } from "@/lib/owner-scope"; +import { classifyCorpusGrounding } from "@/lib/corpus-grounding"; import type { Database, Json } from "@/lib/supabase/database.types"; import { embedTextWithTelemetry, @@ -98,6 +99,7 @@ import type { Citation, ConflictOrGap, ClinicalQueryAnalysis, + CorpusGroundingVerdict, DocumentIndexQuality, DocumentIndexUnitMatch, DocumentMemoryCard, @@ -362,6 +364,9 @@ export type SearchTelemetry = { // long-standing relax-on-zero path; "weak_augment" appends OR recall behind weak-but-nonzero // strict matches (issue: strict-AND could bury the right chunk without ever relaxing). text_or_relaxation_used?: "none" | "empty_fallback" | "weak_augment"; + // Finding #11: corpus-grounding verdict for unsupported-soft-tail queries (absent when the + // query never entered the soft tail). See src/lib/corpus-grounding.ts. + corpus_grounding?: CorpusGroundingVerdict; // RC9 observability: how many final results carry a fabricated (non-cosine) similarity. synthetic_similarity_count?: number; embedding_skipped: boolean; @@ -1320,7 +1325,16 @@ function applyClassifierVerdict(analysis: ClinicalQueryAnalysis, parsed: Classif } satisfies ClinicalQueryAnalysis; } -export async function analyzeQueryWithClassifierFallback(query: string, analysis: ClinicalQueryAnalysis) { +export async function analyzeQueryWithClassifierFallback( + query: string, + analysis: ClinicalQueryAnalysis, + opts?: { + // Finding #11 corpus grounding: when provided, unsupported-soft-tail queries are checked + // against the corpus BEFORE the nondeterministic LLM classifier. Scoped with the exact + // owner_filter retrieval will use so grounding can never see documents retrieval cannot. + corpusGrounding?: { supabase: ReturnType; ownerFilter: string | null }; + }, +) { if ( // Fail closed before any generative model call: an adversarial-manipulation // query is routed to "unsupported" downstream, so never send its text to the @@ -1332,6 +1346,46 @@ export async function analyzeQueryWithClassifierFallback(query: string, analysis ) { return { ...analysis, needsClassifierFallback: false } satisfies ClinicalQueryAnalysis; } + + // Finding #11 corpus-grounded relevance: for queries that would hit the unsupported soft + // tail, the corpus — not the LLM — decides. An in-corpus bare topic ("bipolar disorder") + // deterministically reclassifies to broad_summary (mirroring what an accepted classifier + // verdict would have done, minus the coin flip); a corpus-absent query ("florbizone syndrome + // management") skips the LLM entirely so the soft-tail refusal is deterministic — and typos + // remain rescuable because the short-circuit path still runs trigram correction afterwards. + // "inconclusive" (including DB errors and an unapplied migration) keeps legacy behaviour. + // This deliberately runs before the OPENAI_API_KEY gate: offline/source-only deployments + // still retrieve lexically, so in-corpus bare topics should answer there too. + if (opts?.corpusGrounding && isUnsupportedSoftTailAnalysis(query, analysis)) { + const grounding = await classifyCorpusGrounding({ + supabase: opts.corpusGrounding.supabase, + query, + ownerFilter: opts.corpusGrounding.ownerFilter, + }); + if (grounding.verdict === "in_corpus_topic") { + return { + ...analysis, + queryClass: "broad_summary", + confidence: Math.max(analysis.confidence, 0.62), + needsSynthesis: true, + needsClassifierFallback: false, + corpusGrounding: "in_corpus_topic", + reasons: uniqueTextValues([...analysis.reasons, "corpus_topic_grounding"], 12), + } satisfies ClinicalQueryAnalysis; + } + if (grounding.verdict === "out_of_corpus") { + // Do NOT touch queryClass/confidence/reasons: the existing soft-tail short-circuit (and + // its alias-expansion + trigram-correction escape hatches) must keep firing exactly as + // before — only the LLM lottery is removed. + return { + ...analysis, + needsClassifierFallback: false, + corpusGrounding: "out_of_corpus", + } satisfies ClinicalQueryAnalysis; + } + analysis = { ...analysis, corpusGrounding: "inconclusive" }; + } + if (!analysis.needsClassifierFallback || !env.OPENAI_API_KEY) return analysis; const memoKey = classifierVerdictMemoKey(query, analysis); @@ -1360,16 +1414,35 @@ export async function analyzeQueryWithClassifierFallback(query: string, analysis } } -function shouldShortCircuitUnsupportedSearch(query: string, analysis: ClinicalQueryAnalysis) { - if (unavailableDocumentNoisePattern.test(query)) return true; - if (clearlyOutsideCorpusMedicalPattern.test(query) && analysis.documentTitleTerms.length === 0) return true; +// Shared eligibility gate for the unsupported soft tail: a low-signal unsupported_or_general +// analysis with no title/medication/threshold intent and no non-default reasons. +function unsupportedSoftTailEligible(analysis: ClinicalQueryAnalysis) { if (analysis.queryClass !== "unsupported_or_general") return false; if (analysis.documentTitleIntent || analysis.medications.length || analysis.thresholdTerms.length) return false; if (analysis.reasons.some((reason) => reason !== "no_specific_rag_class_terms")) return false; + return true; +} + +function shouldShortCircuitUnsupportedSearch(query: string, analysis: ClinicalQueryAnalysis) { + if (unavailableDocumentNoisePattern.test(query)) return true; + if (clearlyOutsideCorpusMedicalPattern.test(query) && analysis.documentTitleTerms.length === 0) return true; + if (!unsupportedSoftTailEligible(analysis)) return false; if (clearlyNonClinicalConsumerPattern.test(query)) return true; return analysis.confidence <= 0.42 && analysis.expandedTerms.length <= 5; } +// True only for queries that would short-circuit via the soft tail itself — NOT via the +// pattern guards (out-of-corpus medical list, consumer-noise, unavailable-document noise). +// Corpus grounding is scoped to exactly this branch: the pattern-guarded refusals and every +// higher-confidence class keep their existing behaviour untouched. +function isUnsupportedSoftTailAnalysis(query: string, analysis: ClinicalQueryAnalysis) { + if (unavailableDocumentNoisePattern.test(query)) return false; + if (clearlyOutsideCorpusMedicalPattern.test(query) && analysis.documentTitleTerms.length === 0) return false; + if (!unsupportedSoftTailEligible(analysis)) return false; + if (clearlyNonClinicalConsumerPattern.test(query)) return false; + return analysis.confidence <= 0.42 && analysis.expandedTerms.length <= 5; +} + function scopeKey(args: Pick) { const scope = args.documentIds?.length ? [...args.documentIds].sort().join(",") @@ -5553,7 +5626,29 @@ export async function searchChunksWithTelemetry(args: SearchChunksArgs) { const memoryCardCache: MemoryCardCache = new Map(); const retrievalQuery = queryForClinicalMode(args.query, args.queryMode ?? "auto"); const modeQueryClass = queryClassForClinicalMode(args.queryMode ?? "auto"); - const queryAnalysis = await analyzeQueryWithClassifierFallback(retrievalQuery, analyzeClinicalQuery(retrievalQuery)); + const documentFilterList = args.documentIds?.length + ? args.documentIds + : args.documentId + ? [args.documentId] + : undefined; + // Finding #11: give the classifier fallback the exact owner scope retrieval will use, so + // corpus grounding sees the same corpus. If owner-scope derivation throws (anonymous prod + // call without allowGlobalSearch), grounding is skipped and the retrieval path below raises + // the proper owner-scope error itself. + const corpusGroundingScope = (() => { + try { + return { + supabase, + ownerFilter: + ownerScopeForDocumentFilteredRetrieval(args.ownerId, documentFilterList, args.allowGlobalSearch) ?? null, + }; + } catch { + return undefined; + } + })(); + const queryAnalysis = await analyzeQueryWithClassifierFallback(retrievalQuery, analyzeClinicalQuery(retrievalQuery), { + corpusGrounding: corpusGroundingScope, + }); throwIfAborted(args.signal); if (modeQueryClass) queryAnalysis.queryClass = modeQueryClass; const queryClassification = { @@ -5561,11 +5656,6 @@ export async function searchChunksWithTelemetry(args: SearchChunksArgs) { confidence: queryAnalysis.confidence, reasons: queryAnalysis.reasons, }; - const documentFilterList = args.documentIds?.length - ? args.documentIds - : args.documentId - ? [args.documentId] - : undefined; const telemetry: SearchTelemetry = { search_cache_hit: false, query_class: queryClassification.queryClass, @@ -5605,6 +5695,7 @@ export async function searchChunksWithTelemetry(args: SearchChunksArgs) { weighted_top_score: 0, rrf_top_score: 0, }; + if (queryAnalysis.corpusGrounding) telemetry.corpus_grounding = queryAnalysis.corpusGrounding; const ragAliases = await fetchEnabledRagAliases(supabase, args.ownerId); const ragAliasExpansions = selectRagAliasExpansions(retrievalQuery, ragAliases); diff --git a/src/lib/supabase/database.types.ts b/src/lib/supabase/database.types.ts index 55db0bd383..84252d0e3f 100644 --- a/src/lib/supabase/database.types.ts +++ b/src/lib/supabase/database.types.ts @@ -2105,6 +2105,16 @@ export type Database = { retry_after_seconds: number; }[]; }; + corpus_topic_term_stats: { + Args: { terms: string[]; owner_filter?: string | null }; + Returns: { + term: string; + has_ts_signal: boolean; + title_doc_count: number; + chunk_present: boolean; + total_doc_count: number; + }[]; + }; correct_clinical_query_terms: { Args: { input_query: string; min_sim?: number }; Returns: string; diff --git a/src/lib/types.ts b/src/lib/types.ts index d5f759cedc..41f649ddd7 100644 --- a/src/lib/types.ts +++ b/src/lib/types.ts @@ -635,6 +635,12 @@ export type ClinicalQueryIntent = | "broad_summary" | "general"; +// Finding #11 corpus grounding: how the corpus classified an unsupported-soft-tail query. +// "in_corpus_topic" deterministically reclassifies to broad_summary; "out_of_corpus" skips the +// LLM classifier so the soft-tail refusal is deterministic; "inconclusive" keeps legacy +// behaviour (LLM classifier fallback). Absent = the query never entered the soft tail. +export type CorpusGroundingVerdict = "in_corpus_topic" | "out_of_corpus" | "inconclusive"; + export type ClinicalQueryAnalysis = { originalQuery: string; normalizedQuery: string; @@ -642,6 +648,7 @@ export type ClinicalQueryAnalysis = { intent: ClinicalQueryIntent; confidence: number; reasons: string[]; + corpusGrounding?: CorpusGroundingVerdict; canonicalTerms: string[]; expandedTerms: string[]; typoCorrections: Array<{ from: string; to: string }>; diff --git a/supabase/migrations/20260707100000_corpus_topic_term_stats.sql b/supabase/migrations/20260707100000_corpus_topic_term_stats.sql new file mode 100644 index 0000000000..0ce70a08fe --- /dev/null +++ b/supabase/migrations/20260707100000_corpus_topic_term_stats.sql @@ -0,0 +1,72 @@ +-- Finding #11 (corpus-grounded relevance): deterministic in/out-of-corpus signal for the +-- unsupported-query soft tail. The deterministic query analyzer carries no signal separating +-- in-corpus bare topics ("bipolar disorder") from out-of-corpus or invented ones ("florbizone +-- syndrome management") — only the corpus can tell them apart, and the LLM classifier fallback +-- is nondeterministic. This function reports, per query term and scoped exactly like retrieval +-- (retrieval_owner_matches + status = 'indexed' + committed generation): +-- * has_ts_signal — whether the term survives to_tsquery stemming/stopwording at all +-- (a stopword like "the" produces an empty tsquery and must be ignored, +-- not treated as corpus-absent); +-- * title_doc_count — how many indexed documents match the term in their title tsvector +-- (the corpus's own topic vocabulary; served by documents_title_search_idx); +-- * chunk_present — whether ANY committed chunk matches the term (absence = the corpus has +-- never seen the term, the refusal signal for invented terms); +-- * total_doc_count — scoped corpus size, so callers can derive a genericity share +-- (e.g. "management" titles ~18% of docs = scaffolding, not a topic). +-- Read-only, additive, service_role-only. App-side consumer: src/lib/corpus-grounding.ts. + +create or replace function public.corpus_topic_term_stats( + terms text[], + owner_filter uuid default null +) +returns table ( + term text, + has_ts_signal boolean, + title_doc_count integer, + chunk_present boolean, + total_doc_count integer +) +language sql +stable +set search_path = public, extensions, pg_temp +as $$ + with input_terms as ( + select distinct lower(btrim(t.term)) as term + from unnest(coalesce(terms, array[]::text[])) with ordinality as t(term, ord) + where btrim(t.term) <> '' + and t.ord <= 8 + ), + totals as ( + select count(*)::integer as total_doc_count + from public.documents d + where public.retrieval_owner_matches(owner_filter, d.owner_id) + and d.status = 'indexed' + ) + select + it.term, + plainto_tsquery('english', it.term) <> ''::tsquery as has_ts_signal, + ( + select count(*)::integer + from public.documents d + where public.retrieval_owner_matches(owner_filter, d.owner_id) + and d.status = 'indexed' + and d.title_search_tsv @@ plainto_tsquery('english', it.term) + ) as title_doc_count, + exists ( + select 1 + from public.document_chunks c + join public.documents d on d.id = c.document_id + where public.retrieval_owner_matches(owner_filter, d.owner_id) + and d.status = 'indexed' + and public.is_committed_document_generation(c.index_generation_id, d.metadata) + and c.search_tsv @@ plainto_tsquery('english', it.term) + ) as chunk_present, + totals.total_doc_count + from input_terms it + cross join totals; +$$; + +revoke all on function public.corpus_topic_term_stats(text[], uuid) from public; +revoke all on function public.corpus_topic_term_stats(text[], uuid) from anon; +revoke all on function public.corpus_topic_term_stats(text[], uuid) from authenticated; +grant execute on function public.corpus_topic_term_stats(text[], uuid) to service_role; diff --git a/supabase/schema.sql b/supabase/schema.sql index e01272b398..77bb96eebd 100644 --- a/supabase/schema.sql +++ b/supabase/schema.sql @@ -1917,6 +1917,68 @@ as $$ end; $$; +-- Finding #11 (corpus-grounded relevance): deterministic in/out-of-corpus signal for the +-- unsupported-query soft tail. Reports, per query term and scoped exactly like retrieval, +-- whether the term stems to a usable tsquery, how many indexed document titles match it (the +-- corpus's own topic vocabulary), whether any committed chunk has ever seen it, and the scoped +-- corpus size for genericity shares. Read-only, additive, service_role-only. +-- App-side consumer: src/lib/corpus-grounding.ts. Migration: 20260707100000. +create or replace function public.corpus_topic_term_stats( + terms text[], + owner_filter uuid default null +) +returns table ( + term text, + has_ts_signal boolean, + title_doc_count integer, + chunk_present boolean, + total_doc_count integer +) +language sql +stable +set search_path = public, extensions, pg_temp +as $$ + with input_terms as ( + select distinct lower(btrim(t.term)) as term + from unnest(coalesce(terms, array[]::text[])) with ordinality as t(term, ord) + where btrim(t.term) <> '' + and t.ord <= 8 + ), + totals as ( + select count(*)::integer as total_doc_count + from public.documents d + where public.retrieval_owner_matches(owner_filter, d.owner_id) + and d.status = 'indexed' + ) + select + it.term, + plainto_tsquery('english', it.term) <> ''::tsquery as has_ts_signal, + ( + select count(*)::integer + from public.documents d + where public.retrieval_owner_matches(owner_filter, d.owner_id) + and d.status = 'indexed' + and d.title_search_tsv @@ plainto_tsquery('english', it.term) + ) as title_doc_count, + exists ( + select 1 + from public.document_chunks c + join public.documents d on d.id = c.document_id + where public.retrieval_owner_matches(owner_filter, d.owner_id) + and d.status = 'indexed' + and public.is_committed_document_generation(c.index_generation_id, d.metadata) + and c.search_tsv @@ plainto_tsquery('english', it.term) + ) as chunk_present, + totals.total_doc_count + from input_terms it + cross join totals; +$$; + +revoke all on function public.corpus_topic_term_stats(text[], uuid) from public; +revoke all on function public.corpus_topic_term_stats(text[], uuid) from anon; +revoke all on function public.corpus_topic_term_stats(text[], uuid) from authenticated; +grant execute on function public.corpus_topic_term_stats(text[], uuid) to service_role; + create or replace function public.match_document_chunks( query_embedding extensions.vector(1536), match_count integer default 8, diff --git a/tests/corpus-grounding.test.ts b/tests/corpus-grounding.test.ts new file mode 100644 index 0000000000..3a1e6dac58 --- /dev/null +++ b/tests/corpus-grounding.test.ts @@ -0,0 +1,295 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import type { CorpusTopicTermStats } from "../src/lib/corpus-grounding"; + +// Finding #11 corpus-grounded relevance: the corpus — not the LLM classifier lottery — decides +// whether an unsupported-soft-tail query is an in-corpus bare topic (answer), an +// invented/out-of-corpus query (refuse deterministically), or inconclusive (legacy behaviour). + +afterEach(() => { + vi.unstubAllEnvs(); + vi.resetModules(); +}); + +function stats(overrides: Partial & { term: string }): CorpusTopicTermStats { + return { + has_ts_signal: true, + title_doc_count: 0, + chunk_present: true, + total_doc_count: 2000, + ...overrides, + }; +} + +describe("classifyCorpusGroundingFromStats", () => { + async function load() { + return import("../src/lib/corpus-grounding"); + } + + it("classifies a bare in-corpus topic: title anchor present, nothing absent", async () => { + const { classifyCorpusGroundingFromStats } = await load(); + // "bipolar disorder" — measured live: bipolar 1 title, disorder 33/2065 titles (1.6%). + const result = classifyCorpusGroundingFromStats([ + stats({ term: "bipolar", title_doc_count: 1 }), + stats({ term: "disorder", title_doc_count: 33 }), + ]); + expect(result.verdict).toBe("in_corpus_topic"); + expect(result.anchorTerms).toEqual(["bipolar", "disorder"]); + }); + + it("treats corpus-ubiquitous title words as scaffolding, not topics", async () => { + const { classifyCorpusGroundingFromStats } = await load(); + // "management guideline" — management headlines ~18% of titles, guideline ~20%; neither is + // a topic anchor, so presence alone must not rescue the query. + const result = classifyCorpusGroundingFromStats([ + stats({ term: "management", title_doc_count: 375 }), + stats({ term: "guideline", title_doc_count: 405 }), + ]); + expect(result.verdict).toBe("inconclusive"); + expect(result.anchorTerms).toEqual([]); + }); + + it("refuses when any term is corpus-absent, even next to a real anchor", async () => { + const { classifyCorpusGroundingFromStats } = await load(); + // "florbizone syndrome management" — syndrome IS a title anchor (12 titles), but the + // invented head noun has never been seen by any chunk: absent always vetoes. + const result = classifyCorpusGroundingFromStats([ + stats({ term: "florbizone", chunk_present: false, title_doc_count: 0 }), + stats({ term: "syndrome", title_doc_count: 12 }), + stats({ term: "management", title_doc_count: 375 }), + ]); + expect(result.verdict).toBe("out_of_corpus"); + expect(result.absentTerms).toEqual(["florbizone"]); + }); + + it("is inconclusive for chunk-present terms with no title topic (no gout guideline)", async () => { + const { classifyCorpusGroundingFromStats } = await load(); + const result = classifyCorpusGroundingFromStats([ + stats({ term: "gout", title_doc_count: 0, chunk_present: true }), + stats({ term: "management", title_doc_count: 375 }), + ]); + expect(result.verdict).toBe("inconclusive"); + }); + + it("ignores tokens that stem to an empty tsquery instead of calling them absent", async () => { + const { classifyCorpusGroundingFromStats } = await load(); + const result = classifyCorpusGroundingFromStats([ + stats({ term: "the", has_ts_signal: false, chunk_present: false }), + ]); + expect(result.verdict).toBe("inconclusive"); + }); + + it("is inconclusive when the scoped corpus is empty", async () => { + const { classifyCorpusGroundingFromStats } = await load(); + const result = classifyCorpusGroundingFromStats([ + stats({ term: "bipolar", title_doc_count: 1, total_doc_count: 0 }), + ]); + expect(result.verdict).toBe("inconclusive"); + }); +}); + +describe("corpusGroundingTerms", () => { + it("drops stopwords and numerals, dedupes, and caps the term list", async () => { + const { corpusGroundingTerms } = await load(); + expect(corpusGroundingTerms("what is the bipolar disorder 2027 bipolar")).toEqual(["bipolar", "disorder"]); + const many = corpusGroundingTerms("alpha bravo charlie delta echo foxtrot golf hotel india juliet kilo"); + expect(many.length).toBeLessThanOrEqual(8); + }); + + async function load() { + return import("../src/lib/corpus-grounding"); + } +}); + +describe("classifyCorpusGrounding (RPC + cache)", () => { + async function load() { + return import("../src/lib/corpus-grounding"); + } + + function fakeSupabase(rows: CorpusTopicTermStats[] | (() => CorpusTopicTermStats[])) { + const rpc = vi.fn(async (_fn: string, args: { terms: string[] }) => { + const all = typeof rows === "function" ? rows() : rows; + return { data: all.filter((row) => args.terms.includes(row.term)), error: null }; + }); + return { client: { rpc } as never, rpc }; + } + + it("caches per-term stats so a repeated query does not re-query the corpus", async () => { + const { classifyCorpusGrounding, resetCorpusGroundingCacheForTests } = await load(); + resetCorpusGroundingCacheForTests(); + const { client, rpc } = fakeSupabase([ + stats({ term: "bipolar", title_doc_count: 1 }), + stats({ term: "disorder", title_doc_count: 33 }), + ]); + + const first = await classifyCorpusGrounding({ supabase: client, query: "bipolar disorder", ownerFilter: null }); + const second = await classifyCorpusGrounding({ supabase: client, query: "bipolar disorder", ownerFilter: null }); + + expect(first.verdict).toBe("in_corpus_topic"); + expect(second).toEqual(first); + expect(rpc).toHaveBeenCalledTimes(1); + }); + + it("fails open to inconclusive on RPC errors (missing migration, transient DB failure)", async () => { + const { classifyCorpusGrounding, resetCorpusGroundingCacheForTests } = await load(); + resetCorpusGroundingCacheForTests(); + const rpc = vi.fn(async () => ({ data: null, error: new Error("function does not exist") })); + + const result = await classifyCorpusGrounding({ + supabase: { rpc } as never, + query: "bipolar disorder", + ownerFilter: null, + }); + expect(result.verdict).toBe("inconclusive"); + }); + + it("scopes the cache by owner filter", async () => { + const { classifyCorpusGrounding, resetCorpusGroundingCacheForTests } = await load(); + resetCorpusGroundingCacheForTests(); + const { client, rpc } = fakeSupabase([ + stats({ term: "bipolar", title_doc_count: 1 }), + stats({ term: "disorder", title_doc_count: 33 }), + ]); + + await classifyCorpusGrounding({ supabase: client, query: "bipolar disorder", ownerFilter: null }); + await classifyCorpusGrounding({ + supabase: client, + query: "bipolar disorder", + ownerFilter: "00000000-0000-0000-0000-000000000000", + }); + expect(rpc).toHaveBeenCalledTimes(2); + }); +}); + +describe("analyzeQueryWithClassifierFallback corpus grounding", () => { + async function loadRag(args: { classifierMock?: ReturnType; rows: CorpusTopicTermStats[] }) { + vi.stubEnv("OPENAI_API_KEY", "test-key"); + const classifierMock = + args.classifierMock ?? + vi.fn(async () => { + throw new Error("LLM classifier must not be called for corpus-decided queries"); + }); + vi.doMock("@/lib/openai", async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, generateStructuredTextResult: classifierMock }; + }); + const rag = await import("../src/lib/rag"); + const corpusGrounding = await import("../src/lib/corpus-grounding"); + const { analyzeClinicalQuery } = await import("../src/lib/clinical-search"); + rag.resetClassifierVerdictMemoForTests(); + corpusGrounding.resetCorpusGroundingCacheForTests(); + const rpc = vi.fn(async (_fn: string, rpcArgs: { terms: string[] }) => ({ + data: args.rows.filter((row) => rpcArgs.terms.includes(row.term)), + error: null, + })); + return { + rag, + analyzeClinicalQuery, + classifierMock, + rpc, + opts: { corpusGrounding: { supabase: { rpc } as never, ownerFilter: null } }, + }; + } + + // 60s timeout: the first test in this block pays the one-off vite transform cost of the + // large rag.ts module graph (~15s on a cold worker) before any assertion runs. + it( + "deterministically reclassifies an in-corpus bare topic to broad_summary without the LLM", + { timeout: 60000 }, + async () => { + const { rag, analyzeClinicalQuery, classifierMock, opts } = await loadRag({ + rows: [stats({ term: "bipolar", title_doc_count: 1 }), stats({ term: "disorder", title_doc_count: 33 })], + }); + const analysis = analyzeClinicalQuery("bipolar disorder"); + expect(analysis.queryClass).toBe("unsupported_or_general"); + + const result = await rag.analyzeQueryWithClassifierFallback("bipolar disorder", analysis, opts); + + expect(classifierMock).not.toHaveBeenCalled(); + expect(result.queryClass).toBe("broad_summary"); + expect(result.confidence).toBeGreaterThanOrEqual(0.62); + expect(result.needsSynthesis).toBe(true); + expect(result.corpusGrounding).toBe("in_corpus_topic"); + expect(result.reasons).toContain("corpus_topic_grounding"); + // The reclassified analysis must no longer short-circuit to 0 results. + expect(rag.shouldApplyUnsupportedSearchShortCircuit("bipolar disorder", result, [])).toBe(false); + }, + ); + + it("skips the LLM and keeps the deterministic refusal for corpus-absent invented terms", async () => { + const { rag, analyzeClinicalQuery, classifierMock, opts } = await loadRag({ + rows: [ + stats({ term: "florbizone", chunk_present: false, title_doc_count: 0 }), + stats({ term: "syndrome", title_doc_count: 12 }), + stats({ term: "management", title_doc_count: 375 }), + ], + }); + const analysis = analyzeClinicalQuery("florbizone syndrome management"); + + const result = await rag.analyzeQueryWithClassifierFallback("florbizone syndrome management", analysis, opts); + + expect(classifierMock).not.toHaveBeenCalled(); + expect(result.queryClass).toBe("unsupported_or_general"); + expect(result.needsClassifierFallback).toBe(false); + expect(result.corpusGrounding).toBe("out_of_corpus"); + // The refusal machinery keeps firing exactly as before — the LLM lottery is just removed. + expect(rag.shouldApplyUnsupportedSearchShortCircuit("florbizone syndrome management", result, [])).toBe(true); + // Alias expansions still rescue the query from the short-circuit (escape hatch preserved). + expect(rag.shouldApplyUnsupportedSearchShortCircuit("florbizone syndrome management", result, ["expansion"])).toBe( + false, + ); + }); + + it("falls through to the LLM classifier when grounding is inconclusive", async () => { + const classifierMock = vi.fn(async () => ({ + text: JSON.stringify({ + queryClass: "broad_summary", + confidence: 0.9, + reasons: ["classifier_test"], + expandedTerms: [], + }), + })); + const { rag, analyzeClinicalQuery, opts } = await loadRag({ + classifierMock, + rows: [stats({ term: "gout", title_doc_count: 0 }), stats({ term: "management", title_doc_count: 375 })], + }); + const analysis = analyzeClinicalQuery("gout management"); + + const result = await rag.analyzeQueryWithClassifierFallback("gout management", analysis, opts); + + expect(classifierMock).toHaveBeenCalledTimes(1); + expect(result.queryClass).toBe("broad_summary"); + expect(result.corpusGrounding).toBe("inconclusive"); + }); + + it("never sends pattern-guarded out-of-corpus medical queries to the corpus check or LLM", async () => { + const { rag, analyzeClinicalQuery, classifierMock, rpc, opts } = await loadRag({ rows: [] }); + const query = "What SSRI dose is recommended for adolescent depression?"; + const analysis = analyzeClinicalQuery(query); + + const result = await rag.analyzeQueryWithClassifierFallback(query, analysis, opts); + + expect(rpc).not.toHaveBeenCalled(); + expect(classifierMock).not.toHaveBeenCalled(); + expect(result.needsClassifierFallback).toBe(false); + expect(result.queryClass).toBe("unsupported_or_general"); + }); + + it("keeps legacy behaviour when no corpus grounding scope is provided", async () => { + const classifierMock = vi.fn(async () => ({ + text: JSON.stringify({ + queryClass: "broad_summary", + confidence: 0.9, + reasons: ["classifier_test"], + expandedTerms: [], + }), + })); + const { rag, analyzeClinicalQuery } = await loadRag({ classifierMock, rows: [] }); + const analysis = analyzeClinicalQuery("bipolar disorder"); + + const result = await rag.analyzeQueryWithClassifierFallback("bipolar disorder", analysis); + + expect(classifierMock).toHaveBeenCalledTimes(1); + expect(result.queryClass).toBe("broad_summary"); + expect(result.corpusGrounding).toBeUndefined(); + }); +}); diff --git a/tests/supabase-schema.test.ts b/tests/supabase-schema.test.ts index 522d8d115b..8e4abc7d10 100644 --- a/tests/supabase-schema.test.ts +++ b/tests/supabase-schema.test.ts @@ -534,6 +534,23 @@ describe("Supabase schema Data API grants", () => { expect(functionBody).toContain("f.metadata"); }); + it("declares the corpus topic term stats function with retrieval-equivalent scoping", () => { + // Finding #11 corpus grounding (migration 20260707100000): the stats the unsupported + // soft tail grounds on must be scoped exactly like retrieval — owner filter, indexed + // status, and committed generation — and stay service_role-only. + expect(schema).toContain("create or replace function public.corpus_topic_term_stats("); + const corpusStatsBody = schema.slice( + schema.indexOf("create or replace function public.corpus_topic_term_stats("), + schema.indexOf("create or replace function public.match_document_chunks("), + ); + expect(corpusStatsBody).toContain("public.retrieval_owner_matches(owner_filter, d.owner_id)"); + expect(corpusStatsBody).toContain("d.status = 'indexed'"); + expect(corpusStatsBody).toContain("public.is_committed_document_generation(c.index_generation_id, d.metadata)"); + expect(corpusStatsBody).toContain( + "grant execute on function public.corpus_topic_term_stats(text[], uuid) to service_role;", + ); + }); + it("filters hybrid retrieval by owner inside Postgres", () => { expect(schema).toContain("owner_filter uuid default null"); expect(schema).toContain( diff --git a/tests/universal-search.test.ts b/tests/universal-search.test.ts index 2a6b7f925b..ea5f08c5d2 100644 --- a/tests/universal-search.test.ts +++ b/tests/universal-search.test.ts @@ -10,7 +10,10 @@ async function loadUniversalSearch() { } describe("runUniversalSearch (demo/fixtures path)", () => { - it("returns groups in the fixed domain order without touching Supabase", async () => { + // 60s timeout: the first test in this file pays the one-off vite transform cost of the large + // universal-search module graph (~15s on a cold, loaded worker — right at the global 15s + // limit), which made this the suite's most frequent first-test timeout flake. + it("returns groups in the fixed domain order without touching Supabase", { timeout: 60000 }, async () => { const { runUniversalSearch, universalSearchDomains } = await loadUniversalSearch(); const response = await runUniversalSearch({ query: "clinical", limitPerDomain: 5, demo: true }); From b6b79fc29e7abdda82880c9d50dd0546a188c4e3 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Tue, 7 Jul 2026 03:27:10 +0800 Subject: [PATCH 3/6] =?UTF-8?q?fix(rag):=20RC9=20=E2=80=94=20fabricated=20?= =?UTF-8?q?similarity=20can=20no=20longer=20mint=20high=20answer=20confide?= =?UTF-8?q?nce?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consumer audit of the synthetic text similarity (findings item 21, updated): the headline least(0.95, 0.56 + text_rank*0.39) proxy no longer exists — match_document_chunks_text already returns similarity 0 with hybrid capped at 0.5. Three app-side fabricators remain (document-lookup fast path, memory-card chunk loader, table-fact signal matches), all tagged synthetic_text; their 0.58-floor values are deliberately load-bearing in route/coverage gates (always paired with structural checks), so re-gating those on native signals stays deferred to the telemetry-backed recalibration. The one provable defect is fixed: deriveConfidence let a fabricated 0.82+ (memory-card hybrid reaches 0.89, document-lookup 0.94) grant a "high" clinician-facing confidence label from purely lexical evidence. "high" now requires a genuine-cosine citation; synthetic-origin evidence caps at "medium". Strictly tightening; ordering, routing, and coverage gates untouched. Unit tests in tests/rag-score.test.ts. Also documents why the RC6 alias-group seed (workstream task 5) is deferred: the groups live in the synchronous analyzer while rag_aliases rows flow through the variant/short-circuit path — seeding both double-expands and changes behaviour corpus-wide, so it needs its own eval-gated change. Co-Authored-By: Claude Fable 5 --- docs/rag-hybrid-findings-and-todo.md | 40 +++++++++++++++++----- src/lib/rag.ts | 17 ++++++++-- tests/rag-score.test.ts | 51 +++++++++++++++++++++++++++- 3 files changed, 97 insertions(+), 11 deletions(-) diff --git a/docs/rag-hybrid-findings-and-todo.md b/docs/rag-hybrid-findings-and-todo.md index 358083b156..048e44b7bd 100644 --- a/docs/rag-hybrid-findings-and-todo.md +++ b/docs/rag-hybrid-findings-and-todo.md @@ -92,9 +92,10 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows Note: this only affects the **panel**; the answer-retrieval path (`searchChunksWithTelemetry`) has no per-doc cap and doesn't need one — the comparison gate already enforces ≥2 distinct docs, and single-topic queries _should_ be able to draw multiple chunks from the best document. - - ⏳ **Synthetic text similarity (RC9)** `least(0.95, 0.56 + text_rank*0.39)` still feeds coverage - gates that assume a real cosine — gate text-only paths on `text_rank`/`rrf` instead. (Cleanest - remaining ranking-correctness item.) + - 🔧 **Synthetic text similarity (RC9)** — superseded; see item 21 (2026-07-07 audit): the SQL + formula is gone (`match_document_chunks_text` returns similarity 0), the three remaining + app-side fabricators are tagged, and the fabricated-"high"-confidence defect is fixed; + only the telemetry-gated threshold recalibration remains. - ⏳ **Source-strength as a filter not just a penalty (RC8)**; **threshold floors (RC5)**; **rerank trigger (RC10)** — see item 4's note (marginal without a chunk-level eval metric). - ⏳ **Differentials flowchart-action boost (dropped in the PR #120 merge).** The codex/RAG_FIX @@ -216,7 +217,17 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows management" — chunk-present but no title topic) keep the legacy memoized-LLM behaviour. - ⏳ Still hard-coded (lower priority now the trigram path exists): moving `synonymGroups` / `domainAliasGroups` / `medicationAliasGroups` into `rag_aliases`; generalising the special-case - rewrites off `RagQueryClass`. + rewrites off `RagQueryClass`. **Design constraint (2026-07-07):** this is NOT a plain seed + migration. The groups are consumed inside the synchronous deterministic analyzer + (`analyzeClinicalQuery`), which must keep working in demo mode and in unit tests without a + DB, while `rag_aliases` rows flow through a different mechanism (`fetchEnabledRagAliases` → + retrieval query variants + the unsupported-short-circuit alias guard). Seeding the same + groups into `rag_aliases` while the in-code groups remain would double-expand variants and + change short-circuit behaviour corpus-wide — a behavior change needing its own golden + + rag-only eval run, not a data chore. Deferred from the 2026-07-07 retrieval-correctness + branch for that reason; do it as a dedicated eval-gated change (either an async analyzer + vocabulary refactor, or DB-only expansion with the in-code groups retired from the variant + path in the same change). ## P2 — offline/fallback remainder (Workstream F) @@ -256,10 +267,23 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows regression class (governance metadata weighting selection ordering) is only guarded by the manual PR checklist because `eval:retrieval:quality` needs live keys. Investigate a keys-free structural test (e.g. assert selection sort inputs exclude governance fields). -21. ⏳ **Recalibrate gates for synthetic text-only similarity (RC9 residual).** Text-fast-path - results now carry `similarity_origin: "synthetic_text"` telemetry; once enough data exists, - recalibrate `evaluateEvidenceCoverageGate` / text-fast-path thresholds against real cosine - distributions instead of the `least(0.95, 0.56 + text_rank*0.39)` proxy. +21. ⏳ **Recalibrate gates for synthetic text-only similarity (RC9 residual) — audited 2026-07-07, + scope reduced.** Full consumer audit on `claude/retrieval-correctness`: the headline + `least(0.95, 0.56 + text_rank*0.39)` proxy NO LONGER EXISTS — `match_document_chunks_text` + already returns `similarity = 0` with hybrid capped at 0.5 and the lexical signal isolated in + `lexical_score` (codified in schema.sql with the "do not fabricate" comment). What remains + synthetic are three app-side fabricators, all tagged `similarity_origin: "synthetic_text"`: + the document-lookup fast path (0.58 + documentScore, hybrid ≤ 0.94), memory-card chunk loader + (0.58 + confidence·0.28, hybrid ≤ 0.89), and table-fact signal matches. Their consumers: + `evaluateEvidenceCoverageGate` / `shouldReturnTextFastPath` / `chooseAnswerRoute` / + `shouldUseExtractiveAnswer` (thresholds 0.32–0.76 — the fabricated 0.58 floor is + deliberately load-bearing there, always paired with structural checks like + `directTitleSupport`; re-gating them on native signals is the deferred recalibration and + must not be attempted without the telemetry distributions), `buildRetrievalDiagnostics` + (topScore < 0.5 weak gate — floor also load-bearing), and `deriveConfidence`. **Fixed now:** + `deriveConfidence` no longer lets a fabricated 0.82+ mint a "high" answer-confidence label — + "high" requires a genuine-cosine citation; synthetic-origin evidence caps at "medium" + (strictly tightening, ordering/routing untouched, unit-tested in tests/rag-score.test.ts). 22. ⏳ **Registry-to-corpus embedding (universal search Phase 5).** Medications/services/forms/ differentials are federated into `/api/search/universal` but are not retrieval-corpus entities, so Answer mode cannot cite them. If product wants that: env-flagged ingestion, diff --git a/src/lib/rag.ts b/src/lib/rag.ts index fdfea34fab..4a2739d2b1 100644 --- a/src/lib/rag.ts +++ b/src/lib/rag.ts @@ -751,7 +751,16 @@ function allowedChunkMap(results: SearchResult[]) { // uncited high-similarity chunk grant "high" confidence to an answer built on // weak citations, so the strongest-score scan is scoped to the cited subset. // A citation that maps to no known chunk contributes nothing (fail low). -function deriveConfidence( +// +// RC9: results tagged `similarity_origin: "synthetic_text"` carry a similarity FABRICATED from +// lexical/structural signals (0.58-floor formulas in the document-lookup fast path, memory-card +// chunk loader, and table-fact signal matches), not a real cosine. Those fabrications routinely +// clear the 0.82 bar (memory-card hybrid reaches 0.89, document-lookup 0.94), which let a +// lexical-only citation mint "high" confidence. Synthetic-origin evidence is therefore capped at +// "medium": "high" requires at least one cited result whose similarity is a genuine cosine. +// Ordering, routing, and coverage gates are untouched — this only stops the fabricated scale +// from masquerading as strong semantic evidence in the clinician-facing confidence label. +export function deriveConfidence( results: SearchResult[], acceptedCitations: Array>, ): RagAnswer["confidence"] { @@ -759,7 +768,11 @@ function deriveConfidence( const citedIds = new Set(acceptedCitations.map((citation) => citation.chunk_id)); const citedResults = results.filter((result) => citedIds.has(result.id)); const strongest = citedResults.reduce((max, result) => Math.max(max, scoreValue(result)), 0); - if (strongest >= 0.82 && acceptedCitations.length >= 2) return "high"; + const strongestCosine = citedResults.reduce( + (max, result) => (result.similarity_origin === "synthetic_text" ? max : Math.max(max, scoreValue(result))), + 0, + ); + if (strongestCosine >= 0.82 && acceptedCitations.length >= 2) return "high"; if (strongest >= 0.64) return "medium"; return "low"; } diff --git a/tests/rag-score.test.ts b/tests/rag-score.test.ts index 7768af0df1..a8f5ebc1ee 100644 --- a/tests/rag-score.test.ts +++ b/tests/rag-score.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "vitest"; import type { SearchResult } from "../src/lib/types"; -import { scoreValue } from "../src/lib/rag"; +import { deriveConfidence, scoreValue } from "../src/lib/rag"; describe("scoreValue", () => { const base: SearchResult = { @@ -37,3 +37,52 @@ describe("scoreValue", () => { expect(scoreValue(result)).toBe(0.41); }); }); + +describe("deriveConfidence (RC9 synthetic similarity)", () => { + function result(overrides: Partial & { id: string }): SearchResult { + return { + document_id: "doc-1", + title: "Document", + file_name: "document.pdf", + page_number: 1, + chunk_index: 0, + section_heading: null, + content: "test", + image_ids: [], + similarity: 0, + images: [], + ...overrides, + }; + } + + it("caps synthetic-similarity citations at medium — a fabricated 0.82+ cannot mint high confidence", () => { + // Memory-card / document-lookup fabrications reach 0.89-0.94, which used to satisfy the + // 0.82 "high" bar with purely lexical evidence. + const cited = [ + result({ id: "a", similarity: 0.86, hybrid_score: 0.89, similarity_origin: "synthetic_text" }), + result({ id: "b", similarity: 0.84, hybrid_score: 0.87, similarity_origin: "synthetic_text" }), + ]; + expect(deriveConfidence(cited, [{ chunk_id: "a" }, { chunk_id: "b" }])).toBe("medium"); + }); + + it("still grants high confidence when a genuine cosine citation clears the bar", () => { + const cited = [ + result({ id: "a", similarity: 0.85, hybrid_score: 0.85, similarity_origin: "cosine" }), + result({ id: "b", similarity: 0.7, hybrid_score: 0.72, similarity_origin: "synthetic_text" }), + ]; + expect(deriveConfidence(cited, [{ chunk_id: "a" }, { chunk_id: "b" }])).toBe("high"); + }); + + it("treats untagged results as genuine cosine evidence (vector/hybrid layers set no origin)", () => { + const cited = [ + result({ id: "a", similarity: 0.85, hybrid_score: 0.85 }), + result({ id: "b", similarity: 0.83, hybrid_score: 0.84 }), + ]; + expect(deriveConfidence(cited, [{ chunk_id: "a" }, { chunk_id: "b" }])).toBe("high"); + }); + + it("returns unsupported without citations and low for weak cited evidence", () => { + expect(deriveConfidence([result({ id: "a", similarity: 0.9 })], [])).toBe("unsupported"); + expect(deriveConfidence([result({ id: "a", similarity: 0.4 })], [{ chunk_id: "a" }])).toBe("low"); + }); +}); From 384bd94e6628db1dc2f5257a5dfcc8ca7ac232e3 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Tue, 7 Jul 2026 03:31:42 +0800 Subject: [PATCH 4/6] docs(eval): record the public-owner sentinel requirement for live evals Since the 2026-07-06 public promotion the whole corpus is owner_id NULL; a real RAG_EVAL_OWNER_ID scopes retrieval to zero documents and fails every golden case (observed as a 34-case wipeout before diagnosis). Co-Authored-By: Claude Fable 5 --- docs/retrieval-quality-runbook.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/retrieval-quality-runbook.md b/docs/retrieval-quality-runbook.md index 972fac1740..6081a009e6 100644 --- a/docs/retrieval-quality-runbook.md +++ b/docs/retrieval-quality-runbook.md @@ -50,6 +50,12 @@ The command requires the same live-eval environment as the existing RAG eval scr - `OPENAI_API_KEY` - `RAG_EVAL_OWNER_ID`, `LOCAL_NO_AUTH_OWNER_ID`, or `RAG_EVAL_OWNER_EMAIL` +Since the 2026-07-06 public promotion the live corpus is entirely `owner_id = NULL`, so set +`RAG_EVAL_OWNER_ID=00000000-0000-0000-0000-000000000000` (the public-owner sentinel — +`retrieval_owner_matches` maps it to NULL-owner rows, mirroring anonymous production search). A +real owner UUID now scopes retrieval to zero documents and fails every case; leaving it unset +throws the owner-scope guard. + Optional cost fields: - `RAG_EVAL_INPUT_USD_PER_MILLION` From 1c1ed52ff71c5ac4286b8e867858100af90fdde8 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 7 Jul 2026 03:19:41 +0000 Subject: [PATCH 5/6] ci: retrigger checks after billing fix Co-authored-by: BigSimmo From 17d1754bf528e583b700179e3b701beec9c4b23b Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 7 Jul 2026 05:28:54 +0000 Subject: [PATCH 6/6] fix: resolve merge conflict and update drift-manifest schema_sha256 for merged schema.sql --- docs/rag-hybrid-findings-and-todo.md | 60 +++++++++++----------------- supabase/drift-manifest.json | 2 +- 2 files changed, 24 insertions(+), 38 deletions(-) diff --git a/docs/rag-hybrid-findings-and-todo.md b/docs/rag-hybrid-findings-and-todo.md index 80f68ac979..66fb26b91f 100644 --- a/docs/rag-hybrid-findings-and-todo.md +++ b/docs/rag-hybrid-findings-and-todo.md @@ -259,36 +259,6 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows from `candidate_aliases` → `rag_aliases` is still manual. 18. ⏳ **`document_index_units` vector recall** — no HNSW index (dropped 2026-07-02) and hosted Supabase denies `ALTER FUNCTION … SET hnsw.ef_search` for the `language sql` hybrid RPCs, so -<<<<<<< HEAD - only `match_document_memory_cards_hybrid` pins `ef_search=100`. Quantify the recall impact - before reintroducing an index. -19. ⏳ **Demo fallback can mask live retrieval failures in non-prod.** `/api/search` and - `/api/answer` silently swap in demo data on Supabase errors outside production (only an - `X-Clinical-KB-Fallback` header signals it). Proposal: surface a warning in - `check:production-readiness` output and/or a visible dev-mode banner rather than changing - the fallback behaviour. -20. ⏳ **Automated guard for governance-weighting regressions.** The 23/23 → 16/23 golden-set - regression class (governance metadata weighting selection ordering) is only guarded by the - manual PR checklist because `eval:retrieval:quality` needs live keys. Investigate a - keys-free structural test (e.g. assert selection sort inputs exclude governance fields). -21. ⏳ **Recalibrate gates for synthetic text-only similarity (RC9 residual) — audited 2026-07-07, - scope reduced.** Full consumer audit on `claude/retrieval-correctness`: the headline - `least(0.95, 0.56 + text_rank*0.39)` proxy NO LONGER EXISTS — `match_document_chunks_text` - already returns `similarity = 0` with hybrid capped at 0.5 and the lexical signal isolated in - `lexical_score` (codified in schema.sql with the "do not fabricate" comment). What remains - synthetic are three app-side fabricators, all tagged `similarity_origin: "synthetic_text"`: - the document-lookup fast path (0.58 + documentScore, hybrid ≤ 0.94), memory-card chunk loader - (0.58 + confidence·0.28, hybrid ≤ 0.89), and table-fact signal matches. Their consumers: - `evaluateEvidenceCoverageGate` / `shouldReturnTextFastPath` / `chooseAnswerRoute` / - `shouldUseExtractiveAnswer` (thresholds 0.32–0.76 — the fabricated 0.58 floor is - deliberately load-bearing there, always paired with structural checks like - `directTitleSupport`; re-gating them on native signals is the deferred recalibration and - must not be attempted without the telemetry distributions), `buildRetrievalDiagnostics` - (topScore < 0.5 weak gate — floor also load-bearing), and `deriveConfidence`. **Fixed now:** - `deriveConfidence` no longer lets a fabricated 0.82+ mint a "high" answer-confidence label — - "high" requires a genuine-cosine citation; synthetic-origin evidence caps at "medium" - (strictly tightening, ordering/routing untouched, unit-tested in tests/rag-score.test.ts). -======= only `match_document_memory_cards_hybrid` pins `ef_search=100`. Concrete measurement plan (needs live keys, ~1 hour): run `eval:retrieval:quality` twice with `--force-embedding` (bypasses lexical fast paths, exercising vectors directly) — once as-is and once after @@ -307,13 +277,29 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows `review_due`/`unverified` source outranks a lower-relevance `current`/`reviewed` one. The manual golden-eval checklist remains the live backstop; no further action. 21. 🔶 **Recalibrate gates for synthetic text-only similarity (RC9 residual) — DATA NOW - FLOWING (2026-07-06).** `synthetic_similarity_count` and `text_or_relaxation_used` are now - persisted into `rag_retrieval_logs.metadata` (they were computed but dropped by the - telemetry whitelist in /api/search). Once ~2 weeks of live rows exist, recalibrate - `evaluateEvidenceCoverageGate` / text-fast-path thresholds against real cosine - distributions: query `metadata->>'synthetic_similarity_count'` joined to `is_miss` to see - how often synthetic scores cross the 0.58/0.62 gates on misses vs hits. ->>>>>>> origin/main + FLOWING (2026-07-06); audited 2026-07-07, scope reduced.** `synthetic_similarity_count` and + `text_or_relaxation_used` are now persisted into `rag_retrieval_logs.metadata` (they were + computed but dropped by the telemetry whitelist in /api/search). Once ~2 weeks of live rows + exist, recalibrate `evaluateEvidenceCoverageGate` / text-fast-path thresholds against real + cosine distributions: query `metadata->>'synthetic_similarity_count'` joined to `is_miss` to + see how often synthetic scores cross the 0.58/0.62 gates on misses vs hits. Full consumer + audit on `claude/retrieval-correctness`: the headline `least(0.95, 0.56 + text_rank*0.39)` + proxy NO LONGER EXISTS — `match_document_chunks_text` already returns `similarity = 0` with + hybrid capped at 0.5 and the lexical signal isolated in `lexical_score` (codified in + schema.sql with the "do not fabricate" comment). What remains synthetic are three app-side + fabricators, all tagged `similarity_origin: "synthetic_text"`: the document-lookup fast path + (0.58 + documentScore, hybrid ≤ 0.94), memory-card chunk loader (0.58 + confidence·0.28, + hybrid ≤ 0.89), and table-fact signal matches. Their consumers: + `evaluateEvidenceCoverageGate` / `shouldReturnTextFastPath` / `chooseAnswerRoute` / + `shouldUseExtractiveAnswer` (thresholds 0.32–0.76 — the fabricated 0.58 floor is + deliberately load-bearing there, always paired with structural checks like + `directTitleSupport`; re-gating them on native signals is the deferred recalibration and + must not be attempted without the telemetry distributions), `buildRetrievalDiagnostics` + (topScore < 0.5 weak gate — floor also load-bearing), and `deriveConfidence`. **Fixed + (2026-07-07):** `deriveConfidence` no longer lets a fabricated 0.82+ mint a "high" + answer-confidence label — "high" requires a genuine-cosine citation; synthetic-origin + evidence caps at "medium" (strictly tightening, ordering/routing untouched, unit-tested in + tests/rag-score.test.ts). 22. ⏳ **Registry-to-corpus embedding (universal search Phase 5).** Medications/services/forms/ differentials are federated into `/api/search/universal` but are not retrieval-corpus entities, so Answer mode cannot cite them. Concrete implementation spec (in order): diff --git a/supabase/drift-manifest.json b/supabase/drift-manifest.json index 710da2a313..30fc86da30 100644 --- a/supabase/drift-manifest.json +++ b/supabase/drift-manifest.json @@ -2,7 +2,7 @@ "generated_at": "2026-07-06T18:30:39.629Z", "generator": "scripts/generate-drift-manifest.ts", "postgres_image": "supabase/postgres:17.6.1.127", - "schema_sha256": "9c29e76446f57874b3ea4f474713dcdc2d4bf029213bc96dbebbb60a5057a84c", + "schema_sha256": "8a88db3fcd974fae74807a91da118e8b0f9f4bcbf02df65b8b5670b22c32e24c", "replay_seconds": 15, "snapshot": { "views": [