diff --git a/docs/rag-hybrid-findings-and-todo.md b/docs/rag-hybrid-findings-and-todo.md index 0794142ef0..0f6f45e44c 100644 --- a/docs/rag-hybrid-findings-and-todo.md +++ b/docs/rag-hybrid-findings-and-todo.md @@ -234,28 +234,68 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows ## Follow-ups filed 2026-07-06 (universal-search workstream) -17. ⏳ **Alias promotion pipeline is blocked by privacy redaction.** `rag_query_misses` rows store - hashed/redacted queries with empty `candidate_aliases`, so hardcoded `synonymGroups` / - `domainAliasGroups` / special-case rewrites in `src/lib/clinical-search.ts` cannot be replaced - with data-driven `rag_aliases` rows until a privacy-safe candidate-alias capture is designed. +17. 🔶 **Alias promotion pipeline is blocked by privacy redaction — PARTIALLY UNBLOCKED + (2026-07-06).** Weak-search misses now store `queryVocabularyAliasesForStorage(query)` as + `candidate_aliases` when raw retention is off: only canonical terms from the curated + clinical vocabulary that the query MATCHED are persisted (output text comes from the fixed + vocabulary table, never the raw query, so RET-H4 holds). Remaining: terms OUTSIDE the + curated vocabulary still cannot be captured without a privacy review; promotion tooling + from `candidate_aliases` → `rag_aliases` is still manual. 18. ⏳ **`document_index_units` vector recall** — no HNSW index (dropped 2026-07-02) and hosted Supabase denies `ALTER FUNCTION … SET hnsw.ef_search` for the `language sql` hybrid RPCs, so - only `match_document_memory_cards_hybrid` pins `ef_search=100`. Quantify the recall impact - before reintroducing an index. -19. ⏳ **Demo fallback can mask live retrieval failures in non-prod.** `/api/search` and - `/api/answer` silently swap in demo data on Supabase errors outside production (only an - `X-Clinical-KB-Fallback` header signals it). Proposal: surface a warning in - `check:production-readiness` output and/or a visible dev-mode banner rather than changing - the fallback behaviour. -20. ⏳ **Automated guard for governance-weighting regressions.** The 23/23 → 16/23 golden-set - regression class (governance metadata weighting selection ordering) is only guarded by the - manual PR checklist because `eval:retrieval:quality` needs live keys. Investigate a - keys-free structural test (e.g. assert selection sort inputs exclude governance fields). -21. ⏳ **Recalibrate gates for synthetic text-only similarity (RC9 residual).** Text-fast-path - results now carry `similarity_origin: "synthetic_text"` telemetry; once enough data exists, - recalibrate `evaluateEvidenceCoverageGate` / text-fast-path thresholds against real cosine - distributions instead of the `least(0.95, 0.56 + text_rank*0.39)` proxy. + only `match_document_memory_cards_hybrid` pins `ef_search=100`. Concrete measurement plan + (needs live keys, ~1 hour): run `eval:retrieval:quality` twice with `--force-embedding` + (bypasses lexical fast paths, exercising vectors directly) — once as-is and once after + `create index concurrently` on `document_index_units.embedding` in a Supabase branch — and + compare doc-recall@5 + p90 latency. If recall gain < 1 case, close as not-worth-4.4GB. The + ef_search half can be retested via the plpgsql-wrapper trick that memory_cards already uses + (wrap the `language sql` RPC in a plpgsql shim that SETs it). +19. ✅ **Demo fallback can mask live retrieval failures in non-prod — DONE (2026-07-06).** + `nonProductionSupabaseDemoFallbackReason` (the shared choke point for /api/search, + /api/answer, and /api/answer/stream) now emits a loud `console.warn` naming the env vars to + check whenever the non-prod demo fallback fires; behaviour and the + `X-Clinical-KB-Fallback` header are unchanged. A visible dev-mode banner remains optional. +20. ✅ **Automated guard for governance-weighting regressions — ALREADY COVERED.** A keys-free + structural test exists: `tests/retrieval-selection.test.ts` ("keeps relevance ordering and + does not let source-governance metadata reorder selection") asserts a higher-relevance + `review_due`/`unverified` source outranks a lower-relevance `current`/`reviewed` one. The + manual golden-eval checklist remains the live backstop; no further action. +21. 🔶 **Recalibrate gates for synthetic text-only similarity (RC9 residual) — DATA NOW + FLOWING (2026-07-06).** `synthetic_similarity_count` and `text_or_relaxation_used` are now + persisted into `rag_retrieval_logs.metadata` (they were computed but dropped by the + telemetry whitelist in /api/search). Once ~2 weeks of live rows exist, recalibrate + `evaluateEvidenceCoverageGate` / text-fast-path thresholds against real cosine + distributions: query `metadata->>'synthetic_similarity_count'` joined to `is_miss` to see + how often synthetic scores cross the 0.58/0.62 gates on misses vs hits. 22. ⏳ **Registry-to-corpus embedding (universal search Phase 5).** Medications/services/forms/ differentials are federated into `/api/search/universal` but are not retrieval-corpus - entities, so Answer mode cannot cite them. If product wants that: env-flagged ingestion, - golden-eval + invented-term controls first (depends on 17 for alias hygiene). + entities, so Answer mode cannot cite them. Concrete implementation spec (in order): + 1. Flag `RAG_REGISTRY_CORPUS_EMBEDDING` (default off) in `src/lib/env.ts`. + 2. Ingestion script `scripts/embed-registry-records.ts`: map each registry record to a + synthetic "document" (`metadata.source_kind = 'registry_record'`, title = record title, + one chunk per record from the record's search text, embedded with the standard + `text-embedding-3-small` path) so the existing chunk pipeline/RPCs need no schema change. + 3. Re-embed on registry edit: hook `ensureRegistrySeeded` / record-update routes to enqueue + re-embedding for the changed slug only. + 4. Answer-surface labelling: `sourceGovernanceWarnings` must label registry-backed + citations distinctly (registry records are curated summaries, not source documents). + 5. Gates before enabling anywhere real: `eval:retrieval:quality` 23/23 with the flag ON, + plus invented-term controls ("florbizone syndrome management") still refusing — registry + rows must not become a fabrication surface for unsupported topics. +23. ⏳ **Finding #11 full fix (RAG optimisation Phase 2)** — the classifier-verdict memo (shipped + 2026-07-06) makes zero-result behaviour deterministic per query but does not close the gap: + the deterministic analyzer still cannot tell in-corpus topics from out-of-corpus ones. + Phase-2 spec stands (corpus-grounded relevance: IDF/corpus-frequency weighting of query + terms + data-driven vocabulary), with the added prerequisite that item 17's vocabulary + capture now supplies real miss data to seed the vocabulary from. +24. ⏳ **OCR dropped-letter corruption in table index units** — no reliable detector exists (82% + false positives; guard reverted). Next viable angle: dictionary-based repair at INGESTION + (compare table-cell tokens against the document's own clean chunk text — "p ycho ocial" + aligns to "psychosocial" within the same page's raw text) rather than heuristic detection at + query time. Scope to `worker/` table extraction; requires the Python OCR stack to test. +25. ⏳ **Retrieval latency p90 ~8.6s (local)** — remaining sequential layers after the 2026-07-01 + parallelisation. Cheapest next step (measure first): overlap `embedTextWithTelemetry` with + the text fast path unconditionally (today preload only fires when `shouldPreloadEmbedding`), + and collapse the repeated `attachDocumentRankingMetadata` calls to one batched fetch per + request. Both are perf-only; gate with the golden eval unchanged + p90 from + `rag_retrieval_logs` before/after. diff --git a/src/app/api/search/route.ts b/src/app/api/search/route.ts index 0f2fea96db..044ab1e4ce 100644 --- a/src/app/api/search/route.ts +++ b/src/app/api/search/route.ts @@ -30,6 +30,7 @@ import { queryDerivedTokensForStorage, queryPrivacyMetadata, queryTextForStorage, + queryVocabularyAliasesForStorage, } from "@/lib/query-privacy"; import { safeErrorLogDetails } from "@/lib/privacy"; import { nonProductionSupabaseDemoFallbackReason } from "@/lib/supabase/errors"; @@ -414,8 +415,12 @@ function candidatePromotions(query: string, results: SearchResult[]) { document_id: label.document_id, confidence: label.confidence, })); + const rawTokens = queryDerivedTokensForStorage(Array.from(new Set(queryTerms)).slice(0, 10)); return { - aliases: queryDerivedTokensForStorage(Array.from(new Set(queryTerms)).slice(0, 10)), + // With raw retention off, fall back to curated clinical-vocabulary matches — output text + // comes from the fixed vocabulary table, never the query, so it is RET-H4 safe and keeps + // the alias-promotion pipeline fed (rag-hybrid-findings item 17). + aliases: rawTokens.length ? rawTokens : queryVocabularyAliasesForStorage(query), labels: topLabels, }; } @@ -528,6 +533,11 @@ function retrievalDecisionTelemetry(telemetry: Record) { second_stage_rerank_used: telemetryBoolean(telemetry, "second_stage_rerank_used"), second_stage_rerank_latency_ms: telemetryNumber(telemetry, "second_stage_rerank_latency_ms"), visual_direct_image_count: telemetryNumber(telemetry, "visual_direct_image_count"), + // RC9/P8b observability: these feed the synthetic-similarity gate recalibration and the + // weak-match OR-augmentation review (rag-hybrid-findings items 21 and the P8b extension) — + // without persisting them the recalibration has no data to work from. + text_or_relaxation_used: telemetryString(telemetry, "text_or_relaxation_used"), + synthetic_similarity_count: telemetryNumber(telemetry, "synthetic_similarity_count"), }; } diff --git a/src/components/clinical-dashboard/universal-search-command-surface.tsx b/src/components/clinical-dashboard/universal-search-command-surface.tsx index 066aa98787..72b24e753b 100644 --- a/src/components/clinical-dashboard/universal-search-command-surface.tsx +++ b/src/components/clinical-dashboard/universal-search-command-surface.tsx @@ -34,6 +34,17 @@ const excludedDomainByMode: Partial> = tools: "tools", }; +// Reverse of modeIdByDomain for chip counts: the domain whose live result total a +// cross-mode chip should show. Answer/favourites chips have no countable domain. +const domainByTargetMode: Partial> = { + documents: "documents", + prescribing: "medications", + services: "services", + forms: "forms", + differentials: "differentials", + tools: "tools", +}; + const modeIdByDomain: Record = { documents: "documents", medications: "prescribing", @@ -357,6 +368,7 @@ export function UniversalSearchCommandSurface({ const showSafetyBanner = modeId === "differentials" && differentialRedFlagTerms.some((term) => trimmedQuery.toLowerCase().includes(term)); const showFormCodeHint = modeId === "forms" && isFormCodeQuery(trimmedQuery); + const { groups: universalGroups, query: universalQuery } = universal; const sections = useMemo(() => { if (!config) return []; @@ -453,8 +465,8 @@ export function UniversalSearchCommandSurface({ // search endpoint, excluding this mode's own domain. Selecting an item navigates straight // to the record; each group ends with a cross-mode "view all" that re-runs the query in // the owning mode. Enter with nothing highlighted still runs the mode-scoped search. - if (trimmedQuery && universal.query === trimmedQuery && universal.groups.length) { - for (const group of universal.groups) { + if (trimmedQuery && universalQuery === trimmedQuery && universalGroups.length) { + for (const group of universalGroups) { const targetModeId = modeIdByDomain[group.kind]; const targetMode = appModeDefinition(targetModeId); const GroupIcon = appModeIcons[targetModeId]; @@ -564,6 +576,13 @@ export function UniversalSearchCommandSurface({ items: config.crossModes.map((target) => { const targetMode = appModeDefinition(target); const TargetIcon = appModeIcons[target]; + // Live count from the universal typeahead response ("Forms (2)") — only shown when + // fresh results for this exact query exist, so the chip never shows a stale number. + const targetDomain = domainByTargetMode[target]; + const targetCount = + targetDomain && universalQuery === trimmedQuery + ? universalGroups.find((group) => group.kind === targetDomain)?.total + : undefined; return { id: nextId(), label: targetMode.label, @@ -582,6 +601,7 @@ export function UniversalSearchCommandSurface({ > {targetMode.label} + {typeof targetCount === "number" ? ` (${targetCount})` : ""} ), }; @@ -605,8 +625,8 @@ export function UniversalSearchCommandSurface({ router, showFormCodeHint, trimmedQuery, - universal.groups, - universal.query, + universalGroups, + universalQuery, ]); const flatItems = useMemo(() => sections.flatMap((section) => section.items), [sections]); diff --git a/src/components/clinical-dashboard/use-universal-search.ts b/src/components/clinical-dashboard/use-universal-search.ts index 590b2f8ae7..331e5efc7b 100644 --- a/src/components/clinical-dashboard/use-universal-search.ts +++ b/src/components/clinical-dashboard/use-universal-search.ts @@ -32,18 +32,26 @@ export function useUniversalSearch(args: { const { authorizationHeader } = useAuthSession(); const [result, setResult] = useState<{ groups: UniversalSearchGroup[]; query: string }>({ groups: [], query: "" }); const requestSeqRef = useRef(0); + const prevAuthRef = useRef(authorizationHeader); const trimmedQuery = args.query.trim(); const active = args.enabled && trimmedQuery.length >= minQueryLength; const limitPerDomain = args.limitPerDomain ?? 3; const excludeDomain = args.excludeDomain; useEffect(() => { + const authChanged = prevAuthRef.current !== authorizationHeader; + prevAuthRef.current = authorizationHeader; + if (!active) { // Invalidate any in-flight request; visible state is derived, so no reset needed. requestSeqRef.current += 1; return undefined; } + if (authChanged) { + setResult({ groups: [], query: "" }); + } + const requestId = ++requestSeqRef.current; const timer = window.setTimeout(() => { const domains = (["documents", "medications", "services", "forms", "differentials", "tools"] as const).filter( diff --git a/src/lib/query-privacy.ts b/src/lib/query-privacy.ts index 13cd6bd3c0..02f8d2d2a0 100644 --- a/src/lib/query-privacy.ts +++ b/src/lib/query-privacy.ts @@ -1,4 +1,5 @@ import { createHash, createHmac } from "node:crypto"; +import { clinicalVocabularyMatches } from "@/lib/clinical-vocabulary"; import { env } from "@/lib/env"; export function normalizeQueryText(query: string) { @@ -46,6 +47,16 @@ export function queryDerivedTokensForStorage(tokens: string[]): string[] { return env.RAG_PERSIST_RAW_QUERY_TEXT ? tokens : []; } +// RET-H4-safe candidate aliases for the alias-promotion pipeline (rag-hybrid-findings +// item 17): every returned string is a canonical term from the curated clinical +// vocabulary that the query MATCHED — the output text comes from the fixed vocabulary +// table, never from the raw query — so patient-identifying text cannot leak even with +// raw retention off. This unblocks rag_query_misses.candidate_aliases, which was always +// empty under redaction and starved data-driven promotion into rag_aliases. +export function queryVocabularyAliasesForStorage(query: string, limit = 10): string[] { + return Array.from(new Set(clinicalVocabularyMatches(query, limit).map((entry) => entry.canonical))).slice(0, limit); +} + // Privacy metadata to fold into a logged row's `metadata` jsonb: a stable hash // for joins/dedup and a flag recording whether raw text was retained. export function queryPrivacyMetadata(query: string) { diff --git a/src/lib/supabase/errors.ts b/src/lib/supabase/errors.ts index 24ea4756d3..07120d3b44 100644 --- a/src/lib/supabase/errors.ts +++ b/src/lib/supabase/errors.ts @@ -23,5 +23,14 @@ export function isSupabaseApiKeyConfigurationError(error: unknown) { export function nonProductionSupabaseDemoFallbackReason(error: unknown) { if (process.env.NODE_ENV === "production") return null; if (!isSupabaseApiKeyConfigurationError(error)) return null; + // Item 19 (rag-hybrid-findings): this fallback silently swaps demo data in for live search + // and answer responses outside production, which can make a broken live path look healthy + // during local/dev testing. Keep the behaviour, but make it loud in the server log — the + // only other signal is the easy-to-miss X-Clinical-KB-Fallback response header. + console.warn( + "[clinical-kb] Supabase unavailable — serving DEMO data as a non-production fallback. " + + "Live search/answer paths are NOT being exercised. Check NEXT_PUBLIC_SUPABASE_URL / " + + "SUPABASE_SERVICE_ROLE_KEY if this is unexpected.", + ); return "supabase_api_key_configuration_unavailable"; } diff --git a/tests/privacy.test.ts b/tests/privacy.test.ts index 969377f452..ce9edd29d5 100644 --- a/tests/privacy.test.ts +++ b/tests/privacy.test.ts @@ -146,3 +146,31 @@ describe("query privacy storage helpers", () => { expect(queryDerivedTokensForStorage(["clozapine"])).toEqual(["clozapine"]); }); }); + +describe("queryVocabularyAliasesForStorage (RET-H4-safe candidate aliases)", () => { + it("returns only curated vocabulary canonicals matched by the query, never query text", async () => { + vi.doMock("@/lib/env", () => ({ env: { RAG_PERSIST_RAW_QUERY_TEXT: false } })); + const { queryVocabularyAliasesForStorage } = await import("../src/lib/query-privacy"); + const { clinicalVocabularyEntries } = await import("../src/lib/clinical-vocabulary"); + const canonicals = new Set(clinicalVocabularyEntries().map((entry) => entry.canonical)); + + // A query mixing a patient-identifying name with clinical vocabulary must only ever emit + // the curated canonical — the name cannot appear because output strings come from the + // fixed vocabulary table, not from the query. + const aliases = queryVocabularyAliasesForStorage("John Citizen ANC threshold for depot"); + expect(aliases.length).toBeGreaterThan(0); + for (const alias of aliases) { + expect(canonicals.has(alias)).toBe(true); + expect(alias.toLowerCase()).not.toContain("john"); + expect(alias.toLowerCase()).not.toContain("citizen"); + } + expect(aliases).toContain("absolute neutrophil count"); + expect(aliases).toContain("long acting injectable"); + }); + + it("returns nothing for queries with no vocabulary match", async () => { + vi.doMock("@/lib/env", () => ({ env: { RAG_PERSIST_RAW_QUERY_TEXT: false } })); + const { queryVocabularyAliasesForStorage } = await import("../src/lib/query-privacy"); + expect(queryVocabularyAliasesForStorage("John Citizen follow up appointment")).toEqual([]); + }); +}); diff --git a/tests/universal-search-owner-live.test.ts b/tests/universal-search-owner-live.test.ts new file mode 100644 index 0000000000..156a534522 --- /dev/null +++ b/tests/universal-search-owner-live.test.ts @@ -0,0 +1,69 @@ +import { describe, expect, it } from "vitest"; +import { isUsableBrowserSupabaseKey } from "../src/lib/supabase/client"; + +// Live owner-auth coverage for universal search. The header sign-in UI is magic-link/OAuth +// only (no password field), so browser-login Playwright coverage is not feasible; instead +// this signs in with the E2E password user via supabase-js and exercises the real route +// handler with a genuine bearer token — the same path the typeahead hook uses. +// +// Skips (never fails) when the live env is absent: demo/offline environments and keys-free +// CI run the mocked coverage in tests/universal-search.test.ts instead. + +const liveEnvReady = Boolean( + process.env.E2E_USER_EMAIL && + process.env.E2E_USER_PASSWORD && + process.env.NEXT_PUBLIC_SUPABASE_URL && + isUsableBrowserSupabaseKey(process.env.NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY) && + process.env.SUPABASE_SERVICE_ROLE_KEY && + !/<[^>]+>|^your-|replace-with|placeholder/i.test(process.env.SUPABASE_SERVICE_ROLE_KEY) && + process.env.NEXT_PUBLIC_DEMO_MODE !== "true", +); + +describe.skipIf(!liveEnvReady)("GET /api/search/universal (live owner auth)", () => { + it("serves owner-scoped registry groups through a real session token", { timeout: 45_000 }, async () => { + const { createClient } = await import("@supabase/supabase-js"); + const supabase = createClient( + process.env.NEXT_PUBLIC_SUPABASE_URL!, + process.env.NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY!, + { auth: { persistSession: false } }, + ); + const { data, error } = await supabase.auth.signInWithPassword({ + email: process.env.E2E_USER_EMAIL!, + password: process.env.E2E_USER_PASSWORD!, + }); + if (error) { + // Env advertises live credentials but Supabase rejected them (stale secret, wrong project). + console.warn(`Skipping live owner-auth test: ${error.message}`); + return; + } + const token = data.session?.access_token; + expect(token).toBeTruthy(); + + const { GET } = await import("../src/app/api/search/universal/route"); + const response = await GET( + new Request("http://localhost/api/search/universal?q=acamprosate&limit=3", { + headers: { Authorization: `Bearer ${token}` }, + }), + ); + expect(response.status).toBe(200); + + const payload = (await response.json()) as { + demoMode?: boolean; + publicAccess?: boolean; + groups: Array<{ kind: string; error?: boolean; items: Array<{ href: string }> }>; + }; + // Owner path: neither the demo nor the public-fixture ladder rung served this. + expect(payload.demoMode).toBeUndefined(); + expect(payload.publicAccess).toBeUndefined(); + + const medications = payload.groups.find((group) => group.kind === "medications"); + expect(medications?.error).toBeUndefined(); + expect(medications?.items.length ?? 0).toBeGreaterThan(0); + expect(medications?.items[0]?.href).toContain("/medications/"); + + const documents = payload.groups.find((group) => group.kind === "documents"); + expect(documents?.error).toBeUndefined(); + + await supabase.auth.signOut(); + }); +});