From de684aa36d3fd573d3feb7d07076dc3954708c8f Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 10:54:32 +0000 Subject: [PATCH 1/6] Extract weak-retrieval threshold, reconcile timeout docs, file search follow-ups Phase 0 of the search hardening + universal search workstream: - weakRetrievalTopScoreThreshold (0.48) moves from duplicated magic literals in /api/search into rag-routing.ts beside the routing thresholds, with an ordering test (0.32 < 0.48 < 0.64 < 0.76). - search-rag-master-context.md no longer claims a 12000ms answer budget; the 30000ms default in env.ts is a documented deliberate product decision. - Six follow-ups filed in rag-hybrid-findings-and-todo.md (alias promotion, index-unit HNSW, demo-fallback masking, governance-regression guard, synthetic-similarity recalibration, registry-to-corpus embedding). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_011QiyE8jMm7VnrtknHF6jnJ --- docs/rag-hybrid-findings-and-todo.md | 28 ++++++++++++++++++++++++++++ docs/search-rag-master-context.md | 8 ++++---- src/app/api/search/route.ts | 5 +++-- src/lib/rag-routing.ts | 5 +++++ tests/rag-routing.test.ts | 13 +++++++++++++ 5 files changed, 53 insertions(+), 6 deletions(-) diff --git a/docs/rag-hybrid-findings-and-todo.md b/docs/rag-hybrid-findings-and-todo.md index b8cd13526e..0794142ef0 100644 --- a/docs/rag-hybrid-findings-and-todo.md +++ b/docs/rag-hybrid-findings-and-todo.md @@ -231,3 +231,31 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows 16. ⏳ **ROTATE all secrets** pasted in plaintext this session: OpenAI key, Supabase `service_role` JWT + legacy JWT secret, DB password, E2E password. `.env.local` is gitignored, but the values were exposed in chat. + +## Follow-ups filed 2026-07-06 (universal-search workstream) + +17. ⏳ **Alias promotion pipeline is blocked by privacy redaction.** `rag_query_misses` rows store + hashed/redacted queries with empty `candidate_aliases`, so hardcoded `synonymGroups` / + `domainAliasGroups` / special-case rewrites in `src/lib/clinical-search.ts` cannot be replaced + with data-driven `rag_aliases` rows until a privacy-safe candidate-alias capture is designed. +18. ⏳ **`document_index_units` vector recall** — no HNSW index (dropped 2026-07-02) and hosted + Supabase denies `ALTER FUNCTION … SET hnsw.ef_search` for the `language sql` hybrid RPCs, so + only `match_document_memory_cards_hybrid` pins `ef_search=100`. Quantify the recall impact + before reintroducing an index. +19. ⏳ **Demo fallback can mask live retrieval failures in non-prod.** `/api/search` and + `/api/answer` silently swap in demo data on Supabase errors outside production (only an + `X-Clinical-KB-Fallback` header signals it). Proposal: surface a warning in + `check:production-readiness` output and/or a visible dev-mode banner rather than changing + the fallback behaviour. +20. ⏳ **Automated guard for governance-weighting regressions.** The 23/23 → 16/23 golden-set + regression class (governance metadata weighting selection ordering) is only guarded by the + manual PR checklist because `eval:retrieval:quality` needs live keys. Investigate a + keys-free structural test (e.g. assert selection sort inputs exclude governance fields). +21. ⏳ **Recalibrate gates for synthetic text-only similarity (RC9 residual).** Text-fast-path + results now carry `similarity_origin: "synthetic_text"` telemetry; once enough data exists, + recalibrate `evaluateEvidenceCoverageGate` / text-fast-path thresholds against real cosine + distributions instead of the `least(0.95, 0.56 + text_rank*0.39)` proxy. +22. ⏳ **Registry-to-corpus embedding (universal search Phase 5).** Medications/services/forms/ + differentials are federated into `/api/search/universal` but are not retrieval-corpus + entities, so Answer mode cannot cite them. If product wants that: env-flagged ingestion, + golden-eval + invented-term controls first (depends on 17 for alias hygiene). diff --git a/docs/search-rag-master-context.md b/docs/search-rag-master-context.md index 9d6fbe5b17..5eaa051a65 100644 --- a/docs/search-rag-master-context.md +++ b/docs/search-rag-master-context.md @@ -19,7 +19,7 @@ The desired experience is: Phase 7 performance hardening is implemented: -- `OPENAI_ANSWER_TIMEOUT_MS=12000` is the answer-generation timeout budget. +- `OPENAI_ANSWER_TIMEOUT_MS` is the answer-generation timeout budget. Phase 7 introduced it at 12000ms; the current default is **30000ms** — a deliberate product decision to favour natural, model-written answers over fast degradation to stitched extractive prose (see the rationale comment at `src/lib/env.ts` next to `OPENAI_ANSWER_TIMEOUT_MS`). - `src/lib/rag.ts` passes that timeout to structured answer generation so provider stalls fail into the existing source-backed fallback path faster than the global OpenAI request timeout. - `scripts/eval-rag.ts` excludes `generation_fallback` answers from the intentional routine-extractive latency bucket so provider timeout waits do not distort the model-free extractive metric. - Focused tests, typecheck, production-readiness, and capped RAG eval with threshold failure enabled passed after the change. @@ -33,9 +33,9 @@ Phase 7b latency polish is implemented: Deployment/config note: -- `.env.example` documents `OPENAI_ANSWER_TIMEOUT_MS=12000`. -- Local `.env.local` should also include `OPENAI_ANSWER_TIMEOUT_MS=12000` for explicit local parity. -- Hosted production/deployment environments should set `OPENAI_ANSWER_TIMEOUT_MS=12000` explicitly, or they will rely on the server default from `src/lib/env.ts`. +- `.env.example` documents `OPENAI_ANSWER_TIMEOUT_MS=30000`, matching the server default in `src/lib/env.ts`. +- Local `.env.local` may set it explicitly for parity; unset environments rely on the 30000ms server default. +- The historical 12000ms value in `docs/search-rag-phase-0-baseline.md` and `docs/search-rag-master-plan.md` records the Phase 7 rollout, not current guidance. ## Skill Lenses Used diff --git a/src/app/api/search/route.ts b/src/app/api/search/route.ts index 62ea24d859..0f2fea96db 100644 --- a/src/app/api/search/route.ts +++ b/src/app/api/search/route.ts @@ -9,6 +9,7 @@ import { fetchRelatedDocuments, toDocumentMatch } from "@/lib/document-enrichmen import { jsonError, PublicApiError } from "@/lib/http"; import { isClinicalImageEvidence } from "@/lib/image-filtering"; import { searchChunksWithTelemetry } from "@/lib/rag"; +import { weakRetrievalTopScoreThreshold } from "@/lib/rag-routing"; import { classifyRagQuery, normalizedClinicalSearchTokens } from "@/lib/clinical-search"; import { buildSmartRagApiPlan } from "@/lib/smart-rag-api"; import { SOURCE_ONLY_EMBEDDING_SKIP_REASON } from "@/lib/rag-provider"; @@ -439,7 +440,7 @@ function logWeakSearch(args: { args.results.length === 0 || args.relevance.verdict === "none" || args.relevance.verdict === "nearby" || - topScore < 0.48; + topScore < weakRetrievalTopScoreThreshold; if (!weak) return; const promotions = candidatePromotions(args.query, args.results); void args.supabase @@ -549,7 +550,7 @@ function logRetrievalDiagnostics(args: { args.results.length === 0 || args.relevance.verdict === "none" || args.relevance.verdict === "nearby" || - topScore < 0.48; + topScore < weakRetrievalTopScoreThreshold; const latencyMs = telemetryLatencyMs(args.telemetry); await args.supabase.from("rag_retrieval_logs").insert({ diff --git a/src/lib/rag-routing.ts b/src/lib/rag-routing.ts index 1c518e9038..7f53b158e2 100644 --- a/src/lib/rag-routing.ts +++ b/src/lib/rag-routing.ts @@ -14,6 +14,11 @@ export type AnswerRoute = { const unsupportedSimilarityThreshold = 0.32; const strongRetrievalThreshold = 0.64; const extractiveRetrievalThreshold = 0.76; +// Gates weak-search telemetry (rag_query_misses logging + retrieval diagnostics), not answer +// routing. It deliberately sits between unsupportedSimilarityThreshold (0.32) and +// strongRetrievalThreshold (0.64): a top score below it means retrieval "worked" but was too +// weak to trust, which is the miss signal alias curation feeds on. +export const weakRetrievalTopScoreThreshold = 0.48; const complexClinicalQueryPattern = /\b(compare|compared|versus|vs|conflict|gap|contraindicat\w*|interaction\w*|side effect\w*|adverse|suicid\w*|toxicity|myocarditis|neutropenia|anc|fbc|urgent|escalat\w*|withhold|cease|stop|dose|dosing|prescrib\w*)\b/i; const strongClinicalEscalationPattern = diff --git a/tests/rag-routing.test.ts b/tests/rag-routing.test.ts index 36116b730c..c4f7c9bed2 100644 --- a/tests/rag-routing.test.ts +++ b/tests/rag-routing.test.ts @@ -3,6 +3,7 @@ import { chooseAnswerRoute, hasAdversarialManipulationIntent, shouldRetryWithStrongAfterFast, + weakRetrievalTopScoreThreshold, } from "../src/lib/rag-routing"; import { ragEvalCases } from "../src/lib/rag-eval-cases"; import type { SearchResult } from "../src/lib/types"; @@ -400,3 +401,15 @@ describe("adversarial-manipulation query guard", () => { } }); }); + +describe("weakRetrievalTopScoreThreshold", () => { + it("sits between the unsupported (0.32) and strong (0.64) routing thresholds", () => { + // Telemetry "weak search" labeling must be stricter than the unsupported routing floor + // (otherwise every routed answer would log as a miss) and looser than the strong-route + // bar (otherwise genuinely weak retrievals would never be logged for alias curation). + expect(weakRetrievalTopScoreThreshold).toBeGreaterThan(0.32); + expect(weakRetrievalTopScoreThreshold).toBeLessThan(0.64); + expect(0.64).toBeLessThan(0.76); + expect(weakRetrievalTopScoreThreshold).toBe(0.48); + }); +}); From c9d20ac0540648204b040d101938bd9258781e7b Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 10:54:50 +0000 Subject: [PATCH 2/6] Fix core RAG retrieval defects: classifier determinism, tie-breaking, weak-match recall MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1 of the search hardening workstream — four independently revertable fixes: 1.1 Memoize the LLM query-classifier verdict (finding #11 interim fix per docs/process-hardening.md): successful verdicts (accepted AND rejected) are cached 15 min with in-flight dedup, so the unsupported short-circuit can no longer flip run-to-run and intermittently return 0 results for valid queries like "bipolar disorder". Transport errors stay retryable. 1.2 Complete the saturation tiebreak: rankClinicalResults now emits its pre-clamp boost sum as score_explanation.preClampFinalScore, and selectRetrievalEvidence uses it after score/lexical/rerank ties — fully tied 1.0-saturated candidates no longer order by arbitrary chunk id. Tie-only by construction; the clamped primary score is untouched. 1.3 OR-relaxation for weak-but-nonzero strict text matches (P8b extension): when strict-AND retrieval returns a sparse set (<3) or negligible top text_rank (<0.05), append OR-relaxed recall BEHIND the strict matches (append-only merge; strict results keep precedence). Kill switch: RAG_TEXT_WEAK_OR_RELAXATION=false. Telemetry: text_or_relaxation_used. 1.4 RC9 observability: fabricated (non-cosine) similarities from the document-lookup, memory-card, and table-facts paths are tagged similarity_origin="synthetic_text" and counted in telemetry (synthetic_similarity_count) ahead of any gate recalibration. Eval-gate note: eval:retrieval:quality (23/23) and eval:quality --rag-only need live keys and must be run before merge per the governance preflight; 1.3 ships behind the env kill switch for exactly that reason. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_011QiyE8jMm7VnrtknHF6jnJ --- .env.example | 3 + src/lib/clinical-search.ts | 1 + src/lib/env.ts | 8 + src/lib/rag.ts | 232 ++++++++++++++++++------- src/lib/retrieval-selection.ts | 6 + src/lib/types.ts | 12 ++ tests/clinical-search.test.ts | 14 ++ tests/rag-classifier-memo.test.ts | 126 ++++++++++++++ tests/retrieval-query-variants.test.ts | 37 ++++ tests/retrieval-selection.test.ts | 52 ++++++ 10 files changed, 433 insertions(+), 58 deletions(-) create mode 100644 tests/rag-classifier-memo.test.ts diff --git a/.env.example b/.env.example index 273e16f094..e6062d5bc7 100644 --- a/.env.example +++ b/.env.example @@ -65,6 +65,9 @@ RAG_PROVIDER_MODE=auto # Optional JSON override for app-layer ranking weights (see src/lib/ranking-config.ts). # Omit for current defaults. Example (enable diversity demotion + linear freshness): # RAG_RANKING_CONFIG={"documentDiversityPenalty":0.03,"freshness":{"mode":"linear"}} +# Append OR-relaxed recall behind weak-but-nonzero strict text matches (P8b extension). +# Set false to restore relax-only-on-empty behaviour (golden retrieval eval kill switch). +RAG_TEXT_WEAK_OR_RELAXATION=true RAG_ANSWER_CACHE_TTL_MS=300000 RAG_ANSWER_CACHE_SIZE=100 RAG_SEARCH_CACHE_TTL_MS=60000 diff --git a/src/lib/clinical-search.ts b/src/lib/clinical-search.ts index a1e983d686..9e976f2a1e 100644 --- a/src/lib/clinical-search.ts +++ b/src/lib/clinical-search.ts @@ -1714,6 +1714,7 @@ export function rankClinicalResults(query: string, results: SearchResult[]) { score_explanation: { ...entry.explanation, finalRank: index + 1, + preClampFinalScore: roundScore(entry.preClampFinalScore), }, })); diff --git a/src/lib/env.ts b/src/lib/env.ts index b17748051a..a37d23c856 100644 --- a/src/lib/env.ts +++ b/src/lib/env.ts @@ -73,6 +73,14 @@ const envSchema = z.object({ // Lets tuning/eval experiments adjust the second-stage rerank weights, document-diversity // demotion, and freshness decay WITHOUT a code change. Omitted/malformed => current defaults. RAG_RANKING_CONFIG: z.string().optional(), + // P8b extension: when strict-AND text retrieval returns weak-but-nonzero matches (sparse + // result set or negligible top text_rank), append OR-relaxed recall behind the strict + // matches. Kill switch for the golden retrieval eval: set false to restore + // relax-only-on-empty behaviour without a code change. + RAG_TEXT_WEAK_OR_RELAXATION: z + .enum(["true", "false"]) + .default("true") + .transform((value) => value === "true"), RAG_ANSWER_CACHE_TTL_MS: z.coerce.number().int().nonnegative().default(300000), RAG_ANSWER_CACHE_SIZE: z.coerce.number().int().nonnegative().default(100), RAG_SEARCH_CACHE_TTL_MS: z.coerce.number().int().nonnegative().default(60000), diff --git a/src/lib/rag.ts b/src/lib/rag.ts index 68d07c8c31..b1e21c8271 100644 --- a/src/lib/rag.ts +++ b/src/lib/rag.ts @@ -358,6 +358,12 @@ export type SearchTelemetry = { text_fast_path_latency_ms: number; text_candidate_budget?: number; text_fast_path_reason?: string | null; + // P8b extension: how OR-relaxation participated in the text layer. "empty_fallback" is the + // long-standing relax-on-zero path; "weak_augment" appends OR recall behind weak-but-nonzero + // strict matches (issue: strict-AND could bury the right chunk without ever relaxing). + text_or_relaxation_used?: "none" | "empty_fallback" | "weak_augment"; + // RC9 observability: how many final results carry a fabricated (non-cosine) similarity. + synthetic_similarity_count?: number; embedding_skipped: boolean; embedding_skip_reason?: string | null; embedding_latency_ms: number; @@ -548,6 +554,9 @@ function recordSearchScoreTelemetry(telemetry: SearchTelemetry, results: SearchR telemetry.score_spread = Number(Math.max(0, telemetry.top_score - telemetry.second_top_score).toFixed(4)); telemetry.score_distinct_documents = new Set(results.map((result) => result.document_id)).size; telemetry.retrieval_candidate_count = results.length; + telemetry.synthetic_similarity_count = results.filter( + (result) => result.similarity_origin === "synthetic_text", + ).length; telemetry.retrieval_provenance_counts = results.reduce>((counts, result) => { for (const layer of provenanceLayerKeys(result)) counts[layer] = (counts[layer] ?? 0) + 1; return counts; @@ -1215,7 +1224,103 @@ function uniqueTextValues(values: Array, limit = 32) return output; } -async function analyzeQueryWithClassifierFallback(query: string, analysis: ClinicalQueryAnalysis) { +type ClassifierVerdict = z.infer; + +// Finding #11 interim fix (docs/process-hardening.md): the LLM classifier verdict flips +// run-to-run for the same query, so the unsupported short-circuit downstream intermittently +// returned 0 results for valid in-corpus topics. Memoizing successful verdicts makes the +// verdict — and therefore retrieval behaviour — deterministic per query for the TTL window. +// Only *successful* classifier calls are memoized (accepted and rejected verdicts alike); +// transport errors and timeouts stay retryable, otherwise one transient 6s timeout would pin +// a query's classification for the whole TTL. The full corpus-grounded relevance fix remains +// scoped to RAG optimisation Phase 2. +const classifierVerdictMemoTtlMs = 15 * 60 * 1000; +const classifierVerdictMemoMaxEntries = 500; +const classifierVerdictMemo = new Map(); +const classifierVerdictInflight = new Map>(); + +function classifierVerdictMemoKey(query: string, analysis: ClinicalQueryAnalysis) { + const normalizedQuery = query.normalize("NFKC").toLowerCase().replace(/\s+/g, " ").trim(); + // The deterministic class + confidence bucket are part of the key so a deterministic-analyzer + // change invalidates stale verdicts instead of replaying them against a different baseline. + return `${normalizedQuery}::${analysis.queryClass}::${analysis.confidence.toFixed(2)}`; +} + +function storeClassifierVerdictMemo(key: string, verdict: ClassifierVerdict) { + if (classifierVerdictMemo.size >= classifierVerdictMemoMaxEntries) { + const oldestKey = classifierVerdictMemo.keys().next().value; + if (oldestKey !== undefined) classifierVerdictMemo.delete(oldestKey); + } + classifierVerdictMemo.set(key, { expiresAt: Date.now() + classifierVerdictMemoTtlMs, verdict }); +} + +export function resetClassifierVerdictMemoForTests() { + classifierVerdictMemo.clear(); + classifierVerdictInflight.clear(); +} + +async function requestClassifierVerdict(query: string, analysis: ClinicalQueryAnalysis): Promise { + const result = await generateStructuredTextResult( + [ + { + role: "user", + content: [ + { + type: "input_text", + text: [ + `Query: ${query}`, + `Deterministic query class: ${analysis.queryClass}`, + `Deterministic confidence: ${analysis.confidence}`, + `Known expanded terms: ${analysis.expandedTerms.join(", ") || "none"}`, + ].join("\n"), + }, + ], + }, + ], + queryClassifierOutputSchema, + { + model: env.OPENAI_FAST_ANSWER_MODEL, + maxOutputTokens: 220, + operation: "text_generation", + instructions: + "Classify this query for retrieval routing only. Do not answer the clinical question. Prefer unsupported when the query is not about indexed clinical document retrieval.", + reasoningEffort: "low", + textVerbosity: "low", + schemaName: "clinical_rag_query_classifier", + promptCacheKey: "clinical-rag-query-classifier-v1", + timeoutMs: 6000, + }, + ); + return queryClassifierParseSchema.parse(JSON.parse(result.text)); +} + +function applyClassifierVerdict(analysis: ClinicalQueryAnalysis, parsed: ClassifierVerdict): ClinicalQueryAnalysis { + if (parsed.confidence < 0.58 || parsed.queryClass === "unsupported_or_general") return analysis; + return { + ...analysis, + queryClass: parsed.queryClass, + confidence: Math.max(analysis.confidence, parsed.confidence), + needsClassifierFallback: false, + needsSynthesis: + analysis.needsSynthesis || + parsed.queryClass === "comparison" || + parsed.queryClass === "broad_summary" || + parsed.queryClass === "medication_dose_risk", + expandedTerms: uniqueTextValues([...analysis.expandedTerms, ...parsed.expandedTerms], 36), + queryRewrite: { + ...analysis.queryRewrite, + expansions: uniqueTextValues([...analysis.queryRewrite.expansions, ...parsed.expandedTerms], 48), + searchQuery: uniqueTextValues( + [analysis.queryRewrite.searchQuery, ...analysis.queryRewrite.expansions, ...parsed.expandedTerms], + 60, + ).join(" "), + reasons: uniqueTextValues([...analysis.queryRewrite.reasons, ...parsed.reasons, "classifier_fallback"], 16), + }, + reasons: uniqueTextValues([...analysis.reasons, ...parsed.reasons, "classifier_fallback"], 12), + } satisfies ClinicalQueryAnalysis; +} + +export async function analyzeQueryWithClassifierFallback(query: string, analysis: ClinicalQueryAnalysis) { if ( // Fail closed before any generative model call: an adversarial-manipulation // query is routed to "unsupported" downstream, so never send its text to the @@ -1229,63 +1334,28 @@ async function analyzeQueryWithClassifierFallback(query: string, analysis: Clini } if (!analysis.needsClassifierFallback || !env.OPENAI_API_KEY) return analysis; + const memoKey = classifierVerdictMemoKey(query, analysis); + const memoized = classifierVerdictMemo.get(memoKey); + if (memoized) { + if (memoized.expiresAt > Date.now()) return applyClassifierVerdict(analysis, memoized.verdict); + classifierVerdictMemo.delete(memoKey); + } + + let pending = classifierVerdictInflight.get(memoKey); + if (!pending) { + pending = requestClassifierVerdict(query, analysis).finally(() => { + classifierVerdictInflight.delete(memoKey); + }); + classifierVerdictInflight.set(memoKey, pending); + } + try { - const result = await generateStructuredTextResult( - [ - { - role: "user", - content: [ - { - type: "input_text", - text: [ - `Query: ${query}`, - `Deterministic query class: ${analysis.queryClass}`, - `Deterministic confidence: ${analysis.confidence}`, - `Known expanded terms: ${analysis.expandedTerms.join(", ") || "none"}`, - ].join("\n"), - }, - ], - }, - ], - queryClassifierOutputSchema, - { - model: env.OPENAI_FAST_ANSWER_MODEL, - maxOutputTokens: 220, - operation: "text_generation", - instructions: - "Classify this query for retrieval routing only. Do not answer the clinical question. Prefer unsupported when the query is not about indexed clinical document retrieval.", - reasoningEffort: "low", - textVerbosity: "low", - schemaName: "clinical_rag_query_classifier", - promptCacheKey: "clinical-rag-query-classifier-v1", - timeoutMs: 6000, - }, - ); - const parsed = queryClassifierParseSchema.parse(JSON.parse(result.text)); - if (parsed.confidence < 0.58 || parsed.queryClass === "unsupported_or_general") return analysis; - return { - ...analysis, - queryClass: parsed.queryClass, - confidence: Math.max(analysis.confidence, parsed.confidence), - needsClassifierFallback: false, - needsSynthesis: - analysis.needsSynthesis || - parsed.queryClass === "comparison" || - parsed.queryClass === "broad_summary" || - parsed.queryClass === "medication_dose_risk", - expandedTerms: uniqueTextValues([...analysis.expandedTerms, ...parsed.expandedTerms], 36), - queryRewrite: { - ...analysis.queryRewrite, - expansions: uniqueTextValues([...analysis.queryRewrite.expansions, ...parsed.expandedTerms], 48), - searchQuery: uniqueTextValues( - [analysis.queryRewrite.searchQuery, ...analysis.queryRewrite.expansions, ...parsed.expandedTerms], - 60, - ).join(" "), - reasons: uniqueTextValues([...analysis.queryRewrite.reasons, ...parsed.reasons, "classifier_fallback"], 16), - }, - reasons: uniqueTextValues([...analysis.reasons, ...parsed.reasons, "classifier_fallback"], 12), - } satisfies ClinicalQueryAnalysis; + const verdict = await pending; + storeClassifierVerdictMemo(memoKey, verdict); + return applyClassifierVerdict(analysis, verdict); } catch { + // Transport/parse failures are deliberately NOT memoized: fall back to the deterministic + // analysis for this request only, and let the next request retry the classifier. return analysis; } } @@ -2213,6 +2283,23 @@ export function relaxVariantToOrQuery(variant: string): string | null { return tokens.join(" OR "); } +// Mirrors the minimum meaningful text-signal floor used by answer routing +// (see textSignalFloor usage in rag-routing.ts): a strict-AND result set whose best +// text_rank sits below it carries almost no lexical evidence. +const weakTextMatchTopRankFloor = 0.05; +const weakTextMatchMinResultCount = 3; + +// Strict-AND matched something, but so weakly (sparse set or negligible text rank) that the +// right chunk may be buried outside the candidate pool. In that case OR-relaxed recall is +// appended BEHIND the strict matches (append-only: strict results keep merge precedence), so +// this can widen the pool but never displace a precise match. +export function shouldRelaxWeakTextMatches(merged: SearchResult[]): boolean { + if (merged.length === 0) return false; + if (merged.length < weakTextMatchMinResultCount) return true; + const topTextRank = merged.reduce((top, result) => Math.max(top, result.text_rank ?? 0), 0); + return topTextRank < weakTextMatchTopRankFloor; +} + async function searchTextChunkCandidates(args: { supabase: ReturnType; queryVariants: string[]; @@ -2220,6 +2307,7 @@ async function searchTextChunkCandidates(args: { documentIds?: string[]; allowGlobalSearch?: boolean; matchCount: number; + telemetry?: SearchTelemetry; }) { const runChunkText = async (queryText: string, matchCount: number) => { const { data, error } = await args.supabase.rpc("match_document_chunks_text", { @@ -2241,7 +2329,25 @@ async function searchTextChunkCandidates(args: { (accumulated, resultSet) => mergeSearchResults(resultSet, accumulated), [] as SearchResult[], ); - if (merged.length > 0) return merged; + if (args.telemetry) args.telemetry.text_or_relaxation_used = "none"; + if (merged.length > 0) { + // P8b extension: strict-AND matched, but weakly. Append OR-relaxed recall behind the + // strict matches so a buried chunk can enter the candidate pool without displacing any + // precise match (mergeSearchResults keeps primary/strict precedence). Trigram correction + // is not run here — it exists to rescue *empty* strict retrieval (RC6), and correcting a + // query that already matched would second-guess a working query. + if (env.RAG_TEXT_WEAK_OR_RELAXATION && shouldRelaxWeakTextMatches(merged)) { + const weakRelaxed = relaxVariantToOrQuery(variants[0] ?? ""); + if (weakRelaxed) { + const orResults = await runChunkText(weakRelaxed, Math.min(args.matchCount, 24)); + if (orResults.length > 0) { + if (args.telemetry) args.telemetry.text_or_relaxation_used = "weak_augment"; + return mergeSearchResults(merged, orResults); + } + } + } + return merged; + } // Strict AND variants matched nothing. Two fallbacks, in order: // (item 10, RC6) a typo the hard-coded map misses can block an otherwise-precise query — so first @@ -2267,7 +2373,10 @@ async function searchTextChunkCandidates(args: { const relaxed = relaxVariantToOrQuery(effectivePrimary); if (relaxed) { const relaxedResults = await runChunkText(relaxed, args.matchCount); - if (relaxedResults.length > 0) return relaxedResults; + if (relaxedResults.length > 0) { + if (args.telemetry) args.telemetry.text_or_relaxation_used = "empty_fallback"; + return relaxedResults; + } } return merged; } @@ -2538,8 +2647,10 @@ async function searchDocumentLookupFastPath(args: { if (!document) continue; const documentScore = scoreByDocument.get(chunk.document_id) ?? 0; const chunkScore = documentLookupChunkScore(chunk, terms); + // Not a cosine: fabricated from title/label match strength (RC9 — tagged below). const similarity = Math.min(0.92, 0.58 + documentScore + Math.min(0.12, chunkScore * 0.08)); results.push({ + similarity_origin: "synthetic_text", id: chunk.id, document_id: chunk.document_id, title: document.title, @@ -2713,8 +2824,10 @@ async function loadChunksForMemoryCards( const committedGeneration = committedIndexGeneration(document.metadata); if (chunk.index_generation_id && chunk.index_generation_id !== committedGeneration) return null; const card = bestCardByChunk.get(chunk.id); + // Not a cosine: fabricated from memory-card confidence (RC9 — tagged below). const similarity = Math.min(0.92, 0.58 + (card?.confidence ?? 0.5) * 0.28); return { + similarity_origin: "synthetic_text" as const, id: chunk.id, document_id: chunk.document_id, title: document.title, @@ -2796,6 +2909,8 @@ async function loadChunksForSignalMatches(args: { retrieval_synopsis: chunk.retrieval_synopsis ?? null, image_ids: chunk.image_ids ?? [], source_metadata: normalizeSourceMetadata(document.metadata), + // ChunkSignalMatch similarities are fabricated from table-fact text rank (RC9). + similarity_origin: "synthetic_text" as const, similarity: match.similarity, text_rank: match.textRank, hybrid_score: match.hybridScore, @@ -5558,6 +5673,7 @@ export async function searchChunksWithTelemetry(args: SearchChunksArgs) { documentIds: documentFilterList, allowGlobalSearch: args.allowGlobalSearch, matchCount: textCandidateCount, + telemetry, }); telemetry.text_candidate_count = textData.length; telemetry.text_fast_path_latency_ms = Date.now() - textRpcStartedAt; diff --git a/src/lib/retrieval-selection.ts b/src/lib/retrieval-selection.ts index eaf7bef498..870129f762 100644 --- a/src/lib/retrieval-selection.ts +++ b/src/lib/retrieval-selection.ts @@ -449,6 +449,7 @@ export function buildRetrievalCandidates( lexicalScore: 0, semanticScore: result.similarity, rerankScore: result.score_explanation?.finalScore ?? result.hybrid_score, + preClampScore: result.score_explanation?.preClampFinalScore, matchedSignals: [], sourceHref: documentCitationHref(citationFromResult(result)), }; @@ -557,6 +558,11 @@ export function selectRetrievalEvidence(args: { if ((right.lexicalScore ?? 0) !== (left.lexicalScore ?? 0)) return (right.lexicalScore ?? 0) - (left.lexicalScore ?? 0); if ((right.rerankScore ?? 0) !== (left.rerankScore ?? 0)) return (right.rerankScore ?? 0) - (left.rerankScore ?? 0); + // rerankScore is the clamped finalScore, which saturates at 1.0 for heavily-boosted results; + // the pre-clamp sum still discriminates within that saturated region. Tie-only by + // construction — every primary signal above has already tied when this fires. + if ((right.preClampScore ?? 0) !== (left.preClampScore ?? 0)) + return (right.preClampScore ?? 0) - (left.preClampScore ?? 0); return left.chunkId.localeCompare(right.chunkId); }); const selectedCandidates: RetrievalCandidate[] = []; diff --git a/src/lib/types.ts b/src/lib/types.ts index a00bcf3e01..7202600e35 100644 --- a/src/lib/types.ts +++ b/src/lib/types.ts @@ -173,6 +173,9 @@ export type RetrievalCandidate = { lexicalScore?: number; semanticScore?: number; rerankScore?: number; + // Deep tiebreak only (see SearchScoreExplanation.preClampFinalScore): discriminates + // candidates whose clamped scores all saturate at 1.0. Never a primary ordering signal. + preClampScore?: number; matchedSignals: string[]; sourceHref?: string; }; @@ -302,6 +305,11 @@ export type SearchResult = { retrieval_synopsis?: string | null; image_ids: string[]; similarity: number; + // RC9 observability: "synthetic_text" marks a `similarity` fabricated from lexical/structural + // signals (document-lookup, memory-card, table-facts fast paths) rather than a real cosine. + // Coverage/threshold gates are calibrated for cosine values; this tag lets telemetry measure + // how often synthetic scores cross those gates before any recalibration. + similarity_origin?: "cosine" | "synthetic_text"; text_rank?: number; hybrid_score?: number; // Lexical/keyword relevance (0-1) for text-only fallback rows. This is NOT a @@ -407,6 +415,10 @@ export type SearchScoreExplanation = { rawPenalty?: number; finalScore: number; finalRank?: number; + // Pre-clamp boost sum: finalScore saturates at 1.0 for heavily-boosted results, so this + // carries the discrimination the clamp discards. Used only as a deep tiebreak — it must + // never reorder results above finalScore. + preClampFinalScore?: number; strategy: "weighted_hybrid" | "weighted_hybrid_rrf_blend"; }; diff --git a/tests/clinical-search.test.ts b/tests/clinical-search.test.ts index 2c825f38f3..37ea3ad4cf 100644 --- a/tests/clinical-search.test.ts +++ b/tests/clinical-search.test.ts @@ -871,3 +871,17 @@ describe("clinical rank score bounding and penalty caps (RET-H1, RET-H2)", () => expect(ranked[0].id).toBe("dose-table-row"); }); }); + +describe("pre-clamp final score emission", () => { + it("emits preClampFinalScore on every ranked result for downstream tie-breaking", () => { + const ranked = rankClinicalResults("clozapine monitoring requirements", [ + result({ id: "a", title: "Clozapine Prescribing and Monitoring", hybrid_score: 0.9 }), + result({ id: "b", title: "General Notes", hybrid_score: 0.4 }), + ]); + + for (const item of ranked) { + expect(typeof item.score_explanation?.preClampFinalScore).toBe("number"); + expect(Number.isFinite(item.score_explanation?.preClampFinalScore)).toBe(true); + } + }); +}); diff --git a/tests/rag-classifier-memo.test.ts b/tests/rag-classifier-memo.test.ts new file mode 100644 index 0000000000..a19310dc84 --- /dev/null +++ b/tests/rag-classifier-memo.test.ts @@ -0,0 +1,126 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; + +// Finding #11 interim fix: the LLM classifier verdict must be deterministic per query for +// the memo TTL window, so the unsupported short-circuit cannot flip run-to-run and return +// 0 results for a valid in-corpus topic on some runs but not others. + +afterEach(() => { + vi.unstubAllEnvs(); + vi.resetModules(); + vi.useRealTimers(); +}); + +async function loadWithClassifierMock(mock: ReturnType) { + vi.stubEnv("OPENAI_API_KEY", "test-key"); + vi.doMock("@/lib/openai", async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, generateStructuredTextResult: mock }; + }); + const rag = await import("../src/lib/rag"); + const { analyzeClinicalQuery } = await import("../src/lib/clinical-search"); + rag.resetClassifierVerdictMemoForTests(); + return { rag, analyzeClinicalQuery }; +} + +function classifierResponse(overrides: Record = {}) { + return { + text: JSON.stringify({ + queryClass: "broad_summary", + confidence: 0.9, + reasons: ["classifier_test"], + expandedTerms: ["mood disorder"], + ...overrides, + }), + }; +} + +function fallbackQueryAnalysis( + analyzeClinicalQuery: (typeof import("../src/lib/clinical-search"))["analyzeClinicalQuery"], +) { + // A bare condition query is exactly the class that needs the LLM fallback (deterministic + // confidence below 0.58 with class unsupported_or_general) — the finding #11 shape. + const query = "bipolar disorder"; + const analysis = analyzeClinicalQuery(query); + expect(analysis.needsClassifierFallback).toBe(true); + return { query, analysis }; +} + +describe("classifier verdict memoization", () => { + it("does not re-call the model for a repeated query and returns an identical verdict", async () => { + const mock = vi.fn(async () => classifierResponse()); + const { rag, analyzeClinicalQuery } = await loadWithClassifierMock(mock); + const { query, analysis } = fallbackQueryAnalysis(analyzeClinicalQuery); + + const first = await rag.analyzeQueryWithClassifierFallback(query, analysis); + const second = await rag.analyzeQueryWithClassifierFallback(query, analysis); + + expect(mock).toHaveBeenCalledTimes(1); + expect(first.queryClass).toBe("broad_summary"); + expect(second.queryClass).toBe("broad_summary"); + expect(second).toEqual(first); + }); + + it("memoizes rejected verdicts so a rejection is also deterministic", async () => { + const mock = vi.fn(async () => classifierResponse({ confidence: 0.3 })); + const { rag, analyzeClinicalQuery } = await loadWithClassifierMock(mock); + const { query, analysis } = fallbackQueryAnalysis(analyzeClinicalQuery); + + const first = await rag.analyzeQueryWithClassifierFallback(query, analysis); + const second = await rag.analyzeQueryWithClassifierFallback(query, analysis); + + expect(mock).toHaveBeenCalledTimes(1); + // Rejected verdict (confidence < 0.58) leaves the deterministic analysis untouched. + expect(first).toBe(analysis); + expect(second).toBe(analysis); + }); + + it("does not memoize transport errors — the next request retries the classifier", async () => { + const mock = vi + .fn() + .mockRejectedValueOnce(new Error("timeout")) + .mockResolvedValueOnce(classifierResponse()); + const { rag, analyzeClinicalQuery } = await loadWithClassifierMock(mock); + const { query, analysis } = fallbackQueryAnalysis(analyzeClinicalQuery); + + const first = await rag.analyzeQueryWithClassifierFallback(query, analysis); + const second = await rag.analyzeQueryWithClassifierFallback(query, analysis); + + expect(mock).toHaveBeenCalledTimes(2); + expect(first).toBe(analysis); + expect(second.queryClass).toBe("broad_summary"); + }); + + it("deduplicates concurrent in-flight calls for the same query", async () => { + let resolveCall: ((value: { text: string }) => void) | undefined; + const mock = vi.fn( + () => + new Promise<{ text: string }>((resolve) => { + resolveCall = resolve; + }), + ); + const { rag, analyzeClinicalQuery } = await loadWithClassifierMock(mock); + const { query, analysis } = fallbackQueryAnalysis(analyzeClinicalQuery); + + const firstPromise = rag.analyzeQueryWithClassifierFallback(query, analysis); + const secondPromise = rag.analyzeQueryWithClassifierFallback(query, analysis); + resolveCall?.(classifierResponse()); + const [first, second] = await Promise.all([firstPromise, secondPromise]); + + expect(mock).toHaveBeenCalledTimes(1); + expect(first.queryClass).toBe("broad_summary"); + expect(second.queryClass).toBe("broad_summary"); + }); + + it("re-calls the model after the memo TTL expires", async () => { + vi.useFakeTimers({ now: new Date("2026-07-06T00:00:00Z") }); + const mock = vi.fn(async () => classifierResponse()); + const { rag, analyzeClinicalQuery } = await loadWithClassifierMock(mock); + const { query, analysis } = fallbackQueryAnalysis(analyzeClinicalQuery); + + await rag.analyzeQueryWithClassifierFallback(query, analysis); + vi.setSystemTime(new Date("2026-07-06T00:16:00Z")); + await rag.analyzeQueryWithClassifierFallback(query, analysis); + + expect(mock).toHaveBeenCalledTimes(2); + }); +}); diff --git a/tests/retrieval-query-variants.test.ts b/tests/retrieval-query-variants.test.ts index a2f06474e1..e5b98174e4 100644 --- a/tests/retrieval-query-variants.test.ts +++ b/tests/retrieval-query-variants.test.ts @@ -11,6 +11,7 @@ import { shouldApplyUnsupportedSearchShortCircuit, textCandidateBudgetForQueryClass, relaxVariantToOrQuery, + shouldRelaxWeakTextMatches, } from "../src/lib/rag"; import type { SearchResult } from "../src/lib/types"; @@ -1034,3 +1035,39 @@ describe("relaxVariantToOrQuery (8b over-conjunction fallback)", () => { expect(relaxVariantToOrQuery("clozapine")).toBeNull(); }); }); + +describe("shouldRelaxWeakTextMatches (P8b weak-augment)", () => { + it("never fires on an empty strict result set (that is the empty_fallback path)", () => { + expect(shouldRelaxWeakTextMatches([])).toBe(false); + }); + + it("fires when strict-AND returned a sparse result set", () => { + expect(shouldRelaxWeakTextMatches([result({ text_rank: 0.4 })])).toBe(true); + expect(shouldRelaxWeakTextMatches([result({ id: "a", text_rank: 0.4 }), result({ id: "b", text_rank: 0.3 })])).toBe( + true, + ); + }); + + it("fires when the best strict text rank is below the meaningful-signal floor", () => { + const weak = [ + result({ id: "a", text_rank: 0.01 }), + result({ id: "b", text_rank: 0.02 }), + result({ id: "c", text_rank: 0.04 }), + ]; + expect(shouldRelaxWeakTextMatches(weak)).toBe(true); + }); + + it("does not fire when strict-AND already carries meaningful lexical evidence", () => { + const strong = [ + result({ id: "a", text_rank: 0.4 }), + result({ id: "b", text_rank: 0.2 }), + result({ id: "c", text_rank: 0.1 }), + ]; + expect(shouldRelaxWeakTextMatches(strong)).toBe(false); + }); + + it("treats a missing text_rank as no lexical evidence", () => { + const missing = [result({ id: "a" }), result({ id: "b" }), result({ id: "c" })]; + expect(shouldRelaxWeakTextMatches(missing)).toBe(true); + }); +}); diff --git a/tests/retrieval-selection.test.ts b/tests/retrieval-selection.test.ts index fcc420c4f5..990bc6ed35 100644 --- a/tests/retrieval-selection.test.ts +++ b/tests/retrieval-selection.test.ts @@ -627,3 +627,55 @@ describe("retrieval source selection", () => { expect(selection.results[0].source_metadata?.clinical_validation_status).toBe("unverified"); }); }); + +describe("saturated-score tie-breaking (pre-clamp)", () => { + function saturatedExplanation(preClampFinalScore: number): NonNullable { + return { + vectorScore: 0.9, + textRank: 0.3, + lexicalCoverageScore: 0.5, + metadataMatchScore: 0.2, + sectionTitleMatchBoost: 0.1, + freshnessRecencyBoost: 0, + weightedHybridScore: 0.9, + rrfScore: null, + rrfBoost: 0, + memoryBoost: 0, + titleBoost: 0.3, + metadataBoost: 0.2, + clinicalSignalBoost: 0.3, + penalty: 0, + finalScore: 1, + preClampFinalScore, + strategy: "weighted_hybrid", + }; + } + + it("orders fully-tied saturated candidates by pre-clamp score, not chunk id", () => { + // Both results are identical on every primary signal (score, lexical, rerank all tie at the + // 1.0 clamp). Without the pre-clamp tiebreak, ordering would fall through to + // chunkId.localeCompare and pick "chunk-a" first; the higher pre-clamp sum must win instead. + const higherPreClamp = source({ + id: "chunk-b", + hybrid_score: 1, + similarity: 0.9, + score_explanation: saturatedExplanation(1.8), + }); + const lowerPreClamp = source({ + id: "chunk-a", + hybrid_score: 1, + similarity: 0.9, + score_explanation: saturatedExplanation(1.2), + }); + + const selection = selectRetrievalEvidence({ + query: "clinical guidance", + queryClass: "broad_summary", + results: [lowerPreClamp, higherPreClamp], + topK: 2, + maxResultsPerDocument: 2, + }); + + expect(selection.results.map((item) => item.id)).toEqual(["chunk-b", "chunk-a"]); + }); +}); From 789c3ea3b89fde673af18004f87d8840c56505a5 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 11:15:41 +0000 Subject: [PATCH 3/6] Unify registry catalog search into shared primitives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 2 of the search hardening + universal search workstream: - New src/lib/catalog-search.ts: one canonical normalizeSearchText (the medications superset — NFKD + diacritic strip, keeps + . / - for dose strings) replacing four divergent per-domain normalizers, and a generic rankCatalogRecords weighted ranker returning match signals so each domain wrapper keeps its historical reason labels and match shapes. - medications/services/forms rankers become thin wrappers with their exact historical weights; tests/{medications,services,forms}.test.ts pass unmodified as parity oracles. - Differentials get real relevance ranking (rankDifferentialRecords) with alias expansion, replacing the unranked includes() OR-filter; empty-query full-catalogue browse preserved. - Forms stop masquerading as documents: app-modes gains a "forms" search/result kind, removing the ClinicalDashboard special case; forms no longer qualify as a source-library shortcut (scope-tag clicks fall back to documents mode). Prescribing deliberately keeps kind:"documents" — it searches the corpus for dosing guidance (rationale recorded inline). - Tools dataset deduplicated into src/lib/tools-catalog.ts (icon-free so server code can use it) + rankToolRecords; the launcher derives its apps from it and the mockup fixtures derive identity fields from it. - Amend Phase-1 OR-relaxation heuristic: a sparse strict set anchored by a strong lexical hit (top text_rank >= 0.3) no longer augments — exact single-match lookups stay one-RPC retrievals (caught by the answer coalescing test asserting a single text RPC). New tests: catalog-search, tools-catalog; extended differentials and app-modes contracts; sitemap regenerated (mode kind is embedded there). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_011QiyE8jMm7VnrtknHF6jnJ --- docs/site-map.md | 2 +- src/components/ClinicalDashboard.tsx | 8 +- src/components/applications-launcher-page.tsx | 267 ++-------------- .../clinical-dashboard/dashboard-nav.tsx | 5 +- .../master-search-header.tsx | 2 +- .../tools-page-mockups/tool-fixtures.ts | 65 ++-- src/lib/app-modes.ts | 14 +- src/lib/catalog-search.ts | 145 +++++++++ src/lib/differentials.ts | 97 +++--- src/lib/forms.ts | 75 ++--- src/lib/medications.ts | 91 +++--- src/lib/rag.ts | 18 +- src/lib/services.ts | 79 ++--- src/lib/tools-catalog.ts | 284 ++++++++++++++++++ tests/app-modes.test.ts | 10 +- tests/catalog-search.test.ts | 122 ++++++++ tests/differentials.test.ts | 16 + tests/retrieval-query-variants.test.ts | 16 +- tests/tools-catalog.test.ts | 37 +++ 19 files changed, 882 insertions(+), 471 deletions(-) create mode 100644 src/lib/catalog-search.ts create mode 100644 src/lib/tools-catalog.ts create mode 100644 tests/catalog-search.test.ts create mode 100644 tests/tools-catalog.test.ts diff --git a/docs/site-map.md b/docs/site-map.md index ca5a37db39..c522d23950 100644 --- a/docs/site-map.md +++ b/docs/site-map.md @@ -23,7 +23,7 @@ This file is generated by `npm run sitemap:update`. Run `npm run sitemap:check` - `/?mode=answer` - Answer mode. Search kind: `answer`. Query example: `/?mode=answer&q=example+question&focus=1&run=1`. - `/?mode=documents` - Documents mode. Search kind: `documents`. Query example: `/documents/search?mode=documents&q=lithium+monitoring&focus=1&run=1`. - `/services` - Services mode. Search kind: `services`. Query example: `/services?q=13YARN&focus=1&run=1`. -- `/forms` - Forms mode. Search kind: `documents`. Query example: `/forms?q=transport+forms&focus=1&run=1`. +- `/forms` - Forms mode. Search kind: `forms`. Query example: `/forms?q=transport+forms&focus=1&run=1`. - `/favourites` - Favourites mode. Search kind: `favourites`. Query example: `/favourites?q=clozapine+set&focus=1&run=1`. - `/differentials` - Differentials mode. Search kind: `differentials`. Query example: `/differentials?q=acute+confusion&focus=1&run=1`. - `/?mode=prescribing` - Medication mode. Search kind: `documents`. Query example: `/?mode=prescribing&q=acamprosate+renal+dose&focus=1&run=1`. diff --git a/src/components/ClinicalDashboard.tsx b/src/components/ClinicalDashboard.tsx index d449062956..b9be3256d2 100644 --- a/src/components/ClinicalDashboard.tsx +++ b/src/components/ClinicalDashboard.tsx @@ -1123,7 +1123,10 @@ function buildMobileSectionFabState({ }; } return { - statusLabel: modeSearch.resultKind === "documents" ? modeSearch.statusLabel : "No answer yet", + statusLabel: + modeSearch.resultKind === "documents" || modeSearch.resultKind === "forms" + ? modeSearch.statusLabel + : "No answer yet", statusTone: "empty", nextStep: modeSearch.nextStep, badgeLabel: modeSearch.badgeLabel, @@ -2445,6 +2448,7 @@ export function ClinicalDashboard({ const shouldRun = params.get("run") === "1" || modeSearch.kind === "documents" || + modeSearch.kind === "forms" || modeSearch.kind === "favourites" || modeSearch.kind === "differentials"; if (!shouldRun) return; @@ -2712,7 +2716,7 @@ export function ClinicalDashboard({ setActionNotice({ tone: "success", message: "Favourites filtered from the composer." }); return; } - if (modeSearch.kind === "services" || targetMode === "forms") { + if (modeSearch.kind === "services" || modeSearch.kind === "forms") { resetAnswerThread(); setAnswer(null); setSources([]); diff --git a/src/components/applications-launcher-page.tsx b/src/components/applications-launcher-page.tsx index 6145f852ee..f8b9fbf271 100644 --- a/src/components/applications-launcher-page.tsx +++ b/src/components/applications-launcher-page.tsx @@ -26,31 +26,18 @@ import { type FormEvent, useEffect, useMemo, useState } from "react"; import { ModeHomeVerificationFooter } from "@/components/mode-home-template"; import { cn } from "@/components/ui-primitives"; -type LauncherStatus = "ready" | "recent" | "review_due"; -type LauncherArea = "assessment" | "reference" | "care" | "coordination" | "saved"; +import { + toolCatalogRecords, + type ToolCatalogArea, + type ToolCatalogRecord, + type ToolCatalogStatus, +} from "@/lib/tools-catalog"; + +type LauncherStatus = ToolCatalogStatus; +type LauncherArea = ToolCatalogArea; type LauncherFilter = "all" | LauncherArea | "more"; -type LauncherApp = { - id: string; - title: string; - mobileTitle?: string; - description: string; - bestFor: string; - detail: string; - href: string; - external?: boolean; - icon: LucideIcon; - area: LauncherArea; - status: LauncherStatus; - sourceBacked: boolean; - safetyFirst?: boolean; - highYield?: boolean; - actionLabel: string; - keywords: string[]; - checkFirst: string[]; - neededInput: string[]; - output: string; -}; +type LauncherApp = ToolCatalogRecord & { icon: LucideIcon }; const focusRing = "focus-visible:outline focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-[color:var(--focus)]"; @@ -81,220 +68,26 @@ const iconToneClasses: Record = { + "clinical-kb-search": Search, + differentials: Brain, + documents: FileText, + guidelines: ShieldCheck, + "risk-safety": ShieldCheck, + "medication-prescribing": Pill, + services: Users, + forms: FileCheck2, + "care-plans": ClipboardCheck, + monitoring: Waves, + favourites: Star, +}; + +const launcherApps: LauncherApp[] = toolCatalogRecords.map((record) => ({ + ...record, + icon: launcherIconById[record.id] ?? Sparkles, +})); const toolsLauncherCopy = { heading: "Tools", diff --git a/src/components/clinical-dashboard/dashboard-nav.tsx b/src/components/clinical-dashboard/dashboard-nav.tsx index 4ffe93ec21..3a34cbc91a 100644 --- a/src/components/clinical-dashboard/dashboard-nav.tsx +++ b/src/components/clinical-dashboard/dashboard-nav.tsx @@ -89,7 +89,10 @@ export function buildMobileSectionFabState({ }; } return { - statusLabel: modeSearch.resultKind === "documents" ? modeSearch.statusLabel : "No answer yet", + statusLabel: + modeSearch.resultKind === "documents" || modeSearch.resultKind === "forms" + ? modeSearch.statusLabel + : "No answer yet", statusTone: "empty", nextStep: modeSearch.nextStep, badgeLabel: modeSearch.badgeLabel, diff --git a/src/components/clinical-dashboard/master-search-header.tsx b/src/components/clinical-dashboard/master-search-header.tsx index 6feda36919..ce5c991302 100644 --- a/src/components/clinical-dashboard/master-search-header.tsx +++ b/src/components/clinical-dashboard/master-search-header.tsx @@ -288,7 +288,7 @@ export function MasterSearchHeader({ const isHeroDesktopComposer = desktopSearchPlacement === "hero" && isMobileBottomComposer; const canRunLocalSearch = selectedSearch.kind === "documents" || - searchMode === "forms" || + selectedSearch.kind === "forms" || selectedSearch.kind === "services" || selectedSearch.kind === "tools" || selectedSearch.kind === "favourites"; diff --git a/src/components/tools-page-mockups/tool-fixtures.ts b/src/components/tools-page-mockups/tool-fixtures.ts index 00d1238da9..cbd395bafc 100644 --- a/src/components/tools-page-mockups/tool-fixtures.ts +++ b/src/components/tools-page-mockups/tool-fixtures.ts @@ -1,5 +1,6 @@ import { Brain, FileCheck2, FileText, Pill, Search, Star, type LucideIcon } from "lucide-react"; import { appModeIcons } from "@/lib/app-mode-icons"; +import { toolCatalogRecordById } from "@/lib/tools-catalog"; export type ToolStatus = "ready" | "review_due" | "recent"; export type ToolArea = "reference" | "assessment" | "care" | "coordination" | "personal"; @@ -18,15 +19,25 @@ export type ToolFixture = { secondary: string; }; -export const tools: ToolFixture[] = [ +// Identity (title/description/href/sourceBacked) comes from the shared tools catalog +// (src/lib/tools-catalog.ts); only mockup-specific presentation extras live here. +type ToolFixtureExtras = { + id: string; + icon: LucideIcon; + area: ToolArea; + status: ToolStatus; + lastUsed: string; + primaryAction: string; + secondary: string; + title?: string; + description?: string; +}; + +const fixtureExtras: ToolFixtureExtras[] = [ { id: "clinical-kb-search", - title: "Clinical KB Search", - description: "Ask source-backed clinical questions and move straight to evidence.", - href: "/?mode=answer", icon: Search, area: "reference", - sourceBacked: true, status: "ready", lastUsed: "Today, 7:30 AM", primaryAction: "Ask", @@ -34,12 +45,8 @@ export const tools: ToolFixture[] = [ }, { id: "documents", - title: "Documents", - description: "Search indexed PDFs, policies, guidelines, pages, tables, and images.", - href: "/?mode=documents", icon: FileText, area: "reference", - sourceBacked: true, status: "ready", lastUsed: "May 10, 2025", primaryAction: "Search", @@ -47,12 +54,8 @@ export const tools: ToolFixture[] = [ }, { id: "differentials", - title: "Differentials", - description: "Build and compare diagnostic possibilities with source-aware prompts.", - href: "/differentials", icon: Brain, area: "assessment", - sourceBacked: true, status: "recent", lastUsed: "Today, 8:40 AM", primaryAction: "Compare", @@ -60,12 +63,8 @@ export const tools: ToolFixture[] = [ }, { id: "medication-prescribing", - title: "Medication Prescribing", - description: "Review prescribing context, monitoring, interactions, and cautions.", - href: "/?mode=prescribing", icon: Pill, area: "care", - sourceBacked: true, status: "review_due", lastUsed: "May 12, 2025", primaryAction: "Prescribe", @@ -73,12 +72,8 @@ export const tools: ToolFixture[] = [ }, { id: "services", - title: "Services", - description: "Open source-backed service records, referral routes, and eligibility.", - href: "/services", icon: appModeIcons.services, area: "coordination", - sourceBacked: true, status: "review_due", lastUsed: "Today, 8:15 AM", primaryAction: "Refer", @@ -86,12 +81,8 @@ export const tools: ToolFixture[] = [ }, { id: "forms", - title: "Forms", - description: "Find clinical forms and source-backed readiness pathways.", - href: "/forms", icon: FileCheck2, area: "coordination", - sourceBacked: true, status: "ready", lastUsed: "Today, 8:05 AM", primaryAction: "Open", @@ -99,19 +90,35 @@ export const tools: ToolFixture[] = [ }, { id: "favourites", - title: "Favourites", - description: "Return to saved clinical work, sources, and repeated workflows.", - href: "/favourites", icon: Star, area: "personal", - sourceBacked: false, status: "recent", lastUsed: "Today, 8:45 AM", primaryAction: "Resume", secondary: "Saved items, recent work, pins", + // The mockups keep the shorter historical framing for this entry. + title: "Favourites", + description: "Return to saved clinical work, sources, and repeated workflows.", }, ]; +export const tools: ToolFixture[] = fixtureExtras.map((extras) => { + const record = toolCatalogRecordById(extras.id); + return { + id: record.id, + title: extras.title ?? record.title, + description: extras.description ?? record.description, + href: record.href, + sourceBacked: record.sourceBacked, + icon: extras.icon, + area: extras.area, + status: extras.status, + lastUsed: extras.lastUsed, + primaryAction: extras.primaryAction, + secondary: extras.secondary, + }; +}); + export const pinnedToolIds = ["clinical-kb-search", "documents", "medication-prescribing", "services"] as const; export const areaLabels: Record = { diff --git a/src/lib/app-modes.ts b/src/lib/app-modes.ts index 055f04a983..088a5a37c1 100644 --- a/src/lib/app-modes.ts +++ b/src/lib/app-modes.ts @@ -5,8 +5,8 @@ export type AppModeId = "answer" | "documents" | "services" | "forms" | "favourites" | "differentials" | "prescribing" | "tools"; export type SearchableAppModeId = AppModeId; -export type AppModeSearchKind = "answer" | "documents" | "services" | "favourites" | "differentials" | "tools"; -export type AppModeResultKind = "answer" | "documents" | "services" | "favourites" | "differentials" | "tools"; +export type AppModeSearchKind = "answer" | "documents" | "services" | "forms" | "favourites" | "differentials" | "tools"; +export type AppModeResultKind = "answer" | "documents" | "services" | "forms" | "favourites" | "differentials" | "tools"; export type AppModeSearchConfig = { kind: AppModeSearchKind; @@ -106,7 +106,9 @@ export const appModeDefinitions = [ description: "Clinical forms and pathways", href: "/forms", search: { - kind: "documents", + // Forms are a registry catalogue, not corpus documents. Declaring the honest kind + // removes the ClinicalDashboard special-casing that the old kind:"documents" forced. + kind: "forms", placeholder: "Search forms...", inputAriaLabel: "Search forms, source records, pathways, and criteria", submitIdleLabel: "Forms", @@ -115,7 +117,7 @@ export const appModeDefinitions = [ emptyTitle: "Enter a form search term", readyTitle: "Search forms", progressLabel: "Searching form records.", - resultKind: "documents", + resultKind: "forms", resultHeading: "Form matches", statusLabel: "Forms", nextStep: "Review matching form records", @@ -173,6 +175,9 @@ export const appModeDefinitions = [ description: "Prescribing checks and guidance", href: "/?mode=prescribing", search: { + // Deliberately kind:"documents" (unlike forms): prescribing intentionally searches the + // document corpus for dosing/threshold guidance (defaultQueryMode dose_threshold_lookup). + // The medication registry joins cross-entity search via /api/search/universal instead. kind: "documents", placeholder: "Search medications...", inputAriaLabel: "Search medication guidance", @@ -299,6 +304,7 @@ export function isSearchableAppMode(modeId: string): modeId is SearchableAppMode kind === "answer" || kind === "documents" || kind === "services" || + kind === "forms" || kind === "favourites" || kind === "differentials" || kind === "tools" diff --git a/src/lib/catalog-search.ts b/src/lib/catalog-search.ts new file mode 100644 index 0000000000..8b1e1686d4 --- /dev/null +++ b/src/lib/catalog-search.ts @@ -0,0 +1,145 @@ +// Shared search primitives for the registry catalogs (medications, services, forms, +// differentials, tools). Before this module each domain re-implemented its own text +// normalizer (with divergent regexes, so the same query tokenized differently per domain) +// and its own weighted includes() ranker. The domain rankers are thin wrappers over +// rankCatalogRecords with their historical field weights; the wrapper owns its reason +// labels and match shape so existing API/UI contracts are unchanged. + +// Canonical normalizer (the medications implementation — the superset of the retired +// services/forms variants: NFKD + diacritic strip, and `+ . / -` survive so dose strings +// ("5+5", "0.5mg", "IM/PO") and hyphenated clinical terms stay searchable). +export function normalizeSearchText(value: string) { + return value + .toLowerCase() + .normalize("NFKD") + .replace(/[\u0300-\u036f]/g, "") + .replace(/[^a-z0-9+./\s-]/g, " ") + .replace(/\s+/g, " ") + .trim(); +} + +export function compactSearchText(value: string) { + return value.replace(/\s+/g, ""); +} + +export type CatalogField = { + // Wrapper-facing key used to build human-readable reasons (e.g. "title", "contact"). + id: string; + weight: number; + text: (record: T) => string; +}; + +export type CatalogMatchSignals = { + // Matched term count per field id (only fields with at least one match are present). + fields: Record; + // Matched term count against the full-text haystack. + content: number; + compact: boolean; + phrase: boolean; + exact: boolean; + broad: boolean; +}; + +export type CatalogRankedMatch = { + record: T; + score: number; + signals: CatalogMatchSignals; +}; + +export type RankCatalogOptions = { + fields: Array>; + // The widest haystack for the record; also the compact-match haystack. + fullText: (record: T) => string; + contentWeight?: number; + // Compact-query bonus (query with spaces removed found in the compacted haystack). + // 0 disables. compactExtraText widens the compact haystack (e.g. compacted title). + compactBonus?: number; + compactMinLength?: number; + compactExtraText?: (record: T) => string; + // Whole normalized query found in the full text. + phraseBonus?: number; + // Values compared for exact equality with the normalized query (title/slug). + exactValues?: (record: T) => string[]; + exactBonus?: number; + // Catalogue-wide "broad intent" terms ("forms", "services") granting a flat bonus. + broadTerms?: string[]; + broadBonus?: number; + // Token expansion hook (differential alias table). Receives the deduped query terms. + expandTokens?: (terms: string[]) => string[]; + limit?: number; + // Defaults to input order (stable) when omitted. + tieBreak?: (left: T, right: T) => number; +}; + +export function rankCatalogRecords( + records: T[], + query: string, + options: RankCatalogOptions, +): Array> { + const normalizedQuery = normalizeSearchText(query); + if (!normalizedQuery) return []; + + const contentWeight = options.contentWeight ?? 2; + const compactBonus = options.compactBonus ?? 0; + const compactMinLength = options.compactMinLength ?? 4; + const phraseBonus = options.phraseBonus ?? 4; + const exactBonus = options.exactBonus ?? 10; + const broadBonus = options.broadBonus ?? 1; + + const compactQuery = compactSearchText(normalizedQuery); + const baseTerms = Array.from(new Set(normalizedQuery.split(/\s+/).filter((term) => term.length > 1))); + const terms = options.expandTokens + ? Array.from(new Set(options.expandTokens(baseTerms).filter((term) => term.length > 1))) + : baseTerms; + const broad = Boolean(options.broadTerms?.length && terms.some((term) => options.broadTerms!.includes(term))); + + const ranked = records + .map((record, index) => { + const text = options.fullText(record); + const fields: Record = {}; + let score = 0; + + for (const field of options.fields) { + const haystack = field.text(record); + if (!haystack) continue; + const matched = terms.filter((term) => haystack.includes(term)).length; + if (!matched) continue; + fields[field.id] = matched; + score += matched * field.weight; + } + + const content = terms.filter((term) => text.includes(term)).length; + score += content * contentWeight; + + const compact = + compactBonus > 0 && + compactQuery.length >= compactMinLength && + (compactSearchText(text).includes(compactQuery) || + (options.compactExtraText ? compactSearchText(options.compactExtraText(record)).includes(compactQuery) : false)); + if (compact) score += compactBonus; + + const phrase = phraseBonus > 0 && text.includes(normalizedQuery); + if (phrase) score += phraseBonus; + + const exact = Boolean(options.exactValues?.(record).some((value) => value === normalizedQuery)); + if (exact) score += exactBonus; + + if (broad) score += broadBonus; + + return { + record, + index, + score, + signals: { fields, content, compact, phrase, exact, broad } satisfies CatalogMatchSignals, + }; + }) + .filter((match) => match.score > 0) + .sort( + (left, right) => + right.score - left.score || + (options.tieBreak ? options.tieBreak(left.record, right.record) : left.index - right.index), + ) + .map(({ record, score, signals }) => ({ record, score, signals })); + + return options.limit !== undefined ? ranked.slice(0, options.limit) : ranked; +} diff --git a/src/lib/differentials.ts b/src/lib/differentials.ts index 497df76999..4f47ba0414 100644 --- a/src/lib/differentials.ts +++ b/src/lib/differentials.ts @@ -1,3 +1,4 @@ +import { normalizeSearchText, rankCatalogRecords } from "@/lib/catalog-search"; import { loadDifferentialSnapshot } from "@/lib/differential-fixtures"; import type { DifferentialComparisonCandidate, @@ -98,51 +99,77 @@ export function differentialStaticParams() { return differentialRecords.map((record) => ({ slug: record.slug })); } -function expandQueryTokens(query: string) { +function expandQueryTerms(terms: string[]) { const aliases = differentialSearchAliases(); - const tokens = query.trim().toLowerCase().split(/\s+/).filter(Boolean); - const expanded = new Set(tokens); - for (const token of tokens) { - for (const alias of aliases[token] ?? []) expanded.add(alias); + const expanded = new Set(terms); + for (const term of terms) { + for (const alias of aliases[term] ?? []) { + for (const aliasToken of normalizeSearchText(alias).split(/\s+/).filter(Boolean)) expanded.add(aliasToken); + expanded.add(normalizeSearchText(alias)); + } } return [...expanded]; } function recordSearchText(record: DifferentialRecord) { - return [ - record.title, - record.subtitle, - record.clinicalHinge, - record.safetySnapshot.summary, - ...record.sections.flatMap((section) => [section.title, section.summary, ...section.items]), - ...record.related.flatMap((node) => [node.label, node.note]), - ] - .join(" ") - .toLowerCase(); + return normalizeSearchText( + [ + record.subtitle, + record.clinicalHinge, + record.safetySnapshot.summary, + ...record.sections.flatMap((section) => [section.title, section.summary, ...section.items]), + ...record.related.flatMap((node) => [node.label, node.note]), + ].join(" "), + ); +} + +export type DifferentialSearchMatch = { record: DifferentialRecord; score: number; reasons: string[] }; + +export function rankDifferentialRecords(query: string, limit?: number): DifferentialSearchMatch[] { + return rankCatalogRecords(differentialRecords, query, { + fields: [{ id: "title", weight: 6, text: (record) => normalizeSearchText(`${record.title} ${record.slug}`) }], + fullText: recordSearchText, + contentWeight: 2, + phraseBonus: 4, + exactValues: (record) => [normalizeSearchText(record.title), normalizeSearchText(record.slug)], + exactBonus: 10, + expandTokens: expandQueryTerms, + limit, + tieBreak: (left, right) => left.title.localeCompare(right.title), + }).map(({ record, score, signals }) => ({ + record, + score, + reasons: [ + signals.fields.title ? "title" : "", + signals.exact ? "exact title" : "", + signals.content ? "clinical content" : "", + ].filter(Boolean), + })); } export function searchDifferentialRecords(query: string) { - const tokens = expandQueryTokens(query); - if (!tokens.length) return differentialRecords; - return differentialRecords.filter((record) => { - const text = recordSearchText(record); - return tokens.some((token) => text.includes(token)); - }); + // Empty query keeps the full-catalogue browse behaviour; otherwise results are now + // relevance-ranked (previously an unranked alias OR-filter in snapshot order). + if (!normalizeSearchText(query)) return differentialRecords; + return rankDifferentialRecords(query).map((match) => match.record); } export function searchPresentationWorkflows(query: string) { - const tokens = expandQueryTokens(query); - if (!tokens.length) return differentialPresentations(); - return differentialPresentations().filter((presentation) => { - const text = [ - presentation.title, - presentation.subtitle, - presentation.safetySnapshot.summary, - ...presentation.safetySnapshot.tags, - ...presentation.candidates.map((candidate) => candidate.slug), - ] - .join(" ") - .toLowerCase(); - return tokens.some((token) => text.includes(token)); - }); + if (!normalizeSearchText(query)) return differentialPresentations(); + return rankCatalogRecords(differentialPresentations(), query, { + fields: [{ id: "title", weight: 6, text: (presentation) => normalizeSearchText(presentation.title) }], + fullText: (presentation) => + normalizeSearchText( + [ + presentation.subtitle, + presentation.safetySnapshot.summary, + ...presentation.safetySnapshot.tags, + ...presentation.candidates.map((candidate) => candidate.slug), + ].join(" "), + ), + contentWeight: 2, + phraseBonus: 4, + expandTokens: expandQueryTerms, + tieBreak: (left, right) => left.title.localeCompare(right.title), + }).map((match) => match.record); } diff --git a/src/lib/forms.ts b/src/lib/forms.ts index 13becf3bc8..210928a0cd 100644 --- a/src/lib/forms.ts +++ b/src/lib/forms.ts @@ -1,3 +1,4 @@ +import { normalizeSearchText, rankCatalogRecords } from "@/lib/catalog-search"; import type { ServiceRecord, ServiceSearchMatch } from "@/lib/services"; export type FormRecord = ServiceRecord; @@ -214,14 +215,6 @@ export const formRecords: FormRecord[] = [ }, ]; -function normalizeSearchText(value: string) { - return value - .toLowerCase() - .replace(/[^\w\d\s]+/g, " ") - .replace(/\s+/g, " ") - .trim(); -} - function formRecordSearchText(form: FormRecord) { const values = [ form.title, @@ -287,12 +280,19 @@ export function formNavigatorQuery(form: FormRecord) { export function rankFormRecords(records: FormRecord[], query: string, limit = records.length): FormSearchMatch[] { const normalizedQuery = normalizeSearchText(query); if (!normalizedQuery) return []; + // A bare "service(s)" query belongs to the services catalogue, not forms. if (/^services?$/.test(normalizedQuery)) return []; - const compactQuery = normalizedQuery.replace(/\s+/g, ""); - const terms = Array.from(new Set(normalizedQuery.split(" ").filter((term) => term.length > 1))); - const broadFormsQuery = terms.some((term) => - [ + return rankCatalogRecords(records, query, { + fields: [ + { id: "title", weight: 6, text: (form) => normalizeSearchText(`${form.title} ${form.slug}`) }, + { id: "contact", weight: 5, text: (form) => normalizeSearchText(form.primaryContact?.value ?? "") }, + ], + fullText: formRecordSearchText, + contentWeight: 2, + compactBonus: 5, + phraseBonus: 4, + broadTerms: [ "form", "forms", "checklist", @@ -304,42 +304,21 @@ export function rankFormRecords(records: FormRecord[], query: string, limit = re "examination", "template", "assessment", - ].includes(term), - ); - - return records - .map((form) => { - const title = normalizeSearchText(form.title); - const slug = normalizeSearchText(form.slug); - const contact = normalizeSearchText(form.primaryContact?.value ?? ""); - const text = formRecordSearchText(form); - const compactText = text.replace(/\s+/g, ""); - - const matchedTerms = terms.filter((term) => text.includes(term)); - const titleMatches = terms.filter((term) => title.includes(term) || slug.includes(term)); - const contactMatches = terms.filter((term) => contact.includes(term)); - const compactContactMatch = compactQuery.length >= 4 && compactText.includes(compactQuery); - - let score = 0; - score += titleMatches.length * 6; - score += contactMatches.length * 5; - if (compactContactMatch) score += 5; - score += matchedTerms.length * 2; - if (broadFormsQuery) score += 1; - if (normalizedQuery && text.includes(normalizedQuery)) score += 4; - - const reasons = [ - titleMatches.length ? "title" : "", - contactMatches.length || compactContactMatch ? "contact" : "", - matchedTerms.length ? "record fields" : "", - broadFormsQuery ? "psychiatry forms catalogue" : "", - ].filter(Boolean); - - return { service: form, score, reasons }; - }) - .filter((match) => match.score > 0) - .sort((left, right) => right.score - left.score || records.indexOf(left.service) - records.indexOf(right.service)) - .slice(0, limit); + ], + broadBonus: 1, + limit, + // No tieBreak: forms historically tie-break by catalogue (input) order, which is the + // generic ranker's default. + }).map(({ record, score, signals }) => ({ + service: record, + score, + reasons: [ + signals.fields.title ? "title" : "", + signals.fields.contact || signals.compact ? "contact" : "", + signals.content ? "record fields" : "", + signals.broad ? "psychiatry forms catalogue" : "", + ].filter(Boolean), + })); } export function searchFormRecords(query: string, limit = formRecords.length): FormSearchMatch[] { diff --git a/src/lib/medications.ts b/src/lib/medications.ts index d04a3eea84..29987f6113 100644 --- a/src/lib/medications.ts +++ b/src/lib/medications.ts @@ -1,3 +1,5 @@ +import { normalizeSearchText, rankCatalogRecords } from "@/lib/catalog-search"; + export type MedicationPatientMetadata = { factors?: string[]; action?: string; @@ -69,15 +71,7 @@ export function normalizeMedicationSlug(value: string) { return value.trim().toLowerCase(); } -export function normalizeSearchText(value: string) { - return value - .toLowerCase() - .normalize("NFKD") - .replace(/[\u0300-\u036f]/g, "") - .replace(/[^a-z0-9+./\s-]/g, " ") - .replace(/\s+/g, " ") - .trim(); -} +export { normalizeSearchText }; export function normalizeRecord(record: MedicationRecord): MedicationRecord { return { @@ -196,48 +190,43 @@ export function medicationToSearchResult(match: MedicationSearchMatch): Medicati } export function rankMedicationRecords(records: MedicationRecord[], query: string, limit = 50): MedicationSearchMatch[] { - const normalizedQuery = normalizeSearchText(query); - if (!normalizedQuery) return []; - - const compactQuery = normalizedQuery.replace(/\s+/g, ""); - const terms = Array.from(new Set(normalizedQuery.split(/\s+/).filter((term) => term.length > 1))); - - return records - .map((medication) => { - const title = normalizeSearchText(medication.name); - const slug = normalizeSearchText(medication.slug); - const taxonomy = normalizeSearchText( - [medication.class, medication.subclass, medication.category, medication.tag, medication.schedule].join(" "), - ); - const text = medicationSearchText(medication); - const compactText = text.replace(/\s+/g, ""); - const matchedTerms = terms.filter((term) => text.includes(term)); - const titleMatches = terms.filter((term) => title.includes(term) || slug.includes(term)); - const taxonomyMatches = terms.filter((term) => taxonomy.includes(term)); - const compactTitleMatch = - compactQuery.length >= 4 && - (compactText.includes(compactQuery) || title.replace(/\s+/g, "").includes(compactQuery)); - - let score = 0; - score += titleMatches.length * 8; - if (compactTitleMatch) score += 6; - score += taxonomyMatches.length * 3; - score += matchedTerms.length * 2; - if (normalizedQuery && text.includes(normalizedQuery)) score += 4; - if (title === normalizedQuery || slug === normalizedQuery) score += 10; - - const reasons = [ - titleMatches.length ? "name" : "", - compactTitleMatch ? "exact name" : "", - taxonomyMatches.length ? "class/category" : "", - matchedTerms.length ? "content" : "", - ].filter(Boolean); - - return { medication, score, reasons }; - }) - .filter((match) => match.score > 0) - .sort((left, right) => right.score - left.score || left.medication.name.localeCompare(right.medication.name)) - .slice(0, limit); + return rankCatalogRecords(records, query, { + fields: [ + { + id: "name", + weight: 8, + text: (medication) => normalizeSearchText(`${medication.name} ${medication.slug}`), + }, + { + id: "taxonomy", + weight: 3, + text: (medication) => + normalizeSearchText( + [medication.class, medication.subclass, medication.category, medication.tag, medication.schedule].join( + " ", + ), + ), + }, + ], + fullText: medicationSearchText, + contentWeight: 2, + compactBonus: 6, + compactExtraText: (medication) => normalizeSearchText(medication.name), + phraseBonus: 4, + exactValues: (medication) => [normalizeSearchText(medication.name), normalizeSearchText(medication.slug)], + exactBonus: 10, + limit, + tieBreak: (left, right) => left.name.localeCompare(right.name), + }).map(({ record, score, signals }) => ({ + medication: record, + score, + reasons: [ + signals.fields.name ? "name" : "", + signals.compact ? "exact name" : "", + signals.fields.taxonomy ? "class/category" : "", + signals.content ? "content" : "", + ].filter(Boolean), + })); } export function medicationIdentityBadges(record: MedicationRecord) { diff --git a/src/lib/rag.ts b/src/lib/rag.ts index b1e21c8271..ccfdf32f0f 100644 --- a/src/lib/rag.ts +++ b/src/lib/rag.ts @@ -2288,15 +2288,21 @@ export function relaxVariantToOrQuery(variant: string): string | null { // text_rank sits below it carries almost no lexical evidence. const weakTextMatchTopRankFloor = 0.05; const weakTextMatchMinResultCount = 3; - -// Strict-AND matched something, but so weakly (sparse set or negligible text rank) that the -// right chunk may be buried outside the candidate pool. In that case OR-relaxed recall is -// appended BEHIND the strict matches (append-only: strict results keep merge precedence), so -// this can widen the pool but never displace a precise match. +// Above this rank the best strict match is a precise lexical hit; augmenting a sparse set +// around it only adds OR noise and a needless RPC round-trip (single-strong-match queries +// like exact table lookups must stay one-RPC retrievals). +const strongTextMatchTopRankBar = 0.3; + +// Strict-AND matched something, but so weakly (sparse set of middling matches, or a +// negligible best text rank) that the right chunk may be buried outside the candidate pool. +// In that case OR-relaxed recall is appended BEHIND the strict matches (append-only: strict +// results keep merge precedence), so this can widen the pool but never displace a precise +// match. A sparse set anchored by a strong hit does NOT relax. export function shouldRelaxWeakTextMatches(merged: SearchResult[]): boolean { if (merged.length === 0) return false; - if (merged.length < weakTextMatchMinResultCount) return true; const topTextRank = merged.reduce((top, result) => Math.max(top, result.text_rank ?? 0), 0); + if (topTextRank >= strongTextMatchTopRankBar) return false; + if (merged.length < weakTextMatchMinResultCount) return true; return topTextRank < weakTextMatchTopRankFloor; } diff --git a/src/lib/services.ts b/src/lib/services.ts index 9c3f50c59b..4814ef0a67 100644 --- a/src/lib/services.ts +++ b/src/lib/services.ts @@ -1,3 +1,4 @@ +import { normalizeSearchText, rankCatalogRecords } from "@/lib/catalog-search"; import { defaultServiceRecords } from "@/lib/registry-fixtures"; export type ServiceChipTone = "danger" | "info" | "warning" | "success" | "neutral"; @@ -104,12 +105,6 @@ export function serviceNavigatorQuery(service: ServiceRecord) { ); } -function normalizeSearchText(value: string) { - return value - .toLowerCase() - .replace(/[^a-z0-9]+/g, " ") - .trim(); -} function serviceRecordSearchParts(service: ServiceRecord) { return [ @@ -154,49 +149,35 @@ export function rankServiceRecords( query: string, limit = records.length, ): ServiceSearchMatch[] { - const normalizedQuery = normalizeSearchText(query); - if (!normalizedQuery) return []; - - const compactQuery = normalizedQuery.replace(/\s+/g, ""); - const terms = Array.from(new Set(normalizedQuery.split(/\s+/).filter((term) => term.length > 1))); - const broadServicesQuery = terms.some((term) => ["service", "services", "pathway", "pathways"].includes(term)); - - return records - .map((service) => { - const title = normalizeSearchText(service.title); - const slug = normalizeSearchText(service.slug); - const contact = normalizeSearchText(service.primaryContact?.value ?? ""); - const tags = normalizeSearchText([...(service.tags ?? []), ...(service.catchments ?? [])].join(" ")); - const text = serviceRecordSearchText(service); - const compactText = text.replace(/\s+/g, ""); - const matchedTerms = terms.filter((term) => text.includes(term)); - const titleMatches = terms.filter((term) => title.includes(term) || slug.includes(term)); - const contactMatches = terms.filter((term) => contact.includes(term)); - const tagMatches = terms.filter((term) => tags.includes(term)); - const compactContactMatch = compactQuery.length >= 4 && compactText.includes(compactQuery); - - let score = 0; - score += titleMatches.length * 6; - score += contactMatches.length * 5; - if (compactContactMatch) score += 5; - score += tagMatches.length * 3; - score += matchedTerms.length * 2; - if (broadServicesQuery) score += 1; - if (normalizedQuery && text.includes(normalizedQuery)) score += 4; - - const reasons = [ - titleMatches.length ? "title" : "", - contactMatches.length || compactContactMatch ? "contact" : "", - tagMatches.length ? "tags" : "", - matchedTerms.length ? "record fields" : "", - broadServicesQuery ? "services catalogue" : "", - ].filter(Boolean); - - return { service, score, reasons }; - }) - .filter((match) => match.score > 0) - .sort((left, right) => right.score - left.score || left.service.title.localeCompare(right.service.title)) - .slice(0, limit); + return rankCatalogRecords(records, query, { + fields: [ + { id: "title", weight: 6, text: (service) => normalizeSearchText(`${service.title} ${service.slug}`) }, + { id: "contact", weight: 5, text: (service) => normalizeSearchText(service.primaryContact?.value ?? "") }, + { + id: "tags", + weight: 3, + text: (service) => normalizeSearchText([...(service.tags ?? []), ...(service.catchments ?? [])].join(" ")), + }, + ], + fullText: serviceRecordSearchText, + contentWeight: 2, + compactBonus: 5, + phraseBonus: 4, + broadTerms: ["service", "services", "pathway", "pathways"], + broadBonus: 1, + limit, + tieBreak: (left, right) => left.title.localeCompare(right.title), + }).map(({ record, score, signals }) => ({ + service: record, + score, + reasons: [ + signals.fields.title ? "title" : "", + signals.fields.contact || signals.compact ? "contact" : "", + signals.fields.tags ? "tags" : "", + signals.content ? "record fields" : "", + signals.broad ? "services catalogue" : "", + ].filter(Boolean), + })); } export function searchServiceRecords(query: string, limit = serviceRecords.length): ServiceSearchMatch[] { diff --git a/src/lib/tools-catalog.ts b/src/lib/tools-catalog.ts new file mode 100644 index 0000000000..181671bd87 --- /dev/null +++ b/src/lib/tools-catalog.ts @@ -0,0 +1,284 @@ +import { normalizeSearchText, rankCatalogRecords } from "@/lib/catalog-search"; + +// Canonical Tools dataset. Previously duplicated between the live launcher +// (applications-launcher-page.tsx inline array) and the mockup fixtures +// (tools-page-mockups/tool-fixtures.ts) with divergent fields and two separate filter +// implementations. Icons are UI concerns and stay in the components (keyed by id). + +export type ToolCatalogStatus = "ready" | "recent" | "review_due"; +export type ToolCatalogArea = "assessment" | "reference" | "care" | "coordination" | "saved"; + +export type ToolCatalogRecord = { + id: string; + title: string; + mobileTitle?: string; + description: string; + bestFor: string; + detail: string; + href: string; + external?: boolean; + area: ToolCatalogArea; + status: ToolCatalogStatus; + sourceBacked: boolean; + safetyFirst?: boolean; + highYield?: boolean; + actionLabel: string; + keywords: string[]; + checkFirst: string[]; + neededInput: string[]; + output: string; +}; + +export const toolCatalogRecords: ToolCatalogRecord[] = [ + { + id: "clinical-kb-search", + title: "Clinical KB Search", + mobileTitle: "Clinical KB", + description: "Ask source-backed clinical questions and move straight to evidence.", + bestFor: "Quick answers and guidance", + detail: "Ask source-backed clinical questions and move straight to evidence.", + href: "/?mode=answer", + area: "assessment", + status: "ready", + sourceBacked: true, + highYield: true, + actionLabel: "Ask", + keywords: ["answer", "ask", "source", "knowledge base", "clinical question", "search"], + checkFirst: ["Clinical question or PICO", "Patient context and setting", "Timeframe or guideline scope"], + neededInput: ["Clinical question", "Relevant patient context", "Optional source or document scope"], + output: "Concise answer, key points, citations, and source links.", + }, + { + id: "differentials", + title: "Differentials", + description: "Build and compare diagnostic possibilities with source-aware prompts.", + bestFor: "Broad or complex presentations", + detail: "Compare diagnostic possibilities, supporting features, red flags, and next-step questions.", + href: "/differentials", + area: "assessment", + status: "recent", + sourceBacked: true, + highYield: true, + actionLabel: "Compare", + keywords: ["compare", "diagnosis", "differential", "presentation", "risk"], + checkFirst: ["Red flags", "Key presenting features", "Important negatives"], + neededInput: ["Chief concern", "History and examination features", "Available observations or tests"], + output: "Ranked differentials, rationale, must-not-miss risks, and next steps.", + }, + { + id: "documents", + title: "Documents", + mobileTitle: "Docs", + description: "Search indexed PDFs, policies, guidelines, pages, tables, and images.", + bestFor: "Trusted documents and pages", + detail: "Find the source document, page, table, image, or policy wording behind an answer.", + href: "/?mode=documents", + area: "reference", + status: "ready", + sourceBacked: true, + highYield: true, + actionLabel: "Search", + keywords: ["documents", "docs", "pdf", "policy", "guideline", "source", "pages"], + checkFirst: ["Document title or topic", "Local policy scope", "Page, table, or image need"], + neededInput: ["Source topic", "Optional document name", "Preferred date or local scope"], + output: "Matching documents, page context, snippets, and source links.", + }, + { + id: "guidelines", + title: "Guidelines", + description: "Browse trusted guidelines and clinical pathways.", + bestFor: "Recommendations and standards", + detail: "Move from a clinical question to guideline wording, pathway steps, and source context.", + href: "/?mode=documents&q=guideline&focus=1", + area: "reference", + status: "ready", + sourceBacked: true, + highYield: true, + actionLabel: "Browse", + keywords: ["guidelines", "recommendations", "standards", "pathways"], + checkFirst: ["Guideline topic", "Population or setting", "Local policy relevance"], + neededInput: ["Condition or intervention", "Clinical setting", "Optional source preference"], + output: "Guideline matches, key recommendations, and linked source context.", + }, + { + id: "risk-safety", + title: "Risk & Safety", + mobileTitle: "Safety", + description: "Check risks, contraindications, alerts, and safety guidance.", + bestFor: "Preventing harm", + detail: "Check risks, contraindications, and safety alerts before making clinical decisions.", + href: "/?mode=answer&q=safety%20check&focus=1", + area: "care", + status: "review_due", + sourceBacked: true, + safetyFirst: true, + actionLabel: "Open", + keywords: ["risk", "safety", "contraindications", "red flags", "alerts", "harm"], + checkFirst: [ + "Allergies and adverse reactions", + "Drug-drug and drug-disease interactions", + "Dose adjustments and monitoring needs", + "Safety alerts and warnings", + ], + neededInput: [ + "Patient context and problem list", + "Current medications and doses", + "Allergies and prior reactions", + "Renal/hepatic function if relevant", + ], + output: "Prioritized risks, alerts, and actionable recommendations with source links.", + }, + { + id: "medication-prescribing", + title: "Medication Prescribing", + mobileTitle: "Prescribe", + description: "Review prescribing context, monitoring, interactions, and cautions.", + bestFor: "Safe and effective prescribing", + detail: "Review medication context, dosing, interactions, monitoring, and medication-specific cautions.", + href: "/?mode=prescribing", + area: "care", + status: "review_due", + sourceBacked: true, + safetyFirst: true, + actionLabel: "Prescribe", + keywords: ["medication", "medications", "prescribing", "dose", "monitoring", "interactions"], + checkFirst: ["Current medicines", "Contraindications", "Monitoring requirements"], + neededInput: ["Medicine and indication", "Dose and route if known", "Comorbidities and key labs"], + output: "Prescribing guidance, monitoring plan, cautions, and references.", + }, + { + id: "services", + title: "Services", + description: "Open source-backed service records, referral routes, and eligibility.", + bestFor: "Referrals and coordination", + detail: "Open service records with referral routes, eligibility, source status, and access pathways.", + href: "/services", + area: "coordination", + status: "ready", + sourceBacked: true, + highYield: true, + actionLabel: "Refer", + keywords: ["services", "referral", "eligibility", "pathway", "contact"], + checkFirst: ["Eligibility", "Referral route", "Service source status"], + neededInput: ["Patient location or catchment", "Clinical need", "Urgency and pathway requirements"], + output: "Referral pathway, eligibility notes, service record, and source link.", + }, + { + id: "forms", + title: "Forms", + description: "Find clinical forms and source-backed readiness pathways.", + bestFor: "Forms and workflows", + detail: "Open form search, readiness checks, pathway tasks, and source-backed records.", + href: "/forms", + area: "coordination", + status: "ready", + sourceBacked: true, + highYield: true, + actionLabel: "Open", + keywords: ["forms", "paperwork", "readiness", "pathway"], + checkFirst: ["Current form version", "Required fields", "Linked service pathway"], + neededInput: ["Form type", "Clinical pathway", "Patient or service context"], + output: "Relevant form, readiness tasks, and source-backed pathway details.", + }, + { + id: "care-plans", + title: "Care plans", + description: "Create and review management plans with monitoring and follow-up.", + bestFor: "Ongoing care planning", + detail: "Structure care planning, review milestones, monitoring needs, and follow-up tasks.", + href: "/?mode=answer&q=care%20plan&focus=1", + area: "care", + status: "ready", + sourceBacked: true, + highYield: true, + actionLabel: "Open", + keywords: ["care plan", "management", "follow-up", "monitoring"], + checkFirst: ["Goals of care", "Review date", "Monitoring responsibilities"], + neededInput: ["Diagnosis or working problem", "Current plan", "Follow-up timeframe"], + output: "Care-plan structure, review points, and monitoring prompts.", + }, + { + id: "monitoring", + title: "Monitoring", + description: "Track and review key monitoring parameters and results.", + bestFor: "Ongoing monitoring", + detail: "Review monitoring intervals, parameters, alerts, and follow-up actions.", + href: "/?mode=answer&q=monitoring%20schedule&focus=1", + area: "care", + status: "ready", + sourceBacked: true, + highYield: true, + actionLabel: "Open", + keywords: ["monitoring", "results", "parameters", "schedule", "labs"], + checkFirst: ["Monitoring indication", "Last result date", "Thresholds and alerts"], + neededInput: ["Medication or condition", "Recent results", "Monitoring timeframe"], + output: "Monitoring schedule, thresholds, and review prompts.", + }, + { + id: "favourites", + title: "Saved workflows", + mobileTitle: "Saved", + description: "Return to saved clinical workspaces and repeated workflows.", + bestFor: "Repeated or complex work", + detail: "Resume saved answers, pinned sources, and repeated clinical workflows.", + href: "/favourites", + area: "saved", + status: "recent", + sourceBacked: false, + actionLabel: "View", + keywords: ["favourites", "favorites", "saved", "recent", "pinned"], + checkFirst: ["Saved context", "Last-used status", "Review markers"], + neededInput: ["Saved item or workflow name", "Optional source set", "Review context"], + output: "Saved workspace, pinned source, or recent workflow.", + }, +]; + +export function toolCatalogRecordById(id: string) { + return toolCatalogRecords.find((tool) => tool.id === id) ?? toolCatalogRecords[0]; +} + +export function toolSearchText(tool: ToolCatalogRecord) { + return normalizeSearchText( + [ + tool.title, + tool.mobileTitle, + tool.description, + tool.bestFor, + tool.detail, + tool.area, + ...tool.keywords, + ...tool.checkFirst, + tool.output, + ] + .filter((value): value is string => Boolean(value?.trim())) + .join(" "), + ); +} + +export type ToolSearchMatch = { tool: ToolCatalogRecord; score: number; reasons: string[] }; + +export function rankToolRecords(query: string, limit?: number): ToolSearchMatch[] { + return rankCatalogRecords(toolCatalogRecords, query, { + fields: [ + { + id: "title", + weight: 6, + text: (tool) => normalizeSearchText(`${tool.title} ${tool.mobileTitle ?? ""} ${tool.id}`), + }, + { id: "keywords", weight: 3, text: (tool) => normalizeSearchText(tool.keywords.join(" ")) }, + ], + fullText: toolSearchText, + contentWeight: 2, + phraseBonus: 4, + limit, + tieBreak: (left, right) => left.title.localeCompare(right.title), + }).map(({ record, score, signals }) => ({ + tool: record, + score, + reasons: [ + signals.fields.title ? "title" : "", + signals.fields.keywords ? "keywords" : "", + signals.content ? "description" : "", + ].filter(Boolean), + })); +} diff --git a/tests/app-modes.test.ts b/tests/app-modes.test.ts index b499230998..c39c66275b 100644 --- a/tests/app-modes.test.ts +++ b/tests/app-modes.test.ts @@ -75,8 +75,10 @@ describe("app mode search contract", () => { expect(isSearchableAppMode("forms")).toBe(true); expect(mode?.label).toBe("Forms"); expect(mode?.href).toBe("/forms"); - expect(config.kind).toBe("documents"); - expect(config.resultKind).toBe("documents"); + // Forms are a registry catalogue with their own honest kind — no longer masquerading + // as corpus documents (which forced downstream special-casing). + expect(config.kind).toBe("forms"); + expect(config.resultKind).toBe("forms"); expect(config.placeholder.toLowerCase()).toContain("forms"); }); @@ -107,7 +109,9 @@ describe("app mode search contract", () => { expect(appModeCanUseSourceLibraryShortcut("tools")).toBe(false); expect(appModeCanUseSourceLibraryShortcut("documents")).toBe(true); expect(appModeCanUseSourceLibraryShortcut("services")).toBe(false); - expect(appModeCanUseSourceLibraryShortcut("forms")).toBe(true); + // Forms is a registry catalogue: a scope-tag shortcut falls back to documents mode + // instead of dead-ending in the forms registry branch. + expect(appModeCanUseSourceLibraryShortcut("forms")).toBe(false); expect(appModeCanUseSourceLibraryShortcut("favourites")).toBe(false); expect(appModeCanUseSourceLibraryShortcut("prescribing")).toBe(true); expect(appModeCanUseSourceLibraryShortcut("differentials")).toBe(true); diff --git a/tests/catalog-search.test.ts b/tests/catalog-search.test.ts new file mode 100644 index 0000000000..c788eff509 --- /dev/null +++ b/tests/catalog-search.test.ts @@ -0,0 +1,122 @@ +import { describe, expect, it } from "vitest"; +import { compactSearchText, normalizeSearchText, rankCatalogRecords } from "../src/lib/catalog-search"; + +type Item = { title: string; slug: string; tags: string[]; body: string }; + +const items: Item[] = [ + { title: "Clozapine Monitoring", slug: "clozapine-monitoring", tags: ["antipsychotic"], body: "ANC FBC thresholds" }, + { title: "Lithium Levels", slug: "lithium-levels", tags: ["mood stabiliser"], body: "Serum level monitoring" }, + { title: "Transfer Checklist", slug: "transfer-checklist", tags: ["transport"], body: "Receiving service details" }, +]; + +function rank(query: string, overrides: Partial>[2]> = {}) { + return rankCatalogRecords(items, query, { + fields: [ + { id: "title", weight: 6, text: (item) => normalizeSearchText(`${item.title} ${item.slug}`) }, + { id: "tags", weight: 3, text: (item) => normalizeSearchText(item.tags.join(" ")) }, + ], + fullText: (item) => normalizeSearchText(`${item.title} ${item.tags.join(" ")} ${item.body}`), + ...overrides, + }); +} + +describe("normalizeSearchText (shared)", () => { + it("keeps dose-string characters that the retired per-domain normalizers disagreed on", () => { + expect(normalizeSearchText("0.5mg IM/PO 5+5 co-located")).toBe("0.5mg im/po 5+5 co-located"); + }); + + it("strips diacritics and collapses punctuation to single spaces", () => { + expect(normalizeSearchText("Sérum; Lévels!")).toBe("serum levels"); + }); + + it("compacts whitespace for compact-query matching", () => { + expect(compactSearchText("clozapine monitoring")).toBe("clozapinemonitoring"); + }); +}); + +describe("rankCatalogRecords", () => { + it("returns nothing for an empty or whitespace query", () => { + expect(rank("")).toEqual([]); + expect(rank(" ")).toEqual([]); + }); + + it("weights field matches by their configured weight plus the content weight", () => { + const [top] = rank("clozapine"); + expect(top.record.slug).toBe("clozapine-monitoring"); + // title (6) + content (2) + whole-query phrase (4) — a single-term query IS its own + // phrase, matching the historical per-domain rankers. + expect(top.score).toBe(12); + expect(top.signals.fields.title).toBe(1); + expect(top.signals.content).toBe(1); + expect(top.signals.phrase).toBe(true); + }); + + it("drops records with no matching signal", () => { + const results = rank("clozapine"); + expect(results.some((match) => match.record.slug === "lithium-levels")).toBe(false); + }); + + it("applies the whole-phrase bonus on top of term matches", () => { + const [top] = rank("clozapine monitoring"); + // 2 title terms (12) + 2 content terms (4) + phrase (4). + expect(top.score).toBe(20); + expect(top.signals.phrase).toBe(true); + }); + + it("grants the exact bonus only on strict equality with a configured exact value", () => { + const options = { + exactValues: (item: Item) => [normalizeSearchText(item.title), normalizeSearchText(item.slug)], + exactBonus: 10, + }; + const exact = rank("clozapine monitoring", options)[0]; + const partial = rank("clozapine", options)[0]; + expect(exact.signals.exact).toBe(true); + expect(partial.signals.exact).toBe(false); + expect(exact.score).toBe(30); + }); + + it("grants the compact bonus when the de-spaced query appears in the compacted haystack", () => { + const [top] = rank("clozapinemonitoring", { compactBonus: 6 }); + expect(top.signals.compact).toBe(true); + // The de-spaced term matches nothing term-wise; only the compact bonus scores it, + // which is exactly how a run-together query survived in the historical rankers. + expect(top.score).toBe(6); + }); + + it("adds the broad-catalogue bonus to every record when a broad term is present", () => { + const results = rank("transport checklist", { broadTerms: ["transport"], broadBonus: 1 }); + expect(results[0].record.slug).toBe("transfer-checklist"); + for (const match of results) expect(match.signals.broad).toBe(true); + }); + + it("expands query terms through the expandTokens hook", () => { + const results = rank("cloz", { + expandTokens: (terms) => (terms.includes("cloz") ? [...terms, "clozapine"] : terms), + }); + expect(results[0].record.slug).toBe("clozapine-monitoring"); + }); + + it("breaks score ties by input order unless a tieBreak is supplied", () => { + const tied: Item[] = [ + { title: "Zeta Monitoring", slug: "zeta", tags: [], body: "" }, + { title: "Alpha Monitoring", slug: "alpha", tags: [], body: "" }, + ]; + const byInput = rankCatalogRecords(tied, "monitoring", { + fields: [{ id: "title", weight: 6, text: (item) => normalizeSearchText(item.title) }], + fullText: (item) => normalizeSearchText(item.title), + }); + expect(byInput.map((match) => match.record.slug)).toEqual(["zeta", "alpha"]); + + const byTitle = rankCatalogRecords(tied, "monitoring", { + fields: [{ id: "title", weight: 6, text: (item) => normalizeSearchText(item.title) }], + fullText: (item) => normalizeSearchText(item.title), + tieBreak: (left, right) => left.title.localeCompare(right.title), + }); + expect(byTitle.map((match) => match.record.slug)).toEqual(["alpha", "zeta"]); + }); + + it("applies the limit after ranking", () => { + const results = rank("monitoring checklist", { limit: 1 }); + expect(results).toHaveLength(1); + }); +}); diff --git a/tests/differentials.test.ts b/tests/differentials.test.ts index a04b826e25..3198b8fe06 100644 --- a/tests/differentials.test.ts +++ b/tests/differentials.test.ts @@ -9,6 +9,7 @@ import { getDifferentialRecord, getPresentationWorkflow, loadDifferentialSnapshot, + rankDifferentialRecords, searchDifferentialRecords, } from "@/lib/differentials"; @@ -136,3 +137,18 @@ describe("differential records", () => { } }); }); + +describe("ranked differential search", () => { + it("ranks title matches above content-only matches", () => { + const matches = rankDifferentialRecords("delirium"); + expect(matches.length).toBeGreaterThan(0); + expect(matches[0].record.slug).toContain("delirium"); + expect(matches[0].score).toBeGreaterThanOrEqual(matches[matches.length - 1].score); + expect(matches[0].reasons).toContain("title"); + }); + + it("keeps the full catalogue for an empty query and still honours aliases", () => { + expect(searchDifferentialRecords("")).toEqual(differentialRecords); + expect(searchDifferentialRecords(" ")).toEqual(differentialRecords); + }); +}); diff --git a/tests/retrieval-query-variants.test.ts b/tests/retrieval-query-variants.test.ts index e5b98174e4..723c41897b 100644 --- a/tests/retrieval-query-variants.test.ts +++ b/tests/retrieval-query-variants.test.ts @@ -1041,10 +1041,18 @@ describe("shouldRelaxWeakTextMatches (P8b weak-augment)", () => { expect(shouldRelaxWeakTextMatches([])).toBe(false); }); - it("fires when strict-AND returned a sparse result set", () => { - expect(shouldRelaxWeakTextMatches([result({ text_rank: 0.4 })])).toBe(true); - expect(shouldRelaxWeakTextMatches([result({ id: "a", text_rank: 0.4 }), result({ id: "b", text_rank: 0.3 })])).toBe( - true, + it("fires when strict-AND returned a sparse set of middling matches", () => { + expect(shouldRelaxWeakTextMatches([result({ text_rank: 0.1 })])).toBe(true); + expect( + shouldRelaxWeakTextMatches([result({ id: "a", text_rank: 0.12 }), result({ id: "b", text_rank: 0.08 })]), + ).toBe(true); + }); + + it("does not fire when a sparse set is anchored by a strong lexical hit", () => { + // A single precise match (e.g. an exact table lookup) must stay a one-RPC retrieval. + expect(shouldRelaxWeakTextMatches([result({ text_rank: 1.1 })])).toBe(false); + expect(shouldRelaxWeakTextMatches([result({ id: "a", text_rank: 0.4 }), result({ id: "b", text_rank: 0.05 })])).toBe( + false, ); }); diff --git a/tests/tools-catalog.test.ts b/tests/tools-catalog.test.ts new file mode 100644 index 0000000000..45fe736c0d --- /dev/null +++ b/tests/tools-catalog.test.ts @@ -0,0 +1,37 @@ +import { describe, expect, it } from "vitest"; +import { rankToolRecords, toolCatalogRecordById, toolCatalogRecords } from "../src/lib/tools-catalog"; +import { tools as mockupToolFixtures } from "../src/components/tools-page-mockups/tool-fixtures"; + +describe("tools catalog", () => { + it("has unique ids and the launcher staples", () => { + const ids = toolCatalogRecords.map((tool) => tool.id); + expect(new Set(ids).size).toBe(ids.length); + for (const staple of ["clinical-kb-search", "documents", "medication-prescribing", "services", "forms"]) { + expect(ids).toContain(staple); + } + }); + + it("ranks title matches above keyword-only matches", () => { + const matches = rankToolRecords("forms"); + expect(matches[0].tool.id).toBe("forms"); + expect(matches[0].reasons).toContain("title"); + }); + + it("finds tools through keywords", () => { + const matches = rankToolRecords("contraindications"); + expect(matches.some((match) => match.tool.id === "risk-safety")).toBe(true); + }); + + it("returns nothing for an empty query", () => { + expect(rankToolRecords("")).toEqual([]); + }); + + it("keeps the mockup fixtures derived from catalog identity fields", () => { + for (const fixture of mockupToolFixtures) { + const record = toolCatalogRecordById(fixture.id); + expect(record.id).toBe(fixture.id); + expect(fixture.href).toBe(record.href); + expect(fixture.sourceBacked).toBe(record.sourceBacked); + } + }); +}); From 3422fc394f11cbfcaea3c0a4bb2e06d73feb5518 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 11:21:02 +0000 Subject: [PATCH 4/6] Add federated universal search endpoint across documents and registry catalogs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3 of the universal search workstream: - New src/lib/universal-search.ts: runUniversalSearch() fans out in parallel (Promise.allSettled) to the document retrieval pipeline (searchChunksWithTelemetry + fetchRelatedDocuments) and the shared Phase-2 registry rankers (medications, services, forms, differentials, tools). Per-domain timeouts (2.5s registry / 6s documents); a failed domain yields an empty errored group so one broken adapter never blanks the response. No cross-domain score normalization — registry integer weights and [0,1] document scores are incommensurable, so ordering is by fixed group order, ranked within each group. - New GET /api/search/universal (typeahead-friendly): q (min 2), limit (max 10), domains CSV. Access ladder mirrors /api/registry/records: demo/local -> fixtures, public -> public catalogues, owner -> seeded records, with the shared "registry" rate-limit bucket and Cache-Control: private, no-store. - The owner fetch + lazy-seed blocks are extracted from /api/medications and /api/registry/records into fetchOwnerMedicationRowsWithSeed / fetchOwnerRegistryRowsWithSeed so the routes and universal search share one code path. - Tests: universal-search lib (fixed group order, per-domain fixtures with working hrefs, domain filtering, error isolation, demo document search) and route (demoMode payload, min-length validation, unknown-domain CSV). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_011QiyE8jMm7VnrtknHF6jnJ --- docs/site-map.md | 1 + src/app/api/medications/route.ts | 24 +-- src/app/api/registry/records/route.ts | 30 +-- src/app/api/search/universal/route.ts | 96 +++++++++ src/lib/medication-seed.ts | 33 +++ src/lib/registry-seed.ts | 37 ++++ src/lib/universal-search.ts | 298 ++++++++++++++++++++++++++ tests/universal-search.test.ts | 116 ++++++++++ 8 files changed, 585 insertions(+), 50 deletions(-) create mode 100644 src/app/api/search/universal/route.ts create mode 100644 src/lib/universal-search.ts create mode 100644 tests/universal-search.test.ts diff --git a/docs/site-map.md b/docs/site-map.md index c522d23950..6a83d4c4c6 100644 --- a/docs/site-map.md +++ b/docs/site-map.md @@ -570,6 +570,7 @@ This file is generated by `npm run sitemap:update`. Run `npm run sitemap:check` - `/api/registry/records/[slug]` - Registry record detail. Source: `src/app/api/registry/records/[slug]/route.ts`. - `/api/search` - Search endpoint. Source: `src/app/api/search/route.ts`. - `/api/search/interaction` - Search interaction telemetry. Source: `src/app/api/search/interaction/route.ts`. +- `/api/search/universal` - Route discovered from app directory Source: `src/app/api/search/universal/route.ts`. - `/api/setup-status` - Setup status. Source: `src/app/api/setup-status/route.ts`. - `/api/upload` - Upload endpoint. Source: `src/app/api/upload/route.ts`. - `/auth/callback` - Route discovered from app directory Source: `src/app/auth/callback/route.ts`. diff --git a/src/app/api/medications/route.ts b/src/app/api/medications/route.ts index 88aff92a3f..365b3090f4 100644 --- a/src/app/api/medications/route.ts +++ b/src/app/api/medications/route.ts @@ -8,7 +8,7 @@ import { } from "@/lib/api-rate-limit"; import { isDemoMode, isLocalNoAuthMode } from "@/lib/env"; import { jsonError } from "@/lib/http"; -import { defaultMedicationRecords, ensureMedicationsSeeded } from "@/lib/medication-seed"; +import { defaultMedicationRecords, fetchOwnerMedicationRowsWithSeed } from "@/lib/medication-seed"; import { medicationSourceStatus, medicationValidationStatus, @@ -107,27 +107,7 @@ export async function GET(request: Request) { }); } - const fetchRecords = async () => { - const { data, error } = await supabase - .from("medication_records") - .select("*") - .eq("owner_id", access.ownerId) - .order("name") - .limit(MEDICATION_MAX_RECORDS); - if (error) throw new Error(error.message); - return (data ?? []) as MedicationRecordRow[]; - }; - - let rows = await fetchRecords(); - if (rows.length === 0) { - try { - await ensureMedicationsSeeded(supabase, access.ownerId); - } catch (seedError) { - console.error(`[medications] auto-seed failed for owner ${access.ownerId}`, seedError); - } - rows = await fetchRecords(); - } - + const rows = await fetchOwnerMedicationRowsWithSeed(supabase, access.ownerId, MEDICATION_MAX_RECORDS); const records = rows.map(rowToMedicationRecord); const governanceBySlug = Object.fromEntries(rows.map((row) => [row.slug, rowGovernance(row)])); diff --git a/src/app/api/registry/records/route.ts b/src/app/api/registry/records/route.ts index 373ec658ad..29b864a649 100644 --- a/src/app/api/registry/records/route.ts +++ b/src/app/api/registry/records/route.ts @@ -17,7 +17,7 @@ import { type RegistryRecordKind, type RegistryRecordRow, } from "@/lib/registry-records"; -import { ensureRegistrySeeded } from "@/lib/registry-seed"; +import { fetchOwnerRegistryRowsWithSeed } from "@/lib/registry-seed"; import { rankServiceRecords, serviceRecords, type ServiceRecord, type ServiceSearchMatch } from "@/lib/services"; import { createAdminClient } from "@/lib/supabase/admin"; import { AuthenticationError, unauthorizedResponse } from "@/lib/supabase/auth"; @@ -109,33 +109,7 @@ export async function GET(request: Request) { }); } - const fetchRecords = async () => { - const { data, error } = await supabase - .from("clinical_registry_records") - .select("*") - .eq("owner_id", access.ownerId) - .eq("kind", kind) - .order("title") - .limit(REGISTRY_MAX_RECORDS); - if (error) throw new Error(error.message); - return (data ?? []) as RegistryRecordRow[]; - }; - - let rows = await fetchRecords(); - if (rows.length === 0) { - // First visit for this owner: lazily seed the curated defaults so new - // accounts get populated Services/Forms instead of the empty state. Only - // the seed write is best-effort (a failure falls back to the empty set); - // the re-read stays outside the try so a genuine read failure still - // surfaces as an error rather than a misleading empty registry. - try { - await ensureRegistrySeeded(supabase, access.ownerId, kind); - } catch (seedError) { - console.error(`[registry] auto-seed failed for owner ${access.ownerId} (${kind})`, seedError); - } - rows = await fetchRecords(); - } - + const rows = await fetchOwnerRegistryRowsWithSeed(supabase, access.ownerId, kind, REGISTRY_MAX_RECORDS); const records = rows.map(rowToServiceRecord); const governanceBySlug = Object.fromEntries(rows.map((row) => [row.slug, rowGovernance(row)])); diff --git a/src/app/api/search/universal/route.ts b/src/app/api/search/universal/route.ts new file mode 100644 index 0000000000..fdffec438a --- /dev/null +++ b/src/app/api/search/universal/route.ts @@ -0,0 +1,96 @@ +import { NextResponse } from "next/server"; +import { z } from "zod"; + +import { + allowRateLimitInMemoryFallbackOnUnavailable, + consumeSubjectApiRateLimit, + rateLimitJsonResponse, +} from "@/lib/api-rate-limit"; +import { isDemoMode, isLocalNoAuthMode } from "@/lib/env"; +import { jsonError } from "@/lib/http"; +import { publicAccessContext, shouldResolvePublicCatalogAccess } from "@/lib/public-api-access"; +import { createAdminClient } from "@/lib/supabase/admin"; +import { AuthenticationError, unauthorizedResponse } from "@/lib/supabase/auth"; +import { + runUniversalSearch, + universalSearchDomains, + type UniversalSearchDomain, +} from "@/lib/universal-search"; +import { parseRequestQuery, queryInteger } from "@/lib/validation/query"; + +export const runtime = "nodejs"; + +// Typeahead-friendly GET: cross-entity federated search over documents + the registry +// catalogues. Access ladder mirrors /api/registry/records — demo/local serves fixtures, +// unauthenticated public serves the public catalogues, owners get their seeded records. +const universalSearchQuerySchema = z.object({ + q: z.string().trim().min(2).max(200), + limit: queryInteger({ fallback: 5, min: 1, max: 10 }), + domains: z + .string() + .trim() + .optional() + .transform((value) => { + if (!value) return undefined; + const requested = value + .split(",") + .map((domain) => domain.trim()) + .filter((domain): domain is UniversalSearchDomain => + (universalSearchDomains as string[]).includes(domain), + ); + return requested.length ? requested : undefined; + }), +}); + +function universalResponse(payload: Record) { + return NextResponse.json(payload, { headers: { "Cache-Control": "private, no-store" } }); +} + +export async function GET(request: Request) { + try { + const { q, limit, domains } = parseRequestQuery(request, universalSearchQuerySchema, "Invalid universal query."); + + if (isDemoMode() || isLocalNoAuthMode()) { + const payload = await runUniversalSearch({ query: q, limitPerDomain: limit, domains, demo: true }); + return universalResponse({ ...payload, demoMode: true }); + } + + if (!shouldResolvePublicCatalogAccess(request)) { + const payload = await runUniversalSearch({ query: q, limitPerDomain: limit, domains, demo: true }); + return universalResponse({ ...payload, publicAccess: true }); + } + + const supabase = createAdminClient(); + const access = await publicAccessContext(request, supabase); + + const rateLimit = await consumeSubjectApiRateLimit({ + supabase, + subject: access.rateLimitSubject, + bucket: "registry", + allowInMemoryFallbackOnUnavailable: allowRateLimitInMemoryFallbackOnUnavailable(), + }); + if (rateLimit.limited) { + return rateLimitJsonResponse("Universal search requests are rate limited. Try again shortly.", rateLimit); + } + + if (!access.ownerId) { + const payload = await runUniversalSearch({ query: q, limitPerDomain: limit, domains, demo: true }); + return universalResponse({ ...payload, publicAccess: true }); + } + + const payload = await runUniversalSearch({ + query: q, + limitPerDomain: limit, + domains, + supabase, + ownerId: access.ownerId, + demo: false, + }); + return universalResponse(payload); + } catch (error) { + if (error instanceof AuthenticationError) { + return unauthorizedResponse(); + } + return jsonError(error); + } +} diff --git a/src/lib/medication-seed.ts b/src/lib/medication-seed.ts index 36637d10e2..6f392443e6 100644 --- a/src/lib/medication-seed.ts +++ b/src/lib/medication-seed.ts @@ -18,3 +18,36 @@ export async function ensureMedicationsSeeded(supabase: AdminClient, ownerId: st } export { defaultMedicationRecords }; + +/** + * Fetch an owner's medication rows, lazily seeding the curated defaults on the + * first visit (extracted from /api/medications so the route and universal + * search share one code path). Seed write is best-effort; re-read is not. + */ +export async function fetchOwnerMedicationRowsWithSeed( + supabase: AdminClient, + ownerId: string, + maxRecords = 500, +): Promise { + const fetchRecords = async () => { + const { data, error } = await supabase + .from("medication_records") + .select("*") + .eq("owner_id", ownerId) + .order("name") + .limit(maxRecords); + if (error) throw new Error(error.message); + return (data ?? []) as MedicationRecordRow[]; + }; + + let rows = await fetchRecords(); + if (rows.length === 0) { + try { + await ensureMedicationsSeeded(supabase, ownerId); + } catch (seedError) { + console.error(`[medications] auto-seed failed for owner ${ownerId}`, seedError); + } + rows = await fetchRecords(); + } + return rows; +} diff --git a/src/lib/registry-seed.ts b/src/lib/registry-seed.ts index b61fbca099..875805a4d5 100644 --- a/src/lib/registry-seed.ts +++ b/src/lib/registry-seed.ts @@ -44,3 +44,40 @@ export async function ensureRegistrySeeded( if (error) throw new Error(`Registry seed failed: ${error.message}`); return (data ?? []) as RegistryRecordRow[]; } + +/** + * Fetch an owner's registry rows for a kind, lazily seeding the curated + * defaults on the first visit (the registry API's long-standing behaviour, + * extracted so /api/registry/records and universal search share one code + * path). The seed write is best-effort; the re-read is not, so a genuine + * read failure still surfaces instead of a misleading empty registry. + */ +export async function fetchOwnerRegistryRowsWithSeed( + supabase: AdminClient, + ownerId: string, + kind: RegistryRecordKind, + maxRecords = 500, +): Promise { + const fetchRecords = async () => { + const { data, error } = await supabase + .from("clinical_registry_records") + .select("*") + .eq("owner_id", ownerId) + .eq("kind", kind) + .order("title") + .limit(maxRecords); + if (error) throw new Error(error.message); + return (data ?? []) as RegistryRecordRow[]; + }; + + let rows = await fetchRecords(); + if (rows.length === 0) { + try { + await ensureRegistrySeeded(supabase, ownerId, kind); + } catch (seedError) { + console.error(`[registry] auto-seed failed for owner ${ownerId} (${kind})`, seedError); + } + rows = await fetchRecords(); + } + return rows; +} diff --git a/src/lib/universal-search.ts b/src/lib/universal-search.ts new file mode 100644 index 0000000000..5bb0c329e1 --- /dev/null +++ b/src/lib/universal-search.ts @@ -0,0 +1,298 @@ +import { demoSearch } from "@/lib/demo-data"; +import { fetchRelatedDocuments } from "@/lib/document-enrichment"; +import { documentsSearchHref } from "@/lib/document-flow-routes"; +import { rankDifferentialRecords } from "@/lib/differentials"; +import { formRecords, rankFormRecords, type FormRecord } from "@/lib/forms"; +import { rowToMedicationRecord } from "@/lib/medication-records"; +import { defaultMedicationRecords, fetchOwnerMedicationRowsWithSeed } from "@/lib/medication-seed"; +import { medicationIndication, rankMedicationRecords, type MedicationRecord } from "@/lib/medications"; +import { searchChunksWithTelemetry } from "@/lib/rag"; +import { rowToServiceRecord } from "@/lib/registry-records"; +import { fetchOwnerRegistryRowsWithSeed } from "@/lib/registry-seed"; +import { rankServiceRecords, serviceRecords, type ServiceRecord } from "@/lib/services"; +import { rankToolRecords } from "@/lib/tools-catalog"; +import type { SearchResult } from "@/lib/types"; + +// Server-side federated cross-entity search: one parallel in-process fan-out to the document +// retrieval pipeline plus the shared registry rankers (medications, services, forms, +// differentials, tools). Chosen over client-side federation (N round-trips per keystroke, +// duplicated auth/demo handling) and over ingesting registry rows into the eval-gated +// pgvector corpus (couples registry edits to reindexing; kept as a documented follow-up so +// Answer mode can eventually cite registry entities). + +type AdminClient = ReturnType; + +export type UniversalSearchDomain = "documents" | "medications" | "services" | "forms" | "differentials" | "tools"; + +export const universalSearchDomains: UniversalSearchDomain[] = [ + "documents", + "medications", + "services", + "forms", + "differentials", + "tools", +]; + +export type UniversalSearchItem = { + id: string; + kind: UniversalSearchDomain; + title: string; + subtitle?: string; + href: string; + // Comparable within a group only: registry scores are integer term-weights while document + // scores live in [0,1]. Cross-domain ordering is by fixed group order, never by score. + score: number; + badge?: string; + meta?: string; +}; + +export type UniversalSearchGroup = { + kind: UniversalSearchDomain; + total: number; + items: UniversalSearchItem[]; + latencyMs: number; + error?: boolean; +}; + +export type UniversalSearchResponse = { + query: string; + groups: UniversalSearchGroup[]; + tookMs: number; + demoMode?: boolean; + publicAccess?: boolean; +}; + +export type RunUniversalSearchArgs = { + query: string; + limitPerDomain: number; + domains?: UniversalSearchDomain[]; + // Live mode: both present. Demo/public mode: demo=true and the registry adapters serve + // fixtures without touching Supabase. + supabase?: AdminClient; + ownerId?: string; + demo: boolean; +}; + +const registryDomainTimeoutMs = 2500; +const documentsDomainTimeoutMs = 6000; + +function withTimeout(promise: Promise, timeoutMs: number, label: string): Promise { + return new Promise((resolve, reject) => { + const timer = setTimeout(() => reject(new Error(`${label} search timed out after ${timeoutMs}ms`)), timeoutMs); + promise.then( + (value) => { + clearTimeout(timer); + resolve(value); + }, + (error: unknown) => { + clearTimeout(timer); + reject(error instanceof Error ? error : new Error(String(error))); + }, + ); + }); +} + +function medicationItem(record: MedicationRecord, score: number): UniversalSearchItem { + return { + id: record.slug, + kind: "medications", + title: record.name, + subtitle: medicationIndication(record), + href: `/medications/${record.slug}`, + score, + badge: record.schedule || undefined, + meta: [record.class, record.subclass].filter(Boolean).join(" · ") || undefined, + }; +} + +function serviceItem(record: ServiceRecord, score: number): UniversalSearchItem { + return { + id: record.slug, + kind: "services", + title: record.title, + subtitle: record.subtitle ?? undefined, + href: `/services/${record.slug}`, + score, + badge: record.statusChips?.[0]?.label ?? undefined, + meta: record.primaryContact?.value ?? undefined, + }; +} + +function formItem(record: FormRecord, score: number): UniversalSearchItem { + return { + id: record.slug, + kind: "forms", + title: record.title, + subtitle: record.subtitle ?? undefined, + href: `/forms/${record.slug}`, + score, + badge: record.catalogueLabel ?? undefined, + }; +} + +async function searchMedicationsDomain(args: RunUniversalSearchArgs): Promise { + const records = + !args.demo && args.supabase && args.ownerId + ? (await fetchOwnerMedicationRowsWithSeed(args.supabase, args.ownerId)).map(rowToMedicationRecord) + : defaultMedicationRecords(); + return rankMedicationRecords(records, args.query, args.limitPerDomain).map((match) => + medicationItem(match.medication, match.score), + ); +} + +async function searchServicesDomain(args: RunUniversalSearchArgs): Promise { + const records = + !args.demo && args.supabase && args.ownerId + ? (await fetchOwnerRegistryRowsWithSeed(args.supabase, args.ownerId, "service")).map(rowToServiceRecord) + : serviceRecords; + return rankServiceRecords(records, args.query, args.limitPerDomain).map((match) => + serviceItem(match.service, match.score), + ); +} + +async function searchFormsDomain(args: RunUniversalSearchArgs): Promise { + const records = + !args.demo && args.supabase && args.ownerId + ? (await fetchOwnerRegistryRowsWithSeed(args.supabase, args.ownerId, "form")).map(rowToServiceRecord) + : formRecords; + return rankFormRecords(records, args.query, args.limitPerDomain).map((match) => formItem(match.service, match.score)); +} + +async function searchDifferentialsDomain(args: RunUniversalSearchArgs): Promise { + // Differentials are a static snapshot for list/search purposes (owner edits surface only on + // detail pages today), so demo and live share the in-bundle catalogue. + return rankDifferentialRecords(args.query, args.limitPerDomain).map((match) => ({ + id: match.record.slug, + kind: "differentials", + title: match.record.title, + subtitle: match.record.clinicalHinge || match.record.subtitle || undefined, + href: `/differentials/diagnoses/${match.record.slug}`, + score: match.score, + })); +} + +async function searchToolsDomain(args: RunUniversalSearchArgs): Promise { + return rankToolRecords(args.query, args.limitPerDomain).map((match) => ({ + id: match.tool.id, + kind: "tools", + title: match.tool.title, + subtitle: match.tool.bestFor, + href: match.tool.href, + score: match.score, + badge: match.tool.sourceBacked ? "Source-backed" : undefined, + })); +} + +function documentItemsFromChunks(results: SearchResult[], limit: number): UniversalSearchItem[] { + const byDocument = new Map(); + for (const result of results) { + const score = result.hybrid_score ?? result.similarity ?? 0; + const existing = byDocument.get(result.document_id); + if (existing) { + existing.score = Math.max(existing.score, score); + continue; + } + byDocument.set(result.document_id, { + id: result.document_id, + kind: "documents", + title: result.title, + subtitle: result.section_heading ?? undefined, + href: `/documents/${result.document_id}`, + score, + meta: result.file_name, + }); + } + return Array.from(byDocument.values()) + .sort((left, right) => right.score - left.score) + .slice(0, limit); +} + +async function searchDocumentsDomain(args: RunUniversalSearchArgs): Promise { + if (args.demo || !args.supabase) { + return documentItemsFromChunks(demoSearch(args.query, args.limitPerDomain * 3) as SearchResult[], args.limitPerDomain); + } + const { results } = await searchChunksWithTelemetry({ + query: args.query, + ownerId: args.ownerId, + topK: Math.max(6, args.limitPerDomain), + allowGlobalSearch: !args.ownerId, + }); + const related = await fetchRelatedDocuments({ + supabase: args.supabase, + ownerId: args.ownerId, + query: args.query, + results, + limit: args.limitPerDomain, + }); + if (related.length > 0) { + return related.map((document) => ({ + id: document.document_id, + kind: "documents" as const, + title: document.title, + subtitle: document.summary ?? undefined, + href: `/documents/${document.document_id}`, + score: document.score, + meta: document.match_reason, + })); + } + return documentItemsFromChunks(results, args.limitPerDomain); +} + +const domainAdapters: Record< + UniversalSearchDomain, + { run: (args: RunUniversalSearchArgs) => Promise; timeoutMs: number } +> = { + documents: { run: searchDocumentsDomain, timeoutMs: documentsDomainTimeoutMs }, + medications: { run: searchMedicationsDomain, timeoutMs: registryDomainTimeoutMs }, + services: { run: searchServicesDomain, timeoutMs: registryDomainTimeoutMs }, + forms: { run: searchFormsDomain, timeoutMs: registryDomainTimeoutMs }, + differentials: { run: searchDifferentialsDomain, timeoutMs: registryDomainTimeoutMs }, + tools: { run: searchToolsDomain, timeoutMs: registryDomainTimeoutMs }, +}; + +export async function runUniversalSearch(args: RunUniversalSearchArgs): Promise { + const startedAt = Date.now(); + const requested = args.domains?.length + ? universalSearchDomains.filter((domain) => args.domains!.includes(domain)) + : universalSearchDomains; + + const settled = await Promise.allSettled( + requested.map(async (domain): Promise => { + const domainStartedAt = Date.now(); + const adapter = domainAdapters[domain]; + const items = await withTimeout(adapter.run(args), adapter.timeoutMs, domain); + return { + kind: domain, + total: items.length, + items: items.slice(0, args.limitPerDomain), + latencyMs: Date.now() - domainStartedAt, + }; + }), + ); + + // A failed domain yields an empty errored group — one slow or broken adapter must never + // blank the whole response. + const groups = settled.map((result, index): UniversalSearchGroup => { + if (result.status === "fulfilled") return result.value; + return { kind: requested[index], total: 0, items: [], latencyMs: Date.now() - startedAt, error: true }; + }); + + return { query: args.query, groups, tookMs: Date.now() - startedAt }; +} + +export function universalSearchViewAllHref(domain: UniversalSearchDomain, query: string) { + switch (domain) { + case "documents": + return documentsSearchHref({ query, run: true }); + case "medications": + return `/?mode=prescribing&q=${encodeURIComponent(query)}&run=1`; + case "services": + return `/services?q=${encodeURIComponent(query)}&run=1`; + case "forms": + return `/forms?q=${encodeURIComponent(query)}&run=1`; + case "differentials": + return `/differentials?q=${encodeURIComponent(query)}&run=1`; + case "tools": + return `/?mode=tools&q=${encodeURIComponent(query)}&run=1`; + } +} diff --git a/tests/universal-search.test.ts b/tests/universal-search.test.ts new file mode 100644 index 0000000000..d7d0b30cba --- /dev/null +++ b/tests/universal-search.test.ts @@ -0,0 +1,116 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; + +afterEach(() => { + vi.unstubAllEnvs(); + vi.resetModules(); +}); + +async function loadUniversalSearch() { + return import("../src/lib/universal-search"); +} + +describe("runUniversalSearch (demo/fixtures path)", () => { + it("returns groups in the fixed domain order without touching Supabase", async () => { + const { runUniversalSearch, universalSearchDomains } = await loadUniversalSearch(); + const response = await runUniversalSearch({ query: "clinical", limitPerDomain: 5, demo: true }); + + expect(response.groups.map((group) => group.kind)).toEqual(universalSearchDomains); + for (const group of response.groups) { + expect(group.error).toBeUndefined(); + expect(group.items.length).toBeLessThanOrEqual(5); + } + }); + + it("finds fixture records per domain with working hrefs", async () => { + const { runUniversalSearch } = await loadUniversalSearch(); + const response = await runUniversalSearch({ query: "acamprosate", limitPerDomain: 5, demo: true }); + const medications = response.groups.find((group) => group.kind === "medications"); + expect(medications?.items[0]?.title.toLowerCase()).toContain("acamprosate"); + expect(medications?.items[0]?.href).toBe("/medications/acamprosate"); + + const differentialResponse = await runUniversalSearch({ query: "delirium", limitPerDomain: 5, demo: true }); + const differentials = differentialResponse.groups.find((group) => group.kind === "differentials"); + expect(differentials?.items.length ?? 0).toBeGreaterThan(0); + expect(differentials?.items[0]?.href).toContain("/differentials/diagnoses/"); + + const toolsResponse = await runUniversalSearch({ query: "forms", limitPerDomain: 5, demo: true }); + const forms = toolsResponse.groups.find((group) => group.kind === "forms"); + const tools = toolsResponse.groups.find((group) => group.kind === "tools"); + expect(tools?.items.some((item) => item.id === "forms")).toBe(true); + expect(forms?.items.every((item) => item.href.startsWith("/forms/"))).toBe(true); + }); + + it("filters to requested domains only", async () => { + const { runUniversalSearch } = await loadUniversalSearch(); + const response = await runUniversalSearch({ + query: "monitoring", + limitPerDomain: 3, + domains: ["tools", "differentials"], + demo: true, + }); + expect(response.groups.map((group) => group.kind)).toEqual(["documents", "medications", "services", "forms", "differentials", "tools"].filter((domain) => ["tools", "differentials"].includes(domain))); + }); + + it("isolates a failing domain instead of blanking the response", async () => { + vi.doMock("@/lib/tools-catalog", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + rankToolRecords: () => { + throw new Error("tools adapter exploded"); + }, + }; + }); + const { runUniversalSearch } = await loadUniversalSearch(); + const response = await runUniversalSearch({ query: "monitoring", limitPerDomain: 3, demo: true }); + + const tools = response.groups.find((group) => group.kind === "tools"); + expect(tools?.error).toBe(true); + expect(tools?.items).toEqual([]); + const differentials = response.groups.find((group) => group.kind === "differentials"); + expect(differentials?.error).toBeUndefined(); + }); + + it("uses demo document search when no Supabase client is supplied", async () => { + const { runUniversalSearch } = await loadUniversalSearch(); + const response = await runUniversalSearch({ query: "clozapine monitoring", limitPerDomain: 4, demo: true }); + const documents = response.groups.find((group) => group.kind === "documents"); + expect(documents?.items.length ?? 0).toBeGreaterThan(0); + expect(documents?.items[0]?.href).toContain("/documents/"); + }); +}); + +describe("GET /api/search/universal (demo mode)", () => { + it("serves fixture-backed groups with demoMode flagged", async () => { + vi.stubEnv("NEXT_PUBLIC_DEMO_MODE", "true"); + const { GET } = await import("../src/app/api/search/universal/route"); + const response = await GET(new Request("http://localhost/api/search/universal?q=acamprosate&limit=3")); + expect(response.status).toBe(200); + expect(response.headers.get("Cache-Control")).toBe("private, no-store"); + const payload = (await response.json()) as { + demoMode?: boolean; + groups: Array<{ kind: string; items: Array<{ title: string }> }>; + }; + expect(payload.demoMode).toBe(true); + const medications = payload.groups.find((group) => group.kind === "medications"); + expect(medications?.items[0]?.title.toLowerCase()).toContain("acamprosate"); + }); + + it("rejects queries under the minimum length", async () => { + vi.stubEnv("NEXT_PUBLIC_DEMO_MODE", "true"); + const { GET } = await import("../src/app/api/search/universal/route"); + const response = await GET(new Request("http://localhost/api/search/universal?q=a")); + expect(response.status).toBe(400); + }); + + it("ignores unknown domains in the CSV filter", async () => { + vi.stubEnv("NEXT_PUBLIC_DEMO_MODE", "true"); + const { GET } = await import("../src/app/api/search/universal/route"); + const response = await GET( + new Request("http://localhost/api/search/universal?q=monitoring&domains=tools,bogus"), + ); + expect(response.status).toBe(200); + const payload = (await response.json()) as { groups: Array<{ kind: string }> }; + expect(payload.groups.map((group) => group.kind)).toEqual(["tools"]); + }); +}); From 2d9a2eaa929ad769ac2b979940da409e5554a4d1 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 11:41:48 +0000 Subject: [PATCH 5/6] Surface cross-entity universal search as live typeahead in the command surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 4 of the universal search workstream: - New use-universal-search hook: 250ms-debounced GET /api/search/universal, enabled only while the command dropdown is open with a 2+ char query, excluding the active mode's own domain (its results already come from the mode search). Race handling mirrors the dashboard's monotonic request token, and visible groups are derived from the fetched query so a stale result set is never rendered against a newer query. - UniversalSearchCommandSurface renders the grouped cross-entity results ("Medications · 3", "Forms · 1", ...) between suggestions and mode actions; selecting an item navigates straight to the record, and each group ends with a "View all in {mode}" row reusing the existing cross-mode plumbing. Keyboard navigation and aria come free from the existing flatItems listbox. Enter with nothing highlighted still runs the mode-scoped search; errored/empty domains render nothing; a subtle spinner row shows while the federated request is pending. - use-medication-catalog now sends the auth session header (matching use-registry-records) — an authenticated owner was previously served the public fixture catalogue silently. - New tests/ui-universal-search.spec.ts (registered in the Playwright testMatch): grouped section renders while typing, selection navigates to /medications/{slug}, Enter preserves mode search. Verified green on Chromium against a live demo-mode dev server, along with the full ui-tools suites (40/40) after the Phase-2 tools dedupe. - Verification state and remaining eval debt recorded in docs/process-hardening.md (golden retrieval eval requires live keys and must run before merge per the standing gate). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_011QiyE8jMm7VnrtknHF6jnJ --- docs/process-hardening.md | 7 + playwright.config.ts | 2 +- .../universal-search-command-surface.tsx | 122 +++++++++++++++++- .../use-medication-catalog.ts | 17 ++- .../use-universal-search.ts | 85 ++++++++++++ tests/ui-universal-search.spec.ts | 93 +++++++++++++ 6 files changed, 317 insertions(+), 9 deletions(-) create mode 100644 src/components/clinical-dashboard/use-universal-search.ts create mode 100644 tests/ui-universal-search.spec.ts diff --git a/docs/process-hardening.md b/docs/process-hardening.md index f6a53a4eed..ab225dc90d 100644 --- a/docs/process-hardening.md +++ b/docs/process-hardening.md @@ -161,3 +161,10 @@ All approved render-surface modules are extracted. `ClinicalDashboard.tsx` went - **P2 M13:** `20260702000000_commit_generation_preserve_legacy_artifacts.sql` must be applied to live Supabase before reindex commits can safely purge legacy NULL-generation rows. After apply, run `npm run check:m13-migration`, `npm run reindex:health`, and `npm run check:indexing`. `search_schema_health()` now reports `commit_document_index_generation.preserve_legacy_artifacts_migration` when the live function body is stale. - **P2 upload hardening:** `/api/upload` consumes the `document_upload` rate-limit bucket (12/min owner, 3/min anonymous). - **P3 dispositioned (no code change):** L9 searchable-only `image_count` (documented in `worker/main.ts`); L11 triple `readFile` peak-memory trade-off (documented at the ingestion site); L18 duplicate `audit_logs` policy in an already-applied migration (do not edit applied migrations — consolidate only if migrations are ever squashed); L19 CSP `script-src 'unsafe-inline'` deferred (no active XSS sink today; nonce migration needs dedicated UI verification). + +## Universal search workstream — verification state (2026-07-06) + +- **Shipped on `claude/universal-search-algorithm-ryrps7`:** finding #11 interim fix (classifier-verdict memoization, 15-min TTL, errors not memoized — the "cheaper interim option" from the 2026-07-03 entry above), pre-clamp tiebreak completion in `selectRetrievalEvidence`, weak-match OR-augmentation (kill switch `RAG_TEXT_WEAK_OR_RELAXATION=false`), `similarity_origin` telemetry, shared `catalog-search` primitives replacing the four per-domain rankers, forms mode-kind honesty, tools dataset dedupe, `/api/search/universal` federated endpoint, and the cross-entity typeahead in `UniversalSearchCommandSurface`. +- **Eval debt (blocking merge, not development):** `npm run eval:retrieval:quality` (23/23) and `eval:quality --rag-only` (`unsupported_correct_rate` 1.0) could NOT be run in the authoring environment (no live keys) — they MUST be run before merge per the standing gate above, with special attention to the weak-match OR-augmentation (flag off restores relax-on-empty exactly) and the retrieval-selection tiebreak (tie-only by construction). +- **UI verification run:** new `tests/ui-universal-search.spec.ts` (grouped typeahead renders, item selection navigates, Enter still runs the mode search — universal endpoint mocked), full `ui-tools`/`ui-tools-task-directory` (40/40) and `ui-smoke`/`ui-overlap` suites against a live dev server in demo mode, plus a live curl of `/api/search/universal` (grouped payload, domain filter, 400 on short query). +- **Known limitation:** the typeahead spec mocks the universal endpoint; an end-to-end spec against live seeded registries needs the owner-auth Playwright project (E2E_USER_* keys). diff --git a/playwright.config.ts b/playwright.config.ts index b858b95255..d264bec515 100644 --- a/playwright.config.ts +++ b/playwright.config.ts @@ -5,7 +5,7 @@ const baseURL = getPlaywrightBaseUrl(); export default defineConfig({ testDir: "./tests", - testMatch: /.*ui-(smoke|stress|accessibility|tools|tools-task-directory|overlap)\.spec\.ts/, + testMatch: /.*ui-(smoke|stress|accessibility|tools|tools-task-directory|overlap|universal-search)\.spec\.ts/, timeout: 60_000, retries: process.env.CI ? 1 : 0, expect: { diff --git a/src/components/clinical-dashboard/universal-search-command-surface.tsx b/src/components/clinical-dashboard/universal-search-command-surface.tsx index 2252c2c7e3..066aa98787 100644 --- a/src/components/clinical-dashboard/universal-search-command-surface.tsx +++ b/src/components/clinical-dashboard/universal-search-command-surface.tsx @@ -1,6 +1,7 @@ "use client"; -import { AlertTriangle, Clock, CornerDownLeft, Search, X } from "lucide-react"; +import { AlertTriangle, Clock, CornerDownLeft, Loader2, Search, X } from "lucide-react"; +import { useRouter } from "next/navigation"; import { useEffect, useId, useMemo, useState, type KeyboardEvent as ReactKeyboardEvent, type ReactNode } from "react"; import { @@ -9,6 +10,7 @@ import { type ModeActionSetId, } from "@/components/clinical-dashboard/mode-action-popup"; import { AnswerSuggestionChips } from "@/components/clinical-dashboard/answer-suggestion-chips"; +import { useUniversalSearch } from "@/components/clinical-dashboard/use-universal-search"; import { cn } from "@/components/ui-primitives"; import { appModeDefinition, type AppModeId } from "@/lib/app-modes"; import { appModeIcons } from "@/lib/app-mode-icons"; @@ -18,6 +20,37 @@ import { isFormCodeQuery, searchCommandSurfaceConfig, } from "@/lib/search-command-surface"; +import type { UniversalSearchDomain } from "@/lib/universal-search"; + +// Each mode's own domain is excluded from the cross-entity typeahead — its results already +// come from the mode search itself. Answer mode maps to documents (that is its corpus). +const excludedDomainByMode: Partial> = { + answer: "documents", + documents: "documents", + services: "services", + forms: "forms", + differentials: "differentials", + prescribing: "medications", + tools: "tools", +}; + +const modeIdByDomain: Record = { + documents: "documents", + medications: "prescribing", + services: "services", + forms: "forms", + differentials: "differentials", + tools: "tools", +}; + +const domainHeadings: Record = { + documents: "Documents", + medications: "Medications", + services: "Services", + forms: "Forms", + differentials: "Differentials", + tools: "Tools", +}; const focusRing = "focus-visible:outline focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-[color:var(--focus)]"; @@ -144,6 +177,7 @@ function CommandDropdown({ activeItemId, sections, showSafetyBanner, + universalPending, onHoverItem, placement, }: { @@ -153,6 +187,7 @@ function CommandDropdown({ activeItemId: string | null; sections: Array<{ key: string; heading?: string; layout?: "list" | "chips"; items: DropdownItem[] }>; showSafetyBanner: boolean; + universalPending: boolean; onHoverItem: (id: string) => void; placement: CommandSurfacePlacement; }) { @@ -241,7 +276,16 @@ function CommandDropdown({ ) : null, )} - {!hasItems ? ( + {universalPending ? ( +
+ + Searching across Clinical KB… +
+ ) : null} + {!hasItems && !universalPending ? (
Press Enter to run the full {mode.label.toLowerCase()} search.
@@ -300,9 +344,15 @@ export function UniversalSearchCommandSurface({ }) { const config = searchCommandSurfaceConfig(modeId); const listboxId = useId(); + const router = useRouter(); const [activeIndex, setActiveIndex] = useState(-1); const trimmedQuery = query.trim(); const mode = appModeDefinition(modeId); + const universal = useUniversalSearch({ + query: trimmedQuery, + enabled: dropdownOpen && Boolean(config), + excludeDomain: excludedDomainByMode[modeId], + }); const showSafetyBanner = modeId === "differentials" && differentialRedFlagTerms.some((term) => trimmedQuery.toLowerCase().includes(term)); @@ -399,6 +449,70 @@ export function UniversalSearchCommandSurface({ } } + // Cross-entity typeahead ("Across Clinical KB"): live grouped matches from the universal + // search endpoint, excluding this mode's own domain. Selecting an item navigates straight + // to the record; each group ends with a cross-mode "view all" that re-runs the query in + // the owning mode. Enter with nothing highlighted still runs the mode-scoped search. + if (trimmedQuery && universal.query === trimmedQuery && universal.groups.length) { + for (const group of universal.groups) { + const targetModeId = modeIdByDomain[group.kind]; + const targetMode = appModeDefinition(targetModeId); + const GroupIcon = appModeIcons[targetModeId]; + built.push({ + key: `universal-${group.kind}`, + heading: `${domainHeadings[group.kind]} · ${group.total}`, + items: [ + ...group.items.map((item) => ({ + id: nextId(), + label: item.title, + onSelect: () => { + onDropdownOpenChange(false); + router.push(item.href); + }, + render: (active: boolean) => ( + + + + + + {item.title} + {item.subtitle ? ( + + {item.subtitle} + + ) : null} + + {item.badge ? ( + + {item.badge} + + ) : null} + + ), + })), + { + id: nextId(), + label: `View all in ${targetMode.label}`, + onSelect: () => { + onDropdownOpenChange(false); + onCrossMode(targetModeId, trimmedQuery); + }, + render: (active: boolean) => ( + + + + + + View all in {targetMode.label} + + + ), + }, + ], + }); + } + } + const actionSetId: ModeActionSetId | null = modeId === "documents" || modeId === "forms" || modeId === "prescribing" ? "documents" @@ -488,8 +602,11 @@ export function UniversalSearchCommandSurface({ onRunModeAction, onSearch, recentQueries, + router, showFormCodeHint, trimmedQuery, + universal.groups, + universal.query, ]); const flatItems = useMemo(() => sections.flatMap((section) => section.items), [sections]); @@ -597,6 +714,7 @@ export function UniversalSearchCommandSurface({ activeItemId={activeItemId} sections={sections} showSafetyBanner={showSafetyBanner} + universalPending={universal.loading && Boolean(trimmedQuery)} placement={placement} onHoverItem={(id) => { const index = flatItems.findIndex((item) => item.id === id); diff --git a/src/components/clinical-dashboard/use-medication-catalog.ts b/src/components/clinical-dashboard/use-medication-catalog.ts index 63271e7116..dbf19e0ca5 100644 --- a/src/components/clinical-dashboard/use-medication-catalog.ts +++ b/src/components/clinical-dashboard/use-medication-catalog.ts @@ -3,6 +3,7 @@ import { useEffect, useState } from "react"; import type { MedicationRecord, MedicationSearchResult } from "@/lib/medications"; +import { useAuthSession } from "@/lib/supabase/client"; type MedicationCatalogMatch = { medication: MedicationRecord; @@ -33,8 +34,8 @@ type AsyncState = { error: string | null; }; -async function fetchJson(url: string): Promise { - const response = await fetch(url, { cache: "no-store" }); +async function fetchJson(url: string, headers?: HeadersInit): Promise { + const response = await fetch(url, { cache: "no-store", headers }); if (!response.ok) { throw new Error(`Request failed (${response.status})`); } @@ -43,6 +44,9 @@ async function fetchJson(url: string): Promise { export function useMedicationCatalog(query?: string): AsyncState { const trimmed = query?.trim() ?? ""; + // Auth-aware like use-registry-records: without the header an authenticated owner was + // silently served the public fixture catalogue instead of their seeded records. + const { authorizationHeader } = useAuthSession(); const [prevQuery, setPrevQuery] = useState(trimmed); const [state, setState] = useState>({ data: null, @@ -62,7 +66,7 @@ export function useMedicationCatalog(query?: string): AsyncState { let cancelled = false; const url = trimmed ? `/api/medications?q=${encodeURIComponent(trimmed)}` : "/api/medications"; - fetchJson(url) + fetchJson(url, authorizationHeader) .then((data) => { if (!cancelled) setState({ data, loading: false, error: null }); }) @@ -78,13 +82,14 @@ export function useMedicationCatalog(query?: string): AsyncState { cancelled = true; }; - }, [trimmed]); + }, [trimmed, authorizationHeader]); return state; } export function useMedicationDetail(slug?: string): AsyncState { const normalized = slug?.trim().toLowerCase() ?? ""; + const { authorizationHeader } = useAuthSession(); const [prevSlug, setPrevSlug] = useState(normalized); const [state, setState] = useState>(() => ({ data: null, @@ -106,7 +111,7 @@ export function useMedicationDetail(slug?: string): AsyncState(`/api/medications/${encodeURIComponent(normalized)}`) + fetchJson(`/api/medications/${encodeURIComponent(normalized)}`, authorizationHeader) .then((data) => { if (!cancelled) setState({ data, loading: false, error: null }); }) @@ -122,7 +127,7 @@ export function useMedicationDetail(slug?: string): AsyncState { cancelled = true; }; - }, [normalized]); + }, [normalized, authorizationHeader]); return state; } diff --git a/src/components/clinical-dashboard/use-universal-search.ts b/src/components/clinical-dashboard/use-universal-search.ts new file mode 100644 index 0000000000..590b2f8ae7 --- /dev/null +++ b/src/components/clinical-dashboard/use-universal-search.ts @@ -0,0 +1,85 @@ +"use client"; + +import { useEffect, useRef, useState } from "react"; + +import type { UniversalSearchDomain, UniversalSearchGroup } from "@/lib/universal-search"; +import { useAuthSession } from "@/lib/supabase/client"; + +export type UniversalSearchState = { + groups: UniversalSearchGroup[]; + loading: boolean; + /** The query the current groups were computed for (guards stale renders). */ + query: string; +}; + +const debounceMs = 250; +const minQueryLength = 2; + +/** + * Cross-entity typeahead for the command surface: debounced GET + * /api/search/universal excluding the active mode's own domain (its results + * already come from the mode search itself). Race handling mirrors the + * dashboard's monotonic searchRequestSeqRef — stale responses are dropped, + * never committed; visible groups are derived from the fetched query so a + * stale result set is never rendered against a newer query. + */ +export function useUniversalSearch(args: { + query: string; + enabled: boolean; + excludeDomain?: UniversalSearchDomain; + limitPerDomain?: number; +}): UniversalSearchState { + const { authorizationHeader } = useAuthSession(); + const [result, setResult] = useState<{ groups: UniversalSearchGroup[]; query: string }>({ groups: [], query: "" }); + const requestSeqRef = useRef(0); + const trimmedQuery = args.query.trim(); + const active = args.enabled && trimmedQuery.length >= minQueryLength; + const limitPerDomain = args.limitPerDomain ?? 3; + const excludeDomain = args.excludeDomain; + + useEffect(() => { + if (!active) { + // Invalidate any in-flight request; visible state is derived, so no reset needed. + requestSeqRef.current += 1; + return undefined; + } + + const requestId = ++requestSeqRef.current; + const timer = window.setTimeout(() => { + const domains = (["documents", "medications", "services", "forms", "differentials", "tools"] as const).filter( + (domain) => domain !== excludeDomain, + ); + const params = new URLSearchParams({ + q: trimmedQuery, + limit: String(limitPerDomain), + domains: domains.join(","), + }); + fetch(`/api/search/universal?${params.toString()}`, { headers: authorizationHeader }) + .then(async (response) => { + if (requestId !== requestSeqRef.current) return; + if (!response.ok) { + setResult({ groups: [], query: trimmedQuery }); + return; + } + const payload = (await response.json()) as { groups?: UniversalSearchGroup[] }; + if (requestId !== requestSeqRef.current) return; + setResult({ + groups: (payload.groups ?? []).filter((group) => !group.error && group.items.length > 0), + query: trimmedQuery, + }); + }) + .catch(() => { + if (requestId !== requestSeqRef.current) return; + setResult({ groups: [], query: trimmedQuery }); + }); + }, debounceMs); + + return () => { + window.clearTimeout(timer); + }; + }, [active, trimmedQuery, excludeDomain, limitPerDomain, authorizationHeader]); + + if (!active) return { groups: [], loading: false, query: "" }; + const fresh = result.query === trimmedQuery; + return { groups: fresh ? result.groups : [], loading: !fresh, query: result.query }; +} diff --git a/tests/ui-universal-search.spec.ts b/tests/ui-universal-search.spec.ts new file mode 100644 index 0000000000..557a186742 --- /dev/null +++ b/tests/ui-universal-search.spec.ts @@ -0,0 +1,93 @@ +import { expect, test, type Page } from "playwright/test"; + +// Cross-entity universal typeahead in the command surface. The universal endpoint is +// mocked so this spec exercises the UI contract (grouped sections, navigation, +// mode-search preservation) deterministically in demo mode without live retrieval. + +const universalPayload = { + query: "acamprosate", + tookMs: 12, + demoMode: true, + groups: [ + { + kind: "medications", + total: 1, + latencyMs: 4, + items: [ + { + id: "acamprosate", + kind: "medications", + title: "Acamprosate", + subtitle: "Alcohol dependence — maintenance of abstinence", + href: "/medications/acamprosate", + score: 22, + badge: "S4", + }, + ], + }, + { + kind: "forms", + total: 1, + latencyMs: 3, + items: [ + { + id: "transfer-form", + kind: "forms", + title: "Transfer order form", + href: "/forms/transfer-form", + score: 9, + }, + ], + }, + ], +}; + +async function mockUniversalSearch(page: Page) { + await page.route(/\/api\/search\/universal(?:\?.*)?$/, async (route) => { + await route.fulfill({ json: universalPayload }); + }); +} + +async function openComposer(page: Page) { + await page.goto("/?mode=documents&focus=1"); + const input = page.getByTestId("global-search-input").first(); + await input.click(); + return input; +} + +test.describe("universal search typeahead", () => { + test("shows grouped cross-entity results while typing", async ({ page }) => { + await mockUniversalSearch(page); + const input = await openComposer(page); + await input.fill("acamprosate"); + + await expect(page.getByText("Medications · 1")).toBeVisible(); + await expect(page.getByRole("option", { name: /Acamprosate/ })).toBeVisible(); + await expect(page.getByText("Forms · 1")).toBeVisible(); + await expect(page.getByRole("option", { name: /View all in Medication/ })).toBeVisible(); + }); + + test("selecting a grouped result navigates to the record", async ({ page }) => { + await mockUniversalSearch(page); + const input = await openComposer(page); + await input.fill("acamprosate"); + + const option = page.getByRole("option", { name: /Acamprosate/ }); + await expect(option).toBeVisible(); + await option.click(); + await expect(page).toHaveURL(/\/medications\/acamprosate/); + }); + + test("Enter with nothing highlighted still runs the mode-scoped search", async ({ page }) => { + await mockUniversalSearch(page); + const input = await openComposer(page); + await input.fill("clozapine monitoring"); + await expect(page.getByText("Medications · 1")).toBeVisible(); + await input.press("Enter"); + + // Documents mode routes an Enter submit to the document search flow; the dropdown + // closes and the app stays on a documents surface rather than a registry page. + await expect(page.getByText("Medications · 1")).toBeHidden(); + await expect(page).not.toHaveURL(/\/medications\//); + }); +}); From 716b5964c0a704e9f5291359e03d25cb86e1b51e Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 11:46:20 +0000 Subject: [PATCH 6/6] Apply Prettier formatting across universal-search workstream files npm run format:check is part of the PR verification contract; nine files from the preceding phases needed --write. No behaviour change. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_011QiyE8jMm7VnrtknHF6jnJ --- src/app/api/search/universal/route.ts | 10 ++-------- src/lib/app-modes.ts | 6 ++++-- src/lib/catalog-search.ts | 4 +++- src/lib/medications.ts | 4 +--- src/lib/services.ts | 1 - src/lib/universal-search.ts | 5 ++++- tests/rag-classifier-memo.test.ts | 5 +---- tests/retrieval-query-variants.test.ts | 6 +++--- tests/universal-search.test.ts | 10 ++++++---- 9 files changed, 24 insertions(+), 27 deletions(-) diff --git a/src/app/api/search/universal/route.ts b/src/app/api/search/universal/route.ts index fdffec438a..210cd1bf31 100644 --- a/src/app/api/search/universal/route.ts +++ b/src/app/api/search/universal/route.ts @@ -11,11 +11,7 @@ import { jsonError } from "@/lib/http"; import { publicAccessContext, shouldResolvePublicCatalogAccess } from "@/lib/public-api-access"; import { createAdminClient } from "@/lib/supabase/admin"; import { AuthenticationError, unauthorizedResponse } from "@/lib/supabase/auth"; -import { - runUniversalSearch, - universalSearchDomains, - type UniversalSearchDomain, -} from "@/lib/universal-search"; +import { runUniversalSearch, universalSearchDomains, type UniversalSearchDomain } from "@/lib/universal-search"; import { parseRequestQuery, queryInteger } from "@/lib/validation/query"; export const runtime = "nodejs"; @@ -35,9 +31,7 @@ const universalSearchQuerySchema = z.object({ const requested = value .split(",") .map((domain) => domain.trim()) - .filter((domain): domain is UniversalSearchDomain => - (universalSearchDomains as string[]).includes(domain), - ); + .filter((domain): domain is UniversalSearchDomain => (universalSearchDomains as string[]).includes(domain)); return requested.length ? requested : undefined; }), }); diff --git a/src/lib/app-modes.ts b/src/lib/app-modes.ts index 088a5a37c1..5691a950dc 100644 --- a/src/lib/app-modes.ts +++ b/src/lib/app-modes.ts @@ -5,8 +5,10 @@ export type AppModeId = "answer" | "documents" | "services" | "forms" | "favourites" | "differentials" | "prescribing" | "tools"; export type SearchableAppModeId = AppModeId; -export type AppModeSearchKind = "answer" | "documents" | "services" | "forms" | "favourites" | "differentials" | "tools"; -export type AppModeResultKind = "answer" | "documents" | "services" | "forms" | "favourites" | "differentials" | "tools"; +export type AppModeSearchKind = + "answer" | "documents" | "services" | "forms" | "favourites" | "differentials" | "tools"; +export type AppModeResultKind = + "answer" | "documents" | "services" | "forms" | "favourites" | "differentials" | "tools"; export type AppModeSearchConfig = { kind: AppModeSearchKind; diff --git a/src/lib/catalog-search.ts b/src/lib/catalog-search.ts index 8b1e1686d4..20ad2c1436 100644 --- a/src/lib/catalog-search.ts +++ b/src/lib/catalog-search.ts @@ -115,7 +115,9 @@ export function rankCatalogRecords( compactBonus > 0 && compactQuery.length >= compactMinLength && (compactSearchText(text).includes(compactQuery) || - (options.compactExtraText ? compactSearchText(options.compactExtraText(record)).includes(compactQuery) : false)); + (options.compactExtraText + ? compactSearchText(options.compactExtraText(record)).includes(compactQuery) + : false)); if (compact) score += compactBonus; const phrase = phraseBonus > 0 && text.includes(normalizedQuery); diff --git a/src/lib/medications.ts b/src/lib/medications.ts index 29987f6113..4c4873c2a4 100644 --- a/src/lib/medications.ts +++ b/src/lib/medications.ts @@ -202,9 +202,7 @@ export function rankMedicationRecords(records: MedicationRecord[], query: string weight: 3, text: (medication) => normalizeSearchText( - [medication.class, medication.subclass, medication.category, medication.tag, medication.schedule].join( - " ", - ), + [medication.class, medication.subclass, medication.category, medication.tag, medication.schedule].join(" "), ), }, ], diff --git a/src/lib/services.ts b/src/lib/services.ts index 4814ef0a67..2350d456fc 100644 --- a/src/lib/services.ts +++ b/src/lib/services.ts @@ -105,7 +105,6 @@ export function serviceNavigatorQuery(service: ServiceRecord) { ); } - function serviceRecordSearchParts(service: ServiceRecord) { return [ service.title, diff --git a/src/lib/universal-search.ts b/src/lib/universal-search.ts index 5bb0c329e1..ee2e791509 100644 --- a/src/lib/universal-search.ts +++ b/src/lib/universal-search.ts @@ -209,7 +209,10 @@ function documentItemsFromChunks(results: SearchResult[], limit: number): Univer async function searchDocumentsDomain(args: RunUniversalSearchArgs): Promise { if (args.demo || !args.supabase) { - return documentItemsFromChunks(demoSearch(args.query, args.limitPerDomain * 3) as SearchResult[], args.limitPerDomain); + return documentItemsFromChunks( + demoSearch(args.query, args.limitPerDomain * 3) as SearchResult[], + args.limitPerDomain, + ); } const { results } = await searchChunksWithTelemetry({ query: args.query, diff --git a/tests/rag-classifier-memo.test.ts b/tests/rag-classifier-memo.test.ts index a19310dc84..dc180b878a 100644 --- a/tests/rag-classifier-memo.test.ts +++ b/tests/rag-classifier-memo.test.ts @@ -75,10 +75,7 @@ describe("classifier verdict memoization", () => { }); it("does not memoize transport errors — the next request retries the classifier", async () => { - const mock = vi - .fn() - .mockRejectedValueOnce(new Error("timeout")) - .mockResolvedValueOnce(classifierResponse()); + const mock = vi.fn().mockRejectedValueOnce(new Error("timeout")).mockResolvedValueOnce(classifierResponse()); const { rag, analyzeClinicalQuery } = await loadWithClassifierMock(mock); const { query, analysis } = fallbackQueryAnalysis(analyzeClinicalQuery); diff --git a/tests/retrieval-query-variants.test.ts b/tests/retrieval-query-variants.test.ts index 723c41897b..829b4bdee8 100644 --- a/tests/retrieval-query-variants.test.ts +++ b/tests/retrieval-query-variants.test.ts @@ -1051,9 +1051,9 @@ describe("shouldRelaxWeakTextMatches (P8b weak-augment)", () => { it("does not fire when a sparse set is anchored by a strong lexical hit", () => { // A single precise match (e.g. an exact table lookup) must stay a one-RPC retrieval. expect(shouldRelaxWeakTextMatches([result({ text_rank: 1.1 })])).toBe(false); - expect(shouldRelaxWeakTextMatches([result({ id: "a", text_rank: 0.4 }), result({ id: "b", text_rank: 0.05 })])).toBe( - false, - ); + expect( + shouldRelaxWeakTextMatches([result({ id: "a", text_rank: 0.4 }), result({ id: "b", text_rank: 0.05 })]), + ).toBe(false); }); it("fires when the best strict text rank is below the meaningful-signal floor", () => { diff --git a/tests/universal-search.test.ts b/tests/universal-search.test.ts index d7d0b30cba..2a6b7f925b 100644 --- a/tests/universal-search.test.ts +++ b/tests/universal-search.test.ts @@ -48,7 +48,11 @@ describe("runUniversalSearch (demo/fixtures path)", () => { domains: ["tools", "differentials"], demo: true, }); - expect(response.groups.map((group) => group.kind)).toEqual(["documents", "medications", "services", "forms", "differentials", "tools"].filter((domain) => ["tools", "differentials"].includes(domain))); + expect(response.groups.map((group) => group.kind)).toEqual( + ["documents", "medications", "services", "forms", "differentials", "tools"].filter((domain) => + ["tools", "differentials"].includes(domain), + ), + ); }); it("isolates a failing domain instead of blanking the response", async () => { @@ -106,9 +110,7 @@ describe("GET /api/search/universal (demo mode)", () => { it("ignores unknown domains in the CSV filter", async () => { vi.stubEnv("NEXT_PUBLIC_DEMO_MODE", "true"); const { GET } = await import("../src/app/api/search/universal/route"); - const response = await GET( - new Request("http://localhost/api/search/universal?q=monitoring&domains=tools,bogus"), - ); + const response = await GET(new Request("http://localhost/api/search/universal?q=monitoring&domains=tools,bogus")); expect(response.status).toBe(200); const payload = (await response.json()) as { groups: Array<{ kind: string }> }; expect(payload.groups.map((group) => group.kind)).toEqual(["tools"]);