Universal search: fix core RAG search defects and federate medications/services/forms/differentials/tools into one cross-entity search

[BigSimmo]

Summary

• Core RAG search fixes (each independently revertable):
  • Finding Improve Clinical KB dashboard and RAG hardening #11 interim fix — memoize the LLM query-classifier verdict (15-min TTL, in-flight dedup, errors never cached) so the unsupported short-circuit can no longer flip run-to-run and intermittently return 0 results for valid queries like "bipolar disorder" (src/lib/rag.ts).
  • Complete the 1.0-saturation tiebreak — rankClinicalResults now emits its pre-clamp boost sum (score_explanation.preClampFinalScore) and selectRetrievalEvidence uses it after score/lexical/rerank ties, so fully tied saturated candidates no longer order by arbitrary chunk id. Tie-only by construction; the clamped primary score is untouched.
  • Weak-match OR-augmentation (P8b extension) — strict-AND text retrieval that returns a sparse set of middling matches (or negligible top text_rank) now appends OR-relaxed recall behind the strict matches (append-only merge). A sparse set anchored by a strong lexical hit (top rank ≥ 0.3) does not relax, so exact lookups stay one-RPC retrievals. Kill switch: RAG_TEXT_WEAK_OR_RELAXATION=false.
  • RC9 observability — fabricated (non-cosine) similarities from the document-lookup, memory-card, and table-facts paths are tagged similarity_origin:"synthetic_text" and counted in telemetry ahead of any gate recalibration.
  • Housekeeping — the duplicated weak-search 0.48 literal is now weakRetrievalTopScoreThreshold in rag-routing.ts with an ordering test; stale 12000ms answer-budget docs reconciled with the deliberate 30000ms default; six follow-ups filed in docs/rag-hybrid-findings-and-todo.md.
• Shared catalog-search primitives — one canonical normalizeSearchText and a generic rankCatalogRecords replace the four divergent per-domain rankers. Medications/services/forms wrappers keep their exact historical weights, reason strings, and match shapes (tests/{medications,services,forms}.test.ts pass unmodified as parity oracles). Differentials get real relevance ranking with alias expansion (previously an unranked includes() filter). The duplicated Tools dataset is consolidated into src/lib/tools-catalog.ts (icon-free so server code can use it). Forms stop masquerading as kind:"documents" (new honest forms kind removes the dashboard special-casing); prescribing deliberately keeps kind:"documents" with the rationale recorded inline.
• Universal search endpoint — new GET /api/search/universal federates one parallel in-process fan-out (Promise.allSettled, per-domain timeouts, error isolation) across document retrieval (searchChunksWithTelemetry + fetchRelatedDocuments) and the shared registry rankers. Access ladder mirrors /api/registry/records (demo → fixtures, public → public catalogues, owner → lazily-seeded records, shared rate-limit bucket). No cross-domain score normalization — registry integer weights and [0,1] document scores are incommensurable, so ordering is fixed group order, ranked within group.
• Universal typeahead UI — the live UniversalSearchCommandSurface gains grouped cross-entity results while typing (debounced hook with a monotonic race token, excluding the active mode's own domain). Selecting an item navigates straight to the record; each group ends with a "View all in {mode}" row reusing the existing cross-mode plumbing; Enter with nothing highlighted still runs the mode-scoped search. use-medication-catalog also now sends the auth header (owners were silently served fixture data).

Verification

• npm run verify:cheap — green (lint, typecheck, 1139 vitest, runtime check, sitemap check)
• npm run verify:ui equivalent — Chromium against a live demo-mode dev server: new ui-universal-search.spec.ts (3/3), ui-tools + ui-tools-task-directory (40/40), ui-smoke + ui-overlap (59/59); plus live curl of /api/search/universal (grouped payload, domain CSV filter, 400 on short query)
• npm run verify:release — not run (release gate; needs live keys/Deno)
• npm run format:check
• npm run eval:retrieval:quality (must stay 23/23) — could NOT run in the authoring environment (no live Supabase/OpenAI keys). Must be run before merge. Pay attention to the weak-match OR-augmentation (setting RAG_TEXT_WEAK_OR_RELAXATION=false restores relax-on-empty exactly) and the retrieval-selection tiebreak (tie-only). The classifier memo also needs eval:quality -- --rag-only to confirm unsupported_correct_rate stays 1.0.
• npm run eval:rag -- --limit 15 + npm run eval:quality -- --rag-only — answer generation itself is untouched, but the classifier memo affects routing inputs; run alongside the retrieval eval before merge.
• npm run check:production-readiness behaviour unchanged for env/deployment; search-behaviour verification state recorded in docs/process-hardening.md
• npm run check:deployment-readiness — no deployment/startup behaviour changed

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use (answer pipeline and source rendering untouched)
• No patient-identifiable document workflow was introduced or expanded (universal search queries follow the existing privacy-redaction telemetry paths; the new endpoint logs nothing)
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy) — no env/project changes
• Service-role keys and private document access remain server-only (the universal route uses the same admin-client + owner-scope ladder as the registry routes)
• Demo/synthetic content remains clearly separated (demo mode serves fixtures via the same isDemoMode() ladder and flags demoMode:true)
• Source metadata, review status, and outdated/unknown-source behaviour remain conservative — governance metadata still never weights retrieval selection ordering; candidate scores stay clamped
• TGA SaMD impact checked: no clinical decision-support output changed; registry typeahead surfaces existing records with existing governance badges

Notes

• Eval gate is the one open item: the two golden evals need live keys, so they are a pre-merge manual step per docs/process-hardening.md ("Retrieval changes must pass the golden eval before merge"). Every retrieval change here was engineered tie-only/append-only, and the riskiest one has an env kill switch, specifically to make that gate easy to satisfy or roll back.
• Follow-ups filed (items 17–22 in docs/rag-hybrid-findings-and-todo.md) include the optional Phase-5 registry-to-corpus embedding so Answer mode could eventually cite medications/services/forms — deliberately excluded here to keep the eval-gated pgvector corpus untouched.
• The ui-universal-search spec mocks the universal endpoint; an end-to-end owner-auth spec needs the E2E_USER_* Playwright project.

🤖 Generated with Claude Code

https://claude.ai/code/session_011QiyE8jMm7VnrtknHF6jnJ

---

Generated by Claude Code

---

Note

High Risk
Touches core RAG text retrieval, classifier routing memoization, and evidence selection tiebreaks—areas that previously regressed golden evals without unit-test coverage; merge still depends on live eval:retrieval:quality / RAG quality gates.

Overview
Delivers universal search end-to-end: GET /api/search/universal runs a parallel, per-domain federated search (documents via RAG + medications/services/forms/differentials/tools via shared rankers), with demo/public/owner access aligned to registry routes and isolated failures per domain. The command surface adds debounced “Across Clinical KB” typeahead (excluding the active mode’s domain), direct navigation to records, and “view all in {mode}” rows; medication catalog fetches now send auth so owners get seeded data.

Catalog layer:catalog-search + tools-catalog replace duplicated normalizers/rankers and the launcher/tools fixture split. Forms use an honest forms search kind (not documents); dashboard/nav/composer paths are updated accordingly.

RAG retrieval changes (each with tests and/or kill switches where noted): LLM query-classifier verdict memoization (15m TTL, no error caching) for stable routing; pre-clamp score emitted and used as a tiebreak when clamped scores saturate at 1.0; weak strict-AND text matches can append OR-relaxed recall behind strict hits (RAG_TEXT_WEAK_OR_RELAXATION=false restores relax-on-empty); similarity_origin: synthetic_text and telemetry for non-cosine similarities; shared weakRetrievalTopScoreThreshold for weak-search logging. Registry/medication APIs use shared fetch-with-seed helpers for universal search parity.

Docs/env: RAG_TEXT_WEAK_OR_RELAXATION, answer timeout docs corrected to 30s default; process-hardening notes eval debt before merge.

^{Reviewed by Cursor Bugbot for commit 716b596. Configure here.}

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Auth change stale typeahead
  • When authorizationHeader changes, the hook now clears cached result groups so fresh becomes false and the dropdown shows loading until the auth-scoped refetch completes.

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 716b596. Configure here.}

[cursor]

Auth change stale typeahead

Medium Severity

When authorizationHeader changes (sign-in, sign-out, or token refresh), the hook refetches but keeps the prior result state. While result.query still equals the current input, fresh stays true and loading false, so the dropdown can keep showing the previous access tier’s universal results (e.g. public fixtures) with no spinner until the debounced request finishes.

^{Reviewed by Cursor Bugbot for commit 716b596. Configure here.}