Universal search: fix core RAG search defects and federate medications/services/forms/differentials/tools into one cross-entity search

.env.example

@@ -65,6 +65,9 @@ RAG_PROVIDER_MODE=auto
 # Optional JSON override for app-layer ranking weights (see src/lib/ranking-config.ts).
 # Omit for current defaults. Example (enable diversity demotion + linear freshness):
 # RAG_RANKING_CONFIG={"documentDiversityPenalty":0.03,"freshness":{"mode":"linear"}}
+# Append OR-relaxed recall behind weak-but-nonzero strict text matches (P8b extension).
+# Set false to restore relax-only-on-empty behaviour (golden retrieval eval kill switch).
+RAG_TEXT_WEAK_OR_RELAXATION=true
 RAG_ANSWER_CACHE_TTL_MS=300000
 RAG_ANSWER_CACHE_SIZE=100
 RAG_SEARCH_CACHE_TTL_MS=60000

docs/process-hardening.md

@@ -161,3 +161,10 @@ All approved render-surface modules are extracted. `ClinicalDashboard.tsx` went
 - **P2 M13:** `20260702000000_commit_generation_preserve_legacy_artifacts.sql` must be applied to live Supabase before reindex commits can safely purge legacy NULL-generation rows. After apply, run `npm run check:m13-migration`, `npm run reindex:health`, and `npm run check:indexing`. `search_schema_health()` now reports `commit_document_index_generation.preserve_legacy_artifacts_migration` when the live function body is stale.
 - **P2 upload hardening:** `/api/upload` consumes the `document_upload` rate-limit bucket (12/min owner, 3/min anonymous).
 - **P3 dispositioned (no code change):** L9 searchable-only `image_count` (documented in `worker/main.ts`); L11 triple `readFile` peak-memory trade-off (documented at the ingestion site); L18 duplicate `audit_logs` policy in an already-applied migration (do not edit applied migrations — consolidate only if migrations are ever squashed); L19 CSP `script-src 'unsafe-inline'` deferred (no active XSS sink today; nonce migration needs dedicated UI verification).
+
+## Universal search workstream — verification state (2026-07-06)
+
+- **Shipped on `claude/universal-search-algorithm-ryrps7`:** finding #11 interim fix (classifier-verdict memoization, 15-min TTL, errors not memoized — the "cheaper interim option" from the 2026-07-03 entry above), pre-clamp tiebreak completion in `selectRetrievalEvidence`, weak-match OR-augmentation (kill switch `RAG_TEXT_WEAK_OR_RELAXATION=false`), `similarity_origin` telemetry, shared `catalog-search` primitives replacing the four per-domain rankers, forms mode-kind honesty, tools dataset dedupe, `/api/search/universal` federated endpoint, and the cross-entity typeahead in `UniversalSearchCommandSurface`.
+- **Eval debt (blocking merge, not development):** `npm run eval:retrieval:quality` (23/23) and `eval:quality --rag-only` (`unsupported_correct_rate` 1.0) could NOT be run in the authoring environment (no live keys) — they MUST be run before merge per the standing gate above, with special attention to the weak-match OR-augmentation (flag off restores relax-on-empty exactly) and the retrieval-selection tiebreak (tie-only by construction).
+- **UI verification run:** new `tests/ui-universal-search.spec.ts` (grouped typeahead renders, item selection navigates, Enter still runs the mode search — universal endpoint mocked), full `ui-tools`/`ui-tools-task-directory` (40/40) and `ui-smoke`/`ui-overlap` suites against a live dev server in demo mode, plus a live curl of `/api/search/universal` (grouped payload, domain filter, 400 on short query).
+- **Known limitation:** the typeahead spec mocks the universal endpoint; an end-to-end spec against live seeded registries needs the owner-auth Playwright project (E2E_USER_* keys).

docs/rag-hybrid-findings-and-todo.md

@@ -231,3 +231,31 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows
 16. ⏳ **ROTATE all secrets** pasted in plaintext this session: OpenAI key, Supabase `service_role`
  JWT + legacy JWT secret, DB password, E2E password. `.env.local` is gitignored, but the values
  were exposed in chat.
+
+## Follow-ups filed 2026-07-06 (universal-search workstream)
+
+17. ⏳ **Alias promotion pipeline is blocked by privacy redaction.** `rag_query_misses` rows store
+ hashed/redacted queries with empty `candidate_aliases`, so hardcoded `synonymGroups` /
+ `domainAliasGroups` / special-case rewrites in `src/lib/clinical-search.ts` cannot be replaced
+ with data-driven `rag_aliases` rows until a privacy-safe candidate-alias capture is designed.
+18. ⏳ **`document_index_units` vector recall** — no HNSW index (dropped 2026-07-02) and hosted
+ Supabase denies `ALTER FUNCTION … SET hnsw.ef_search` for the `language sql` hybrid RPCs, so
+ only `match_document_memory_cards_hybrid` pins `ef_search=100`. Quantify the recall impact
+ before reintroducing an index.
+19. ⏳ **Demo fallback can mask live retrieval failures in non-prod.** `/api/search` and
+ `/api/answer` silently swap in demo data on Supabase errors outside production (only an
+ `X-Clinical-KB-Fallback` header signals it). Proposal: surface a warning in
+ `check:production-readiness` output and/or a visible dev-mode banner rather than changing
+ the fallback behaviour.
+20. ⏳ **Automated guard for governance-weighting regressions.** The 23/23 → 16/23 golden-set
+ regression class (governance metadata weighting selection ordering) is only guarded by the
+ manual PR checklist because `eval:retrieval:quality` needs live keys. Investigate a
+ keys-free structural test (e.g. assert selection sort inputs exclude governance fields).
+21. ⏳ **Recalibrate gates for synthetic text-only similarity (RC9 residual).** Text-fast-path
+ results now carry `similarity_origin: "synthetic_text"` telemetry; once enough data exists,
+ recalibrate `evaluateEvidenceCoverageGate` / text-fast-path thresholds against real cosine
+ distributions instead of the `least(0.95, 0.56 + text_rank*0.39)` proxy.
+22. ⏳ **Registry-to-corpus embedding (universal search Phase 5).** Medications/services/forms/
+ differentials are federated into `/api/search/universal` but are not retrieval-corpus
+ entities, so Answer mode cannot cite them. If product wants that: env-flagged ingestion,
+ golden-eval + invented-term controls first (depends on 17 for alias hygiene).

docs/search-rag-master-context.md

@@ -19,7 +19,7 @@ The desired experience is:
 
 Phase 7 performance hardening is implemented:
 
-- `OPENAI_ANSWER_TIMEOUT_MS=12000` is the answer-generation timeout budget.
+- `OPENAI_ANSWER_TIMEOUT_MS` is the answer-generation timeout budget. Phase 7 introduced it at 12000ms; the current default is **30000ms** — a deliberate product decision to favour natural, model-written answers over fast degradation to stitched extractive prose (see the rationale comment at `src/lib/env.ts` next to `OPENAI_ANSWER_TIMEOUT_MS`).
 - `src/lib/rag.ts` passes that timeout to structured answer generation so provider stalls fail into the existing source-backed fallback path faster than the global OpenAI request timeout.
 - `scripts/eval-rag.ts` excludes `generation_fallback` answers from the intentional routine-extractive latency bucket so provider timeout waits do not distort the model-free extractive metric.
 - Focused tests, typecheck, production-readiness, and capped RAG eval with threshold failure enabled passed after the change.
@@ -33,9 +33,9 @@ Phase 7b latency polish is implemented:
 
 Deployment/config note:
 
-- `.env.example` documents `OPENAI_ANSWER_TIMEOUT_MS=12000`.
-- Local `.env.local` should also include `OPENAI_ANSWER_TIMEOUT_MS=12000` for explicit local parity.
-- Hosted production/deployment environments should set `OPENAI_ANSWER_TIMEOUT_MS=12000` explicitly, or they will rely on the server default from `src/lib/env.ts`.
+- `.env.example` documents `OPENAI_ANSWER_TIMEOUT_MS=30000`, matching the server default in `src/lib/env.ts`.
+- Local `.env.local` may set it explicitly for parity; unset environments rely on the 30000ms server default.
+- The historical 12000ms value in `docs/search-rag-phase-0-baseline.md` and `docs/search-rag-master-plan.md` records the Phase 7 rollout, not current guidance.
 
 ## Skill Lenses Used
 

docs/site-map.md

@@ -23,7 +23,7 @@ This file is generated by `npm run sitemap:update`. Run `npm run sitemap:check`
 - `/?mode=answer` - Answer mode. Search kind: `answer`. Query example: `/?mode=answer&q=example+question&focus=1&run=1`.
 - `/?mode=documents` - Documents mode. Search kind: `documents`. Query example: `/documents/search?mode=documents&q=lithium+monitoring&focus=1&run=1`.
 - `/services` - Services mode. Search kind: `services`. Query example: `/services?q=13YARN&focus=1&run=1`.
-- `/forms` - Forms mode. Search kind: `documents`. Query example: `/forms?q=transport+forms&focus=1&run=1`.
+- `/forms` - Forms mode. Search kind: `forms`. Query example: `/forms?q=transport+forms&focus=1&run=1`.
 - `/favourites` - Favourites mode. Search kind: `favourites`. Query example: `/favourites?q=clozapine+set&focus=1&run=1`.
 - `/differentials` - Differentials mode. Search kind: `differentials`. Query example: `/differentials?q=acute+confusion&focus=1&run=1`.
 - `/?mode=prescribing` - Medication mode. Search kind: `documents`. Query example: `/?mode=prescribing&q=acamprosate+renal+dose&focus=1&run=1`.
@@ -570,6 +570,7 @@ This file is generated by `npm run sitemap:update`. Run `npm run sitemap:check`
 - `/api/registry/records/[slug]` - Registry record detail. Source: `src/app/api/registry/records/[slug]/route.ts`.
 - `/api/search` - Search endpoint. Source: `src/app/api/search/route.ts`.
 - `/api/search/interaction` - Search interaction telemetry. Source: `src/app/api/search/interaction/route.ts`.
+- `/api/search/universal` - Route discovered from app directory Source: `src/app/api/search/universal/route.ts`.
 - `/api/setup-status` - Setup status. Source: `src/app/api/setup-status/route.ts`.
 - `/api/upload` - Upload endpoint. Source: `src/app/api/upload/route.ts`.
 - `/auth/callback` - Route discovered from app directory Source: `src/app/auth/callback/route.ts`.

playwright.config.ts

@@ -5,7 +5,7 @@ const baseURL = getPlaywrightBaseUrl();
 
 export default defineConfig({
  testDir: "./tests",
- testMatch: /.*ui-(smoke|stress|accessibility|tools|tools-task-directory|overlap)\.spec\.ts/,
+ testMatch: /.*ui-(smoke|stress|accessibility|tools|tools-task-directory|overlap|universal-search)\.spec\.ts/,
  timeout: 60_000,
  retries: process.env.CI ? 1 : 0,
  expect: {

src/app/api/medications/route.ts

@@ -8,13 +8,13 @@
 } from "@/lib/api-rate-limit";
 import { isDemoMode, isLocalNoAuthMode } from "@/lib/env";
 import { jsonError } from "@/lib/http";
-import { defaultMedicationRecords, ensureMedicationsSeeded } from "@/lib/medication-seed";
+import { defaultMedicationRecords, fetchOwnerMedicationRowsWithSeed } from "@/lib/medication-seed";
 import {
  medicationSourceStatus,
  medicationValidationStatus,
  rowGovernance,
  rowToMedicationRecord,
  type MedicationRecordRow,
 } from "@/lib/medication-records";
 import { medicationToSearchResult, rankMedicationRecords, type MedicationSearchMatch } from "@/lib/medications";
 import { publicAccessContext, shouldResolvePublicCatalogAccess } from "@/lib/public-api-access";
@@ -107,27 +107,7 @@
  });
  }
 
- const fetchRecords = async () => {
- const { data, error } = await supabase
- .from("medication_records")
- .select("*")
- .eq("owner_id", access.ownerId)
- .order("name")
- .limit(MEDICATION_MAX_RECORDS);
- if (error) throw new Error(error.message);
- return (data ?? []) as MedicationRecordRow[];
- };
-
- let rows = await fetchRecords();
- if (rows.length === 0) {
- try {
- await ensureMedicationsSeeded(supabase, access.ownerId);
- } catch (seedError) {
- console.error(`[medications] auto-seed failed for owner ${access.ownerId}`, seedError);
- }
- rows = await fetchRecords();
- }
-
+ const rows = await fetchOwnerMedicationRowsWithSeed(supabase, access.ownerId, MEDICATION_MAX_RECORDS);
  const records = rows.map(rowToMedicationRecord);
  const governanceBySlug = Object.fromEntries(rows.map((row) => [row.slug, rowGovernance(row)]));
 

src/app/api/registry/records/route.ts

@@ -15,9 +15,9 @@
  rowGovernance,
  rowToServiceRecord,
  type RegistryRecordKind,
  type RegistryRecordRow,
 } from "@/lib/registry-records";
-import { ensureRegistrySeeded } from "@/lib/registry-seed";
+import { fetchOwnerRegistryRowsWithSeed } from "@/lib/registry-seed";
 import { rankServiceRecords, serviceRecords, type ServiceRecord, type ServiceSearchMatch } from "@/lib/services";
 import { createAdminClient } from "@/lib/supabase/admin";
 import { AuthenticationError, unauthorizedResponse } from "@/lib/supabase/auth";
@@ -109,33 +109,7 @@
  });
  }
 
- const fetchRecords = async () => {
- const { data, error } = await supabase
- .from("clinical_registry_records")
- .select("*")
- .eq("owner_id", access.ownerId)
- .eq("kind", kind)
- .order("title")
- .limit(REGISTRY_MAX_RECORDS);
- if (error) throw new Error(error.message);
- return (data ?? []) as RegistryRecordRow[];
- };
-
- let rows = await fetchRecords();
- if (rows.length === 0) {
- // First visit for this owner: lazily seed the curated defaults so new
- // accounts get populated Services/Forms instead of the empty state. Only
- // the seed write is best-effort (a failure falls back to the empty set);
- // the re-read stays outside the try so a genuine read failure still
- // surfaces as an error rather than a misleading empty registry.
- try {
- await ensureRegistrySeeded(supabase, access.ownerId, kind);
- } catch (seedError) {
- console.error(`[registry] auto-seed failed for owner ${access.ownerId} (${kind})`, seedError);
- }
- rows = await fetchRecords();
- }
-
+ const rows = await fetchOwnerRegistryRowsWithSeed(supabase, access.ownerId, kind, REGISTRY_MAX_RECORDS);
  const records = rows.map(rowToServiceRecord);
  const governanceBySlug = Object.fromEntries(rows.map((row) => [row.slug, rowGovernance(row)]));
 

src/app/api/search/route.ts

@@ -9,6 +9,7 @@ import { fetchRelatedDocuments, toDocumentMatch } from "@/lib/document-enrichmen
 import { jsonError, PublicApiError } from "@/lib/http";
 import { isClinicalImageEvidence } from "@/lib/image-filtering";
 import { searchChunksWithTelemetry } from "@/lib/rag";
+import { weakRetrievalTopScoreThreshold } from "@/lib/rag-routing";
 import { classifyRagQuery, normalizedClinicalSearchTokens } from "@/lib/clinical-search";
 import { buildSmartRagApiPlan } from "@/lib/smart-rag-api";
 import { SOURCE_ONLY_EMBEDDING_SKIP_REASON } from "@/lib/rag-provider";
@@ -439,7 +440,7 @@ function logWeakSearch(args: {
  args.results.length === 0 ||
  args.relevance.verdict === "none" ||
  args.relevance.verdict === "nearby" ||
- topScore < 0.48;
+ topScore < weakRetrievalTopScoreThreshold;
  if (!weak) return;
  const promotions = candidatePromotions(args.query, args.results);
  void args.supabase
@@ -549,7 +550,7 @@ function logRetrievalDiagnostics(args: {
  args.results.length === 0 ||
  args.relevance.verdict === "none" ||
  args.relevance.verdict === "nearby" ||
- topScore < 0.48;
+ topScore < weakRetrievalTopScoreThreshold;
  const latencyMs = telemetryLatencyMs(args.telemetry);
 
  await args.supabase.from("rag_retrieval_logs").insert({

src/app/api/search/universal/route.ts

@@ -0,0 +1,90 @@
+import { NextResponse } from "next/server";
+import { z } from "zod";
+
+import {
+ allowRateLimitInMemoryFallbackOnUnavailable,
+ consumeSubjectApiRateLimit,
+ rateLimitJsonResponse,
+} from "@/lib/api-rate-limit";
+import { isDemoMode, isLocalNoAuthMode } from "@/lib/env";
+import { jsonError } from "@/lib/http";
+import { publicAccessContext, shouldResolvePublicCatalogAccess } from "@/lib/public-api-access";
+import { createAdminClient } from "@/lib/supabase/admin";
+import { AuthenticationError, unauthorizedResponse } from "@/lib/supabase/auth";
+import { runUniversalSearch, universalSearchDomains, type UniversalSearchDomain } from "@/lib/universal-search";
+import { parseRequestQuery, queryInteger } from "@/lib/validation/query";
+
+export const runtime = "nodejs";
+
+// Typeahead-friendly GET: cross-entity federated search over documents + the registry
+// catalogues. Access ladder mirrors /api/registry/records — demo/local serves fixtures,
+// unauthenticated public serves the public catalogues, owners get their seeded records.
+const universalSearchQuerySchema = z.object({
+ q: z.string().trim().min(2).max(200),
+ limit: queryInteger({ fallback: 5, min: 1, max: 10 }),
+ domains: z
+ .string()
+ .trim()
+ .optional()
+ .transform((value) => {
+ if (!value) return undefined;
+ const requested = value
+ .split(",")
+ .map((domain) => domain.trim())
+ .filter((domain): domain is UniversalSearchDomain => (universalSearchDomains as string[]).includes(domain));
+ return requested.length ? requested : undefined;
+ }),
+});
+
+function universalResponse(payload: Record<string, unknown>) {
+ return NextResponse.json(payload, { headers: { "Cache-Control": "private, no-store" } });
+}
+
+export async function GET(request: Request) {
+ try {
+ const { q, limit, domains } = parseRequestQuery(request, universalSearchQuerySchema, "Invalid universal query.");
+
+ if (isDemoMode() || isLocalNoAuthMode()) {
+ const payload = await runUniversalSearch({ query: q, limitPerDomain: limit, domains, demo: true });
+ return universalResponse({ ...payload, demoMode: true });
+ }
+
+ if (!shouldResolvePublicCatalogAccess(request)) {
+ const payload = await runUniversalSearch({ query: q, limitPerDomain: limit, domains, demo: true });
+ return universalResponse({ ...payload, publicAccess: true });
+ }
+
+ const supabase = createAdminClient();
+ const access = await publicAccessContext(request, supabase);
+
+ const rateLimit = await consumeSubjectApiRateLimit({
+ supabase,
+ subject: access.rateLimitSubject,
+ bucket: "registry",
+ allowInMemoryFallbackOnUnavailable: allowRateLimitInMemoryFallbackOnUnavailable(),
+ });
+ if (rateLimit.limited) {
+ return rateLimitJsonResponse("Universal search requests are rate limited. Try again shortly.", rateLimit);
+ }
+
+ if (!access.ownerId) {
+ const payload = await runUniversalSearch({ query: q, limitPerDomain: limit, domains, demo: true });
+ return universalResponse({ ...payload, publicAccess: true });
+ }
+
+ const payload = await runUniversalSearch({
+ query: q,
+ limitPerDomain: limit,
+ domains,
+ supabase,
+ ownerId: access.ownerId,
+ demo: false,
+ });
+ return universalResponse(payload);
+ } catch (error) {
+ if (error instanceof AuthenticationError) {
+ return unauthorizedResponse();
+ }
+ return jsonError(error);
+ }
+}

src/components/ClinicalDashboard.tsx

@@ -1123,7 +1123,10 @@
  };
  }
  return {
- statusLabel: modeSearch.resultKind === "documents" ? modeSearch.statusLabel : "No answer yet",
+ statusLabel:
+ modeSearch.resultKind === "documents" || modeSearch.resultKind === "forms"
+ ? modeSearch.statusLabel
+ : "No answer yet",
  statusTone: "empty",
  nextStep: modeSearch.nextStep,
  badgeLabel: modeSearch.badgeLabel,
@@ -2445,6 +2448,7 @@
  const shouldRun =
  params.get("run") === "1" ||
  modeSearch.kind === "documents" ||
+ modeSearch.kind === "forms" ||
  modeSearch.kind === "favourites" ||
  modeSearch.kind === "differentials";
  if (!shouldRun) return;
@@ -2453,7 +2457,7 @@
  urlDocumentSearchBootstrappedRef.current = true;
  void executeSearch(searchText, mode, scopeFilters);
  // URL search intentionally runs once when the selected mode can execute.
  }, [canRunSearch, answerThreadBootstrapped]);
 
  useEffect(() => {
  const updateHash = () => {
@@ -2712,7 +2716,7 @@
  setActionNotice({ tone: "success", message: "Favourites filtered from the composer." });
  return;
  }
- if (modeSearch.kind === "services" || targetMode === "forms") {
+ if (modeSearch.kind === "services" || modeSearch.kind === "forms") {
  resetAnswerThread();
  setAnswer(null);
  setSources([]);