diff --git a/docs/process-hardening.md b/docs/process-hardening.md index 89157972aa..1d9fb15e6c 100644 --- a/docs/process-hardening.md +++ b/docs/process-hardening.md @@ -212,7 +212,33 @@ All approved render-surface modules are extracted. `ClinicalDashboard.tsx` went - **Trap logged:** a backgrounded `tsc … | tail; echo EXIT` reports the pipeline's (echo's) exit, not tsc's — always read the captured EXIT marker, never trust the background "exit 0" notification. This masked both the true baseline (56 fast-check errors) and move 1's first-cut breakage (9 unexported functions + a missed aliased import) until vitest failed. - **Move 2 (shipped): retrieval variants H → `rag-retrieval-variants.ts`.** Lines 1999–2283 moved verbatim (empty block diff modulo the `export` keyword added to 5 declarations): `buildRetrievalQueryVariants`, alias normalize/expand + live `fetchEnabledRagAliases`, `assertGlobalSearchAllowed`/`ownerScopeForDocumentFilteredRetrieval`, `shouldRelaxWeakTextMatches`, `textCandidateBudgetForQueryClass`, budget consts. rag.ts imports the 11 names it calls, re-exports the 6 `retrieval-query-variants.test.ts` imports. Back-edges: `SearchChunksArgs` (type-only, no runtime cycle) + `shouldShortCircuitUnsupportedSearch` (one call-time value fn, gained an `export`; belongs to the soft-tail family staying in rag.ts). 4 orphaned imports removed from rag.ts. rag.ts 6,025 → 5,750 lines. Gate: tsc clean, focused vitest 102/102 (incl. `retrieval-query-variants.test.ts`), lint (7 pre-existing warnings only), prettier clean. - **Move 3 (shipped, final): cache region F + context-packing J → `rag-cache.ts`.** Cache is interleaved in rag.ts with the query-classifier, unsupported-soft-tail, and query-expansion regions, so this was a **multi-region verbatim move** (4 contiguous regions assembled into one module, each byte-identical to source; diff shows only the `export` keyword on 11 boundary-crossing declarations): cache maps/consts + cache-key helpers + main cache block (get/set answer+search, shared-cache, `stableHash`, `retrievalPlanCacheQuery`, invalidation) + context-packing. rag.ts imports the 12 names it calls (incl. the `answerInflight` map) and re-exports 4 (`invalidateRagCachesForOwner`/`…ForDocumentMutation` for API routes; `packedContextCacheKey`/`retrievalPlanCacheQuery` for tests). Back-edges: `SearchChunksArgs` + `SearchTelemetry` (type-only) and one call-time value fn `retrievalPlanForQueryClass` (gained `export`). Notably the interleaved classifier/soft-tail/expansion helpers are cleanly _not_ referenced by the cache union — only those 3 back-edges. 3 orphaned imports removed. rag.ts 5,750 → 5,108 (**7,550 → 5,108 across part 2, −2,442 / −32%**; the file now holds the orchestrator + retrieval-diagnostics + classifier + coverage-gate + memory/enrichment + logging). Gate: tsc clean, focused vitest 118/118, lint (7 pre-existing warnings only; module 0), prettier clean. -- **Golden retrieval eval environment-blocked — root cause is a data/owner mismatch, not code or OpenAI:** `npm run eval:retrieval:quality` FAILS ~36/36 with every retrieval layer at 0 (`text_candidates:0`, `hybrid_vector:0`, …). Proven not a regression: pristine `origin/main` rag.ts (moved modules inert) reproduces the **byte-identical** failure output (24/24 printed cases, latency aside). Direct Supabase probe with the configured service key: **2,065 documents / 69,334 chunks present, but `owner_id` is `null` (global) on the sampled docs while `RAG_EVAL_OWNER_ID` (2bac05f1-…) owns 0**, and owner-scoped retrieval now **fail-closes** on a null owner (privacy hardening) — so no owner value retrieves the null-owner corpus and the eval returns 0 candidates. Restoring OpenAI credits fixed embeddings (`embedding:1` succeeds) but not this. To run a real 36/36, the corpus docs must be assigned to the eval owner (or the eval must target the actual owner / allow global search). Behavior preservation for moves 1–3 is evidenced by byte-identical moves + tsc + focused vitest + **pristine-eval equivalence**; the mandated clean 36/36 + `eval:quality --rag-only` must run once the eval-owner/corpus config is fixed (or in CI), before merge. +- **Golden retrieval eval environment-blocked — root cause is a data/owner mismatch, not code or OpenAI:** `npm run eval:retrieval:quality` FAILS ~36/36 with every retrieval layer at 0 (`text_candidates:0`, `hybrid_vector:0`, …). Proven not a regression: pristine `origin/main` rag.ts (moved modules inert) reproduces the **byte-identical** failure output (24/24 printed cases, latency aside). Direct Supabase probe with the configured service key: **2,065 documents / 69,334 chunks present, but `owner_id` is `null` (global) on the sampled docs while `RAG_EVAL_OWNER_ID` (2bac05f1-…) owns 0**, and owner-scoped retrieval now **fail-closes** on a null owner (privacy hardening) — so no owner value retrieves the null-owner corpus and the eval returns 0 candidates. Restoring OpenAI credits fixed embeddings (`embedding:1` succeeds) but not this. To run a real 36/36, the corpus docs must be assigned to the eval owner (or the eval must target the actual owner / allow global search). Behavior preservation for moves 1–3 is evidenced by byte-identical moves + tsc + focused vitest + **pristine-eval equivalence**; the mandated clean 36/36 + `eval:quality --rag-only` must run once the eval-owner/corpus config is fixed (or in CI), before merge. **RESOLVED 2026-07-07** — the eval now defaults its owner to the public sentinel and the mandated clean **36/36, failed_cases=0** ran green against the live corpus (see "Golden retrieval eval owner default = public sentinel" below); this closes the #347 part-2 eval debt. + +## Golden retrieval eval owner default = public sentinel (2026-07-07) + +- **Fix committed:** the golden retrieval eval now defaults its owner to the public-owner sentinel + `00000000-0000-0000-0000-000000000000` (`DEFAULT_EVAL_OWNER_ID` in `scripts/eval-retrieval.ts`, + applied as the final fallback after env/CLI/email resolution). Since the 2026-07-06 public + promotion the live corpus is entirely `owner_id = NULL`; `retrieval_owner_matches` maps the + sentinel to NULL-owner rows, mirroring anonymous production search, so **no future session has to + set `RAG_EVAL_OWNER_ID` by hand**. An explicit `RAG_EVAL_OWNER_ID` / `LOCAL_NO_AUTH_OWNER_ID` / + `RAG_EVAL_OWNER_EMAIL` (or `--owner-id` / `--owner-email`) still overrides it — the email-lookup + path is preserved. Change is localized to eval config + docs; retrieval/ranking/app runtime code + is untouched. This is the fix the retrieval-correctness workstream (#343) established and the + runbook already recommended; it makes that recommendation the committed default. +- **Offline gates (2026-07-07):** `npm run typecheck` clean; `npm run verify:cheap` green + (`check:runtime`, `sitemap:check`, lint — 8 pre-existing warnings, 0 errors — typecheck, unit + tests 1251 passed / 3 skipped). +- **Live 36/36 CONFIRMED (2026-07-07):** ran `npm run check:supabase-project` (live mode against + `Clinical KB Database` / `sjrfecxgysukkwxsowpy`) then `npm run eval:retrieval:quality` with the + new default owner (no `RAG_EVAL_OWNER_ID` set). Result: **`cases=36`, `failed_cases=0`, + `latency_failed_cases=0`**, `document_recall@5=1`, `content_recall@5=1`, `top_k_hit_rate=1`, + `mrr@10=0.8148`, `content_mrr@10=0.9244`, strategies `{text_fast_path:25, document_lookup_fast_path:1, +hybrid:10}`, all 10 forced-embedding vector cases passed (`force_embedding_failure_count=0`). A + direct read-only DB probe confirmed the premise: all **2,065** documents have `owner_id = NULL`; + the old eval owner `2bac05f1-…` owns 0. **This closes the #347 part-2 eval debt** — the golden + retrieval eval is the behavior-preservation proof the rag.ts decomposition owed, and it now runs + green with no manual owner setup. ## Answer-thread Back button: URL and visible answer can disagree (2026-07-06) diff --git a/docs/retrieval-quality-runbook.md b/docs/retrieval-quality-runbook.md index 6081a009e6..648332aa26 100644 --- a/docs/retrieval-quality-runbook.md +++ b/docs/retrieval-quality-runbook.md @@ -48,13 +48,16 @@ The command requires the same live-eval environment as the existing RAG eval scr - Supabase server env values for `Clinical KB Database` - `OPENAI_API_KEY` -- `RAG_EVAL_OWNER_ID`, `LOCAL_NO_AUTH_OWNER_ID`, or `RAG_EVAL_OWNER_EMAIL` - -Since the 2026-07-06 public promotion the live corpus is entirely `owner_id = NULL`, so set -`RAG_EVAL_OWNER_ID=00000000-0000-0000-0000-000000000000` (the public-owner sentinel — -`retrieval_owner_matches` maps it to NULL-owner rows, mirroring anonymous production search). A -real owner UUID now scopes retrieval to zero documents and fails every case; leaving it unset -throws the owner-scope guard. +- Owner is **optional** — see below. + +Since the 2026-07-06 public promotion the live corpus is entirely `owner_id = NULL`, the eval now +**defaults its owner to the public-owner sentinel** `00000000-0000-0000-0000-000000000000` +(`DEFAULT_EVAL_OWNER_ID` in `scripts/eval-retrieval.ts`). `retrieval_owner_matches` maps that +sentinel to NULL-owner rows, mirroring anonymous production search, so no session has to set +`RAG_EVAL_OWNER_ID` by hand. An explicit `RAG_EVAL_OWNER_ID`, `LOCAL_NO_AUTH_OWNER_ID`, or +`RAG_EVAL_OWNER_EMAIL` (or `--owner-id` / `--owner-email`) still overrides the default. Note a real +owner UUID now scopes retrieval to zero documents and fails every case, so only override when the +corpus ownership actually changes. Optional cost fields: diff --git a/scripts/eval-retrieval.ts b/scripts/eval-retrieval.ts index 2a46d96f25..2e6d4de957 100644 --- a/scripts/eval-retrieval.ts +++ b/scripts/eval-retrieval.ts @@ -9,6 +9,13 @@ import { findOwnerIdByEmail, loadAdminClient, percentile, withProviderBackoff } loadEnvConfig(process.cwd()); +// Committed default eval owner. Since the 2026-07-06 public promotion the live corpus is +// entirely owner_id = NULL, so owner-scoped retrieval must run as the public-owner sentinel +// (retrieval_owner_matches maps it to NULL-owner rows, mirroring anonymous production search). +// An explicit RAG_EVAL_OWNER_ID / LOCAL_NO_AUTH_OWNER_ID / RAG_EVAL_OWNER_EMAIL (or --owner-id / +// --owner-email) still overrides this. See docs/retrieval-quality-runbook.md. +const DEFAULT_EVAL_OWNER_ID = "00000000-0000-0000-0000-000000000000"; + const contentExpectationSchema = z.union([z.string().min(1), z.array(z.string().min(1)).min(1)]); const goldenCaseSchema = z.object({ @@ -812,7 +819,10 @@ async function main() { requireServerEnv(); requireOpenAIEnv(); - const ownerId = args.ownerId ?? (args.ownerEmail ? await findOwnerIdByEmail(supabase, args.ownerEmail) : undefined); + const ownerId = + args.ownerId ?? + (args.ownerEmail ? await findOwnerIdByEmail(supabase, args.ownerEmail) : undefined) ?? + DEFAULT_EVAL_OWNER_ID; const capturedCaseClient = supabase as unknown as SupabaseEvalCaseClient; const capturedCases = await loadCapturedRagEvalCases({ supabase: capturedCaseClient, ownerId, limit: args.limit }); const allCases = [...capturedCases.map(capturedRagCaseToGoldenCase), ...loadGoldenRetrievalCases(args.fixture)];