fix(eval): default golden retrieval eval owner to public sentinel - #348
Merged
Conversation
Since the 2026-07-06 public promotion the live corpus is entirely owner_id = NULL, and owner-scoped retrieval fail-closes on a null owner. The previously-configured RAG_EVAL_OWNER_ID owns 0 documents, so `npm run eval:retrieval:quality` returned 0/36. Default DEFAULT_EVAL_OWNER_ID to the public-owner sentinel 00000000-0000-0000-0000-000000000000 as the final fallback in scripts/eval-retrieval.ts, so no session has to set RAG_EVAL_OWNER_ID by hand. Explicit RAG_EVAL_OWNER_ID / LOCAL_NO_AUTH_OWNER_ID / RAG_EVAL_OWNER_EMAIL (and --owner-id / --owner-email) still override it; the email-lookup path is preserved. Change is localized to eval config + docs; retrieval, ranking, and app runtime code are untouched. Updates the retrieval-quality runbook and records the change in process-hardening.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdP9DwsGdvtEvyVh3juYZ5
This pull request has been ignored for the connected project Preview Branches by Supabase. |
… debt Ran check:supabase-project (live mode) then eval:retrieval:quality with the new public-sentinel default owner (no RAG_EVAL_OWNER_ID set): cases=36, failed_cases=0, latency_failed_cases=0, document_recall@5=1, content_recall@5=1, top_k_hit_rate=1. A direct read-only probe confirmed all 2065 documents are owner_id=NULL. This settles the PR #347 part-2 eval debt. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdP9DwsGdvtEvyVh3juYZ5
BigSimmo
marked this pull request as ready for review
July 7, 2026 15:59
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
BigSimmo
enabled auto-merge
July 7, 2026 15:59
Uh oh!
There was an error while loading. Please reload this page.
BigSimmo pushed a commit
that referenced
this pull request
Jul 8, 2026
…scripts Hoist DEFAULT_EVAL_OWNER_ID + resolveEvalOwnerId() into scripts/eval-utils.ts and apply it at the final owner-resolution point in eval-retrieval, eval-quality, eval-rag, eval-answer-quality, and eval-search. Previously only eval-retrieval (PR #348) defaulted to the public-owner sentinel, so the answer-path evals still returned 0/N against the all-public (owner_id=NULL) live corpus unless RAG_EVAL_OWNER_ID was set by hand. The helper preserves precedence (explicit --owner-id / RAG_EVAL_OWNER_ID / LOCAL_NO_AUTH_OWNER_ID, then --owner-email / RAG_EVAL_OWNER_EMAIL, then the sentinel) and emits a one-line warning on the sentinel fallback so the narrowing to public-only scope is visible, not silent. Scope labels in eval-rag/eval-search now key off the explicit input so a sentinel run still reads "public". Read/eval scripts only — write/backfill scripts are intentionally untouched. Adds first-ever resolveEvalOwnerId unit coverage to tests/eval-utils.test.ts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdP9DwsGdvtEvyVh3juYZ5
BigSimmo pushed a commit
that referenced
this pull request
Jul 8, 2026
…h gate Adds the 2026-07-08 follow-up: DEFAULT_EVAL_OWNER_ID + resolveEvalOwnerId() are now shared across all read/eval scripts (item 1), with a one-line sentinel-fallback warning (item 3). Records the live eval:quality --rag-only result closing #347's answer-path gate (item 2): cases=44, unsupported_correct=1.0, citation_failure=0.0455, numeric_grounding=0.0227 (all identical to #343 baseline); grounded_supported=0.90, with the 5 failing cases all in #343's documented latency/route live-variance set. Corrects the earlier overstatement so #348 reads as closing the retrieval half only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdP9DwsGdvtEvyVh3juYZ5
14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
00000000-0000-0000-0000-000000000000(DEFAULT_EVAL_OWNER_IDinscripts/eval-retrieval.ts), applied as the final fallback after env/CLI/email resolution — so no future session has to setRAG_EVAL_OWNER_IDby hand.owner_id = NULL, and owner-scoped retrieval fail-closes on a null owner (privacy hardening). The oldRAG_EVAL_OWNER_ID(2bac05f1-…) now owns 0 documents, soeval:retrieval:qualityreturned 0/36 with every retrieval layer empty.retrieval_owner_matchesmaps the public sentinel to NULL-owner rows, mirroring anonymous production search — this is the fix the retrieval-correctness workstream (Retrieval correctness: repair #325 golden regression, corpus-grounded finding #11 fix, RC9 confidence hardening #343) established and the runbook already recommended.RAG_EVAL_OWNER_ID/LOCAL_NO_AUTH_OWNER_ID/RAG_EVAL_OWNER_EMAIL(and--owner-id/--owner-email) still overrides the default; the email-lookup path is preserved.docs/retrieval-quality-runbook.mdand records the change + confirming eval result indocs/process-hardening.md.Verification
npm run verify:cheap— green (check:runtime,sitemap:check, lint 0 errors / 8 pre-existing warnings, typecheck, unit tests 1251 passed / 3 skipped)npm run verify:ui— N/A (no UI, routing, styling, or browser behavior changed)npm run verify:release— not run (no release/handoff claim)npm run format:check— Prettier clean on changed filesnpm run eval:retrieval:quality— 36/36,failed_cases=0✅ (ran live againstClinical KB Database/sjrfecxgysukkwxsowpywith the new default owner, noRAG_EVAL_OWNER_IDset). This also settles the refactor(rag): decompose rag.ts part 2 (extractive-answer, retrieval-variants, cache) — move-only #347 part-2 eval debt (the behavior-preservation proof the rag.ts decomposition owed). Summary:Also confirmed live via
npm run check:supabase-project(live mode) and a read-only DB probe: all 2,065 documents haveowner_id = NULL; the old eval owner owns 0.npm run eval:rag/eval:quality --rag-only— N/A (answer generation unchanged by this PR)npm run check:production-readiness— N/A (no clinical/privacy/env/Supabase/governance runtime change)npm run check:deployment-readiness— N/AClinical Governance Preflight
This change touches only the eval harness's owner configuration, not ingestion, answer generation, search/ranking runtime, source rendering, document access, privacy, production env, or clinical output. The safe invariants below are unaffected:
Clinical KB Database(sjrfecxgysukkwxsowpy) — no Supabase config touched, reads only.env.local, never committed)Notes
PUBLIC_OWNER_FILTER_SENTINELinsrc/lib/owner-scope.ts, theretrieval_owner_matchesSQL predicate,scripts/check-retrieval-owner-migration.ts). This PR does not import from runtime code — it defines a localDEFAULT_EVAL_OWNER_IDconstant in the eval script to keep the change minimal and confined to eval config.🤖 Generated with Claude Code
https://claude.ai/code/session_01BdP9DwsGdvtEvyVh3juYZ5