Skip to content

fix(eval): default golden retrieval eval owner to public sentinel - #348

Merged
BigSimmo merged 2 commits into
mainfrom
claude/fix-eval-owner-default-m7zoww
Jul 7, 2026
Merged

fix(eval): default golden retrieval eval owner to public sentinel#348
BigSimmo merged 2 commits into
mainfrom
claude/fix-eval-owner-default-m7zoww

Conversation

@BigSimmo

@BigSimmoBigSimmo commented Jul 7, 2026

Copy link
Copy Markdown
Owner

Summary

  • Default the golden retrieval eval's owner to the public-owner sentinel 00000000-0000-0000-0000-000000000000 (DEFAULT_EVAL_OWNER_ID in scripts/eval-retrieval.ts), applied as the final fallback after env/CLI/email resolution — so no future session has to set RAG_EVAL_OWNER_ID by hand.
  • Rationale: since the 2026-07-06 public promotion the live corpus is entirely owner_id = NULL, and owner-scoped retrieval fail-closes on a null owner (privacy hardening). The old RAG_EVAL_OWNER_ID (2bac05f1-…) now owns 0 documents, so eval:retrieval:quality returned 0/36 with every retrieval layer empty. retrieval_owner_matches maps the public sentinel to NULL-owner rows, mirroring anonymous production search — this is the fix the retrieval-correctness workstream (Retrieval correctness: repair #325 golden regression, corpus-grounded finding #11 fix, RC9 confidence hardening #343) established and the runbook already recommended.
  • An explicit RAG_EVAL_OWNER_ID / LOCAL_NO_AUTH_OWNER_ID / RAG_EVAL_OWNER_EMAIL (and --owner-id / --owner-email) still overrides the default; the email-lookup path is preserved.
  • Scope is localized to eval config + docs. Retrieval, ranking, selection, and app runtime code are untouched. Also updates docs/retrieval-quality-runbook.md and records the change + confirming eval result in docs/process-hardening.md.

Verification

  • npm run verify:cheap — green (check:runtime, sitemap:check, lint 0 errors / 8 pre-existing warnings, typecheck, unit tests 1251 passed / 3 skipped)
  • npm run verify:ui — N/A (no UI, routing, styling, or browser behavior changed)
  • npm run verify:release — not run (no release/handoff claim)
  • npm run format:check — Prettier clean on changed files
  • npm run eval:retrieval:quality — 36/36, failed_cases=0 ✅ (ran live against Clinical KB Database / sjrfecxgysukkwxsowpy with the new default owner, no RAG_EVAL_OWNER_ID set). This also settles the refactor(rag): decompose rag.ts part 2 (extractive-answer, retrieval-variants, cache) — move-only #347 part-2 eval debt (the behavior-preservation proof the rag.ts decomposition owed). Summary:
Golden retrieval eval summary:
cases=36
document_recall@5=1
content_recall@5=1
top_k_hit_rate=1
mrr@10=0.8148
content_mrr@10=0.9244 (over 36 content-term case(s))
median_latency_ms=3898
p90_latency_ms=19088
retrieval_strategy_counts={"text_fast_path":25,"document_lookup_fast_path":1,"hybrid":10}
force_embedding_case_count=10
force_embedding_failure_count=0
failed_cases=0
latency_failed_cases=0

Also confirmed live via npm run check:supabase-project (live mode) and a read-only DB probe: all 2,065 documents have owner_id = NULL; the old eval owner owns 0.

  • npm run eval:rag / eval:quality --rag-only — N/A (answer generation unchanged by this PR)
  • npm run check:production-readiness — N/A (no clinical/privacy/env/Supabase/governance runtime change)
  • npm run check:deployment-readiness — N/A

Clinical Governance Preflight

This change touches only the eval harness's owner configuration, not ingestion, answer generation, search/ranking runtime, source rendering, document access, privacy, production env, or clinical output. The safe invariants below are unaffected:

  • Source-backed claims still require linked source verification before clinical use (unchanged)
  • No patient-identifiable document workflow introduced or expanded
  • Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy) — no Supabase config touched, reads only
  • Service-role keys and private document access remain server-only (unchanged; secrets used for the eval were kept in gitignored .env.local, never committed)
  • Demo/synthetic content remains clearly separated from real clinical sources (unchanged)
  • Source metadata, review status, and outdated/unknown-source behavior remain conservative (unchanged)
  • Deployment classification/TGA SaMD impact — no clinical decision-support behavior changed

Notes

  • The public-owner sentinel already exists across the codebase (PUBLIC_OWNER_FILTER_SENTINEL in src/lib/owner-scope.ts, the retrieval_owner_matches SQL predicate, scripts/check-retrieval-owner-migration.ts). This PR does not import from runtime code — it defines a local DEFAULT_EVAL_OWNER_ID constant in the eval script to keep the change minimal and confined to eval config.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BdP9DwsGdvtEvyVh3juYZ5

Since the 2026-07-06 public promotion the live corpus is entirely
owner_id = NULL, and owner-scoped retrieval fail-closes on a null owner.
The previously-configured RAG_EVAL_OWNER_ID owns 0 documents, so
`npm run eval:retrieval:quality` returned 0/36.
Default DEFAULT_EVAL_OWNER_ID to the public-owner sentinel
00000000-0000-0000-0000-000000000000 as the final fallback in
scripts/eval-retrieval.ts, so no session has to set RAG_EVAL_OWNER_ID by
hand. Explicit RAG_EVAL_OWNER_ID / LOCAL_NO_AUTH_OWNER_ID /
RAG_EVAL_OWNER_EMAIL (and --owner-id / --owner-email) still override it;
the email-lookup path is preserved.
Change is localized to eval config + docs; retrieval, ranking, and app
runtime code are untouched. Updates the retrieval-quality runbook and
records the change in process-hardening.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdP9DwsGdvtEvyVh3juYZ5
@supabase

supabaseBot commented Jul 7, 2026

Copy link
Copy Markdown

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.


Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

… debt
Ran check:supabase-project (live mode) then eval:retrieval:quality with the
new public-sentinel default owner (no RAG_EVAL_OWNER_ID set): cases=36,
failed_cases=0, latency_failed_cases=0, document_recall@5=1, content_recall@5=1,
top_k_hit_rate=1. A direct read-only probe confirmed all 2065 documents are
owner_id=NULL. This settles the PR #347 part-2 eval debt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdP9DwsGdvtEvyVh3juYZ5
@BigSimmo
BigSimmo marked this pull request as ready for review July 7, 2026 15:59
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@BigSimmo
BigSimmo enabled auto-merge July 7, 2026 15:59
@BigSimmo
BigSimmo merged commit c356487 into mainJul 7, 2026
5 checks passed
BigSimmo pushed a commit that referenced this pull request Jul 8, 2026
…scripts
Hoist DEFAULT_EVAL_OWNER_ID + resolveEvalOwnerId() into scripts/eval-utils.ts
and apply it at the final owner-resolution point in eval-retrieval, eval-quality,
eval-rag, eval-answer-quality, and eval-search. Previously only eval-retrieval
(PR #348) defaulted to the public-owner sentinel, so the answer-path evals still
returned 0/N against the all-public (owner_id=NULL) live corpus unless
RAG_EVAL_OWNER_ID was set by hand.
The helper preserves precedence (explicit --owner-id / RAG_EVAL_OWNER_ID /
LOCAL_NO_AUTH_OWNER_ID, then --owner-email / RAG_EVAL_OWNER_EMAIL, then the
sentinel) and emits a one-line warning on the sentinel fallback so the narrowing
to public-only scope is visible, not silent. Scope labels in eval-rag/eval-search
now key off the explicit input so a sentinel run still reads "public".
Read/eval scripts only — write/backfill scripts are intentionally untouched.
Adds first-ever resolveEvalOwnerId unit coverage to tests/eval-utils.test.ts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdP9DwsGdvtEvyVh3juYZ5
BigSimmo pushed a commit that referenced this pull request Jul 8, 2026
…h gate
Adds the 2026-07-08 follow-up: DEFAULT_EVAL_OWNER_ID + resolveEvalOwnerId() are
now shared across all read/eval scripts (item 1), with a one-line sentinel-fallback
warning (item 3). Records the live eval:quality --rag-only result closing #347's
answer-path gate (item 2): cases=44, unsupported_correct=1.0, citation_failure=0.0455,
numeric_grounding=0.0227 (all identical to #343 baseline); grounded_supported=0.90,
with the 5 failing cases all in #343's documented latency/route live-variance set.
Corrects the earlier overstatement so #348 reads as closing the retrieval half only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdP9DwsGdvtEvyVh3juYZ5
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@BigSimmo@claude