Skip to content

Harden RAG retrieval and answer reliability - #1288

Closed
BigSimmo wants to merge 3 commits into
mainfrom
codex/rag-reliability-51e6fd
Closed

Harden RAG retrieval and answer reliability#1288
BigSimmo wants to merge 3 commits into
mainfrom
codex/rag-reliability-51e6fd

Conversation

@BigSimmo

Copy link
Copy Markdown
Owner

Summary

  • Harden RAG evidence binding, clinical-value verification, comparison fallback, extractive formatting, citation preservation, and fail-closed behavior across current clinical fixtures.
  • Resolve #019 admission/discharge source loss and #029 review-fallback stubs, with a zero-tolerance quality gate for recurrence.
  • Improve eval diagnostics, irrelevant-at-10 evidence, source-governance reporting, and path-specific coverage floors.
  • Add the auditable BMJ third-party attestation policy, schema/API support, migration, and pending-human-review manifest without treating BMJ content as clinically verified.
  • Reconcile the RAG backlog and evidence documents; lithium is closed, while ADHD/metabolic evidence debt, browser-matrix evidence, and human label review remain open.

RAG impact: behaviour change — canary pair output/rag-retrieval-post-final.json -> exact current-head post canary pending before merge

The protected retrieval change uses the expanded chunk query only for the measured clozapine blood-count action shape. Discharge and other red-section document lookups retain raw intent terms. Ranking scores, comparator order, clamps, and semantic reranking are unchanged.

Verification

  • npm run verify:cheap — PASS: 25 gates, including the full unit suite.
  • npm run test:coverage — PASS: 399 files; 4,062 passed, 2 skipped; RAG coverage 86.83% statements and 90.79% lines.
  • npm run eval:rag:offline — PASS: 36 cases, 21 suites, 560 assertions.
  • npm run check:production-readiness — PASS: 8 checks, no blockers; two isolated-worktree environment-location warnings.
  • npm run check:migration-role — PASS.
  • Final focused post-review Vitest set — PASS: 5 files, 83 tests.
  • Focused source-review lint — PASS.
  • npm run verify:pr-local — merge gate. It passed on the pre-upstream-merge implementation tree; exact reconciled-head rerun is waiting for the repository's current exclusive browser lease and install alignment.
  • npm run eval:retrieval:quality — merge gate. Prior 36-case live evidence passed with document/content recall 1.0 and zero per-case reciprocal-rank regressions; exact current-head canary remains pending.
  • Recorded 44-case live answer gate — 30/30 supported substantive and grounded, 14/14 unsupported correct, zero citation/numeric failures, zero review fallbacks, zero route-ceiling failures, p95 7,494 ms.
  • npm run eval:rag -- --limit 15 + npm run eval:quality -- --rag-only — superseded by the recorded full 44-case answer gate above; no weaker sample is claimed.
  • npm run verify:ui — UI verification not run: no UI, routing, styling, or browser-rendering surface changed.
  • npm run verify:release — not run: it includes additional provider-backed release work; exact-head PR-local, the protected retrieval canary, and hosted required checks are the scoped merge gates.
  • npm run check:production-readiness — result recorded above.
  • npm run check:deployment-readiness — not run: deployment startup and hosting configuration were unchanged.

Earlier npm run typecheck passed before the final narrowed chunk-query patch; exact-head typechecking is included in the pending verify:pr-local rerun.

Risk and rollout

  • Risk: High clinical/reliability surface. Main risks are incorrect evidence selection, misplaced numeric values, lost citations, overconfident extraction, and schema/API version skew. New fixtures and fail-closed gates cover these paths.
  • Rollback: Revert the application changes. The Supabase migration is currently unapplied, so no database rollback is required. If applied later, leave the additive schema in place and use a separately reviewed forward migration rather than dropping data.
  • Provider or production effects: Authorized live evals and read-only hosted RPC profiling were performed. No hosted migration, source attestation, or production-data write was performed. Merging to main will follow the repository's normal application/worker auto-deploy path.

Migration 20260727010000_bmj_third_party_source_attestation.sql is a separate, deliberate operator step and must not be applied automatically. Until applied, legacy owner-scoped review behavior remains available and v2-only public/attestation requests fail closed with 503 source_review_v2_unavailable.

Clinical Governance Preflight

  • Source-backed claims still require linked source verification before clinical use
  • No patient-identifiable document workflow was introduced or expanded without explicit governance approval
  • Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
  • Service-role keys and private document access remain server-only
  • Demo/synthetic content remains clearly separated from real clinical sources
  • Source metadata, review status, and outdated/unknown-source behavior remain conservative
  • Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

Notes

  • Current reconciled head: 33a69d4dbf7818e896c42bf58f1967a1652c2530.
  • BMJ sources remain clinical_validation_status=unverified; qualified human review, hosted apply, attestation, and warning-rate remeasurement remain #022.
  • Firefox/WebKit evidence and human irrelevant-at-10 disposition remain #023.
  • ADHD corpus/table accessibility and metabolic schedule evidence remain the open parts of #018.

@gitguardian

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian idGitGuardian statusSecretCommitFilename
35223466TriggeredBearer Token235974dtests/profile-retrieval-rpcs.test.tsView secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@coderabbitai

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7d18ee81-83a9-46fc-b3c3-e04c944482a6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/rag-reliability-51e6fd

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@supabase

supabaseBot commented Jul 27, 2026

Copy link
Copy Markdown

Updates to Preview Branch (codex/rag-reliability-51e6fd) ↗︎

DeploymentsStatusUpdated
DatabaseMon, 27 Jul 2026 23:07:49 UTC
ServicesMon, 27 Jul 2026 23:07:49 UTC
APIsMon, 27 Jul 2026 23:07:49 UTC

Tasks are run on every commit but only new migration files are pushed.
Close and reopen this PR if you want to apply changes from existing seed or migration files.

TasksStatusUpdated
ConfigurationsMon, 27 Jul 2026 23:07:59 UTC
MigrationsMon, 27 Jul 2026 23:09:26 UTC
SeedingMon, 27 Jul 2026 23:09:30 UTC
Edge FunctionsMon, 27 Jul 2026 23:09:33 UTC

View logs for this Workflow Run ↗︎.
Learn more about Supabase for Git ↗︎.

@BigSimmo

Copy link
Copy Markdown
OwnerAuthor

Superseded by #1289. The replacement preserves the reviewed final tree in a clean one-commit branch so the token-shaped redaction fixture is absent from every PR commit; no force-push or branch deletion was used.

@BigSimmo
BigSimmo deleted the codex/rag-reliability-51e6fd branch July 30, 2026 09:10
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@BigSimmo