fix(rag): don't rescue bare cross-reference answers in source-backed recovery

[BigSimmo]

Summary

Fixes a latent RAG grounding false-positive: the source-backed recovery gate was rescuing a bare cross-reference extract — "Refer to the RKPG Guidelines to Writing for Clinical Policy for further information about Scope of Practice." — to grounded=true purely on the cited sources' structured-chunk signals, even though the answer text has zero query overlap and answers nothing itself. This surfaced as the nightly Eval Canary's flapping discharge-documentation swing case (grounds locally, source-only in CI); the same boilerplate also affected duress-procedure.

Fix (src/lib/rag-extractive-answer.ts): new isBareCrossReferenceAnswer guard. shouldPreserveSourceBackedGeneratedAnswer no longer preserves a "refer to <document> … for further information" pointer. The guard is deliberately narrow — the lead sentence must be both a directive (refer to/see/consult/as per…) and a "for further information"-style redirect — so it leaves untouched the terse paraphrases the gate legitimately rescues (e.g. "Depot antipsychotic follow-up is covered by the cited local pathway.", which shares no query tokens yet answers the query). It evaluates the same sanitized text the quality gate judged, so a stripped leading-noise fragment can't hide the redirect lead. A blanket "minimum token overlap" rule was intentionally avoided because it would regress those valid paraphrase rescues.

Eval (src/lib/rag-eval-cases.ts): the full quality run showed duress-procedure is the same false-positive (identical RKPG extract), so it correctly degrades to source-only. Extended the existing discharge acceptSourceOnly treatment to the two duress cases (duress-procedure + quality-duress-pathway), guarded by expectedHit && citations>0 so a real retrieval regression still fails.

Verification

Offline (this branch):

• npm run typecheck — clean
• npm run lint (eslint --max-warnings 0) — clean
• npm run format:check (prettier) — clean
• Vitest: new tests/source-backed-recovery-cross-reference.test.ts (10 cases: pure predicate + control/treatment through finalizeRagAnswerQuality) + tests/rag-eval-cases.test.ts duress assertion + tests/rag-answer-fallback.test.ts (existing recovery regression guard) all green

Provider-side (run against the equivalent change; retrieval is unaffected by an answer-post-processing change, so results carry over):

• npm run eval:retrieval:quality → 36/36 PASS, content_mrr@10=0.9255 (baseline unchanged)
• npm run eval:quality -- --rag-only --fail-on-threshold → PASS, grounded_supported_rate 0.9667 ≥ 0.9, Failed cases: 0

A diagnostic run confirmed the guard fires only on the RKPG boilerplate and correctly spares other off-topic-but-not-cross-reference extracts.

• Offline gates above (verify:cheap equivalents run individually)
• npm run eval:retrieval:quality (36/36) — answer-post-processing change; ran anyway for safety
• npm run eval:quality -- --rag-only — grounded-supported 0.9667, citation-failure 0

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use — change makes grounding more conservative (off-topic pointer now degrades to source-only)
• No patient-identifiable document workflow was introduced or expanded
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy) — unchanged
• Service-role keys and private document access remain server-only — untouched
• Demo/synthetic content remains clearly separated from real clinical sources — untouched
• Source metadata, review status, and outdated/unknown-source behavior remain conservative — unchanged; eval acceptSourceOnly still requires expected docs to be cited
• Deployment classification/TGA SaMD impact checked — no decision-support behavior change; a non-answering redirect is replaced with a source-only "review the cited documents" response

Notes

• Pure follow-on to the already-merged acceptSourceOnly machinery; no schema, migration, env, or dependency changes.

🤖 Generated with Claude Code

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 23760b37-a08c-4d10-b3d5-e0db5d3e6e2b

📥 Commits

Reviewing files that changed from the base of the PR and between bd41d4f and abb648e.

📒 Files selected for processing (2)

• src/lib/rag-extractive-answer.ts
• tests/source-backed-recovery-cross-reference.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• src/lib/rag-extractive-answer.ts

---

📝 Walkthrough

Summary by CodeRabbit

• Bug Fixes

  • Prevented bare document redirects from being treated as valid clinical answers during source-backed recovery.
  • Preserved valid source-backed paraphrased answers while rejecting off-topic cross-reference responses.
  • Updated duress evaluation scenarios to accept appropriately grounded source-only responses.

• Tests

  • Added coverage for cross-reference detection, recovery behavior, and duress documentation evaluation.
  • Verified expected citations continue to reference the duress documentation.

Walkthrough

The change updates duress evaluation cases to allow source-only answers and adds detection that prevents bare cross-reference responses from being accepted by the source-backed recovery gate. Tests cover fixture configuration and recovery behavior.

Changes

RAG quality evaluation

Layer / File(s)	Summary
Duress source-only evaluation src/lib/rag-eval-cases.ts, tests/rag-eval-cases.test.ts	Duress cases now accept source-only results, remove keyword assertions where applicable, and verify the expected MHSP.Duress.pdf document.
Cross-reference recovery guard src/lib/rag-extractive-answer.ts, tests/source-backed-recovery-cross-reference.test.ts	Adds bare cross-reference detection, excludes those answers from source-backed recovery, and verifies that legitimate source-backed paraphrases remain recoverable.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
participant RagAnswer
participant finalizeRagAnswerQuality
participant shouldPreserveSourceBackedGeneratedAnswer
RagAnswer->>finalizeRagAnswerQuality: provide cleaned answer and source signals
finalizeRagAnswerQuality->>shouldPreserveSourceBackedGeneratedAnswer: evaluate recovery eligibility
shouldPreserveSourceBackedGeneratedAnswer->>finalizeRagAnswerQuality: reject bare cross-reference or preserve paraphrase

Loading

---

Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

• Ignore

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
Verification Claims	❌ Error	The PR summary says "Verification passed for typecheck, lint, formatting..." before the command-specific bullets, so it still contains a vague verification claim.	Rewrite that summary into explicit command/result lines (e.g. Ran npm run typecheck: passed) or remove the blanket verification sentence.

✅ Passed checks (10 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the main change: blocking bare cross-reference answers from source-backed recovery.
Description check	✅ Passed	The description covers Summary, Verification, Clinical Governance Preflight, and Notes with relevant details.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Generated And Sensitive Files	✅ Passed	The commit only adds/changes .ts source and test files; scans found no secrets, env files, logs, caches, or machine-local artifacts.
Risky Git Or Deployment Actions	✅ Passed	PASS: The PR only touches RAG source/test files; repo guidance forbids force-push/reset/cleanup and deploy runbooks use explicit PAUSE gates.
Supabase Project And Schema Safety	✅ Passed	Only answer-postprocessing code and tests changed; no migrations, RLS/policy, env-example, or Supabase project-ref files were modified, and the live ref is unchanged.
Runtime And Package Manager Integrity	✅ Passed	PR only touches a source file and a test; no package.json/lockfile/.npmrc/runtime edits or install-flag changes were introduced.
Api Route Failure Handling	✅ Passed	PASS: only pure RAG helper/eval fixture changes; the new guard and acceptSourceOnly logic are deterministic and test-covered, with no new uncaught failure path.

✨ Finishing Touches

📝 Generate docstrings

• ✅ Generated successfully - (🔄 Check to regenerate)
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/rag-cross-reference-guard

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch claude/rag-cross-reference-guard

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:74d2226a01

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:bd41d4f12c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:abb648e0bd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[coderabbitai]

Note

Docstrings generation - SUCCESS
Generated docstrings for this pull request at #539