feat(rag): record the structured generation-quality verdict on fallback (#231 phase 1)

[BigSimmo]

Summary

• Instruments the structured generation-quality failure behind ledger #231 without changing retrieval, ranking, route budgets, retry policy, quality thresholds, caching, or the conservative source-only fallback.
• Adds src/lib/rag/rag-generation-quality-diagnostics.ts, including a typed GenerationQualityError with the failure stage, a provider-safe gate reason, and prose-free answer-shape metadata.
• Propagates the structured verdict into answer_retry_reasons, fallback rag_queries.metadata, and eval diagnostics while keeping the existing user-visible degraded reason and error message stable.
• Adds offline reproductions for numeric_faithfulness_gap and provider_source_gap, plus focused unit coverage of the diagnostics module.
• Adds the provider-safe scripts/probe-generation-quality.ts operator probe and documents the diagnostic path in docs/rag-behaviour/behaviour-map.md and docs/scripts-index.md.

Conflict-resolution note

Current main was merged at a5f03ccc3390803a2b9de1d91701cf3a9d0455a8 in commit f20b898f4024cf61281c76b132f27be105378f06. The canonical docs/branch-review-ledger.md and docs/outstanding-issues.md changes were deliberately not replayed over current main. The branch's old #231 text contained stale PR-state claims that described draft PR #1861 as superseded and awaiting manual closure. The post-resolution diff therefore contains eight files and keeps the current repository ledger authoritative.

RAG impact: no retrieval behaviour change. This is additive structured diagnostics around existing generation-quality gates. Retrieval, ranking, selection, ordering, thresholds, retry behaviour, caching, and fallback construction are unchanged.

Verification

• Prior exact implementation: npm run verify:pr-local completed all selected phases with failed: (none), including lint, typecheck, full unit suite, build, client-bundle secret checks, and offline RAG validation.
• Prior focused coverage: tests/rag-answer-fallback.test.ts and tests/rag-generation-quality-diagnostics.test.ts passed 92/92 tests.
• Fresh conflict check: GitHub reports head f20b898f4024cf61281c76b132f27be105378f06 mergeable against current main.
• Fresh hosted checks: SAST and Secret Scan passed. CI is running on the conflict-resolution head.
• Provider-backed retrieval and answer evaluations were not run. The change does not alter retrieval or answer synthesis, and live provider dispatch requires separate approval.

Risk and rollout

• Risk: Low. The retained code is additive diagnostics on the generation-failure path. The conservative source-only fallback remains the terminal behaviour if diagnostics extraction or generation fails.
• Rollback: revert this PR. No schema, migration, configuration, cache dependency, or provider-side mutation was introduced.
• Provider or production effects: none during this conflict-resolution sweep.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use
• No patient-identifiable document workflow was introduced or expanded
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo and synthetic content remain separated from real clinical sources
• Source metadata, review status, and unknown-source behaviour remain conservative
• Deployment classification and TGA SaMD impact were checked

Notes

This remains phase 1 of the residual #231 investigation. It makes the binding quality gate observable and reproducible offline. It does not justify a timeout increase, gate weakening, additional provider attempts, or caching a generation fallback.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in:17 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f8e01bbb-f2c3-4688-9921-0d1e61d8c231

📥 Commits

Reviewing files that changed from the base of the PR and between a6bc420 and 5216965.

📒 Files selected for processing (9)

• docs/rag-behaviour/behaviour-map.md
• docs/scripts-index.md
• scripts/probe-generation-quality.ts
• src/lib/rag/rag-answer-instructions.ts
• src/lib/rag/rag-eval-diagnostics.ts
• src/lib/rag/rag-generation-quality-diagnostics.ts
• src/lib/rag/rag.ts
• tests/rag-answer-fallback.test.ts
• tests/rag-generation-quality-diagnostics.test.ts

---

_{Comment @coderabbitai help to get the list of available commands.}

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:90658711b8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

CI triage

CI failed on this PR. Automated classification of the 2 failed job(s):

• Static PR checks — needs investigation: inspect the failing step and uploaded diagnostics; rerun only after classifying the cause.
• PR required — needs investigation: inspect the failing step and uploaded diagnostics; rerun only after classifying the cause.

Compared with main CI run #10430 (cancelled).

Classification is evidence routing, not permission to ignore a failure. Exact quarantined Playwright identities remain governed by the flake ledger.

[BigSimmo]

@copilot Fix the unresolved P2 on exact head a3bc4daa880063f65bb63b1fd76c56751fa9174c and push one focused commit to claude/lithium-generation-quality-debug-ji1vce. Do not merge, close, rebase, force-push, deploy, alter auto-merge, or run live/provider-backed evaluations.

The new diagnostics currently inspect only the terminal error in the outer generation catch. When an initial strong answer fails generatedAnswerQualityFailureReason, the code appends only strong_quality_retry; if the repair attempt then truncates, fails parsing, or throws a provider error, generationQualityFailureDiagnostics(error) sees only that terminal non-quality error and the original quality gate disappears from fallback metadata/eval diagnostics.

Required fix:

1. Capture the first GenerationQualityError diagnostics before entering the existing quality repair/retry and preserve it through the terminal fallback path.
2. Emit exactly one generation_quality_gate:<gateReason> entry in answerRetryReasons and populate the fallback generation_quality_gate_reason, stage and provider-safe answer shape from the preserved verdict when the later retry fails. Do not change retry admission, provider attempts, timeout, degraded token, cache behaviour, quality thresholds or fallback construction.
3. Add a focused offline regression using a real gate-failing first response followed by a truncated or parse-failing repair response; assert the original gate reason/stage/shape survives into fallback log/eval metadata and the conservative source-only fallback/degraded token remain unchanged.
4. Run the focused RAG diagnostics/fallback tests, typecheck and offline RAG fixtures only. Report the pushed 40-character SHA and exact outcomes.

[BigSimmo]

@copilot The P2 repair is verified on current head 423e7ee7151b469dc1cb07c227aea9531093d738. Now merge main at 2ae5a0aa5d339a7dc9089db134c2d9d0220444ae into this branch once, resolve conflicts minimally, commit and push to claude/lithium-generation-quality-debug-ji1vce. Never rebase or force-push. Do not merge/close the PR, deploy, alter auto-merge, run live/provider-backed evaluations, or broaden behaviour.

Resolution rules:

• Preserve current-main RAG routing, budgets, retries, caching, safety gates and every newer rag.ts change.
• Layer only this PR’s typed generation-quality diagnostics and the verified preservation of the first quality verdict through a failed/truncated repair. Do not change retry admission or fallback/degraded tokens.
• Keep current-main docs/outstanding-issues.md and immutable branch-review history authoritative; reapply only still-current fix(ci): bump checkout/setup-node/cache actions to v5 #231 diagnostics wording without deleting or duplicating newer rows. Regenerate docs/scripts-index.md if the probe remains.
• Preserve the focused offline regressions, including the gate-failing first answer followed by a truncated repair.

Run focused diagnostics/fallback tests, offline RAG fixtures, typecheck, format, ledger/docs checks and the smallest PR-local gate. Report the pushed 40-character SHA and exact outcomes.