feat(rag): B1 telemetry gap assessment + B2 offline adversarial harness (packet S5)

[BigSimmo]

Summary

• Packet S5 (B1): telemetry gap assessment. The four dashboard questions in docs/rag-improvement/README.md §B1 (stage timeout rate, fallback rate, candidate-count distribution, p50/p95 stage latency) were mapped against what PR feat(rag): record the structured generation-quality verdict on fallback (#231 phase 1) #1899 and earlier instrumentation already persist to rag_queries.metadata. Three are fully answerable today; the single proven gap is verification_latency_ms, which was measured into latencyTimings (rag.tsfinalizeAnswer) but dropped at the persistence boundary. It now ships behind the new typed flag RAG_TELEMETRY_EXTENDED (src/lib/env.ts, default false) through an allow-listed numeric projection in the new module src/lib/rag/rag-answer-telemetry-metadata.ts — only allow-listed keys, only finite numbers, so free text can never pass. rag.ts shrinks by 3 lines (the three tail latency fields extract into the module; 4359/4362 against the maintainability budget), with byte-identical metadata keys when the flag is off.
• Canary-absence tests (the data-flow register's named B1 deliverable).tests/rag-telemetry-canary-absence.test.ts imports the real canaryRegistry from the S4 fixtures and proves contaminated timings objects (canary tokens as values under allow-listed keys, as extra keys, and as keys themselves) never reach the emitted object, flag on or off, and that summarizeGenerationQualityAnswerShape (PR feat(rag): record the structured generation-quality verdict on fallback (#231 phase 1) #1899) stays canary-free.
• Packet S5 (B2): offline adversarial regression harness as the packet-mandated plain-Vitest substitute (no Promptfoo, no dependency change). New command eval:rag:adversarial:offline runs scripts/check-rag-adversarial-fixtures.mjs first (fail-closed on missing/invalid fixture, proven: removing the fixture exits 1 before any test) and then tests/rag-adversarial-harness.test.ts, which drives all 24 S4 cases through answerQuestionWithScope with a mocked Supabase client and provider module. Every case asserts: zero network attempts (fetch stubbed to throw), a measured per-case Supabase round-trip ceiling of 14 (breach fails with the breakdown), the fixture's behaviour/citation/forbidden-substring contract via deterministic assertion helpers (tests/helpers/rag-adversarial-assertions.ts, unit-tested in tests/rag-adversarial-assertions.test.ts), and canary absence across the entire captured rag_queries insert row.
• Three known divergences pinned, not hidden. The first execution of the S4 cases against the real pipeline found three cases where current behaviour differs from the authored normative expectation (cite-mismatched-attribution: the offline document-match listing cites every retrieved document; scope-other-owner-document and scope-guessed-chunk-id: review fallbacks cite in-scope evidence / echo query text instead of refusing). The tenancy invariant held in all three (no cross-scope read, no identifier echo). They are pinned in a self-expiring KNOWN_DIVERGENCES register: the harness asserts the normative contract still fails and the observed behaviour stays exactly as pinned, so movement in either direction goes red. Two provider-failure expectations were recalibrated in the fixture itself with in-file rationale (provider-malformed-payload → error_contained, matching the observed safe containment; provider-refusal-prose-as-answer → the genuinely retrieved chunk joins the citation allow-list, since citing real evidence on failure is this repository's designed conservative shape). The fixture validator stays green.
• Routing to RAG-surface PRs only.scripts/ci-change-scope.mjs gains the adversarial validator/contract/harness paths in ragEvalPatterns plus four new self-test cases; .github/workflows/ci.yml adds one step to the existing required safety job gated on rag_eval_changed == 'true'; scripts/verify-pr-local.mjs selects the command for the same scope (pinned in its plan self-test). Non-RAG PRs never run the harness.
• Phoenix decision record closed as deferred (README §B1, dated block): existing instrumentation traces without content capture, the assessment found no unanswerable dashboard question, and a second tracing vendor would add a Gate A data-processing route before proving value.
• Answer-gate denominator reconciled to 44/44.baseline-record.md §3 flagged the 44-vs-45 discrepancy and named S5 as the place to resolve it. Canary run 32025082010's own report (job 95372702769) prints Answer Metrics | Cases | 44, Failing Answer Cases — None, and exactly 44 diagnostics rows; HANDOVER's "45/45" was a transcription error and is corrected.
• HANDOVER §2 status rows updated (S5 opened; S4 and fix(supabase): make GUC-set migration hosted-safe; neutralize obsolete N+1 migration #212 T4 corrected to merged, both verified by content on origin/main; S1 evidence corrected to 44/44); docs/rag-improvement/data-flow-register.md sink row and known-gap Bump vitest from 4.1.6 to 4.1.7 #4 updated; docs/scripts-index.md and generated inventory refreshed.

RAG impact: no retrieval behaviour change — offline harness and allow-listed telemetry only. The rag.ts diff extracts three metadata fields into a pure projection module with identical flag-off output; no retrieval, ranking, routing, gating, or prompt logic is touched, and the fixture edits change test expectations only. No canary pair required.

Verification

• npm run verify:pr-local — heavy offline scope selected (script + RAG-surface changes fail closed). Decisive closing summary, verbatim: completed: check:runtime, check:installed-lock-parity, format:changed, check:npm-ci-dry-run, sitemap:check, docs:check-index, docs:check-inventory, docs:check-scripts, docs:check-links, check:branch-review-ledger, check:outstanding-issues, check:ledger-write-discipline, check:github-actions, check:ci-scope, check:gitleaks-pinned, check:ci-triage, check:pr-policy, check:gate-manifest, check:skills, check:pr-mergeability, check:verification-plan, lint, typecheck, test, build, eval:rag:offline, eval:rag:adversarial:offline, check:medication-interactions, check:medication-lexicon-report — failed: (none) — not reached: (none).
• npm run eval:rag:adversarial:offline — decisive lines: Test Files 1 passed (1) / Tests 25 passed (25) / Offline adversarial fixture validation and regression harness passed. Fail-closed proof: with the fixture file temporarily removed the command exits 1 before any test runs.
• npm run check:rag:adversarial-fixtures — Adversarial fixture contract passed (24 synthetic cases, 8 categories, 6 canaries). after the two expectation recalibrations; assertion-helper suite Tests 13 passed (13).
• node scripts/check-maintainability-budgets.mjs — src/lib/rag/rag.ts: 4359/4362 lines.
• node scripts/ci-change-scope.mjs --self-test — CI change scope self-test passed. (4 new adversarial routing cases); node scripts/verify-pr-local.mjs --self-test — PR-local verification plan self-test passed.
• npm run check:production-readiness — run in this credential-free environment; the two FAIL items are the documented provider capability gaps (Missing server environment variables: NEXT_PUBLIC_SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY and Missing OPENAI_API_KEY), not regressions from this diff — no provider credentials exist in this session by design.
• UI verification not run: no UI, routing, styling, or browser-facing change (server-side telemetry, eval scripts, tests, docs only).
• npm run eval:retrieval:quality not run: provider-backed and not required — no retrieval, ranking, selection, chunking, or scoring behaviour changed.

Risk and rollout

• Risk: low. The only production-code change is the extraction of three rag_queries.metadata latency fields into a pure module plus one flag-gated additional numeric field; flag-off output is byte-identical and pinned by test. Everything else is eval scripts, tests, CI routing, and docs. The main residual risk is harness brittleness on future RAG changes, which is the point of the gate — and it runs only on RAG-surface PRs.
• Rollback: RAG_TELEMETRY_EXTENDED=false is the whole-surface rollback for the B1 field (it is also the default, so no action is needed unless the flag was enabled); the harness and routing revert with the commit.
• Provider or production effects: None. All new gates are offline and network-rejecting; no provider-backed command was run in this session.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use
• No patient-identifiable document workflow was introduced or expanded without explicit governance approval
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative
• Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

Notes

• The harness test file also runs inside the full unit suite (repo convention, same as S4's fixture test); the dedicated eval:rag:adversarial:offline step is the routed RAG-surface gate with the fail-closed validator front-end.
• The three KNOWN_DIVERGENCES entries are candidate /issues follow-ups (review-fallback citation/echo behaviour on scope-shaped and attribution-shaped requests); each register entry self-expires when pipeline behaviour reaches the fixture's normative expectation.

Proposed HANDOVER edits for S6/S7 (handoff — last section by design)

For the coordinator to apply (or the next session to carry into docs/rag-improvement/HANDOVER.md):

1. S6 (B3 Docling lab) is unblocked: precondition "S4 merged (shared report-key convention)" is satisfied, and S5's harness is now a second consumer of the reportKey six-field convention — S6's benchmark reports should stamp the identical field set and order (pinned by tests/rag-adversarial-fixtures.test.ts). Suggested packet note: reuse scripts/rag-adversarial-contract.mjs's reportKeyFields export rather than restating the list.
2. S6 prompt addition: the S5 harness's fail-closed triple (missing fixture / network attempt / budget breach) is a working template for B3's sandbox contract; scripts/eval-rag-adversarial-offline.mjs shows the validator-then-runner shape.
3. S7+ stays owner-gated, unchanged — but add to the B5 (Ragas) row that the offline adversarial report from this PR is the natural comparison input ("retained only if it flags failures the deterministic gates miss"), and to the B2 follow-up space that a Promptfoo experiment, if ever approved, is its own PR with the package/lockfile change isolated (already the packet rule).
4. New follow-up candidates for the status table / /issues: (a) the three pinned adversarial divergences; (b) enabling RAG_TELEMETRY_EXTENDED=true in production is an owner decision once a dashboard consumer exists — until then the flag stays false.

---

Generated by Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in:19 minutes

Limit details: You’ve used all 1 included review currently available under your plan. You completed 94 included PR reviews in the past 7 days; at that activity level, included reviews refill at 1 review per hour.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a7d0677a-1301-470d-a43e-b8e609c10a46

📥 Commits

Reviewing files that changed from the base of the PR and between ec87a74 and 5e24200.

📒 Files selected for processing (20)

• .env.example
• .github/workflows/ci.yml
• docs/branch-review-records/b976a45d26170b11c35eb497940d68bd4d2ba8679fd0d427732e1b1d73032bf9.record.md
• docs/rag-improvement/HANDOVER.md
• docs/rag-improvement/README.md
• docs/rag-improvement/baseline-record.md
• docs/rag-improvement/data-flow-register.md
• docs/scripts-index.md
• package.json
• scripts/ci-change-scope.mjs
• scripts/eval-rag-adversarial-offline.mjs
• scripts/fixtures/rag-adversarial-cases.v1.json
• scripts/verify-pr-local.mjs
• src/lib/env.ts
• src/lib/rag/rag-answer-telemetry-metadata.ts
• src/lib/rag/rag.ts
• tests/helpers/rag-adversarial-assertions.ts
• tests/rag-adversarial-assertions.test.ts
• tests/rag-adversarial-harness.test.ts
• tests/rag-telemetry-canary-absence.test.ts

---

_{Comment @coderabbitai help to get the list of available commands.}