test(rag): pin Supabase round-trip budgets on the answer path

[BigSimmo]

Summary

Ledger #098 — the enabler it sequences before #099 and #101. Tests and harness registration only; no production code changes.

Every finding in the 2026-07-28 latency audit was argued from reading code, and the fixes in PR #1377 were verified the same way. Nothing pinned the resulting round-trip counts, so a later refactor could add one to a hot path and no gate would notice. #098's stated outcome is "per-scenario Supabase round-trip counts are pinned by a test, so an extra round trip on a hot path is a red gate rather than an inference" — this delivers that.

• tests/helpers/supabase-round-trip-counter.ts — wraps whatever Supabase stub a suite already builds and records every .rpc(name) and .from(table). Builder methods (.select, .eq, .order, …) are deliberately not counted: they are fluent and issue nothing, so counting them would measure query style rather than network cost and would churn whenever someone added a filter. One chain is one trip. The wrapper only observes — it delegates every call to the original unmodified.
• tests/rag-round-trip-budget.test.ts — drives answerQuestionWithScope through the counted client on the offline source-only path and pins the counts.
• scripts/rag-offline-contract.mjs + scripts/fixtures/rag-offline-contract-tests.json — registers the suite so it runs as a standing guard rather than an orphan test.

The measurement

Measured, not chosen. The offline source-only answer path issues 14 round trips:

from:rag_aliases 1
rpc:match_document_chunks_text_v2 3
rpc:get_related_document_metadata_v2 2
from:document_index_quality 1
from:document_memory_cards 1
from:document_images 3
rpc:match_document_table_facts_text_v2 3

The budget does not endorse that count. Three calls each to the text-chunk, table-fact and image lookups for a single query is the repeated-triple shape #101 already records — hydration repeated across branches while rag.ts:2751 already parallelises three RPCs elsewhere. So this gives #101 a number to improve against instead of an inference. If someone reduces it, the test fails and they lower the budget in the same commit; that is the intended workflow, and the file header says so.

Second scenario guards shape, not magnitude: five sources must cost the same 14 trips as one. A per-source round trip is precisely the regression a single-source budget cannot see, and it is measured to hold today.

Non-vacuity — checked, not assumed

This session produced two guards that passed while proving nothing, so each budget here first asserts the scenario actually ran:

• the answer came back grounded, and
• at least one round trip was issued, and
• the text-retrieval RPC specifically ran.

Without those, a path that silently stopped running would pass as "within budget" — the exact failure mode. Confirmed live: with the constant at 0 the assertion failed at expected 14 to be +0 and printed the full breakdown, so the comparison is real and its failure message is actionable.

A third test pins the mechanism, because a budget guard is only worth having if an added round trip moves it: one extra .from() must increment the count, and wrapping must not mutate the caller's stub (spread, not mutate — otherwise two scenarios in one file share counting state). Proving that on the real path would mean editing src/lib/rag/** to inject a call, which is not worth doing to a protected surface for a mechanism this simple.

RAG impact: no retrieval behaviour change — this adds a test helper and a test, and appends one suite to the offline contract list. No scoring, ordering, selection, alias, citation or RPC logic is touched, and no file under src/lib/rag/** changes. The counter delegates every call to the original client unmodified, so the suites' existing stub behaviour is identical with or without it.

Verification

• npm run verify:cheap — exit 0, Test Files 436 passed (436), Tests 4572 passed | 4 skipped (4576).
• node scripts/eval-rag-offline.mjs — Test Files 22 passed (22), Tests 570 passed (570), Offline RAG fixture and production-contract checks passed. (21 → 22 suites, confirming the registration took effect.)
• validateOfflineContractTests against the fixture — contract list and fixture agree (22 suites), so the required list and the JSON cannot drift.
• npm run typecheck, npx prettier --write then clean.
• Budgets confirmed non-vacuous by observing a real failure at 0, plus the dedicated mechanism test.

Not run, and why: no browser or build gate applies to a test-only change. check:drift not run — supabase/** untouched. Every provider-backed gate (eval:*, verify:release, check:supabase-project, test:live) not run and not implicated: #098 is explicitly scoped "no providers, no DB", and this runs entirely against the suite's existing offline stub.

Risk and rollout

• Risk: Low. Test-only, and additive to the offline suite. Its failure mode is a red gate, never shipped behaviour.
• The one ongoing cost worth naming: 14 is a measured constant, so any intended change to the answer path's traffic will fail this test until the budget is updated. That is the point, but it means the number must be maintained deliberately rather than reflexively relaxed — stated in the file header.
• Rollback: revert the single commit; the offline suite returns to 21 files.
• Provider or production effects: none.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use
• No patient-identifiable document workflow was introduced or expanded without explicit governance approval
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative
• Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

Notes on the above: this PR adds tests and registers one suite, so every item is satisfied by construction rather than by inspection of new behaviour. No answer generation, citation, verification, source-governance or document-access logic changes; the fixture source is the existing synthetic clozapine record already used by tests/rag-offline-answer.test.ts, clearly separated from real clinical content. No credential or Supabase value appears in the diff and no live client is constructed — the pinned target is referenced only as unchanged context. Clinical decision-support behaviour is unaffected, so the SaMD classification is unchanged.

Notes

• Branch restarted from origin/main after docs(issues): re-apply #102's canary-gated correction lost to a merge #1440 merged, keeping the same name, per the merged-PR rule.
• #098's row named the correct files thanks to its own earlier correction: an earlier version pointed at test-cache-path.mjs and check-rag-fixtures.mjs, neither of which exercises a RAG request, so a harness built on them "would have counted nothing". Building on rag-offline-contract instead is what makes these counts real.
• Follow-on value: #101 can now be attempted against a measured baseline, and #099's remaining round trips have a place to be pinned as they are removed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01361jh3eYVjJCzXWjAhdZiF

---

Generated by Claude Code

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in:40 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b0e93755-b9c6-4b5a-afb5-318fa239e4ca

📥 Commits

Reviewing files that changed from the base of the PR and between c5c1a86 and 4646dfd.

📒 Files selected for processing (4)

• scripts/fixtures/rag-offline-contract-tests.json
• scripts/rag-offline-contract.mjs
• tests/helpers/supabase-round-trip-counter.ts
• tests/rag-round-trip-budget.test.ts

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:d40f95b8fe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".