Skip to content

test(coverage): re-ratchet domain coverage floors past their 5pp drift trigger - #1964

Merged
BigSimmo merged 3 commits into
mainfrom
claude/coverage-floors-x6
Aug 14, 2026
Merged

test(coverage): re-ratchet domain coverage floors past their 5pp drift trigger#1964
BigSimmo merged 3 commits into
mainfrom
claude/coverage-floors-x6

Conversation

@BigSimmo

Copy link
Copy Markdown
Owner

Summary

  • Ledger [WIP] Update direct-push guidance for non-bypass users #192 (maturity work-order X6): raise CI-enforced coverage floors for clinical-safety/retrieval/answer surfaces. Investigation found this outcome was already substantially in place on mainvitest.config.mts already carries four per-domain coverage-threshold groups (retrieval, evidence/verification, core RAG/answer, clinical-safety/governance) plus a whole-repo floor, all enforced by the required CI coverage job (npm run test:coverage, gated in .github/workflows/ci.yml).
  • Re-measured every domain group against a fresh full npm run test:coverage run (606 files / 6549 tests passed, all current thresholds green — confirmed no false failure from the existing config before touching anything).
  • Two branch-coverage thresholds had drifted past this file's own documented 5pp re-ratchet trigger (the same rule already used for the whole-repo floor's own comment: "raise these when the gap grows past ~5pp again; never lower them"), so those two were raised:
    • Evidence/verification group (answer-verification, evidence, evidence-relevance, rag-claim-support, rag-evidence-gates, rag-quote-verification, rag-source-segmentation): branches 81 → 84 (measured 86.04%, gap was 5.04pp)
    • Core RAG/answer group (rag, rag-extractive-answer, rag-comparison, rag-answer-support): branches 72 → 76 (measured 78.22%, gap was 6.22pp)
  • Every other threshold in the file already sat within the 5pp band and is unchanged — raising a floor that already tracks its baseline closely risks a false failure from ordinary branch-count variance, not closing a real gap.
  • No file in the four domain groups showed a genuine untested critical path (lowest points: 76.4% statements on rag-quote-verification.ts, 57.7% branches on rag-answer-support.ts — well clear of a "scary" <30% bar), so no tests were added, per the ledger's own stop condition ("expand tests only where gaps are real... do not lower floors to pass").
  • Marked docs/maturity-backlog-workorders.md X6 as DONE with the full evidence trail (measured numbers, which floors moved and why, which didn't and why not).

Before/after coverage numbers

Whole-repo (unchanged, for reference): Statements 54.85% → 54.86%, Branches 51.91% → 51.92%, Functions 57.00% → 57.00%, Lines 56.26% → 56.26% (negligible run-to-run noise; floor unchanged at 52/49/54/53, gap 2-3pp, within the documented 5pp trigger).

Domain groupMetricBefore floorMeasuredAfter floorGap after
Retrieval (9 files)statements/branches/functions/lines86/78/88/9090.32/81.02/92.86/93.88unchanged4.32/3.02/4.86/3.88
Evidence/verification (7 files)statements/branches/functions/lines92/81/94/9493.89/86.04/95.24/96.4492/84/94/941.89/2.04/1.24/2.44
Core RAG/answer (4 files)statements/branches/functions/lines83/72/90/8887.55/78.22/92.57/91.8483/76/90/884.55/2.22/2.57/3.84
Clinical-safety/governance (5 files)statements/branches/functions/lines94/83/96/9796.72/85.11/98.73/98.95unchanged2.72/2.11/2.73/1.95

All numbers above were measured with fraction-level precision from the coverage HTML report (coverage/lcov-report/**), not approximated from LCOV's line-only export, since v8's "Statements" metric is not the same as its "Lines" metric.

RAG impact: no retrieval behaviour change — this PR only edits vitest.config.mts coverage-threshold numbers (which reference RAG file paths as glob strings for measurement purposes) and a maturity-tracking doc; no file under src/lib/rag/** or any other RAG-ranking-protected surface is touched, confirmed against scripts/pr-policy.mjs's ragRankingPatterns (only operationalRiskPatterns matches vitest.config.mts).

Verification

  • npm run test:coverage — before: 606 files / 6549 tests passed, all thresholds green (confirms the existing floors are already honest, no false failure). After: 606 files / 6549 tests passed, all thresholds green (confirms the two raised floors don't cause a false failure either). Whole-repo summary before/after: Statements 54.85%→54.86%, Branches 51.91%→51.92%, Functions 57%→57%, Lines 56.26%→56.26%.
  • npm run verify:pr-local — completed: check:runtime, check:installed-lock-parity, format:changed, sitemap:check, docs:check-index, docs:check-inventory, docs:check-scripts, docs:check-links, check:branch-review-ledger, check:outstanding-issues, check:ledger-write-discipline, lint, typecheck, test (606 files / 6549 tests passed), check:rag:fixtures (36 golden cases, 23 suites passed), check:medication-interactions. One unrelated pre-existing failure: check:medication-lexicon-report — confirmed via git stash that this fails identically on unmodified origin/main (docs/medication-interaction-lexicon-review.md is stale from an unrelated prior change); this PR touches no medication file and does not cause or fix that drift.
  • npm run format — no changes (both files already formatted).
  • Build/offline-RAG-contract steps were skipped by verify:pr-local's own scope classifier: no build-affecting change, no RAG-scoped change.

Risk and rollout

  • Risk: low. Coverage-threshold numbers only — a gate that already passes gets marginally tighter on two specific metrics, with real measured headroom (~2pp) preserved. No production, application, or RAG-behaviour code changed.
  • Rollback: revert this commit; the two branch floors return to 81/72 and the workorder doc reverts to its prior wording.
  • Provider or production effects: None. No deployment, Supabase, or OpenAI interaction.

Notes

  • Pre-existing, unrelated gate failure observed during verification: npm run check:medication-lexicon-report reports docs/medication-interaction-lexicon-review.md is stale on current origin/main (confirmed via git stash before/after this diff). Not fixed here — out of this PR's scope per the ledger sweep's "do not touch anything outside your own scope" instruction; flagging for a separate follow-up.
  • Also observed (not acted on, out of scope): src/lib/rag/rag-hydration.ts sits at 58.71% statements / 42.15% branches and isn't covered by any of the four domain threshold groups — worth a future look, but adding it now would have meant either padding tests to clear an arbitrary bar or setting a floor without real headroom, neither of which this ledger item's stop condition allows.

Generated by Claude Code

…t trigger (ledger #192)
Ledger #192 (maturity X6) asked for higher, CI-enforced coverage floors on the
clinical-safety/retrieval/answer surfaces. That was already substantially in
place: vitest.config.mts already carries four per-domain threshold groups plus
a whole-repo floor, all enforced by the required CI "coverage" job.
Re-measured with `npm run test:coverage` (606 files, 6549 tests passed, all
current thresholds green — no false failure). Two branch thresholds had
drifted past this file's own documented 5pp re-ratchet trigger, so those two
were raised (never lowered): evidence/verification branches 81 -> 84 (measured
86.04%, gap 5.04pp) and core RAG/answer branches 72 -> 76 (measured 78.22%,
gap 6.22pp). Every other value already sat within the 5pp band and is
unchanged. No file in the four domain groups showed a genuine untested
critical path (lowest: 76.4% statements, 57.7% branches), so no tests were
added — only the floors were re-measured and tightened where real.
Marks maturity-backlog-workorders.md X6 DONE with the evidence trail.
@supabase

supabaseBot commented Aug 14, 2026

Copy link
Copy Markdown

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.


Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

@coderabbitai

coderabbitaiBot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in:40 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 217a74f1-956f-4813-b5dd-adcb1020a03f

📥 Commits

Reviewing files that changed from the base of the PR and between 71fa97d and 6fa7532.

📒 Files selected for processing (3)
  • docs/branch-review-records/023564085ea84fa8f81b78f3ec3e137053c64dbd30e5af948b0b041bd6ba537b.record.md
  • docs/maturity-backlog-workorders.md
  • vitest.config.mts

Comment @coderabbitai help to get the list of available commands.

@BigSimmo
BigSimmo enabled auto-merge (squash) August 14, 2026 16:19
@BigSimmo
BigSimmo merged commit adc5182 into mainAug 14, 2026
24 checks passed
@BigSimmo
BigSimmo deleted the claude/coverage-floors-x6 branch August 14, 2026 16:25
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@BigSimmo@claude