fix(rag-eval): split the readability metric into fragmentation and a v19-derived length bound

[BigSimmo]

Summary

• Split the readability answer-quality metric into two independently scored checks.
  scoreAnswerQualityEvalCase in src/lib/rag/rag-eval-cases.ts scored readability as a single
  boolean — wordCount >= 5 && wordCount <= 220 && !fragmentPattern.test(text) — over
  answerTextForQuality, which sums the answer field plus every section heading and body, and
  reported one conflated reason, "fragmented or too long". Packet S2 (feat(rag): intent-conditioned related-information menu and moderate answer length (packet S2 / A2 + A3) #2097, dda4956ff, prompt
  clinical-rag-answer-v19) raised the answer field to 60–110 words and sections to three-to-six,
  so a correctly shaped v19 answer can exceed 220 words. At that point the metric could no longer
  separate intended length from the fragmentation regression it exists to catch. Fragmentation and
  length are now evaluated independently and each failure reports its own reason.
• Fragmentation is unchanged.fragmentPattern and its effect on the score are identical; only
  its reporting is now separate.
• Bumped eval_config_version from rag-eval-config-v1 to rag-eval-config-v2 in
  scripts/fixtures/rag-adversarial-baseline.v1.json and docs/rag-improvement/baseline-record.md,
  because this is a gate-semantic change: a readability rate recorded under v1 is not comparable to
  one recorded under v2. The answer_quality gate's priorRun note and baseline-record.md §4,
  which both recorded the 220-word confound as an artefact to adjudicate, now record it as resolved
  (a new §4a carries the derivation).
• Set the packet S3 row in docs/rag-improvement/HANDOVER.md §2 to
  Merged 2026-08-18 (squash 511d22f4d), landed by content; Track A complete. Verified against git
  and the GitHub PR record: PR feat(answer): compose follow-up chips from the S2 menu, gate them on evidence (packet S3 / A4) #2108 is MERGED with merge commit
  511d22f4d59ea6838bae2983d5c3d27137d042bb, already an ancestor of this branch's base. This is the
  only HANDOVER edit in this PR.

RAG impact: no retrieval behaviour change — evaluation scorer only; gate semantics change is carried by the eval_config_version bump

Derivation of the 900-word length ceiling

The ceiling is derived from S2's own stated targets and the enforced response schema, not raised by
judgement:

Input	Value	Source
Answer field upper target	110 words	src/lib/rag/rag-answer-instructions.ts ("about 60-110 words")
Maximum sections	6	answerSections.maxItems in src/lib/rag/rag.ts (matches the prompt's "three to six")
Maximum chars per section	648	heading maxLength 48 + body maxLength 600, both in src/lib/rag/rag.ts
Characters per word	5	Deliberately low, so the conversion overstates the word ceiling and the bound cannot fail a well-formed answer

110 + 6 × (648 / 5) = 887.6, rounded up to 900 words. The >= 5 word floor for empty or stub
answers is unchanged.

This is a contract ceiling, not a style ceiling. Conciseness is enforced by the prompt itself and
measured by scoreAnswerTargeting; this bound only asks whether the text could have come from a
schema-conformant v19 generation at all. Above 900 words it could not, so the check still catches
runaway duplication and the deterministic composition paths (rag-extractive-answer.ts,
rag-comparison.ts) that build a RagAnswer in code without passing through the JSON schema.

Why split rather than simply raise the ceiling

Raising 220 to 900 alone would have removed the false positive but left the actual defect in place:
one boolean and one reason string, so a length failure and a fragmentation failure would still be
indistinguishable in the report. The metric exists to catch fragmentation, and a length-driven zero
that cannot be told apart from a fragmentation-driven zero silently degrades that. The split fixes
the diagnostic and the new ceiling fixes the false positive — both were needed, and the split is the
part with lasting value.

One constraint worth recording

The two checks share the single readability metric key rather than becoming a sixth metric.
AnswerQualityMetric is a closed union consumed by scripts/eval-answer-quality.ts as a total
Record<AnswerQualityMetric, number>, and two existing tests pin the exact five-key set. A sixth key
would therefore break that aggregation and those pins while adding no evaluative power — the two
checks are already scored and reported independently. scripts/eval-answer-quality.ts emits only
metric and score per case, never reason, so the report shape is unchanged; only the score
semantics move, which is exactly what the eval_config_version bump records.

Verification

• npm run verify:pr-local

Heavy scope selected (executable source changed): runtime, installed-lock parity, format, docs and
workflow contracts, ledger checks, lint, typecheck, the full unit suite, and the RAG offline gates.

• npx vitest run tests/rag-eval-cases.test.ts — Test Files 1 passed (1) / Tests 32 passed (32)
  (27 pre-existing plus 5 new).
• npm run eval:rag:offline — Test Files 26 passed (26) / Tests 623 passed (623);
  Offline RAG fixture and manifest validation passed (36 golden cases, 26 suites).
• npm run check:rag:fixtures —
  Offline RAG fixture and manifest validation passed (36 golden cases, 26 suites).
• npm run eval:rag:adversarial:offline —
  Adversarial fixture contract passed (24 synthetic cases, 8 categories, 6 canaries).,
  Test Files 1 passed (1) / Tests 25 passed (25),
  Offline adversarial fixture validation and regression harness passed. The three
  KNOWN_DIVERGENCES pins (cite-mismatched-attribution, scope-other-owner-document,
  scope-guessed-chunk-id) remain pinned and untouched.

The new tests were mutation-checked rather than trusted for being green. Restoring the old 220-word
ceiling fails scores a long but clean v19-shaped answer as readable with
AssertionError: expected +0 to be 1, and forcing fragmentedText = false fails both fragmentation
tests. Each new test therefore detects the specific regression it claims to.

Note on the 623 count: tests/rag-eval-cases.test.ts is not one of the 26 offline contract suites
listed in scripts/fixtures/rag-offline-contract-tests.json, which is why eval:rag:offline reports
the same 623 as the S2 baseline record rather than 628. The five new tests are covered by the full
unit suite inside verify:pr-local and by the focused run above — this is suite membership, not a
silent skip.

UI verification not run: no UI, routing, styling, browser, reduced-motion, or forced-colors behaviour
changed.

• npm run verify:ui when UI, routing, styling, browser behavior, reduced-motion, or forced-colors behavior changed
• npm run verify:release before release or handoff confidence claims

Verification not run — provider-backed and not authorised for this task: npm run eval:retrieval:quality,
npm run eval:rag, npm run eval:quality, npm run eval:answer-quality, npm run verify:release,
and npm run check:supabase-project. No retrieval, ranking, selection, or generation behaviour
changed, so this diff requires no live eval-canary pair.

• npm run eval:retrieval:quality (must stay 36/36) when retrieval, ranking, selection, chunking, or scoring behavior changed — CI cannot run it (needs live keys), so run it locally and paste the summary. A metadata/governance-weighting change once buried correct docs (recall 1.0→0.76) and only this eval caught it.
• npm run eval:rag -- --limit 15 + npm run eval:quality -- --rag-only when answer generation, the synthesis prompt, or answer post-processing changed (grounded-supported must not drop; citation-failure 0)
• npm run check:production-readiness when clinical workflow, privacy, environment, Supabase, source governance, or deployment behavior changed
• npm run check:deployment-readiness when deployment startup, hosting, or rollout behavior changed

Risk and rollout

• Risk: Low, and confined to evaluation reporting. The change touches only
  scoreAnswerQualityEvalCase's readability branch, which is consumed exclusively by
  scripts/eval-answer-quality.ts — a provider-backed evaluation script, not the runtime answer
  path. No retrieval, ranking, selection, generation prompt, ragAnswerPromptVersion, or blocking
  eval-canary threshold was touched. The residual risk is evaluative: the 900-word ceiling is
  deliberately permissive, so a bloated but schema-conformant answer will now score readable where
  the old 220 ceiling would have flagged it. That ceiling was already unreliable after S2 because it
  flagged correct answers too, and conciseness remains measured by scoreAnswerTargeting.
• Rollback: git revert this PR. It restores the previous single-boolean scorer and the
  rag-eval-config-v1 key together; no data migration, cache invalidation, or canary is involved.
• Provider or production effects: None. No provider-backed command was run, no Supabase or OpenAI
  access occurred, and no canary was dispatched.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use
• No patient-identifiable document workflow was introduced or expanded without explicit governance approval
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative
• Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

Every item above holds unchanged. This diff alters how an offline evaluation scorer reports one
metric; it does not touch citation requirements, document access, privacy scoping, Supabase
configuration, credential handling, demo-corpus separation, source governance, or any clinical
decision-support behaviour that would change the SaMD/TGA assessment.

Notes

• docs/rag-improvement/baseline-record.md gains a §4a recording the split, the derivation, and the
  explicit consequence that readability rates are not comparable across the v1/v2 boundary.
• Files changed are limited to the authorised set: src/lib/rag/rag-eval-cases.ts,
  tests/rag-eval-cases.test.ts, scripts/fixtures/rag-adversarial-baseline.v1.json,
  docs/rag-improvement/baseline-record.md, and the single S3 row in
  docs/rag-improvement/HANDOVER.md.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in:30 minutes

Limit details: You’ve used all 1 included review currently available under your plan. You completed 101 included PR reviews in the past 7 days; at that activity level, included reviews refill at 1 review per hour.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 9a131335-3551-4979-930b-d28c740283be

📥 Commits

Reviewing files that changed from the base of the PR and between fe38c47 and 28b2ece.

📒 Files selected for processing (6)

• docs/branch-review-records/2399b67bef6792a2d01d66efd20576ba87265a3855f2cd37fc64641934115b87.record.md
• docs/rag-improvement/HANDOVER.md
• docs/rag-improvement/baseline-record.md
• scripts/fixtures/rag-adversarial-baseline.v1.json
• src/lib/rag/rag-eval-cases.ts
• tests/rag-eval-cases.test.ts

---

_{Comment @coderabbitai help to get the list of available commands.}

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.