fix(rag-eval): split the readability metric into fragmentation and a v19-derived length bound

docs/branch-review-records/2399b67bef6792a2d01d66efd20576ba87265a3855f2cd37fc64641934115b87.record.md

@@ -0,0 +1 @@
+| 2026-08-18 | 2129 | eb9cb7481674c436464b0a81b560c3c79cc93501 | RAG eval scorer: readability metric split (fragmentation vs v19-derived length bound), eval_config_version v1->v2, HANDOVER S3 row | Approved — evaluation scorer only; no retrieval/ranking/selection/prompt change; RAG impact declared as no behaviour change; pr-policy evaluatePullRequestPolicy ok:true with 7/7 governance items | verify:pr-local heavy scope failed:(none) not-reached:(none) incl lint+typecheck+full unit suite+build; eval:rag:offline 26 suites/623 tests; check:rag:fixtures 36 golden/26 suites; eval:rag:adversarial:offline 24 cases/8 categories/6 canaries + 25 tests, 3 KNOWN_DIVERGENCES still pinned; vitest tests/rag-eval-cases.test.ts 32 passed (27+5 new, mutation-checked) |

docs/rag-improvement/baseline-record.md

@@ -18,7 +18,7 @@ so two reports can be compared without guessing what changed between them.
 | --------------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------- |
 | `commit_sha` | `b7aa925f0ae19e89a9f0acf842b4a80d84083fb5` | `git rev-parse HEAD` of the evaluated tree. Full 40 characters, enforced. |
 | `dataset_version` | `rag-adversarial-cases.v1` | The fixture dataset's own `datasetVersion`; cross-checked against the file. |
-| `eval_config_version` | `rag-eval-config-v1` | Bumped by hand whenever a case list, threshold, or gate semantic changes. |
+| `eval_config_version` | `rag-eval-config-v2` | Bumped by hand whenever a case list, threshold, or gate semantic changes. |
 | `model_version` | `answer=gpt-5.6-terra; fast=gpt-5.6-terra; strong=gpt-5.6-sol` | The resolved answer-model defaults in `src/lib/env.ts`. |
 | `embedding_version` | `text-embedding-3-small@1536` | `OPENAI_EMBEDDING_MODEL` and `EMBEDDING_DIMENSIONS` in `src/lib/env.ts`. |
 | `index_version` | `20260818090000_schema_drift_snapshot_history_probe` | The latest applied migration — the index shape the retrieval RPCs run on. |
@@ -74,12 +74,44 @@ commit `b7aa925f0ae19e89a9f0acf842b4a80d84083fb5`. Offline gates were re-run at
 the three provider-backed gates stay `pending_owner_run`, carrying run `32100681177` at
 `4ea310e48` (prompt v18) as `priorRun` — that run is the baseline half of the S2 canary pair.
 
-One caveat travels with the `answer_quality` gate: `scoreAnswerQualityEvalCase`
-(`src/lib/rag/rag-eval-cases.ts`) scores readability over the answer **plus every section
-body** with a 220-word ceiling. The S2 targets can exceed that by design, so a readability=0
-flag caused only by total length is a metric artefact to adjudicate (raise the ceiling with an
-`eval_config_version` bump, or accept), not evidence of a regression. The scorer was left
-untouched in S2 so the before/after comparison runs under one definition.
+One caveat travelled with the `answer_quality` gate at `rag-eval-config-v1`:
+`scoreAnswerQualityEvalCase` (`src/lib/rag/rag-eval-cases.ts`) scored readability over the
+answer **plus every section body** as a single boolean combining a fragmentation regex with a
+flat 220-word ceiling, under one reason string ("fragmented or too long"). The S2 targets can
+exceed 220 by design, so a readability=0 flag caused only by total length was a metric artefact
+to adjudicate, not evidence of a regression — and it was not separable from the fragmentation
+regression the metric exists to catch. The scorer was deliberately left untouched in S2 so the
+before/after comparison ran under one definition.
+
+## 4a. Readability metric split (2026-08-18, `rag-eval-config-v2`)
+
+That caveat is now resolved, and the resolution is the reason this record reads
+`rag-eval-config-v2`. `scoreAnswerQualityEvalCase` keeps the five metric keys — `readability`
+still reports as one score, because `AnswerQualityMetric` is a closed union consumed by
+`scripts/eval-answer-quality.ts` as a total `Record<AnswerQualityMetric, number>` — but it now
+evaluates two independent checks under that key and reports whichever failed:
+
+- **Fragmentation** — the existing `fragmentPattern`, unchanged in both pattern and effect.
+- **Length** — `>= 5` words (the empty/stub floor, unchanged) and `<= 900` words.
+
+The 900-word ceiling is derived from the v19 contract rather than raised by judgement: the
+answer field's stated upper target is 110 words (`rag-answer-instructions.ts`), sections are
+capped at 6 (`answerSections.maxItems` in `rag.ts`, matching the prompt's "three to six"), and
+each section can carry a 48-character heading plus a 600-character body (both schema maxima in
+`rag.ts`). At a deliberately low 5 characters per word — chosen so the conversion overstates the
+word ceiling and the bound can never fail a well-formed answer — that is
+110 + 6 x (648 / 5) = 887.6, rounded up to 900.
+
+This is a contract ceiling, not a style ceiling: conciseness is enforced by the prompt and
+measured by `scoreAnswerTargeting`. An answer above 900 words could not have come from a
+schema-conformant generation, so the bound still catches runaway duplication and the
+deterministic composition paths (`rag-extractive-answer.ts`, `rag-comparison.ts`) that build a
+`RagAnswer` in code without the JSON schema.
+
+Consequence for comparisons: a `readability` rate recorded under `rag-eval-config-v1` is not
+comparable to one recorded under `v2`. No retrieval, ranking, selection, or generation
+behaviour changed — this is an evaluation-scorer change only, so it carries no canary
+requirement of its own.
 
 ## 5. Related
 

scripts/fixtures/rag-adversarial-baseline.v1.json

@@ -6,7 +6,7 @@
  "reportKey": {
  "commit_sha": "b7aa925f0ae19e89a9f0acf842b4a80d84083fb5",
  "dataset_version": "rag-adversarial-cases.v1",
- "eval_config_version": "rag-eval-config-v1",
+ "eval_config_version": "rag-eval-config-v2",
  "model_version": "answer=gpt-5.6-terra; fast=gpt-5.6-terra; strong=gpt-5.6-sol",
  "embedding_version": "text-embedding-3-small@1536",
  "index_version": "20260818090000_schema_drift_snapshot_history_probe"
@@ -31,7 +31,7 @@
  "caseCount": 30,
  "status": "pending_owner_run",
  "blockedReason": "npm run eval:answer-quality over answerQualityEvalCases is a provider-backed OpenAI/Supabase evaluation requiring explicit owner approval per run. Requested (not executed) by packet S2 as the Gate E before/after comparison.",
- "priorRun": "No before/after comparison has been recorded. Note for the S2 comparison: scoreAnswerQualityEvalCase's readability metric caps answer + section text at 220 words (src/lib/rag/rag-eval-cases.ts), which the S2 length targets can exceed by design; readability=0 flags driven only by total length are a metric artefact to adjudicate, not a regression."
+ "priorRun": "No before/after comparison has been recorded. The 220-word readability confound noted at eval_config_version rag-eval-config-v1 is resolved: scoreAnswerQualityEvalCase now scores fragmentation and length as two independent checks with separate reasons, and the length ceiling is derived from the v19 contract (110-word answer target + 6 sections at the schema heading/body maxima) rather than the pre-S2 flat 220. A readability=0 flag now names which check failed, so length and fragmentation no longer have to be adjudicated apart by hand. This gate semantic change is why the key reads rag-eval-config-v2; the first run under v2 has no v1-comparable readability rate."
  },
  {
  "id": "offline_contract",

src/lib/rag/rag-eval-cases.ts

@@ -87,7 +87,7 @@ export type AnswerQualityMetricScore = {
 
 export const answerQualityMetricLabels: Record<AnswerQualityMetric, string> = {
  relevance: "Answer addresses the requested entity and task.",
- readability: "Answer is grammatical, concise, and not fragment-like.",
+ readability: "Answer is not fragment-like, and its length is within the v19 answer+sections contract.",
  artifact_leaks: "Answer avoids backend, admin, provenance, and template wording.",
  intent_coverage: "Answer includes the action, dose, schedule, document list, or gap required by intent.",
  fail_closed: "Unsupported or weakly supported answers refuse specifically instead of guessing.",
@@ -138,6 +138,39 @@ function citesOrNamesExpectedDocument(testCase: AnswerQualityEvalCase, answer: R
  );
 }
 
+// Readability is scored as TWO independent checks that share one metric key, because
+// `AnswerQualityMetric` is a closed union consumed by `scripts/eval-answer-quality.ts` as a total
+// `Record<AnswerQualityMetric, number>` — a sixth key would break that aggregation (and the metric-key
+// pins in tests/rag-eval-cases.test.ts) without adding evaluative power. Each check therefore reports
+// its own reason so a failure names which contract it broke.
+//
+// Check 1 — FRAGMENTATION (unchanged): the regression this metric exists to catch. Answer text that
+// carries OCR/table run-together artefacts.
+//
+// Check 2 — LENGTH: a floor for empty/stub answers, and a ceiling derived from what prompt
+// `clinical-rag-answer-v19` can legitimately emit. Before packet S2 the ceiling was a flat 220 words
+// over `answerTextForQuality` (answer + every section heading and body). S2 (#2097, `dda4956ff`) moved
+// the answer field to 60–110 words and sections to three-to-six, so a correctly shaped v19 answer can
+// exceed 220 — at which point one conflated boolean could no longer separate "longer by design" from
+// "fragmented", the regression it is here for.
+//
+// Derivation of ANSWER_MAX_WORDS, from S2's own targets and the enforced response schema:
+// answer field 110 words `rag-answer-instructions.ts` upper target ("about 60-110 words")
+// sections 6 `answerSections.maxItems` in `rag.ts` (= the prompt's "three to six")
+// per section 648 chars `heading` maxLength 48 + `body` maxLength 600, both in `rag.ts`
+// chars per word 5 deliberately low, so the char->word conversion OVERSTATES the word
+// ceiling; this bound must never fail a well-formed answer.
+// => 110 + 6 * (648 / 5) = 110 + 777.6 -> 900 words (rounded up)
+//
+// This is a CONTRACT ceiling, not a style ceiling: conciseness is enforced by the prompt itself and
+// measured by `scoreAnswerTargeting`. Exceeding 900 words means the answer could not have come from a
+// schema-conformant generation — runaway duplication, or a deterministic composition path
+// (`rag-extractive-answer.ts`, `rag-comparison.ts`) that builds a RagAnswer in code without the JSON
+// schema. Both are real defects. Raising or lowering either bound is a gate-semantic change and needs
+// an `eval_config_version` bump in `scripts/fixtures/rag-adversarial-baseline.v1.json`.
+const ANSWER_MIN_WORDS = 5;
+const ANSWER_MAX_WORDS = 900;
+
 export function scoreAnswerQualityEvalCase(testCase: AnswerQualityEvalCase, answer: RagAnswer) {
  const text = answerTextForQuality(answer);
  const sourceBackedReviewStub = isSourceBackedReviewFallback(answer);
@@ -160,7 +193,14 @@ export function scoreAnswerQualityEvalCase(testCase: AnswerQualityEvalCase, answ
  expectedFileCoverage(testCase.expectedFiles, answer.citations, answer.citations.length).anyHit
  : answer.grounded && answer.citations.length >= testCase.minCitations && expectedClassOk
  : unsupported;
- const readabilityOk = wordCount >= 5 && wordCount <= 220 && !fragmentPattern.test(text);
+ const fragmentedText = fragmentPattern.test(text);
+ const lengthOk = wordCount >= ANSWER_MIN_WORDS && wordCount <= ANSWER_MAX_WORDS;
+ const readabilityOk = !fragmentedText && lengthOk;
+ const readabilityReasons = [
+ ...(fragmentedText ? ["fragmented"] : []),
+ ...(wordCount < ANSWER_MIN_WORDS ? [`too short (${wordCount} words < ${ANSWER_MIN_WORDS})`] : []),
+ ...(wordCount > ANSWER_MAX_WORDS ? [`too long (${wordCount} words > ${ANSWER_MAX_WORDS})`] : []),
+ ];
  const artifactOk = !artifactPattern.test(text) && containsNone(text, testCase.mustNotContain);
  const intentOk = !sourceBackedReviewStub && containsAny(text, testCase.mustContainAny);
  const failClosedOk =
@@ -171,7 +211,7 @@ export function scoreAnswerQualityEvalCase(testCase: AnswerQualityEvalCase, answ
  {
  metric: "readability",
  score: readabilityOk ? 1 : 0,
- reason: readabilityOk ? "readable" : "fragmented or too long",
+ reason: readabilityOk ? "readable" : readabilityReasons.join("; "),
  },
  { metric: "artifact_leaks", score: artifactOk ? 1 : 0, reason: artifactOk ? "clean" : "artifact wording present" },
  {

tests/rag-eval-cases.test.ts

@@ -326,6 +326,96 @@ describe("captured RAG eval cases", () => {
  expect(scores.every((score) => score.score === 1)).toBe(true);
  });
 
+ describe("readability: fragmentation and length are scored independently", () => {
+ // Packet S2 (#2097, dda4956ff, prompt clinical-rag-answer-v19) moved the answer field to
+ // 60-110 words and sections to three-to-six. The old scorer conflated fragmentation and a flat
+ // 220-word ceiling into one boolean with one reason ("fragmented or too long"), so a correctly
+ // shaped v19 answer could fail for length and be indistinguishable from the fragmentation
+ // regression the metric exists to catch. These tests pin the split.
+ const qualityCase = answerQualityEvalCases.find((item) => item.id === "quality-discharge-documentation")!;
+
+ function readabilityOf(text: string, sections: RagAnswer["answerSections"] = []) {
+ const answer = {
+ answer: text,
+ grounded: true,
+ confidence: "high",
+ citations: [],
+ sources: [],
+ routingMode: "fast",
+ queryClass: "document_lookup",
+ answerSections: sections,
+ } satisfies RagAnswer;
+ return scoreAnswerQualityEvalCase(qualityCase, answer).find((score) => score.metric === "readability")!;
+ }
+
+ // Clean clinical prose with no fragmentation artefacts: no run-together digit lists, no
+ // "? monitoring" break, no glued-together heading token.
+ const cleanSentence =
+ "Review the admission record and confirm the responsible consultant has documented the current plan. ";
+
+ it("scores a long but clean v19-shaped answer as readable", () => {
+ // ~110-word answer plus six sections, the maximum shape prompt v19 asks for. Comfortably over
+ // the retired 220-word ceiling, comfortably under the derived 900-word contract ceiling.
+ const answerField = cleanSentence.repeat(8);
+ const sections = Array.from({ length: 6 }, (_, index) => ({
+ heading: `Section ${String.fromCharCode(65 + index)}`,
+ kind: "required_actions" as const,
+ supportLevel: "direct" as const,
+ body: cleanSentence.repeat(5),
+ citation_chunk_ids: [],
+ })) satisfies RagAnswer["answerSections"];
+
+ const score = readabilityOf(answerField, sections);
+ const wordCount = [answerField, ...sections.map((section) => `${section.heading}: ${section.body}`)]
+ .join(" ")
+ .split(/\s+/)
+ .filter(Boolean).length;
+
+ // Guard the guard: this fixture must actually exercise the regression it claims to.
+ expect(wordCount).toBeGreaterThan(220);
+ expect(wordCount).toBeLessThan(900);
+ expect(score.score).toBe(1);
+ expect(score.reason).toBe("readable");
+ });
+
+ it("still fails a genuinely fragmented answer, and names fragmentation as the reason", () => {
+ const score = readabilityOf(
+ "Clozapine monitoring anyMANAGEMENT of the neutrophil result follows the escalation pathway.",
+ );
+
+ expect(score.score).toBe(0);
+ expect(score.reason).toContain("fragmented");
+ // Distinguishable from the length failure by the reason alone.
+ expect(score.reason).not.toContain("too long");
+ expect(score.reason).not.toContain("too short");
+ });
+
+ it("still fails a runaway-length answer, and names length as the reason", () => {
+ const score = readabilityOf(cleanSentence.repeat(80));
+
+ expect(score.score).toBe(0);
+ expect(score.reason).toContain("too long");
+ // Distinguishable from the fragmentation failure by the reason alone.
+ expect(score.reason).not.toContain("fragmented");
+ });
+
+ it("still fails an empty-stub answer as too short", () => {
+ const score = readabilityOf("No source.");
+
+ expect(score.score).toBe(0);
+ expect(score.reason).toContain("too short");
+ expect(score.reason).not.toContain("fragmented");
+ });
+
+ it("reports both reasons when an answer is fragmented AND over length", () => {
+ const score = readabilityOf(`anyMANAGEMENT ${cleanSentence.repeat(80)}`);
+
+ expect(score.score).toBe(0);
+ expect(score.reason).toContain("fragmented");
+ expect(score.reason).toContain("too long");
+ });
+ });
+
  describe("scoreAnswerTargeting (structural per-intent targeting)", () => {
  const doseCase = {
  id: "t-dose",