From eb9cb7481674c436464b0a81b560c3c79cc93501 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Tue, 18 Aug 2026 22:34:34 +0800 Subject: [PATCH 1/2] fix(rag-eval): split the readability metric into fragmentation and a v19-derived length bound scoreAnswerQualityEvalCase scored readability as one boolean -- wordCount >= 5 && wordCount <= 220 && !fragmentPattern.test(text) -- over answerTextForQuality, which sums the answer field plus every section heading and body, and reported one conflated reason ("fragmented or too long"). Packet S2 (#2097, dda4956ff, prompt clinical-rag-answer-v19) raised the answer field to 60-110 words and sections to three-to-six, so a correctly shaped v19 answer can exceed 220 words. The metric could then no longer separate intended length from the fragmentation regression it exists to catch. Fragmentation and length are now evaluated independently and each failure reports its own reason. Fragmentation is unchanged. The 900-word ceiling is derived from the v19 contract rather than raised by judgement: the 110-word answer upper target, 6 sections (answerSections.maxItems), and each section's 48-char heading plus 600-char body schema maxima, converted at a deliberately low 5 chars/word so the bound cannot fail a well-formed answer -- 110 + 6 * (648 / 5) = 887.6, rounded up to 900. The two checks share the single readability metric key because AnswerQualityMetric is a closed union consumed by scripts/eval-answer-quality.ts as a total Record, and two existing tests pin the five-key set; a sixth key would break both while adding no evaluative power. Bumps eval_config_version to rag-eval-config-v2 in the adversarial baseline and baseline-record (gate-semantic change: readability rates are not comparable across the boundary), and sets the packet S3 row in HANDOVER.md to merged. RAG impact: no retrieval behaviour change -- evaluation scorer only; gate semantics change is carried by the eval_config_version bump Co-Authored-By: Claude Opus 5 --- docs/rag-improvement/HANDOVER.md | 38 ++++---- docs/rag-improvement/baseline-record.md | 46 ++++++++-- .../fixtures/rag-adversarial-baseline.v1.json | 4 +- src/lib/rag/rag-eval-cases.ts | 46 +++++++++- tests/rag-eval-cases.test.ts | 90 +++++++++++++++++++ 5 files changed, 193 insertions(+), 31 deletions(-) diff --git a/docs/rag-improvement/HANDOVER.md b/docs/rag-improvement/HANDOVER.md index b9093fc0c6..260855b4de 100644 --- a/docs/rag-improvement/HANDOVER.md +++ b/docs/rag-improvement/HANDOVER.md @@ -76,25 +76,25 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured ## 2. Status table — update in every programme PR -| Packet | Scope | Branch | PR | State | Canary / evidence refs | -| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | -| Handover | Multi-session handover + coordination | `claude/rag-plan-review-guide-vhrls9` | #1908 / #2024 | Merged 2026-08-13; coordination layer PR #2024 | docs-only | -| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | -| S1 | A1 phase 2: rung-1 verification-faithfulness fixes | `claude/s1-rag-mitigation-231-86c182` | #2022 | Merged 2026-08-17 (squash `2bd146eed`, landed by content) | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; canary pair run 31964560921 (baseline `8f8d111ab`) -> run 32025082010 (`2bd146eed`): recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44 (denominator reconciled by S5; see baseline-record §3); rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | -| S1b | A1 rung 3 (R1): pre-deadline strong routing for dosing class | `claude/s1b-rag-dosing-routing-6u1mik` | #2035 | Merged 2026-08-17 (PR #2035, merge `92f7618`) | canary pair pending: baseline run 32025082010 (`2bd146eed`) -> post-merge dispatch (owner-approved); offline 586/586 + verify:pr-local heavy scope green | -| S1c | A1 residuals R2 + R3: claim-support strictness | `claude/s1c-residuals-r2-r3-4pb1at` | #2052 | Merged 2026-08-17 (merge `b8e774bcd`; follow-up #2063 kept; follow-up #2065 reverted by PR #2088 after canary regression) | canary pair: baseline run 32049952885 -> post run 32052479537 (`084f63799`): recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44 | -| S1d | A1 final-gate gap recovery: hedged cited low-confidence fast answers must recover extractively, not collapse to a citation-free `provider_source_gap` | `claude/s1d-final-gate-gap-recovery-dxgrn2` | #2054 | Merged 2026-08-17 (merge `0bbd64fbc`); landed by content; canary pair green after the #2065 revert (PR #2088) | canary pair 32052479537 -> 32100681177 (`4ea310e48`) green: recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44. Interim post run 32097916649 (`9904fbda8`) was RED on `agitation-im-po-route-short-terms` — bisected live to PR #2065 (S1c follow-up condition-first regex), not S1d; reverted by PR #2088; the confirmation run is 32100681177 | -| G1 | Governance: provenance tag for document-summary rows (Option B) | `claude/g1-rag-document-context-qn9ubx` | #2053 | Merged 2026-08-17 (merge `125e98526`); rows #J912J9 / #0MSNT8 closed at reconcile | no canary (no behaviour change); `document_context` tag at `types.ts` + `rag-row-contracts.ts`, deriveConfidence pinned | -| S2 | A2 + A3: composition menu + moderate length | `claude/s2-rag-composition-7330b0` | #2097 | Merged 2026-08-18 (squash dda4956ff), landed by content; A2 and A3 together; prompt clinical-rag-answer-v19, schema v4, answerSections.maxItems 6 | offline: composition 9/9 + prompt pins 5/5, eval:rag:offline 26 suites / 623 tests, eval:rag:adversarial:offline 25/25 (3 divergences still pinned), rag.ts 4362/4362; canary pair 32100681177 (4ea310e48) -> 32111839806 GREEN (document/content recall 1.0/1.0, zero per-case rr regressions, answer gate clean; one non-blocking 20 s latency advisory on neuroleptic-side-effect-escalation); eval:answer-quality v18 (4ea310e48) vs v19 run 2026-08-18 neutral within nondeterminism (relevance 0.633->0.567 on two nondeterministic timeout/gap cases, targeting 0.409->0.429, readability at ceiling), owner blinded read pending. Note: the 220-word total-length readability ceiling in scoreAnswerQualityEvalCase is a known metric confound for A3 (baseline-record §4) | -| S2b | A3: moderate length (if separate review needed) | — | — | Not needed — A3 shipped inside S2 (combined diff stayed reviewable) | — | -| S3 | A4: follow-up suggestion refinement | `claude/s3-follow-up-suggestions-95e160` | #2108 | PR open 2026-08-18 (menu-derived, evidence-gated, already-answered-suppressed chips in src/lib/answer-follow-up.ts; no ClinicalDashboard change, no new module/field/render block); owner merges | offline only: answer-follow-up 27/27 (menu alignment across all 48 class×intent cells, evidence gate positive + discriminating negatives, suppression), answer-follow-up-chips DOM 6/6 (desktop + phone composer surfaces), answer-composition 9/9 unchanged; no canary — deterministic composition only, generation prompt untouched | -| S4 | B0: adversarial fixtures + baseline + register | `claude/packet-s4-adversarial-fixtures-5ho5tp` | #2036 | Merged 2026-08-17 (squash `f5b093291`) | Offline only: `check:rag:adversarial-fixtures` 24 cases / 8 categories / 6 canaries; `eval:rag:offline` 24 suites, 597 tests. Baseline `scripts/fixtures/rag-adversarial-baseline.v1.json` marks the three provider-backed gates `pending_owner_run` | -| S5 | B1+B2: telemetry assessment + offline harness | `claude/s5-rag-telemetry-harness-2wvis7` | #2056 | Merged 2026-08-17 (merge `093f9340c`); post-merge canary run 32049952885 | Offline only: `eval:rag:adversarial:offline` 25/25 (24 cases + canary-free report; 3 divergences pinned in `KNOWN_DIVERGENCES`); B1 gap = `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED` (default false); canary-absence tests green; 44/44 denominator reconciled | -| S6 | B3: Docling lab benchmark | `claude/packet-s6-docling-lab-d6foa6` | #2057 | Merged 2026-08-17 (merge `5a6418636`) | Offline only: `check:docling-lab` 36 fixtures / 10 hostile / 6 canaries + Gate B template valid; `verify:pr-local` heavy plan failed:(none); contract test 20/20; legacy smoke 46 docs, 10/10 hostile contained, canary-clean report. Verdict is a separate owner dispatch of `docling-lab.yml` | -| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | -| #212 T1–T3 | Runtime row contracts (rag.ts, rag-candidate-sources.ts, src/app/api) — sibling stream sharing `src/lib/rag/**` | — | #1946 / #1981 / #2023 | Merged (T3 squash `440a34f71` 2026-08-17) | see the #212 ledger row; RAG surface complete for the cast class | -| #212 T4 | Runtime row contracts: `worker/main.ts` (11 casts) — sibling stream | `claude/ledger-212-tranche-4-worker-q3y6i4` | #2037 | Merged 2026-08-17 (squash `1726537b7`); #212 closed by reconcile PR #2045 | Governance Preflight complete; audit: 1 inbound cast (claim rows, per-row fail-soft) + 2 read-back param casts contracted, 9 outbound/interop left; closes #212 (inbox `done` queued in the PR) | +| Packet | Scope | Branch | PR | State | Canary / evidence refs | +| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | +| Handover | Multi-session handover + coordination | `claude/rag-plan-review-guide-vhrls9` | #1908 / #2024 | Merged 2026-08-13; coordination layer PR #2024 | docs-only | +| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | +| S1 | A1 phase 2: rung-1 verification-faithfulness fixes | `claude/s1-rag-mitigation-231-86c182` | #2022 | Merged 2026-08-17 (squash `2bd146eed`, landed by content) | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; canary pair run 31964560921 (baseline `8f8d111ab`) -> run 32025082010 (`2bd146eed`): recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44 (denominator reconciled by S5; see baseline-record §3); rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | +| S1b | A1 rung 3 (R1): pre-deadline strong routing for dosing class | `claude/s1b-rag-dosing-routing-6u1mik` | #2035 | Merged 2026-08-17 (PR #2035, merge `92f7618`) | canary pair pending: baseline run 32025082010 (`2bd146eed`) -> post-merge dispatch (owner-approved); offline 586/586 + verify:pr-local heavy scope green | +| S1c | A1 residuals R2 + R3: claim-support strictness | `claude/s1c-residuals-r2-r3-4pb1at` | #2052 | Merged 2026-08-17 (merge `b8e774bcd`; follow-up #2063 kept; follow-up #2065 reverted by PR #2088 after canary regression) | canary pair: baseline run 32049952885 -> post run 32052479537 (`084f63799`): recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44 | +| S1d | A1 final-gate gap recovery: hedged cited low-confidence fast answers must recover extractively, not collapse to a citation-free `provider_source_gap` | `claude/s1d-final-gate-gap-recovery-dxgrn2` | #2054 | Merged 2026-08-17 (merge `0bbd64fbc`); landed by content; canary pair green after the #2065 revert (PR #2088) | canary pair 32052479537 -> 32100681177 (`4ea310e48`) green: recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44. Interim post run 32097916649 (`9904fbda8`) was RED on `agitation-im-po-route-short-terms` — bisected live to PR #2065 (S1c follow-up condition-first regex), not S1d; reverted by PR #2088; the confirmation run is 32100681177 | +| G1 | Governance: provenance tag for document-summary rows (Option B) | `claude/g1-rag-document-context-qn9ubx` | #2053 | Merged 2026-08-17 (merge `125e98526`); rows #J912J9 / #0MSNT8 closed at reconcile | no canary (no behaviour change); `document_context` tag at `types.ts` + `rag-row-contracts.ts`, deriveConfidence pinned | +| S2 | A2 + A3: composition menu + moderate length | `claude/s2-rag-composition-7330b0` | #2097 | Merged 2026-08-18 (squash dda4956ff), landed by content; A2 and A3 together; prompt clinical-rag-answer-v19, schema v4, answerSections.maxItems 6 | offline: composition 9/9 + prompt pins 5/5, eval:rag:offline 26 suites / 623 tests, eval:rag:adversarial:offline 25/25 (3 divergences still pinned), rag.ts 4362/4362; canary pair 32100681177 (4ea310e48) -> 32111839806 GREEN (document/content recall 1.0/1.0, zero per-case rr regressions, answer gate clean; one non-blocking 20 s latency advisory on neuroleptic-side-effect-escalation); eval:answer-quality v18 (4ea310e48) vs v19 run 2026-08-18 neutral within nondeterminism (relevance 0.633->0.567 on two nondeterministic timeout/gap cases, targeting 0.409->0.429, readability at ceiling), owner blinded read pending. Note: the 220-word total-length readability ceiling in scoreAnswerQualityEvalCase is a known metric confound for A3 (baseline-record §4) | +| S2b | A3: moderate length (if separate review needed) | — | — | Not needed — A3 shipped inside S2 (combined diff stayed reviewable) | — | +| S3 | A4: follow-up suggestion refinement | `claude/s3-follow-up-suggestions-95e160` | #2108 | Merged 2026-08-18 (squash 511d22f4d), landed by content; Track A complete | offline only: answer-follow-up 27/27 (menu alignment across all 48 class×intent cells, evidence gate positive + discriminating negatives, suppression), answer-follow-up-chips DOM 6/6 (desktop + phone composer surfaces), answer-composition 9/9 unchanged; no canary — deterministic composition only, generation prompt untouched | +| S4 | B0: adversarial fixtures + baseline + register | `claude/packet-s4-adversarial-fixtures-5ho5tp` | #2036 | Merged 2026-08-17 (squash `f5b093291`) | Offline only: `check:rag:adversarial-fixtures` 24 cases / 8 categories / 6 canaries; `eval:rag:offline` 24 suites, 597 tests. Baseline `scripts/fixtures/rag-adversarial-baseline.v1.json` marks the three provider-backed gates `pending_owner_run` | +| S5 | B1+B2: telemetry assessment + offline harness | `claude/s5-rag-telemetry-harness-2wvis7` | #2056 | Merged 2026-08-17 (merge `093f9340c`); post-merge canary run 32049952885 | Offline only: `eval:rag:adversarial:offline` 25/25 (24 cases + canary-free report; 3 divergences pinned in `KNOWN_DIVERGENCES`); B1 gap = `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED` (default false); canary-absence tests green; 44/44 denominator reconciled | +| S6 | B3: Docling lab benchmark | `claude/packet-s6-docling-lab-d6foa6` | #2057 | Merged 2026-08-17 (merge `5a6418636`) | Offline only: `check:docling-lab` 36 fixtures / 10 hostile / 6 canaries + Gate B template valid; `verify:pr-local` heavy plan failed:(none); contract test 20/20; legacy smoke 46 docs, 10/10 hostile contained, canary-clean report. Verdict is a separate owner dispatch of `docling-lab.yml` | +| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | +| #212 T1–T3 | Runtime row contracts (rag.ts, rag-candidate-sources.ts, src/app/api) — sibling stream sharing `src/lib/rag/**` | — | #1946 / #1981 / #2023 | Merged (T3 squash `440a34f71` 2026-08-17) | see the #212 ledger row; RAG surface complete for the cast class | +| #212 T4 | Runtime row contracts: `worker/main.ts` (11 casts) — sibling stream | `claude/ledger-212-tranche-4-worker-q3y6i4` | #2037 | Merged 2026-08-17 (squash `1726537b7`); #212 closed by reconcile PR #2045 | Governance Preflight complete; audit: 1 inbound cast (claim rows, per-row fail-soft) + 2 read-back param casts contracted, 9 outbound/interop left; closes #212 (inbox `done` queued in the PR) | Update rule: the session that opens a packet's PR edits its row (branch, PR number, state) in the same PR. A later session updating another packet may also correct stale rows diff --git a/docs/rag-improvement/baseline-record.md b/docs/rag-improvement/baseline-record.md index 3536270a33..0b4ea51987 100644 --- a/docs/rag-improvement/baseline-record.md +++ b/docs/rag-improvement/baseline-record.md @@ -18,7 +18,7 @@ so two reports can be compared without guessing what changed between them. | --------------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------- | | `commit_sha` | `b7aa925f0ae19e89a9f0acf842b4a80d84083fb5` | `git rev-parse HEAD` of the evaluated tree. Full 40 characters, enforced. | | `dataset_version` | `rag-adversarial-cases.v1` | The fixture dataset's own `datasetVersion`; cross-checked against the file. | -| `eval_config_version` | `rag-eval-config-v1` | Bumped by hand whenever a case list, threshold, or gate semantic changes. | +| `eval_config_version` | `rag-eval-config-v2` | Bumped by hand whenever a case list, threshold, or gate semantic changes. | | `model_version` | `answer=gpt-5.6-terra; fast=gpt-5.6-terra; strong=gpt-5.6-sol` | The resolved answer-model defaults in `src/lib/env.ts`. | | `embedding_version` | `text-embedding-3-small@1536` | `OPENAI_EMBEDDING_MODEL` and `EMBEDDING_DIMENSIONS` in `src/lib/env.ts`. | | `index_version` | `20260818090000_schema_drift_snapshot_history_probe` | The latest applied migration — the index shape the retrieval RPCs run on. | @@ -74,12 +74,44 @@ commit `b7aa925f0ae19e89a9f0acf842b4a80d84083fb5`. Offline gates were re-run at the three provider-backed gates stay `pending_owner_run`, carrying run `32100681177` at `4ea310e48` (prompt v18) as `priorRun` — that run is the baseline half of the S2 canary pair. -One caveat travels with the `answer_quality` gate: `scoreAnswerQualityEvalCase` -(`src/lib/rag/rag-eval-cases.ts`) scores readability over the answer **plus every section -body** with a 220-word ceiling. The S2 targets can exceed that by design, so a readability=0 -flag caused only by total length is a metric artefact to adjudicate (raise the ceiling with an -`eval_config_version` bump, or accept), not evidence of a regression. The scorer was left -untouched in S2 so the before/after comparison runs under one definition. +One caveat travelled with the `answer_quality` gate at `rag-eval-config-v1`: +`scoreAnswerQualityEvalCase` (`src/lib/rag/rag-eval-cases.ts`) scored readability over the +answer **plus every section body** as a single boolean combining a fragmentation regex with a +flat 220-word ceiling, under one reason string ("fragmented or too long"). The S2 targets can +exceed 220 by design, so a readability=0 flag caused only by total length was a metric artefact +to adjudicate, not evidence of a regression — and it was not separable from the fragmentation +regression the metric exists to catch. The scorer was deliberately left untouched in S2 so the +before/after comparison ran under one definition. + +## 4a. Readability metric split (2026-08-18, `rag-eval-config-v2`) + +That caveat is now resolved, and the resolution is the reason this record reads +`rag-eval-config-v2`. `scoreAnswerQualityEvalCase` keeps the five metric keys — `readability` +still reports as one score, because `AnswerQualityMetric` is a closed union consumed by +`scripts/eval-answer-quality.ts` as a total `Record` — but it now +evaluates two independent checks under that key and reports whichever failed: + +- **Fragmentation** — the existing `fragmentPattern`, unchanged in both pattern and effect. +- **Length** — `>= 5` words (the empty/stub floor, unchanged) and `<= 900` words. + +The 900-word ceiling is derived from the v19 contract rather than raised by judgement: the +answer field's stated upper target is 110 words (`rag-answer-instructions.ts`), sections are +capped at 6 (`answerSections.maxItems` in `rag.ts`, matching the prompt's "three to six"), and +each section can carry a 48-character heading plus a 600-character body (both schema maxima in +`rag.ts`). At a deliberately low 5 characters per word — chosen so the conversion overstates the +word ceiling and the bound can never fail a well-formed answer — that is +110 + 6 x (648 / 5) = 887.6, rounded up to 900. + +This is a contract ceiling, not a style ceiling: conciseness is enforced by the prompt and +measured by `scoreAnswerTargeting`. An answer above 900 words could not have come from a +schema-conformant generation, so the bound still catches runaway duplication and the +deterministic composition paths (`rag-extractive-answer.ts`, `rag-comparison.ts`) that build a +`RagAnswer` in code without the JSON schema. + +Consequence for comparisons: a `readability` rate recorded under `rag-eval-config-v1` is not +comparable to one recorded under `v2`. No retrieval, ranking, selection, or generation +behaviour changed — this is an evaluation-scorer change only, so it carries no canary +requirement of its own. ## 5. Related diff --git a/scripts/fixtures/rag-adversarial-baseline.v1.json b/scripts/fixtures/rag-adversarial-baseline.v1.json index 7a1e9bf7ca..5c797ccd1f 100644 --- a/scripts/fixtures/rag-adversarial-baseline.v1.json +++ b/scripts/fixtures/rag-adversarial-baseline.v1.json @@ -6,7 +6,7 @@ "reportKey": { "commit_sha": "b7aa925f0ae19e89a9f0acf842b4a80d84083fb5", "dataset_version": "rag-adversarial-cases.v1", - "eval_config_version": "rag-eval-config-v1", + "eval_config_version": "rag-eval-config-v2", "model_version": "answer=gpt-5.6-terra; fast=gpt-5.6-terra; strong=gpt-5.6-sol", "embedding_version": "text-embedding-3-small@1536", "index_version": "20260818090000_schema_drift_snapshot_history_probe" @@ -31,7 +31,7 @@ "caseCount": 30, "status": "pending_owner_run", "blockedReason": "npm run eval:answer-quality over answerQualityEvalCases is a provider-backed OpenAI/Supabase evaluation requiring explicit owner approval per run. Requested (not executed) by packet S2 as the Gate E before/after comparison.", - "priorRun": "No before/after comparison has been recorded. Note for the S2 comparison: scoreAnswerQualityEvalCase's readability metric caps answer + section text at 220 words (src/lib/rag/rag-eval-cases.ts), which the S2 length targets can exceed by design; readability=0 flags driven only by total length are a metric artefact to adjudicate, not a regression." + "priorRun": "No before/after comparison has been recorded. The 220-word readability confound noted at eval_config_version rag-eval-config-v1 is resolved: scoreAnswerQualityEvalCase now scores fragmentation and length as two independent checks with separate reasons, and the length ceiling is derived from the v19 contract (110-word answer target + 6 sections at the schema heading/body maxima) rather than the pre-S2 flat 220. A readability=0 flag now names which check failed, so length and fragmentation no longer have to be adjudicated apart by hand. This gate semantic change is why the key reads rag-eval-config-v2; the first run under v2 has no v1-comparable readability rate." }, { "id": "offline_contract", diff --git a/src/lib/rag/rag-eval-cases.ts b/src/lib/rag/rag-eval-cases.ts index 34732e9be1..5279881320 100644 --- a/src/lib/rag/rag-eval-cases.ts +++ b/src/lib/rag/rag-eval-cases.ts @@ -87,7 +87,7 @@ export type AnswerQualityMetricScore = { export const answerQualityMetricLabels: Record = { relevance: "Answer addresses the requested entity and task.", - readability: "Answer is grammatical, concise, and not fragment-like.", + readability: "Answer is not fragment-like, and its length is within the v19 answer+sections contract.", artifact_leaks: "Answer avoids backend, admin, provenance, and template wording.", intent_coverage: "Answer includes the action, dose, schedule, document list, or gap required by intent.", fail_closed: "Unsupported or weakly supported answers refuse specifically instead of guessing.", @@ -138,6 +138,39 @@ function citesOrNamesExpectedDocument(testCase: AnswerQualityEvalCase, answer: R ); } +// Readability is scored as TWO independent checks that share one metric key, because +// `AnswerQualityMetric` is a closed union consumed by `scripts/eval-answer-quality.ts` as a total +// `Record` — a sixth key would break that aggregation (and the metric-key +// pins in tests/rag-eval-cases.test.ts) without adding evaluative power. Each check therefore reports +// its own reason so a failure names which contract it broke. +// +// Check 1 — FRAGMENTATION (unchanged): the regression this metric exists to catch. Answer text that +// carries OCR/table run-together artefacts. +// +// Check 2 — LENGTH: a floor for empty/stub answers, and a ceiling derived from what prompt +// `clinical-rag-answer-v19` can legitimately emit. Before packet S2 the ceiling was a flat 220 words +// over `answerTextForQuality` (answer + every section heading and body). S2 (#2097, `dda4956ff`) moved +// the answer field to 60–110 words and sections to three-to-six, so a correctly shaped v19 answer can +// exceed 220 — at which point one conflated boolean could no longer separate "longer by design" from +// "fragmented", the regression it is here for. +// +// Derivation of ANSWER_MAX_WORDS, from S2's own targets and the enforced response schema: +// answer field 110 words `rag-answer-instructions.ts` upper target ("about 60-110 words") +// sections 6 `answerSections.maxItems` in `rag.ts` (= the prompt's "three to six") +// per section 648 chars `heading` maxLength 48 + `body` maxLength 600, both in `rag.ts` +// chars per word 5 deliberately low, so the char->word conversion OVERSTATES the word +// ceiling; this bound must never fail a well-formed answer. +// => 110 + 6 * (648 / 5) = 110 + 777.6 -> 900 words (rounded up) +// +// This is a CONTRACT ceiling, not a style ceiling: conciseness is enforced by the prompt itself and +// measured by `scoreAnswerTargeting`. Exceeding 900 words means the answer could not have come from a +// schema-conformant generation — runaway duplication, or a deterministic composition path +// (`rag-extractive-answer.ts`, `rag-comparison.ts`) that builds a RagAnswer in code without the JSON +// schema. Both are real defects. Raising or lowering either bound is a gate-semantic change and needs +// an `eval_config_version` bump in `scripts/fixtures/rag-adversarial-baseline.v1.json`. +const ANSWER_MIN_WORDS = 5; +const ANSWER_MAX_WORDS = 900; + export function scoreAnswerQualityEvalCase(testCase: AnswerQualityEvalCase, answer: RagAnswer) { const text = answerTextForQuality(answer); const sourceBackedReviewStub = isSourceBackedReviewFallback(answer); @@ -160,7 +193,14 @@ export function scoreAnswerQualityEvalCase(testCase: AnswerQualityEvalCase, answ expectedFileCoverage(testCase.expectedFiles, answer.citations, answer.citations.length).anyHit : answer.grounded && answer.citations.length >= testCase.minCitations && expectedClassOk : unsupported; - const readabilityOk = wordCount >= 5 && wordCount <= 220 && !fragmentPattern.test(text); + const fragmentedText = fragmentPattern.test(text); + const lengthOk = wordCount >= ANSWER_MIN_WORDS && wordCount <= ANSWER_MAX_WORDS; + const readabilityOk = !fragmentedText && lengthOk; + const readabilityReasons = [ + ...(fragmentedText ? ["fragmented"] : []), + ...(wordCount < ANSWER_MIN_WORDS ? [`too short (${wordCount} words < ${ANSWER_MIN_WORDS})`] : []), + ...(wordCount > ANSWER_MAX_WORDS ? [`too long (${wordCount} words > ${ANSWER_MAX_WORDS})`] : []), + ]; const artifactOk = !artifactPattern.test(text) && containsNone(text, testCase.mustNotContain); const intentOk = !sourceBackedReviewStub && containsAny(text, testCase.mustContainAny); const failClosedOk = @@ -171,7 +211,7 @@ export function scoreAnswerQualityEvalCase(testCase: AnswerQualityEvalCase, answ { metric: "readability", score: readabilityOk ? 1 : 0, - reason: readabilityOk ? "readable" : "fragmented or too long", + reason: readabilityOk ? "readable" : readabilityReasons.join("; "), }, { metric: "artifact_leaks", score: artifactOk ? 1 : 0, reason: artifactOk ? "clean" : "artifact wording present" }, { diff --git a/tests/rag-eval-cases.test.ts b/tests/rag-eval-cases.test.ts index 423dd038b2..8e438c58da 100644 --- a/tests/rag-eval-cases.test.ts +++ b/tests/rag-eval-cases.test.ts @@ -326,6 +326,96 @@ describe("captured RAG eval cases", () => { expect(scores.every((score) => score.score === 1)).toBe(true); }); + describe("readability: fragmentation and length are scored independently", () => { + // Packet S2 (#2097, dda4956ff, prompt clinical-rag-answer-v19) moved the answer field to + // 60-110 words and sections to three-to-six. The old scorer conflated fragmentation and a flat + // 220-word ceiling into one boolean with one reason ("fragmented or too long"), so a correctly + // shaped v19 answer could fail for length and be indistinguishable from the fragmentation + // regression the metric exists to catch. These tests pin the split. + const qualityCase = answerQualityEvalCases.find((item) => item.id === "quality-discharge-documentation")!; + + function readabilityOf(text: string, sections: RagAnswer["answerSections"] = []) { + const answer = { + answer: text, + grounded: true, + confidence: "high", + citations: [], + sources: [], + routingMode: "fast", + queryClass: "document_lookup", + answerSections: sections, + } satisfies RagAnswer; + return scoreAnswerQualityEvalCase(qualityCase, answer).find((score) => score.metric === "readability")!; + } + + // Clean clinical prose with no fragmentation artefacts: no run-together digit lists, no + // "? monitoring" break, no glued-together heading token. + const cleanSentence = + "Review the admission record and confirm the responsible consultant has documented the current plan. "; + + it("scores a long but clean v19-shaped answer as readable", () => { + // ~110-word answer plus six sections, the maximum shape prompt v19 asks for. Comfortably over + // the retired 220-word ceiling, comfortably under the derived 900-word contract ceiling. + const answerField = cleanSentence.repeat(8); + const sections = Array.from({ length: 6 }, (_, index) => ({ + heading: `Section ${String.fromCharCode(65 + index)}`, + kind: "required_actions" as const, + supportLevel: "direct" as const, + body: cleanSentence.repeat(5), + citation_chunk_ids: [], + })) satisfies RagAnswer["answerSections"]; + + const score = readabilityOf(answerField, sections); + const wordCount = [answerField, ...sections.map((section) => `${section.heading}: ${section.body}`)] + .join(" ") + .split(/\s+/) + .filter(Boolean).length; + + // Guard the guard: this fixture must actually exercise the regression it claims to. + expect(wordCount).toBeGreaterThan(220); + expect(wordCount).toBeLessThan(900); + expect(score.score).toBe(1); + expect(score.reason).toBe("readable"); + }); + + it("still fails a genuinely fragmented answer, and names fragmentation as the reason", () => { + const score = readabilityOf( + "Clozapine monitoring anyMANAGEMENT of the neutrophil result follows the escalation pathway.", + ); + + expect(score.score).toBe(0); + expect(score.reason).toContain("fragmented"); + // Distinguishable from the length failure by the reason alone. + expect(score.reason).not.toContain("too long"); + expect(score.reason).not.toContain("too short"); + }); + + it("still fails a runaway-length answer, and names length as the reason", () => { + const score = readabilityOf(cleanSentence.repeat(80)); + + expect(score.score).toBe(0); + expect(score.reason).toContain("too long"); + // Distinguishable from the fragmentation failure by the reason alone. + expect(score.reason).not.toContain("fragmented"); + }); + + it("still fails an empty-stub answer as too short", () => { + const score = readabilityOf("No source."); + + expect(score.score).toBe(0); + expect(score.reason).toContain("too short"); + expect(score.reason).not.toContain("fragmented"); + }); + + it("reports both reasons when an answer is fragmented AND over length", () => { + const score = readabilityOf(`anyMANAGEMENT ${cleanSentence.repeat(80)}`); + + expect(score.score).toBe(0); + expect(score.reason).toContain("fragmented"); + expect(score.reason).toContain("too long"); + }); + }); + describe("scoreAnswerTargeting (structural per-intent targeting)", () => { const doseCase = { id: "t-dose", From 2750bc3322e30cf4a87b2591d5039bf07f44bb28 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Tue, 18 Aug 2026 22:49:15 +0800 Subject: [PATCH 2/2] docs(ledger): record the PR #2129 review (readability metric split) Immutable review record for the RAG eval scorer readability split, travelling with its owning product PR rather than a ledger-only branch. Co-Authored-By: Claude Opus 5 --- ...2a2d01d66efd20576ba87265a3855f2cd37fc64641934115b87.record.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/branch-review-records/2399b67bef6792a2d01d66efd20576ba87265a3855f2cd37fc64641934115b87.record.md diff --git a/docs/branch-review-records/2399b67bef6792a2d01d66efd20576ba87265a3855f2cd37fc64641934115b87.record.md b/docs/branch-review-records/2399b67bef6792a2d01d66efd20576ba87265a3855f2cd37fc64641934115b87.record.md new file mode 100644 index 0000000000..0ec886d02c --- /dev/null +++ b/docs/branch-review-records/2399b67bef6792a2d01d66efd20576ba87265a3855f2cd37fc64641934115b87.record.md @@ -0,0 +1 @@ +| 2026-08-18 | 2129 | eb9cb7481674c436464b0a81b560c3c79cc93501 | RAG eval scorer: readability metric split (fragmentation vs v19-derived length bound), eval_config_version v1->v2, HANDOVER S3 row | Approved — evaluation scorer only; no retrieval/ranking/selection/prompt change; RAG impact declared as no behaviour change; pr-policy evaluatePullRequestPolicy ok:true with 7/7 governance items | verify:pr-local heavy scope failed:(none) not-reached:(none) incl lint+typecheck+full unit suite+build; eval:rag:offline 26 suites/623 tests; check:rag:fixtures 36 golden/26 suites; eval:rag:adversarial:offline 24 cases/8 categories/6 canaries + 25 tests, 3 KNOWN_DIVERGENCES still pinned; vitest tests/rag-eval-cases.test.ts 32 passed (27+5 new, mutation-checked) |