diff --git a/docs/branch-review-records/2399b67bef6792a2d01d66efd20576ba87265a3855f2cd37fc64641934115b87.record.md b/docs/branch-review-records/2399b67bef6792a2d01d66efd20576ba87265a3855f2cd37fc64641934115b87.record.md new file mode 100644 index 0000000000..0ec886d02c --- /dev/null +++ b/docs/branch-review-records/2399b67bef6792a2d01d66efd20576ba87265a3855f2cd37fc64641934115b87.record.md @@ -0,0 +1 @@ +| 2026-08-18 | 2129 | eb9cb7481674c436464b0a81b560c3c79cc93501 | RAG eval scorer: readability metric split (fragmentation vs v19-derived length bound), eval_config_version v1->v2, HANDOVER S3 row | Approved — evaluation scorer only; no retrieval/ranking/selection/prompt change; RAG impact declared as no behaviour change; pr-policy evaluatePullRequestPolicy ok:true with 7/7 governance items | verify:pr-local heavy scope failed:(none) not-reached:(none) incl lint+typecheck+full unit suite+build; eval:rag:offline 26 suites/623 tests; check:rag:fixtures 36 golden/26 suites; eval:rag:adversarial:offline 24 cases/8 categories/6 canaries + 25 tests, 3 KNOWN_DIVERGENCES still pinned; vitest tests/rag-eval-cases.test.ts 32 passed (27+5 new, mutation-checked) | diff --git a/docs/rag-improvement/HANDOVER.md b/docs/rag-improvement/HANDOVER.md index b9093fc0c6..260855b4de 100644 --- a/docs/rag-improvement/HANDOVER.md +++ b/docs/rag-improvement/HANDOVER.md @@ -76,25 +76,25 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured ## 2. Status table — update in every programme PR -| Packet | Scope | Branch | PR | State | Canary / evidence refs | -| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | -| Handover | Multi-session handover + coordination | `claude/rag-plan-review-guide-vhrls9` | #1908 / #2024 | Merged 2026-08-13; coordination layer PR #2024 | docs-only | -| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | -| S1 | A1 phase 2: rung-1 verification-faithfulness fixes | `claude/s1-rag-mitigation-231-86c182` | #2022 | Merged 2026-08-17 (squash `2bd146eed`, landed by content) | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; canary pair run 31964560921 (baseline `8f8d111ab`) -> run 32025082010 (`2bd146eed`): recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44 (denominator reconciled by S5; see baseline-record §3); rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | -| S1b | A1 rung 3 (R1): pre-deadline strong routing for dosing class | `claude/s1b-rag-dosing-routing-6u1mik` | #2035 | Merged 2026-08-17 (PR #2035, merge `92f7618`) | canary pair pending: baseline run 32025082010 (`2bd146eed`) -> post-merge dispatch (owner-approved); offline 586/586 + verify:pr-local heavy scope green | -| S1c | A1 residuals R2 + R3: claim-support strictness | `claude/s1c-residuals-r2-r3-4pb1at` | #2052 | Merged 2026-08-17 (merge `b8e774bcd`; follow-up #2063 kept; follow-up #2065 reverted by PR #2088 after canary regression) | canary pair: baseline run 32049952885 -> post run 32052479537 (`084f63799`): recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44 | -| S1d | A1 final-gate gap recovery: hedged cited low-confidence fast answers must recover extractively, not collapse to a citation-free `provider_source_gap` | `claude/s1d-final-gate-gap-recovery-dxgrn2` | #2054 | Merged 2026-08-17 (merge `0bbd64fbc`); landed by content; canary pair green after the #2065 revert (PR #2088) | canary pair 32052479537 -> 32100681177 (`4ea310e48`) green: recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44. Interim post run 32097916649 (`9904fbda8`) was RED on `agitation-im-po-route-short-terms` — bisected live to PR #2065 (S1c follow-up condition-first regex), not S1d; reverted by PR #2088; the confirmation run is 32100681177 | -| G1 | Governance: provenance tag for document-summary rows (Option B) | `claude/g1-rag-document-context-qn9ubx` | #2053 | Merged 2026-08-17 (merge `125e98526`); rows #J912J9 / #0MSNT8 closed at reconcile | no canary (no behaviour change); `document_context` tag at `types.ts` + `rag-row-contracts.ts`, deriveConfidence pinned | -| S2 | A2 + A3: composition menu + moderate length | `claude/s2-rag-composition-7330b0` | #2097 | Merged 2026-08-18 (squash dda4956ff), landed by content; A2 and A3 together; prompt clinical-rag-answer-v19, schema v4, answerSections.maxItems 6 | offline: composition 9/9 + prompt pins 5/5, eval:rag:offline 26 suites / 623 tests, eval:rag:adversarial:offline 25/25 (3 divergences still pinned), rag.ts 4362/4362; canary pair 32100681177 (4ea310e48) -> 32111839806 GREEN (document/content recall 1.0/1.0, zero per-case rr regressions, answer gate clean; one non-blocking 20 s latency advisory on neuroleptic-side-effect-escalation); eval:answer-quality v18 (4ea310e48) vs v19 run 2026-08-18 neutral within nondeterminism (relevance 0.633->0.567 on two nondeterministic timeout/gap cases, targeting 0.409->0.429, readability at ceiling), owner blinded read pending. Note: the 220-word total-length readability ceiling in scoreAnswerQualityEvalCase is a known metric confound for A3 (baseline-record §4) | -| S2b | A3: moderate length (if separate review needed) | — | — | Not needed — A3 shipped inside S2 (combined diff stayed reviewable) | — | -| S3 | A4: follow-up suggestion refinement | `claude/s3-follow-up-suggestions-95e160` | #2108 | PR open 2026-08-18 (menu-derived, evidence-gated, already-answered-suppressed chips in src/lib/answer-follow-up.ts; no ClinicalDashboard change, no new module/field/render block); owner merges | offline only: answer-follow-up 27/27 (menu alignment across all 48 class×intent cells, evidence gate positive + discriminating negatives, suppression), answer-follow-up-chips DOM 6/6 (desktop + phone composer surfaces), answer-composition 9/9 unchanged; no canary — deterministic composition only, generation prompt untouched | -| S4 | B0: adversarial fixtures + baseline + register | `claude/packet-s4-adversarial-fixtures-5ho5tp` | #2036 | Merged 2026-08-17 (squash `f5b093291`) | Offline only: `check:rag:adversarial-fixtures` 24 cases / 8 categories / 6 canaries; `eval:rag:offline` 24 suites, 597 tests. Baseline `scripts/fixtures/rag-adversarial-baseline.v1.json` marks the three provider-backed gates `pending_owner_run` | -| S5 | B1+B2: telemetry assessment + offline harness | `claude/s5-rag-telemetry-harness-2wvis7` | #2056 | Merged 2026-08-17 (merge `093f9340c`); post-merge canary run 32049952885 | Offline only: `eval:rag:adversarial:offline` 25/25 (24 cases + canary-free report; 3 divergences pinned in `KNOWN_DIVERGENCES`); B1 gap = `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED` (default false); canary-absence tests green; 44/44 denominator reconciled | -| S6 | B3: Docling lab benchmark | `claude/packet-s6-docling-lab-d6foa6` | #2057 | Merged 2026-08-17 (merge `5a6418636`) | Offline only: `check:docling-lab` 36 fixtures / 10 hostile / 6 canaries + Gate B template valid; `verify:pr-local` heavy plan failed:(none); contract test 20/20; legacy smoke 46 docs, 10/10 hostile contained, canary-clean report. Verdict is a separate owner dispatch of `docling-lab.yml` | -| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | -| #212 T1–T3 | Runtime row contracts (rag.ts, rag-candidate-sources.ts, src/app/api) — sibling stream sharing `src/lib/rag/**` | — | #1946 / #1981 / #2023 | Merged (T3 squash `440a34f71` 2026-08-17) | see the #212 ledger row; RAG surface complete for the cast class | -| #212 T4 | Runtime row contracts: `worker/main.ts` (11 casts) — sibling stream | `claude/ledger-212-tranche-4-worker-q3y6i4` | #2037 | Merged 2026-08-17 (squash `1726537b7`); #212 closed by reconcile PR #2045 | Governance Preflight complete; audit: 1 inbound cast (claim rows, per-row fail-soft) + 2 read-back param casts contracted, 9 outbound/interop left; closes #212 (inbox `done` queued in the PR) | +| Packet | Scope | Branch | PR | State | Canary / evidence refs | +| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | +| Handover | Multi-session handover + coordination | `claude/rag-plan-review-guide-vhrls9` | #1908 / #2024 | Merged 2026-08-13; coordination layer PR #2024 | docs-only | +| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | +| S1 | A1 phase 2: rung-1 verification-faithfulness fixes | `claude/s1-rag-mitigation-231-86c182` | #2022 | Merged 2026-08-17 (squash `2bd146eed`, landed by content) | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; canary pair run 31964560921 (baseline `8f8d111ab`) -> run 32025082010 (`2bd146eed`): recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44 (denominator reconciled by S5; see baseline-record §3); rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | +| S1b | A1 rung 3 (R1): pre-deadline strong routing for dosing class | `claude/s1b-rag-dosing-routing-6u1mik` | #2035 | Merged 2026-08-17 (PR #2035, merge `92f7618`) | canary pair pending: baseline run 32025082010 (`2bd146eed`) -> post-merge dispatch (owner-approved); offline 586/586 + verify:pr-local heavy scope green | +| S1c | A1 residuals R2 + R3: claim-support strictness | `claude/s1c-residuals-r2-r3-4pb1at` | #2052 | Merged 2026-08-17 (merge `b8e774bcd`; follow-up #2063 kept; follow-up #2065 reverted by PR #2088 after canary regression) | canary pair: baseline run 32049952885 -> post run 32052479537 (`084f63799`): recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44 | +| S1d | A1 final-gate gap recovery: hedged cited low-confidence fast answers must recover extractively, not collapse to a citation-free `provider_source_gap` | `claude/s1d-final-gate-gap-recovery-dxgrn2` | #2054 | Merged 2026-08-17 (merge `0bbd64fbc`); landed by content; canary pair green after the #2065 revert (PR #2088) | canary pair 32052479537 -> 32100681177 (`4ea310e48`) green: recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44. Interim post run 32097916649 (`9904fbda8`) was RED on `agitation-im-po-route-short-terms` — bisected live to PR #2065 (S1c follow-up condition-first regex), not S1d; reverted by PR #2088; the confirmation run is 32100681177 | +| G1 | Governance: provenance tag for document-summary rows (Option B) | `claude/g1-rag-document-context-qn9ubx` | #2053 | Merged 2026-08-17 (merge `125e98526`); rows #J912J9 / #0MSNT8 closed at reconcile | no canary (no behaviour change); `document_context` tag at `types.ts` + `rag-row-contracts.ts`, deriveConfidence pinned | +| S2 | A2 + A3: composition menu + moderate length | `claude/s2-rag-composition-7330b0` | #2097 | Merged 2026-08-18 (squash dda4956ff), landed by content; A2 and A3 together; prompt clinical-rag-answer-v19, schema v4, answerSections.maxItems 6 | offline: composition 9/9 + prompt pins 5/5, eval:rag:offline 26 suites / 623 tests, eval:rag:adversarial:offline 25/25 (3 divergences still pinned), rag.ts 4362/4362; canary pair 32100681177 (4ea310e48) -> 32111839806 GREEN (document/content recall 1.0/1.0, zero per-case rr regressions, answer gate clean; one non-blocking 20 s latency advisory on neuroleptic-side-effect-escalation); eval:answer-quality v18 (4ea310e48) vs v19 run 2026-08-18 neutral within nondeterminism (relevance 0.633->0.567 on two nondeterministic timeout/gap cases, targeting 0.409->0.429, readability at ceiling), owner blinded read pending. Note: the 220-word total-length readability ceiling in scoreAnswerQualityEvalCase is a known metric confound for A3 (baseline-record §4) | +| S2b | A3: moderate length (if separate review needed) | — | — | Not needed — A3 shipped inside S2 (combined diff stayed reviewable) | — | +| S3 | A4: follow-up suggestion refinement | `claude/s3-follow-up-suggestions-95e160` | #2108 | Merged 2026-08-18 (squash 511d22f4d), landed by content; Track A complete | offline only: answer-follow-up 27/27 (menu alignment across all 48 class×intent cells, evidence gate positive + discriminating negatives, suppression), answer-follow-up-chips DOM 6/6 (desktop + phone composer surfaces), answer-composition 9/9 unchanged; no canary — deterministic composition only, generation prompt untouched | +| S4 | B0: adversarial fixtures + baseline + register | `claude/packet-s4-adversarial-fixtures-5ho5tp` | #2036 | Merged 2026-08-17 (squash `f5b093291`) | Offline only: `check:rag:adversarial-fixtures` 24 cases / 8 categories / 6 canaries; `eval:rag:offline` 24 suites, 597 tests. Baseline `scripts/fixtures/rag-adversarial-baseline.v1.json` marks the three provider-backed gates `pending_owner_run` | +| S5 | B1+B2: telemetry assessment + offline harness | `claude/s5-rag-telemetry-harness-2wvis7` | #2056 | Merged 2026-08-17 (merge `093f9340c`); post-merge canary run 32049952885 | Offline only: `eval:rag:adversarial:offline` 25/25 (24 cases + canary-free report; 3 divergences pinned in `KNOWN_DIVERGENCES`); B1 gap = `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED` (default false); canary-absence tests green; 44/44 denominator reconciled | +| S6 | B3: Docling lab benchmark | `claude/packet-s6-docling-lab-d6foa6` | #2057 | Merged 2026-08-17 (merge `5a6418636`) | Offline only: `check:docling-lab` 36 fixtures / 10 hostile / 6 canaries + Gate B template valid; `verify:pr-local` heavy plan failed:(none); contract test 20/20; legacy smoke 46 docs, 10/10 hostile contained, canary-clean report. Verdict is a separate owner dispatch of `docling-lab.yml` | +| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | +| #212 T1–T3 | Runtime row contracts (rag.ts, rag-candidate-sources.ts, src/app/api) — sibling stream sharing `src/lib/rag/**` | — | #1946 / #1981 / #2023 | Merged (T3 squash `440a34f71` 2026-08-17) | see the #212 ledger row; RAG surface complete for the cast class | +| #212 T4 | Runtime row contracts: `worker/main.ts` (11 casts) — sibling stream | `claude/ledger-212-tranche-4-worker-q3y6i4` | #2037 | Merged 2026-08-17 (squash `1726537b7`); #212 closed by reconcile PR #2045 | Governance Preflight complete; audit: 1 inbound cast (claim rows, per-row fail-soft) + 2 read-back param casts contracted, 9 outbound/interop left; closes #212 (inbox `done` queued in the PR) | Update rule: the session that opens a packet's PR edits its row (branch, PR number, state) in the same PR. A later session updating another packet may also correct stale rows diff --git a/docs/rag-improvement/baseline-record.md b/docs/rag-improvement/baseline-record.md index 3536270a33..0b4ea51987 100644 --- a/docs/rag-improvement/baseline-record.md +++ b/docs/rag-improvement/baseline-record.md @@ -18,7 +18,7 @@ so two reports can be compared without guessing what changed between them. | --------------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------- | | `commit_sha` | `b7aa925f0ae19e89a9f0acf842b4a80d84083fb5` | `git rev-parse HEAD` of the evaluated tree. Full 40 characters, enforced. | | `dataset_version` | `rag-adversarial-cases.v1` | The fixture dataset's own `datasetVersion`; cross-checked against the file. | -| `eval_config_version` | `rag-eval-config-v1` | Bumped by hand whenever a case list, threshold, or gate semantic changes. | +| `eval_config_version` | `rag-eval-config-v2` | Bumped by hand whenever a case list, threshold, or gate semantic changes. | | `model_version` | `answer=gpt-5.6-terra; fast=gpt-5.6-terra; strong=gpt-5.6-sol` | The resolved answer-model defaults in `src/lib/env.ts`. | | `embedding_version` | `text-embedding-3-small@1536` | `OPENAI_EMBEDDING_MODEL` and `EMBEDDING_DIMENSIONS` in `src/lib/env.ts`. | | `index_version` | `20260818090000_schema_drift_snapshot_history_probe` | The latest applied migration — the index shape the retrieval RPCs run on. | @@ -74,12 +74,44 @@ commit `b7aa925f0ae19e89a9f0acf842b4a80d84083fb5`. Offline gates were re-run at the three provider-backed gates stay `pending_owner_run`, carrying run `32100681177` at `4ea310e48` (prompt v18) as `priorRun` — that run is the baseline half of the S2 canary pair. -One caveat travels with the `answer_quality` gate: `scoreAnswerQualityEvalCase` -(`src/lib/rag/rag-eval-cases.ts`) scores readability over the answer **plus every section -body** with a 220-word ceiling. The S2 targets can exceed that by design, so a readability=0 -flag caused only by total length is a metric artefact to adjudicate (raise the ceiling with an -`eval_config_version` bump, or accept), not evidence of a regression. The scorer was left -untouched in S2 so the before/after comparison runs under one definition. +One caveat travelled with the `answer_quality` gate at `rag-eval-config-v1`: +`scoreAnswerQualityEvalCase` (`src/lib/rag/rag-eval-cases.ts`) scored readability over the +answer **plus every section body** as a single boolean combining a fragmentation regex with a +flat 220-word ceiling, under one reason string ("fragmented or too long"). The S2 targets can +exceed 220 by design, so a readability=0 flag caused only by total length was a metric artefact +to adjudicate, not evidence of a regression — and it was not separable from the fragmentation +regression the metric exists to catch. The scorer was deliberately left untouched in S2 so the +before/after comparison ran under one definition. + +## 4a. Readability metric split (2026-08-18, `rag-eval-config-v2`) + +That caveat is now resolved, and the resolution is the reason this record reads +`rag-eval-config-v2`. `scoreAnswerQualityEvalCase` keeps the five metric keys — `readability` +still reports as one score, because `AnswerQualityMetric` is a closed union consumed by +`scripts/eval-answer-quality.ts` as a total `Record` — but it now +evaluates two independent checks under that key and reports whichever failed: + +- **Fragmentation** — the existing `fragmentPattern`, unchanged in both pattern and effect. +- **Length** — `>= 5` words (the empty/stub floor, unchanged) and `<= 900` words. + +The 900-word ceiling is derived from the v19 contract rather than raised by judgement: the +answer field's stated upper target is 110 words (`rag-answer-instructions.ts`), sections are +capped at 6 (`answerSections.maxItems` in `rag.ts`, matching the prompt's "three to six"), and +each section can carry a 48-character heading plus a 600-character body (both schema maxima in +`rag.ts`). At a deliberately low 5 characters per word — chosen so the conversion overstates the +word ceiling and the bound can never fail a well-formed answer — that is +110 + 6 x (648 / 5) = 887.6, rounded up to 900. + +This is a contract ceiling, not a style ceiling: conciseness is enforced by the prompt and +measured by `scoreAnswerTargeting`. An answer above 900 words could not have come from a +schema-conformant generation, so the bound still catches runaway duplication and the +deterministic composition paths (`rag-extractive-answer.ts`, `rag-comparison.ts`) that build a +`RagAnswer` in code without the JSON schema. + +Consequence for comparisons: a `readability` rate recorded under `rag-eval-config-v1` is not +comparable to one recorded under `v2`. No retrieval, ranking, selection, or generation +behaviour changed — this is an evaluation-scorer change only, so it carries no canary +requirement of its own. ## 5. Related diff --git a/scripts/fixtures/rag-adversarial-baseline.v1.json b/scripts/fixtures/rag-adversarial-baseline.v1.json index 7a1e9bf7ca..5c797ccd1f 100644 --- a/scripts/fixtures/rag-adversarial-baseline.v1.json +++ b/scripts/fixtures/rag-adversarial-baseline.v1.json @@ -6,7 +6,7 @@ "reportKey": { "commit_sha": "b7aa925f0ae19e89a9f0acf842b4a80d84083fb5", "dataset_version": "rag-adversarial-cases.v1", - "eval_config_version": "rag-eval-config-v1", + "eval_config_version": "rag-eval-config-v2", "model_version": "answer=gpt-5.6-terra; fast=gpt-5.6-terra; strong=gpt-5.6-sol", "embedding_version": "text-embedding-3-small@1536", "index_version": "20260818090000_schema_drift_snapshot_history_probe" @@ -31,7 +31,7 @@ "caseCount": 30, "status": "pending_owner_run", "blockedReason": "npm run eval:answer-quality over answerQualityEvalCases is a provider-backed OpenAI/Supabase evaluation requiring explicit owner approval per run. Requested (not executed) by packet S2 as the Gate E before/after comparison.", - "priorRun": "No before/after comparison has been recorded. Note for the S2 comparison: scoreAnswerQualityEvalCase's readability metric caps answer + section text at 220 words (src/lib/rag/rag-eval-cases.ts), which the S2 length targets can exceed by design; readability=0 flags driven only by total length are a metric artefact to adjudicate, not a regression." + "priorRun": "No before/after comparison has been recorded. The 220-word readability confound noted at eval_config_version rag-eval-config-v1 is resolved: scoreAnswerQualityEvalCase now scores fragmentation and length as two independent checks with separate reasons, and the length ceiling is derived from the v19 contract (110-word answer target + 6 sections at the schema heading/body maxima) rather than the pre-S2 flat 220. A readability=0 flag now names which check failed, so length and fragmentation no longer have to be adjudicated apart by hand. This gate semantic change is why the key reads rag-eval-config-v2; the first run under v2 has no v1-comparable readability rate." }, { "id": "offline_contract", diff --git a/src/lib/rag/rag-eval-cases.ts b/src/lib/rag/rag-eval-cases.ts index 34732e9be1..5279881320 100644 --- a/src/lib/rag/rag-eval-cases.ts +++ b/src/lib/rag/rag-eval-cases.ts @@ -87,7 +87,7 @@ export type AnswerQualityMetricScore = { export const answerQualityMetricLabels: Record = { relevance: "Answer addresses the requested entity and task.", - readability: "Answer is grammatical, concise, and not fragment-like.", + readability: "Answer is not fragment-like, and its length is within the v19 answer+sections contract.", artifact_leaks: "Answer avoids backend, admin, provenance, and template wording.", intent_coverage: "Answer includes the action, dose, schedule, document list, or gap required by intent.", fail_closed: "Unsupported or weakly supported answers refuse specifically instead of guessing.", @@ -138,6 +138,39 @@ function citesOrNamesExpectedDocument(testCase: AnswerQualityEvalCase, answer: R ); } +// Readability is scored as TWO independent checks that share one metric key, because +// `AnswerQualityMetric` is a closed union consumed by `scripts/eval-answer-quality.ts` as a total +// `Record` — a sixth key would break that aggregation (and the metric-key +// pins in tests/rag-eval-cases.test.ts) without adding evaluative power. Each check therefore reports +// its own reason so a failure names which contract it broke. +// +// Check 1 — FRAGMENTATION (unchanged): the regression this metric exists to catch. Answer text that +// carries OCR/table run-together artefacts. +// +// Check 2 — LENGTH: a floor for empty/stub answers, and a ceiling derived from what prompt +// `clinical-rag-answer-v19` can legitimately emit. Before packet S2 the ceiling was a flat 220 words +// over `answerTextForQuality` (answer + every section heading and body). S2 (#2097, `dda4956ff`) moved +// the answer field to 60–110 words and sections to three-to-six, so a correctly shaped v19 answer can +// exceed 220 — at which point one conflated boolean could no longer separate "longer by design" from +// "fragmented", the regression it is here for. +// +// Derivation of ANSWER_MAX_WORDS, from S2's own targets and the enforced response schema: +// answer field 110 words `rag-answer-instructions.ts` upper target ("about 60-110 words") +// sections 6 `answerSections.maxItems` in `rag.ts` (= the prompt's "three to six") +// per section 648 chars `heading` maxLength 48 + `body` maxLength 600, both in `rag.ts` +// chars per word 5 deliberately low, so the char->word conversion OVERSTATES the word +// ceiling; this bound must never fail a well-formed answer. +// => 110 + 6 * (648 / 5) = 110 + 777.6 -> 900 words (rounded up) +// +// This is a CONTRACT ceiling, not a style ceiling: conciseness is enforced by the prompt itself and +// measured by `scoreAnswerTargeting`. Exceeding 900 words means the answer could not have come from a +// schema-conformant generation — runaway duplication, or a deterministic composition path +// (`rag-extractive-answer.ts`, `rag-comparison.ts`) that builds a RagAnswer in code without the JSON +// schema. Both are real defects. Raising or lowering either bound is a gate-semantic change and needs +// an `eval_config_version` bump in `scripts/fixtures/rag-adversarial-baseline.v1.json`. +const ANSWER_MIN_WORDS = 5; +const ANSWER_MAX_WORDS = 900; + export function scoreAnswerQualityEvalCase(testCase: AnswerQualityEvalCase, answer: RagAnswer) { const text = answerTextForQuality(answer); const sourceBackedReviewStub = isSourceBackedReviewFallback(answer); @@ -160,7 +193,14 @@ export function scoreAnswerQualityEvalCase(testCase: AnswerQualityEvalCase, answ expectedFileCoverage(testCase.expectedFiles, answer.citations, answer.citations.length).anyHit : answer.grounded && answer.citations.length >= testCase.minCitations && expectedClassOk : unsupported; - const readabilityOk = wordCount >= 5 && wordCount <= 220 && !fragmentPattern.test(text); + const fragmentedText = fragmentPattern.test(text); + const lengthOk = wordCount >= ANSWER_MIN_WORDS && wordCount <= ANSWER_MAX_WORDS; + const readabilityOk = !fragmentedText && lengthOk; + const readabilityReasons = [ + ...(fragmentedText ? ["fragmented"] : []), + ...(wordCount < ANSWER_MIN_WORDS ? [`too short (${wordCount} words < ${ANSWER_MIN_WORDS})`] : []), + ...(wordCount > ANSWER_MAX_WORDS ? [`too long (${wordCount} words > ${ANSWER_MAX_WORDS})`] : []), + ]; const artifactOk = !artifactPattern.test(text) && containsNone(text, testCase.mustNotContain); const intentOk = !sourceBackedReviewStub && containsAny(text, testCase.mustContainAny); const failClosedOk = @@ -171,7 +211,7 @@ export function scoreAnswerQualityEvalCase(testCase: AnswerQualityEvalCase, answ { metric: "readability", score: readabilityOk ? 1 : 0, - reason: readabilityOk ? "readable" : "fragmented or too long", + reason: readabilityOk ? "readable" : readabilityReasons.join("; "), }, { metric: "artifact_leaks", score: artifactOk ? 1 : 0, reason: artifactOk ? "clean" : "artifact wording present" }, { diff --git a/tests/rag-eval-cases.test.ts b/tests/rag-eval-cases.test.ts index 423dd038b2..8e438c58da 100644 --- a/tests/rag-eval-cases.test.ts +++ b/tests/rag-eval-cases.test.ts @@ -326,6 +326,96 @@ describe("captured RAG eval cases", () => { expect(scores.every((score) => score.score === 1)).toBe(true); }); + describe("readability: fragmentation and length are scored independently", () => { + // Packet S2 (#2097, dda4956ff, prompt clinical-rag-answer-v19) moved the answer field to + // 60-110 words and sections to three-to-six. The old scorer conflated fragmentation and a flat + // 220-word ceiling into one boolean with one reason ("fragmented or too long"), so a correctly + // shaped v19 answer could fail for length and be indistinguishable from the fragmentation + // regression the metric exists to catch. These tests pin the split. + const qualityCase = answerQualityEvalCases.find((item) => item.id === "quality-discharge-documentation")!; + + function readabilityOf(text: string, sections: RagAnswer["answerSections"] = []) { + const answer = { + answer: text, + grounded: true, + confidence: "high", + citations: [], + sources: [], + routingMode: "fast", + queryClass: "document_lookup", + answerSections: sections, + } satisfies RagAnswer; + return scoreAnswerQualityEvalCase(qualityCase, answer).find((score) => score.metric === "readability")!; + } + + // Clean clinical prose with no fragmentation artefacts: no run-together digit lists, no + // "? monitoring" break, no glued-together heading token. + const cleanSentence = + "Review the admission record and confirm the responsible consultant has documented the current plan. "; + + it("scores a long but clean v19-shaped answer as readable", () => { + // ~110-word answer plus six sections, the maximum shape prompt v19 asks for. Comfortably over + // the retired 220-word ceiling, comfortably under the derived 900-word contract ceiling. + const answerField = cleanSentence.repeat(8); + const sections = Array.from({ length: 6 }, (_, index) => ({ + heading: `Section ${String.fromCharCode(65 + index)}`, + kind: "required_actions" as const, + supportLevel: "direct" as const, + body: cleanSentence.repeat(5), + citation_chunk_ids: [], + })) satisfies RagAnswer["answerSections"]; + + const score = readabilityOf(answerField, sections); + const wordCount = [answerField, ...sections.map((section) => `${section.heading}: ${section.body}`)] + .join(" ") + .split(/\s+/) + .filter(Boolean).length; + + // Guard the guard: this fixture must actually exercise the regression it claims to. + expect(wordCount).toBeGreaterThan(220); + expect(wordCount).toBeLessThan(900); + expect(score.score).toBe(1); + expect(score.reason).toBe("readable"); + }); + + it("still fails a genuinely fragmented answer, and names fragmentation as the reason", () => { + const score = readabilityOf( + "Clozapine monitoring anyMANAGEMENT of the neutrophil result follows the escalation pathway.", + ); + + expect(score.score).toBe(0); + expect(score.reason).toContain("fragmented"); + // Distinguishable from the length failure by the reason alone. + expect(score.reason).not.toContain("too long"); + expect(score.reason).not.toContain("too short"); + }); + + it("still fails a runaway-length answer, and names length as the reason", () => { + const score = readabilityOf(cleanSentence.repeat(80)); + + expect(score.score).toBe(0); + expect(score.reason).toContain("too long"); + // Distinguishable from the fragmentation failure by the reason alone. + expect(score.reason).not.toContain("fragmented"); + }); + + it("still fails an empty-stub answer as too short", () => { + const score = readabilityOf("No source."); + + expect(score.score).toBe(0); + expect(score.reason).toContain("too short"); + expect(score.reason).not.toContain("fragmented"); + }); + + it("reports both reasons when an answer is fragmented AND over length", () => { + const score = readabilityOf(`anyMANAGEMENT ${cleanSentence.repeat(80)}`); + + expect(score.score).toBe(0); + expect(score.reason).toContain("fragmented"); + expect(score.reason).toContain("too long"); + }); + }); + describe("scoreAnswerTargeting (structural per-intent targeting)", () => { const doseCase = { id: "t-dose",