From 6ddd45e5fd0725638c55684ca85833ddf78560c4 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Mon, 17 Aug 2026 16:38:22 +0800 Subject: [PATCH 1/4] fix(rag): stop verification rejecting verbatim-faithful answers (#231 S1) Two measured text-normalization artifacts made the deterministic quality gates discard answers whose figures were present verbatim in their cited chunks, degrading dosing queries to source-only: - Markdown emphasis (including the pipeline's own high-yield bolding) split clinical value atoms: "**200 mg**/day" extracted a bare 200mg atom while the cited source's "200 mg/day" carried the per-day denominator, so the exact atom-key match failed. Atom and numeric token extraction now fold emphasis markers first. - Claim-support segmentation split source sentences at PDF visual line wraps, so no single segment carried every atom of a claim restating a wrapped sentence. Segmentation now rejoins bounded visual wraps, joining only visible sentence continuations (lowercase/digit/paren starts) so separate capitalized source lines cannot manufacture support. Evidence: 8 pre-fix live probes on healthy latency (post index restore) attributed fallbacks to numeric_faithfulness_gap (3), claim_support_ high_risk_gap (1), provider_timeout (3); both artifacts reproduced deterministically offline. Post-fix, the sertraline maximum-dose probe returns grounded model_synthesis. Residuals (strong-retry timeout inside the fast route budget; directive-normativity and topic-overlap strictness) are recorded in the PR body for follow-up. Co-Authored-By: Claude Fable 5 --- docs/rag-improvement/HANDOVER.md | 32 ++++++++++++--------- src/lib/answer-verification.ts | 17 +++++++++-- src/lib/rag/rag-claim-support.ts | 12 +++++++- src/lib/rag/rag-source-segmentation.ts | 15 +++++++++- tests/answer-verification.test.ts | 40 ++++++++++++++++++++++++++ tests/rag-claim-support.test.ts | 26 +++++++++++++++++ 6 files changed, 125 insertions(+), 17 deletions(-) diff --git a/docs/rag-improvement/HANDOVER.md b/docs/rag-improvement/HANDOVER.md index 954600281f..212644a22b 100644 --- a/docs/rag-improvement/HANDOVER.md +++ b/docs/rag-improvement/HANDOVER.md @@ -39,19 +39,19 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured ## 2. Status table — update in every programme PR -| Packet | Scope | Branch | PR | State | Canary / evidence refs | -| -------- | ----------------------------------------------- | ------------------------------------------------ | ----- | ---------------------- | ---------------------- | -| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | -| Handover | Multi-session handover | `claude/rag-plan-review-guide-vhrls9` | #1908 | Open — this PR | docs-only | -| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | -| S1 | A1 phase 2: evidence-chosen mitigation | `claude/rag-a1-mitigation-` | — | Not started | — | -| S2 | A2: composition menu | `claude/rag-a2-composition-` | — | Blocked on S1 evidence | — | -| S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-` | — | Blocked on S2 | — | -| S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-` | — | Blocked on S2 + S2b | — | -| S4 | B0: adversarial fixtures + baseline + register | `claude/rag-b0-adversarial-fixtures-` | — | Ready (parallel-safe) | — | -| S5 | B1+B2: telemetry assessment + offline harness | `claude/rag-b1-b2-harness-` | — | Blocked on S4 | — | -| S6 | B3: Docling lab benchmark | `claude/rag-b3-docling-lab-` | — | Blocked on S4 | — | -| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | +| Packet | Scope | Branch | PR | State | Canary / evidence refs | +| -------- | ----------------------------------------------- | ------------------------------------------------ | ----- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | +| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | +| Handover | Multi-session handover | `claude/rag-plan-review-guide-vhrls9` | #1908 | Open — this PR | docs-only | +| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | +| S1 | A1 phase 2: evidence-chosen mitigation | `claude/s1-rag-mitigation-231-86c182` | TBD | PR open — rung 1 verification-faithfulness fixes | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | +| S2 | A2: composition menu | `claude/rag-a2-composition-` | — | Blocked on S1 evidence | — | +| S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-` | — | Blocked on S2 | — | +| S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-` | — | Blocked on S2 + S2b | — | +| S4 | B0: adversarial fixtures + baseline + register | `claude/rag-b0-adversarial-fixtures-` | — | Ready (parallel-safe) | — | +| S5 | B1+B2: telemetry assessment + offline harness | `claude/rag-b1-b2-harness-` | — | Blocked on S4 | — | +| S6 | B3: Docling lab benchmark | `claude/rag-b3-docling-lab-` | — | Blocked on S4 | — | +| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | Update rule: the session that opens a packet's PR edits its row (branch, PR number, state) in the same PR. A later session updating another packet may also correct stale rows @@ -65,6 +65,12 @@ owner merges) and never at watching CI. ### S1 — A1 phase 2: choose and implement the mitigation from evidence (`#231`) +- **Evidence update (2026-08-14 incident):** the mitigation ladder's rung 2 (pre-generation + latency) has a live measurement — `supabase_rpc_latency_ms` 31,610 ms from two dropped + trigram indexes, restored the same day (now 1,535 ms text / 8,519 ms hybrid). See + `docs/audit/live-drift-forensics-2026-08.md` (Phases 1.3 and 5) for the before/after + probes; any rung-2 reasoning must start from that file, not from the pre-incident + assumption that retrieval latency was healthy. - **Precondition:** PR #1899 merged; its diagnostics available. If live `generation_quality_gate:*` distributions exist in `rag_queries.metadata`, ask the owner for the aggregate counts (reading live Supabase is provider-gated — do not query it diff --git a/src/lib/answer-verification.ts b/src/lib/answer-verification.ts index b87bd381c4..5d1dfe601f 100644 --- a/src/lib/answer-verification.ts +++ b/src/lib/answer-verification.ts @@ -62,6 +62,19 @@ function foldSuperscripts(text: string): string { return text.replace(/[⁰¹²³⁴⁵⁶⁷⁸⁹]/g, (ch) => SUPERSCRIPT_DIGITS[ch] ?? ch); } +// Markdown emphasis markers reach answer prose from two writers — the model itself and the +// pipeline's own high-yield bolding (boldRagAnswerHighYieldText in rag.ts), which runs BEFORE +// numeric verification. A marker between a quantity and its per-time denominator makes +// `**200 mg**/day` extract a bare `200mg` atom while the cited source's `200 mg/day` extracts +// `200mg/day`, so a verbatim-faithful figure fails the exact atom-key match and the whole +// dosing answer is discarded (#231, measured live 2026-08-17). Fold the markers (asterisks, +// backticks, heading hashes, double underscores) to nothing before matching, mirroring the +// claim-splitting `cleanText` in rag-claim-support.ts; single underscores are kept so +// identifier-like tokens are not merged. +function foldMarkdownEmphasis(text: string): string { + return /[*`#_]/.test(text) ? text.replace(/[*`#]+|_{2,}/g, "") : text; +} + function normalizeNumericToken(raw: string): string { return foldSuperscripts(raw) .toLowerCase() @@ -190,7 +203,7 @@ function canonicalFrequency(count: string | undefined, phrase: string) { /** Extract meaning-preserving clinical value atoms for exact evidence matching. */ export function extractClinicalValueAtoms(text: string): ClinicalValueAtom[] { if (!text) return []; - const normalized = foldSuperscripts(text).normalize("NFKC"); + const normalized = foldSuperscripts(foldMarkdownEmphasis(text)).normalize("NFKC"); const atoms: ClinicalValueAtom[] = []; const occupied: Array<[number, number]> = []; @@ -1257,7 +1270,7 @@ function clinicalValueAtomDisplay(atom: ClinicalValueAtom) { export function extractNumericTokens(text: string): string[] { if (!text) return []; - const folded = foldSuperscripts(text); + const folded = foldSuperscripts(foldMarkdownEmphasis(text)); const tokens = new Set(); for (const match of folded.matchAll(NUMERIC_TOKEN_PATTERN)) { const normalized = normalizeNumericToken(match[0]); diff --git a/src/lib/rag/rag-claim-support.ts b/src/lib/rag/rag-claim-support.ts index 20d238069f..714848fb84 100644 --- a/src/lib/rag/rag-claim-support.ts +++ b/src/lib/rag/rag-claim-support.ts @@ -505,9 +505,19 @@ function usesSourceBoundComparisonReflow(source: SearchResult, claim: string) { function sourceEvidenceClaimSegments(source: SearchResult, claim: string) { const split = (value: string | null | undefined, context?: string | null, reflowVisualLines = false) => { const rawValue = reflowWrappedAgitationDoseLines(reflowWrappedEscalationRecipientLines(value ?? "")); + // Always rejoin PDF visual line wraps before sentence-splitting. Splitting raw extracted + // text on `\n+` fragments a source sentence at its hard wrap ("…for adults is 500 mg nocte + // and for patients over 65 years it\nis 250 mg nocte"), so no single segment carries every + // atom of a claim restating that sentence — a verbatim-faithful high-risk claim then reads + // as unsupported and the whole answer degrades to source-only (#231, measured live + // 2026-08-17). Blank lines, bullets, terminal punctuation, colons, and numbered headings + // stay hard boundaries, and the general path joins only visible sentence continuations + // (lowercase/digit/parenthesis starts) so separate capitalized source lines cannot + // manufacture support. The comparison-reflow path keeps its historical aggressive join + // plus low-yield noise stripping for its two known policy documents, unchanged. const blocks = reflowVisualLines ? reflowBoundedSourceLines(sourceTextForClinicalProsePreservingBreaks(rawValue)) - : [rawValue]; + : reflowBoundedSourceLines(rawValue, { requireContinuationStart: true }); return blocks.flatMap((block) => { const sharedConditional = block.match(/^\s*((?:when|whenever|if|unless)\b[^,;.!?]+),?/i)?.[1]; return block diff --git a/src/lib/rag/rag-source-segmentation.ts b/src/lib/rag/rag-source-segmentation.ts index 99c7b79a21..6465224163 100644 --- a/src/lib/rag/rag-source-segmentation.ts +++ b/src/lib/rag/rag-source-segmentation.ts @@ -85,8 +85,20 @@ export function reflowWrappedAgitationDoseLines(value: string) { * Reflow OCR/PDF visual line wraps without joining across semantic source * boundaries. Blank lines, bullets, terminal punctuation, colons, and numbered * headings remain hard boundaries. + * + * With `requireContinuationStart`, a line additionally joins the previous one + * only when it visibly continues its sentence (starts with a lowercase letter, + * digit, or opening parenthesis). Capitalized lines then start a new block: + * joining separate capitalized items ("Stop clozapine" / "Starting dose + * 12.5 mg") would manufacture claim support from unrelated source lines. The + * default keeps the historical aggressive join for the sanitized policy-source + * and table-row callers, whose known texts wrap across capitalized proper + * nouns ("…Emergency\nDepartment (AHS-ED)…"). */ -export function reflowBoundedSourceLines(value: string) { +const wrapContinuationLineStartPattern = /^[a-z0-9(]/; + +export function reflowBoundedSourceLines(value: string, options?: { requireContinuationStart?: boolean }) { + const requireContinuationStart = options?.requireContinuationStart ?? false; const blocks: string[] = []; let wrappedLines: string[] = []; const flush = () => { @@ -107,6 +119,7 @@ export function reflowBoundedSourceLines(value: string) { continue; } if (sourceBulletLinePattern.test(line)) flush(); + if (requireContinuationStart && wrappedLines.length > 0 && !wrapContinuationLineStartPattern.test(line)) flush(); wrappedLines.push(line); if (/[.!?]\s*$/.test(line) || /:\s*$/.test(line)) flush(); } diff --git a/tests/answer-verification.test.ts b/tests/answer-verification.test.ts index 1dc3dd8706..39b674e04e 100644 --- a/tests/answer-verification.test.ts +++ b/tests/answer-verification.test.ts @@ -28,6 +28,46 @@ function source(overrides: Partial = {}): SearchResult { } describe("answer-verification (GEN-C2 / GEN-H2)", () => { + it("extracts the same clinical value atom through markdown emphasis markers (#231)", () => { + // Measured live 2026-08-17: the pipeline's own high-yield bolding produced + // "**200 mg**/day", which extracted a bare 200mg atom while the cited source's + // "200 mg/day" carried denominatorTime — a verbatim-faithful maximum-dose answer + // then failed the exact atom-key match and degraded to source-only. + const [plain] = extractClinicalValueAtoms("adjust dose according to response, maximum 200 mg/day"); + const [bolded] = extractClinicalValueAtoms("The maximum recommended dose of sertraline is **200 mg**/day."); + expect(plain?.denominatorTime).toBe("day"); + expect(bolded?.denominatorTime).toBe("day"); + expect(bolded?.canonicalValue).toBe("200"); + expect(bolded?.canonicalUnit).toBe("mg"); + }); + + it("verifies bolded figures against unformatted source text (#231)", () => { + const doseSource = source({ + id: "chunk-dose", + content: "sertraline: 50 mg orally once daily initially, adjust dose according to response, maximum 200 mg/day", + }); + const verification = verifyAnswerNumbers( + "The maximum recommended dose of sertraline is **200 mg**/day.", + [{ chunk_id: "chunk-dose" }], + [doseSource], + ); + expect(verification.unverifiedTokens).toEqual([]); + expect(verification.hasUnverifiedNumbers).toBe(false); + }); + + it("still fails a bolded figure absent from the cited source", () => { + const doseSource = source({ + id: "chunk-dose", + content: "sertraline: 50 mg orally once daily initially, adjust dose according to response, maximum 200 mg/day", + }); + const verification = verifyAnswerNumbers( + "The maximum recommended dose of sertraline is **300 mg**/day.", + [{ chunk_id: "chunk-dose" }], + [doseSource], + ); + expect(verification.hasUnverifiedNumbers).toBe(true); + }); + it("detects overlapping labelled score bands without inferring corrected cutoffs", () => { const conflicts = detectLabelledNumericBandConflicts( "Escalate when the LUNSERS score is medium (41-89), high (81-100), or very high (>101).", diff --git a/tests/rag-claim-support.test.ts b/tests/rag-claim-support.test.ts index 9b478f5041..2ebcb21d00 100644 --- a/tests/rag-claim-support.test.ts +++ b/tests/rag-claim-support.test.ts @@ -106,6 +106,32 @@ describe("deterministic claim support", () => { ]); }); + it("supports a claim restating a source sentence split by a PDF visual line wrap (#231)", () => { + // Measured live 2026-08-17: the EMHS lithium guideline's starting-dose bullet wraps + // mid-sentence, so segment-splitting on raw newlines separated 500 mg/over-65 from + // 250 mg and a verbatim-faithful high-risk claim read as unsupported. + const wrappedDoseBullet = source( + "wrapped-dose-bullet", + [ + "Dosage and administration", + "", + "• The usual oral starting dose for adults is 500 mg nocte and for patients over 65 years it", + "is 250 mg nocte.", + "• Dose should be titrated against target serum levels.", + ].join("\n"), + { title: "Lithium Clinical Guideline(EMHS)", file_name: "Lithium Clinical Guideline(EMHS).pdf" }, + ); + const input = answer( + "The usual oral starting dose for adults is 500 mg nocte and for patients over 65 years it is 250 mg nocte.", + [wrappedDoseBullet], + ); + + const { claims } = assessClaimSupport(input); + expect(claims).toHaveLength(1); + expect(claims[0]?.supportStatus).toBe("direct"); + expect(claims[0]?.supportingChunkIds).toEqual(["wrapped-dose-bullet"]); + }); + it("keeps a wrapped escalation recipient in the directly supporting source segment", () => { const wrappedRule = source( "wrapped-escalation-rule", From 6c3baf02b8cf37bb7fe4ef8f6fcf891caa175d5a Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Mon, 17 Aug 2026 17:05:06 +0800 Subject: [PATCH 2/4] docs: record S1 PR #2022 in the HANDOVER status row and review ledger Co-Authored-By: Claude Fable 5 --- ...1a271101bd68b0da1eac8b1633ed57ff72ad389f420fd20a2b.record.md | 1 + docs/rag-improvement/HANDOVER.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 docs/branch-review-records/e60e67c8bd9bd71a271101bd68b0da1eac8b1633ed57ff72ad389f420fd20a2b.record.md diff --git a/docs/branch-review-records/e60e67c8bd9bd71a271101bd68b0da1eac8b1633ed57ff72ad389f420fd20a2b.record.md b/docs/branch-review-records/e60e67c8bd9bd71a271101bd68b0da1eac8b1633ed57ff72ad389f420fd20a2b.record.md new file mode 100644 index 0000000000..ba213204b7 --- /dev/null +++ b/docs/branch-review-records/e60e67c8bd9bd71a271101bd68b0da1eac8b1633ed57ff72ad389f420fd20a2b.record.md @@ -0,0 +1 @@ +| 2026-08-17 | claude/s1-rag-mitigation-231-86c182 | 6ddd45e5fd0725638c55684ca85833ddf78560c4 | RAG answer-verification faithfulness fixes (#231 S1): markdown-emphasis atom folding + claim-support wrap reflow, tests, HANDOVER S1 row | PR #2022 open — rung 1 mitigation; residuals recorded (unbudgeted strong retry timeout, directive-normativity, topic dilution) | eval:rag:offline 583/583; check:production-readiness READY; verify:pr-local heavy scope green except 2 pre-existing host-env unit failures reproduced at merge-base d02767184; 8 pre-fix + 5 post-fix owner-approved live probes | diff --git a/docs/rag-improvement/HANDOVER.md b/docs/rag-improvement/HANDOVER.md index 212644a22b..6bb3e97976 100644 --- a/docs/rag-improvement/HANDOVER.md +++ b/docs/rag-improvement/HANDOVER.md @@ -44,7 +44,7 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured | Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | | Handover | Multi-session handover | `claude/rag-plan-review-guide-vhrls9` | #1908 | Open — this PR | docs-only | | S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | -| S1 | A1 phase 2: evidence-chosen mitigation | `claude/s1-rag-mitigation-231-86c182` | TBD | PR open — rung 1 verification-faithfulness fixes | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | +| S1 | A1 phase 2: evidence-chosen mitigation | `claude/s1-rag-mitigation-231-86c182` | #2022 | PR open — rung 1 verification-faithfulness fixes | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | | S2 | A2: composition menu | `claude/rag-a2-composition-` | — | Blocked on S1 evidence | — | | S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-` | — | Blocked on S2 | — | | S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-` | — | Blocked on S2 + S2b | — | From d0e7bbf0246dbf3e3f7cb92dd541aecc33178ef8 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Mon, 17 Aug 2026 17:31:01 +0800 Subject: [PATCH 3/4] test(rag): assert extractNumericTokens folds emphasis (#231 S1 review nit) Covers the modified numeric-token path directly per CodeRabbit review: the bolded dose yields the same normalized 200mg/day token as the unformatted source text. Co-Authored-By: Claude Fable 5 --- tests/answer-verification.test.ts | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/answer-verification.test.ts b/tests/answer-verification.test.ts index 39b674e04e..4ccf04b287 100644 --- a/tests/answer-verification.test.ts +++ b/tests/answer-verification.test.ts @@ -39,6 +39,12 @@ describe("answer-verification (GEN-C2 / GEN-H2)", () => { expect(bolded?.denominatorTime).toBe("day"); expect(bolded?.canonicalValue).toBe("200"); expect(bolded?.canonicalUnit).toBe("mg"); + // The direct numeric-token path folds emphasis too: the bolded dose yields the same + // normalized per-day token as the unformatted source text. + expect(extractNumericTokens("The maximum recommended dose of sertraline is **200 mg**/day.")).toContain( + "200mg/day", + ); + expect(extractNumericTokens("adjust dose according to response, maximum 200 mg/day")).toContain("200mg/day"); }); it("verifies bolded figures against unformatted source text (#231)", () => { From 5f1ccbd51024d5b3c23b35e014c27644b86727e1 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Mon, 17 Aug 2026 17:34:20 +0800 Subject: [PATCH 4/4] docs: supersede S1 review record after CodeRabbit nit fix Co-Authored-By: Claude Fable 5 --- ...ac930771cdeca2f76774bb555c73d9c5bb9469e702af2734790.record.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/branch-review-records/c41785428c51bac930771cdeca2f76774bb555c73d9c5bb9469e702af2734790.record.md diff --git a/docs/branch-review-records/c41785428c51bac930771cdeca2f76774bb555c73d9c5bb9469e702af2734790.record.md b/docs/branch-review-records/c41785428c51bac930771cdeca2f76774bb555c73d9c5bb9469e702af2734790.record.md new file mode 100644 index 0000000000..86e86f4ecb --- /dev/null +++ b/docs/branch-review-records/c41785428c51bac930771cdeca2f76774bb555c73d9c5bb9469e702af2734790.record.md @@ -0,0 +1 @@ +| 2026-08-17 | claude/s1-rag-mitigation-231-86c182 | d0e7bbf0246dbf3e3f7cb92dd541aecc33178ef8 | RAG answer-verification faithfulness fixes (#231 S1) + review-nit test coverage | PR #2022 CI green (PR required SUCCESS, CLEAN/MERGEABLE); CodeRabbit nitpick addressed; Codex/CodeRabbit both usage-limited so no further bot review; ready for owner merge | eval:rag:offline 583/583 on merged head 9098cc7c5; focused vitest answer-verification+rag-claim-support 227/227 after nit; check:production-readiness READY |