fix(rag): stop verification rejecting verbatim-faithful answers (#231 S1)

docs/branch-review-records/c41785428c51bac930771cdeca2f76774bb555c73d9c5bb9469e702af2734790.record.md

@@ -0,0 +1 @@
+| 2026-08-17 | claude/s1-rag-mitigation-231-86c182 | d0e7bbf0246dbf3e3f7cb92dd541aecc33178ef8 | RAG answer-verification faithfulness fixes (#231 S1) + review-nit test coverage | PR #2022 CI green (PR required SUCCESS, CLEAN/MERGEABLE); CodeRabbit nitpick addressed; Codex/CodeRabbit both usage-limited so no further bot review; ready for owner merge | eval:rag:offline 583/583 on merged head 9098cc7c5; focused vitest answer-verification+rag-claim-support 227/227 after nit; check:production-readiness READY |

docs/branch-review-records/e60e67c8bd9bd71a271101bd68b0da1eac8b1633ed57ff72ad389f420fd20a2b.record.md

@@ -0,0 +1 @@
+| 2026-08-17 | claude/s1-rag-mitigation-231-86c182 | 6ddd45e5fd0725638c55684ca85833ddf78560c4 | RAG answer-verification faithfulness fixes (#231 S1): markdown-emphasis atom folding + claim-support wrap reflow, tests, HANDOVER S1 row | PR #2022 open — rung 1 mitigation; residuals recorded (unbudgeted strong retry timeout, directive-normativity, topic dilution) | eval:rag:offline 583/583; check:production-readiness READY; verify:pr-local heavy scope green except 2 pre-existing host-env unit failures reproduced at merge-base d02767184; 8 pre-fix + 5 post-fix owner-approved live probes |

docs/rag-improvement/HANDOVER.md

@@ -39,19 +39,19 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured
 
 ## 2. Status table — update in every programme PR
 
-| Packet | Scope | Branch | PR | State | Canary / evidence refs |
-| -------- | ----------------------------------------------- | ------------------------------------------------ | ----- | ---------------------- | ---------------------- |
-| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only |
-| Handover | Multi-session handover | `claude/rag-plan-review-guide-vhrls9` | #1908 | Open — this PR | docs-only |
-| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused |
-| S1 | A1 phase 2: evidence-chosen mitigation | `claude/rag-a1-mitigation-<suffix>`  | — | Not started | —  |
-| S2 | A2: composition menu | `claude/rag-a2-composition-<suffix>` | — | Blocked on S1 evidence | — |
-| S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-<suffix>` | — | Blocked on S2 | — |
-| S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-<suffix>` | — | Blocked on S2 + S2b | — |
-| S4 | B0: adversarial fixtures + baseline + register | `claude/rag-b0-adversarial-fixtures-<suffix>` | — | Ready (parallel-safe) | — |
-| S5 | B1+B2: telemetry assessment + offline harness | `claude/rag-b1-b2-harness-<suffix>` | — | Blocked on S4 | — |
-| S6 | B3: Docling lab benchmark | `claude/rag-b3-docling-lab-<suffix>` | — | Blocked on S4 | — |
-| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — |
+| Packet | Scope | Branch | PR | State | Canary / evidence refs |
+| -------- | ----------------------------------------------- | ------------------------------------------------ | ----- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
+| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only |
+| Handover | Multi-session handover | `claude/rag-plan-review-guide-vhrls9` | #1908 | Open — this PR | docs-only |
+| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused |
+| S1 | A1 phase 2: evidence-chosen mitigation | `claude/s1-rag-mitigation-231-86c182` | #2022 | PR open — rung 1 verification-faithfulness fixes | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 |
+| S2 | A2: composition menu | `claude/rag-a2-composition-<suffix>` | — | Blocked on S1 evidence | — |
+| S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-<suffix>` | — | Blocked on S2 | — |
+| S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-<suffix>` | — | Blocked on S2 + S2b | — |
+| S4 | B0: adversarial fixtures + baseline + register | `claude/rag-b0-adversarial-fixtures-<suffix>` | — | Ready (parallel-safe) | — |
+| S5 | B1+B2: telemetry assessment + offline harness | `claude/rag-b1-b2-harness-<suffix>` | — | Blocked on S4 | — |
+| S6 | B3: Docling lab benchmark | `claude/rag-b3-docling-lab-<suffix>` | — | Blocked on S4 | — |
+| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — |
 
 Update rule: the session that opens a packet's PR edits its row (branch, PR number,
 state) in the same PR. A later session updating another packet may also correct stale rows
@@ -65,6 +65,12 @@ owner merges) and never at watching CI.
 
 ### S1 — A1 phase 2: choose and implement the mitigation from evidence (`#231`)
 
+- **Evidence update (2026-08-14 incident):** the mitigation ladder's rung 2 (pre-generation
+ latency) has a live measurement — `supabase_rpc_latency_ms` 31,610 ms from two dropped
+ trigram indexes, restored the same day (now 1,535 ms text / 8,519 ms hybrid). See
+ `docs/audit/live-drift-forensics-2026-08.md` (Phases 1.3 and 5) for the before/after
+ probes; any rung-2 reasoning must start from that file, not from the pre-incident
+ assumption that retrieval latency was healthy.
 - **Precondition:** PR #1899 merged; its diagnostics available. If live
  `generation_quality_gate:*` distributions exist in `rag_queries.metadata`, ask the owner
  for the aggregate counts (reading live Supabase is provider-gated — do not query it

src/lib/answer-verification.ts

@@ -62,6 +62,19 @@ function foldSuperscripts(text: string): string {
  return text.replace(/[⁰¹²³⁴⁵⁶⁷⁸⁹]/g, (ch) => SUPERSCRIPT_DIGITS[ch] ?? ch);
 }
 
+// Markdown emphasis markers reach answer prose from two writers — the model itself and the
+// pipeline's own high-yield bolding (boldRagAnswerHighYieldText in rag.ts), which runs BEFORE
+// numeric verification. A marker between a quantity and its per-time denominator makes
+// `**200 mg**/day` extract a bare `200mg` atom while the cited source's `200 mg/day` extracts
+// `200mg/day`, so a verbatim-faithful figure fails the exact atom-key match and the whole
+// dosing answer is discarded (#231, measured live 2026-08-17). Fold the markers (asterisks,
+// backticks, heading hashes, double underscores) to nothing before matching, mirroring the
+// claim-splitting `cleanText` in rag-claim-support.ts; single underscores are kept so
+// identifier-like tokens are not merged.
+function foldMarkdownEmphasis(text: string): string {
+ return /[*`#_]/.test(text) ? text.replace(/[*`#]+|_{2,}/g, "") : text;
+}
+
 function normalizeNumericToken(raw: string): string {
  return foldSuperscripts(raw)
  .toLowerCase()
@@ -190,7 +203,7 @@ function canonicalFrequency(count: string | undefined, phrase: string) {
 /** Extract meaning-preserving clinical value atoms for exact evidence matching. */
 export function extractClinicalValueAtoms(text: string): ClinicalValueAtom[] {
  if (!text) return [];
- const normalized = foldSuperscripts(text).normalize("NFKC");
+ const normalized = foldSuperscripts(foldMarkdownEmphasis(text)).normalize("NFKC");
  const atoms: ClinicalValueAtom[] = [];
  const occupied: Array<[number, number]> = [];
 
@@ -1257,7 +1270,7 @@ function clinicalValueAtomDisplay(atom: ClinicalValueAtom) {
 
 export function extractNumericTokens(text: string): string[] {
  if (!text) return [];
- const folded = foldSuperscripts(text);
+ const folded = foldSuperscripts(foldMarkdownEmphasis(text));
  const tokens = new Set<string>();
  for (const match of folded.matchAll(NUMERIC_TOKEN_PATTERN)) {
  const normalized = normalizeNumericToken(match[0]);

src/lib/rag/rag-claim-support.ts

@@ -505,9 +505,19 @@ function usesSourceBoundComparisonReflow(source: SearchResult, claim: string) {
 function sourceEvidenceClaimSegments(source: SearchResult, claim: string) {
  const split = (value: string | null | undefined, context?: string | null, reflowVisualLines = false) => {
  const rawValue = reflowWrappedAgitationDoseLines(reflowWrappedEscalationRecipientLines(value ?? ""));
+ // Always rejoin PDF visual line wraps before sentence-splitting. Splitting raw extracted
+ // text on `\n+` fragments a source sentence at its hard wrap ("…for adults is 500 mg nocte
+ // and for patients over 65 years it\nis 250 mg nocte"), so no single segment carries every
+ // atom of a claim restating that sentence — a verbatim-faithful high-risk claim then reads
+ // as unsupported and the whole answer degrades to source-only (#231, measured live
+ // 2026-08-17). Blank lines, bullets, terminal punctuation, colons, and numbered headings
+ // stay hard boundaries, and the general path joins only visible sentence continuations
+ // (lowercase/digit/parenthesis starts) so separate capitalized source lines cannot
+ // manufacture support. The comparison-reflow path keeps its historical aggressive join
+ // plus low-yield noise stripping for its two known policy documents, unchanged.
  const blocks = reflowVisualLines
  ? reflowBoundedSourceLines(sourceTextForClinicalProsePreservingBreaks(rawValue))
- : [rawValue];
+ : reflowBoundedSourceLines(rawValue, { requireContinuationStart: true });
  return blocks.flatMap((block) => {
  const sharedConditional = block.match(/^\s*((?:when|whenever|if|unless)\b[^,;.!?]+),?/i)?.[1];
  return block

src/lib/rag/rag-source-segmentation.ts

@@ -85,8 +85,20 @@ export function reflowWrappedAgitationDoseLines(value: string) {
  * Reflow OCR/PDF visual line wraps without joining across semantic source
  * boundaries. Blank lines, bullets, terminal punctuation, colons, and numbered
  * headings remain hard boundaries.
+ *
+ * With `requireContinuationStart`, a line additionally joins the previous one
+ * only when it visibly continues its sentence (starts with a lowercase letter,
+ * digit, or opening parenthesis). Capitalized lines then start a new block:
+ * joining separate capitalized items ("Stop clozapine" / "Starting dose
+ * 12.5 mg") would manufacture claim support from unrelated source lines. The
+ * default keeps the historical aggressive join for the sanitized policy-source
+ * and table-row callers, whose known texts wrap across capitalized proper
+ * nouns ("…Emergency\nDepartment (AHS-ED)…").
  */
-export function reflowBoundedSourceLines(value: string) {
+const wrapContinuationLineStartPattern = /^[a-z0-9(]/;
+
+export function reflowBoundedSourceLines(value: string, options?: { requireContinuationStart?: boolean }) {
+ const requireContinuationStart = options?.requireContinuationStart ?? false;
  const blocks: string[] = [];
  let wrappedLines: string[] = [];
  const flush = () => {
@@ -107,6 +119,7 @@ export function reflowBoundedSourceLines(value: string) {
  continue;
  }
  if (sourceBulletLinePattern.test(line)) flush();
+ if (requireContinuationStart && wrappedLines.length > 0 && !wrapContinuationLineStartPattern.test(line)) flush();
  wrappedLines.push(line);
  if (/[.!?]\s*$/.test(line) || /:\s*$/.test(line)) flush();
  }

tests/answer-verification.test.ts

@@ -28,6 +28,52 @@ function source(overrides: Partial<SearchResult> = {}): SearchResult {
 }
 
 describe("answer-verification (GEN-C2 / GEN-H2)", () => {
+ it("extracts the same clinical value atom through markdown emphasis markers (#231)", () => {
+ // Measured live 2026-08-17: the pipeline's own high-yield bolding produced
+ // "**200 mg**/day", which extracted a bare 200mg atom while the cited source's
+ // "200 mg/day" carried denominatorTime — a verbatim-faithful maximum-dose answer
+ // then failed the exact atom-key match and degraded to source-only.
+ const [plain] = extractClinicalValueAtoms("adjust dose according to response, maximum 200 mg/day");
+ const [bolded] = extractClinicalValueAtoms("The maximum recommended dose of sertraline is **200 mg**/day.");
+ expect(plain?.denominatorTime).toBe("day");
+ expect(bolded?.denominatorTime).toBe("day");
+ expect(bolded?.canonicalValue).toBe("200");
+ expect(bolded?.canonicalUnit).toBe("mg");
+ // The direct numeric-token path folds emphasis too: the bolded dose yields the same
+ // normalized per-day token as the unformatted source text.
+ expect(extractNumericTokens("The maximum recommended dose of sertraline is **200 mg**/day.")).toContain(
+ "200mg/day",
+ );
+ expect(extractNumericTokens("adjust dose according to response, maximum 200 mg/day")).toContain("200mg/day");
+ });
+
+ it("verifies bolded figures against unformatted source text (#231)", () => {
+ const doseSource = source({
+ id: "chunk-dose",
+ content: "sertraline: 50 mg orally once daily initially, adjust dose according to response, maximum 200 mg/day",
+ });
+ const verification = verifyAnswerNumbers(
+ "The maximum recommended dose of sertraline is **200 mg**/day.",
+ [{ chunk_id: "chunk-dose" }],
+ [doseSource],
+ );
+ expect(verification.unverifiedTokens).toEqual([]);
+ expect(verification.hasUnverifiedNumbers).toBe(false);
+ });
+
+ it("still fails a bolded figure absent from the cited source", () => {
+ const doseSource = source({
+ id: "chunk-dose",
+ content: "sertraline: 50 mg orally once daily initially, adjust dose according to response, maximum 200 mg/day",
+ });
+ const verification = verifyAnswerNumbers(
+ "The maximum recommended dose of sertraline is **300 mg**/day.",
+ [{ chunk_id: "chunk-dose" }],
+ [doseSource],
+ );
+ expect(verification.hasUnverifiedNumbers).toBe(true);
+ });
+
  it("detects overlapping labelled score bands without inferring corrected cutoffs", () => {
  const conflicts = detectLabelledNumericBandConflicts(
  "Escalate when the LUNSERS score is medium (41-89), high (81-100), or very high (>101).",

tests/rag-claim-support.test.ts

@@ -106,6 +106,32 @@ describe("deterministic claim support", () => {
  ]);
  });
 
+ it("supports a claim restating a source sentence split by a PDF visual line wrap (#231)", () => {
+ // Measured live 2026-08-17: the EMHS lithium guideline's starting-dose bullet wraps
+ // mid-sentence, so segment-splitting on raw newlines separated 500 mg/over-65 from
+ // 250 mg and a verbatim-faithful high-risk claim read as unsupported.
+ const wrappedDoseBullet = source(
+ "wrapped-dose-bullet",
+ [
+ "Dosage and administration",
+ "",
+ "• The usual oral starting dose for adults is 500 mg nocte and for patients over 65 years it",
+ "is 250 mg nocte.",
+ "• Dose should be titrated against target serum levels.",
+ ].join("\n"),
+ { title: "Lithium Clinical Guideline(EMHS)", file_name: "Lithium Clinical Guideline(EMHS).pdf" },
+ );
+ const input = answer(
+ "The usual oral starting dose for adults is 500 mg nocte and for patients over 65 years it is 250 mg nocte.",
+ [wrappedDoseBullet],
+ );
+
+ const { claims } = assessClaimSupport(input);
+ expect(claims).toHaveLength(1);
+ expect(claims[0]?.supportStatus).toBe("direct");
+ expect(claims[0]?.supportingChunkIds).toEqual(["wrapped-dose-bullet"]);
+ });
+
  it("keeps a wrapped escalation recipient in the directly supporting source segment", () => {
  const wrappedRule = source(
  "wrapped-escalation-rule",