From 11d9cdd12b133ae5e80d013160ebc03ed6fa843e Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 02:23:08 +0000 Subject: [PATCH 1/6] docs: add reviewed/updated RAG improvement programme guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of the uploaded Clinical RAG Improvement Programme PDF against the current repository, with an updated two-track implementation guide under docs/rag-improvement/: Track A (answer quality — intent-conditioned related information, moderate length increase, follow-up suggestions, #231 budget prerequisite) sequenced before the corrected eval/safety infra track (adversarial fixtures, telemetry gap assessment, Docling lab/shadow, Ragas, reranker, DSPy). Adds planned-path allowlist entries to the docs link and script-ref checkers. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01UfJeqoq59y5xKmgRGHjqyA --- docs/README.md | 1 + docs/rag-improvement/README.md | 352 +++++++++++++++++++++++++++++ scripts/check-docs-links.mjs | 5 + scripts/check-docs-script-refs.mjs | 1 + 4 files changed, 359 insertions(+) create mode 100644 docs/rag-improvement/README.md diff --git a/docs/README.md b/docs/README.md index b01b72507..d1e0b0f16 100644 --- a/docs/README.md +++ b/docs/README.md @@ -95,6 +95,7 @@ npm run docs:check-links - [maturity-backlog-workorders.md](maturity-backlog-workorders.md) — actionable work orders tracking the repository-maturity audit backlog - [framework-dependency-modernization-checklist.md](framework-dependency-modernization-checklist.md) — ordered Next.js 16, runtime, dependency, Turbopack, and verification migration program - [search-rag-master-plan.md](search-rag-master-plan.md) / [search-rag-master-context.md](search-rag-master-context.md) — search/RAG roadmap and shared context +- [rag-improvement/README.md](rag-improvement/README.md) — reviewed/updated RAG improvement programme: answer-quality track (intent-aware related information, length) + corrected eval/safety infra track - [rag-hybrid-findings-and-todo.md](rag-hybrid-findings-and-todo.md) — hybrid retrieval findings backlog - [reindex-shadow-harness-design.md](reindex-shadow-harness-design.md) — designed-only shadow reindex harness (driver not built) - [ingestion-concurrency-fix-workorder.md](ingestion-concurrency-fix-workorder.md) — ingestion concurrency workorder diff --git a/docs/rag-improvement/README.md b/docs/rag-improvement/README.md new file mode 100644 index 000000000..384775a02 --- /dev/null +++ b/docs/rag-improvement/README.md @@ -0,0 +1,352 @@ +# Clinical RAG Improvement Programme — reviewed and updated implementation guide + +**Status:** maintained plan (2026-08-13). Supersedes the uploaded "Clinical RAG Improvement +Programme" PDF, which was pinned to main `130c7746` (main is now past `67db5a1`) and predates +several repo changes. This guide is grounded in the current codebase and is the working +reference for elevating the RAG: first the answer itself (intent-aware related information, +higher clinical yield, moderately longer output), then the PDF's evaluation/safety +infrastructure, corrected. + +**Scope discipline:** every Track A item touches protected RAG surfaces +(`docs/rag-behaviour/safeguards.md`). Each PR must flag RAG impact before editing, carry a +`RAG impact:` line in its body (`scripts/pr-policy.mjs` blocks the merge otherwise), and any +behaviour change needs a live eval-canary pair (provider-backed, ~$1–2, explicit approval per +run). Nothing here authorises reindexing, migrations, or provider calls by itself. + +--- + +## 1. Review verdict on the original PDF + +### 1.1 Agreed — retained (with corrections applied in Track B) + +| PDF item | Verdict | Why | +| ------------------------------------------------------------------------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Additive, gate-first philosophy; no production change without a measured promotion gate | **Keep** | Matches the repo's standing rule: "offline green is necessary, never sufficient" (`docs/rag-behaviour/README.md`) — the Phase C regression passed 121/121 offline and failed 3/36 live. | +| Phoenix deferred; assess existing Sentry first with an allow-listed telemetry contract | **Keep** | `src/instrumentation.ts` + `src/lib/observability/agent-monitoring.ts` already trace without content capture; a second tracing vendor adds a data-processing route before proving value. | +| Docling as isolated lab benchmark → worker shadow mode only after Gate B | **Keep** | The known weak point is tables/layout/OCR; `src/lib/index-quality.ts` signals can select a small shadow cohort instead of mass reprocessing. | +| Offline adversarial (Promptfoo) harness, synthetic-only, network-free | **Keep — genuinely missing** | The injection defences exist at runtime (`answerInstructions` §"Source excerpts are untrusted data" in `src/lib/rag/rag.ts`, `hasAdversarialManipulationIntent` in `src/lib/rag/rag-routing.ts:329`) but no fixture suite pins them; a prompt edit could silently weaken them today. | +| Reranker benchmark constraints: 36/36, zero per-case regression, latency/cost headroom | **Keep** | Consistent with refutations 1–2 and outstanding issue `#001` (semantic rerank stays off without an approved ambiguity comparison). | +| DSPy deferred until ≥100 labelled cases | **Keep, re-scoped** | See disagreement 2 — the proposed "query intent classification" target already exists in triplicate. | +| CI tiering through `scripts/ci-change-scope.mjs`, no duplicate workflows, cancel-superseded | **Keep** | Matches the existing risk-scoped CI design. | +| PHI posture: synthetic/de-identified fixtures only; canary strings in every sink | **Keep** | Aligns with the privacy boundary and `#053`. | + +### 1.2 Disagreed — corrected in this guide + +1. **Stale pin and a name collision.** The PDF's PR 0 proposes adding + `npm run check:rag:fixtures`. That command **already exists** + (`scripts/check-rag-fixtures.mjs`, validating the golden retrieval fixture and ranking + snapshot). The adversarial fixture validator must be a new command + (`check:rag:adversarial-fixtures`) or an extension of the existing script — never a + silent replacement. +2. **"Add query-intent classification" duplicates existing machinery.** Intent detection + already exists in three layers: `RagQueryClass` (6 classes, `src/lib/types.ts:718`, with + an LLM classifier fallback at `rag.ts` `analyzeQueryWithClassifierFallback`), + `ClinicalQueryIntent` (8 intents, `src/lib/clinical-search.ts:838`), and the + user-selectable `ClinicalQueryMode` (7 modes whose `clinicalModePrompt` is already + injected into generation). The genuine gap is **per-intent answer composition** — what + the answer shows for each question type — not detection. Track A2 targets that gap; + the PDF's DSPy lab is re-scoped away from classification. +3. **The PDF optimises everything except the answer.** It contains no item on related + information, answer length, clinical yield, or follow-up questions — the owner's actual + priority. Track A (new) fills this and is sequenced **first**. +4. **No engagement with the refuted-approaches ledger.** `docs/rag-behaviour/refuted-approaches.md` + records live-refuted shapes the PDF's conditional experiments could re-walk: + per-class feature-weight tuning (live no-op), comparator-key spread (live regression, + doc-recall 1.0 → 0.9167), governance currentness penalties/boosts (measured harm — + deliberately shipped at 0), and token streaming (removed as a clinical-safety control; + raw tokens bypass post-generation verification). Binding constraints are now embedded in + Track B items 6–7: any new ranking discriminator sits **strictly below `relevance.score`**; + benchmark fixtures must use differently-relevant candidates (identical-content fixtures + are how Phase C's regression escaped offline detection); a live canary pair is mandatory. +5. **Ignores outstanding issue `#231`.** Live answers already degrade to source-only when + the fast route budget (`answerRouteBudgetMs.fast = 25_000` in + `src/lib/rag/rag-route-budget.ts`) binds while retrieval is healthy. This is the single + highest-yield answer-quality defect in the queue, and a hard prerequisite for longer + answers: more output tokens means more truncation/timeout exposure, which converts + directly into more source-only fallbacks. Track A1 sequences it first. +6. **Roles table presumes a team.** Product/privacy/application/worker/evaluation owners are + one person here. Collapsed to: **owner** (clinical intent, privacy sign-off, release + thresholds, canary approval) and **agent sessions** (implementation, gates, evidence). +7. **Follow-up assumptions were wrong in both directions.** Suggested follow-up questions do + **not** exist (`src/lib/answer-follow-up.ts` only rewrites the user's own short + follow-ups by prepending the prior question), while `relatedDocuments` **does** exist + (`buildRelatedDocumentsSafe` in `rag.ts`, rendered under `trustCaps[trust].related` in + `src/lib/answer-render-policy.ts`). Track A4 builds the missing surface on the existing one. + +--- + +## 2. Current-state anchor (what the pipeline already does) + +Read `docs/rag-behaviour/` first for the protected mechanics. The short version relevant to +this plan: + +- **Flow:** `/api/answer` → `answerQuestionWithScopeUncoalesced` (`src/lib/rag/rag.ts`) → + hybrid retrieval (vector + tsvector + trigram + aliases + table facts + index units) → + `selectRetrievalEvidence` (`src/lib/retrieval-selection.ts`) → deterministic second stage + (`src/lib/rag/rag-second-stage.ts`) → `chooseAnswerRoute` (`src/lib/rag/rag-routing.ts`) → + fast/strong generation (reasoning-effort routing, not different models) → numeric + verification, claim support, citation sanitisation → render policy trust ladder. +- **Answer shape today:** the `answer` field is prompted to 1–3 sentences (~35–75 words); + `answerSections` carries 0–1 sections for simple facts, 2–5 for complex questions. The + prompt (`answerInstructions`, `rag.ts:3150`) and the "Interpreted clinical task" block + built by `buildAnswerInput` already carry `intent`, `query_class`, `answer_focus`, + `answer_scope`, and the full `answer_plan.*` fields. +- **Budgets:** `unsupported 0 / extractive 12s / fast 25s / strong 35s`; a + truncation self-heal retries with `strongRetryMaxOutputTokens`. Source-only fallback + (`source_backed_review_fallback`) fires on quality-gate failure, ungrounded extractive + fallback, or post-generation claim-support gaps — conservative by design. +- **Evals:** 36-case golden retrieval fixture (`scripts/fixtures/rag-retrieval-golden.json`, + zero-tolerance), 44-case `ragEvalCases` + 30-case `answerQualityEvalCases` + (`src/lib/rag/rag-eval-cases.ts`), ranking-candidate snapshot with lockstep pin and 30-day + freshness, live canary pairs via the `eval-canary` repository dispatch only. + +--- + +## 3. Track A — answer quality (sequenced first) + +Goal: for any clinical question, the system should (a) answer it directly, (b) surface the +_related_ high-yield information a psychiatrist colleague would append unprompted — chosen by +question type, and (c) be moderately longer (~1.5×) where evidence supports it, without +raising the source-only fallback rate or weakening a single grounding gate. + +### A1 — Budget headroom before length (prerequisite; resolves/mitigates `#231`) + +**Problem.** Longer answers cost tokens and seconds; the fast route already times out into +source-only fallbacks on healthy retrieval. Increasing length before fixing this makes the +product worse (more fallbacks), not better. + +**Work.** + +- Instrument (allow-listed metadata only — stage, latency_ms, timeout, fallback_used, + candidate_count; never query/answer text) the fast-route stages via the existing + `answer-telemetry` path to attribute where the 25s goes: retrieval, context packing + (`packContextForGeneration`), generation, verification. +- Candidate mitigations, in preference order: (1) raise `answerRouteBudgetMs.fast` + modestly with evidence that p95 generation fits; (2) route length-heavy query classes + (broad_summary, comparison) to the strong budget earlier via + `shouldRetryWithStrongAfterFast` predicates; (3) trim context-pack latency. Do **not** + reintroduce token streaming — the only admissible perceived-latency fix is progressive + disclosure of already-verified units over the existing `progress` SSE event + (refutation 6, ledger `#100`). +- **Files:** `src/lib/rag/rag-route-budget.ts`, `src/lib/rag/rag-routing.ts`, + `src/lib/answer-telemetry.ts`, targeted tests beside each. +- **Gate:** offline 44-case + 30-case suites unchanged; live observation window showing + fallback-rate non-inferiority; `RAG impact:` line (behaviour change → canary pair). + +### A2 — Intent-conditioned answer composition ("related information") + +**Design.** A small pure module, `src/lib/rag/answer-composition.ts`, mapping +(`RagQueryClass`, `ClinicalQueryIntent`) → a **composition menu**: which `answerSections` +kinds the model should attempt when — and only when — the retrieved evidence supports them. +Illustrative menu (tune during implementation): + +| Question type | Related information to offer | +| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `medication_dose_risk` / `drug_dosing` | monitoring schedule, contraindications/cautions, escalation & stop triggers, dose-adjustment populations (renal/hepatic/older adults), related documents | +| `table_threshold` | adjacent thresholds in the same scale, required actions per band, escalation pathway | +| `comparison` | decision factors, per-source differences/conflicts, switching/washout considerations | +| `broad_summary` / `protocol` | weighted management map (risk → first-line → adjuncts → monitoring → special populations), documentation/forms, source gaps | +| `escalation_risk` | immediate actions, thresholds, who to contact/refer, documentation | +| `definition` / `document_lookup` | stays narrow — zero or one section, exactly as today | + +**Mechanism (three small changes, no new pipeline stage):** + +1. Serialise the selected menu into the "Interpreted clinical task" block in + `buildAnswerInput` (one new line, e.g. `related_information_menu: monitoring, contraindications, escalation`). +2. Add one paragraph to `answerInstructions` §"Answer sections": attempt the listed kinds + when the excerpts support them; omit silently when they don't; never pad. +3. Grounding contract unchanged: every related item carries `citation_chunk_ids` or is + omitted. Verification (`applyNumericVerification`, `assessAndEnforceClaimSupport`) and the + render trust ladder apply to related sections exactly as to the core answer, so failure + still degrades conservatively. + +The render side needs nothing new: `answerSections` and `relatedDocuments` blocks already +exist under `answer-render-policy.ts`; check `trustCaps` section limits accommodate the +larger menus at `medium`/`high` trust. + +- **Files:** new `src/lib/rag/answer-composition.ts` (+ unit test), `src/lib/rag/rag.ts` + (prompt + `buildAnswerInput`), possibly `src/lib/answer-render-policy.ts` caps. +- **Gate:** offline 30/30 + 44-case suites re-baselined; retrieval untouched → 36/36 stays + trivially green; live canary pair before trusting (prompt changes alter answer behaviour); + Clinical Governance Preflight in the PR body. + +### A3 — Moderate length increase (~1.5×) + +- Prompt targets in `answerInstructions`: answer field 35–75 → **~60–110 words** (2–4 + sentences); complex-question sections 2–5 → **3–6** where evidence supports. Keep the + "narrow question → narrow answer" rule verbatim — a definition or single threshold must + not bloat; the length increase applies to management/comparison/threshold questions where + yield is real. +- Verify headroom: `OPENAI_MAX_OUTPUT_TOKENS` (16000) is ample; the binding constraint is + route time, which is why A1 lands first. Check `trustCaps` and any verification heuristics + that assume the current shape (quote-card counts, section caps). +- Bundle with A2 in one PR if the diff stays reviewable — both are prompt-surface changes + sharing one canary pair; otherwise ship A2 first, A3 second with its own pair. +- **Files:** `src/lib/rag/rag.ts` (prompt), `src/lib/rag/rag-versioning.ts` (bump + `ragAnswerPromptVersion` so the response cache and prompt cache key roll), eval baselines. + +### A4 — Suggested follow-up questions (new surface) + +- Deterministic-first generation (no extra provider call, no added latency): derive 2–4 + candidate next questions from `queryAnalysis` (medications, canonical terms), the + composition menu of A2 (e.g. dosing answered → offer "monitoring for X", "contraindications + for X"), and retrieved section headings. Template-based phrasing; only offer a suggestion + whose subject actually appears in the retrieved evidence. +- Return on `RagAnswer` (new optional field), render near `relatedDocuments`, gated by the + same trust ladder (suppress at `unsupported`/`low`). Clicking a suggestion submits through + the existing composer path; `buildAnswerFollowUpQuery` already handles topic carry-over. + Cross-mode deep links (prescribing, differentials, dsm) go through + `src/lib/cross-mode-links.ts` / `src/lib/app-modes.ts` hrefs — never raw ``. +- **Files:** new `src/lib/answer-follow-up-suggestions.ts` (+ test), `src/lib/types.ts` + (`RagAnswer` field), `rag.ts` wiring, one render block, UI wiring per + `docs/wiring-conventions.md`. +- **Gate:** additive field → `RAG impact: no retrieval behaviour change — additive answer +metadata` if generation prompt untouched; UI proof via `npm run ensure` + focused journey, + `verify:phone-chrome` if composer chrome is affected. + +**Track A sequencing:** A1 → (A2 + A3) → A4. Each PR: `npm run format` + commit, +`verify:pr-local`, offline eval re-baseline, canary pair where behaviour changes, ledger +append, one PR at a time (no bundling across RAG-impact boundaries). + +--- + +## 4. Track B — evaluation & safety infrastructure (corrected PDF Orders 0–7) + +Sequenced after Track A's A1–A3 unless a Track A canary surfaces a safety gap first. All +items are offline/lab work with no production behaviour change until their gate passes. + +### B0 — Baseline, adversarial fixture contract, data-flow register + +As PDF PR 0, corrected: + +- Command is **`npm run check:rag:adversarial-fixtures`** (new script + `scripts/check-rag-adversarial-fixtures.mjs`); the existing `check:rag:fixtures` is + untouched. +- Fixtures at `scripts/fixtures/rag-adversarial-cases.v1.json` + schema; 20–30 synthetic + cases in the PDF's 8 categories (injection, citation fabrication, unsupported claim, + empty/conflicting evidence, scope/tenant, provider failure, adversarial metadata, + cost/timeout abuse). Include PHI-like canary strings; validator rejects canary literals in + any reportable output. +- Baseline record: commit SHA, 36/36 result, 30/30 + 44-case results, prompt version, + `RAG_SEMANTIC_RERANK_ENABLED=false`, report key + `{commit_sha, dataset_version, eval_config_version, model_version, embedding_version, index_version}`. +- Data-flow register in this directory (`data-flow-register.md`): each input/process/sink, + retention, and whether de-identified data is permitted — including temp worker paths, CI + artifacts, caches, provider logs. + +### B1 — Telemetry gap assessment (no Phoenix) + +As PDF PR 1: dashboard questions first (stage timeout rate, fallback rate, candidate-count +distribution, p50/p95 stage latency); map to existing Sentry/answer-telemetry fields; add +`RAG_TELEMETRY_EXTENDED` (typed in `src/lib/env.ts`, default `false`) only for proven gaps; +unit tests assert canaries never appear in emitted objects. Much of this is shared with +Track A1's instrumentation — build once. Phoenix decision record: **deferred**. + +### B2 — Offline adversarial regression harness + +As PDF PR 2: Promptfoo pinned as a dev dependency, custom offline provider around the +repository's own offline harness (`RAG_PROVIDER_MODE=offline`, fetch/network rejected), +deterministic assertion functions with their own tests, fed by B0 fixtures. New command +`eval:rag:adversarial:offline`, routed by `scripts/ci-change-scope.mjs` to RAG-surface PRs +only; fails closed on missing fixture, network attempt, or budget breach. If Promptfoo's +dependency footprint proves heavy, a plain Vitest harness over the same fixtures is an +acceptable substitute — the fixtures and assertions are the asset, not the runner. + +### B3 — Docling lab benchmark (isolated) + +As PDF PR 3, unchanged in substance: `eval/docling/` with hashed lockfile, sandboxed +(non-root, no egress, resource limits), 30–50 public/synthetic fixtures stratified by +document difficulty plus a hostile corpus; compare against the legacy extractor +(`src/lib/extractors/document.ts`, `worker/python/extract_pdf_assets.py`) on parse success, +resource bounds, table precision/recall, exact number/unit/comparator checks. **Do not touch +worker requirements, Dockerfile.worker, or the database.** Gate B: non-inferiority on all +safety/exactness measures + pre-agreed table-heavy improvement. + +### B4 — Docling worker shadow mode (conditional on Gate B) + +As PDF PR 4: `WORKER_DOCUMENT_EXTRACTOR_MODE=legacy|shadow` (typed, default `legacy`), +shadow runs after legacy success on a 1–5% cohort selected by `src/lib/index-quality.ts` +signals, aggregate metadata only, no chunks/embeddings/index writes, kill switch, one-step +rollback to `legacy`. Worker-reviewer subagent (`ingestion-worker-reviewer`) reviews the PR. + +### B5 — Ragas calibration pilot (optional, offline) + +As PDF PR 5: adapter reads existing evaluation JSON, egress denied, judge-model use requires +Gate A approval. Retained only if it flags failures the deterministic gates miss at agreed +precision; otherwise archived. Never a release gate. + +### B6 — Local cross-encoder reranker benchmark (conditional, offline) + +As PDF PR 6, with refutation constraints made explicit: candidates exported with +**differently-relevant** content per case; any serving proposal keeps its score contribution +strictly below `relevance.score` in every comparator chain; requires 36/36, zero per-case +regression, citation-support parity, latency/cost headroom, then a separate default-off +serving PR (`RAG_LOCAL_RERANK_ENABLED=false`) with its own canary pair. Coordinate with +issue `#001` — the existing ambiguity-band semantic reranker stays independent and off. + +### B7 — DSPy lab (deferred, re-scoped) + +Entry criterion unchanged (≥100 clinician-reviewed cases, 60/20/20 split). **Re-scoped:** +optimise only answer phrasing/section-composition prompts (Track A2/A3 surfaces) — not +query-intent classification, which exists, and never authorization, evidence gates, +citations, or abstention policy. Candidates ship as static reviewed prompt PRs behind +default-off flags. + +### Gates A–F (retained from the PDF, collapsed roles) + +- **A — data-flow/privacy:** register complete; canary strings absent from every sink; + owner sign-off before any vendor/provider use. +- **B — Docling extraction:** non-inferior safety/exactness, improved table-heavy metric, + no budget breach. +- **C — adversarial behaviour:** B2 suite green on the expected refusal/abstention/citation + contract. +- **D — retrieval/ranking:** 36/36, zero per-case regression, p95 latency and cost in budget. +- **E — clinical quality:** owner's blinded review on representative sources — evidence, + citations, harmful-advice, abstention. For Track A: before/after answer comparison on a + fixed question set (the 30 `answerQualityEvalCases` plus ~10 owner-chosen live questions). +- **F — operations:** flag, one-step rollback, runbook, cost cap, redacted telemetry. + +--- + +## 5. Sequencing summary + +| Order | Item | Depends on | Behaviour change? | +| ----- | ----------------------------------------------- | ------------------------------- | -------------------------------------- | +| 1 | A1 budget headroom (`#231`) | — | Yes → canary pair | +| 2 | A2 + A3 intent-conditioned composition + length | A1 | Yes → canary pair + Gate E comparison | +| 3 | A4 follow-up suggestions | A2 | Additive (no pair if prompt untouched) | +| 4 | B0 baseline + adversarial fixtures | — (can run parallel to Track A) | No | +| 5 | B1 telemetry assessment | B0 (shares A1 instrumentation) | No | +| 6 | B2 adversarial harness | B0 | No | +| 7 | B3 Docling lab | B0 | No | +| 8 | B4 Docling shadow | Gate B | Worker-only, shadow | +| 9 | B5/B6 Ragas/reranker | conditional | No until separately promoted | +| 10 | B7 DSPy | ≥100 labelled cases | No until separately promoted | + +## 6. Verification commands (per PR, smallest first) + +```bash +npm run format # and COMMIT the result before push +npm run test:focused -- --files +npm run check:rag:fixtures # existing golden/snapshot validator +npm run eval:rag:offline # offline RAG suite (Track A PRs) +npm run verify:pr-local -- --dry-run --files # then run selected gate +npm run check:production-readiness # domain changes (env flags, answer path) +``` + +Live canary pairs fire only via the `eval-canary` repository dispatch with explicit owner +approval per run; regression → single-commit revert + confirmation run. + +## 7. Rollback map + +| Change | Rollback | +| -------------------------- | ----------------------------------------------------------------------- | +| A1 budgets/routing | revert commit; budgets are constants in `rag-route-budget.ts` | +| A2/A3 prompt + composition | revert commit; `ragAnswerPromptVersion` bump isolates caches | +| A4 suggestions | additive field — revert or hide render block | +| B1 telemetry | `RAG_TELEMETRY_EXTENDED=false` | +| B4 shadow | `WORKER_DOCUMENT_EXTRACTOR_MODE=legacy`; no migration or reindex needed | +| B6 reranker | `RAG_LOCAL_RERANK_ENABLED=false` | + +No item in this guide requires an irreversible action; index/database changes and any +cloud-vendor activation remain separate, explicit approval points. diff --git a/scripts/check-docs-links.mjs b/scripts/check-docs-links.mjs index 23bd96468..74f027115 100644 --- a/scripts/check-docs-links.mjs +++ b/scripts/check-docs-links.mjs @@ -45,6 +45,11 @@ const ROOT_PREFIXES = [ // designed-but-unbuilt drivers and hypothetical future splits. const ALLOWLIST = new Set([ "scripts/reindex-shadow.ts", // designed-only harness driver (docs/reindex-shadow-harness-design.md) + // Planned-but-unbuilt files named by the RAG improvement guide (docs/rag-improvement/README.md): + "src/lib/rag/answer-composition.ts", + "src/lib/answer-follow-up-suggestions.ts", + "scripts/check-rag-adversarial-fixtures.mjs", + "scripts/fixtures/rag-adversarial-cases.v1.json", "docs/site-map.generated.md", // hypothetical future split named in docs/process-hardening.md // Legacy pre-(search-app) paths still cited in docs/ledger/redesign records: "src/app/page.tsx", diff --git a/scripts/check-docs-script-refs.mjs b/scripts/check-docs-script-refs.mjs index f3a49b3e7..3985d53c8 100644 --- a/scripts/check-docs-script-refs.mjs +++ b/scripts/check-docs-script-refs.mjs @@ -36,6 +36,7 @@ const ALLOWLIST = new Set([ "", "your-script", "test:e2e:advisory", // renamed to test:e2e:regression (2026-07); kept for historical ledger accuracy + "check:rag:adversarial-fixtures", // planned command named by docs/rag-improvement/README.md (not yet built) ]); /** Script names defined in package.json. */ From a5d11cdb801eeaab436862417d5ff0e3167e7367 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 02:24:00 +0000 Subject: [PATCH 2/6] docs: record review ledger entry for claude/rag-plan-review-guide-vhrls9 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01UfJeqoq59y5xKmgRGHjqyA --- ...76575ed299d92b17501eef76d85f7b516d5d1ec95279fca7162.record.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/branch-review-records/c31311559848676575ed299d92b17501eef76d85f7b516d5d1ec95279fca7162.record.md diff --git a/docs/branch-review-records/c31311559848676575ed299d92b17501eef76d85f7b516d5d1ec95279fca7162.record.md b/docs/branch-review-records/c31311559848676575ed299d92b17501eef76d85f7b516d5d1ec95279fca7162.record.md new file mode 100644 index 000000000..53af63d30 --- /dev/null +++ b/docs/branch-review-records/c31311559848676575ed299d92b17501eef76d85f7b516d5d1ec95279fca7162.record.md @@ -0,0 +1 @@ +| 2026-08-13 | claude/rag-plan-review-guide-vhrls9 | 11d9cdd12b133ae5e80d013160ebc03ed6fa843e | docs: RAG improvement programme review + updated guide (docs/rag-improvement/, docs checker allowlists) | clean | verify:pr-local full scope green (lint, typecheck, test, docs gates, check:rag:fixtures 36 golden cases) | From 27714b593582f2c2685946dc1ae28ccfd826e668 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Thu, 13 Aug 2026 10:50:56 +0800 Subject: [PATCH 3/6] docs(rag): correct refuted budget and existing follow-up guidance --- docs/rag-improvement/README.md | 164 ++++++++++++++++++--------------- 1 file changed, 91 insertions(+), 73 deletions(-) diff --git a/docs/rag-improvement/README.md b/docs/rag-improvement/README.md index 384775a02..d6370bfa0 100644 --- a/docs/rag-improvement/README.md +++ b/docs/rag-improvement/README.md @@ -58,20 +58,24 @@ run). Nothing here authorises reindexing, migrations, or provider calls by itsel Track B items 6–7: any new ranking discriminator sits **strictly below `relevance.score`**; benchmark fixtures must use differently-relevant candidates (identical-content fixtures are how Phase C's regression escaped offline detection); a live canary pair is mandatory. -5. **Ignores outstanding issue `#231`.** Live answers already degrade to source-only when - the fast route budget (`answerRouteBudgetMs.fast = 25_000` in - `src/lib/rag/rag-route-budget.ts`) binds while retrieval is healthy. This is the single - highest-yield answer-quality defect in the queue, and a hard prerequisite for longer - answers: more output tokens means more truncation/timeout exposure, which converts - directly into more source-only fallbacks. Track A1 sequences it first. +5. **Ignores outstanding issue `#231`'s actual stop condition.** The 35–40-second route-budget + probes were already tested and rejected. In the decisive probe, generation completed + within budget with `route_deadline_exceeded=false`, but generation quality still failed + and the answer degraded to source-only. The prerequisite is therefore structured + generation-failure attribution and provider-safe instrumentation — not a larger timeout. + Track A1 starts there and forbids budget increases unless new evidence directly overturns + the recorded result. 6. **Roles table presumes a team.** Product/privacy/application/worker/evaluation owners are one person here. Collapsed to: **owner** (clinical intent, privacy sign-off, release thresholds, canary approval) and **agent sessions** (implementation, gates, evidence). -7. **Follow-up assumptions were wrong in both directions.** Suggested follow-up questions do - **not** exist (`src/lib/answer-follow-up.ts` only rewrites the user's own short - follow-ups by prepending the prior question), while `relatedDocuments` **does** exist +7. **Follow-up assumptions were wrong.** The repository already has + `buildAnswerFollowUpSuggestions` and `buildAnswerFollowUpQuery` in + `src/lib/answer-follow-up.ts`; `ClinicalDashboard.tsx` computes the suggestions and + renders wired phone and desktop chips. The genuine gap is stronger evidence gating and + intent/composition-aware selection on that existing surface, not a duplicate module, + `RagAnswer` field, or render block. `relatedDocuments` also already exists (`buildRelatedDocumentsSafe` in `rag.ts`, rendered under `trustCaps[trust].related` in - `src/lib/answer-render-policy.ts`). Track A4 builds the missing surface on the existing one. + `src/lib/answer-render-policy.ts`). Track A4 refines what is present. --- @@ -109,29 +113,39 @@ _related_ high-yield information a psychiatrist colleague would append unprompte question type, and (c) be moderately longer (~1.5×) where evidence supports it, without raising the source-only fallback rate or weakening a single grounding gate. -### A1 — Budget headroom before length (prerequisite; resolves/mitigates `#231`) +### A1 — Diagnose generation-quality fallbacks before changing length (prerequisite; `#231`) -**Problem.** Longer answers cost tokens and seconds; the fast route already times out into -source-only fallbacks on healthy retrieval. Increasing length before fixing this makes the -product worse (more fallbacks), not better. +**Problem.** Healthy retrieval still sometimes ends in a source-only fallback. The decisive +extended-budget probe completed generation inside the route deadline and still failed the +quality path, so route duration is not established as the binding cause. Increasing output +length before identifying the structured failure reason risks increasing fallbacks without +fixing the mechanism. **Work.** -- Instrument (allow-listed metadata only — stage, latency_ms, timeout, fallback_used, - candidate_count; never query/answer text) the fast-route stages via the existing - `answer-telemetry` path to attribute where the 25s goes: retrieval, context packing - (`packContextForGeneration`), generation, verification. -- Candidate mitigations, in preference order: (1) raise `answerRouteBudgetMs.fast` - modestly with evidence that p95 generation fits; (2) route length-heavy query classes - (broad_summary, comparison) to the strong budget earlier via - `shouldRetryWithStrongAfterFast` predicates; (3) trim context-pack latency. Do **not** - reintroduce token streaming — the only admissible perceived-latency fix is progressive - disclosure of already-verified units over the existing `progress` SSE event +- Complete allow-listed timing and decision metadata across the whole route: pre-retrieval + cache/version work, retrieval phase latencies, search total, route budget/deadline state, + generation failure reason/detail, and retry count/reasons. Never record query, answer, + provider-error, or source text. +- Attribute each source-only fallback to a stable stage and structured reason before changing + behaviour. Preserve the existing conservative fallback while diagnosing it. +- Choose the mitigation from evidence, in this order: + 1. fix the specific generation-quality, verification, or composition failure; + 2. reduce pre-generation latency if measurements show it is starving generation; + 3. when a length-heavy class such as `broad_summary` or `comparison` genuinely requires the + strong route, select it in `chooseAnswerRoute` **before** the route deadline is created — + not in `shouldRetryWithStrongAfterFast`, which runs only after the fast attempt; + 4. change a timeout or `answerRouteBudgetMs` only if new measurements directly show the + deadline is binding and explicitly rebut `#231`'s recorded stop condition. +- Do **not** reintroduce token streaming. The only admissible perceived-latency improvement is + progressive disclosure of already-verified units over the existing `progress` SSE event (refutation 6, ledger `#100`). -- **Files:** `src/lib/rag/rag-route-budget.ts`, `src/lib/rag/rag-routing.ts`, - `src/lib/answer-telemetry.ts`, targeted tests beside each. -- **Gate:** offline 44-case + 30-case suites unchanged; live observation window showing - fallback-rate non-inferiority; `RAG impact:` line (behaviour change → canary pair). +- **Files:** principally `src/lib/rag/rag.ts`, `src/lib/types.ts`, existing answer-telemetry + helpers, `src/lib/rag/rag-routing.ts` only if initial routing changes, and targeted tests. + `src/lib/rag/rag-route-budget.ts` is out of scope unless the evidence threshold above is met. +- **Gate:** telemetry-only work must leave offline 44-case + 30-case behaviour unchanged and + prove no sensitive text enters metadata. Any routing, prompt, quality-gate, or budget change + is a separate behaviour change with a `RAG impact:` statement and live canary pair. ### A2 — Intent-conditioned answer composition ("related information") @@ -177,36 +191,39 @@ larger menus at `medium`/`high` trust. "narrow question → narrow answer" rule verbatim — a definition or single threshold must not bloat; the length increase applies to management/comparison/threshold questions where yield is real. -- Verify headroom: `OPENAI_MAX_OUTPUT_TOKENS` (16000) is ample; the binding constraint is - route time, which is why A1 lands first. Check `trustCaps` and any verification heuristics - that assume the current shape (quote-card counts, section caps). +- Verify A1's evidence first: generation-quality reasons, available headroom, and fallback + rate are the binding evidence, not an assumed need for a longer route timeout. Check + `trustCaps` and any verification heuristics that assume the current shape (quote-card + counts, section caps). - Bundle with A2 in one PR if the diff stays reviewable — both are prompt-surface changes sharing one canary pair; otherwise ship A2 first, A3 second with its own pair. - **Files:** `src/lib/rag/rag.ts` (prompt), `src/lib/rag/rag-versioning.ts` (bump `ragAnswerPromptVersion` so the response cache and prompt cache key roll), eval baselines. -### A4 — Suggested follow-up questions (new surface) - -- Deterministic-first generation (no extra provider call, no added latency): derive 2–4 - candidate next questions from `queryAnalysis` (medications, canonical terms), the - composition menu of A2 (e.g. dosing answered → offer "monitoring for X", "contraindications - for X"), and retrieved section headings. Template-based phrasing; only offer a suggestion - whose subject actually appears in the retrieved evidence. -- Return on `RagAnswer` (new optional field), render near `relatedDocuments`, gated by the - same trust ladder (suppress at `unsupported`/`low`). Clicking a suggestion submits through - the existing composer path; `buildAnswerFollowUpQuery` already handles topic carry-over. - Cross-mode deep links (prescribing, differentials, dsm) go through - `src/lib/cross-mode-links.ts` / `src/lib/app-modes.ts` hrefs — never raw ``. -- **Files:** new `src/lib/answer-follow-up-suggestions.ts` (+ test), `src/lib/types.ts` - (`RagAnswer` field), `rag.ts` wiring, one render block, UI wiring per - `docs/wiring-conventions.md`. -- **Gate:** additive field → `RAG impact: no retrieval behaviour change — additive answer -metadata` if generation prompt untouched; UI proof via `npm run ensure` + focused journey, - `verify:phone-chrome` if composer chrome is affected. - -**Track A sequencing:** A1 → (A2 + A3) → A4. Each PR: `npm run format` + commit, -`verify:pr-local`, offline eval re-baseline, canary pair where behaviour changes, ledger -append, one PR at a time (no bundling across RAG-impact boundaries). +### A4 — Improve the existing suggested follow-up questions + +- Keep `buildAnswerFollowUpSuggestions` and `buildAnswerFollowUpQuery` in + `src/lib/answer-follow-up.ts` as the single implementation. The current deterministic + surface already derives up to four suggestions from medications, canonical terms, query + class, comparison intent, and source gaps, and `ClinicalDashboard.tsx` already renders + wired phone and desktop chips. +- Improve the existing function rather than duplicating it: incorporate A2's composition menu, + require the suggested subject to be supported by retrieved evidence, suppress redundant or + already-answered suggestions, and retain deterministic phrasing with no extra provider call + or latency. +- Keep submission on the current composer path. Cross-mode deep links, when appropriate, go + through `src/lib/cross-mode-links.ts` / `src/lib/app-modes.ts` hrefs — never raw ``. +- Do **not** add a second follow-up module, a new `RagAnswer` field, or another render block. + Change `ClinicalDashboard.tsx` only if the existing function's input contract must expand. +- **Files:** `src/lib/answer-follow-up.ts` and its focused tests; optionally the existing + `ClinicalDashboard.tsx` call site and A2 composition types. +- **Gate:** `RAG impact: no retrieval behaviour change — deterministic follow-up composition + only` when the generation prompt is untouched; focused DOM proof for both existing chip + surfaces and `verify:phone-chrome` only if shared composer chrome changes. + +**Track A sequencing:** A1 diagnosis → (A2 + A3) → A4 refinement. Each PR: +`npm run format` + commit, `verify:pr-local`, offline eval re-baseline, canary pair where +behaviour changes, ledger append, one PR at a time (no bundling across RAG-impact boundaries). --- @@ -310,18 +327,18 @@ default-off flags. ## 5. Sequencing summary -| Order | Item | Depends on | Behaviour change? | -| ----- | ----------------------------------------------- | ------------------------------- | -------------------------------------- | -| 1 | A1 budget headroom (`#231`) | — | Yes → canary pair | -| 2 | A2 + A3 intent-conditioned composition + length | A1 | Yes → canary pair + Gate E comparison | -| 3 | A4 follow-up suggestions | A2 | Additive (no pair if prompt untouched) | -| 4 | B0 baseline + adversarial fixtures | — (can run parallel to Track A) | No | -| 5 | B1 telemetry assessment | B0 (shares A1 instrumentation) | No | -| 6 | B2 adversarial harness | B0 | No | -| 7 | B3 Docling lab | B0 | No | -| 8 | B4 Docling shadow | Gate B | Worker-only, shadow | -| 9 | B5/B6 Ragas/reranker | conditional | No until separately promoted | -| 10 | B7 DSPy | ≥100 labelled cases | No until separately promoted | +| Order | Item | Depends on | Behaviour change? | +| ----- | ------------------------------------------------ | ------------------------------- | ------------------------------------------ | +| 1 | A1 structured fallback diagnosis (`#231`) | — | No for telemetry; separate pair for fixes | +| 2 | A2 + A3 intent-conditioned composition + length | A1 evidence | Yes → canary pair + Gate E comparison | +| 3 | A4 improve existing follow-up suggestions | A2 | Deterministic composition only | +| 4 | B0 baseline + adversarial fixtures | — (can run parallel to Track A) | No | +| 5 | B1 telemetry assessment | B0 (shares A1 instrumentation) | No | +| 6 | B2 adversarial harness | B0 | No | +| 7 | B3 Docling lab | B0 | No | +| 8 | B4 Docling shadow | Gate B | Worker-only, shadow | +| 9 | B5/B6 Ragas/reranker | conditional | No until separately promoted | +| 10 | B7 DSPy | ≥100 labelled cases | No until separately promoted | ## 6. Verification commands (per PR, smallest first) @@ -339,14 +356,15 @@ approval per run; regression → single-commit revert + confirmation run. ## 7. Rollback map -| Change | Rollback | -| -------------------------- | ----------------------------------------------------------------------- | -| A1 budgets/routing | revert commit; budgets are constants in `rag-route-budget.ts` | -| A2/A3 prompt + composition | revert commit; `ragAnswerPromptVersion` bump isolates caches | -| A4 suggestions | additive field — revert or hide render block | -| B1 telemetry | `RAG_TELEMETRY_EXTENDED=false` | -| B4 shadow | `WORKER_DOCUMENT_EXTRACTOR_MODE=legacy`; no migration or reindex needed | -| B6 reranker | `RAG_LOCAL_RERANK_ENABLED=false` | +| Change | Rollback | +| -------------------------- | ------------------------------------------------------------------------- | +| A1 telemetry/diagnosis | revert additive metadata commit; no budget change is implied | +| A1 behavioural mitigation | revert the separately measured routing/quality fix | +| A2/A3 prompt + composition | revert commit; `ragAnswerPromptVersion` bump isolates caches | +| A4 suggestions | revert changes to the existing `answer-follow-up.ts` implementation | +| B1 telemetry | `RAG_TELEMETRY_EXTENDED=false` | +| B4 shadow | `WORKER_DOCUMENT_EXTRACTOR_MODE=legacy`; no migration or reindex needed | +| B6 reranker | `RAG_LOCAL_RERANK_ENABLED=false` | No item in this guide requires an irreversible action; index/database changes and any cloud-vendor activation remain separate, explicit approval points. From 7ae5ae3e43c9393f131f7a11fe109e9365af4cf1 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Thu, 13 Aug 2026 10:51:30 +0800 Subject: [PATCH 4/6] chore(docs): remove duplicate follow-up module allowlist --- scripts/check-docs-links.mjs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/scripts/check-docs-links.mjs b/scripts/check-docs-links.mjs index 74f027115..baa16e0e9 100644 --- a/scripts/check-docs-links.mjs +++ b/scripts/check-docs-links.mjs @@ -10,7 +10,7 @@ * — resolved relative to the file containing the link and required to stay * inside the repository. * - * Scanned by default: README.md, AGENTS.md, and docs/**\/*.md excluding + * Scanned by default: README.md, AGENTS.md, and docs/**/*.md excluding * docs/archive/, docs/audit/, dated point-in-time filenames * (docs/README.md classifies those as historical records that intentionally * reference the repo as it was), and docs/prompts/codex-cloud-review/ (verbatim @@ -47,7 +47,6 @@ const ALLOWLIST = new Set([ "scripts/reindex-shadow.ts", // designed-only harness driver (docs/reindex-shadow-harness-design.md) // Planned-but-unbuilt files named by the RAG improvement guide (docs/rag-improvement/README.md): "src/lib/rag/answer-composition.ts", - "src/lib/answer-follow-up-suggestions.ts", "scripts/check-rag-adversarial-fixtures.mjs", "scripts/fixtures/rag-adversarial-cases.v1.json", "docs/site-map.generated.md", // hypothetical future split named in docs/process-hardening.md From e5ad09dc54aa741841558a9fffeeec695d837064 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Thu, 13 Aug 2026 11:00:23 +0800 Subject: [PATCH 5/6] fix(docs): avoid block-comment terminator in glob prose --- scripts/check-docs-links.mjs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/check-docs-links.mjs b/scripts/check-docs-links.mjs index baa16e0e9..2a6d8d9cb 100644 --- a/scripts/check-docs-links.mjs +++ b/scripts/check-docs-links.mjs @@ -10,8 +10,8 @@ * — resolved relative to the file containing the link and required to stay * inside the repository. * - * Scanned by default: README.md, AGENTS.md, and docs/**/*.md excluding - * docs/archive/, docs/audit/, dated point-in-time filenames + * Scanned by default: README.md, AGENTS.md, and all Markdown files under docs/, + * excluding docs/archive/, docs/audit/, dated point-in-time filenames * (docs/README.md classifies those as historical records that intentionally * reference the repo as it was), and docs/prompts/codex-cloud-review/ (verbatim * as-provided prompt inputs whose paths must not be edited). Pass --all to scan From 460401a31c7be0d0a1f962bca92b60d39268bdc3 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Thu, 13 Aug 2026 11:09:26 +0800 Subject: [PATCH 6/6] fix: format RAG improvement guide --- docs/rag-improvement/README.md | 44 +++++++++++++++++----------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/docs/rag-improvement/README.md b/docs/rag-improvement/README.md index d6370bfa0..28b5f464e 100644 --- a/docs/rag-improvement/README.md +++ b/docs/rag-improvement/README.md @@ -218,7 +218,7 @@ larger menus at `medium`/`high` trust. - **Files:** `src/lib/answer-follow-up.ts` and its focused tests; optionally the existing `ClinicalDashboard.tsx` call site and A2 composition types. - **Gate:** `RAG impact: no retrieval behaviour change — deterministic follow-up composition - only` when the generation prompt is untouched; focused DOM proof for both existing chip +only` when the generation prompt is untouched; focused DOM proof for both existing chip surfaces and `verify:phone-chrome` only if shared composer chrome changes. **Track A sequencing:** A1 diagnosis → (A2 + A3) → A4 refinement. Each PR: @@ -327,18 +327,18 @@ default-off flags. ## 5. Sequencing summary -| Order | Item | Depends on | Behaviour change? | -| ----- | ------------------------------------------------ | ------------------------------- | ------------------------------------------ | -| 1 | A1 structured fallback diagnosis (`#231`) | — | No for telemetry; separate pair for fixes | -| 2 | A2 + A3 intent-conditioned composition + length | A1 evidence | Yes → canary pair + Gate E comparison | -| 3 | A4 improve existing follow-up suggestions | A2 | Deterministic composition only | -| 4 | B0 baseline + adversarial fixtures | — (can run parallel to Track A) | No | -| 5 | B1 telemetry assessment | B0 (shares A1 instrumentation) | No | -| 6 | B2 adversarial harness | B0 | No | -| 7 | B3 Docling lab | B0 | No | -| 8 | B4 Docling shadow | Gate B | Worker-only, shadow | -| 9 | B5/B6 Ragas/reranker | conditional | No until separately promoted | -| 10 | B7 DSPy | ≥100 labelled cases | No until separately promoted | +| Order | Item | Depends on | Behaviour change? | +| ----- | ----------------------------------------------- | ------------------------------- | ----------------------------------------- | +| 1 | A1 structured fallback diagnosis (`#231`) | — | No for telemetry; separate pair for fixes | +| 2 | A2 + A3 intent-conditioned composition + length | A1 evidence | Yes → canary pair + Gate E comparison | +| 3 | A4 improve existing follow-up suggestions | A2 | Deterministic composition only | +| 4 | B0 baseline + adversarial fixtures | — (can run parallel to Track A) | No | +| 5 | B1 telemetry assessment | B0 (shares A1 instrumentation) | No | +| 6 | B2 adversarial harness | B0 | No | +| 7 | B3 Docling lab | B0 | No | +| 8 | B4 Docling shadow | Gate B | Worker-only, shadow | +| 9 | B5/B6 Ragas/reranker | conditional | No until separately promoted | +| 10 | B7 DSPy | ≥100 labelled cases | No until separately promoted | ## 6. Verification commands (per PR, smallest first) @@ -356,15 +356,15 @@ approval per run; regression → single-commit revert + confirmation run. ## 7. Rollback map -| Change | Rollback | -| -------------------------- | ------------------------------------------------------------------------- | -| A1 telemetry/diagnosis | revert additive metadata commit; no budget change is implied | -| A1 behavioural mitigation | revert the separately measured routing/quality fix | -| A2/A3 prompt + composition | revert commit; `ragAnswerPromptVersion` bump isolates caches | -| A4 suggestions | revert changes to the existing `answer-follow-up.ts` implementation | -| B1 telemetry | `RAG_TELEMETRY_EXTENDED=false` | -| B4 shadow | `WORKER_DOCUMENT_EXTRACTOR_MODE=legacy`; no migration or reindex needed | -| B6 reranker | `RAG_LOCAL_RERANK_ENABLED=false` | +| Change | Rollback | +| -------------------------- | ----------------------------------------------------------------------- | +| A1 telemetry/diagnosis | revert additive metadata commit; no budget change is implied | +| A1 behavioural mitigation | revert the separately measured routing/quality fix | +| A2/A3 prompt + composition | revert commit; `ragAnswerPromptVersion` bump isolates caches | +| A4 suggestions | revert changes to the existing `answer-follow-up.ts` implementation | +| B1 telemetry | `RAG_TELEMETRY_EXTENDED=false` | +| B4 shadow | `WORKER_DOCUMENT_EXTRACTOR_MODE=legacy`; no migration or reindex needed | +| B6 reranker | `RAG_LOCAL_RERANK_ENABLED=false` | No item in this guide requires an irreversible action; index/database changes and any cloud-vendor activation remain separate, explicit approval points.