feat(rag): B1 telemetry gap assessment + B2 offline adversarial harness (packet S5)

.env.example

@@ -150,6 +150,9 @@ RAG_INCREMENTAL_EVIDENCE_PREVIEW=false
 #NEXT_PUBLIC_RAG_INCREMENTAL_EVIDENCE_PREVIEW_RENDER=false
 # Ambiguity-only structured semantic reranking. Keep false until the retrieval canary is approved.
 RAG_SEMANTIC_RERANK_ENABLED=false
+# B1 extended answer telemetry (allow-listed numeric fields in rag_queries.metadata).
+# Default false; false is also the whole-surface rollback for packet B1.
+RAG_TELEMETRY_EXTENDED=false
 # Append OR-relaxed recall behind weak-but-nonzero strict text matches (P8b extension).
 # Opt-in experiment ONLY and OFF by default (matches the src/lib/env.ts default): it is known to
 # regress the golden retrieval eval (buries some correct docs after re-ranking). Leave false;

.github/workflows/ci.yml

@@ -417,6 +417,13 @@ jobs:
  if: needs.changes.outputs.rag_eval_changed == 'true'
  run: npm run eval:rag:offline
 
+ # Packet B2: the adversarial regression harness runs only for RAG-surface
+ # PRs — fixture validation first, then the offline Vitest harness; fails
+ # closed on missing fixture, network attempt, or round-trip budget breach.
+ - name: Offline RAG adversarial regression harness
+ if: needs.changes.outputs.rag_eval_changed == 'true'
+ run: npm run eval:rag:adversarial:offline
+
  coverage:
  name: Unit coverage
  needs: changes

docs/branch-review-records/b976a45d26170b11c35eb497940d68bd4d2ba8679fd0d427732e1b1d73032bf9.record.md

@@ -0,0 +1 @@
+| 2026-08-17 | claude/s5-rag-telemetry-harness-2wvis7 | 254132140dd8b19d8eb3669de88cf1d9ad9cf609 | Packet S5 (B1+B2): telemetry gap assessment + offline adversarial harness + routing | PR #2056 open; offline gates green; 3 divergences pinned in KNOWN_DIVERGENCES; 44/44 denominator reconciled | verify:pr-local heavy chain (lint, typecheck, test, build, eval:rag:offline, eval:rag:adversarial:offline) failed:(none) not-reached:(none); check:rag:adversarial-fixtures 24/8/6; maintainability rag.ts 4359/4362 |

docs/rag-improvement/README.md

@@ -257,7 +257,22 @@ As PDF PR 1: dashboard questions first (stage timeout rate, fallback rate, candi
 distribution, p50/p95 stage latency); map to existing Sentry/answer-telemetry fields; add
 `RAG_TELEMETRY_EXTENDED` (typed in `src/lib/env.ts`, default `false`) only for proven gaps;
 unit tests assert canaries never appear in emitted objects. Much of this is shared with
-Track A1's instrumentation — build once. Phoenix decision record: **deferred**.
+Track A1's instrumentation — build once.
+
+**Delivered by packet S5 (2026-08-17):** the assessment found three of the four dashboard
+questions fully answerable from the fields PR #1899 and earlier instrumentation already
+persist to `rag_queries.metadata`; the single proven gap was `verification_latency_ms`
+(measured into `latencyTimings`, dropped at the persistence boundary). It ships behind
+`RAG_TELEMETRY_EXTENDED` through the allow-listed numeric projection in
+`src/lib/rag/rag-answer-telemetry-metadata.ts`, with canary-absence tests in
+`tests/rag-telemetry-canary-absence.test.ts`.
+
+**Phoenix decision record — closed 2026-08-17: deferred.** `src/instrumentation.ts` and
+`src/lib/observability/agent-monitoring.ts` already trace without content capture; the B1
+assessment showed the named dashboard questions answerable from existing fields plus one
+flag-gated addition; a second tracing vendor would add a data-processing route (Gate A)
+before proving value. Revisit only if a dashboard question becomes unanswerable from
+`rag_queries.metadata` plus the existing Sentry surface.
 
 ### B2 — Offline adversarial regression harness
 
@@ -269,6 +284,16 @@ only; fails closed on missing fixture, network attempt, or budget breach. If Pro
 dependency footprint proves heavy, a plain Vitest harness over the same fixtures is an
 acceptable substitute — the fixtures and assertions are the asset, not the runner.
 
+**Delivered by packet S5 (2026-08-17) as the plain-Vitest substitute** (no dependency
+change; a Promptfoo experiment would be its own PR): `eval:rag:adversarial:offline` runs
+`scripts/check-rag-adversarial-fixtures.mjs` then `tests/rag-adversarial-harness.test.ts`
+over the 24 B0 cases with a stubbed-throwing `fetch`, a per-case Supabase round-trip
+ceiling, and canary-absence assertions on every persisted telemetry row. CI runs it in the
+`safety` job only when `rag_eval_changed` is true; `verify:pr-local` selects it for the
+same scope. Three fixture expectations diverge from current pipeline behaviour and are
+pinned in the harness's self-expiring `KNOWN_DIVERGENCES` register rather than being
+recalibrated away.
+
 ### B3 — Docling lab benchmark (isolated)
 
 As PDF PR 3, unchanged in substance: `eval/docling/` with hashed lockfile, sandboxed
@@ -349,7 +374,9 @@ work packets, the live status table, checklists, and paste-ready prompts.
 npm run format # and COMMIT the result before push
 npm run test:focused -- --files <changed source + tests>
 npm run check:rag:fixtures # existing golden/snapshot validator
+npm run check:rag:adversarial-fixtures # adversarial fixture contract (B0)
 npm run eval:rag:offline # offline RAG suite (Track A PRs)
+npm run eval:rag:adversarial:offline # offline adversarial harness (B2, RAG-surface PRs)
 npm run verify:pr-local -- --dry-run --files <paths> # then run selected gate
 npm run check:production-readiness # domain changes (env flags, answer path)
 ```

docs/rag-improvement/baseline-record.md

@@ -53,16 +53,14 @@ and explicitly not a result for this tree.
 **To complete the baseline:** with owner approval, dispatch the `eval-canary` pair at the merge
 commit, then flip each pending gate to `recorded` with its run id, and update `commit_sha`.
 
-## 3. Open discrepancy — the answer gate denominator
-
-`HANDOVER.md` §1 records "answer gate 45/45" for canary run `32025082010` at `2bd146eed`.
-`src/lib/rag/rag-eval-cases.ts` defines **44** `ragEvalCases`, both at `2bd146eed` and at this
-baseline commit, and `scripts/eval-rag.ts` reports its rates over `results.length`. The two do
-not reconcile, and this record does not guess which is right: the gate is recorded as 44 cases
-with the discrepancy stated.
-
-Reconcile it against run `32025082010`'s own report before treating either number as the
-baseline. Packet S5 is the natural place, since its harness consumes this record.
+## 3. Resolved discrepancy — the answer gate denominator is 44
+
+Reconciled by packet S5 (2026-08-17) against run `32025082010`'s own report (job
+`95372702769`): the report prints `## Answer Metrics | Cases | 44 |`, `Failing Answer
+Cases — None`, and its per-case diagnostics table lists exactly 44 rows — matching the 44
+`ragEvalCases` in `src/lib/rag/rag-eval-cases.ts` at both `2bd146eed` and this baseline
+commit. `HANDOVER.md`'s original "45/45" was a transcription error and has been corrected
+to 44/44. The gate denominator recorded here (44) stands.
 
 ## 4. Related
 

docs/rag-improvement/data-flow-register.md

@@ -42,18 +42,18 @@ expected category — that is what the adversarial fixtures' canary strings exis
 
 ## 3. Sinks and retention
 
-| Sink | What lands there | Retention | De-identified data permitted? | Enforcement / notes |
-| ---------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
-| Supabase Storage (`clinical-documents`) | Original uploaded files | Until the owner deletes the document | N/A — originals are the product | Private bucket; access is owner-scoped and signed-URL mediated. |
-| Postgres tables (chunks, pages, images, embeddings, index units) | Derived clinical text and vectors | Until reindex or document deletion; reindex commits per generation | N/A — derived from the owner's own documents | `reindex-pipeline.ts` commits atomically per generation. |
-| `rag_response_cache` | Generated answers keyed by query + prompt version | Until invalidated by a prompt-version bump or eviction | Yes | Generation fallbacks are excluded from the cache (`#231`), so a degraded answer cannot be served later. |
-| `rag_queries` / answer telemetry | Query metadata, routing, gate reasons, latency | Operational retention | Yes — metadata only | `query-privacy.ts`. Packet B1 adds fields only behind `RAG_TELEMETRY_EXTENDED`, with canary-absence tests.  |
-| `ingestion_jobs` / `document_index_quality` | Job state, quality gate outcomes | Job lifetime + audit history | Yes — metadata only | No document body text. |
-| Worker temp paths | Extracted text, page images, OCR intermediates | Process lifetime; removed when the job completes or fails | N/A — transient | Container-local. A crashed job must not leave text behind; this is the register's weakest verified claim and is called out in §5. |
-| Application logs / Sentry | Errors, stack traces, structured diagnostics | Provider retention | Yes — metadata only; never excerpt text | `docs/error-tracking.md`. Excerpt text must never be attached to an event. |
-| CI artifacts (GitHub Actions) | Eval reports, Playwright traces, coverage | GitHub's artifact retention | Yes — aggregates only | Eval reports are aggregate; canary literals must not appear (Gate A). |
-| Local eval output / stdout | Gate counts, case ids, aggregate metrics | Session only | Yes — aggregates and ids only | `check:rag:adversarial-fixtures` scans its own report for canaries before printing it. |
-| Repository files | Fixtures, baseline record, docs | Permanent (git history) | Synthetic only | The adversarial dataset is synthetic-only by contract; a real source name is a hard validation failure. |
+| Sink | What lands there | Retention | De-identified data permitted? | Enforcement / notes |
+| ---------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Supabase Storage (`clinical-documents`) | Original uploaded files | Until the owner deletes the document | N/A — originals are the product | Private bucket; access is owner-scoped and signed-URL mediated. |
+| Postgres tables (chunks, pages, images, embeddings, index units) | Derived clinical text and vectors | Until reindex or document deletion; reindex commits per generation | N/A — derived from the owner's own documents | `reindex-pipeline.ts` commits atomically per generation. |
+| `rag_response_cache` | Generated answers keyed by query + prompt version | Until invalidated by a prompt-version bump or eviction | Yes | Generation fallbacks are excluded from the cache (`#231`), so a degraded answer cannot be served later. |
+| `rag_queries` / answer telemetry | Query metadata, routing, gate reasons, latency | Operational retention | Yes — metadata only | `query-privacy.ts`. Packet B1 (shipped 2026-08-17): extended fields flow only through the allow-listed numeric projection in `src/lib/rag/rag-answer-telemetry-metadata.ts` behind `RAG_TELEMETRY_EXTENDED` (default false), pinned by `tests/rag-telemetry-canary-absence.test.ts`. |
+| `ingestion_jobs` / `document_index_quality` | Job state, quality gate outcomes | Job lifetime + audit history | Yes — metadata only | No document body text. |
+| Worker temp paths | Extracted text, page images, OCR intermediates | Process lifetime; removed when the job completes or fails | N/A — transient | Container-local. A crashed job must not leave text behind; this is the register's weakest verified claim and is called out in §5. |
+| Application logs / Sentry | Errors, stack traces, structured diagnostics | Provider retention | Yes — metadata only; never excerpt text | `docs/error-tracking.md`. Excerpt text must never be attached to an event. |
+| CI artifacts (GitHub Actions) | Eval reports, Playwright traces, coverage | GitHub's artifact retention | Yes — aggregates only | Eval reports are aggregate; canary literals must not appear (Gate A). |
+| Local eval output / stdout | Gate counts, case ids, aggregate metrics | Session only | Yes — aggregates and ids only | `check:rag:adversarial-fixtures` scans its own report for canaries before printing it. |
+| Repository files | Fixtures, baseline record, docs | Permanent (git history) | Synthetic only | The adversarial dataset is synthetic-only by contract; a real source name is a hard validation failure. |
 
 ## 4. Provider egress
 
@@ -83,5 +83,9 @@ work. Gate A sign-off must account for them explicitly.
 3. **Upload does not screen for identifiers.** Nothing at `/api/upload` detects a patient
  identifier inside an uploaded document. The system's design assumption is that uploads are
  guidelines, not records; that assumption is unenforced.
-4. **Telemetry canary-absence tests do not exist yet.** They are packet B1's deliverable. Today
- the only enforced canary boundary is the adversarial validator's own report.
+4. **Telemetry canary-absence tests — closed by packet B1/S5 (2026-08-17).**
+ `tests/rag-telemetry-canary-absence.test.ts` proves contaminated inputs cannot push a
+ registered canary token through the telemetry projection or the generation-quality answer
+ shape, and `tests/rag-adversarial-harness.test.ts` asserts every persisted `rag_queries`
+ row is canary-free for all 24 adversarial cases. The adversarial validator's own report
+ scan remains the boundary for the fixture-report sink.

docs/scripts-index.md

@@ -1,6 +1,6 @@
 # Scripts index
 
-Curated map of `scripts/` (245 files) and the `package.json` script surface (248 entries),
+Curated map of `scripts/` (246 files) and the `package.json` script surface (249 entries),
 grouped by purpose. This is orientation, not an exhaustive per-file listing — the authoritative
 command list is `package.json`, and `npm run docs:check-scripts` verifies every `npm run <x>`
 referenced in docs resolves to a real script. `npm run docs:update` refreshes the exact counts above.
@@ -93,7 +93,11 @@ For executable phone-chrome changes, use `verify:phone-chrome` before the broad
 `generation_quality_gate_reasons` added for `/issues` `#231` — provider-backed, refuses demo mode),
 `check-rag-adversarial-fixtures.mjs` + `rag-adversarial-contract.mjs` (offline, network-free
 validation of the synthetic adversarial fixture dataset and its baseline record; separate from
-`check-rag-fixtures.mjs`, which is untouched).
+`check-rag-fixtures.mjs`, which is untouched),
+`eval-rag-adversarial-offline.mjs` (packet B2: fixture validation then the offline Vitest
+adversarial harness `tests/rag-adversarial-harness.test.ts`; `npm run eval:rag:adversarial:offline`,
+routed by `ci-change-scope.mjs` to RAG-surface PRs only; fails closed on missing fixture,
+network attempt, or round-trip budget breach).
 Golden fixtures:
 `scripts/fixtures/rag-retrieval-golden.json`, `scripts/fixtures/assertion-golden.json`.
 Adversarial fixtures: `scripts/fixtures/rag-adversarial-cases.v1.json` (+ its schema) and