From b14adae27648fc95308e0aed1439940b4ea62515 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Fri, 31 Jul 2026 05:40:48 +0800 Subject: [PATCH 1/2] docs(rag): record refuted lexical probe collapse (#098) --- docs/outstanding-issues.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/outstanding-issues.md b/docs/outstanding-issues.md index 955e78f8ff..9789eb0cf2 100644 --- a/docs/outstanding-issues.md +++ b/docs/outstanding-issues.md @@ -132,7 +132,7 @@ removed after current-main verification; it is not missing recommended work. | #092 | P3 | task | Refetch pulse deferred on auth-backed registries (privacy invariant) | **Outcome:** a background refresh keeps the prior count visible instead of a skeleton, without weakening identity clearing. **Detail:** the `refetching` status is built in the band and adopted only on `formulation-home-page.tsx`, where the lag is `useDeferredValue` over static data. It is deliberately NOT adopted on `use-registry-records.ts:85`, `use-medication-catalog.ts:76` or `use-differential-catalog.ts:133`, which all clear data on entering loading. `use-differential-catalog.ts:122` states why: "Auth must clear prior identity's matches immediately", and `:164` that "a later retype of any prior query cannot resurrect authorized matches." **Next:** if adopted, guard preservation on identity AND query equality, and pin with a test that an identity change still clears immediately. **Stop:** never hold records across an auth transition. | PR #1316 plan phase 6; session 2026-07-28 | 2026-07-28 | | #093 | P2 | issue | Next streaming `S:` clone causes Playwright strict-mode violations under CI load | **Outcome:** duplicate-element strict-mode failures stop appearing on loaded CI runs. **Detail:** under full-suite CI load Next.js leaves a hidden duplicate page root in the stream, so a `getByTestId` that is unique locally resolves to 2 elements in CI (seen as `differentials-search-results` on PR #1316, and previously noted on PR #1294 against main). It does not reproduce in isolation, on a single spec, or locally. The documented workaround is to scope the locator to the visible root. **Reproduced locally 2026-07-28** (isolated _production_ build via `run-playwright.mjs`, full `verify:ui`): `ui-tools.spec.ts:563` duplicated `forms-home` and `ui-smoke.spec.ts:3001` duplicated `favourite-row-lithium-monitoring-guideline`; in both, copy 1 is nested under `mobile-composer-reserve-pad`. Both pass when run alone, so it is load/order-dependent, not build-mode dependent — this also corrects an earlier note that CI uses `next dev`; it does not. **Strongest evidence (CI run `30345484316`, 2026-07-28): `ui-overlap.spec.ts:199` on `/` asserted `toHaveCount(1)` successfully and then the same `header#search` locator resolved to 2 a statement later, one of them hidden.** A duplicate that appears _after_ a passing count assertion is a stream/hydration artifact by construction, not a static double mount and not something a CSS or component change can cause. That makes four distinct testids across four specs with the identical shape. **Mitigated, not fixed, on `main` (2026-07-28):** `3a8edb93` rewrapped `gotoHome` in `tests/ui-overlap.spec.ts` to retry count-and-visibility together via `toPass`, so a transient second header no longer trips strict mode there — its own note says "checking count then immediately calling waitFor races that flicker into a strict-mode violation". That hardens one helper; the duplicate root itself is unchanged and other specs remain exposed. **Confirmed pre-existing:** at `631d90d2`, the commit before PR #1316's first commit, that spec already documented "two `header#search` nodes" and "a second transient `header#search` can exist briefly" — so this predates that branch. **Next:** with a full-suite repro now available, bisect the preceding specs to find the state that triggers the second mount, then either scope the shared helpers to the visible root once or fix the mount. **Stop:** do not paper over new occurrences with `.first()` before the duplicate itself is explained. | PR #1316 CI runs; PR #1294 note on main; session 2026-07-28 | 2026-07-28 | | #094 | P2 | rec | Design-system gates assert structure, not rendered effect | **Outcome:** a style contract cannot pass while the style is inert. **Detail:** PR #1316's accent rail shipped inert because `.search-band` sat in `@layer components`, which loses to Tailwind's utilities layer regardless of specificity — and the test asserted `toHaveClass("search-band")`, i.e. class presence, not effect. Computed style showed `1px rgb(229,231,235)` where `2px rgb(11,111,134)` was intended. The same shape of gap let a rail-colour assertion compare a colour against a width and pass unconditionally. **Next:** for contracts where the visual IS the requirement (rails, forced-colors thickness, tap targets), assert `getComputedStyle` in a Playwright case rather than class names in a DOM test, and add the unlayered-component convention to the design-system contract check. **Stop:** do not convert existing passing DOM tests wholesale; add computed-style proof only where the effect carries the meaning. | PR #1316 Codex P2 finding; session 2026-07-28 | 2026-07-28 | -| #098 | P2 | task | Offline round-trip budget harness for the hot routes | **Outcome:** per-scenario Supabase round-trip counts are pinned by a test, so an extra round trip on a hot path is a red gate rather than an inference. **Done 2026-07-29:** the measurement gap is closed — `Server-Timing` now covers `auth`/`ratelimit`/`scope` on `/api/answer`, `auth`/`ratelimit`/`search`/`total` on `/api/search`, and `auth`/`ratelimit` on `/api/answer/stream` (previously the route the UI actually calls emitted no header at all). Headers flush before the first SSE frame, so in-stream stages cannot reach a header and must NOT be routed through the governed `progress`/`final` contract. `tests/answer-route-preamble.test.ts` pins admission-before-scope (no scope call while the limiter is pending or after a deny) and the client-disconnect abort signal. **Done 2026-07-30 (PR #1450, `1bff4c78`):** the counting proxy exists and the answer path is budgeted. `tests/helpers/supabase-round-trip-counter.ts` counts on **execution, not construction** — a builder that is never awaited costs zero, one awaited twice costs two — which is the distinction that makes the count mean "requests issued". `tests/rag-round-trip-budget.test.ts` pins two offline answer-path scenarios (a single-source source-only answer, and that trips do not scale with the number of retrieved sources) plus three self-tests of the counter, and is registered in `scripts/fixtures/rag-offline-contract-tests.json` so it runs inside the offline contract rather than only on demand. Verified locally, provider-free: `Test Files 1 passed (1)`, `Tests 5 passed (5)`. Its documented blind spot is worth repeating before anyone cites a budget as total cost: it sees only traffic through the wrapped client, so a trip issued via another client instance, a direct `fetch`, or a provider SDK is invisible to it. **Done 2026-07-30 (PR #1450, `1bff4c78`):** the counting proxy exists and the answer path is budgeted. `tests/helpers/supabase-round-trip-counter.ts` counts on **execution, not construction** — a builder that is never awaited costs zero, one awaited twice costs two. `tests/rag-round-trip-budget.test.ts` pins two offline answer-path scenarios plus three self-tests of the counter, registered in `scripts/fixtures/rag-offline-contract-tests.json`. Its documented blind spot: it sees only traffic through the wrapped client, so a trip via another client instance, a direct `fetch`, or a provider SDK is invisible to it. **Done 2026-07-30 (search *retrieval core*, not the endpoint):** `tests/search-round-trip-budget.test.ts` pins `searchChunksWithTelemetry` — what `/api/search` calls to retrieve — registered in both the contract fixture and `scripts/rag-offline-contract.mjs`. **Corrected after Codex review on PR #1464:** an earlier version of this row and the test itself claimed to pin `/api/search`. They do not. The route's auth, rate limiting, scope resolution, related-document enrichment and telemetry write are all invisible to this suite, so a round trip added to any of them leaves it green — and the refusal budget below is about *retrieval*, not about an adversarial HTTP request, which still pays the route preamble. **The measured shape is itself the finding:** one search costs **11 round trips** — `rag_aliases` 1, `match_document_chunks_text_v2` **3**, `match_document_table_facts_text_v2` **3**, `get_related_document_metadata_v2` 1, `document_index_quality` 1, `document_images` 2 — so the two text RPCs are each issued three times per search. Pinned by total *and* breakdown, because a refactor swapping one probe for an unrelated query would keep the total at 11 while changing the traffic. Deterministic across three consecutive runs. The refusal budget asserts **zero** Supabase traffic, matching `rag.ts`'s claim that prompt-injection intent is refused before any query issues, and was proven against the broken shape: with a non-refused query it fails on the round-trip assertion (`expected 11 to be +0`), which is why that assertion is ordered ahead of the results assertion. **Next:** (a) add the route-level budget this suite does not provide — drive `POST` from `src/app/api/search/route.ts` with counted clients, following the `tests/answer-route-preamble.test.ts` pattern, so a round trip added to the route preamble or post-processing is a red gate; (b) decide whether `match_document_chunks_text_v2` ×3 and `match_document_table_facts_text_v2` ×3 per search are intended or a probe that should be collapsed — a latency question this budget surfaced but does not answer, and one that touches retrieval, so any change needs the usual RAG gate. (c) `scripts/eval-rag-offline.mjs` and `scripts/test-rag-offline.mjs` remain unwired; the offline contract runner is now the de-facto single home for budgets, so either adopt that explicitly here or wire them. | `docs/audit/latency-audit-2026-07-28.md` measurement plan; `src/lib/server-timing.ts`; `src/lib/answer-stream-contract.ts:18-21` | 2026-07-29 | +| #098 | P2 | task | Offline round-trip budget harness for the hot routes | **Outcome:** per-scenario Supabase round-trip counts are pinned by a test, so an extra round trip on a hot path is a red gate rather than an inference. **Done 2026-07-29:** the measurement gap is closed — `Server-Timing` now covers `auth`/`ratelimit`/`scope` on `/api/answer`, `auth`/`ratelimit`/`search`/`total` on `/api/search`, and `auth`/`ratelimit` on `/api/answer/stream` (previously the route the UI actually calls emitted no header at all). Headers flush before the first SSE frame, so in-stream stages cannot reach a header and must NOT be routed through the governed `progress`/`final` contract. `tests/answer-route-preamble.test.ts` pins admission-before-scope (no scope call while the limiter is pending or after a deny) and the client-disconnect abort signal. **Done 2026-07-30 (PR #1450, `1bff4c78`):** the counting proxy exists and the answer path is budgeted. `tests/helpers/supabase-round-trip-counter.ts` counts on **execution, not construction** — a builder that is never awaited costs zero, one awaited twice costs two — which is the distinction that makes the count mean "requests issued". `tests/rag-round-trip-budget.test.ts` pins two offline answer-path scenarios (a single-source source-only answer, and that trips do not scale with the number of retrieved sources) plus three self-tests of the counter, and is registered in `scripts/fixtures/rag-offline-contract-tests.json` so it runs inside the offline contract rather than only on demand. Verified locally, provider-free: `Test Files 1 passed (1)`, `Tests 5 passed (5)`. Its documented blind spot is worth repeating before anyone cites a budget as total cost: it sees only traffic through the wrapped client, so a trip issued via another client instance, a direct `fetch`, or a provider SDK is invisible to it. **Done 2026-07-30 (PR #1450, `1bff4c78`):** the counting proxy exists and the answer path is budgeted. `tests/helpers/supabase-round-trip-counter.ts` counts on **execution, not construction** — a builder that is never awaited costs zero, one awaited twice costs two. `tests/rag-round-trip-budget.test.ts` pins two offline answer-path scenarios plus three self-tests of the counter, registered in `scripts/fixtures/rag-offline-contract-tests.json`. Its documented blind spot: it sees only traffic through the wrapped client, so a trip via another client instance, a direct `fetch`, or a provider SDK is invisible to it. **Done 2026-07-30 (search *retrieval core*, not the endpoint):** `tests/search-round-trip-budget.test.ts` pins `searchChunksWithTelemetry` — what `/api/search` calls to retrieve — registered in both the contract fixture and `scripts/rag-offline-contract.mjs`. **Corrected after Codex review on PR #1464:** an earlier version of this row and the test itself claimed to pin `/api/search`. They do not. The route's auth, rate limiting, scope resolution, related-document enrichment and telemetry write are all invisible to this suite, so a round trip added to any of them leaves it green — and the refusal budget below is about *retrieval*, not about an adversarial HTTP request, which still pays the route preamble. **The measured shape is itself the finding:** one search costs **11 round trips** — `rag_aliases` 1, `match_document_chunks_text_v2` **3**, `match_document_table_facts_text_v2` **3**, `get_related_document_metadata_v2` 1, `document_index_quality` 1, `document_images` 2 — so the two text RPCs are each issued three times per search. Pinned by total *and* breakdown, because a refactor swapping one probe for an unrelated query would keep the total at 11 while changing the traffic. Deterministic across three consecutive runs. The refusal budget asserts **zero** Supabase traffic, matching `rag.ts`'s claim that prompt-injection intent is refused before any query issues, and was proven against the broken shape: with a non-refused query it fails on the round-trip assertion (`expected 11 to be +0`), which is why that assertion is ordered ahead of the results assertion. **Refuted 2026-07-30:** collapsing both text surfaces to the canonical expanded query reduced the offline budget from 11 to 7, but live canary `30579804611` regressed document recall from 1.0 to 0.9444 and content recall from 1.0 to 0.9699 across four cases. The change was immediately reverted in `a0cd00b`; confirmation canary `30580564419` passed. The ×3 sibling variants therefore carry real recall and must not be removed wholesale. **Next:** (a) add the route-level budget this suite does not provide — drive `POST` from `src/app/api/search/route.ts` with counted clients, following the `tests/answer-route-preamble.test.ts` pattern, so a round trip added to the route preamble or post-processing is a red gate; (b) decide whether `match_document_chunks_text_v2` ×3 and `match_document_table_facts_text_v2` ×3 per search are intended or a probe that should be collapsed — a latency question this budget surfaced but does not answer, and one that touches retrieval, so any change needs the usual RAG gate. (c) `scripts/eval-rag-offline.mjs` and `scripts/test-rag-offline.mjs` remain unwired; the offline contract runner is now the de-facto single home for budgets, so either adopt that explicitly here or wire them. | `docs/audit/latency-audit-2026-07-28.md` measurement plan; `src/lib/server-timing.ts`; `src/lib/answer-stream-contract.ts:18-21` | 2026-07-29 | | #099 | P2 | task | Remove the remaining fixed per-request round trips | **Outcome:** the answer path stops paying avoidable per-request Supabase round trips. **Done 2026-07-29:** shared-cache-hit promotion deferred off the response path with its mid-request staleness guard intact and documented (`rag.ts:3234`, `rag-cache.ts`); scope resolution overlapped with the rate-limit RPC, signal threaded so a client disconnect finally cancels its paginated queries (`answer/route.ts`). **REFUTED on PR #1377 review — do not retry:** the same pass also overlapped scope with the rate-limit RPC and aborted it on deny, claiming the limiter could "deny for free". It cannot. With caller-supplied `filters` or explicit ids, scope passes its zero-query early returns (`search-scope.ts:242,253`) into the paginated `documents` loop at `:269`, and an `AbortSignal` cancels the client request without un-executing a statement Postgres already began — so throttled traffic kept burning database capacity while collecting 429s, against `capacity-review.md:106-113`'s first-soft-failure warning. Scope is behind admission again, pinned by `tests/answer-route-preamble.test.ts`. Re-attempting the overlap requires a non-database admission gate ahead of the durable limiter first. **Remaining:** (a) the 8 `setCachedSearch` awaits — deferring changes `throwIfAborted` semantics and widens a real mutation window because the clone happens after an `await`, so each branch needs discharging individually; (b) batch the anonymous subject+global rate-limit pair, which needs a NEW atomic RPC modelled on `consume_summary_rate_limits_atomic` and cannot be called until the operator applies it — `Promise.all` is the WRONG fix because it consumes the global bucket even when the subject bucket already denied; (c) stop the proxy and route handler resolving identity twice per authenticated request — no in-process memo can do this (different `Request` objects), so the proxy must forward unspoofable verified claims via a header it controls. Cross-references #011: halving auth resolutions eases the ~10-connection Auth cap that `capacity-review.md:106-113` calls the first hard failure. | `docs/audit/latency-audit-2026-07-28.md` L1-1/L1-3/L1-4; `src/lib/api-rate-limit.ts:276-282`; `src/proxy.ts:125` | 2026-07-29 | | #100 | P2 | rec | Buffered answer generation has no incremental verified delivery | **Design complete; runtime work remains provider-gated.** [`verified-answer-incremental-delivery-design.md`](verified-answer-incremental-delivery-design.md) records the clinical-governance decision and staged contract: keep the `progress`/`final`/`error` allowlist; disclose bounded, owner-scoped evidence only after the canonical danger-level source-governance refusal permits it, then emit complete answer sections only after each reuses the full production verification boundary; reconcile every preview byte-for-byte with the authoritative `final`; discard all previews on error/cancel/retry; deploy behind separate parse/emission/render flags. Phase 0 contract proof and Phase 1 evidence preview can be developed offline, but visible rollout still needs clinical/browser proof. Phase 2 changes generation architecture and requires explicit approval for answer-quality evals plus a baseline/post live canary pair. **Naive token streaming remains REFUTED:** never re-land `token`, `revising`, provisional prose, or a weaker stream-only verifier. Cross-references #021. | `docs/verified-answer-incremental-delivery-design.md`; `docs/audit/latency-audit-2026-07-28.md` L0-1; `src/lib/answer-stream-contract.ts:18-21` | 2026-07-30 | | #101 | P3 | rec | Canary-gated retrieval parallelisation candidates | **Outcome:** independent retrieval stages stop running serially, proven by a live canary pair. Candidates: metadata/memory/visual hydration triples repeated on four branches (`rag.ts:2460,2493,2521` and three more) while `rag.ts:2751-2804` already parallelises three RPCs in one `Promise.all`, so the omission is inconsistency rather than intent; the nested `await`-in-loop scope enumeration (`search-scope.ts:202,328`); typeahead results never cached (`rag.ts:2698-2711`); universal-search coalescing (`/api/search` has it, `/api/search/universal` does not). Each changes candidate assembly, truncation, or what the next keystroke returns, so each needs 36/36 retrieval plus recall 1.0 and zero per-case rr regressions. Distinct from #001 (semantic rerank). Resolved #075 and #083 are the precedents for why these are gated rather than free. **Stop:** needs the #098 harness and explicit canary approval first. | `docs/audit/latency-audit-2026-07-28.md` L2-1/L2-2/L2-8/L1-5 | 2026-07-29 | From ed4afc44355115892a45a8e04d69bcdf799c1078 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 31 Jul 2026 00:13:56 +0000 Subject: [PATCH 2/2] docs: record PR #1503 reopen-ready review Append the branch-review ledger verdict for the main-synced tip that keeps the #098 wholesale-collapse ban. Co-authored-by: BigSimmo --- docs/branch-review-ledger.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 02fb182afc..ba7627d9a4 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -329,3 +329,6 @@ Records before 2026-07-28 were written by hand and had drifted: 146 lines carrie | 2026-07-31 | claude/root-dir-coverage-gate-v2 | 398660144d93aeefc2e5649c156948a68925cb64 | docs:check-index repo-root coverage, stale script counts, ledger correction | MERGED as PR #1458 (squash 907fd9f4a). Root-directory coverage pass for docs:check-index, red-then-green proven (flagged .cursor/.design-sync/.vscode, then 49 entries vs 31). Main landed an equivalent pass independently in #1480, so the two overlapped; no duplication reached main. Row not recorded at the time - appended retrospectively | verify:cheap exit 0, 435 test files / 4574 tests pass; codebase-index-coverage 10/10 incl 4 new root cases; eslint clean; docs gates green; prettier clean | | 2026-07-31 | claude/pre-commit-fail-open | 7b96a09b8500adc917cf5549b1c61142b2244b39 | pre-commit hook fail-open when the inventory script is absent | MERGED as PR #1494 (squash 387c3b653). Resolves ledger #153: core.hooksPath is absolute to the primary checkout, so the hook ran in worktrees lacking scripts/update-docs-inventory.mjs and aborted with MODULE_NOT_FOUND. Guard drops the inventory task and re-checks the all-tasks-empty exit; grep carries \|\| true because set -e treats a fully-filtering grep as failure | isolated-repo probe with the script genuinely absent: prints skipping inventory sync, commit succeeds; sh -n clean; no-op when the script is present; prettier does not parse shell so format:check skips it | | 2026-07-31 | claude/ledger-relanding | 30ec06964e4235d9f0b4bb782f357e6b4fb59430 | re-land the three session findings lost when PR #1490 was closed | MERGED as PR #1508 (squash 7b551abc4). Ledger-only: #151 corrects the claim that CI is unreadable (PAT has Actions:read though not Checks:read), #152 re-lands the at-risk worktree inventory with the four preservation snapshots, #153 archives the hook fix. Verified landed by content on main, not by PR state or row id | CI, PR Policy, PR mergeability, SAST, Secret Scan all completed/success via the Actions API; check:outstanding-issues 151 rows 45 open unique ids next-id=154; docs:check-links 1414 refs; prettier clean | +| 2026-07-31 | codex/fix-search-retrieval-issues-and-run-canary | c45cd227be6d852865eb111a2a06486d1f2c62a0 | pr-1503 reopen-ready | approved: #098 next-action (b) prohibits wholesale collapse; main synced clean; docs-only delta; PR stays closed | format:check; check:outstanding-issues; check:branch-review-ledger; merge-tree clean; bugbot+codex P2 fixed | +| 2026-07-31 | codex/fix-search-retrieval-issues-and-run-canary | c45cd227be6d852865eb111a2a06486d1f2c62a0 | pr-1503-review | re-reviewed: Codex P2 fixed; merge-tree clean vs origin/main; docs-only; no Bugbot/P0-P1; PR remains CLOSED (GitHub headRefOid may lag closed PR) | merge-tree+diff-vs-main+#098-text; no push/reopen | +| 2026-07-31 | codex/fix-search-retrieval-issues-and-run-canary | e8de1aebd1b32d5e901853a3473a792a66aa82cf | pr-1503 reopen-ready | approved: #098 collapse ban kept after main sync; merge-tree clean; docs-only; PR stays closed | format:check; check:outstanding-issues; merge-tree clean; codex P2 fixed; bugbot clean |