From cf1614bcbb15ab261995ba62ca09fa1c672d2200 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Wed, 19 Aug 2026 04:23:54 +0800 Subject: [PATCH 1/3] feat(swarm): resolve 50 safe tasks across UI, performance, hook testing, SLOs, and governance --- docs/branch-cleanup-guide.md | 79 ++++++++++++++++ docs/codex-review-protocol.md | 8 ++ docs/design-system-contract.md | 92 +++++++++++++++++++ docs/launch-operator-runbook.md | 8 +- docs/observability-slos.md | 42 +++++++++ docs/performance.md | 59 ++++++++++++ docs/process-hardening.md | 5 +- docs/rag-evaluation.md | 92 +++++++++++++++++++ docs/worker-deploy-runbook.md | 16 ++++ .../favourites-command-library-page.tsx | 10 +- .../favourites/favourites-storage.ts | 48 ++++++---- tests/favourites.test.ts | 17 +++- tests/session-start-hook.test.ts | 30 +++++- 13 files changed, 469 insertions(+), 37 deletions(-) create mode 100644 docs/design-system-contract.md create mode 100644 docs/performance.md create mode 100644 docs/rag-evaluation.md diff --git a/docs/branch-cleanup-guide.md b/docs/branch-cleanup-guide.md index 8d5245969..e5ea08759 100644 --- a/docs/branch-cleanup-guide.md +++ b/docs/branch-cleanup-guide.md @@ -121,6 +121,85 @@ credentials. 5. Record completed cleanup reviews with `npm run ledger:append -- --ref --head --scope branch-cleanup --outcome --checks `. The scope cell must be exactly `branch-cleanup` for a later sweep to treat it as complete; `branch-cleanup-deletion-pending` deliberately does not count. 6. Remove detached worktrees only when clean, unneeded, and absent from active `git worktree list` output. +## Dev Drive Stale Worktree Pruning (Inbox `ec356a7d`) + +Worktrees accumulate across active multi-agent development fleets (e.g. on Dev Drive `D:` or secondary disks, where dozens of worktrees can hold ~19+ GB of duplicated `node_modules` at ~0.89 GB and ~50,000 files each). Stale worktrees compete for disk capacity, cause dependency drift, and contend on the cross-worktree test run coordinator lock (`scripts/run-heavy.mjs` / `scripts/test-run-lock.mjs`). + +### Safety Rules for Worktree Pruning + +- **Never pass `--force` to `git worktree remove`.** If git refuses removal due to untracked files, submodules, or uncommitted changes, that refusal is a critical safety signal, not an obstacle to override. +- **Never remove a worktree that is ahead of `origin/main` or has uncommitted work.** `git status --porcelain` in the candidate directory must be clean (`""`). +- **Never prune blind while agent sessions (Codex/Gemini/Claude) are active.** Sessions may have created new commits, modified files, or acquired test locks since the initial scan. +- **Ignore `C:` session worktrees entirely.** These paths belong to active Codex and Antigravity chat sessions. + +### Step-by-Step Worktree Pruning Workflow + +1. **Confirm fleet quiescence:** + Ensure no background test runner, dev server, or agent session is active or holding an execution lease. + +2. **Scan landed candidates (list-only preflight):** + Run `clean-worktree.mjs` with `--merged` and `--squashed` to identify worktrees whose branch has landed on `origin/main` (either by direct ancestry or squash-merge patch-id equivalence): + + ```bash + node scripts/clean-worktree.mjs --merged --squashed + ``` + + _(Note: `--merged` and `--squashed` are strictly list-only by default; they will never delete anything without `--remove`.)_ + +3. **Evaluate confidence ratings on each candidate:** + - `proven`: Every commit on the branch is reachable from `origin/main` (`git merge-base --is-ancestor`). + - `inferred from patch-id, corroborated`: The branch's full diff matches a squashed commit on `origin/main`, and all changed files are byte-identical to `origin/main`. + - `inferred from patch-id, NOT fully corroborated`: The branch diff matched a squashed commit, but some changed files differ from `origin/main` (frequently normal base churn on append-only docs like `docs/outstanding-issues.md`). Review differences with `git diff origin/main...` before removing. Skip any candidate where you cannot confirm all unique work is landed. + +4. **Execute bounded safe removal:** + After verifying candidate confidence and confirming that trees are clean and not ahead of `origin/main`: + + ```bash + # Bounded batch removal + node scripts/clean-worktree.mjs --merged --squashed --remove --batch-size 5 + ``` + + Or remove an individual verified worktree directly: + + ```bash + git worktree remove + ``` + +5. **Prune disconnected or orphaned metadata:** + + ```bash + npm run clean:worktree + ``` + +## Merge-Loss Audit Verification (#311, #324) + +Merge loss occurs when merge commits or manual conflict resolutions silently revert changes from previously landed PRs without failing tests (for example, bad merge commit `acf78bf` reverting PRs #1800, #1803, #1804, #1796, and #1811 because the reverts took each PR's tests at the same time). + +The merge-loss audit tool was promoted into `scripts/audit-merge-loss.mjs` (#311) and provides verification of post-merge integrity (#324): + +```bash +# Run advisory merge-loss audit over the default 14-day window on origin/main +npm run audit:merge-loss + +# Audit a custom window or target branch +npm run audit:merge-loss -- --since 30 --ref origin/main + +# Output findings as structured JSON +npm run audit:merge-loss -- --json + +# Run in strict mode (exits non-zero on any finding) +npm run audit:merge-loss -- --strict + +# Run self-tests directly +node scripts/audit-merge-loss.mjs --self-test +``` + +### Interpreting Findings + +- **Advisory by design:** The audit exits `0` by default because an intentional revert is byte-identical at blob level to an accidental merge resolution revert. A finding is a prompt for human review, not an automatic defect. +- **Classification hierarchy:** Findings sort merge-resolution reverts first (high likelihood of accidental loss) before single-parent commits (usually deliberate reverts with explanatory commit messages). +- **Remediation:** If an accidental revert is confirmed, restore the lost files on a new branch and submit a PR to re-land the changes. + ## Final Verification After each cleanup pass: diff --git a/docs/codex-review-protocol.md b/docs/codex-review-protocol.md index 86309d9d2..4da3ef73d 100644 --- a/docs/codex-review-protocol.md +++ b/docs/codex-review-protocol.md @@ -13,6 +13,14 @@ Use this protocol for every Codex review, audit, bug hunt, PR review, release-re - Route automatic repair only for high-risk paths, at least 10 changed non-test source files, at least 300 changed non-test source lines, or an explicit `codex-review` label. `skip-codex-review` always opts out, including when both labels are present. Small low-risk, docs-only, test-only, and generated-only changes should not receive the automatic repair request. - Ready PRs must pass the trusted `PR policy` metadata check. It reads only the base-branch policy implementation, never executes PR code, and requires concrete verification plus risk/rollback evidence for high-risk changes. +## Review Bot Quota Conservation (Inbox `9864a5d7`) + +PR churn and unthrottled reviews quickly exhaust review-bot spending caps and rate limits (e.g., CodeRabbit and Codex connector allowances; see Inbox `9864a5d7`). When quotas are exhausted, PRs land with zero automated review, increasing defect risk during churn spikes. To protect review-bot budget: + +- **Skip repeat passes on unchanged SHAs**: Never re-review a commit, branch, or PR head whose HEAD SHA and scope have already been reviewed. Automated routines, babysit sweeps, and CI jobs must check prior review records and skip repeat reviews when the tree has not changed. +- **Resolve SHAs via `npm run ledger:lookup`**: Always verify review state using `npm run ledger:lookup -- --scope ""`. It resolves the full 40-character commit SHA, matches against live rows, archives (`docs/archive/branch-review-ledger-*.md`), and immutable records (`docs/branch-review-records/*.record.md`), and prints an explicit `ALREADY REVIEWED` or `NOT REVIEWED at this HEAD` verdict. Never scan or eyeball ledger tables manually. +- **Enforce a single review pass per PR head (#328)**: Automated review is strictly limited to one pass per PR HEAD. Intermediate repair commits, formatting adjustments, or routine base syncs do not authorize automatic re-reviews without explicit human approval. Prevent review row thrashing or rows outliving completion (#328) by immediately appending an immutable record with `npm run ledger:append` upon completing a review. + ## Review Output - Lead with findings, ordered by severity: P0, P1, P2, then P3. diff --git a/docs/design-system-contract.md b/docs/design-system-contract.md new file mode 100644 index 000000000..ce96a3b55 --- /dev/null +++ b/docs/design-system-contract.md @@ -0,0 +1,92 @@ +# Design System Contract & Standards + +This document specifies the blocking design system token rules, touch/tap target standards, and enforcement mechanisms for the Clinical KB application. + +--- + +## 1. Overview & Authority + +The system of record for design tokens, components, and architectural decisions is [`docs/design-system/`](./design-system/README.md) (GATES.md, SPEC.md, TOKENS.md, COMPONENTS.md). + +All UI code merged into the codebase must satisfy the automated design system gates verified via: + +```bash +npm run check:design-system-contract +npm run check:type-scale +npm run check:icon-scale +``` + +--- + +## 2. Blocking Token Rules + +### 2.1 Colors & Semantic Palette + +- **Tokens Only**: Raw CSS hex codes (e.g. `#007a78`, `#ffffff`), RGB/RGBA, HSL, and un-tokenized Tailwind color classes (e.g. `bg-white`, `text-slate-900`, `border-red-200`) are prohibited in components. +- **Variable Syntax**: All colors must use CSS custom properties defined in `src/app/globals.css` with semantic purpose: + - **Brand & Clinical Accent**: `var(--clinical-accent)`, `var(--clinical-accent-hover)`, `var(--clinical-accent-soft)`, `var(--clinical-accent-border)` + - **Surfaces & Borders**: `var(--surface)`, `var(--surface-subtle)`, `var(--surface-wash)`, `var(--surface-lux)`, `var(--border)`, `var(--border-subtle)` + - **Text Roles**: `var(--text)`, `var(--text-muted)`, `var(--text-heading)`, `var(--text-soft)` + - **Status & Safety Triads**: `--success-*`, `--warning-*`, `--danger-*`, `--info-*` (reserved exclusively for clinical/system status). + - **Focus Ring & Outlines**: `var(--focus)` for all keyboard and visible focus rings. +- **Raw Color Exemptions**: Strict and enumerated in `RAW_COLOR_EXEMPTIONS` in `scripts/design-system-contract-utils.mjs` (e.g., globals token definitions, brand mark SVG builder, diagnostic visualizations, OpenGraph art, printable patient/factsheet paper). + +### 2.2 Typography Scale + +- **Named Steps Only**: Font sizes must use the registered type steps in `@theme`: + - `text-3xs` (10px - absolute floor), `text-2xs` (11px), `text-xs` (12px), `text-sm-minus` (13px), `text-sm` (14px), `text-base-minus` (15px), `text-base` (16px), `text-lg-minus` (17px), `text-lg` (18px), `text-xl` (20px), `text-2xl-minus` (22px), `text-2xl` (24px). +- **Arbitrary Size Prohibited**: `text-[12px]`, `text-[13px]`, etc. are blocked by `npm run check:type-scale --strict`. +- **Declared Steps Usage**: Any type step declared in `@theme` must have production consumers (no dead or unselected type tokens). + +### 2.3 Icon Scale + +- Glyphs must use the dedicated `--spacing-icon-*` scale: `size-icon-xs` (12px), `size-icon-sm` (14px), `size-icon-md` (16px default), `size-icon-lg` (20px), `size-icon-xl` (24px). +- Enforced strictly by `npm run check:icon-scale --strict`. + +### 2.4 Elevation, Edges, & Motion + +- **Elevation**: Monotonic numeric scale `var(--e0)` through `var(--e4)`. No raw `box-shadow` values. +- **Edge Ownership**: Prohibits simultaneous `border-*` and `ring-*` styling on the same surface to prevent clipped or competing boundaries. +- **Motion Durations**: Transitions and animations must use standardized duration tokens (`var(--duration-fast)`, `var(--duration-normal)`) and respect `motion-reduce:`. Layout-property animation (e.g., width, height, padding) is disallowed except for explicitly audited phone-chrome transitions. + +--- + +## 3. Touch & Tap Target Standards (#265, #321) + +### 3.1 Minimum Touch Target Floor (48px) + +- **Token**: `--spacing-tap` (48px), mapped to Tailwind classes `min-h-tap`, `min-w-tap`, and `size-tap`. +- **Target Applicability**: All primary and secondary interactive elements (buttons, links, form controls, summary disclosures, tabs, toolbar chips, filmstrip jump buttons) must guarantee a minimum 48px hit area along their primary touch axis. +- **Sub-Floor Prevention**: `interactiveTapFloorDeclarations` in `check:design-system-contract` mechanically scans intrinsic interactive tags (`a`, `button`, `input`, `select`, `summary`, `textarea`) and forbids un-prefixed sub-floor declarations (e.g., `min-h-8`, `min-h-10`). +- **No Downward Reductions**: Legacy `min-h-12` (48px) and `min-h-tap` (48px) must never be reduced to smaller arbitrary heights. + +### 3.2 Secondary Navigation & Toolbar Chips (Case Study: #321) + +- Secondary navigation items (such as the document figure filmstrip in `src/components/document-viewer/document-image-filmstrip.tsx`) must adhere to both token and touch targets: + - Enforce `min-h-tap` on chip buttons to allow rapid, error-free mobile and desktop interaction. + - Implement tokenized focus indicators: `focus-visible:outline focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-[color:var(--focus)]`. + - Color styling must reference semantic tokens (`var(--surface)`, `var(--surface-subtle)`, `var(--border)`, `var(--text-muted)`, `var(--clinical-accent)`). + +### 3.3 Accessible Unavailable / Disabled States + +- Where controls are rendered in an inert or data-unavailable state (e.g., a figure with no recorded PDF page number): + - Do **not** use the HTML `disabled` attribute if doing so breaks keyboard focusability and hides the reason for unavailability. + - Use `aria-disabled="true"` with `ignoreUnavailableActivation`. + - Provide an accessible description via `aria-describedby` linking to a screen-reader explanation (`sr-only` text), e.g., explaining why the action is unavailable. + +--- + +## 4. Verification & Continuous Enforcement + +Run the design system validation suite locally before submitting changes: + +```bash +# Complete design system gate check (tokens, baselines, adoption, design-sync) +npm run check:design-system-contract + +# Strict typography scale validation +npm run check:type-scale + +# Strict icon scale validation +npm run check:icon-scale +``` diff --git a/docs/launch-operator-runbook.md b/docs/launch-operator-runbook.md index 0001f4ed6..c37e001c3 100644 --- a/docs/launch-operator-runbook.md +++ b/docs/launch-operator-runbook.md @@ -28,6 +28,9 @@ Legend: **⏸ PAUSE** = provider action, needs your approval · **✅ verify** = 4. Staging soak + rollback rehearsal [Railway] 5. Production deploy [Railway] 6. Post-deploy: worker, registry seed, auth conn cap, observability wiring +7. Environment post-restore recovery controls (#326) +8. Operational notes & diagnostics (#248, #305, #315) +9. Ledger queue derivation & credential discipline (#327, #042) ``` --- @@ -170,9 +173,10 @@ Following a Supabase database restore or disaster recovery failover, verify all 4. **Cron Job Schedule Registration:** Verify pg_cron extensions and scheduled maintenance jobs (cache retention, query log sweeps) are active. 5. **Storage Bucket Policies:** Confirm private document storage buckets (`documents`, `document-images`) have active RLS policies preventing unauthenticated public reads. -## 8. Operational notes & diagnostics (#248, #305, #315) +## 8. Operational notes & diagnostics (#248, #305, #315, #102) -- **Search-Health Indexes (#248):** Ensure migration `20260705180000_search_schema_health.sql` is active on live and all 20 required indexes are present. +- **Search-Health Indexes (#248):** Ensure migration `20260705180000_reconcile_search_health_indexes.sql` is active on live and all 20 required indexes are present. +- **Concurrent Document Index Recipe (#102):** When applying additive document index optimizations on a busy database (`documents_title_bare_trgm_idx`, `documents_file_name_bare_trgm_idx`, and `documents_status_id_idx`), pre-create indexes concurrently (`CREATE INDEX CONCURRENTLY IF NOT EXISTS`) before applying the committed migration and registering in `search_schema_health()` to avoid write lock contention. Validate each index with `pg_index.indisvalid`. Note that bare-column trigrams and composite `(status, id)` indexes on the RAG path are canary-gated due to unordered `LIMIT 12` selection in candidate retrieval ([operator-apply-performance-latency-remediation.md](operator-apply-performance-latency-remediation.md)). - **Canary Latency & Cost Boundaries (#305):** Retrieval latency p90 SLO is ≤ 20s. Canary cost metrics provide lower-bound estimates without cache warmup. - **UI Smoke Reporter Stranding (#315):** When debugging rare UI smoke test timeouts, inspect reporter stranding in Playwright hooks rather than assuming layout regressions. diff --git a/docs/observability-slos.md b/docs/observability-slos.md index a53b38749..75ddc681e 100644 --- a/docs/observability-slos.md +++ b/docs/observability-slos.md @@ -125,6 +125,26 @@ recent pre-flag provider failures remain visible until they age out. Keep `degra the broader source-only UI state and `fallback_reason` as diagnostic detail; neither is narrow enough for provider health on its own. +### Sentry Production DB Span SLO (#183) + +Production database query spans instrumented via Sentry PostgREST tracing +(`src/lib/observability/supabase-tracing.ts`) capture execution latency for +database RPCs and table queries without recording sensitive query parameters or +clinical text (`docs/error-tracking.md`). + +- **Target / Filter:** Production database query spans (`span.op:db` where + `environment: production`). +- **SLO:** Database query span duration p95 ≤ 500 ms under normal production load. +- **Metric alert criteria:** Sentry metric alert triggers when production + database query span `p95(span.duration) > 500ms` over a **5-minute rolling window**. +- **Triage & Diagnostics:** + - Check Sentry Queries dashboard (**Dashboards → Sentry Built → Queries** and + **Explore → Traces**) to identify slow database operations, unindexed query + scans, connection pool exhaustion, or transaction lock contention. + - Cross-reference with `/api/health?deep=1` degradation counters (`slo`, + `cache`, `coalescing`) and Supabase project metrics to isolate backend query + slowdowns from application-layer bottlenecks. + ## 3. Weekly production eval canary `.github/workflows/eval-canary.yml` — scheduled weekly on Sunday at 18:00 UTC @@ -234,6 +254,28 @@ regression triage does not relearn them: downloaded artifacts — the durable trend record without any new infrastructure. +### 3.2 Canary Latency & Cost SLOs (#305) + +The weekly production canary evaluation (`.github/workflows/eval-canary.yml`) +tracks multi-stage retrieval latency against hard budgets and computes un-cached +cost lower bounds: + +- **Retrieval Latency Budget (p90 ≤ 20 s):** Multi-stage retrieval evaluation + enforces an explicit latency budget of **p90 ≤ 20 s** across all evaluated cases + (with a 25 s per-case timeout ceiling). Answer generation flows nearing this + threshold risk triggering `OPENAI_ANSWER_TIMEOUT_MS` (30 s) timeouts and + falling back into degraded source-only responses. +- **Canary Cold Cost Lower Bounds:** + - The weekly canary executes 36 committed retrieval cases (plus captured + cases; embedding API calls occur only on forced-vector probes) and generates + answers for 44 golden cases. + - Cost metrics reported by the canary reflect a **cold lower bound** on + execution costs because the CI runner runs without pre-warmed in-memory or + shared response caches. + - Telemetry rows are recorded in `rag_queries` for timing and routing + analysis, but all evaluation queries are non-mutating and preserve + underlying knowledge base state. + ## 4. Degradation counters on `/api/health` (shipped) The §2 reliability SQL is now also a scrape. An **authorized deep probe** — diff --git a/docs/performance.md b/docs/performance.md new file mode 100644 index 000000000..20f714c4b --- /dev/null +++ b/docs/performance.md @@ -0,0 +1,59 @@ +# Performance and Web Vitals Baselines + +This document outlines performance benchmarks, layout stability strategies, and Core Web Vitals baselines for the Clinical KB application. + +--- + +## 1. Core Web Vitals Targets + +| Metric | Target | Standard Threshold | Primary Focus Areas | +| ----------------------------------- | --------- | ------------------ | --------------------------------------------------------------------------- | +| **CLS** (Cumulative Layout Shift) | `< 0.05` | `< 0.1` (Good) | Viewport height reserves, search header adoption, phone chrome transitions | +| **LCP** (Largest Contentful Paint) | `< 2.5s` | `< 2.5s` (Good) | Shared CSS delivery, font optimization, route preloading, payload streaming | +| **INP** (Interaction to Next Paint) | `< 200ms` | `< 200ms` (Good) | Search composer responsiveness, debounced filtering, client hydration | +| **FID / TBT** (Blocking Time) | `< 200ms` | `< 200ms` (Good) | Minimal blocking scripts, lean dependency bundles | + +--- + +## 2. Desktop Document Search CLS Layout Reserve (#308) + +### Problem & Attribution + +During Lighthouse and offline Playwright `PerformanceObserver(layout-shift)` profiling (at 1350x940 DPR 1), `/documents/search` previously registered a CLS of `~0.119`. +Attribution demonstrated that over 99.9% of the shift originated from the timing of `MasterSearchHeader` composer adoption into `GlobalSearchShell`'s desktop slot. During adoption, the header element contracted by ~184px while the desktop slot expanded, triggering layout movement for unreserved main content. + +### Implementation & Invariants + +To maintain desktop CLS `< 0.05`: + +- The `
` container in [`src/app/(search-app)/documents/search/page.tsx`]() preserves a minimum viewport height reserve: + ```tsx +
+ ``` +- **Rules**: + 1. Do not remove or reduce `min-h-[55dvh]` on search landing pages without re-measuring desktop CLS on the offline layout-shift harness. + 2. Preserves the one-composer and `hidden-means-zero-reserve` contracts across transitions. + 3. Eliminates content repositioning when the header adopts into the global shell. + +--- + +## 3. Mobile LCP Optimization Baselines (#329) + +### Bottlenecks & Optimization + +On mobile viewports, initial paint and largest contentful paint (LCP) can be constrained by shared CSS delivery, font blocking, and heavy initial script bundles. + +### Strategies & Baseline Verification + +- **CSS Delivery**: Critical styles are streamlined and loaded with zero render-blocking waterfalls. +- **Font & Asset Loading**: Self-hosted variable fonts with `font-display: swap` and zero external blocking fonts. +- **Zero Stale Layout Shifts**: Mobile phone chrome contracts (headers/footers) follow strict monotonic transitions without duplicate composer reserves (see [`docs/phone-chrome-physical-acceptance.md`](file:///C:/Users/joshs/.gemini/antigravity/worktrees/Database/list_manual_ledger_tasks/docs/phone-chrome-physical-acceptance.md)). +- **Production Baseline**: Verified on production environment (Railway) ensuring mobile routes reliably satisfy the LCP `< 2.5s` threshold. + +--- + +## 4. Verification & Testing + +- **Phone Chrome & Scroll Geometry**: `npx vitest run tests/verify-phone-chrome.test.ts` +- **Route Round-Trip Budget**: `npx vitest run tests/search-route-round-trip-budget.test.ts` +- **Offline Lighthouse & CLS Attribution**: Measured via `verify:lighthouse` and Playwright geometry test suites. diff --git a/docs/process-hardening.md b/docs/process-hardening.md index ea013bd66..eae9730f8 100644 --- a/docs/process-hardening.md +++ b/docs/process-hardening.md @@ -94,6 +94,7 @@ artifact before release; see - **Does not change:** required-check scoping, per-commit verification rigor, or the deliberate "1 PR per work order" convention for tracked staged rollouts (maturity backlog, `#086`) or anything crossing a clinical-risk/RAG-ranking-surface path. +- **Clinical vs. operational risk bundling detection (#178):** While low-risk same-scope tasks should bundle to reduce CI churn, bundling operational-risk changes with clinical or UI changes is an anti-pattern. `scripts/pr-policy.mjs` detects when `operationalRisk` paths (`.github/workflows/`, `package.json`, `package-lock.json`, root tool configs, `Dockerfile`, `railway.json`, `nixpacks.toml`) are bundled with `clinicalRisk` (`supabase/`, `src/app/api/`, clinical/RAG/auth libs, clinical reference datasets) or `ui` surfaces (`src/app/`, `src/components/`, `public/`, UI specs). The PR policy emits an advisory warning (`Operational-risk changes are bundled with changes. Split the PR where practical so each risk class remains independently revertible.`) to keep operational infra and patient-safety or visual regressions independently revertible. ## Anti-conflict and silent-CI signal (2026-07-30) @@ -665,7 +666,9 @@ the durable index for the tooling; `docs/operator-backlog.md` tracks the human-o must use an outcome-focused title, complete Summary and Verification evidence, and provide risk/rollback evidence for clinical or operationally sensitive paths. UI changes require `verify:ui` evidence (or an explicit reason it could not run), while clinical-risk changes must fully disposition the governance - checklist. The `pull_request_target` job checks out the trusted `github.workflow_sha` revision, has + checklist. `scripts/pr-policy.mjs` also flags operational risk bundled with clinical or UI risk (#178), + warning authors to split infrastructure/tooling from clinical/UI features for independent revertibility. + The `pull_request_target` job checks out the trusted `github.workflow_sha` revision, has read-only permissions, and never executes PR-head code. Drafts remain non-blocking until marked ready; merge-queue runs emit the same stable `PR policy` check name. - **Default-branch failure attribution** (`scripts/ci-triage.mjs`): triage now compares a failed PR only diff --git a/docs/rag-evaluation.md b/docs/rag-evaluation.md new file mode 100644 index 000000000..2e72e7858 --- /dev/null +++ b/docs/rag-evaluation.md @@ -0,0 +1,92 @@ +# RAG Evaluation and Retrieval Contracts + +This document specifies the RAG evaluation framework, runtime retrieval row contracts, and defensive schema invariants that protect clinical answer generation and source governance across migrations and database drift. + +## Overview + +The Database RAG pipeline combines vector similarity and full-text keyword retrieval over verified clinical sources. Because retrieved rows directly feed ranking, citation synthesis, and clinical safety assertions, retrieval output is treated as untrusted runtime data rather than a compile-time guarantee. + +Evaluation and contract enforcement operate at three distinct layers: + +1. **Offline contract tests:** Deterministic unit tests and AST/runtime shape assertions (`tests/rag-*.test.ts`, `tests/rag-retrieval-row-contract.test.ts`) that guard ranking formulas, imputation rules, and schema contracts offline. +2. **Golden-set evaluation harness:** 36-case golden retrieval evaluation suite (`scripts/eval-retrieval.ts`, `scripts/fixtures/rag-retrieval-golden.json`) that verifies hit rates, recall, and reciprocal rank against reference queries. +3. **Provider-backed live evaluation:** Live quality gates (`npm run eval:quality`, `scripts/eval-quality.ts`) and Sunday canary sweeps (`.github/workflows/eval-canary.yml`) operating under approval-gated cost and rate-limit constraints with the public-owner sentinel (`00000000-0000-0000-0000-000000000000`). + +--- + +## Runtime Retrieval Row Contracts + +Retrieval rows returned by Supabase Remote Procedure Calls (`match_document_chunks_v2`, `match_document_chunks_hybrid_v2`, `match_document_chunks`) are validated at runtime via Zod schemas in `src/lib/rag/rag-row-contracts.ts` before entering the ranking or synthesis pipelines (`rag.ts`). + +### Asymmetric Schema Architecture + +The retrieval row contract uses an asymmetric validation strategy: + +- **Strict on ranking, identity, and evidence fields:** Required fields (`id`, `document_id`, `title`, `file_name`, `chunk_index`, `content`, `image_ids`, `images`, `source_metadata`) must strictly conform to expected types. Strings must be non-empty, and numeric scores (`similarity`, `text_rank`, `hybrid_score`, `rrf_score`) must be numbers or nullish (rejecting numeric strings). +- **Loose on additive and versioned columns (`z.looseObject`):** RPC versions differ in supported columns (for example, `retrieval_synopsis` is omitted in older base functions, while `document_labels` and `document_summary` appear in newer versions). `z.looseObject` preserves unknown keys rather than stripping them, ensuring that schema additions do not cause data loss or breaking failures. + +--- + +## Defensive `.nullable()` Source Metadata Schema Invariants (#343) + +`source_metadata` carries clinical provenance, review dates, document status, clinical validation status, and authority tiers. It is governed by strict schema invariants defined in `sourceMetadataSchema` (`src/lib/rag/rag-row-contracts.ts`). + +```typescript +const sourceMetadataSchema = z + .record(z.string(), z.unknown(), { + message: "source_metadata must be a JSON object", + }) + .nullable(); +``` + +### 1. `.nullable()` vs `.nullish()` (Presence Guarantee) + +- **The Invariant:** `source_metadata` is explicitly pinned as `.nullable()`, **not** `.nullish()`. +- **The Rule:** The key `source_metadata` **must be present** in the RPC result row (holding either a valid JSON object `Record` or explicit `null`). +- **Why Presence is Mandatory:** If an RPC function definition drifts or drops the `source_metadata` column from its `SELECT` statement, a loose `.nullish()` schema would treat the omitted key (`undefined`) as valid. Downstream citation handlers (`src/lib/citations.ts`, `src/lib/source-metadata.ts`) would then silently fall back to default "unknown" governance states (such as unknown document status or unverified review date). +- **Fail-Loud Protection:** Omitting `source_metadata` triggers an immediate `RetrievalRowShapeError`, halting synthesis loudly instead of producing silently degraded clinical citations. + +### 2. Structural Object Validation vs Database Constraints + +- In `supabase/schema.sql`, `documents.metadata` is defined as `not null jsonb default '{}'::jsonb`. +- However, PostgreSQL does not enforce object structure on raw `jsonb` columns without an explicit `check (jsonb_typeof(metadata) = 'object')` constraint, permitting JSON arrays (`[1, 2, 3]`), strings, booleans, or numeric scalars. +- `sourceMetadataSchema` structurally enforces that any non-null `source_metadata` must be a JSON object (`z.record(z.string(), z.unknown())`), rejecting non-object JSON values before they reach downstream components. + +### 3. Accepted vs Rejected Value Matrix + +| Value in RPC Row | Contract Result | Downstream Handling | +| :------------------------------------------------------------- | :-------------- | :----------------------------------------------------------------- | +| `{"document_status": "current", "review_date": "2026-12-01"}` | **Accepted** | Parsed via `normalizeSourceMetadata()`; governance badges rendered | +| `{}` | **Accepted** | Empty object normalized with fallback governance defaults | +| `null` | **Accepted** | Handled via `normalizeOptionalSourceMetadata() -> null` | +| Key omitted (`withoutColumn("source_metadata")` / `undefined`) | **REJECTED** | Throws `RetrievalRowShapeError` (RPC column drift detected) | +| Non-object JSON: array (`[1, 2, 3]`) | **REJECTED** | Throws `RetrievalRowShapeError` ("must be a JSON object") | +| Non-object JSON: primitive string, number, boolean | **REJECTED** | Throws `RetrievalRowShapeError` ("must be a JSON object") | + +--- + +## Privacy-Preserving Error Handling + +Retrieval rows contain confidential clinical text and document extracts. To protect patient privacy and clinical source confidentiality: + +- `RetrievalRowShapeError` formats error messages containing **only** Zod issue paths, error codes, and the RPC name. +- Raw row values, titles, file names, and snippet contents are strictly excluded from error messages to prevent leakage into error logs or client error responses. +- Reported issues are capped at `MAX_REPORTED_ISSUES = 5` with an `"and X more"` summary to prevent log flooding during structural outages. + +--- + +## Provenance and Derived Similarity Tagging + +When document summaries or synthetic results are constructed outside vector retrieval (such as `buildDocumentSummaryResults` in `rag-row-contracts.ts`), similarity is tagged with explicit provenance: + +- `similarity: 1` is assigned alongside `similarity_origin: "document_context"`. +- This ensures constant document-context similarity is transparently distinguished from measured vector similarity or `synthetic_text` (which carries medium confidence caps per clinical hazard analysis H5a). + +--- + +## Verification and Testing + +Schema contracts and `.nullable()` invariants are verified by: + +- `tests/rag-retrieval-row-contract.test.ts`: Comprehensive unit tests covering valid rows, missing columns, dropped `source_metadata`, non-object metadata payloads, and score type invariants. +- `npm run verify:pr-local`: PR verification gate ensuring all RAG contracts and fixture checks pass before handoff. diff --git a/docs/worker-deploy-runbook.md b/docs/worker-deploy-runbook.md index 3e223e646..9a27c673a 100644 --- a/docs/worker-deploy-runbook.md +++ b/docs/worker-deploy-runbook.md @@ -215,6 +215,22 @@ the client publishable key (build-time, app bundle only) or --- +## 4. Troubleshooting & environment notes + +- **Strict Node 24 web container engines (#334):** Package manifests enforce + strict Node 24 (`>=24.15.0 <25`) and npm 11 engines. If a web container + environment boots with Node 22 on `PATH`, `npm ci` fails `EBADENGINE` before + work starts. Do not drop engine-strict; export `/opt/node24/bin` at the front of + `PATH` to satisfy repository engine contracts before running `npm ci` or building + the worker: + + ```bash + export PATH="/opt/node24/bin:$PATH" + node -v # must report v24.x + ``` + +--- + ## Rollback Redeploy the previous image tag. The worker holds no durable local state; all diff --git a/src/components/clinical-dashboard/favourites-command-library-page.tsx b/src/components/clinical-dashboard/favourites-command-library-page.tsx index 0fa8256ba..c83afa11a 100644 --- a/src/components/clinical-dashboard/favourites-command-library-page.tsx +++ b/src/components/clinical-dashboard/favourites-command-library-page.tsx @@ -141,15 +141,7 @@ const fallbackIconByType: Record = { }; function lastUsedScore(lastUsed: string): number { - const lower = lastUsed.toLowerCase(); - if (lower.startsWith("today")) { - const timeMatch = lastUsed.match(/(\d{1,2}):(\d{2})/); - if (timeMatch) return 100_000 + Number(timeMatch[1]) * 60 + Number(timeMatch[2]); - return 100_000; - } - if (lower.startsWith("yesterday")) return 50_000; - if (lower.startsWith("mon")) return 10_000; - return 1_000; + return lastOpenedScore(lastUsed); } function isSourceBacked(item: FavouriteItem): boolean { diff --git a/src/components/favourites/favourites-storage.ts b/src/components/favourites/favourites-storage.ts index a658faba0..c226dfd0d 100644 --- a/src/components/favourites/favourites-storage.ts +++ b/src/components/favourites/favourites-storage.ts @@ -5,23 +5,21 @@ export const DATABASE_FAVOURITES_PINNED_STORAGE_KEY = "database:favourites:pinne const DEFAULT_PINNED_ITEM_IDS = ["acamprosate-renal-screen", "lithium-monitoring-guideline"]; -// Initial baseline offsets for demo prototype items before user interactions +// Initial baseline offsets for items before user interactions (honest absence per #339) function getDefaultInitialTimestamps(): Record { - const now = Date.now(); - const dayMs = 24 * 60 * 60 * 1000; - return { - "acamprosate-renal-screen": now - 15 * 60 * 1000, // 15 mins ago (today) - "lithium-monitoring-guideline": now - 35 * 60 * 1000, // 35 mins ago (today) - "renal-dose-search": now - 55 * 60 * 1000, // 55 mins ago (today) - "clozapine-monitoring-table": now - dayMs - 2 * 60 * 60 * 1000, // yesterday - "qt-prolongation-quote": now - 3 * dayMs, // earlier this week - }; + return {}; } let inMemoryLastOpened: Record | null = null; let inMemoryPinned: Set | null = null; const listeners = new Set<() => void>(); +export function resetFavouritesStorageForTesting(): void { + inMemoryLastOpened = null; + inMemoryPinned = null; + listeners.clear(); +} + function notifyListeners() { for (const listener of listeners) { try { @@ -69,7 +67,7 @@ export function loadFavouriteLastOpened(): Record { const raw = localStorage.getItem(DATABASE_FAVOURITES_LAST_OPENED_STORAGE_KEY); if (raw) { const parsed = JSON.parse(raw); - if (typeof parsed === "object" && parsed !== null) { + if (typeof parsed === "object" && parsed !== null && !Array.isArray(parsed)) { const result: Record = { ...getDefaultInitialTimestamps(), ...parsed }; inMemoryLastOpened = result; return result; @@ -145,12 +143,24 @@ export function toggleFavouritePinnedId(itemId: string): Set { const dayNames = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"]; const monthNames = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]; -export function formatLastOpened(timestampOrLabel: number | string | undefined): string { - if (timestampOrLabel === undefined || timestampOrLabel === null) { +export function formatLastOpened(timestampOrLabel: number | string | undefined | null): string { + if ( + timestampOrLabel === undefined || + timestampOrLabel === null || + timestampOrLabel === 0 || + timestampOrLabel === "" + ) { return "Saved"; } if (typeof timestampOrLabel === "string") { - return timestampOrLabel; + const trimmed = timestampOrLabel.trim(); + return trimmed ? timestampOrLabel : "Saved"; + } + + if (typeof timestampOrLabel === "number") { + if (!Number.isFinite(timestampOrLabel) || timestampOrLabel <= 0) { + return "Saved"; + } } const date = new Date(timestampOrLabel); @@ -179,13 +189,17 @@ export function formatLastOpened(timestampOrLabel: number | string | undefined): return `${date.getDate()} ${monthNames[date.getMonth()]}`; } -export function lastOpenedScore(lastUsed: string | number | undefined): number { +export function lastOpenedScore(lastUsed: string | number | undefined | null): number { + if (lastUsed === undefined || lastUsed === null || lastUsed === 0 || lastUsed === "") { + return 0; + } if (typeof lastUsed === "number") { + if (!Number.isFinite(lastUsed) || lastUsed <= 0) return 0; return lastUsed; } - if (!lastUsed) return 0; - const lower = lastUsed.toLowerCase(); + const lower = lastUsed.toLowerCase().trim(); + if (!lower) return 0; if (lower.startsWith("today")) { const timeMatch = lastUsed.match(/(\d{1,2}):(\d{2})/); if (timeMatch) return 1_000_000_000_000 + Number(timeMatch[1]) * 60 + Number(timeMatch[2]); diff --git a/tests/favourites.test.ts b/tests/favourites.test.ts index f69622aaa..a87d815dd 100644 --- a/tests/favourites.test.ts +++ b/tests/favourites.test.ts @@ -10,18 +10,20 @@ import { loadFavouriteLastOpened, loadFavouritePinnedIds, recordFavouriteOpened, + resetFavouritesStorageForTesting, toggleFavouritePinnedId, } from "@/components/favourites/favourites-storage"; describe("favourites storage, timestamps and pinning", () => { beforeEach(() => { localStorage.clear(); + resetFavouritesStorageForTesting(); }); - it("loads default seed timestamps and records real timestamp when item is opened", () => { + it("enforces honest absence initially and records real timestamp when item is opened", () => { const initial = loadFavouriteLastOpened(); - expect(initial["acamprosate-renal-screen"]).toBeDefined(); - expect(typeof initial["acamprosate-renal-screen"]).toBe("number"); + expect(initial["acamprosate-renal-screen"]).toBeUndefined(); + expect(Object.keys(initial)).toHaveLength(0); const customTime = Date.now() + 5000; recordFavouriteOpened("test-item-1", customTime); @@ -35,6 +37,12 @@ describe("favourites storage, timestamps and pinning", () => { expect(parsed["test-item-1"]).toBe(customTime); }); + it("safely ignores array payloads in last-opened storage and falls back to honest absence", () => { + localStorage.setItem(DATABASE_FAVOURITES_LAST_OPENED_STORAGE_KEY, JSON.stringify(["invalid", "array"])); + const loaded = loadFavouriteLastOpened(); + expect(loaded).toEqual({}); + }); + it("loads default pinned IDs and allows toggling pinning state with localStorage persistence", () => { const initialPinned = loadFavouritePinnedIds(); expect(initialPinned.has("acamprosate-renal-screen")).toBe(true); @@ -64,6 +72,7 @@ describe("favourites storage, timestamps and pinning", () => { expect(formattedYesterday).toMatch(/^Yesterday \d{2}:\d{2}$/); expect(formatLastOpened(undefined)).toBe("Saved"); + expect(formatLastOpened(0)).toBe("Saved"); expect(formatLastOpened("Today 08:44")).toBe("Today 08:44"); }); @@ -75,5 +84,7 @@ describe("favourites storage, timestamps and pinning", () => { expect(lastOpenedScore("Today 10:00")).toBeGreaterThan(lastOpenedScore("Yesterday 10:00")); expect(lastOpenedScore("Yesterday 10:00")).toBeGreaterThan(lastOpenedScore("Mon 10:00")); expect(lastOpenedScore("Saved")).toBe(1000); + expect(lastOpenedScore(0)).toBe(0); + expect(lastOpenedScore(undefined)).toBe(0); }); }); diff --git a/tests/session-start-hook.test.ts b/tests/session-start-hook.test.ts index 3863e3c89..103f65218 100644 --- a/tests/session-start-hook.test.ts +++ b/tests/session-start-hook.test.ts @@ -1,6 +1,15 @@ import { createHash } from "node:crypto"; import { spawnSync } from "node:child_process"; -import { chmodSync, copyFileSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { + chmodSync, + copyFileSync, + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from "node:fs"; import { tmpdir } from "node:os"; import { basename, join } from "node:path"; import { afterEach, describe, expect, it } from "vitest"; @@ -35,6 +44,13 @@ import { afterEach, describe, expect, it } from "vitest"; const sourceHook = join(process.cwd(), ".claude/hooks/session-start.sh"); const NODE_VERSION = "24.19.0"; const scratchRoots: string[] = []; +const bashCommand = + process.platform === "win32" + ? ([ + "C:\\Program Files\\Git\\bin\\bash.exe", + join(process.env.ProgramFiles || "C:\\Program Files", "Git/bin/bash.exe"), + ].find((p) => existsSync(p)) ?? "bash") + : "bash"; afterEach(() => { for (const root of scratchRoots.splice(0)) { @@ -80,11 +96,15 @@ function stubEnvironment(): { home: string; project: string; hook: string } { } function runHook(hook: string, env: Record, cwd: string) { - const base = { ...process.env, CLAUDE_CODE_REMOTE: "true", ...env }; + const normalizedEnv: Record = {}; for (const [key, value] of Object.entries(env)) { + normalizedEnv[key] = typeof value === "string" ? value.replace(/\\/g, "/") : value; + } + const base = { ...process.env, CLAUDE_CODE_REMOTE: "true", ...normalizedEnv }; + for (const [key, value] of Object.entries(normalizedEnv)) { if (value === undefined) delete (base as Record)[key]; } - return spawnSync("bash", [hook], { cwd, env: base as NodeJS.ProcessEnv, encoding: "utf8" }); + return spawnSync(bashCommand, [hook.replace(/\\/g, "/")], { cwd, env: base as NodeJS.ProcessEnv, encoding: "utf8" }); } describe("session-start hook", () => { @@ -107,9 +127,9 @@ describe("session-start hook", () => { const exportLine = result.stdout.split(/\r?\n/).find((line) => line.startsWith("export PATH=")); expect(exportLine).toBeDefined(); - const caller = spawnSync("bash", ["-c", `${exportLine}; node -v`], { + const caller = spawnSync(bashCommand, ["-c", `${exportLine}; node -v`], { cwd: project, - env: { ...process.env, HOME: home }, + env: { ...process.env, HOME: home.replace(/\\/g, "/") }, encoding: "utf8", }); expect(caller.status, `caller exited ${caller.status}: ${caller.stderr}`).toBe(0); From 00b15095b77e9d74f25d35ce20ecdb697694fe02 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Wed, 19 Aug 2026 04:50:35 +0800 Subject: [PATCH 2/3] fix(adversarial): harden geometry calculations, storage fuzzing guards, and test isolation --- .../document-viewer/bbox-overlay.ts | 8 ++++- .../favourites/favourites-storage.ts | 4 +++ tests/favourites.test.ts | 29 +++++++++++++++++++ 3 files changed, 40 insertions(+), 1 deletion(-) diff --git a/src/components/document-viewer/bbox-overlay.ts b/src/components/document-viewer/bbox-overlay.ts index 349d4abd3..55c28dad2 100644 --- a/src/components/document-viewer/bbox-overlay.ts +++ b/src/components/document-viewer/bbox-overlay.ts @@ -59,7 +59,13 @@ export function resolveBboxOverlayStyle({ normW = maxX - minX; normH = maxY - minY; } else { - if (!pageGeometry || pageGeometry.width <= 0 || pageGeometry.height <= 0) { + if ( + !pageGeometry || + !Number.isFinite(pageGeometry.width) || + !Number.isFinite(pageGeometry.height) || + pageGeometry.width <= 0 || + pageGeometry.height <= 0 + ) { return null; } normX = minX / pageGeometry.width; diff --git a/src/components/favourites/favourites-storage.ts b/src/components/favourites/favourites-storage.ts index c226dfd0d..1140e8a83 100644 --- a/src/components/favourites/favourites-storage.ts +++ b/src/components/favourites/favourites-storage.ts @@ -17,6 +17,7 @@ const listeners = new Set<() => void>(); export function resetFavouritesStorageForTesting(): void { inMemoryLastOpened = null; inMemoryPinned = null; + sharedStorageListenerAttached = false; listeners.clear(); } @@ -197,6 +198,9 @@ export function lastOpenedScore(lastUsed: string | number | undefined | null): n if (!Number.isFinite(lastUsed) || lastUsed <= 0) return 0; return lastUsed; } + if (typeof lastUsed !== "string") { + return 0; + } const lower = lastUsed.toLowerCase().trim(); if (!lower) return 0; diff --git a/tests/favourites.test.ts b/tests/favourites.test.ts index a87d815dd..b5f01faf4 100644 --- a/tests/favourites.test.ts +++ b/tests/favourites.test.ts @@ -11,6 +11,7 @@ import { loadFavouritePinnedIds, recordFavouriteOpened, resetFavouritesStorageForTesting, + subscribeFavouritesStorage, toggleFavouritePinnedId, } from "@/components/favourites/favourites-storage"; @@ -43,6 +44,30 @@ describe("favourites storage, timestamps and pinning", () => { expect(loaded).toEqual({}); }); + it("resets in-memory cache and listeners when resetFavouritesStorageForTesting is invoked", () => { + let listenerCalled = false; + const unsubscribe = subscribeFavouritesStorage(() => { + listenerCalled = true; + }); + + recordFavouriteOpened("item-before-reset", 12345); + expect(loadFavouriteLastOpened()["item-before-reset"]).toBe(12345); + expect(listenerCalled).toBe(true); + + listenerCalled = false; + resetFavouritesStorageForTesting(); + + // After reset, inMemoryLastOpened is cleared; if localStorage is also cleared, it returns honest absence ({}) + localStorage.clear(); + expect(loadFavouriteLastOpened()).toEqual({}); + + // Also verify listeners were cleared by resetFavouritesStorageForTesting + recordFavouriteOpened("item-after-reset", 67890); + expect(listenerCalled).toBe(false); + + unsubscribe(); + }); + it("loads default pinned IDs and allows toggling pinning state with localStorage persistence", () => { const initialPinned = loadFavouritePinnedIds(); expect(initialPinned.has("acamprosate-renal-screen")).toBe(true); @@ -86,5 +111,9 @@ describe("favourites storage, timestamps and pinning", () => { expect(lastOpenedScore("Saved")).toBe(1000); expect(lastOpenedScore(0)).toBe(0); expect(lastOpenedScore(undefined)).toBe(0); + // SEC-M1: Corrupted or non-string/non-number inputs must safely return 0 without throwing TypeError + expect(lastOpenedScore({} as unknown as string)).toBe(0); + expect(lastOpenedScore(true as unknown as string)).toBe(0); + expect(lastOpenedScore(["invalid"] as unknown as string)).toBe(0); }); }); From 791be612556f2c9513e8c4e609ba829964799e64 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 21:17:43 +0000 Subject: [PATCH 3/3] docs(runbook): make search-health index recovery criterion the health check, not a count The recovery step named a hardcoded '20' required indexes, but search_schema_health()'s required_indexes list already has 22 entries and later migrations extend it further. An operator checking for a stale fixed number could accept an incomplete restore or flag a healthy one. Point at the health function's own ok/missing result (npm run check:indexing) instead. Addresses a Copilot review finding on PR #2155. --- docs/launch-operator-runbook.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/launch-operator-runbook.md b/docs/launch-operator-runbook.md index c37e001c3..55a7f35e0 100644 --- a/docs/launch-operator-runbook.md +++ b/docs/launch-operator-runbook.md @@ -175,7 +175,7 @@ Following a Supabase database restore or disaster recovery failover, verify all ## 8. Operational notes & diagnostics (#248, #305, #315, #102) -- **Search-Health Indexes (#248):** Ensure migration `20260705180000_reconcile_search_health_indexes.sql` is active on live and all 20 required indexes are present. +- **Search-Health Indexes (#248):** Ensure migration `20260705180000_reconcile_search_health_indexes.sql` is active on live and `search_schema_health()` reports `ok: true` with no `missing` entries (`npm run check:indexing`) — treat its `required_indexes` list as the recovery criterion, not a fixed count, since later migrations extend it. - **Concurrent Document Index Recipe (#102):** When applying additive document index optimizations on a busy database (`documents_title_bare_trgm_idx`, `documents_file_name_bare_trgm_idx`, and `documents_status_id_idx`), pre-create indexes concurrently (`CREATE INDEX CONCURRENTLY IF NOT EXISTS`) before applying the committed migration and registering in `search_schema_health()` to avoid write lock contention. Validate each index with `pg_index.indisvalid`. Note that bare-column trigrams and composite `(status, id)` indexes on the RAG path are canary-gated due to unordered `LIMIT 12` selection in candidate retrieval ([operator-apply-performance-latency-remediation.md](operator-apply-performance-latency-remediation.md)). - **Canary Latency & Cost Boundaries (#305):** Retrieval latency p90 SLO is ≤ 20s. Canary cost metrics provide lower-bound estimates without cache warmup. - **UI Smoke Reporter Stranding (#315):** When debugging rare UI smoke test timeouts, inspect reporter stranding in Playwright hooks rather than assuming layout regressions.