fix: remediate 2026-07 repo audit findings (clinical answer path, data integrity, privacy)

.env.example

@@ -56,6 +56,13 @@ RAG_SEARCH_CACHE_TTL_MS=60000
 RAG_SEARCH_CACHE_SIZE=200
 RAG_AWAIT_QUERY_LOGS=false
 
+# Server-side key for the redacted query-hash placeholder (min 16 chars).
+# When set, stored query hashes are HMAC-SHA256 keyed pseudonyms — not
+# offline-reversible and not correlatable outside this deployment. Strongly
+# recommended wherever real clinical queries are logged. Changing or setting
+# the key changes future hashes, so historical dedup/joins reset from then on.
+#RAG_QUERY_HASH_SECRET=
+
 # Private buckets created by supabase/schema.sql.
 SUPABASE_DOCUMENT_BUCKET=clinical-documents
 SUPABASE_IMAGE_BUCKET=clinical-images

docs/rag-hybrid-findings-and-todo.md

@@ -63,8 +63,8 @@ outstanding backlog. See also the master plan
 
 4. 🔍 **Answer-path ranking investigated (2026-07-01) — healthy; low mrr is a sibling-doc artifact,
  NOT a defect.** Probed every low-`rr@10` golden case. In each, the docs ranked above the pinned
- one are **legitimate siblings** the corpus genuinely contains: several *Safety Planning* guidelines
- (KEMH/RKPG/AKG), multiple hospital versions of *Active Community Patients in ED*, multiple
+ one are **legitimate siblings** the corpus genuinely contains: several _Safety Planning_ guidelines
+ (KEMH/RKPG/AKG), multiple hospital versions of _Active Community Patients in ED_, multiple
  opioid-pharmacotherapy guidelines, and the two agitation guidelines. Recall stays 1.0 and the model
  gets correct context; forcing the pinned doc to #1 over equally-valid siblings would be overfitting.
  **So items 1/6 (query-class weighting to raise mrr) are deprioritized** — chasing that metric on
@@ -73,25 +73,25 @@ outstanding backlog. See also the master plan
  (`clinical-search.ts:1362`) — base + the ~40 stacked boosts routinely exceed 1.0, so many strong
  matches tie at 1.0 and order by an arbitrary `document_id` tiebreak. It doesn't hurt these cases
  (the tied docs are all relevant), but it wastes the boost engineering. If ever revisited, break
- ties by the *pre-clamp* score rather than raising the ceiling (downstream gates assume [0,1]).
+ ties by the _pre-clamp_ score rather than raising the ceiling (downstream gates assume [0,1]).
  - The second-stage rerank (which uses unclamped scoring + a strong dose-amount/title boost) rarely
  fires for document_lookup/broad_summary (`shouldUseSecondStageRerank` needs `topScoresClose &&
-hasVisualEvidence`, `rag.ts:548`). Widening it (RC10) could restore discrimination among the
+hasVisualEvidence`, `rag.ts:548`). Widening it (RC10) could restore discrimination among the
  1.0-tied group, but since the tied docs are valid siblings the payoff is marginal and unvalidatable
  on the current golden set — do it only alongside a chunk-level "best-passage-first" eval metric.
 5. ⏸️ **`ef_search` policy inconsistent — BLOCKED, deferred.** Attempted `ALTER FUNCTION … SET
-hnsw.ef_search='100'` on the three sql functions; **hosted Supabase denies it (`42501 permission
-denied to set parameter`)** — the RC11 blocker. The only method hosted allows is the plpgsql-wrapper
-+ runtime `PERFORM set_config('hnsw.ef_search','100',true)` pattern (what memory_cards uses; measured
- latency-neutral: chunks 76→79ms warm). Deferred: the recall gain is unquantified (golden already 1.0)
- and there's no hard-query eval set to justify adding three plpgsql wrappers. Revisit once an
- expanded/hard eval set exists (see P2.8).
+hnsw.ef_search='100'` on the three sql functions; **hosted Supabase denies it (`42501 permission
+denied to set parameter`)** — the RC11 blocker. The only method hosted allows is the plpgsql-wrapper
+- runtime `PERFORM set_config('hnsw.ef_search','100',true)` pattern (what memory_cards uses; measured
+latency-neutral: chunks 76→79ms warm). Deferred: the recall gain is unquantified (golden already 1.0)
+and there's no hard-query eval set to justify adding three plpgsql wrappers. Revisit once an
+expanded/hard eval set exists (see P2.8).
 6. **RC5–RC13 ranking tuning** — partially addressed / re-scoped after the item-4 investigation:
  - ✅ **Same-document crowding (RC7)** — the `/api/search` results panel cap was lowered
  `maxPerDocument 4→3` (`app/api/search/route.ts`, backfill-protected so result count is unchanged).
  Note: this only affects the **panel**; the answer-retrieval path (`searchChunksWithTelemetry`) has
  no per-doc cap and doesn't need one — the comparison gate already enforces ≥2 distinct docs, and
- single-topic queries *should* be able to draw multiple chunks from the best document.
+ single-topic queries _should_ be able to draw multiple chunks from the best document.
  - ⏳ **Synthetic text similarity (RC9)** `least(0.95, 0.56 + text_rank*0.39)` still feeds coverage
  gates that assume a real cosine — gate text-only paths on `text_rank`/`rrf` instead. (Cleanest
  remaining ranking-correctness item.)
@@ -137,12 +137,13 @@ outstanding backlog. See also the master plan
  working precise match. Unit tests in `tests/retrieval-query-variants.test.ts` + the
  `alcohol-ciwa-threshold` golden case guard it. **This is a general recall win, not just CIWA** —
  any long multi-term query previously risked silent 0-match FTS.
+
 9. ⚠️ **OCR "dropped-s" defect — real but NOT reliably heuristically-detectable; guard attempted then
  REVERTED (2026-07-01). Honest post-mortem below.**
  - **What's true:** real dropped-'s' corruption exists in some table-derived index units
  ("psychosocial"→"p ycho ocial", "1st mood stabiliser"→"1 t mood tabili er"). The **raw
  `document_chunks` (answer context) are clean** — 0 docs below 0.025 s-ratio — so **generated
- answer text is not degraded**; the defect only touches structured *table* units (OCR'd from
+ answer text is not degraded**; the defect only touches structured _table_ units (OCR'd from
  images), and the intact numbers survive ("CIWA-Ar **core** <10" keeps the "<10").
  - **The detection is the hard part — every heuristic false-positives.** First tried an s-ratio
  detector (`'s'`/letter < 0.03): it flagged 772 units but **only 135 were real (82% false
@@ -164,8 +165,8 @@ outstanding backlog. See also the master plan
  - **If ever pursued (low priority, modest impact):** reliable detection needs a **dictionary/
  spellcheck approach** ("fraction of tokens that aren't valid English/clinical words") or fixing
  the **upstream table-OCR** step — not a token heuristic. Neither is warranted by the impact.
- Remaining true enrichment items: confirm `20260627000000_retrieval_hnsw_ef_search.sql` on live; run
- `enrich:backfill` / `tags:backfill` for any genuinely missing synopsis/labels.
+Remaining true enrichment items: confirm `20260627000000_retrieval_hnsw_ef_search.sql` on live; run
+`enrich:backfill` / `tags:backfill` for any genuinely missing synopsis/labels.
 10. 🔧 **Query understanding (RC6/E) — pg_trgm typo correction started (2026-07-01).**
  - **Data-driven promotion is blocked:** `rag_query_misses` (71 rows) are privacy-redacted hashes
  with empty `candidate_aliases`, so the plan's "promote real misses to aliases" path can't run.
@@ -179,18 +180,18 @@ outstanding backlog. See also the master plan
  min_sim 0.45. Validated: clozapin→clozapine, agitaton→agitation, schizophrenai→schizophrenia,
  bipoler→bipolar, withdrawl→withdrawal, lithiun→lithium; clean queries unchanged. ~85ms.
  - ✅ **Wired as a text-search fallback** in `searchTextChunkCandidates` (`rag.ts`): when strict AND
- variants return nothing, correct the query and retry (strictly, then OR-relaxed) *before* the 8b
+ variants return nothing, correct the query and retry (strictly, then OR-relaxed) _before_ the 8b
  OR-relaxation, so a typo like "clozapin monitoring" resolves to clozapine rather than OR-matching
  generic "monitoring" docs. Verified end-to-end: "clozapin anc threshold"→Clozapine docs, "dischage
  planning"→Discharge Planning. Golden set unchanged (23/23, no regression); 682 tests pass.
  - ✅ **Correction before the unsupported short-circuit (2026-07-01).** `searchChunksWithTelemetry`
  (`rag.ts:4986`) now, when a query would short-circuit as unsupported, trigram-corrects it and —
  if it changed — re-runs the whole retrieval once on the corrected text (guarded by an internal
  `typoCorrected` flag; only fires for would-be-unsupported queries so no hot-path cost). Rescues
- typo queries whose corrected form is a *supported* class (e.g. a typo'd clozapine/dose query
+ typo queries whose corrected form is a _supported_ class (e.g. a typo'd clozapine/dose query
  → table_threshold). Golden 23/23 unchanged, 682 tests pass.
  - ⚠️ **Pre-existing bug surfaced (NEW, finding #11):** unsupported-classified queries retrieve
- **nondeterministically** — the *same* query in the *same* process alternates
+ **nondeterministically** — the _same_ query in the _same_ process alternates
  `unsupported_short_circuit` (0 results) vs `text_fast_path`/`hybrid` (real results), e.g.
  "anorexia management" (no typo). Classification is pure and all caches honour `skipCache`, so the
  variance is elsewhere in the unsupported-query path (candidate: alias fetch/expansion or an async

next.config.ts

@@ -28,7 +28,7 @@ const securityHeaders = [
  upgradeInsecureRequests +
  "img-src 'self' data: blob: https:; " +
  "media-src 'self' https:; " +
- "connect-src 'self' https://sjrfecxgysukkwxsowpy.supabase.co https://*.supabase.co https://api.openai.com; " +
+ "connect-src 'self' https://*.supabase.co https://api.openai.com; " +
  scriptSrc +
  "style-src 'self' 'unsafe-inline'",
  },

scripts/purge-query-logs.ts

@@ -27,7 +27,11 @@ function parseArgs(argv: string[]): PurgeArgs {
  index += 1;
 
  if (token === "--owner-email") args.ownerEmail = value;
- if (token === "--older-than-days") args.olderThanDays = Number.parseInt(value, 10);
+ else if (token === "--older-than-days") args.olderThanDays = Number.parseInt(value, 10);
+ // Audit L1: fail loudly on unknown flags. A typo'd flag used to silently
+ // swallow its value (e.g. --owner-emial ate the email), and the purge then
+ // ran against the env-configured owner instead of the intended one.
+ else throw new Error(`Unknown argument ${token}`);
  }
 
  if (!args.ownerEmail) throw new Error("Provide --owner-email.");

scripts/recover-ingestion-queue.ts

@@ -35,18 +35,47 @@ function supabaseStageError(stage: string, error: unknown) {
  return wrapped;
 }
 
+const booleanFlags = new Set(["--apply", "--yes"]);
+const valueFlags = new Set(["--stale-after-minutes", "--limit"]);
+
+// Audit L2 (hardened after diff review): this script mutates ingestion state,
+// so argument parsing fails loudly on ANY surprise —
+// - unknown/typo'd flag names ("--limt 5" used to be ignored, silently
+// recovering up to 20 jobs instead of the intended 5),
+// - a value-flag with a missing or empty value ("--limit" at the end of the
+// line, "--limit="),
+// - provided-but-malformed numeric values ("--limit 5O").
 function parseArgs(argv: string[]) {
- const valueFor = (name: string) => {
- const inline = argv.find((arg) => arg.startsWith(`--${name}=`))?.split("=")[1];
- if (inline) return inline;
- const index = argv.indexOf(`--${name}`);
- return index >= 0 ? argv[index + 1] : undefined;
+ const values = new Map<string, string>();
+ const booleans = new Set<string>();
+ for (let index = 0; index < argv.length; index += 1) {
+ const token = argv[index];
+ if (booleanFlags.has(token)) {
+ booleans.add(token);
+ continue;
+ }
+ const equalsIndex = token.indexOf("=");
+ const name = equalsIndex >= 0 ? token.slice(0, equalsIndex) : token;
+ if (!valueFlags.has(name)) throw new Error(`Unknown argument ${token}`);
+ const value = equalsIndex >= 0 ? token.slice(equalsIndex + 1) : argv[index + 1];
+ if (equalsIndex < 0) index += 1;
+ if (!value || value.startsWith("--")) throw new Error(`Missing value for ${name}`);
+ values.set(name, value);
+ }
+ const positiveIntFor = (name: string) => {
+ const raw = values.get(`--${name}`);
+ if (raw === undefined) return undefined;
+ const parsed = Number.parseInt(raw, 10);
+ if (!Number.isInteger(parsed) || parsed <= 0 || String(parsed) !== raw.trim()) {
+ throw new Error(`--${name} must be a positive integer (received "${raw}").`);
+ }
+ return parsed;
  };
  return {
- apply: argv.includes("--apply"),
- yes: argv.includes("--yes"),
- staleAfterMinutes: Number.parseInt(valueFor("stale-after-minutes") ?? "", 10),
- limit: Number.parseInt(valueFor("limit") ?? "", 10),
+ apply: booleans.has("--apply"),
+ yes: booleans.has("--yes"),
+ staleAfterMinutes: positiveIntFor("stale-after-minutes"),
+ limit: positiveIntFor("limit"),
  };
 }
 
@@ -64,10 +93,8 @@ async function main() {
  ]);
  requireServerEnv();
  const args = parseArgs(process.argv.slice(2));
- const staleAfterMinutes = Number.isFinite(args.staleAfterMinutes)
- ? args.staleAfterMinutes
- : env.WORKER_STALE_AFTER_MINUTES;
- const limit = Number.isFinite(args.limit) ? args.limit : 20;
+ const staleAfterMinutes = args.staleAfterMinutes ?? env.WORKER_STALE_AFTER_MINUTES;
+ const limit = args.limit ?? 20;
  const supabase = createAdminClient();
 
  console.log("=== Ingestion Queue Recovery ===");

src/app/api/documents/[id]/route.ts

@@ -496,16 +496,24 @@ export async function DELETE(request: Request, { params }: { params: Promise<{ i
  if (documentError) throw new Error(documentError.message);
  if (!document) return NextResponse.json({ error: "Document not found." }, { status: 404 });
 
+ // Audit M9: block deletion on PENDING jobs too, matching the reindex
+ // routes' checkIngestionMutationSafety predicate. A just-queued reindex
+ // job (status "pending") racing this DELETE let the worker upload a new
+ // generation of image objects after the storage paths were enumerated,
+ // orphaning them permanently.
  const { data: activeJobs, error: activeJobsError } = await supabase
  .from("ingestion_jobs")
  .select("id,status")
  .eq("document_id", id)
- .eq("status", "processing")
+ .in("status", ["pending", "processing"])
  .limit(1);
 
  if (activeJobsError) throw new Error(activeJobsError.message);
  if ((activeJobs ?? []).length > 0) {
- throw new PublicApiError("Document is currently indexing. Stop or wait for the worker before deleting.", 409);
+ throw new PublicApiError(
+ "Document has pending or processing indexing work. Stop or wait for the worker before deleting.",
+ 409,
+ );
  }
 
  const [images, chunks] = await Promise.all([

src/app/api/search/route.ts

@@ -680,8 +680,12 @@ async function buildScopedSearchPayload(
  limit: isSourceLibrarySearchMode(body.mode) ? body.documentLimit : undefined,
  })
  : [];
- const smartPanel = buildSmartPanel(searchFocusQuery, results);
+ // Audit L10: compute relevance/visual evidence ONCE and share with the
+ // smart panel — the panel's own recomputation was discarded by the spread
+ // at payload build time anyway.
  const relevance = buildEvidenceRelevance(searchFocusQuery, results);
+ const visualEvidence = buildVisualEvidence(results);
+ const smartPanel = buildSmartPanel(searchFocusQuery, results, { relevance, visualEvidence });
  const documentMatches = isSourceLibrarySearchMode(body.mode)
  ? annotateDocumentMatches(searchFocusQuery, relatedDocuments.map(toDocumentMatch), results)
  : [];
@@ -707,7 +711,7 @@ async function buildScopedSearchPayload(
  const payload = {
  results: compactSearchResults(searchFocusQuery, results),
  facets: buildSearchFacets(results),
- visualEvidence: buildVisualEvidence(results),
+ visualEvidence,
  relevance,
  relatedDocuments: relatedDocuments.map((document) => ({
  document_id: document.document_id,
@@ -828,12 +832,16 @@ export async function POST(request: Request) {
  results,
  )
  : [];
+ const cachedVisualEvidence = buildVisualEvidence(results);
  return NextResponse.json({
  results: compactSearchResults(searchFocusQuery, results),
  facets: buildSearchFacets(results),
- visualEvidence: buildVisualEvidence(results),
+ visualEvidence: cachedVisualEvidence,
  relevance,
- smartPanel: { ...buildSmartPanel(searchFocusQuery, results), relevance },
+ smartPanel: {
+ ...buildSmartPanel(searchFocusQuery, results, { relevance, visualEvidence: cachedVisualEvidence }),
+ relevance,
+ },
  smartApiPlan: buildSmartRagApiPlan({
  query: searchFocusQuery,
  queryClass,

src/components/AccessibleTable.tsx

@@ -333,13 +333,21 @@ export function AccessibleTable({
  const restoreFocusRef = useRef<HTMLElement | null>(null);
  const [open, setOpen] = useState(false);
  const canExpand = useMobileTableExpansion(expandOnMobile);
- const parsed = rows?.length ? rows : parseMarkdownTable(markdown);
+ const hasExplicitRows = Boolean(rows?.length);
+ const parsed = hasExplicitRows ? rows : parseMarkdownTable(markdown);
  const normalized = useMemo(() => {
  if (!parsed?.length) return null;
- const table = normalizeAccessibleTable(parsed, columns);
+ // Audit M8/H4 parity (diff review): markdown-parsed rows include their
+ // own header line as row 0 — passing explicit columns alongside them made
+ // the markdown header render as the first DATA row on screen, and let the
+ // on-screen and copied-ward-note normalizations disagree (different
+ // headers, potentially different lowConfidence caveats). Columns are the
+ // header only for explicit row arrays, matching clinicalTableToTextRows
+ // in ward-output.ts.
+ const table = normalizeAccessibleTable(parsed, hasExplicitRows ? columns : null);
  if (!table) return null;
  return clinicalOnly ? clinicalOnlyTable(table) : table;
- }, [clinicalOnly, columns, parsed]);
+ }, [clinicalOnly, columns, hasExplicitRows, parsed]);
 
  const dialogOpen = open && canExpand;