fix: restore RAG search health indexes

[BigSimmo]

Summary

• Record the four already-completed production RAG/search index repairs in repository migration history without rebuilding them inside a transaction.
• Make the forward migration fail fast when any required index is absent, invalid (indisvalid/indisready), or non-canonical versus the pinned definition; a drifted hosted target must prebuild missing indexes with CREATE INDEX CONCURRENTLY outside the migration transaction, validate them, and only then mark the version applied.
• Keep fresh replays deterministic through the earlier canonical reconciliation migration and schema.sql.
• Pin normalized full index definitions across the canonical migration, schema.sql, and drift-manifest.json, while separately pinning the mark-applied guard.

RAG impact: no retrieval behaviour change — full definitions are pinned but unchanged; the guard validates presence, validity, and canonical match only.

Verification

• Secret Scan / SAST at prior heads
• Prior CI run 8043 passed at f932897760ff667ddecd5d5db405303761a1b82a
• npm run check:rag:fixtures — 36 golden cases, 23 suites
• npm run check:migration-role
• npm run test -- tests/supabase-schema.test.ts — 74/74 after validation + normalizer fix
• Prior live search_schema_health() — ok: true, missing: [] (pre-fix evidence)
• Prior live index validation — all four indexes indisvalid=true and indisready=true (pre-fix evidence)
• Fresh GitHub CI at exact current head after the validation repair push
• Verification not run: npm run eval:retrieval:quality — not required because no retrieval/ranking SQL, comparator, or index definition changes
• UI verification not run: no UI, routing, styling, or browser behavior changes

Risk and rollout

• Risk: medium; clinical-risk path via supabase migration history, but the migration creates no indexes and only validates already-built objects.
• Rollback: leave this version unapplied, or ship a separately approved forward migration/operator procedure; do not drop or rebuild live indexes as rollback for this history marker.
• Provider or production effects: merging alone performs no provider calls and no database writes. A later migration apply only validates object presence/validity/definition on a target where this version is not already recorded.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use
• No patient-identifiable document workflow was introduced or expanded without explicit governance approval
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative
• Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

Merge gate

Do not merge until fresh exact-head CI passes and migration-safety review threads are verified against the current head.

Summary by CodeRabbit

• Bug Fixes

  • Added safeguards to verify that required search-health indexes exist, are valid, and match expected definitions.
  • Improved error reporting for missing, invalid, or mismatched indexes.

• Tests

  • Added coverage for detecting incorrect search indexes and validating repair workflows.

• Documentation

  • Documented an outstanding production issue involving absent search-health indexes and tracked it for follow-up.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ada2c3a4-54fb-401b-a041-a02472b9b499

📥 Commits

Reviewing files that changed from the base of the PR and between a87cec9 and 56cacc5.

📒 Files selected for processing (2)

• docs/outstanding-issues.md
• tests/supabase-schema.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/supabase-schema.test.ts

---

📝 Walkthrough

Walkthrough

The migration validates four required RAG search-health indexes without creating them. It reports missing, invalid, or mismatched indexes. Schema tests compare normalized definitions across the schema, migration, and drift manifest. Documentation records the drift investigation.

Changes

RAG index restoration

Layer / File(s)	Summary
Index restoration migration guard supabase/migrations/20260804110240_restore_rag_search_health_indexes.sql	The migration checks index existence, readiness, validity, and canonical definitions. It raises categorized errors and requires concurrent repair before migration execution.
Schema and migration validation tests/supabase-schema.test.ts	Tests normalize and compare index definitions across the schema, canonical migration, and drift manifest. They verify missing, invalid, and mismatched index detection and reject transactional index creation.
Production drift tracking docs/outstanding-issues.md	The issue tracker records investigation of missing production search-health indexes and advances the next issue identifier.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• BigSimmo/Database#840: Both changes modify Supabase migration replay and schema/index validation tests.
• BigSimmo/Database#1377: This change extends the index migration and schema-drift sequencing described in the operator documentation.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely identifies the primary change: restoring RAG search-health indexes.
Description check	✅ Passed	The description covers the change, verification evidence, risks, rollback, production effects, governance checks, and pending exact-head CI.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[supabase]

Updates to Preview Branch (codex/restore-rag-indexes-20260804) ↗︎

Deployments	Status	Updated
Database	✅	Wed, 05 Aug 2026 14:21:12 UTC
Services	✅	Wed, 05 Aug 2026 14:21:12 UTC
APIs	✅	Wed, 05 Aug 2026 14:21:12 UTC

Tasks are run on every commit but only new migration files are pushed.
Close and reopen this PR if you want to apply changes from existing seed or migration files.

Tasks	Status	Updated
Configurations	✅	Wed, 05 Aug 2026 14:21:14 UTC
Migrations	✅	Wed, 05 Aug 2026 14:21:16 UTC
Seeding	✅	Wed, 05 Aug 2026 14:21:18 UTC
Edge Functions	✅	Wed, 05 Aug 2026 14:21:19 UTC

---

View logs for this Workflow Run ↗︎.
Learn more about Supabase for Git ↗︎.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

[BigSimmo]

@codex review the exact current head. Focus on migration safety, idempotency, index definitions, rollback implications, and whether the tests actually pin the repository-to-production repair. Do not modify files or provider data.

[BigSimmo]

@codex resolve actionable Codex review findings for this pull request and current head using the repository instructions. This is the pull request's single automatic repair pass: do not perform a fresh review, create new standalone findings, or request another review. Work only the existing unresolved Codex threads on the current head. The only repository destination is BigSimmo/Database, and the only branch destination is the pull request head branch codex/restore-rag-indexes-20260804 at starting commit f932897; never publish fixes to a detached or synthetic work branch and never create a stacked pull request. Use the authenticated GitHub connector to commit each approved fix to BigSimmo/Database:codex/restore-rag-indexes-20260804, then verify that the pull request head contains the published commit before reporting success. Always fix P0 and P1 findings. For P2 and lower findings, fix only clear, scoped, low-risk issues; otherwise disposition them with a concise reason. For a fixed thread, reply with as the first line and as the second line. For a no-code disposition, use followed by . These result markers authorize the workflow to close that exact thread only after it verifies a fixed commit is the pull request head; a local-only commit is not a fix. If publication or verification fails, do not use either result marker, do not claim success, and leave the thread open with the blocker. If human input or new authorization is required, do the same. Finish only after every actionable thread is fixed or dispositioned and closed, or explicitly left open for a human decision. Do not update the branch from main, address unrelated reviews, broaden scope, or create more than one scoped fix commit. Do not use external APIs, paid services, credentials, dependency changes, or broad refactors unless explicitly authorized. Add targeted tests where behavior changes and run the narrowest relevant validation.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit:f932897760

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

CI triage

CI failed on this PR. Automated classification of the 2 failed job(s):

• Unit coverage — needs investigation: inspect the failing step and uploaded diagnostics; rerun only after classifying the cause.
• PR required — needs investigation: inspect the failing step and uploaded diagnostics; rerun only after classifying the cause.

Compared with main CI run #8060 (cancelled).

Classification is evidence routing, not permission to ignore a failure. Exact quarantined Playwright identities remain governed by the flake ledger.

[BigSimmo]

@codex review the exact current head 46533a9a1b6a524fa2d56f9c2cb3cfb54aa78542 read-only. Verify that the repair migration cannot start a write-blocking index build, fails fast when operator prebuild evidence is absent, remains safe on fresh replay, and that the test compares complete normalized index definitions across the canonical migration, schema, and drift manifest. Do not modify files or provider data.

[BigSimmo]

@codex please resolve failing CI testing and all comments and address any remaining issues with this PR

[chatgpt-codex-connector]

Summary

• Updated the Supabase schema index-definition normalizer to canonicalize pg_get_indexdef spacing between a qualified table name and its column list, so drift-manifest definitions like public.document_index_units (...) compare correctly against repository SQL definitions like public.document_index_units(...). tests/supabase-schema.test.tsL186-L194
• Committed the fix on the current branch with commit 3a07930d7a5d79554f9b10d1120a47a54f7679e9 (test: normalize restored index definitions).
• I could not create the requested pull request because no make_pr tool is available in this environment, and gh is not authenticated (gh auth status reported “You are not logged into any GitHub hosts.”).

Testing

• ❌ npm run test -- tests/supabase-schema.test.ts --runInBand — failed because Vitest does not support the Jest-style --runInBand option in this repo.
• ✅ npm run test -- tests/supabase-schema.test.ts
• ✅ npm run format
• ✅ npm run test -- tests/supabase-schema.test.ts

View task →

[BigSimmo]

@cursoragent Work the current open PR end-to-end. Confirm the PR number and GitHub head first from context. If more than one open PR could apply, stop and say which one you would use and why.

Fetch and start from the remote tip that matches that GitHub head. If the named branch ref is missing or stale, use the PR head ref. Preserve unrelated local WIP, including any local-only ledger commits; do not discard dirty work, and do not treat a local-ahead commit as the reviewed tip. Do not merge the PR, force-push, rebase, or arm auto-merge unless I explicitly ask. No provider-backed gates without separate approval. If you cannot push or resolve threads, diagnose and comment only; if inline replies fail, resolve when possible and put dispositions in the summary comment. If auto-merge is already armed, push only for a real blocker, and avoid pushes that would cancel in-flight required CI unless the push itself clears that blocker.

If the PR is already merged or closed: confirm the head and merge commit, note required-CI outcome, post one summary, and stop.

Goal: deep review plus Bugbot, fix actionable issues with the smallest correct changes, clear merge / required-CI / thread blockers, run strong local offline verification, push fixes, append the review ledger, and post one PR summary. Prefer thoroughness over speed. Regenerate large assets only when a fix requires it; then run the asset check and keep compatibility aliases byte-identical where the repo uses them.

Snapshot the GitHub head SHA: tip, base, behind/ahead, mergeable state, merge-tree versus origin/main (real conflict versus behind-but-clean), required checks on that tip including Production UI when selected, advisory separately, unresolved actionable threads. Missing checks while dirty are not green. If the tip moves mid-work, re-snapshot and continue from the new head.

Ledger-lookup against that GitHub head under the heavy review-and-fix scope for this PR. Already reviewed at this head with clean merge-tree, green required checks, and no new actionable threads → summarize, comment, stop unless I asked for a fresh superseding pass. Follow the repo review protocol.

Unblock once: real conflict → merge origin/main (prefer main’s shared queues; keep this PR’s notes); behind-but-clean → one sync, late if required CI is in flight; clean → leave. Labels like skip-branch-sync do not block a manual sync needed to clear a real blocker. No sync thrash. After any sync or push, re-snapshot tip, merge-tree, and required checks before declaring done. Dedupe the ledger if a merge touched it.

Review high-confidence delta risks only. Separate PR-introduced defects from pre-existing re-emitted debt. For generated assets, review contracts, aliases, cache, and manifests—not every generated line. If protected RAG or ranking surfaces are touched, say so before editing; fix PR-body policy text only when wrong or missing. Ignore bot noise. No nit spam or broad rewrites. Escalate verification by touched risk: clinical / RAG / privacy / migrations / auth → domain check plus production-readiness when warranted; UI / phone-chrome / routing / styling → ensure plus phone-chrome or UI gates when warranted; generated assets or docs inventory/links → asset or docs checks when warranted.

Fix P0/P1 always; clear scoped P2 when locally provable; else disposition and resolve. Required-check failures on this tip only; ignore advisory. Prefer reverting a bad autofix commit. After fixes: format and commit before push; smallest targeted proof; repo cheap gate; repo PR-local gate. No release, lighthouse, live eval, or live provider gates without approval.

Push only this PR’s fix commits. Append ledger for the final GitHub head under the heavy scope. Never push a ledger-only tip—include ledger in a real fix push, or leave a no-change ledger append local and unpushed.

One PR comment: tip, sync/merge-tree, fixed versus dispositioned, threads, required CI, decisive local gate lines, residual risks; merge left to me. Inline only for remaining human-needed P0/P1.

Stop when merge-tree is clean on the current GitHub tip, actionable threads are clear, required checks are green or clearly in progress, heavy gates for scope passed, summary posted, merge left to me.

REMEMBER TO GROUND YOUR RESPONSES AGAINST THE CURRENT REPO AND ENSURE THAT THE EDITS DO NOT CAUSE REGRESSION AND ADDITIONALLY, the edits make sense and take the entire repo and branch into account.

[cursor]

PR #1614 is merge-ready on tip ab3c7a30 — merge-tree clean, required CI green, threads cleared. Merge left to you.

What I fixed

• Migration guard now checks presence + indisvalid/indisready + normalized pg_get_indexdef (no transactional index builds)
• Schema normalizer fixes drift-style table ( vs table( spacing (the Unit coverage failure)
• PR policy body completed (all 7 governance items) via temporary PR_POLICY_BODY.md, then deleted

Status

Item	Result
Tip	ab3c7a30fad1a2dba536635391cf301b61aca674
Behind/ahead	0 / 12 · merge-tree clean · CLEAN
Threads	6/6 resolved
Required CI	PR policy, Unit coverage, Static PR, Migration replay, PR required — all SUCCESS
Local	verify:cheap + verify:pr-local — 5097 passed; schema tests 74/74

Residual

• Ledger append is local only (not pushed — ledger-only tip policy)
• Inline thread replies were forbidden for this token; dispositions are in the PR summary comment
• No live provider re-check this pass; prior live index health evidence retained in the PR body

[cursor]

PR #1614 review-and-fix summary

Tip:ab3c7a30fad1a2dba536635391cf301b61aca674 on codex/restore-rag-indexes-20260804
Base:main · ahead/behind: 12 / 0 · merge-tree: clean · mergeable: MERGEABLE (merge left to you)

Fixed

• P1 presence-only guard — migration now fails closed on missing, invalid (indisvalid/indisready), or definition-mismatched indexes via normalized pg_get_indexdef (no transactional CREATE INDEX).
• P1 normalizer spacing — normalizeIndexDefinition canonicalizes drift-style table ( vs schema table( so all four index defs pin across reconcile migration / schema.sql / drift-manifest.json.
• PR policy body — complete 7-item Clinical Governance Preflight + Risk: / Rollback: synced via temporary PR_POLICY_BODY.md, then deleted so it cannot overwrite other PRs (Role-pure categorical tone scale for service pathway pills #230).

Dispositioned / threads

• All six review threads resolved (Copilot + CodeRabbit + prior Codex). Inline reply posts were forbidden for this token; dispositions are here.
• Bugbot: no residual P0–P2 in the fixed tip beyond the above.

Required CI on tip

• PR policy SUCCESS
• Unit coverage SUCCESS
• Static PR checks SUCCESS
• Migration replay SUCCESS
• PR required SUCCESS
• Production UI: skipped (out of scope)

Local gates (decisive)

• npm run test -- tests/supabase-schema.test.ts → Tests 74 passed (74)
• npm run verify:cheap → Test Files 489 passed (489) / 5097 passed | 4 skipped
• npm run verify:pr-local → same unit suite + Offline RAG fixture and manifest validation passed (36 golden cases, 23 suites)
• npm run check:migration-role → hosted migration-role guard passed
• npm run check:rag:fixtures → passed

Residual risks

• PL/pgSQL index-def normalizer is duplicated beside the TS helper; exotic pg_get_indexdef shapes would fail closed (safe).
• Ledger row appended locally for this tip; not pushed (ledger-only tip policy).
• No provider-backed live re-validation in this pass; prior live search_schema_health / indisvalid evidence retained in the PR body.

Merge left to you.

[cursor]

Review findings disposition (pasted follow-up + babysit)

GitHub review threads were already resolved on tip ab3c7a30. This pass addressed the additional findings you listed against the restore-guard migration.

Finding	Verdict	Action
Session-scoped lock_timeout / statement_timeout leak	fixed	Switched to SET LOCAL (also for search_path) so later migrations on the same CLI session do not inherit the 5s/30s caps
Guard requires canonical names while health accepts aliases	fixed (contract pin)	None of the four restored indexes are in index_aliases today; schema test now fails if an alias key is added for any of them without updating the guard
Normalization verified against real pg_get_indexdef	replied — already green	Confirmed against pinned drift-manifest defs; no code change
WHERE-clause nested-paren / INCLUDE limitation	replied + documented	Flat $-anchored unwrap is enough for the two pinned predicates; scope called out in the migration header rather than inventing a nested parser
SQL/TS normalizer duplication	fixed	Migration comments + INDEX_DEFINITION_NORMALIZER_SQL_STEPS pin every SQL transform literal in the schema contract test
Root cause of earlier drift not addressed	captured	Outstanding issue #237 — out-of-band investigation of why 20260705180000 was applied yet indexes were missing
Index defs match schema.sql byte-for-byte	replied — already green	Confirmed by existing cross-source pin; no code change

Pushed tip:ea98ef85 on codex/restore-rag-indexes-20260804

Local proof

• npm run test -- tests/supabase-schema.test.ts → Tests 74 passed (74)
• npm run check:migration-role → hosted migration-role guard passed
• npm run verify:pr-local → Test Files 489 passed (489) / 5097 passed | 4 skipped + Offline RAG fixture and manifest validation passed (36 golden cases, 23 suites)

Skills

• /ce-resolve-pr-feedback + /ce-babysit-pr (checkpoint): no unresolved GitHub threads; findings above disposed on the PR head
• /ce-sweep: not configured in this repo (.compound-engineering/config.local.yaml missing feedback_sources) — first run needs interactive setup; skipped

Merge left to you after fresh exact-head CI is green.

[BigSimmo]

@cursoragent resolve merge conflict

[cursor]

Merge conflict on PR #1614 is resolved and pushed.

Conflict: only docs/outstanding-issues.md (real dirty merge vs main08595cc0)

Resolution: kept main’s #237–#245 from PR #1616; renumbered this PR’s search-health follow-up #237 → #246 (issues:next-id=247)

Tip:a87cec9c · merge-tree clean · mergeable: MERGEABLE · CI re-running · merge left to you

[cursor]

Merge conflict resolved

Tip:a87cec9c3efb01deecc8eb70db0d896aaff46f9f
Conflict:docs/outstanding-issues.md only (real dirty merge vs origin/main08595cc0)

Resolution: kept main’s #237–#245 rows from PR #1616; renumbered this PR’s search-health history follow-up from colliding #237 → #246 via issues:add; marker is now issues:next-id=247.

Checks:check:outstanding-issues passed · merge-tree clean · GitHub mergeable: MERGEABLE (CI re-running; merge left to you).