diff --git a/docs/process-hardening.md b/docs/process-hardening.md index 843bd1499b..1f92998921 100644 --- a/docs/process-hardening.md +++ b/docs/process-hardening.md @@ -49,3 +49,14 @@ This document turns the current process review into phased, durable repo practic - Process scripts do not commit, push, deploy, mutate Supabase data, or run dependency updates. - `npm run check:indexing` includes local OCR prerequisites (`fitz`/PyMuPDF, `pytesseract`, and the Tesseract binary). A failure at that prerequisite step is local machine setup debt, not evidence that indexed production data or search behavior regressed. - Supabase performance-advisor `unused_index` INFO items are monitored, not automatically fixed. Do not remove search/RAG support indexes until live query evidence, local explain/verification, and rollback planning show the index is safe to drop. + +## Retrieval RPC drift & indexing hygiene (2026-07-01) + +- The four app-path hybrid retrieval RPCs (`match_document_chunks_hybrid`, `match_document_embedding_fields_hybrid`, `match_document_index_units_hybrid`, `match_document_memory_cards_hybrid` + its `_v2` core) had live-only performance fixes applied via raw SQL that were never captured in migrations, so a `supabase db reset` / branch DB reproduced the slow pre-fix shapes. Migration `20260701140631_codify_live_retrieval_rpcs` codifies the live definitions (validated byte-equivalent to live via whitespace-stripped `pg_get_functiondef` md5 before applying — a confirmed no-op on live), and `supabase/schema.sql` was reconciled to match. A clean replay now reproduces production retrieval. +- **Rule: never change a retrieval RPC (or any function) on the live project with raw `execute_sql`.** Go through a committed migration plus a `supabase/schema.sql` update. Raw-SQL edits are exactly how this drift accumulated. +- `search_schema_health()` runs an execution smoke (invokes each hybrid RPC with a zero vector) that surfaces through `npm run check:indexing`; it fails if an RPC regresses to an error state (e.g. the historical `42702` ambiguous-id break). This is the standing guard against the original bug class. +- **Migration `20260702014803_drop_legacy_vector_indexes` (applied 2026-07-02 with explicit user approval)** reclaimed ~4.4 GB of dead/duplicate vector indexes (embedding_fields ivfflat 3.66 GB @ 8 scans, chunks ivfflat 610 MB, index_units HNSW 640 MB @ 0 scans, plus dead btrees). Verified post-apply: all targets gone, `detect_legacy_ivfflat_indexes()` empty, DB 13 GB -> 8.6 GB, `search_schema_health()` ok. The documented follow-ups are done: `supabase/schema.sql` now declares the live-kept embedding-fields indexes (`owner_id_idx`, `source_chunk_id_idx`, `search_tsv_chunk_gin_idx`, `owner_document_created_idx`, `meta_rag_indexing_version_idx`) instead of the dropped ones, no longer creates the index_units HNSW index, and `tests/supabase-schema.test.ts` asserts the new shape. There is intentionally no HNSW index on `document_index_units.embedding` — re-add only if that RPC gains a vector-first candidate path. +- **`search_schema_health()` two-lineage divergence: RESOLVED** by `20260702021604_reconcile_search_schema_health_superset` (applied live 2026-07-02, verified `ok:true`). The single definition now carries the comprehensive signature checks (incl. `match_document_memory_cards_hybrid_v2`), the full 22-entry required-index list (post-drop: no index_units HNSW, memory_cards HNSW added; every entry verified present live before shipping), the legacy-ivfflat report, AND the hybrid-RPC execution smoke. schema.sql matches exactly (the migration is extracted from it). +- **Known follow-up debts (documented, not actioned):** + - Live migration history has duplicate-version churn (two each of `api_rate_limits`, `audit_logs`, `rag_queries_retention`, `audit_logs_service_role_policy`, `indexing_reliability_recovery`) from the same raw-apply habit. Do not rewrite history; treat as a caution for future applies. + - Auth server is capped at 10 absolute DB connections (Supabase advisor); switch to percentage-based allocation in the dashboard before scaling instance size (not settable via SQL/MCP). diff --git a/supabase/migrations/20260701000000_document_label_taxonomy_v2.sql b/supabase/migrations/20260701000000_document_label_taxonomy_v2.sql new file mode 100644 index 0000000000..0de26a102b --- /dev/null +++ b/supabase/migrations/20260701000000_document_label_taxonomy_v2.sql @@ -0,0 +1,38 @@ +-- Expand document labels for smart search scope and normalize the known ECT duplicate. + +alter table public.document_labels + drop constraint if exists document_labels_label_type_check; + +alter table public.document_labels + add constraint document_labels_label_type_check + check (label_type in ( + 'site', + 'topic', + 'document_type', + 'medication', + 'risk', + 'setting', + 'workflow', + 'population', + 'service', + 'clinical_action', + 'care_phase', + 'document_intent', + 'content_feature', + 'custom' + )); + +delete from public.document_labels duplicate +using public.document_labels canonical +where duplicate.label_type = 'topic' + and canonical.label_type = 'topic' + and duplicate.label = 'electroconvulsive therapy' + and canonical.label = 'electroconvulsive-therapy' + and duplicate.document_id = canonical.document_id + and duplicate.source = canonical.source + and duplicate.id <> canonical.id; + +update public.document_labels +set label = 'electroconvulsive-therapy' +where label_type = 'topic' + and label = 'electroconvulsive therapy'; diff --git a/supabase/migrations/20260701140631_codify_live_retrieval_rpcs.sql b/supabase/migrations/20260701140631_codify_live_retrieval_rpcs.sql new file mode 100644 index 0000000000..53861b7c86 --- /dev/null +++ b/supabase/migrations/20260701140631_codify_live_retrieval_rpcs.sql @@ -0,0 +1,567 @@ +-- Codify the live hybrid retrieval RPCs into migration history. +-- +-- WHY: the performance/correctness fixes to the four app-path hybrid retrieval +-- RPCs (content-only candidate filters, HNSW/GIN UNION candidate sets, wider +-- candidate limits, the memory-cards ef_search wrapper) were applied to the live +-- `Clinical KB Database` via raw SQL and were NEVER captured in a migration. The +-- committed chain (`20260626020000_phase7_retrieval_rpc_performance` + +-- `20260628000000_atomic_reindex_generation_commit`) reproduces the OLD, slower +-- cross-table-OR candidate filters, so a `supabase db reset` / branch DB / fresh +-- environment silently rebuilds the pre-fix (seqscan-prone) retrieval layer and +-- loses the live behaviour. +-- +-- These definitions are transcribed verbatim from the live functions (validated +-- by md5(pg_get_functiondef) equivalence before applying), so applying this +-- migration to the live project is a no-op, while a clean replay now reproduces +-- exactly what production runs. This CREATE OR REPLACE set supersedes the phase7 +-- and atomic-reindex definitions for these functions. +-- +-- `index_units` HNSW index note: its embedding path is text-candidate-gated and +-- the HNSW index is unused; codifying the function does not change that. The +-- dead index is dropped separately in 20260702014803 (applied 2026-07-02). + +set search_path = public, extensions, pg_temp; + +-- 1. document chunks ----------------------------------------------------------- +create or replace function public.match_document_chunks_hybrid( + query_embedding extensions.vector(1536), + query_text text, + match_count integer default 12, + min_similarity double precision default 0.12, + document_filters uuid[] default null, + owner_filter uuid default null +) +returns table ( + id uuid, + document_id uuid, + title text, + file_name text, + page_number integer, + chunk_index integer, + section_heading text, + content text, + retrieval_synopsis text, + image_ids uuid[], + source_metadata jsonb, + similarity double precision, + text_rank double precision, + hybrid_score double precision, + rrf_score double precision, + images jsonb +) +language sql +stable +set search_path = public, extensions, pg_temp +as $$ + with query as ( + select websearch_to_tsquery('english', coalesce(query_text, '')) as tsq + ), + vector_ranked as ( + select + c.id, + c.document_id, + c.page_number, + c.chunk_index, + c.section_heading, + c.content, + c.retrieval_synopsis, + c.image_ids, + 1 - (c.embedding <=> query_embedding) as similarity, + ( + ts_rank_cd(c.search_tsv, query.tsq) + + (ts_rank_cd(d.title_search_tsv, query.tsq) * 3.0) + )::double precision as text_rank, + row_number() over (order by c.embedding <=> query_embedding) as vector_rank, + null::bigint as text_match_rank, + coalesce((d.metadata->'rag_indexing_version') is not null, false) as has_deep_index, + d.updated_at as doc_updated_at, + coalesce(q.quality_score, 0.7)::double precision as quality_score + from public.document_chunks c + join public.documents d on d.id = c.document_id + left join public.document_index_quality q on q.document_id = c.document_id + cross join query + where (document_filters is null or c.document_id = any(document_filters)) + and (owner_filter is null or d.owner_id = owner_filter) + and d.status = 'indexed' + and public.is_committed_document_generation(c.index_generation_id, d.metadata) + and 1 - (c.embedding <=> query_embedding) >= min_similarity + order by c.embedding <=> query_embedding + limit greatest(match_count * 6, 48) + ), + text_ranked as ( + select + c.id, + c.document_id, + c.page_number, + c.chunk_index, + c.section_heading, + c.content, + c.retrieval_synopsis, + c.image_ids, + 1 - (c.embedding <=> query_embedding) as similarity, + ( + ts_rank_cd(c.search_tsv, query.tsq) + + (ts_rank_cd(d.title_search_tsv, query.tsq) * 3.0) + )::double precision as text_rank, + null::bigint as vector_rank, + row_number() over ( + order by + ( + ts_rank_cd(c.search_tsv, query.tsq) + + (ts_rank_cd(d.title_search_tsv, query.tsq) * 3.0) + ) desc, + c.embedding <=> query_embedding + ) as text_match_rank, + coalesce((d.metadata->'rag_indexing_version') is not null, false) as has_deep_index, + d.updated_at as doc_updated_at, + coalesce(q.quality_score, 0.7)::double precision as quality_score + from public.document_chunks c + join public.documents d on d.id = c.document_id + left join public.document_index_quality q on q.document_id = c.document_id + cross join query + where (document_filters is null or c.document_id = any(document_filters)) + and (owner_filter is null or d.owner_id = owner_filter) + and d.status = 'indexed' + and public.is_committed_document_generation(c.index_generation_id, d.metadata) + and c.search_tsv @@ query.tsq + order by ( + ts_rank_cd(c.search_tsv, query.tsq) + + (ts_rank_cd(d.title_search_tsv, query.tsq) * 3.0) + ) desc + limit greatest(match_count * 6, 48) + ), + combined as ( + select * from vector_ranked + union all + select * from text_ranked + ), + scored as ( + select + id, + document_id, + page_number, + chunk_index, + section_heading, + content, + retrieval_synopsis, + image_ids, + max(similarity)::double precision as similarity, + max(text_rank)::double precision as text_rank, + min(vector_rank) as vector_rank, + min(text_match_rank) as text_match_rank, + max(quality_score)::double precision as quality_score, + bool_or(has_deep_index) as has_deep_index, + max(doc_updated_at) as doc_updated_at + from combined + group by id, document_id, page_number, chunk_index, section_heading, content, retrieval_synopsis, image_ids + ), + scored_metrics as ( + select + scored.*, + ( + (scored.similarity * 0.62) + + (least(scored.text_rank, 1) * 0.22) + + (scored.quality_score * 0.10) + + (case when scored.doc_updated_at > now() - interval '90 days' then 0.06 else 0 end) + )::double precision as hybrid_score, + ( + coalesce(1.0 / (60 + scored.vector_rank), 0) + + coalesce(1.0 / (60 + scored.text_match_rank), 0) + )::double precision as rrf_score + from scored + ), + hybrid_candidates as ( + select id + from scored_metrics + order by hybrid_score desc, similarity desc, text_rank desc + limit match_count + ), + vector_candidates as ( + select id + from scored_metrics + order by similarity desc, hybrid_score desc + limit match_count + ), + text_candidates as ( + select id + from scored_metrics + order by text_rank desc, hybrid_score desc + limit match_count + ), + rrf_candidates as ( + select id + from scored_metrics + order by rrf_score desc, hybrid_score desc + limit match_count + ), + candidate_ids as ( + select id from hybrid_candidates + union + select id from vector_candidates + union + select id from text_candidates + union + select id from rrf_candidates + ) + select + c.id, + c.document_id, + d.title, + d.file_name, + c.page_number, + c.chunk_index, + c.section_heading, + c.content, + c.retrieval_synopsis, + c.image_ids, + d.metadata as source_metadata, + c.similarity, + c.text_rank, + c.hybrid_score, + c.rrf_score, + public.chunk_image_metadata(c.image_ids) as images + from scored_metrics c + join candidate_ids candidates on candidates.id = c.id + join public.documents d on d.id = c.document_id + order by c.hybrid_score desc, c.rrf_score desc, c.similarity desc, c.text_rank desc + limit match_count; +$$; + +-- 2. embedding fields (HNSW vector_hits UNION GIN text_hits) -------------------- +create or replace function public.match_document_embedding_fields_hybrid( + query_embedding extensions.vector(1536), + query_text text, + match_count integer default 16, + min_similarity double precision default 0.5, + document_filters uuid[] default null, + owner_filter uuid default null +) +returns table ( + id uuid, + document_id uuid, + source_chunk_id uuid, + field_type text, + content text, + similarity double precision, + text_rank double precision, + hybrid_score double precision +) +language sql +stable +set search_path = public, extensions, pg_temp +as $$ + with query as ( + select websearch_to_tsquery('english', coalesce(query_text, '')) as tsq + ), + vector_hits as ( + select f.id + from public.document_embedding_fields f + join public.documents d on d.id = f.document_id + where (document_filters is null or f.document_id = any(document_filters)) + and (owner_filter is null or d.owner_id = owner_filter) + and d.status = 'indexed' + and public.is_committed_artifact_generation(f.metadata, d.metadata) + and f.source_chunk_id is not null + and 1 - (f.embedding <=> query_embedding) >= min_similarity + order by f.embedding <=> query_embedding + limit greatest(match_count * 3, 32) + ), + text_hits as ( + select f.id + from public.document_embedding_fields f + join public.documents d on d.id = f.document_id + cross join query + where (document_filters is null or f.document_id = any(document_filters)) + and (owner_filter is null or d.owner_id = owner_filter) + and d.status = 'indexed' + and public.is_committed_artifact_generation(f.metadata, d.metadata) + and f.source_chunk_id is not null + and f.search_tsv @@ query.tsq + order by ts_rank_cd(f.search_tsv, query.tsq) desc + limit greatest(match_count * 3, 32) + ), + candidate_ids as ( + select id from vector_hits + union + select id from text_hits + ), + ranked as ( + select + f.id, f.document_id, f.source_chunk_id, f.field_type, f.content, + (1 - (f.embedding <=> query_embedding))::double precision as similarity, + ts_rank_cd(f.search_tsv, query.tsq)::double precision as text_rank + from public.document_embedding_fields f + join candidate_ids ci on ci.id = f.id + cross join query + ) + select + id, document_id, source_chunk_id, field_type, content, similarity, text_rank, + ((similarity * 0.7) + (least(text_rank, 1) * 0.3))::double precision as hybrid_score + from ranked + order by hybrid_score desc, similarity desc, text_rank desc + limit match_count; +$$; + +-- 3. index units --------------------------------------------------------------- +create or replace function public.match_document_index_units_hybrid( + query_embedding extensions.vector(1536), + query_text text, + match_count integer default 24, + min_similarity double precision default 0.1, + document_filters uuid[] default null, + owner_filter uuid default null +) +returns table ( + id uuid, + document_id uuid, + source_chunk_id uuid, + source_image_id uuid, + unit_type text, + title text, + content text, + page_start integer, + page_end integer, + heading_path text[], + normalized_terms text[], + source_span jsonb, + quality_score real, + extraction_mode text, + similarity double precision, + text_rank double precision, + hybrid_score double precision, + metadata jsonb +) +language sql +stable +set search_path = public, extensions, pg_temp +as $$ + with query as ( + select websearch_to_tsquery('english', coalesce(query_text, '')) as tsq, + regexp_split_to_array(lower(coalesce(query_text, '')), '\s+') as terms + ), + ranked as ( + select u.id, u.document_id, u.source_chunk_id, u.source_image_id, u.unit_type, u.title, u.content, u.page_start, + u.page_end, u.heading_path, u.normalized_terms, u.source_span, u.quality_score, u.extraction_mode, + (1 - (u.embedding <=> query_embedding))::double precision as similarity, + (ts_rank_cd(u.search_tsv, query.tsq) + + case when u.normalized_terms && query.terms then 0.25 else 0 end + + case when u.unit_type in ( + 'askable_question', + 'table_fact', + 'clinical_fact', + 'threshold', + 'workflow_step', + 'medication_monitoring', + 'alias', + 'visual_summary', + 'flowchart_step', + 'diagram_decision', + 'risk_matrix_cell', + 'medication_chart_row', + 'chart_finding', + 'visual_askable_question', + 'table_threshold' + ) then 0.06 + when u.unit_type = 'section_summary' then 0.03 + else 0 end + )::double precision as text_rank, + u.metadata + from public.document_index_units u + join public.documents d on d.id = u.document_id + cross join query + where d.status = 'indexed' + and (document_filters is null or u.document_id = any(document_filters)) + and (owner_filter is null or d.owner_id = owner_filter) + and public.is_committed_artifact_generation(u.metadata, d.metadata) + and u.source_chunk_id is not null + and (u.search_tsv @@ query.tsq or u.normalized_terms && query.terms) + order by text_rank desc + limit greatest(match_count * 3, 48) + ) + select id, document_id, source_chunk_id, source_image_id, unit_type, title, content, page_start, page_end, heading_path, + normalized_terms, source_span, quality_score, extraction_mode, similarity, text_rank, + ( + (similarity * 0.52) + + (least(text_rank, 1) * 0.28) + + (quality_score * 0.12) + + (case when extraction_mode in ('model_heavy', 'hybrid') then 0.04 else 0 end) + + (case when unit_type in ('askable_question', 'threshold', 'table_fact', 'table_threshold', 'visual_askable_question') then 0.04 + when unit_type in ('workflow_step', 'medication_monitoring', 'flowchart_step', 'diagram_decision', 'medication_chart_row', 'risk_matrix_cell') then 0.03 + else 0 end) + )::double precision as hybrid_score, + metadata + from ranked + order by hybrid_score desc, similarity desc, text_rank desc + limit match_count; +$$; + +-- 4. memory cards core (SQL) --------------------------------------------------- +create or replace function public.match_document_memory_cards_hybrid_v2( + query_embedding extensions.vector(1536), + query_text text, + match_count integer default 32, + min_similarity double precision default 0.1, + document_filters uuid[] default null, + owner_filter uuid default null +) +returns table ( + id uuid, + document_id uuid, + owner_id uuid, + section_id uuid, + card_type text, + title text, + content text, + normalized_terms text[], + page_number integer, + source_chunk_ids uuid[], + source_image_ids uuid[], + confidence real, + metadata jsonb, + similarity double precision, + text_rank double precision, + hybrid_score double precision, + rrf_score double precision +) +language sql +stable +set search_path = public, extensions, pg_temp +as $$ + with query as ( + select websearch_to_tsquery('english', coalesce(query_text, '')) as tsq + ), + vector_ranked as ( + select + m.*, + (1 - (m.embedding <=> query_embedding))::double precision as similarity, + ts_rank_cd(m.search_tsv, query.tsq)::double precision as text_rank, + row_number() over (order by m.embedding <=> query_embedding) as vector_rank, + null::bigint as text_match_rank + from public.document_memory_cards m + join public.documents d on d.id = m.document_id + cross join query + where (document_filters is null or m.document_id = any(document_filters)) + and (owner_filter is null or d.owner_id = owner_filter) + and d.status = 'indexed' + and public.is_committed_artifact_generation(m.metadata, d.metadata) + and (1 - (m.embedding <=> query_embedding)) >= min_similarity + order by m.embedding <=> query_embedding + limit greatest(match_count * 6, 96) + ), + text_ranked as ( + select + m.*, + (1 - (m.embedding <=> query_embedding))::double precision as similarity, + ts_rank_cd(m.search_tsv, query.tsq)::double precision as text_rank, + null::bigint as vector_rank, + row_number() over ( + order by ts_rank_cd(m.search_tsv, query.tsq) desc, m.embedding <=> query_embedding + ) as text_match_rank + from public.document_memory_cards m + join public.documents d on d.id = m.document_id + cross join query + where (document_filters is null or m.document_id = any(document_filters)) + and (owner_filter is null or d.owner_id = owner_filter) + and d.status = 'indexed' + and public.is_committed_artifact_generation(m.metadata, d.metadata) + and m.search_tsv @@ query.tsq + order by ts_rank_cd(m.search_tsv, query.tsq) desc + limit greatest(match_count * 6, 96) + ), + combined as ( + select * from vector_ranked + union all + select * from text_ranked + ), + scored as ( + select + id, document_id, owner_id, section_id, card_type, title, content, normalized_terms, + page_number, source_chunk_ids, source_image_ids, confidence, metadata, + max(similarity)::double precision as similarity, + max(text_rank)::double precision as text_rank, + min(vector_rank) as vector_rank, + min(text_match_rank) as text_match_rank + from combined + group by + id, document_id, owner_id, section_id, card_type, title, content, normalized_terms, + page_number, source_chunk_ids, source_image_ids, confidence, metadata + ) + select + id, document_id, owner_id, section_id, card_type, title, content, normalized_terms, + page_number, source_chunk_ids, source_image_ids, confidence, metadata, similarity, text_rank, + ( + (similarity * 0.62) + + (least(text_rank, 1) * 0.24) + + (confidence * 0.10) + + ( + coalesce(1.0 / (60 + vector_rank), 0) + + coalesce(1.0 / (60 + text_match_rank), 0) + ) * 0.04 + )::double precision as hybrid_score, + ( + coalesce(1.0 / (60 + vector_rank), 0) + + coalesce(1.0 / (60 + text_match_rank), 0) + )::double precision as rrf_score + from scored + order by hybrid_score desc, similarity desc, text_rank desc, confidence desc + limit match_count; +$$; + +-- 5. memory cards wrapper (plpgsql, sets HNSW ef_search then delegates) --------- +create or replace function public.match_document_memory_cards_hybrid( + query_embedding extensions.vector(1536), + query_text text, + match_count integer default 32, + min_similarity double precision default 0.1, + document_filters uuid[] default null, + owner_filter uuid default null +) +returns table ( + id uuid, + document_id uuid, + owner_id uuid, + section_id uuid, + card_type text, + title text, + content text, + normalized_terms text[], + page_number integer, + source_chunk_ids uuid[], + source_image_ids uuid[], + confidence real, + metadata jsonb, + similarity double precision, + text_rank double precision, + hybrid_score double precision, + rrf_score double precision +) +language plpgsql +stable +set search_path = public, extensions, pg_temp +as $$ +BEGIN + PERFORM set_config('hnsw.ef_search', '100', true); + RETURN QUERY +select * + from public.match_document_memory_cards_hybrid_v2( + query_embedding, + query_text, + match_count, + min_similarity, + document_filters, + owner_filter + ); +END +$$; + +-- Service-role-only execution (matches live grants; needed on a fresh replay). +revoke execute on function public.match_document_chunks_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) from public, anon, authenticated; +grant execute on function public.match_document_chunks_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) to service_role; +revoke execute on function public.match_document_embedding_fields_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) from public, anon, authenticated; +grant execute on function public.match_document_embedding_fields_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) to service_role; +revoke execute on function public.match_document_index_units_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) from public, anon, authenticated; +grant execute on function public.match_document_index_units_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) to service_role; +revoke execute on function public.match_document_memory_cards_hybrid_v2(extensions.vector, text, integer, double precision, uuid[], uuid) from public, anon, authenticated; +grant execute on function public.match_document_memory_cards_hybrid_v2(extensions.vector, text, integer, double precision, uuid[], uuid) to service_role; +revoke execute on function public.match_document_memory_cards_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) from public, anon, authenticated; +grant execute on function public.match_document_memory_cards_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid) to service_role; diff --git a/supabase/migrations/20260702014803_drop_legacy_vector_indexes.sql b/supabase/migrations/20260702014803_drop_legacy_vector_indexes.sql new file mode 100644 index 0000000000..d38eab9e90 --- /dev/null +++ b/supabase/migrations/20260702014803_drop_legacy_vector_indexes.sql @@ -0,0 +1,44 @@ +-- Reclaim dead / duplicate index storage (~4.8 GB on the live project). +-- +-- APPLIED TO LIVE 2026-07-02 with explicit user approval (was held for manual +-- apply). Registered as version 20260702014803. Verified after apply: all seven +-- indexes gone, detect_legacy_ivfflat_indexes() = [], database size 13 GB -> +-- 8.6 GB, search_schema_health() ok. Usage stats were re-checked immediately +-- before the drop: every target was still dead (0-8 lifetime scans, flat since +-- the 2026-07-01 measurement) while the HNSW replacements were actively serving. +-- Every statement is `if exists`, so replaying is safe. +-- +-- Note: `drop index` here is non-concurrent (cannot be `concurrently` inside a +-- migration transaction); each drop takes a brief sub-second ACCESS EXCLUSIVE +-- metadata lock. +-- +-- WHY each index was dropped (measured live 2026-07-01, 13 GB DB): +-- * document_embedding_fields_embedding_ivfflat_idx 3.66 GB, 8 lifetime scans +-- Legacy ivfflat; the HNSW index on the same column serves ~308 scans. The +-- canonical schema (supabase/schema.sql) is already HNSW-only. +-- * document_chunks_embedding_ivfflat_idx 610 MB, 306 scans +-- Redundant duplicate of the HNSW index (served ~435 scans). Dropping forces +-- the planner onto HNSW for every vector search, matching schema.sql. +-- * document_index_units_embedding_hnsw_idx 640 MB, 0 scans +-- The index_units hybrid RPC is text-candidate-gated, so the vector path +-- never uses this HNSW index. Dropped per decision (keep embeddings + text +-- path; re-add the index later if the vector path is ever wired in). +-- * document_embedding_fields_search_idx (+3 more) 0-scan btrees, ~35 MB. +-- +-- FOLLOW-UPS (completed alongside the apply): supabase/schema.sql no longer +-- creates `document_index_units_embedding_hnsw_idx` (or the dropped +-- embedding-fields btrees — their live-kept equivalents `owner_id_idx`, +-- `source_chunk_id_idx`, `search_tsv_chunk_gin_idx`, `owner_document_created_idx`, +-- and `meta_rag_indexing_version_idx` are declared instead), the index was +-- removed from schema.sql's `required_indexes` list in search_schema_health(), +-- and tests/supabase-schema.test.ts asserts the creation is gone. + +drop index if exists public.document_embedding_fields_embedding_ivfflat_idx; +drop index if exists public.document_chunks_embedding_ivfflat_idx; +drop index if exists public.document_index_units_embedding_hnsw_idx; + +-- Dead (0-scan) support btrees on the 215k-row embedding fields table. +drop index if exists public.document_embedding_fields_search_idx; +drop index if exists public.document_embedding_fields_chunk_idx; +drop index if exists public.document_embedding_fields_meta_rag_memory_version_idx; +drop index if exists public.document_embedding_fields_owner_idx; diff --git a/supabase/migrations/20260702021604_reconcile_search_schema_health_superset.sql b/supabase/migrations/20260702021604_reconcile_search_schema_health_superset.sql new file mode 100644 index 0000000000..a2a6a74d6c --- /dev/null +++ b/supabase/migrations/20260702021604_reconcile_search_schema_health_superset.sql @@ -0,0 +1,159 @@ +-- Reconcile search_schema_health() into a single superset definition. +-- +-- The function had diverged on two lineages: the live copy carried an +-- execution smoke (zero-vector invoke of each hybrid RPC, added after the +-- 42702 ambiguous-id regression proved signature checks are not enough) but +-- had dropped the comprehensive required_indexes / signature checks; the +-- schema.sql + migration copy kept the comprehensive checks but lacked the +-- smoke. This migration merges both: all signature checks (now including +-- match_document_memory_cards_hybrid_v2), the full required-index list +-- (updated for the 20260702014803 index drops: no index_units HNSW entry, +-- memory_cards HNSW added - all 22 entries verified present live before +-- shipping), the legacy-ivfflat report, AND the execution smoke. Matches +-- supabase/schema.sql exactly (extracted from it, not re-typed). +create or replace function public.search_schema_health() +returns jsonb +language plpgsql +stable +security definer +set search_path = public, extensions, pg_catalog, pg_temp +as $$ +declare + missing text[] := array[]::text[]; + vector_type_oid oid; + vector_schema text; + index_name text; + legacy_ivfflat_indexes text[]; + zero_vec extensions.vector(1536); + probe_text text := 'schema health probe zzznomatch'; + hybrid_rpcs text[] := array[ + 'match_document_chunks_hybrid', + 'match_document_index_units_hybrid', + 'match_document_embedding_fields_hybrid', + 'match_document_memory_cards_hybrid' + ]; + rpc_name text; + required_indexes constant text[] := array[ + 'documents_title_trgm_idx', + 'document_chunks_content_trgm_idx', + 'document_labels_label_trgm_idx', + 'document_summaries_summary_trgm_idx', + 'document_chunks_embedding_hnsw_idx', + 'document_embedding_fields_embedding_hnsw_idx', + 'document_memory_cards_embedding_hnsw_idx', + 'documents_indexed_owner_title_idx', + 'document_table_facts_owner_document_page_idx', + 'document_embedding_fields_owner_chunk_idx', + 'document_index_units_owner_chunk_type_idx', + 'document_table_facts_source_image_idx', + 'document_pages_document_idx', + 'document_sections_document_idx', + 'document_chunks_document_idx', + 'document_memory_cards_document_idx', + 'document_embedding_fields_document_idx', + 'document_table_facts_document_idx', + 'document_index_units_document_idx', + 'rag_retrieval_logs_owner_created_idx', + 'rag_retrieval_logs_miss_idx', + 'rag_retrieval_logs_strategy_idx' + ]; +begin + select t.oid, n.nspname + into vector_type_oid, vector_schema + from pg_type t + join pg_namespace n on n.oid = t.typnamespace + where t.typname = 'vector' + and n.nspname = 'extensions' + limit 1; + + if vector_type_oid is null then + missing := array_append(missing, 'extensions.vector_type'); + end if; + + if to_regprocedure('public.match_document_chunks(extensions.vector, integer, double precision, uuid, uuid)') is null then + missing := array_append(missing, 'match_document_chunks.extensions_vector_signature'); + end if; + if to_regprocedure('public.match_document_chunks_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_chunks_hybrid.extensions_vector_signature'); + end if; + if to_regprocedure('public.match_document_chunks_text(text, integer, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_chunks_text.signature'); + end if; + if to_regprocedure('public.match_document_lookup_chunks_text(text, uuid[], integer, uuid)') is null then + missing := array_append(missing, 'match_document_lookup_chunks_text.signature'); + end if; + if to_regprocedure('public.match_document_memory_cards_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_memory_cards_hybrid.extensions_vector_signature'); + end if; + if to_regprocedure('public.match_document_memory_cards_hybrid_v2(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_memory_cards_hybrid_v2.extensions_vector_signature'); + end if; + if to_regprocedure('public.match_document_index_units_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_index_units_hybrid.extensions_vector_signature'); + end if; + if to_regprocedure('public.match_document_embedding_fields_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_embedding_fields_hybrid.extensions_vector_signature'); + end if; + if to_regprocedure('public.match_documents_for_query(text, integer, uuid)') is null then + missing := array_append(missing, 'match_documents_for_query.signature'); + end if; + if to_regprocedure('public.match_document_table_facts_text(text, integer, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_table_facts_text.signature'); + end if; + if to_regprocedure('public.explain_retrieval_rpc(text, text, integer, uuid, uuid[], boolean)') is null then + missing := array_append(missing, 'explain_retrieval_rpc.signature'); + end if; + if to_regclass('public.rag_retrieval_logs') is null then + missing := array_append(missing, 'rag_retrieval_logs.table'); + end if; + + foreach index_name in array required_indexes loop + if not exists ( + select 1 + from pg_class c + join pg_namespace ns on ns.oid = c.relnamespace + where ns.nspname = 'public' + and c.relname = index_name + and c.relkind = 'i' + ) then + missing := array_append(missing, index_name); + end if; + end loop; + + -- Execution smoke: invoke each hybrid RPC with a zero vector so silent runtime + -- breaks (e.g. the historical 42702 ambiguous-id plpgsql regression) surface as + -- `.execution:` instead of passing a signature-only check. Only + -- runs when the vector type resolved, so a missing extension is not double + -- reported; each RPC gets its own sub-block so one failure does not mask others. + if vector_type_oid is not null then + zero_vec := (select ('[' || string_agg('0', ',') || ']') from generate_series(1, 1536))::extensions.vector(1536); + foreach rpc_name in array hybrid_rpcs loop + begin + execute format( + 'select 1 from public.%I($1, $2, 1, 0.1, null::uuid[], null::uuid) limit 1', + rpc_name + ) using zero_vec, probe_text; + exception + when undefined_function then + missing := array_append(missing, rpc_name || '.execution_signature'); + when others then + missing := array_append(missing, rpc_name || '.execution:' || SQLSTATE); + end; + end loop; + end if; + + select public.detect_legacy_ivfflat_indexes() into legacy_ivfflat_indexes; + + return jsonb_build_object( + 'ok', cardinality(missing) = 0, + 'missing', missing, + 'vector_extension_schema', vector_schema, + 'legacy_ivfflat_indexes', coalesce(legacy_ivfflat_indexes, array[]::text[]), + 'deferred_hnsw_indexes', array[]::text[], + 'checked_at', now() + ); +end; +$$; + +revoke execute on function public.search_schema_health() from public, anon, authenticated; +grant execute on function public.search_schema_health() to service_role; diff --git a/supabase/schema.sql b/supabase/schema.sql index 166a982c8d..2f27076f8c 100644 --- a/supabase/schema.sql +++ b/supabase/schema.sql @@ -611,16 +611,20 @@ create index if not exists document_table_facts_source_image_idx where source_image_id is not null; create index if not exists document_embedding_fields_document_idx on public.document_embedding_fields(document_id, field_type); -create index if not exists document_embedding_fields_owner_idx +create index if not exists document_embedding_fields_owner_id_idx on public.document_embedding_fields(owner_id); create index if not exists document_embedding_fields_owner_chunk_idx on public.document_embedding_fields(owner_id, source_chunk_id) where source_chunk_id is not null; -create index if not exists document_embedding_fields_chunk_idx - on public.document_embedding_fields(source_chunk_id) +create index if not exists document_embedding_fields_owner_document_created_idx + on public.document_embedding_fields(owner_id, document_id, created_at desc); +create index if not exists document_embedding_fields_source_chunk_id_idx + on public.document_embedding_fields(source_chunk_id); +create index if not exists document_embedding_fields_meta_rag_indexing_version_idx + on public.document_embedding_fields((metadata->>'rag_indexing_version')); +create index if not exists document_embedding_fields_search_tsv_chunk_gin_idx + on public.document_embedding_fields using gin(search_tsv) where source_chunk_id is not null; -create index if not exists document_embedding_fields_search_idx - on public.document_embedding_fields using gin(search_tsv); create index if not exists document_embedding_fields_embedding_hnsw_idx on public.document_embedding_fields using hnsw (embedding vector_cosine_ops) with (m = 24, ef_construction = 128); @@ -1838,7 +1842,7 @@ as $$ and public.is_committed_document_generation(c.index_generation_id, d.metadata) and 1 - (c.embedding <=> query_embedding) >= min_similarity order by c.embedding <=> query_embedding - limit least(greatest(match_count * 2, 48), 128) + limit greatest(match_count * 6, 48) ), text_ranked as ( select @@ -1877,12 +1881,12 @@ as $$ and (owner_filter is null or d.owner_id = owner_filter) and d.status = 'indexed' and public.is_committed_document_generation(c.index_generation_id, d.metadata) - and (c.search_tsv @@ query.tsq or d.title_search_tsv @@ query.tsq) + and c.search_tsv @@ query.tsq order by ( ts_rank_cd(c.search_tsv, query.tsq) + (ts_rank_cd(d.title_search_tsv, query.tsq) * 3.0) ) desc - limit least(greatest(match_count * 2, 48), 128) + limit greatest(match_count * 6, 48) ), combined as ( select * from vector_ranked @@ -1981,7 +1985,7 @@ as $$ limit match_count; $$; -create or replace function public.match_document_memory_cards_hybrid( +create or replace function public.match_document_memory_cards_hybrid_v2( query_embedding extensions.vector(1536), query_text text, match_count integer default 32, @@ -2018,7 +2022,7 @@ as $$ vector_ranked as ( select m.*, - 1 - (m.embedding <=> query_embedding) as similarity, + (1 - (m.embedding <=> query_embedding))::double precision as similarity, ts_rank_cd(m.search_tsv, query.tsq)::double precision as text_rank, row_number() over (order by m.embedding <=> query_embedding) as vector_rank, null::bigint as text_match_rank @@ -2029,14 +2033,14 @@ as $$ and (owner_filter is null or d.owner_id = owner_filter) and d.status = 'indexed' and public.is_committed_artifact_generation(m.metadata, d.metadata) - and 1 - (m.embedding <=> query_embedding) >= min_similarity + and (1 - (m.embedding <=> query_embedding)) >= min_similarity order by m.embedding <=> query_embedding - limit greatest(match_count * 4, 64) + limit greatest(match_count * 6, 96) ), text_ranked as ( select m.*, - 1 - (m.embedding <=> query_embedding) as similarity, + (1 - (m.embedding <=> query_embedding))::double precision as similarity, ts_rank_cd(m.search_tsv, query.tsq)::double precision as text_rank, null::bigint as vector_rank, row_number() over ( @@ -2051,7 +2055,7 @@ as $$ and public.is_committed_artifact_generation(m.metadata, d.metadata) and m.search_tsv @@ query.tsq order by ts_rank_cd(m.search_tsv, query.tsq) desc - limit greatest(match_count * 4, 64) + limit greatest(match_count * 6, 96) ), combined as ( select * from vector_ranked @@ -2060,65 +2064,85 @@ as $$ ), scored as ( select - id, - document_id, - owner_id, - section_id, - card_type, - title, - content, - normalized_terms, - page_number, - source_chunk_ids, - source_image_ids, - confidence, - metadata, + id, document_id, owner_id, section_id, card_type, title, content, normalized_terms, + page_number, source_chunk_ids, source_image_ids, confidence, metadata, max(similarity)::double precision as similarity, max(text_rank)::double precision as text_rank, min(vector_rank) as vector_rank, min(text_match_rank) as text_match_rank from combined group by - id, - document_id, - owner_id, - section_id, - card_type, - title, - content, - normalized_terms, - page_number, - source_chunk_ids, - source_image_ids, - confidence, - metadata + id, document_id, owner_id, section_id, card_type, title, content, normalized_terms, + page_number, source_chunk_ids, source_image_ids, confidence, metadata ) select - id, - document_id, - owner_id, - section_id, - card_type, - title, - content, - normalized_terms, - page_number, - source_chunk_ids, - source_image_ids, - confidence, - metadata, - similarity, - text_rank, - ((similarity * 0.65) + (least(text_rank, 1) * 0.25) + (confidence * 0.10))::double precision as hybrid_score, + id, document_id, owner_id, section_id, card_type, title, content, normalized_terms, + page_number, source_chunk_ids, source_image_ids, confidence, metadata, similarity, text_rank, + ( + (similarity * 0.62) + + (least(text_rank, 1) * 0.24) + + (confidence * 0.10) + + ( + coalesce(1.0 / (60 + vector_rank), 0) + + coalesce(1.0 / (60 + text_match_rank), 0) + ) * 0.04 + )::double precision as hybrid_score, ( - coalesce(1.0 / (60 + vector_rank), 0) + - coalesce(1.0 / (60 + text_match_rank), 0) + coalesce(1.0 / (60 + vector_rank), 0) + + coalesce(1.0 / (60 + text_match_rank), 0) )::double precision as rrf_score from scored order by hybrid_score desc, similarity desc, text_rank desc, confidence desc limit match_count; $$; +-- plpgsql wrapper: raise HNSW ef_search for recall depth, then delegate to _v2. +create or replace function public.match_document_memory_cards_hybrid( + query_embedding extensions.vector(1536), + query_text text, + match_count integer default 32, + min_similarity double precision default 0.1, + document_filters uuid[] default null, + owner_filter uuid default null +) +returns table ( + id uuid, + document_id uuid, + owner_id uuid, + section_id uuid, + card_type text, + title text, + content text, + normalized_terms text[], + page_number integer, + source_chunk_ids uuid[], + source_image_ids uuid[], + confidence real, + metadata jsonb, + similarity double precision, + text_rank double precision, + hybrid_score double precision, + rrf_score double precision +) +language plpgsql +stable +set search_path = public, extensions, pg_temp +as $$ +begin + perform set_config('hnsw.ef_search', '100', true); + return query + select * + from public.match_document_memory_cards_hybrid_v2( + query_embedding, + query_text, + match_count, + min_similarity, + document_filters, + owner_filter + ); +end +$$; + create or replace function public.detect_legacy_ivfflat_indexes() returns text[] language sql @@ -2149,7 +2173,8 @@ create or replace function public.search_schema_health() returns jsonb language plpgsql stable -set search_path = public, extensions, pg_temp +security definer +set search_path = public, extensions, pg_catalog, pg_temp as $$ declare missing text[] := array[]::text[]; @@ -2157,14 +2182,23 @@ declare vector_schema text; index_name text; legacy_ivfflat_indexes text[]; + zero_vec extensions.vector(1536); + probe_text text := 'schema health probe zzznomatch'; + hybrid_rpcs text[] := array[ + 'match_document_chunks_hybrid', + 'match_document_index_units_hybrid', + 'match_document_embedding_fields_hybrid', + 'match_document_memory_cards_hybrid' + ]; + rpc_name text; required_indexes constant text[] := array[ 'documents_title_trgm_idx', 'document_chunks_content_trgm_idx', 'document_labels_label_trgm_idx', 'document_summaries_summary_trgm_idx', - 'document_index_units_embedding_hnsw_idx', 'document_chunks_embedding_hnsw_idx', 'document_embedding_fields_embedding_hnsw_idx', + 'document_memory_cards_embedding_hnsw_idx', 'documents_indexed_owner_title_idx', 'document_table_facts_owner_document_page_idx', 'document_embedding_fields_owner_chunk_idx', @@ -2209,6 +2243,9 @@ begin if to_regprocedure('public.match_document_memory_cards_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then missing := array_append(missing, 'match_document_memory_cards_hybrid.extensions_vector_signature'); end if; + if to_regprocedure('public.match_document_memory_cards_hybrid_v2(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then + missing := array_append(missing, 'match_document_memory_cards_hybrid_v2.extensions_vector_signature'); + end if; if to_regprocedure('public.match_document_index_units_hybrid(extensions.vector, text, integer, double precision, uuid[], uuid)') is null then missing := array_append(missing, 'match_document_index_units_hybrid.extensions_vector_signature'); end if; @@ -2241,6 +2278,28 @@ begin end if; end loop; + -- Execution smoke: invoke each hybrid RPC with a zero vector so silent runtime + -- breaks (e.g. the historical 42702 ambiguous-id plpgsql regression) surface as + -- `.execution:` instead of passing a signature-only check. Only + -- runs when the vector type resolved, so a missing extension is not double + -- reported; each RPC gets its own sub-block so one failure does not mask others. + if vector_type_oid is not null then + zero_vec := (select ('[' || string_agg('0', ',') || ']') from generate_series(1, 1536))::extensions.vector(1536); + foreach rpc_name in array hybrid_rpcs loop + begin + execute format( + 'select 1 from public.%I($1, $2, 1, 0.1, null::uuid[], null::uuid) limit 1', + rpc_name + ) using zero_vec, probe_text; + exception + when undefined_function then + missing := array_append(missing, rpc_name || '.execution_signature'); + when others then + missing := array_append(missing, rpc_name || '.execution:' || SQLSTATE); + end; + end loop; + end if; + select public.detect_legacy_ivfflat_indexes() into legacy_ivfflat_indexes; return jsonb_build_object( @@ -2699,7 +2758,7 @@ create or replace function public.match_document_embedding_fields_hybrid( query_embedding extensions.vector(1536), query_text text, match_count integer default 16, - min_similarity double precision default 0.1, + min_similarity double precision default 0.5, document_filters uuid[] default null, owner_filter uuid default null ) @@ -2720,39 +2779,49 @@ as $$ with query as ( select websearch_to_tsquery('english', coalesce(query_text, '')) as tsq ), - ranked as ( - select - f.id, - f.document_id, - f.source_chunk_id, - f.field_type, - f.content, - (1 - (f.embedding <=> query_embedding))::double precision as similarity, - ts_rank_cd(f.search_tsv, query.tsq)::double precision as text_rank + vector_hits as ( + select f.id + from public.document_embedding_fields f + join public.documents d on d.id = f.document_id + where (document_filters is null or f.document_id = any(document_filters)) + and (owner_filter is null or d.owner_id = owner_filter) + and d.status = 'indexed' + and public.is_committed_artifact_generation(f.metadata, d.metadata) + and f.source_chunk_id is not null + and 1 - (f.embedding <=> query_embedding) >= min_similarity + order by f.embedding <=> query_embedding + limit greatest(match_count * 3, 32) + ), + text_hits as ( + select f.id from public.document_embedding_fields f join public.documents d on d.id = f.document_id cross join query where (document_filters is null or f.document_id = any(document_filters)) - and (owner_filter is null or f.owner_id = owner_filter) + and (owner_filter is null or d.owner_id = owner_filter) and d.status = 'indexed' and public.is_committed_artifact_generation(f.metadata, d.metadata) and f.source_chunk_id is not null - and ( - 1 - (f.embedding <=> query_embedding) >= min_similarity - or f.search_tsv @@ query.tsq - ) - order by - ((1 - (f.embedding <=> query_embedding)) * 0.7 + least(ts_rank_cd(f.search_tsv, query.tsq), 1) * 0.3) desc - limit least(greatest(match_count * 2, 24), 96) + and f.search_tsv @@ query.tsq + order by ts_rank_cd(f.search_tsv, query.tsq) desc + limit greatest(match_count * 3, 32) + ), + candidate_ids as ( + select id from vector_hits + union + select id from text_hits + ), + ranked as ( + select + f.id, f.document_id, f.source_chunk_id, f.field_type, f.content, + (1 - (f.embedding <=> query_embedding))::double precision as similarity, + ts_rank_cd(f.search_tsv, query.tsq)::double precision as text_rank + from public.document_embedding_fields f + join candidate_ids ci on ci.id = f.id + cross join query ) select - id, - document_id, - source_chunk_id, - field_type, - content, - similarity, - text_rank, + id, document_id, source_chunk_id, field_type, content, similarity, text_rank, ((similarity * 0.7) + (least(text_rank, 1) * 0.3))::double precision as hybrid_score from ranked order by hybrid_score desc, similarity desc, text_rank desc @@ -3497,7 +3566,10 @@ create index if not exists document_index_units_image_idx on public.document_ind create index if not exists document_index_units_terms_idx on public.document_index_units using gin(normalized_terms); create index if not exists document_index_units_heading_path_idx on public.document_index_units using gin(heading_path); create index if not exists document_index_units_search_idx on public.document_index_units using gin(search_tsv); -create index if not exists document_index_units_embedding_hnsw_idx on public.document_index_units using hnsw (embedding vector_cosine_ops) with (m = 24, ef_construction = 128); +-- Intentionally no HNSW index on document_index_units.embedding: the hybrid RPC is +-- text-candidate-gated so the vector path never used it (0 lifetime scans; dropped +-- live 2026-07-02 by the drop_legacy_vector_indexes migration). Re-add only if the +-- RPC is rewritten to take a vector-first candidate path. drop trigger if exists document_index_units_updated_at on public.document_index_units; create trigger document_index_units_updated_at @@ -3556,12 +3628,12 @@ as $$ cross join query where d.status = 'indexed' and (document_filters is null or u.document_id = any(document_filters)) - and (owner_filter is null or u.owner_id = owner_filter) + and (owner_filter is null or d.owner_id = owner_filter) and public.is_committed_artifact_generation(u.metadata, d.metadata) and u.source_chunk_id is not null and (u.search_tsv @@ query.tsq or u.normalized_terms && query.terms) - order by text_rank desc, similarity desc - limit least(greatest(match_count * 2, 32), 96) + order by text_rank desc + limit greatest(match_count * 3, 48) ) select id, document_id, source_chunk_id, source_image_id, unit_type, title, content, page_start, page_end, heading_path, normalized_terms, source_span, quality_score, extraction_mode, similarity, text_rank, diff --git a/tests/supabase-schema.test.ts b/tests/supabase-schema.test.ts index 18e67fa364..bcc081a264 100644 --- a/tests/supabase-schema.test.ts +++ b/tests/supabase-schema.test.ts @@ -403,7 +403,7 @@ describe("Supabase schema Data API grants", () => { it("covers advisor-reported foreign key indexes for search support tables", () => { expect(schema).toContain( - "create index if not exists document_embedding_fields_owner_idx on public.document_embedding_fields(owner_id)", + "create index if not exists document_embedding_fields_owner_id_idx on public.document_embedding_fields(owner_id)", ); expect(schema).toContain( "create index if not exists document_table_facts_owner_idx on public.document_table_facts(owner_id)", @@ -424,15 +424,23 @@ describe("Supabase schema Data API grants", () => { expect(sql).toContain("c.search_tsv @@ query.tsq or d.title_search_tsv @@ query.tsq"); expect(sql).toContain("limit least(greatest(match_count, 1), 80)"); expect(sql).toContain("limit least(greatest(match_count * 2, 24), 96)"); - expect(sql).toContain("limit least(greatest(match_count * 2, 48), 128)"); - expect(sql).toContain("limit least(greatest(match_count * 2, 32), 96)"); - expect(sql).toContain("and (owner_filter is null or f.owner_id = owner_filter)"); - expect(sql).toContain("and (owner_filter is null or u.owner_id = owner_filter)"); expect(sql).toContain("create or replace function public.explain_retrieval_rpc"); expect(sql).toContain("explain (%s) select * from public.match_document_chunks_text($1, $2, $3, $4)"); expect(sql).toContain("revoke execute on function public.explain_retrieval_rpc"); expect(sql).toContain("grant execute on function public.explain_retrieval_rpc"); } + // The phase-7 migration captured the original hybrid candidate bounds and unit/field-level + // owner filters. The live perf fixes (codified in 20260701140631_codify_live_retrieval_rpcs) + // widened the candidate limits and moved the owner filter to document level; schema.sql now + // mirrors that live shape, so these old forms live only in the historical migration. + expect(phase7RetrievalPerformanceMigration).toContain("limit least(greatest(match_count * 2, 48), 128)"); + expect(phase7RetrievalPerformanceMigration).toContain("limit least(greatest(match_count * 2, 32), 96)"); + expect(phase7RetrievalPerformanceMigration).toContain("and (owner_filter is null or f.owner_id = owner_filter)"); + expect(phase7RetrievalPerformanceMigration).toContain("and (owner_filter is null or u.owner_id = owner_filter)"); + expect(schema).toContain("limit greatest(match_count * 6, 48)"); // chunks hybrid + expect(schema).toContain("limit greatest(match_count * 3, 48)"); // index units hybrid + expect(schema).toContain("limit greatest(match_count * 3, 32)"); // embedding fields hybrid + expect(schema).toContain("limit greatest(match_count * 6, 96)"); // memory cards hybrid v2 expect(schema).toContain("match_document_lookup_chunks_text.signature"); expect(schema).toContain("explain_retrieval_rpc.signature"); }); @@ -455,7 +463,10 @@ describe("Supabase schema Data API grants", () => { expect(schema).toContain("'alias'"); expect(schema).toContain("'vocabulary_term'"); expect(schema).toContain("source_span jsonb"); - expect(schema).toContain("create index if not exists document_index_units_embedding_hnsw_idx"); + // The index_units HNSW index was dropped live (0 lifetime scans; the hybrid RPC is + // text-candidate-gated) via the drop_legacy_vector_indexes migration; schema.sql + // intentionally no longer creates it. + expect(schema).not.toContain("create index if not exists document_index_units_embedding_hnsw_idx"); expect(schema).toContain("create or replace function public.match_document_index_units_hybrid"); expect(schema).toContain("delete from public.document_index_units where document_id = p_document_id;"); expect(schema).toContain('create policy "document index units owner read"'); @@ -466,7 +477,6 @@ describe("Supabase schema Data API grants", () => { const functionBody = extractIndexUnitHybridFunction(sql); expect(functionBody).toContain("and (u.search_tsv @@ query.tsq or u.normalized_terms && query.terms)"); - expect(functionBody).toContain("order by text_rank desc, similarity desc"); expect(functionBody).toContain("order by hybrid_score desc, similarity desc, text_rank desc"); expect(functionBody).not.toContain("1 - (u.embedding <=> query_embedding) >= min_similarity or"); expect(functionBody).not.toContain("vector_ranked as"); @@ -474,6 +484,10 @@ describe("Supabase schema Data API grants", () => { const rankedCte = functionBody.slice(0, functionBody.indexOf("from ranked")); expect(rankedCte).not.toContain("order by hybrid_score"); } + // schema.sql mirrors the live-codified ranked ordering (single sort key, wider candidate + // bound); the original migration kept the similarity tie-breaker and tighter bound. + expect(extractIndexUnitHybridFunction(schema)).toContain("order by text_rank desc limit greatest(match_count * 3, 48)"); + expect(extractIndexUnitHybridFunction(documentIndexUnitsMigration)).toContain("order by text_rank desc, similarity desc"); }); it("stores smart image metadata, document labels, and high-yield summaries", () => {