diff --git a/docs/redesign/m2-atomic-reindex-migration-note.md b/docs/redesign/m2-atomic-reindex-migration-note.md new file mode 100644 index 0000000000..0e5660a7ef --- /dev/null +++ b/docs/redesign/m2-atomic-reindex-migration-note.md @@ -0,0 +1,55 @@ +# M2 Atomic Reindex Migration Note + +## Goal + +Make reindex completion atomic so search, evaluation, telemetry, and document views never observe a partially replaced index generation. + +This branch is intentionally docs-only before implementation. Code changes should start only after this note is reviewed. + +## Current Risk + +The existing reindex path can delete or replace index artifacts before the replacement generation is fully available. A failure between removal and completion can leave a document with missing chunks, stale retrieval metadata, or inconsistent `index_generation_id` references. + +## API And Config Touchpoints + +- `src/app/api/documents/[id]/reindex/route.ts`: single-document reindex entrypoint and response contract. +- `src/app/api/documents/bulk/reindex/route.ts`: bulk reindex request shape and partial-failure reporting. +- `src/lib/reindex-pipeline.ts`: orchestration boundary for extraction, chunking, embeddings, artifact writes, and final status. +- `src/lib/ingestion.ts` and `worker/main.ts`: shared indexing behavior that must preserve compatibility with upload and background ingestion. +- `supabase/functions/indexing-v3-agent/index.ts` and `supabase/functions/indexing-v3-agent/behavior.ts`: remote worker behavior and retry/claim semantics. +- Supabase tables: `documents`, `document_chunks`, `document_index_units`, `document_memory_cards`, `document_table_facts`, `document_embedding_fields`, and related visual/image tables. +- Supabase RPCs/functions: `match_document_chunks`, `match_document_chunks_hybrid`, `match_document_chunks_text`, `match_documents_for_query`, `claim_indexing_v3_agent_jobs`, and indexing completion/failure RPCs. +- Config/env: Supabase project ref `sjrfecxgysukkwxsowpy`, storage buckets, `OPENAI_API_KEY`, worker retry settings, and local no-auth owner defaults used by eval scripts. + +## Compatibility Assumptions + +- Existing public API payloads remain backwards compatible: reindex routes may add fields, but must not remove current fields or change success/failure status semantics. +- Existing search and eval flows continue to accept documents without a new generation until the atomic cutover completes. +- Retrieval RPCs must ignore staged, incomplete generations by default. +- Existing telemetry keys remain valid. New generation metadata can be additive. +- Rollback must preserve the last completed generation until the new generation is fully committed. + +## Proposed Migration Shape + +1. Add a durable generation record or equivalent metadata state for `pending`, `committing`, `completed`, and `failed`. +2. Write all replacement artifacts with a new generation id while leaving the previous completed generation readable. +3. Switch the document's active generation in one database transaction or RPC after all required artifacts validate. +4. Delete or mark stale artifacts only after the active-generation switch succeeds. +5. Update retrieval RPC filters so default reads use only the active completed generation. +6. Add repair/recovery logic for stuck `pending` or `committing` generations. + +## Required Tests + +- Unit tests for generation-state transitions and rollback behavior. +- Route tests for single and bulk reindex response compatibility. +- Worker tests for retry after extraction, embedding, and DB-write failures. +- Retrieval tests proving staged generations are invisible until commit. +- Eval/search smoke tests proving existing telemetry and source links remain compatible. + +## Acceptance Criteria + +- A failed reindex leaves the previous searchable generation intact. +- A successful reindex switches all retrieval surfaces to one completed generation. +- No search result mixes chunks from old and new generations for the same document. +- Recovery tooling can identify and clean abandoned staged generations. +- `npm run verify:cheap` passes, plus the smallest relevant retrieval/indexing checks.