feat(sqlite): vector search index

[LeeroyHannigan]

What

Implements vector indexes and SearchVectors on the SQLite backend, making it the first backend to declare the optional VectorSearchEngine capability added by the contract #243 this is stacked on.

Working end to end:

• CreateTable with VectorIndexes, and UpdateTable create/delete with a real backfill of items already in the table
• DescribeTable reports the indexes and their status
• Exact-scan SearchVectors for COSINE, EUCLIDEAN and DOT_PRODUCT, with
  per-metric ordering, partition scoping, TopK, inline equality filters and ProjectionExpression
• Index maintenance at all six write sites, including the three TransactWriteItems branches
• Crash recovery: a vector index left CREATING is dropped and rebuilt at startup
• Cleanup: dropping a table takes its vector data tables with it

Storage layout is one row per vector, chosen on write amplification rather than read speed. A packed per-partition blob reads 2 to 4 times faster but would rewrite the whole blob to insert one vector: roughly 390 MB per PutItem for a
100k-vector partition at 1024 dimensions. Recorded in the ADR.

The backfill status sequence follows what the service was measured to report, not what seemed reasonable: CREATING with Backfilling: false, then CREATING with true, then ACTIVE with the member absent. Observed on 2026-08-06 by seeding 3000 items of 1024 dimensions so the backfill took 8.5 minutes and could be sampled. false comes first, so presence does not imply backfilling and a client must read the value rather than test for the member.

Why

The contract PR models the vector index surface but no backend implements it, so no vector index can be created in tree. This makes it real on SQLite, which is the backend that ships in the dev/test image.

It also closes a hole the contract's post-condition guard exposed: this backend declared the capability while ignoring vector_index_updates, so an UpdateTable create returned 200 and created nothing, and a delete returned 200 while the index stayed ACTIVE and kept returning hits.

Stacked on the vector contract PR. Review that one first.

Closes #

Testing done

Verified against a live SQLite-backed server over HTTP, not in process:

# server on the SQLite backend, throttling enforcement on as CI does
extenddb init --backend sqlite && extenddb settings set throttling_enabled true
extenddb serve
cd tests/rust && EXTENDDB_EXPECT_VECTORS=1 cargo test -- --test-threads=1
# 445 passed; 0 failed; 0 filtered out
cargo test # 773 passed; 0 failed; 0 filtered out
cargo fmt --all --check # clean
cargo clippy --all-targets -- -D warnings # clean
cargo clippy --all-targets --no-default-features --features sqlite -- -D warnings # clean

25 vector wire tests, covering nearest-first ordering, TopK, overwrite, delete, unindexed items, tenant isolation in both directions, partition moves when the HASH attribute changes, the dot-product ordering inversion, backfill of pre-existing
items (including one without a vector that must be skipped rather than break the scan), later writes still indexed, index delete stopping both serving and reporting, base items surviving an index delete, duplicate-create and missing-delete rejection, and the f32 narrowing.

Three things were mutation-checked rather than trusted green:

• Neutering the backfill to write nothing fails both backfill tests, and the second then reports only the later-written item, so it genuinely distinguishes backfilled rows from live-written ones.
• Removing the drop from the reconciler fails the partial-table test. That test exists because the crash that actually happens leaves the data table holding some rows, and resuming would index those items twice; the simpler no-table crash case cannot catch it, since the drop is a no-op there.
• Weakening the unscoped-partition sentinel fails the invariant test

Also verified directly against the database rather than inferred:

• After the full suite: 21 vector data tables against exactly 21 catalog rows, zero orphans, zero missing.
• The catalog version bump was exercised by migrating a live deployment 0.0.2 to 0.0.3 and confirming the server starts.
• Row payload at 1024 dimensions is 37 bytes against a 4,096-byte blob, down from 19,862 bytes before the vector was removed from the payload.

Measured scan throughput is roughly 1,000 vectors per 10 ms at 1024 dimensions on one core. Worth stating plainly because an earlier modelled figure of 36,000 was wrong by 10 to 30 times: the cost is the SQLite read path, not the arithmetic, and a zero-copy &[f32] view measured no faster.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
  
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk
  format, or public CLI surface, an RFC has been accepted or is linked
  below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: docs/adr/0004-vector-search-exact-scan.md (added here), RFC #236

Breaking changes

On-disk format. The SQLite catalog version moves from 0.0.3 to the version carrying the vector_indexes table, so an existing deployment must run extenddb migrate before the server will start. The migration is the idempotent schema apply; it was exercised on a live deployment rather than reasoned about.

No wire-protocol or trait changes here: those are all in the contract PR.

Known gaps, stated rather than discovered

• Index maintenance is synchronous, where the service is eventually consistent like a GSI. Being fresher cannot return a wrong answer, so this is safe, but it is a parity divergence and it keeps vector work on the write's critical path. The async path should reuse the existing gsi_pending queue, which already has crash recovery and per-key FIFO. Not in this PR yet.
• Wire coverage is still missing for BatchWriteItem, composite base keys, the scoped-index SearchConditionExpression requirement, and wrong-dimension requests.

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

[yesyayen]

Question: where must the index-build lifecycle code live?

The GSI build is synchronous and backend builds the index in one transaction, and the table is locked for writes until the build is complete. Each backend has its own copy of this logic, and each copy is approximately 100 lines.

The vector backfill runs as a detached task, and the table stays open for writes during the build. Because writes can occur during the build, this PR adds approximately 1,500 lines of new logic: the CREATING to ACTIVE state machine, the queue hold and replay, the poison-row handling, the watchdog, and the startup reconciliation.
When postgres implements VectorSearchEngine, a second copy of all this logic is necessary. The GSI code shows what occurs when a backend keeps its own copy. The two backfill_gsi copies do not agree today: the postgres copy does not report CREATING, it has no crash reconciler, and it keeps the all-0xFF bound defect that this PR removed from sqlite.

My proposal is this: put the state machine in shared engine code, and let each backend supply only the storage primitives (scan a batch, insert a row, drop a table). If the decision is to do this later, lets write that decision in ADR-0004.

[LeeroyHannigan]

Thanks for this review — every finding held up, and two turned out worse than reported. All six are addressed across both branches.

R3 (if_not_exists bypass) was the most severe: the expression-form check also missed the identical hole on TransactWriteItems, and apply_update turned out to have eight call sites across the three backends. The fix is one shared apply_update_validated in core that all eight use, validating the post-update image (the same convention the backends already use for secondary-index keys), so a future backend cannot forget it. Nine tests, negative-controlled: with the wrapper reverted, exactly the rejection tests fail.

R1 (capacity) was half right — search-side existed, write-side had zero occurrences. Rather than guess, I measured against the real service (20 probes): VectorWriteRequestBytes = max(dimensions * 4 + projected_non_vector_bytes, 1024), doubled when the search-schema HASH value moves the entry between partitions. Two things the public docs get wrong, both measured: the charge follows the PROJECTED entry changing (a non-indexed attribute change is charged under ALL, not under KEYS_ONLY), and VectorIndexes is reported for INDEXES only, not TOTAL. Wired through Put/Update/Delete with six wire tests pinning exact figures. One honest gap remains: BatchWriteItem and TransactWriteItems aggregate per-table units without item images, so they don't yet report vector charges — the same documented follow-up that already covers their missing per-GSI breakdowns, not a claim of support.

R2 (validation gaps) was real but the rules already existed on the child branch with a better billing-mode check (absent BillingMode defaults to PROVISIONED and is rejected). The contract branch now carries only what was genuinely missing — the update-path enforcement, where the service diverges from create: the sixth index is LimitExceededException on UpdateTable vs ValidationException on CreateTable, and the key-collision message embeds both schemas. All messages probed byte-exact.

R5 (poison rows) was worse than reported: one malformed item also froze all async index maintenance for the table via the CREATING hold. Backfill now skips, counts, and surfaces poison rows instead of wedging.

R6 — transient errors now propagate so the worker's existing retry-next-pass loop handles them; only genuinely malformed items are dropped, with classification at the drop site.

R4 — agreed on the divergence risk, but with one backend implementing the lifecycle today the shared state machine would be an abstraction with a single consumer (~490 lines moved speculatively). Recorded as ADR-0005 (0004 was taken) with the concrete trigger: the machinery moves to the engine when a second backend implements vector indexes, and the Postgres divergences you cite are listed there as the evidence.

Happy to walk through the capacity probes if useful — the raw figures are in the commit.

completed

[robinnsc]

Approving with one real finding flagged inline (a collision between inline maintenance and an in flight backfill when the propagation delay is 0), plus two small notes:

1. On the UpdateTable delete path the catalog row is removed inside the transaction but the data table drop happens after commit. If the drop errors, the caller gets an error for a delete that already took effect, and the data table is orphaned with nothing that ever reclaims it, since the reconciler only looks at CREATING rows. Probably the same shape as the GSI sibling, but worth a tracking note.
2. The two consecutive sightings rule in the stuck build sweep depends on the registry insert in build_vector_index staying before the spawn. The sweep side explains this nicely; the registration site does not mention that the sweep relies on its ordering, so a future refactor could move it without knowing what breaks. One sentence there would tie them together.

[robinnsc]

This comment is not quite true, and the gap behind it is reachable. The backfill calls insert_vector_row directly from backfill_vector_batch, without going through apply_vector_index, so there is a caller that never does the preceding delete.

That matters once writes can land during a backfill. The CREATING hold protects the queued path, but with index_propagation_delay_ms set to 0 (the documented synchronous mode) maintain_vector_indexes applies inline, and fetch_vector_indexes_for_table has no status filter, so a live write inserts a row into the still CREATING index's table. When the backfill later reaches that base rowid, this plain INSERT hits a primary key violation, the batch fails, and the index sits in CREATING until the stuck build sweep rebuilds it. Under sustained writes at delay 0 the rebuild can keep colliding, so the index may never reach ACTIVE while searches are refused the whole time.

Two ways out, either seems fine. The inline path could treat CREATING indexes the way the queue does and enqueue instead of applying, so the hold covers both paths uniformly. Or the backfill could tolerate an existing row, since a row present at backfill time was written from a base value at least as new as the one the batch just read, so replacing or skipping are both sound. That trades away the loud failure property this comment argues for, so the first option feels more in keeping with the design.

Not blocking since the default delay is nonzero and the window needs a write during a backfill, but the failure mode is an index that never activates, which is worth closing before this pattern gets copied to the Postgres backend.

[LeeroyHannigan]

@robinnsc your approval got auto-dismissed by one more commit (8c53454), so this needs a re-approve. The delta since what you reviewed is small and self-contained: while resolving the merge I probed the service and found it distinguishes the two missing-HASH failure modes, so the guard now fires only when no expression is supplied, the validator carries the full measured message for an expression that omits the HASH ('...in configured SearchSchema'), and the HASH-completeness check runs before the out-of-schema check to match the service's precedence (probed both orderings today). One new unit test pins the two-message split. Full gates on the tip: fmt/clippy clean both feature sets, 970 lib tests 0 filtered, 45/45 vector integration live.