Uh oh!
There was an error while loading. Please reload this page.
RFC: vector search support across storage backends - #236
Conversation
DynamoDB reached GA with native vector search on 2026-08-05: a new index type over an attribute holding an embedding, plus a SearchVectors operation returning nearest neighbours ranked by a distance function with exact-match inline filtering. An application that adopts it cannot use ExtendDB for that workload at all, since CreateTable rejects the index type and SearchVectors is unknown. The RFC separates the parts that must be identical everywhere from the part that cannot be. Wire surface, validation, score sign convention, index lifecycle and consistency are defined once in the engine. ANN execution is a declared per-backend capability (Ann, ExactScan, Unsupported), because pgvector and sqlite-vec are operator-installed extensions and MongoDB's vector search exists in Atlas but not in community mongod. Mandating ANN would convert a feature gap into a backend gap. Two design points carry the most risk and are called out deliberately. Score semantics are asymmetric: lower is more similar for Cosine and Euclidean, higher for DotProduct, so normalising to a single similarity inverts ranking for two of three functions. And a vector index is eventually consistent like a GSI, so the existing pending-index queue and gsi_propagation_delay_ms should carry it rather than a second async mechanism, with the index forbidden from reporting ACTIVE while its backfill is still draining. Recall is not assertable by equality, so ExactScan is proposed as the test oracle with ANN held to a recall threshold against it. Five service behaviours are listed as unresolved and needing live verification rather than guessed at, including the dimension-mismatch error on PutItem and whether a missing vector attribute yields sparse-index behaviour.
| `CreateTable` on an unknown index type, or at `SearchVectors` on an unknown | ||
| operation. Local development, CI, and any offline or edge deployment that | ||
| depends on ExtendDB stops being able to exercise the code path at all. The gap | ||
| is not a missing optimisation, it is a missing feature that silently pushes |
There was a problem hiding this comment.
This feels like a bit of an overstatement, especially for a feature that's been out for a week. :-) I think this can be simplified to vector search being a new first-class feature of DynamoDB, so ExtendDB should support it to support its mission of letting applications written against DynamoDB run unchanged against ExtendDB. it's that simple.
| Evidence the surface is real and stable enough to target: it is GA in all | ||
| commercial Regions plus GovCloud (US), it is documented in the DynamoDB | ||
| Developer Guide, and it is exposed in the console, CLI, SDKs and |
There was a problem hiding this comment.
It'd be good to provide links to documentation references.
| 1. Index creation is asynchronous and the index reports `CREATING` until the | ||
| backfill over existing items completes, then `ACTIVE`. | ||
| 2. The index must not report `ACTIVE` while its backfill is still running. A |
There was a problem hiding this comment.
Is this specific to vector indexes?
| 1. `crates/core`: index configuration and `SearchVectors` request and response | ||
| types, dimension and top K validation, distance function enum. | ||
| 2. `crates/storage`: extend `TableEngine` for vector index create, describe and | ||
| delete; add the search entry point to `DataEngine`; add the capability |
There was a problem hiding this comment.
It sounds like it's not just "capability declaration", but actually "detection". Is that right? In other words, since for at least some backends ANN capability is based on configuration or product version, there needs to be a runtime check: we can't compile the capability directly into the plugin because the plugin will run on different instances with different capabilities. Correct?
| types, dimension and top K validation, distance function enum. | ||
| 2. `crates/storage`: extend `TableEngine` for vector index create, describe and | ||
| delete; add the search entry point to `DataEngine`; add the capability | ||
| declaration to `Backend`. This is the breaking trait change and should land |
There was a problem hiding this comment.
Can it default to something reasonable so it's not breaking?
| our build. | ||
| - **A correct-but-slow mode invites misreading.** `ExactScan` will be measured | ||
| by somebody as though it were ANN, and we will be compared unfavourably. The | ||
| mitigation is documentation and an explicit capability in `DescribeTable`, not |
There was a problem hiding this comment.
The DynamoDB DescribeTable? How does that work?
| Design questions for reviewers specifically: | ||
| 6. Should vector search be optional for backend acceptance? I propose yes. |
There was a problem hiding this comment.
Yes. Maybe in 2-10 years we can revisit.
| 6. Should vector search be optional for backend acceptance? I propose yes. | ||
| 7. What recall threshold should ANN backends be held to in conformance, and | ||
| should it be a hard gate or a reported metric? | ||
| 8. Should `ExactScan` be allowed in a non-development build at all, or should it |
There was a problem hiding this comment.
Is the concern "cost"? Is the cost evident to the developer (kind of like 'scan' is its own operation in part to enforce understanding that its performance and costs is quite different from other operations).
jcshepherd
left a comment
There was a problem hiding this comment.
A few naive questions below. Also, I wonder if it'd be helpful to outline a PG or MySQL implementation: in particular, what the physical index schema is and what the backend query looks like. I don't have a good feel for either.
Summary
DynamoDB reached GA with native vector search on 2026-08-05: a new index type created over an attribute holding an embedding, plus a
SearchVectorsoperation returning nearest neighbours ranked by a distance function, with optional exact-match inline filtering.An application that adopts it cannot use ExtendDB for that workload at all today.
CreateTablerejects the unknown index type andSearchVectorsis an unknown operation, so local development, CI, and offline or edge deployments lose the ability to exercise the code path.This RFC proposes adding that surface.
The central design question
The RFC separates what must be identical across backends from what cannot be.
Wire surface, validation, score sign convention, index lifecycle and the consistency model are defined once in the engine. ANN execution is a declared per-backend capability (
Ann,ExactScan,Unsupported), because the stores diverge sharply:pgvector, but only if the operator installed itsqlite-vec, likewise a loadable extensionmongodMandating ANN everywhere would convert a feature gap into a backend gap. The recommendation is that vector search be optional for backend acceptance, provided a backend that cannot do it fails closed at
CreateTablewith a typed error rather than returning wrong results later.Two points that carry most of the risk
Score semantics are asymmetric. Lower is more similar for
CosineandEuclidean(0 is identical); higher is more similar forDotProduct. Normalising all three into one "similarity" number inverts the ranking for two of them. I have seen this exact defect in a first-party client library against the real service, so the RFC asks for the sign convention to be asserted per distance function rather than inferred from result sets.A vector index is eventually consistent, the same model as a GSI. So the proposal reuses the existing pending-index queue and
gsi_propagation_delay_msrather than adding a second asynchronous mechanism. It also requires that an index must not reportACTIVEwhile its backfill is still draining, with the transition causally ordered after the drain rather than scheduled on a wall clock beside it.Testing
Recall is not assertable by equality, so
ExactScanis proposed as the oracle (perfect recall) with ANN backends held to a recall threshold against it. Everything deterministic (sign convention, top K truncation, partition-key scoping, filter rejection, lifecycle) is asserted exactly.What I am asking reviewers for
ExactScanbe reachable in a production configuration at all, or gated?Five service behaviours are listed as unresolved rather than guessed at, including the dimension-mismatch error on
PutItemand whether a missing vector attribute gives sparse-index behaviour. I am happy to verify each against the real service before the corresponding code is written.Note on numbering: the file is
0000-vector-search.mdper the process indocs/rfcs/README.md, which keeps0000until acceptance.