feat(engine): Vector index and SearchVectors contract for storage backends

[LeeroyHannigan]

What this does

Models DynamoDB's vector index surface in extenddb-core and makes it implementable by a storage backend, without implementing it for any. Both in-tree backends still refuse every vector operation, and those refusals are proven over the wire rather than by calling the validator directly.

CreateTable and UpdateTable accept VectorIndexes / VectorIndexUpdates, DescribeTable reports them, SearchVectors is a first-class operation with its own consumed-capacity shape, and item writes validate the vector attribute and the search-schema attributes against the index definition.

Design docs: docs/adr/0004-vector-search-exact-scan.md (added on the stacked implementation branch), RFC #236.

Vector search is an optional capability

This is the first optional feature on a trait surface where every other method is mandatory, so the shape is worth a look:

// A separate trait, with NO default bodies.pubtraitVectorSearchEngine:Send + Sync{fnsearch_vectors(&self,req:VectorSearch<'_>) -> BoxedFuture<'_,VectorSearchResult>;}// One accessor on DataEngine, defaulting to None.fnas_vector_search(&self) -> Option<&dynVectorSearchEngine>{None}

Declaring support therefore requires handing over an implementation. An earlier revision used a supports_vector_indexes() -> bool, which let a backend return true and never implement the method: an honour system where the type system should be doing the work.

The accessor deliberately sits on DataEngine, notStorageEngine. StorageEngine comes from a blanket impl over the six focused traits, so a defaulted method there could never be overridden by a backend. It would have looked correct and been unoverridable.

StorageError::Unsupported is added so a backend declining a feature is not reported as an internal fault. If we ever want a backend that declines transactions or streams, this is the template: peel the feature into its own trait with no defaults, add one accessor to a trait backends already implement, gate in core so the refusal never reaches storage.

Measured against the live service

Most specifics were measured against real DynamoDB rather than inferred, and in nearly every case the measurement contradicted the inference:

Fact	Inferred	Measured
DistanceFunction order	alphabetical, or declaration order	[DOT_PRODUCT, COSINE, EUCLIDEAN], neither
Inline filter cap	20, from the query-side limit	18
SearchSchema HASH elements	exactly one, required	at most one, and optional
SearchConditionExpression	always required	required only when a HASH is declared
Component range	-	representable as f32; f32::MAX exactly is accepted
Excess decimal precision	rejected	accepted
N output form	may use an exponent	never; the 38-digit limit bounds significant digits
TableThroughputMode	a CreateTable member	not a member at all

That last one mattered: an alias for it made ExtendDB accept a request AWS ignores.

Error messages are pinned by whole-string equality, not by fragment. Fragment assertions that began after the One or more parameter values were invalid prefix hid three divergences: a colon where the service uses a full stop (twice), and one message missing the prefix entirely.

Deliberately absent

Please read these as scope, not omissions:

• No backend implements this, so no vector index can be created in tree yet. The SQLite implementation is a stacked branch and will follow.
• Backfill state and the UpdateTable create/delete paths are modelled but unvalidated, because validating them needs a backend that actually performs a backfill. This is the portion most likely to move, and reviewer time is probably better spent elsewhere.
• Async propagation of index writes is not modelled.
• Errors report the bare field name where the service reports a positional path such as vectorIndexes.1.member.distanceFunction. A serde deserializer for the enum cannot know its position, so closing this means deserialising permissively and validating positionally, as the Projection check already does. Its own change.

Verification

Refusals are asserted over the wire against a live server, using a SigV4 raw-request helper because no SDK version carries the vector types. That suite immediately earned its place: it found that UpdateTable's at-least-one check omitted VectorIndexUpdates, so a request carrying only vector index changes was rejected as empty, contradicting what the live service accepts. The unit tests call the validator directly and never reach that check, so it was invisible to them.

Both vector suites self-skip on the wrong kind of backend, and the expectation is pinned per CI job (EXTENDDB_EXPECT_VECTORS), so a backend that silently lost or gained the capability fails rather than skipping green.

• 427 Rust integration tests, 0 filtered out
• 735 workspace tests, 0 filtered out
• cargo fmt --check clean, clippy --all-targets -D warnings clean on both feature sets

What I would most like reviewed

1. The optional-capability shape, since it becomes the precedent for every future optional feature.
2. Whether the accessor belongs on DataEngine or somewhere better.
3. Wire parity: anything modelled that the service does differently.

[yesyayen]

I believe capacity metering is not yes implemented. (VectorWriteRequestBytes / VectorSearchRequestBytes)

[LeeroyHannigan]

Re the review note on capacity metering: VectorSearchRequestBytes was already implemented; VectorWriteRequestBytes was genuinely missing and is now in. Measured against the real service rather than modelled: max(dimensions * 4 + projected_non_vector_bytes, 1024), doubled when the SearchSchema HASH value moves the entry between partitions, reported under ConsumedCapacity.VectorIndexes at INDEXES granularity only. Model + 17 pinned tests land on this branch; the write-op wiring and six wire tests are on #244. Full detail in the summary on #244.

Completed.

[robinnsc]

Approving, but Fable did highlight a gap in the search handler that seems reasonable to fix, at lease as a fast follow. Left that comment in line

A few smaller things, none blocking:

1. validate_vector_index_attribute_definitions says it applies on both paths, but only validate_vector_index_updates calls it. On the CreateTable path the search schema rule is covered indirectly through the key correspondence check, which produces different error text, and the conflicting vector attribute rule does not run at all there. Either the doc or the call sites should change.
2. Nothing validates the shape of the SearchSchema element list at creation time. There is no element count cap and nothing rejects a second HASH element, while the search handler assumes at most one (it takes the first with .find(), and a condition on any second HASH element would quietly land in filters). Worth tightening, or documenting as deliberate, before a backend lands.
3. The module doc on vector_opt_out_tests in storage/src/lib.rs still describes the earlier boolean design, talking about a default capability of false and a defaulted search_vectors that must fail. The tests themselves exercise the accessor design, so just the prose is stale.

[robinnsc]

This guard means the required HASH check never runs when the caller supplies no SearchConditionExpression at all. An index that declares a HASH element, searched with no expression, passes through here and reaches the backend with hash_key: None.

That breaks two promises this PR makes elsewhere. The comment further down says the index having a HASH element guarantees the search supplied it, and the VectorSearch::hash_key doc in storage/src/lib.rs tells backend authors it is always populated when the index declares one, so they may treat Some as a mandatory predicate. A backend written against that doc, including the stacked SQLite one, would serve an unscoped search where a scoped one was promised.

I think the fix is just to drop the guard and call validate_conditions_against_search_schema unconditionally. With empty conditions the membership loop does nothing and the HASH presence loop correctly rejects, and the case with no schema and no expression still passes. Unreachable in tree today since no backend implements search, but this file is the contract the next PR builds on, so better fixed now.