Document $vectorSearch, $graphLookup, $project, and $limit

[GuanzhouSong]

Relates to documentdb/documentdb.github.io#130. First slice of the missing-stage gap; the remaining 10 stages are still undocumented.

Why these four

The MQL reference documents 25 aggregation stages. pg_documentdb registers 40 public ones (StageDefinitions[] in bson_aggregation_pipeline.c holds 41 entries, one of which is $_internalInhibitOptimization), so 14 have no page at all.

These four are the ones a reader hits first:

• $vectorSearch — promoted on the documentdb.io /ai landing page, and getting-started/python-setup.md ships a working $vectorSearch pipeline for a stage that has no reference page. A reader who follows the AI page into getting-started and then wants the operator's parameters currently has nowhere to land.
• $graphLookup — also promoted on /ai: "traverse related entities with $graphLookup in one query pipeline".
• $project and $limit — core stages.

How the content was derived

Parameters, defaults, and constraints come from the upstream implementation, not from MongoDB's docs:

• ParseAndValidateNativeVectorSearchSpec (bson_aggregation_search.c) — the accepted $vectorSearch fields, the numCandidates bounds (HNSW_MIN_EF_SEARCH..HNSW_MAX_EF_SEARCH = 1..1000), the 16000-element cap on queryVector, and the fact that index is parsed but ignored.
• ParseGraphLookupStage (bson_aggregation_nested_pipeline.c) — the eight accepted $graphLookup parameters, which five are required, and the unbounded maxDepth default.
• HandleLimit (bson_aggregation_pipeline.c) — must be numeric, 64-bit representable, non-negative, non-zero.
  Corrected in review: the minimum rule holds only for adjacent $limit stages. HandleLimit sets requiresSubQuery, so stages nest rather than merge, and a document-multiplying stage between two $limits breaks it.
• bson_projection_tree.c — the inclusion/exclusion mixing rule and the top-level _id exemption.

Verification

Every example and every documented error message was executed against documentdb-local and its real output recorded. A clean-room replay then re-ran the whole set on a fresh database, asserting each documented result and error string:

--- $project --- 7 assertions
--- $limit --- 6 assertions
--- $graphLookup --- 8 assertions
--- $vectorSearch --- 14 assertions
RESULT: 35 passed, 0 failed

Every output block was produced by the engine, with one exception added during the review round: the unprojected $vectorSearch result showing __cosmos_meta__ is derived from AddScoreFieldToDocumentEntry and the upstream expected-output files rather than from a run.

Two caveats on the numbers above. The suite predates the review round and has not been re-run against the current text. And it did not catch the $project Example 3 defect, because the assertion was written against the same fabricated sample document as the page — a replay is only as good as its fixture.

One finding worth flagging

$vectorSearch.filter requires an index on each field used in the filter, not just on the embedding path. Without it the query fails outright rather than falling back to a scan:

The index for filter path 'category' was not found, please check whether the index is created.

The /ai landing page's $vectorSearch sample includes filter: { category: "technical" } with no mention of this, so it is the kind of thing a reader will hit immediately. The $vectorSearch page creates the filter index before using it, and documents the error.

Also, numCandidates is documented as choosing a default from the index's efConstruction and the collection size rather than a flat 40 — GetDefaultSearchParamForIndex picks between the two depending on EnableVectorCalculateDefaultSearchParameter and whether the collection is under the small-collection row threshold.

Sharpened in review: that threshold is a cliff, not a curve. efConstruction feeds the default only below VECTOR_SEARCH_SMALL_COLLECTION_ROWS (10,000); at or above it the default is a flat HNSW_DEFAULT_EF_SEARCH of 40, so an index built with efConstruction: 512 silently searches at 40 on a 50,000-document collection. The page now states the threshold and recommends setting numCandidates by hand on production-sized collections.

Review round

A source-validated review against documentdb upstream raised 22 findings; all are addressed in 21a7838, 8ee5fba and 0daa22a, and every thread is resolved with the specific change noted inline.

The substantive corrections, in rough order of how badly they would have misled a reader:

• $vectorSearch returns an undocumented __cosmos_meta__ field on every document. Every example happened to hide it behind an inclusion $project, and an exclusion projection does not strip it — so an application round-tripping results would have persisted it silently.
• filter was described as a true pre-filter with a limit guarantee. The engine intersects an ANN scan with a separate filter query and applies limit to the join, so reaching limit is best-effort and depends on pgvector ≥ 0.8.0 iterative scan.
• $project Example 3's output was wrong, because the sample document invented a tags field and dropped promotionEvents. Now uses the canonical fixture.
• $graphLookup claimed the as array order is undefined while publishing three exact arrays as results. The page now documents the dedup-to-lowest-depth guarantee it was hiding.
• Undocumented hard failures:$graphLookupfrom cannot be sharded, $vectorSearchfilter is rejected on sharded collections, and restrictSearchWithMatch rejects $near / $nearSphere / $geoNear.
• Index constraints that were never stated:dimensions is capped at 2000 without compression, not 16000 — which is exactly the wall a reader with 3072-dimension embeddings hits; and efConstruction must be at least 2 * m.
• numCandidates on vector-ivf does nothing — accepted and validated, then dropped, because the IVF path reads only nProbes.

Also corrected outside the original diff: $meta.md claimed { $meta: "indexKey" } returns nothing without an index; it is rejected unconditionally (0daa22a).

One review candidate was refuted rather than applied — a claimed inclusion/exclusion message swap, which traces to an unreachable ternary arm in bson_projection_tree.c.

Conventions

Pages follow the existing $addfields.md structure — frontmatter, intro, ## Syntax, ## Parameters, ## Examples with sample data, and a closing ## Related. Filenames are lowercase to match the existing slugs. Every cross-link target was checked to exist. Corrected in review: the targets existed, but the link form was wrong — ../%24limit/ resolves one directory too high, into a path that holds only category folders. All 9 now use the same-directory ./%24name.md form taken from the $bucketauto.md precedent. The repo has no link-checking CI, so this would have shipped silently.

Still missing after this PR

$replaceRoot, $setWindowFields, $unionWith, $search, $currentOp, $inverseMatch, plus four that arguably do not warrant public pages: $searchMeta, $listLocalSessions, and $listSessions are registered with .mutateFunc = NULL and error unconditionally, and $listSearchIndexes is gated off by default (DEFAULT_ENABLE_EXTENDED_INDEXES is false).

Note also that $replaceWith is documented while $replaceRoot is not — in MongoDB $replaceWith is the alias, so the reference currently documents the alias and omits the canonical form. Upstream gives them separate handlers rather than treating them as aliases.