Skip to content

feat(driver-mongodb): index lookup joins off the canonical reference key#13633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zhuangjianguo merged 1 commit into from
Aug 31, 2026
+298 −129
Merged

feat(driver-mongodb): index lookup joins off the canonical reference key #13633

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions .changeset/mongodb-lookup-index-canonical-reference.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
---
"@objectstack/driver-mongodb": minor
---

feat(driver-mongodb): index `lookup` joins off the canonical `reference` key (#13222)

`syncCollectionSchema`'s field-level join-index arm gated on `field.reference_to`.
That is a REJECTED ALIAS — `FieldSchema` answers `unrecognized_keys` for it on any
field type — and this driver's own schema door refuses it outright. So the arm's
`lookup` conjunct could not be satisfied by any input at all, and no authored
lookup had ever been indexed on MongoDB. The arm now reads `reference`, the only
relationship spelling the spec declares.

Measured as a complete case split over the key's value domain rather than a
sample: every `reference_to` value except `undefined` is refused at the door, and
`undefined` is falsy, so the old conjunct was unreachable for every possible
input. The `user` disjunct was unaffected and is why the feature looked healthy —
it needs no relationship key, so `idx_owner_lookup`-shaped indexes were always
created.

The refusal door itself is unchanged: predicate, `VALIDATION_ERROR`/400 envelope,
placement and instruction are all as they were. Only the tail of its message
moved, because it told the reader that renaming the key would not by itself get
the field an index — true when written, false now.

A `lookup` that declares no `reference` is still not indexed. Measured on
`FieldSchema`: `{ type: 'lookup' }` and `{ type: 'lookup', reference: '' }` both
parse successfully — the spec's prose calls `reference` required for these types
but the schema does not enforce it — so this is a real authorable shape, and an
index for a join with no declared target would cost every write and buy no read.
`master_detail` and `tree` are unchanged: they reach this arm on neither spelling.

## ⚠️ OPERATORS — this changes boot behaviour on existing deployments

**What it costs.** The first `syncSchema` after this upgrade CREATES these
indexes on collections that already hold data. Each one is an awaited
`createIndex`: a full collection scan plus an external sort of the extracted
keys, followed by permanent index storage and a small write amplification on
every subsequent insert/update of the indexed field. ("Awaited" describes the
driver, not the server — see the build-feature note below for what the server
does during it.) Later boots are free —
`createIndex` is idempotent for an index that already exists, and this driver
already relies on that.

⚠️ **The builds are SERIALIZED, so the times ADD.** `syncCollectionSchema` awaits
`createIndex` once per index in a sequential loop, and `syncSchemasBatch` awaits
`syncSchema` once per object the same way. Startup is extended by the SUM of
every build, not by the slowest one. This is the figure to plan the maintenance
window around.

**How much.** Measured on this tree: **65 lookup fields carrying `reference`
across the 52 exported platform objects** — every one gains an index, and this is
the floor, not the total. Add the objects of any enabled plugin (`plugin-security`
14 lookup fields, `plugin-approvals` 13, `plugin-audit` 6, `plugin-sharing` 4) and
one index per lookup field on your own authored objects.

Per index, as an order-of-magnitude planning figure and **not** a benchmark — get
your own numbers before sizing a window, because they depend entirely on document
count, storage and cache:

| collection size | build time, one index | index storage added |
|---|---|---|
| empty / a few thousand docs | effectively instant (metadata-only) | negligible |
| ~1M docs | seconds | tens of MB |
| ~10M docs | ~a minute | hundreds of MB |
| ~100M docs | tens of minutes | a few GB |

Most `sys_*` collections are small and will finish instantly. Budget for the ones
that accumulate: `sys_metadata_history`, `sys_metadata_audit`, `sys_notification`,
`sys_email`, `sys_session`, and any audit-log object. Count them first
(`db.<collection>.estimatedDocumentCount()`) and multiply by the lookup fields
each carries; watch a live boot with `db.currentOp()`.

**Build feature — hybrid builds, unconditionally.** `@objectstack/driver-mongodb`
depends on `mongodb@^7.5.0`, whose own compatibility statement is "the driver
currently supports 4.2+ servers". MongoDB 4.2 is exactly the release that made
index builds hybrid, so **every server version this driver can connect to builds
these indexes with the hybrid builder**: an exclusive lock is taken only briefly
at the start and end of each build, and the collection accepts reads AND writes
throughout the rest of it. This is not a full write stall. Two caveats that
remain: the brief exclusive lock at each end is real, and on a replica set the
build runs on every member.

**To take the cost outside the boot window,** create the indexes ahead of the
upgrade, with the names and specs the driver uses — `{ name: 'idx_FIELD_lookup' }`
over `{ FIELD: 1 }`, no `unique` and no `sparse`. The driver's `createIndex` is
then a no-op and startup is unaffected. Matching the options matters: a
pre-existing index of the same name with different options raises
`IndexOptionsConflict`, which this driver deliberately swallows and skips, leaving
your index in place unchanged.
37 changes: 24 additions & 13 deletions content/docs/protocol/objectql/types.mdx
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -731,23 +731,34 @@ contacts:

<Callout type="warn">
A relationship field authored with `reference:` gets **no database-level
`FOREIGN KEY` constraint** and **no MongoDB join index**. Both gaps have one
root cause: the driver branches that would have built them were gated on
`reference_to` — a key the spec REFUSES (`FieldSchema` answers
`unrecognized_keys` for it, on any field type) and one that `reference` never
populates. `master_detail` / `tree` did not reach either branch at all.
`FOREIGN KEY` constraint** on any driver. It does get a **MongoDB join index**.
Both branches were once gated on `reference_to` — a key the spec REFUSES
(`FieldSchema` answers `unrecognized_keys` for it, on any field type) and one
that `reference` never populates — so neither fired for authored metadata; the
two were then settled in opposite directions, on their own merits.

- **SQL:** the `FOREIGN KEY` DDL is retired (#11567). A field that still carries
`reference_to` when it reaches DDL is refused at the driver's door, in the
schema's own words (`400 VALIDATION_ERROR`), instead of silently changing the
physical schema.
- **MongoDB:** the field-level join index `idx_FIELD_lookup` is gated on that
same refused key, so a canonically-spelled `reference` lookup is **not
indexed**. A `user` field still is — that arm needs no relationship key, which
is why the feature looked healthy. `reference_to` is refused at this driver's
door too, with the same verdict (#13222); whether a canonical `reference`
lookup should start building the index is tracked separately, because it
changes boot behaviour for deployments that already hold large collections.
physical schema. Declare an index in `indexes[]` if you want one on the
foreign-key column.
- **MongoDB:** a `lookup` field declaring `reference:` gets the field-level join
index `idx_FIELD_lookup` (#13222). A `user` field gets one too — that arm
needs no relationship key. `reference_to` is refused at this driver's door as
well, with the same verdict as the spec's.

⚠️ **Upgrading an existing MongoDB deployment:** this index is newer than the
driver, so the first `syncSchema` after the upgrade BUILDS it across
collections that already hold data — a real one-off IO and time cost on large
collections, and index storage that persists. The driver's `CHANGELOG.md`
entry for the release that added it carries the operational detail. Later
boots are no-ops: `createIndex` is idempotent for an index that already
exists.

A `lookup` that omits `reference:` gets no join index — there is no declared
target to join to. `master_detail` and `tree` reach neither branch: they get no
`FOREIGN KEY` and no join index. Declare an `indexes[]` entry if a
`master_detail` child is queried by parent often enough to need one.

Referential integrity is enforced by the **engine** instead: `deleteBehavior` is
applied on delete, which is what produces the `409 DELETE_RESTRICTED` above.
Expand Down
14 changes: 10 additions & 4 deletions packages/drivers/driver-mongodb/README.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -163,9 +163,14 @@ object's `indexes[]` — the one surface an index is declared on (a field-level
`indexed` flag is not a `FieldSchema` key and never built an index,
#2377 / #6810).

⚠️ Lookup fields are **not** indexed today. The lookup arm gates on
`reference_to`, a spelling `FieldSchema` refuses, so a canonically-spelled
`reference` lookup gets no join index — see #13222, which owns that fix.
A `lookup` field that declares `reference` also indexes itself, as
`idx_FIELD_lookup`, for join performance; so does a `user` field. A `lookup`
with no `reference` does not — there is no declared target to join to.

⚠️ **Upgrading:** this join index is newer than the driver. The first
`syncSchema` after the upgrade builds it across collections that already hold
data, which costs real IO and time on large ones. See `CHANGELOG.md` for the
operational detail. Later boots are no-ops.

```typescript
await driver.syncSchema('account', {
Expand All@@ -177,7 +182,8 @@ await driver.syncSchema('account', {
},
indexes: [{ fields: ['email'] }],
});
// Creates: idx_id_unique, idx_name_unique, idx_email
// Creates: idx_id_unique, idx_created_at, idx_updated_at,
// idx_name_unique, idx_company_id_lookup, idx_email
```

### Aggregation
Expand Down
89 changes: 61 additions & 28 deletions packages/drivers/driver-mongodb/src/mongodb-13222-reference-to-refusal.test.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -19,13 +19,14 @@
// module between two suites that pin OPPOSITE halves of one arm would couple
// them for no gain.
//
// ⛔ NOT pinned here: whether a canonically-spelled `reference` lookup should
// GET `idx_FIELD_lookup`. That is part (2) of #13222 — a separate, still-open
// ruling (it is a boot-time behaviour change for existing deployments: index
// builds on large collections). The last case below is this PR's own NO-CHANGE
// control for it, and is expected to flip in the PR that takes part (2),
// alongside `mongodb-schema-declared-indexes.test.ts`'s #12252 pin, which owns
// the fact.
// Part (2) of #13222 has since been ruled and taken: the join-index arm now
// gates on the canonical `reference`, so a lookup an author can really publish
// GETS `idx_FIELD_lookup`. The last case below was this file's NO-CHANGE
// control for that question and has flipped accordingly, in the same stroke as
// `mongodb-schema-declared-indexes.test.ts`'s #12252 pin and the real-server
// pin in `mongodb-driver.test.ts`. It still belongs here: it is what proves the
// door and the arm read DIFFERENT keys and disagree about them on purpose —
// `reference_to` refused, `reference` indexed, from one call.

import { describe, it, expect } from 'vitest';
import type { Db } from 'mongodb';
Expand DownExpand Up@@ -187,40 +188,72 @@ describe('#13222 part (1) — driver-mongodb refuses `reference_to` at the schem
expect(names(created)).toEqual(['idx_id_unique', 'idx_created_at', 'idx_updated_at']);
});

it('leaves the join-index arm exactly as it was — part (2) is NOT taken here', async () => {
// ⚠️ THE NO-CHANGE CONTROL for this PR, and load-bearing in both directions.
it('refuses `reference_to` and INDEXES `reference` — two keys, two answers, one call', async () => {
// ⚠️ THE PAIRED CONTROL for the door, and load-bearing in both directions.
//
// Positive half: a `user` field still gets `idx_owner_id_lookup`, which
// proves the arm still executes and that the harness is wired to something —
// without it the negative half below would pass just as happily against a
// function that created no indexes at all.
// The door refuses one relationship spelling; the arm indexes the other. A
// suite that only ever proves the refusal cannot tell "the door works" from
// "syncCollectionSchema throws on everything", so the two answers are taken
// from ONE call here on purpose.
//
// Negative half: a canonically-spelled `reference` lookup still gets NO join
// index. That is the divergence #12252 pinned and part (2) of #13222 owns.
// ⛔ This case records what the driver DOES, not what it should do: the door
// added in this PR makes the arm's `field.reference_to` conjunct unreachable
// but deliberately does not delete it, because deleting it would start
// building indexes on existing deployments' large collections — an unruled
// behaviour change. When part (2) lands, this case is expected to flip to
// `toContain`, in the same stroke as the #12252 pin in
// `mongodb-schema-declared-indexes.test.ts`.
// `user` half: still `idx_owner_id_lookup`, and still the unconditional
// disjunct — it proves the arm executed and names the shape the assertion
// below is spelled in, so neither half can pass vacuously.
//
// Bound through a variable rather than written inline: the driver's own
// `FieldDef` declares no `reference` key, so a fresh object literal carrying
// it trips TypeScript's excess-property check.
const canonicalLookup = { type: 'lookup', reference: 'company' };

// `reference` half: FLIPPED by part (2) of #13222. This case used to assert
// the canonical lookup got NO index — the divergence #12252 pinned, which
// held because the arm gated on `reference_to`, a key this very door
// refuses, making the conjunct unreachable for every input. The arm now
// gates on `reference`, so the lookup is indexed.
const { db, created } = fakeDb();
await syncCollectionSchema(db, 'lead', {
name: 'lead',
fields: { company_id: canonicalLookup, owner_id: { type: 'user' } },
fields: {
company_id: { type: 'lookup', reference: 'company' },
owner_id: { type: 'user' },
},
});

// Exact set, in creation order — closes the vacuity routes a `toContain`
// pair leaves open: an index appearing under another name, the two lookup
// indexes swapping fields, or a stray fourth index nobody declared.
expect(names(created)).toEqual([
'idx_id_unique',
'idx_created_at',
'idx_updated_at',
'idx_company_id_lookup',
'idx_owner_id_lookup',
]);
});

it('does not index a `lookup` that declares no target — `reference` is read for truth, not presence', async () => {
// Measured on `FieldSchema` built from this tree: `{ type: 'lookup' }` with
// no `reference`, and `{ type: 'lookup', reference: '' }`, BOTH parse
// successfully — the spec's prose calls `reference` required for these
// types, but the schema does not enforce it. So this is a shape an author
// can really publish, not a hypothetical, and the arm has to answer for it.
//
// It answers by declining: `idx_FIELD_lookup` exists to serve a join, and a
// lookup with no declared target has no join to serve — the index would
// cost every write and buy no read. This is why the arm gates on
// truthiness and not on `!== undefined` like the door above does; the two
// predicates differ deliberately, because they are answering different
// questions about different keys.
for (const target of [undefined, '']) {
const { db, created } = fakeDb();
await syncCollectionSchema(db, 'lead', {
name: 'lead',
fields: { company_id: { type: 'lookup', reference: target }, owner_id: { type: 'user' } },
});

// The `user` control fires, so the arm ran and the zero below is a real
// zero rather than a harness that called nothing.
expect(names(created)).toEqual([
'idx_id_unique',
'idx_created_at',
'idx_updated_at',
'idx_owner_id_lookup',
]);
}
});
});
42 changes: 21 additions & 21 deletions packages/drivers/driver-mongodb/src/mongodb-driver.test.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -362,31 +362,31 @@ describe.skipIf(!sharedMongod)('MongoDBDriver', () => {
expect(indexNames).toContain('idx_name_unique');
expect(indexNames).toContain('idx_email');
/**
* ⚠️ [#12252] DIVERGENCE PINNED, DISPOSITION OPEN (#13222) — a
* canonically-spelled lookup gets NO join index here.
* ⚠️ DIVERGENCE RETIRED (#13222) — a canonically-spelled lookup gets its
* join index here, verified against a REAL server rather than a recorder.
*
* This fixture used to spell the field `reference_to: 'company'` and
* assert `idx_company_id_lookup` was CREATED. `reference_to` is a key
* `FieldSchema` REFUSES (`unrecognized_keys`), so the object it described
* was one no author could publish — and the assertion passed only because
* this fixture was the sole thing in the tree reaching the lookup arm of
* `mongodb-schema.ts`, which gates on `field.reference_to` and reads no
* other relationship key.
* This fixture once spelled the field `reference_to: 'company'`. That is
* a key `FieldSchema` REFUSES (`unrecognized_keys`), so the object it
* described was one no author could publish — and the assertion passed
* only because this fixture was the sole thing in the tree reaching the
* lookup arm of `mongodb-schema.ts`, which gated on `field.reference_to`
* and read no other relationship key. Correcting the spelling (#13224)
* therefore did not leave the outcome alone: the index disappeared, and
* the line was inverted to record that, with the disposition left open.
*
* So correcting the spelling does not leave the outcome alone. Measured
* differentially against the real `syncCollectionSchema`,
* `idx_company_id_lookup` is the ONE index that disappears; a
* `type: 'user'` field still gets its index, so the gate is live rather
* than dead code. The consequence in production is that EVERY authored
* lookup on MongoDB is unindexed — 57/57 relationship fields across the
* 44 exported platform objects (#13222).
* #13222 settled the disposition in two parts: `reference_to` is now
* refused at the driver's door, and the arm gates on the canonical
* `reference`. So the line is inverted back — by a ruling, not by a
* re-baseline.
*
* ⛔ This records what the driver DOES, not what it SHOULD do. Whether
* the lookup arm learns to read `reference` is #13222's to settle, ⛔ not
* this pin's — when it lands, this line flips back to `toContain`
* deliberately rather than the divergence reopening in silence.
* ⚠️ This copy is the one that runs against real MongoDB, which is what
* it is FOR: it proves the server actually materialized the index under
* that name, not merely that the driver asked for it. The recorder-driven
* twin in `mongodb-schema-declared-indexes.test.ts` is the copy that runs
* on every ordinary CI lane — this suite is `describe.skipIf(!sharedMongod)`.
* TWIN PIN — edit either, edit both.
*/
expect(indexNames).not.toContain('idx_company_id_lookup');
expect(indexNames).toContain('idx_company_id_lookup');
});

it('should be idempotent (safe to call multiple times)', async () => {
Expand Down
Loading
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
feat(driver-mongodb): index lookup joins off the canonical `reference` key by claude[bot] · Pull Request #13633 · objectstack-ai/objectstack · GitHub
Skip to content
feat(driver-mongodb): index lookup joins off the canonical reference key#13633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

zhuangjianguo
merged 1 commit into

from

Aug 31, 2026

+298

−129

 Merged


feat(driver-mongodb): index lookup joins off the canonical reference key
#13633
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


90 changes: 90 additions & 0 deletions
.changeset/mongodb-lookup-index-canonical-reference.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
---
"@objectstack/driver-mongodb": minor
---

feat(driver-mongodb): index `lookup` joins off the canonical `reference` key (#13222)

`syncCollectionSchema`'s field-level join-index arm gated on `field.reference_to`.
That is a REJECTED ALIAS — `FieldSchema` answers `unrecognized_keys` for it on any
field type — and this driver's own schema door refuses it outright. So the arm's
`lookup` conjunct could not be satisfied by any input at all, and no authored
lookup had ever been indexed on MongoDB. The arm now reads `reference`, the only
relationship spelling the spec declares.

Measured as a complete case split over the key's value domain rather than a
sample: every `reference_to` value except `undefined` is refused at the door, and
`undefined` is falsy, so the old conjunct was unreachable for every possible
input. The `user` disjunct was unaffected and is why the feature looked healthy —
it needs no relationship key, so `idx_owner_lookup`-shaped indexes were always
created.

The refusal door itself is unchanged: predicate, `VALIDATION_ERROR`/400 envelope,
placement and instruction are all as they were. Only the tail of its message
moved, because it told the reader that renaming the key would not by itself get
the field an index — true when written, false now.

A `lookup` that declares no `reference` is still not indexed. Measured on
`FieldSchema`: `{ type: 'lookup' }` and `{ type: 'lookup', reference: '' }` both
parse successfully — the spec's prose calls `reference` required for these types
but the schema does not enforce it — so this is a real authorable shape, and an
index for a join with no declared target would cost every write and buy no read.
`master_detail` and `tree` are unchanged: they reach this arm on neither spelling.

## ⚠️ OPERATORS — this changes boot behaviour on existing deployments

**What it costs.** The first `syncSchema` after this upgrade CREATES these
indexes on collections that already hold data. Each one is an awaited
`createIndex`: a full collection scan plus an external sort of the extracted
keys, followed by permanent index storage and a small write amplification on
every subsequent insert/update of the indexed field. ("Awaited" describes the
driver, not the server — see the build-feature note below for what the server
does during it.) Later boots are free —
`createIndex` is idempotent for an index that already exists, and this driver
already relies on that.

⚠️ **The builds are SERIALIZED, so the times ADD.** `syncCollectionSchema` awaits
`createIndex` once per index in a sequential loop, and `syncSchemasBatch` awaits
`syncSchema` once per object the same way. Startup is extended by the SUM of
every build, not by the slowest one. This is the figure to plan the maintenance
window around.

**How much.** Measured on this tree: **65 lookup fields carrying `reference`
across the 52 exported platform objects** — every one gains an index, and this is
the floor, not the total. Add the objects of any enabled plugin (`plugin-security`
14 lookup fields, `plugin-approvals` 13, `plugin-audit` 6, `plugin-sharing` 4) and
one index per lookup field on your own authored objects.

Per index, as an order-of-magnitude planning figure and **not** a benchmark — get
your own numbers before sizing a window, because they depend entirely on document
count, storage and cache:

| collection size | build time, one index | index storage added |
|---|---|---|
| empty / a few thousand docs | effectively instant (metadata-only) | negligible |
| ~1M docs | seconds | tens of MB |
| ~10M docs | ~a minute | hundreds of MB |
| ~100M docs | tens of minutes | a few GB |

Most `sys_*` collections are small and will finish instantly. Budget for the ones
that accumulate: `sys_metadata_history`, `sys_metadata_audit`, `sys_notification`,
`sys_email`, `sys_session`, and any audit-log object. Count them first
(`db.<collection>.estimatedDocumentCount()`) and multiply by the lookup fields
each carries; watch a live boot with `db.currentOp()`.

**Build feature — hybrid builds, unconditionally.** `@objectstack/driver-mongodb`
depends on `mongodb@^7.5.0`, whose own compatibility statement is "the driver
currently supports 4.2+ servers". MongoDB 4.2 is exactly the release that made
index builds hybrid, so **every server version this driver can connect to builds
these indexes with the hybrid builder**: an exclusive lock is taken only briefly
at the start and end of each build, and the collection accepts reads AND writes
throughout the rest of it. This is not a full write stall. Two caveats that
remain: the brief exclusive lock at each end is real, and on a replica set the
build runs on every member.

**To take the cost outside the boot window,** create the indexes ahead of the
upgrade, with the names and specs the driver uses — `{ name: 'idx_FIELD_lookup' }`
over `{ FIELD: 1 }`, no `unique` and no `sparse`. The driver's `createIndex` is
then a no-op and startup is unaffected. Matching the options matters: a
pre-existing index of the same name with different options raises
`IndexOptionsConflict`, which this driver deliberately swallows and skips, leaving
your index in place unchanged.

37 changes: 24 additions & 13 deletions
content/docs/protocol/objectql/types.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -731,23 +731,34 @@ contacts:

<Callout type="warn">
A relationship field authored with `reference:` gets **no database-level
`FOREIGN KEY` constraint** and **no MongoDB join index**. Both gaps have one
root cause: the driver branches that would have built them were gated on
`reference_to` — a key the spec REFUSES (`FieldSchema` answers
`unrecognized_keys` for it, on any field type) and one that `reference` never
populates. `master_detail` / `tree` did not reach either branch at all.
`FOREIGN KEY` constraint** on any driver. It does get a **MongoDB join index**.
Both branches were once gated on `reference_to` — a key the spec REFUSES
(`FieldSchema` answers `unrecognized_keys` for it, on any field type) and one
that `reference` never populates — so neither fired for authored metadata; the
two were then settled in opposite directions, on their own merits.

- **SQL:** the `FOREIGN KEY` DDL is retired (#11567). A field that still carries
 `reference_to` when it reaches DDL is refused at the driver's door, in the
 schema's own words (`400 VALIDATION_ERROR`), instead of silently changing the
 physical schema.
- **MongoDB:** the field-level join index `idx_FIELD_lookup` is gated on that
 same refused key, so a canonically-spelled `reference` lookup is **not
 indexed**. A `user` field still is — that arm needs no relationship key, which
 is why the feature looked healthy. `reference_to` is refused at this driver's
 door too, with the same verdict (#13222); whether a canonical `reference`
 lookup should start building the index is tracked separately, because it
 changes boot behaviour for deployments that already hold large collections.
 physical schema. Declare an index in `indexes[]` if you want one on the
 foreign-key column.
- **MongoDB:** a `lookup` field declaring `reference:` gets the field-level join
 index `idx_FIELD_lookup` (#13222). A `user` field gets one too — that arm
 needs no relationship key. `reference_to` is refused at this driver's door as
 well, with the same verdict as the spec's.

 ⚠️ **Upgrading an existing MongoDB deployment:** this index is newer than the
 driver, so the first `syncSchema` after the upgrade BUILDS it across
 collections that already hold data — a real one-off IO and time cost on large
 collections, and index storage that persists. The driver's `CHANGELOG.md`
 entry for the release that added it carries the operational detail. Later
 boots are no-ops: `createIndex` is idempotent for an index that already
 exists.

A `lookup` that omits `reference:` gets no join index — there is no declared
target to join to. `master_detail` and `tree` reach neither branch: they get no
`FOREIGN KEY` and no join index. Declare an `indexes[]` entry if a
`master_detail` child is queried by parent often enough to need one.

Referential integrity is enforced by the **engine** instead: `deleteBehavior` is
applied on delete, which is what produces the `409 DELETE_RESTRICTED` above.
Expand Down

14 changes: 10 additions & 4 deletions
packages/drivers/driver-mongodb/README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -163,9 +163,14 @@ object's `indexes[]` — the one surface an index is declared on (a field-level
`indexed` flag is not a `FieldSchema` key and never built an index,
#2377 / #6810).

⚠️ Lookup fields are **not** indexed today. The lookup arm gates on
`reference_to`, a spelling `FieldSchema` refuses, so a canonically-spelled
`reference` lookup gets no join index — see #13222, which owns that fix.
A `lookup` field that declares `reference` also indexes itself, as
`idx_FIELD_lookup`, for join performance; so does a `user` field. A `lookup`
with no `reference` does not — there is no declared target to join to.

⚠️ **Upgrading:** this join index is newer than the driver. The first
`syncSchema` after the upgrade builds it across collections that already hold
data, which costs real IO and time on large ones. See `CHANGELOG.md` for the
operational detail. Later boots are no-ops.

```typescript
await driver.syncSchema('account', {
Expand All@@ -177,7 +182,8 @@ await driver.syncSchema('account', {
 },
 indexes: [{ fields: ['email'] }],
});
// Creates: idx_id_unique, idx_name_unique, idx_email
// Creates: idx_id_unique, idx_created_at, idx_updated_at,
// idx_name_unique, idx_company_id_lookup, idx_email
```

### Aggregation
Expand Down

89 changes: 61 additions & 28 deletions
packages/drivers/driver-mongodb/src/mongodb-13222-reference-to-refusal.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,13 +19,14 @@
// module between two suites that pin OPPOSITE halves of one arm would couple
// them for no gain.
//
// ⛔ NOT pinned here: whether a canonically-spelled `reference` lookup should
// GET `idx_FIELD_lookup`. That is part (2) of #13222 — a separate, still-open
// ruling (it is a boot-time behaviour change for existing deployments: index
// builds on large collections). The last case below is this PR's own NO-CHANGE
// control for it, and is expected to flip in the PR that takes part (2),
// alongside `mongodb-schema-declared-indexes.test.ts`'s #12252 pin, which owns
// the fact.
// Part (2) of #13222 has since been ruled and taken: the join-index arm now
// gates on the canonical `reference`, so a lookup an author can really publish
// GETS `idx_FIELD_lookup`. The last case below was this file's NO-CHANGE
// control for that question and has flipped accordingly, in the same stroke as
// `mongodb-schema-declared-indexes.test.ts`'s #12252 pin and the real-server
// pin in `mongodb-driver.test.ts`. It still belongs here: it is what proves the
// door and the arm read DIFFERENT keys and disagree about them on purpose —
// `reference_to` refused, `reference` indexed, from one call.

import { describe, it, expect } from 'vitest';
import type { Db } from 'mongodb';
Expand DownExpand Up@@ -187,40 +188,72 @@ describe('#13222 part (1) — driver-mongodb refuses `reference_to` at the schem
 expect(names(created)).toEqual(['idx_id_unique', 'idx_created_at', 'idx_updated_at']);
 });

 it('leaves the join-index arm exactly as it was — part (2) is NOT taken here', async () => {
 // ⚠️ THE NO-CHANGE CONTROL for this PR, and load-bearing in both directions.
 it('refuses `reference_to` and INDEXES `reference` — two keys, two answers, one call', async () => {
 // ⚠️ THE PAIRED CONTROL for the door, and load-bearing in both directions.
 //
 // Positive half: a `user` field still gets `idx_owner_id_lookup`, which
 // proves the arm still executes and that the harness is wired to something —
 // without it the negative half below would pass just as happily against a
 // function that created no indexes at all.
 // The door refuses one relationship spelling; the arm indexes the other. A
 // suite that only ever proves the refusal cannot tell "the door works" from
 // "syncCollectionSchema throws on everything", so the two answers are taken
 // from ONE call here on purpose.
 //
 // Negative half: a canonically-spelled `reference` lookup still gets NO join
 // index. That is the divergence #12252 pinned and part (2) of #13222 owns.
 // ⛔ This case records what the driver DOES, not what it should do: the door
 // added in this PR makes the arm's `field.reference_to` conjunct unreachable
 // but deliberately does not delete it, because deleting it would start
 // building indexes on existing deployments' large collections — an unruled
 // behaviour change. When part (2) lands, this case is expected to flip to
 // `toContain`, in the same stroke as the #12252 pin in
 // `mongodb-schema-declared-indexes.test.ts`.
 // `user` half: still `idx_owner_id_lookup`, and still the unconditional
 // disjunct — it proves the arm executed and names the shape the assertion
 // below is spelled in, so neither half can pass vacuously.
 //
 // Bound through a variable rather than written inline: the driver's own
 // `FieldDef` declares no `reference` key, so a fresh object literal carrying
 // it trips TypeScript's excess-property check.
const canonicalLookup = { type: 'lookup', reference: 'company' };

 // `reference` half: FLIPPED by part (2) of #13222. This case used to assert
 // the canonical lookup got NO index — the divergence #12252 pinned, which
 // held because the arm gated on `reference_to`, a key this very door
// refuses, making the conjunct unreachable for every input. The arm now
 // gates on `reference`, so the lookup is indexed.
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: canonicalLookup, owner_id: { type: 'user' } },
 fields: {
 company_id: { type: 'lookup', reference: 'company' },
 owner_id: { type: 'user' },
 },
 });

 // Exact set, in creation order — closes the vacuity routes a `toContain`
 // pair leaves open: an index appearing under another name, the two lookup
 // indexes swapping fields, or a stray fourth index nobody declared.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_company_id_lookup',
 'idx_owner_id_lookup',
 ]);
 });

 it('does not index a `lookup` that declares no target — `reference` is read for truth, not presence', async () => {
 // Measured on `FieldSchema` built from this tree: `{ type: 'lookup' }` with
 // no `reference`, and `{ type: 'lookup', reference: '' }`, BOTH parse
 // successfully — the spec's prose calls `reference` required for these
 // types, but the schema does not enforce it. So this is a shape an author
 // can really publish, not a hypothetical, and the arm has to answer for it.
 //
 // It answers by declining: `idx_FIELD_lookup` exists to serve a join, and a
 // lookup with no declared target has no join to serve — the index would
 // cost every write and buy no read. This is why the arm gates on
 // truthiness and not on `!== undefined` like the door above does; the two
 // predicates differ deliberately, because they are answering different
 // questions about different keys.
 for (const target of [undefined, '']) {
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: { type: 'lookup', reference: target }, owner_id: { type: 'user' } },
 });

 // The `user` control fires, so the arm ran and the zero below is a real
 // zero rather than a harness that called nothing.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_owner_id_lookup',
 ]);
 }
 });
});

42 changes: 21 additions & 21 deletions
packages/drivers/driver-mongodb/src/mongodb-driver.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -362,31 +362,31 @@ describe.skipIf(!sharedMongod)('MongoDBDriver', () => {
 expect(indexNames).toContain('idx_name_unique');
 expect(indexNames).toContain('idx_email');
 /**
 * ⚠️ [#12252] DIVERGENCE PINNED, DISPOSITION OPEN (#13222) — a
 * canonically-spelled lookup gets NO join index here.
 * ⚠️ DIVERGENCE RETIRED (#13222) — a canonically-spelled lookup gets its
 * join index here, verified against a REAL server rather than a recorder.
 *
 * This fixture used to spell the field `reference_to: 'company'` and
 * assert `idx_company_id_lookup` was CREATED. `reference_to` is a key
 * `FieldSchema` REFUSES (`unrecognized_keys`), so the object it described
 * was one no author could publish — and the assertion passed only because
 * this fixture was the sole thing in the tree reaching the lookup arm of
 * `mongodb-schema.ts`, which gates on `field.reference_to` and reads no
 * other relationship key.
 * This fixture once spelled the field `reference_to: 'company'`. That is
 * a key `FieldSchema` REFUSES (`unrecognized_keys`), so the object it
 * described was one no author could publish — and the assertion passed
 * only because this fixture was the sole thing in the tree reaching the
 * lookup arm of `mongodb-schema.ts`, which gated on `field.reference_to`
 * and read no other relationship key. Correcting the spelling (#13224)
 * therefore did not leave the outcome alone: the index disappeared, and
 * the line was inverted to record that, with the disposition left open.
 *
 * So correcting the spelling does not leave the outcome alone. Measured
 * differentially against the real `syncCollectionSchema`,
 * `idx_company_id_lookup` is the ONE index that disappears; a
 * `type: 'user'` field still gets its index, so the gate is live rather
 * than dead code. The consequence in production is that EVERY authored
 * lookup on MongoDB is unindexed — 57/57 relationship fields across the
 * 44 exported platform objects (#13222).
 * #13222 settled the disposition in two parts: `reference_to` is now
 * refused at the driver's door, and the arm gates on the canonical
 * `reference`. So the line is inverted back — by a ruling, not by a
 * re-baseline.
 *
 * ⛔ This records what the driver DOES, not what it SHOULD do. Whether
 * the lookup arm learns to read `reference` is #13222's to settle, ⛔ not
 * this pin's — when it lands, this line flips back to `toContain`
 * deliberately rather than the divergence reopening in silence.
 * ⚠️ This copy is the one that runs against real MongoDB, which is what
 * it is FOR: it proves the server actually materialized the index under
 * that name, not merely that the driver asked for it. The recorder-driven
 * twin in `mongodb-schema-declared-indexes.test.ts` is the copy that runs
 * on every ordinary CI lane — this suite is `describe.skipIf(!sharedMongod)`.
 * TWIN PIN — edit either, edit both.
 */
 expect(indexNames).not.toContain('idx_company_id_lookup');
 expect(indexNames).toContain('idx_company_id_lookup');
 });

 it('should be idempotent (safe to call multiple times)', async () => {
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
feat(driver-mongodb): index lookup joins off the canonical `reference` key by claude[bot] · Pull Request #13633 · objectstack-ai/objectstack · GitHub
Skip to content
feat(driver-mongodb): index lookup joins off the canonical reference key#13633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

zhuangjianguo
merged 1 commit into

from

Aug 31, 2026

+298

−129

 Merged


feat(driver-mongodb): index lookup joins off the canonical reference key
#13633
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


90 changes: 90 additions & 0 deletions
.changeset/mongodb-lookup-index-canonical-reference.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
---
"@objectstack/driver-mongodb": minor
---

feat(driver-mongodb): index `lookup` joins off the canonical `reference` key (#13222)

`syncCollectionSchema`'s field-level join-index arm gated on `field.reference_to`.
That is a REJECTED ALIAS — `FieldSchema` answers `unrecognized_keys` for it on any
field type — and this driver's own schema door refuses it outright. So the arm's
`lookup` conjunct could not be satisfied by any input at all, and no authored
lookup had ever been indexed on MongoDB. The arm now reads `reference`, the only
relationship spelling the spec declares.

Measured as a complete case split over the key's value domain rather than a
sample: every `reference_to` value except `undefined` is refused at the door, and
`undefined` is falsy, so the old conjunct was unreachable for every possible
input. The `user` disjunct was unaffected and is why the feature looked healthy —
it needs no relationship key, so `idx_owner_lookup`-shaped indexes were always
created.

The refusal door itself is unchanged: predicate, `VALIDATION_ERROR`/400 envelope,
placement and instruction are all as they were. Only the tail of its message
moved, because it told the reader that renaming the key would not by itself get
the field an index — true when written, false now.

A `lookup` that declares no `reference` is still not indexed. Measured on
`FieldSchema`: `{ type: 'lookup' }` and `{ type: 'lookup', reference: '' }` both
parse successfully — the spec's prose calls `reference` required for these types
but the schema does not enforce it — so this is a real authorable shape, and an
index for a join with no declared target would cost every write and buy no read.
`master_detail` and `tree` are unchanged: they reach this arm on neither spelling.

## ⚠️ OPERATORS — this changes boot behaviour on existing deployments

**What it costs.** The first `syncSchema` after this upgrade CREATES these
indexes on collections that already hold data. Each one is an awaited
`createIndex`: a full collection scan plus an external sort of the extracted
keys, followed by permanent index storage and a small write amplification on
every subsequent insert/update of the indexed field. ("Awaited" describes the
driver, not the server — see the build-feature note below for what the server
does during it.) Later boots are free —
`createIndex` is idempotent for an index that already exists, and this driver
already relies on that.

⚠️ **The builds are SERIALIZED, so the times ADD.** `syncCollectionSchema` awaits
`createIndex` once per index in a sequential loop, and `syncSchemasBatch` awaits
`syncSchema` once per object the same way. Startup is extended by the SUM of
every build, not by the slowest one. This is the figure to plan the maintenance
window around.

**How much.** Measured on this tree: **65 lookup fields carrying `reference`
across the 52 exported platform objects** — every one gains an index, and this is
the floor, not the total. Add the objects of any enabled plugin (`plugin-security`
14 lookup fields, `plugin-approvals` 13, `plugin-audit` 6, `plugin-sharing` 4) and
one index per lookup field on your own authored objects.

Per index, as an order-of-magnitude planning figure and **not** a benchmark — get
your own numbers before sizing a window, because they depend entirely on document
count, storage and cache:

| collection size | build time, one index | index storage added |
|---|---|---|
| empty / a few thousand docs | effectively instant (metadata-only) | negligible |
| ~1M docs | seconds | tens of MB |
| ~10M docs | ~a minute | hundreds of MB |
| ~100M docs | tens of minutes | a few GB |

Most `sys_*` collections are small and will finish instantly. Budget for the ones
that accumulate: `sys_metadata_history`, `sys_metadata_audit`, `sys_notification`,
`sys_email`, `sys_session`, and any audit-log object. Count them first
(`db.<collection>.estimatedDocumentCount()`) and multiply by the lookup fields
each carries; watch a live boot with `db.currentOp()`.

**Build feature — hybrid builds, unconditionally.** `@objectstack/driver-mongodb`
depends on `mongodb@^7.5.0`, whose own compatibility statement is "the driver
currently supports 4.2+ servers". MongoDB 4.2 is exactly the release that made
index builds hybrid, so **every server version this driver can connect to builds
these indexes with the hybrid builder**: an exclusive lock is taken only briefly
at the start and end of each build, and the collection accepts reads AND writes
throughout the rest of it. This is not a full write stall. Two caveats that
remain: the brief exclusive lock at each end is real, and on a replica set the
build runs on every member.

**To take the cost outside the boot window,** create the indexes ahead of the
upgrade, with the names and specs the driver uses — `{ name: 'idx_FIELD_lookup' }`
over `{ FIELD: 1 }`, no `unique` and no `sparse`. The driver's `createIndex` is
then a no-op and startup is unaffected. Matching the options matters: a
pre-existing index of the same name with different options raises
`IndexOptionsConflict`, which this driver deliberately swallows and skips, leaving
your index in place unchanged.

37 changes: 24 additions & 13 deletions
content/docs/protocol/objectql/types.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -731,23 +731,34 @@ contacts:

<Callout type="warn">
A relationship field authored with `reference:` gets **no database-level
`FOREIGN KEY` constraint** and **no MongoDB join index**. Both gaps have one
root cause: the driver branches that would have built them were gated on
`reference_to` — a key the spec REFUSES (`FieldSchema` answers
`unrecognized_keys` for it, on any field type) and one that `reference` never
populates. `master_detail` / `tree` did not reach either branch at all.
`FOREIGN KEY` constraint** on any driver. It does get a **MongoDB join index**.
Both branches were once gated on `reference_to` — a key the spec REFUSES
(`FieldSchema` answers `unrecognized_keys` for it, on any field type) and one
that `reference` never populates — so neither fired for authored metadata; the
two were then settled in opposite directions, on their own merits.

- **SQL:** the `FOREIGN KEY` DDL is retired (#11567). A field that still carries
 `reference_to` when it reaches DDL is refused at the driver's door, in the
 schema's own words (`400 VALIDATION_ERROR`), instead of silently changing the
 physical schema.
- **MongoDB:** the field-level join index `idx_FIELD_lookup` is gated on that
 same refused key, so a canonically-spelled `reference` lookup is **not
 indexed**. A `user` field still is — that arm needs no relationship key, which
 is why the feature looked healthy. `reference_to` is refused at this driver's
 door too, with the same verdict (#13222); whether a canonical `reference`
 lookup should start building the index is tracked separately, because it
 changes boot behaviour for deployments that already hold large collections.
 physical schema. Declare an index in `indexes[]` if you want one on the
 foreign-key column.
- **MongoDB:** a `lookup` field declaring `reference:` gets the field-level join
 index `idx_FIELD_lookup` (#13222). A `user` field gets one too — that arm
 needs no relationship key. `reference_to` is refused at this driver's door as
 well, with the same verdict as the spec's.

 ⚠️ **Upgrading an existing MongoDB deployment:** this index is newer than the
 driver, so the first `syncSchema` after the upgrade BUILDS it across
 collections that already hold data — a real one-off IO and time cost on large
 collections, and index storage that persists. The driver's `CHANGELOG.md`
 entry for the release that added it carries the operational detail. Later
 boots are no-ops: `createIndex` is idempotent for an index that already
 exists.

A `lookup` that omits `reference:` gets no join index — there is no declared
target to join to. `master_detail` and `tree` reach neither branch: they get no
`FOREIGN KEY` and no join index. Declare an `indexes[]` entry if a
`master_detail` child is queried by parent often enough to need one.

Referential integrity is enforced by the **engine** instead: `deleteBehavior` is
applied on delete, which is what produces the `409 DELETE_RESTRICTED` above.
Expand Down

14 changes: 10 additions & 4 deletions
packages/drivers/driver-mongodb/README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -163,9 +163,14 @@ object's `indexes[]` — the one surface an index is declared on (a field-level
`indexed` flag is not a `FieldSchema` key and never built an index,
#2377 / #6810).

⚠️ Lookup fields are **not** indexed today. The lookup arm gates on
`reference_to`, a spelling `FieldSchema` refuses, so a canonically-spelled
`reference` lookup gets no join index — see #13222, which owns that fix.
A `lookup` field that declares `reference` also indexes itself, as
`idx_FIELD_lookup`, for join performance; so does a `user` field. A `lookup`
with no `reference` does not — there is no declared target to join to.

⚠️ **Upgrading:** this join index is newer than the driver. The first
`syncSchema` after the upgrade builds it across collections that already hold
data, which costs real IO and time on large ones. See `CHANGELOG.md` for the
operational detail. Later boots are no-ops.

```typescript
await driver.syncSchema('account', {
Expand All@@ -177,7 +182,8 @@ await driver.syncSchema('account', {
 },
 indexes: [{ fields: ['email'] }],
});
// Creates: idx_id_unique, idx_name_unique, idx_email
// Creates: idx_id_unique, idx_created_at, idx_updated_at,
// idx_name_unique, idx_company_id_lookup, idx_email
```

### Aggregation
Expand Down

89 changes: 61 additions & 28 deletions
packages/drivers/driver-mongodb/src/mongodb-13222-reference-to-refusal.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,13 +19,14 @@
// module between two suites that pin OPPOSITE halves of one arm would couple
// them for no gain.
//
// ⛔ NOT pinned here: whether a canonically-spelled `reference` lookup should
// GET `idx_FIELD_lookup`. That is part (2) of #13222 — a separate, still-open
// ruling (it is a boot-time behaviour change for existing deployments: index
// builds on large collections). The last case below is this PR's own NO-CHANGE
// control for it, and is expected to flip in the PR that takes part (2),
// alongside `mongodb-schema-declared-indexes.test.ts`'s #12252 pin, which owns
// the fact.
// Part (2) of #13222 has since been ruled and taken: the join-index arm now
// gates on the canonical `reference`, so a lookup an author can really publish
// GETS `idx_FIELD_lookup`. The last case below was this file's NO-CHANGE
// control for that question and has flipped accordingly, in the same stroke as
// `mongodb-schema-declared-indexes.test.ts`'s #12252 pin and the real-server
// pin in `mongodb-driver.test.ts`. It still belongs here: it is what proves the
// door and the arm read DIFFERENT keys and disagree about them on purpose —
// `reference_to` refused, `reference` indexed, from one call.

import { describe, it, expect } from 'vitest';
import type { Db } from 'mongodb';
Expand DownExpand Up@@ -187,40 +188,72 @@ describe('#13222 part (1) — driver-mongodb refuses `reference_to` at the schem
 expect(names(created)).toEqual(['idx_id_unique', 'idx_created_at', 'idx_updated_at']);
 });

 it('leaves the join-index arm exactly as it was — part (2) is NOT taken here', async () => {
 // ⚠️ THE NO-CHANGE CONTROL for this PR, and load-bearing in both directions.
 it('refuses `reference_to` and INDEXES `reference` — two keys, two answers, one call', async () => {
 // ⚠️ THE PAIRED CONTROL for the door, and load-bearing in both directions.
 //
 // Positive half: a `user` field still gets `idx_owner_id_lookup`, which
 // proves the arm still executes and that the harness is wired to something —
 // without it the negative half below would pass just as happily against a
 // function that created no indexes at all.
 // The door refuses one relationship spelling; the arm indexes the other. A
 // suite that only ever proves the refusal cannot tell "the door works" from
 // "syncCollectionSchema throws on everything", so the two answers are taken
 // from ONE call here on purpose.
 //
 // Negative half: a canonically-spelled `reference` lookup still gets NO join
 // index. That is the divergence #12252 pinned and part (2) of #13222 owns.
 // ⛔ This case records what the driver DOES, not what it should do: the door
 // added in this PR makes the arm's `field.reference_to` conjunct unreachable
 // but deliberately does not delete it, because deleting it would start
 // building indexes on existing deployments' large collections — an unruled
 // behaviour change. When part (2) lands, this case is expected to flip to
 // `toContain`, in the same stroke as the #12252 pin in
 // `mongodb-schema-declared-indexes.test.ts`.
 // `user` half: still `idx_owner_id_lookup`, and still the unconditional
 // disjunct — it proves the arm executed and names the shape the assertion
 // below is spelled in, so neither half can pass vacuously.
 //
 // Bound through a variable rather than written inline: the driver's own
 // `FieldDef` declares no `reference` key, so a fresh object literal carrying
 // it trips TypeScript's excess-property check.
const canonicalLookup = { type: 'lookup', reference: 'company' };

 // `reference` half: FLIPPED by part (2) of #13222. This case used to assert
 // the canonical lookup got NO index — the divergence #12252 pinned, which
 // held because the arm gated on `reference_to`, a key this very door
// refuses, making the conjunct unreachable for every input. The arm now
 // gates on `reference`, so the lookup is indexed.
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: canonicalLookup, owner_id: { type: 'user' } },
 fields: {
 company_id: { type: 'lookup', reference: 'company' },
 owner_id: { type: 'user' },
 },
 });

 // Exact set, in creation order — closes the vacuity routes a `toContain`
 // pair leaves open: an index appearing under another name, the two lookup
 // indexes swapping fields, or a stray fourth index nobody declared.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_company_id_lookup',
 'idx_owner_id_lookup',
 ]);
 });

 it('does not index a `lookup` that declares no target — `reference` is read for truth, not presence', async () => {
 // Measured on `FieldSchema` built from this tree: `{ type: 'lookup' }` with
 // no `reference`, and `{ type: 'lookup', reference: '' }`, BOTH parse
 // successfully — the spec's prose calls `reference` required for these
 // types, but the schema does not enforce it. So this is a shape an author
 // can really publish, not a hypothetical, and the arm has to answer for it.
 //
 // It answers by declining: `idx_FIELD_lookup` exists to serve a join, and a
 // lookup with no declared target has no join to serve — the index would
 // cost every write and buy no read. This is why the arm gates on
 // truthiness and not on `!== undefined` like the door above does; the two
 // predicates differ deliberately, because they are answering different
 // questions about different keys.
 for (const target of [undefined, '']) {
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: { type: 'lookup', reference: target }, owner_id: { type: 'user' } },
 });

 // The `user` control fires, so the arm ran and the zero below is a real
 // zero rather than a harness that called nothing.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_owner_id_lookup',
 ]);
 }
 });
});

42 changes: 21 additions & 21 deletions
packages/drivers/driver-mongodb/src/mongodb-driver.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -362,31 +362,31 @@ describe.skipIf(!sharedMongod)('MongoDBDriver', () => {
 expect(indexNames).toContain('idx_name_unique');
 expect(indexNames).toContain('idx_email');
 /**
 * ⚠️ [#12252] DIVERGENCE PINNED, DISPOSITION OPEN (#13222) — a
 * canonically-spelled lookup gets NO join index here.
 * ⚠️ DIVERGENCE RETIRED (#13222) — a canonically-spelled lookup gets its
 * join index here, verified against a REAL server rather than a recorder.
 *
 * This fixture used to spell the field `reference_to: 'company'` and
 * assert `idx_company_id_lookup` was CREATED. `reference_to` is a key
 * `FieldSchema` REFUSES (`unrecognized_keys`), so the object it described
 * was one no author could publish — and the assertion passed only because
 * this fixture was the sole thing in the tree reaching the lookup arm of
 * `mongodb-schema.ts`, which gates on `field.reference_to` and reads no
 * other relationship key.
 * This fixture once spelled the field `reference_to: 'company'`. That is
 * a key `FieldSchema` REFUSES (`unrecognized_keys`), so the object it
 * described was one no author could publish — and the assertion passed
 * only because this fixture was the sole thing in the tree reaching the
 * lookup arm of `mongodb-schema.ts`, which gated on `field.reference_to`
 * and read no other relationship key. Correcting the spelling (#13224)
 * therefore did not leave the outcome alone: the index disappeared, and
 * the line was inverted to record that, with the disposition left open.
 *
 * So correcting the spelling does not leave the outcome alone. Measured
 * differentially against the real `syncCollectionSchema`,
 * `idx_company_id_lookup` is the ONE index that disappears; a
 * `type: 'user'` field still gets its index, so the gate is live rather
 * than dead code. The consequence in production is that EVERY authored
 * lookup on MongoDB is unindexed — 57/57 relationship fields across the
 * 44 exported platform objects (#13222).
 * #13222 settled the disposition in two parts: `reference_to` is now
 * refused at the driver's door, and the arm gates on the canonical
 * `reference`. So the line is inverted back — by a ruling, not by a
 * re-baseline.
 *
 * ⛔ This records what the driver DOES, not what it SHOULD do. Whether
 * the lookup arm learns to read `reference` is #13222's to settle, ⛔ not
 * this pin's — when it lands, this line flips back to `toContain`
 * deliberately rather than the divergence reopening in silence.
 * ⚠️ This copy is the one that runs against real MongoDB, which is what
 * it is FOR: it proves the server actually materialized the index under
 * that name, not merely that the driver asked for it. The recorder-driven
 * twin in `mongodb-schema-declared-indexes.test.ts` is the copy that runs
 * on every ordinary CI lane — this suite is `describe.skipIf(!sharedMongod)`.
 * TWIN PIN — edit either, edit both.
 */
 expect(indexNames).not.toContain('idx_company_id_lookup');
 expect(indexNames).toContain('idx_company_id_lookup');
 });

 it('should be idempotent (safe to call multiple times)', async () => {
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
feat(driver-mongodb): index lookup joins off the canonical `reference` key by claude[bot] · Pull Request #13633 · objectstack-ai/objectstack · GitHub
Skip to content
feat(driver-mongodb): index lookup joins off the canonical reference key#13633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

zhuangjianguo
merged 1 commit into

from

Aug 31, 2026

+298

−129

 Merged


feat(driver-mongodb): index lookup joins off the canonical reference key
#13633
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


90 changes: 90 additions & 0 deletions
.changeset/mongodb-lookup-index-canonical-reference.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
---
"@objectstack/driver-mongodb": minor
---

feat(driver-mongodb): index `lookup` joins off the canonical `reference` key (#13222)

`syncCollectionSchema`'s field-level join-index arm gated on `field.reference_to`.
That is a REJECTED ALIAS — `FieldSchema` answers `unrecognized_keys` for it on any
field type — and this driver's own schema door refuses it outright. So the arm's
`lookup` conjunct could not be satisfied by any input at all, and no authored
lookup had ever been indexed on MongoDB. The arm now reads `reference`, the only
relationship spelling the spec declares.

Measured as a complete case split over the key's value domain rather than a
sample: every `reference_to` value except `undefined` is refused at the door, and
`undefined` is falsy, so the old conjunct was unreachable for every possible
input. The `user` disjunct was unaffected and is why the feature looked healthy —
it needs no relationship key, so `idx_owner_lookup`-shaped indexes were always
created.

The refusal door itself is unchanged: predicate, `VALIDATION_ERROR`/400 envelope,
placement and instruction are all as they were. Only the tail of its message
moved, because it told the reader that renaming the key would not by itself get
the field an index — true when written, false now.

A `lookup` that declares no `reference` is still not indexed. Measured on
`FieldSchema`: `{ type: 'lookup' }` and `{ type: 'lookup', reference: '' }` both
parse successfully — the spec's prose calls `reference` required for these types
but the schema does not enforce it — so this is a real authorable shape, and an
index for a join with no declared target would cost every write and buy no read.
`master_detail` and `tree` are unchanged: they reach this arm on neither spelling.

## ⚠️ OPERATORS — this changes boot behaviour on existing deployments

**What it costs.** The first `syncSchema` after this upgrade CREATES these
indexes on collections that already hold data. Each one is an awaited
`createIndex`: a full collection scan plus an external sort of the extracted
keys, followed by permanent index storage and a small write amplification on
every subsequent insert/update of the indexed field. ("Awaited" describes the
driver, not the server — see the build-feature note below for what the server
does during it.) Later boots are free —
`createIndex` is idempotent for an index that already exists, and this driver
already relies on that.

⚠️ **The builds are SERIALIZED, so the times ADD.** `syncCollectionSchema` awaits
`createIndex` once per index in a sequential loop, and `syncSchemasBatch` awaits
`syncSchema` once per object the same way. Startup is extended by the SUM of
every build, not by the slowest one. This is the figure to plan the maintenance
window around.

**How much.** Measured on this tree: **65 lookup fields carrying `reference`
across the 52 exported platform objects** — every one gains an index, and this is
the floor, not the total. Add the objects of any enabled plugin (`plugin-security`
14 lookup fields, `plugin-approvals` 13, `plugin-audit` 6, `plugin-sharing` 4) and
one index per lookup field on your own authored objects.

Per index, as an order-of-magnitude planning figure and **not** a benchmark — get
your own numbers before sizing a window, because they depend entirely on document
count, storage and cache:

| collection size | build time, one index | index storage added |
|---|---|---|
| empty / a few thousand docs | effectively instant (metadata-only) | negligible |
| ~1M docs | seconds | tens of MB |
| ~10M docs | ~a minute | hundreds of MB |
| ~100M docs | tens of minutes | a few GB |

Most `sys_*` collections are small and will finish instantly. Budget for the ones
that accumulate: `sys_metadata_history`, `sys_metadata_audit`, `sys_notification`,
`sys_email`, `sys_session`, and any audit-log object. Count them first
(`db.<collection>.estimatedDocumentCount()`) and multiply by the lookup fields
each carries; watch a live boot with `db.currentOp()`.

**Build feature — hybrid builds, unconditionally.** `@objectstack/driver-mongodb`
depends on `mongodb@^7.5.0`, whose own compatibility statement is "the driver
currently supports 4.2+ servers". MongoDB 4.2 is exactly the release that made
index builds hybrid, so **every server version this driver can connect to builds
these indexes with the hybrid builder**: an exclusive lock is taken only briefly
at the start and end of each build, and the collection accepts reads AND writes
throughout the rest of it. This is not a full write stall. Two caveats that
remain: the brief exclusive lock at each end is real, and on a replica set the
build runs on every member.

**To take the cost outside the boot window,** create the indexes ahead of the
upgrade, with the names and specs the driver uses — `{ name: 'idx_FIELD_lookup' }`
over `{ FIELD: 1 }`, no `unique` and no `sparse`. The driver's `createIndex` is
then a no-op and startup is unaffected. Matching the options matters: a
pre-existing index of the same name with different options raises
`IndexOptionsConflict`, which this driver deliberately swallows and skips, leaving
your index in place unchanged.

37 changes: 24 additions & 13 deletions
content/docs/protocol/objectql/types.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -731,23 +731,34 @@ contacts:

<Callout type="warn">
A relationship field authored with `reference:` gets **no database-level
`FOREIGN KEY` constraint** and **no MongoDB join index**. Both gaps have one
root cause: the driver branches that would have built them were gated on
`reference_to` — a key the spec REFUSES (`FieldSchema` answers
`unrecognized_keys` for it, on any field type) and one that `reference` never
populates. `master_detail` / `tree` did not reach either branch at all.
`FOREIGN KEY` constraint** on any driver. It does get a **MongoDB join index**.
Both branches were once gated on `reference_to` — a key the spec REFUSES
(`FieldSchema` answers `unrecognized_keys` for it, on any field type) and one
that `reference` never populates — so neither fired for authored metadata; the
two were then settled in opposite directions, on their own merits.

- **SQL:** the `FOREIGN KEY` DDL is retired (#11567). A field that still carries
 `reference_to` when it reaches DDL is refused at the driver's door, in the
 schema's own words (`400 VALIDATION_ERROR`), instead of silently changing the
 physical schema.
- **MongoDB:** the field-level join index `idx_FIELD_lookup` is gated on that
 same refused key, so a canonically-spelled `reference` lookup is **not
 indexed**. A `user` field still is — that arm needs no relationship key, which
 is why the feature looked healthy. `reference_to` is refused at this driver's
 door too, with the same verdict (#13222); whether a canonical `reference`
 lookup should start building the index is tracked separately, because it
 changes boot behaviour for deployments that already hold large collections.
 physical schema. Declare an index in `indexes[]` if you want one on the
 foreign-key column.
- **MongoDB:** a `lookup` field declaring `reference:` gets the field-level join
 index `idx_FIELD_lookup` (#13222). A `user` field gets one too — that arm
 needs no relationship key. `reference_to` is refused at this driver's door as
 well, with the same verdict as the spec's.

 ⚠️ **Upgrading an existing MongoDB deployment:** this index is newer than the
 driver, so the first `syncSchema` after the upgrade BUILDS it across
 collections that already hold data — a real one-off IO and time cost on large
 collections, and index storage that persists. The driver's `CHANGELOG.md`
 entry for the release that added it carries the operational detail. Later
 boots are no-ops: `createIndex` is idempotent for an index that already
 exists.

A `lookup` that omits `reference:` gets no join index — there is no declared
target to join to. `master_detail` and `tree` reach neither branch: they get no
`FOREIGN KEY` and no join index. Declare an `indexes[]` entry if a
`master_detail` child is queried by parent often enough to need one.

Referential integrity is enforced by the **engine** instead: `deleteBehavior` is
applied on delete, which is what produces the `409 DELETE_RESTRICTED` above.
Expand Down

14 changes: 10 additions & 4 deletions
packages/drivers/driver-mongodb/README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -163,9 +163,14 @@ object's `indexes[]` — the one surface an index is declared on (a field-level
`indexed` flag is not a `FieldSchema` key and never built an index,
#2377 / #6810).

⚠️ Lookup fields are **not** indexed today. The lookup arm gates on
`reference_to`, a spelling `FieldSchema` refuses, so a canonically-spelled
`reference` lookup gets no join index — see #13222, which owns that fix.
A `lookup` field that declares `reference` also indexes itself, as
`idx_FIELD_lookup`, for join performance; so does a `user` field. A `lookup`
with no `reference` does not — there is no declared target to join to.

⚠️ **Upgrading:** this join index is newer than the driver. The first
`syncSchema` after the upgrade builds it across collections that already hold
data, which costs real IO and time on large ones. See `CHANGELOG.md` for the
operational detail. Later boots are no-ops.

```typescript
await driver.syncSchema('account', {
Expand All@@ -177,7 +182,8 @@ await driver.syncSchema('account', {
 },
 indexes: [{ fields: ['email'] }],
});
// Creates: idx_id_unique, idx_name_unique, idx_email
// Creates: idx_id_unique, idx_created_at, idx_updated_at,
// idx_name_unique, idx_company_id_lookup, idx_email
```

### Aggregation
Expand Down

89 changes: 61 additions & 28 deletions
packages/drivers/driver-mongodb/src/mongodb-13222-reference-to-refusal.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,13 +19,14 @@
// module between two suites that pin OPPOSITE halves of one arm would couple
// them for no gain.
//
// ⛔ NOT pinned here: whether a canonically-spelled `reference` lookup should
// GET `idx_FIELD_lookup`. That is part (2) of #13222 — a separate, still-open
// ruling (it is a boot-time behaviour change for existing deployments: index
// builds on large collections). The last case below is this PR's own NO-CHANGE
// control for it, and is expected to flip in the PR that takes part (2),
// alongside `mongodb-schema-declared-indexes.test.ts`'s #12252 pin, which owns
// the fact.
// Part (2) of #13222 has since been ruled and taken: the join-index arm now
// gates on the canonical `reference`, so a lookup an author can really publish
// GETS `idx_FIELD_lookup`. The last case below was this file's NO-CHANGE
// control for that question and has flipped accordingly, in the same stroke as
// `mongodb-schema-declared-indexes.test.ts`'s #12252 pin and the real-server
// pin in `mongodb-driver.test.ts`. It still belongs here: it is what proves the
// door and the arm read DIFFERENT keys and disagree about them on purpose —
// `reference_to` refused, `reference` indexed, from one call.

import { describe, it, expect } from 'vitest';
import type { Db } from 'mongodb';
Expand DownExpand Up@@ -187,40 +188,72 @@ describe('#13222 part (1) — driver-mongodb refuses `reference_to` at the schem
 expect(names(created)).toEqual(['idx_id_unique', 'idx_created_at', 'idx_updated_at']);
 });

 it('leaves the join-index arm exactly as it was — part (2) is NOT taken here', async () => {
 // ⚠️ THE NO-CHANGE CONTROL for this PR, and load-bearing in both directions.
 it('refuses `reference_to` and INDEXES `reference` — two keys, two answers, one call', async () => {
 // ⚠️ THE PAIRED CONTROL for the door, and load-bearing in both directions.
 //
 // Positive half: a `user` field still gets `idx_owner_id_lookup`, which
 // proves the arm still executes and that the harness is wired to something —
 // without it the negative half below would pass just as happily against a
 // function that created no indexes at all.
 // The door refuses one relationship spelling; the arm indexes the other. A
 // suite that only ever proves the refusal cannot tell "the door works" from
 // "syncCollectionSchema throws on everything", so the two answers are taken
 // from ONE call here on purpose.
 //
 // Negative half: a canonically-spelled `reference` lookup still gets NO join
 // index. That is the divergence #12252 pinned and part (2) of #13222 owns.
 // ⛔ This case records what the driver DOES, not what it should do: the door
 // added in this PR makes the arm's `field.reference_to` conjunct unreachable
 // but deliberately does not delete it, because deleting it would start
 // building indexes on existing deployments' large collections — an unruled
 // behaviour change. When part (2) lands, this case is expected to flip to
 // `toContain`, in the same stroke as the #12252 pin in
 // `mongodb-schema-declared-indexes.test.ts`.
 // `user` half: still `idx_owner_id_lookup`, and still the unconditional
 // disjunct — it proves the arm executed and names the shape the assertion
 // below is spelled in, so neither half can pass vacuously.
 //
 // Bound through a variable rather than written inline: the driver's own
 // `FieldDef` declares no `reference` key, so a fresh object literal carrying
 // it trips TypeScript's excess-property check.
const canonicalLookup = { type: 'lookup', reference: 'company' };

 // `reference` half: FLIPPED by part (2) of #13222. This case used to assert
 // the canonical lookup got NO index — the divergence #12252 pinned, which
 // held because the arm gated on `reference_to`, a key this very door
// refuses, making the conjunct unreachable for every input. The arm now
 // gates on `reference`, so the lookup is indexed.
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: canonicalLookup, owner_id: { type: 'user' } },
 fields: {
 company_id: { type: 'lookup', reference: 'company' },
 owner_id: { type: 'user' },
 },
 });

 // Exact set, in creation order — closes the vacuity routes a `toContain`
 // pair leaves open: an index appearing under another name, the two lookup
 // indexes swapping fields, or a stray fourth index nobody declared.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_company_id_lookup',
 'idx_owner_id_lookup',
 ]);
 });

 it('does not index a `lookup` that declares no target — `reference` is read for truth, not presence', async () => {
 // Measured on `FieldSchema` built from this tree: `{ type: 'lookup' }` with
 // no `reference`, and `{ type: 'lookup', reference: '' }`, BOTH parse
 // successfully — the spec's prose calls `reference` required for these
 // types, but the schema does not enforce it. So this is a shape an author
 // can really publish, not a hypothetical, and the arm has to answer for it.
 //
 // It answers by declining: `idx_FIELD_lookup` exists to serve a join, and a
 // lookup with no declared target has no join to serve — the index would
 // cost every write and buy no read. This is why the arm gates on
 // truthiness and not on `!== undefined` like the door above does; the two
 // predicates differ deliberately, because they are answering different
 // questions about different keys.
 for (const target of [undefined, '']) {
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: { type: 'lookup', reference: target }, owner_id: { type: 'user' } },
 });

 // The `user` control fires, so the arm ran and the zero below is a real
 // zero rather than a harness that called nothing.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_owner_id_lookup',
 ]);
 }
 });
});

42 changes: 21 additions & 21 deletions
packages/drivers/driver-mongodb/src/mongodb-driver.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -362,31 +362,31 @@ describe.skipIf(!sharedMongod)('MongoDBDriver', () => {
 expect(indexNames).toContain('idx_name_unique');
 expect(indexNames).toContain('idx_email');
 /**
 * ⚠️ [#12252] DIVERGENCE PINNED, DISPOSITION OPEN (#13222) — a
 * canonically-spelled lookup gets NO join index here.
 * ⚠️ DIVERGENCE RETIRED (#13222) — a canonically-spelled lookup gets its
 * join index here, verified against a REAL server rather than a recorder.
 *
 * This fixture used to spell the field `reference_to: 'company'` and
 * assert `idx_company_id_lookup` was CREATED. `reference_to` is a key
 * `FieldSchema` REFUSES (`unrecognized_keys`), so the object it described
 * was one no author could publish — and the assertion passed only because
 * this fixture was the sole thing in the tree reaching the lookup arm of
 * `mongodb-schema.ts`, which gates on `field.reference_to` and reads no
 * other relationship key.
 * This fixture once spelled the field `reference_to: 'company'`. That is
 * a key `FieldSchema` REFUSES (`unrecognized_keys`), so the object it
 * described was one no author could publish — and the assertion passed
 * only because this fixture was the sole thing in the tree reaching the
 * lookup arm of `mongodb-schema.ts`, which gated on `field.reference_to`
 * and read no other relationship key. Correcting the spelling (#13224)
 * therefore did not leave the outcome alone: the index disappeared, and
 * the line was inverted to record that, with the disposition left open.
 *
 * So correcting the spelling does not leave the outcome alone. Measured
 * differentially against the real `syncCollectionSchema`,
 * `idx_company_id_lookup` is the ONE index that disappears; a
 * `type: 'user'` field still gets its index, so the gate is live rather
 * than dead code. The consequence in production is that EVERY authored
 * lookup on MongoDB is unindexed — 57/57 relationship fields across the
 * 44 exported platform objects (#13222).
 * #13222 settled the disposition in two parts: `reference_to` is now
 * refused at the driver's door, and the arm gates on the canonical
 * `reference`. So the line is inverted back — by a ruling, not by a
 * re-baseline.
 *
 * ⛔ This records what the driver DOES, not what it SHOULD do. Whether
 * the lookup arm learns to read `reference` is #13222's to settle, ⛔ not
 * this pin's — when it lands, this line flips back to `toContain`
 * deliberately rather than the divergence reopening in silence.
 * ⚠️ This copy is the one that runs against real MongoDB, which is what
 * it is FOR: it proves the server actually materialized the index under
 * that name, not merely that the driver asked for it. The recorder-driven
 * twin in `mongodb-schema-declared-indexes.test.ts` is the copy that runs
 * on every ordinary CI lane — this suite is `describe.skipIf(!sharedMongod)`.
 * TWIN PIN — edit either, edit both.
 */
 expect(indexNames).not.toContain('idx_company_id_lookup');
 expect(indexNames).toContain('idx_company_id_lookup');
 });

 it('should be idempotent (safe to call multiple times)', async () => {
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
feat(driver-mongodb): index lookup joins off the canonical `reference` key by claude[bot] · Pull Request #13633 · objectstack-ai/objectstack · GitHub
Skip to content
feat(driver-mongodb): index lookup joins off the canonical reference key#13633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

zhuangjianguo
merged 1 commit into

from

Aug 31, 2026

+298

−129

 Merged


feat(driver-mongodb): index lookup joins off the canonical reference key
#13633
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


90 changes: 90 additions & 0 deletions
.changeset/mongodb-lookup-index-canonical-reference.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
---
"@objectstack/driver-mongodb": minor
---

feat(driver-mongodb): index `lookup` joins off the canonical `reference` key (#13222)

`syncCollectionSchema`'s field-level join-index arm gated on `field.reference_to`.
That is a REJECTED ALIAS — `FieldSchema` answers `unrecognized_keys` for it on any
field type — and this driver's own schema door refuses it outright. So the arm's
`lookup` conjunct could not be satisfied by any input at all, and no authored
lookup had ever been indexed on MongoDB. The arm now reads `reference`, the only
relationship spelling the spec declares.

Measured as a complete case split over the key's value domain rather than a
sample: every `reference_to` value except `undefined` is refused at the door, and
`undefined` is falsy, so the old conjunct was unreachable for every possible
input. The `user` disjunct was unaffected and is why the feature looked healthy —
it needs no relationship key, so `idx_owner_lookup`-shaped indexes were always
created.

The refusal door itself is unchanged: predicate, `VALIDATION_ERROR`/400 envelope,
placement and instruction are all as they were. Only the tail of its message
moved, because it told the reader that renaming the key would not by itself get
the field an index — true when written, false now.

A `lookup` that declares no `reference` is still not indexed. Measured on
`FieldSchema`: `{ type: 'lookup' }` and `{ type: 'lookup', reference: '' }` both
parse successfully — the spec's prose calls `reference` required for these types
but the schema does not enforce it — so this is a real authorable shape, and an
index for a join with no declared target would cost every write and buy no read.
`master_detail` and `tree` are unchanged: they reach this arm on neither spelling.

## ⚠️ OPERATORS — this changes boot behaviour on existing deployments

**What it costs.** The first `syncSchema` after this upgrade CREATES these
indexes on collections that already hold data. Each one is an awaited
`createIndex`: a full collection scan plus an external sort of the extracted
keys, followed by permanent index storage and a small write amplification on
every subsequent insert/update of the indexed field. ("Awaited" describes the
driver, not the server — see the build-feature note below for what the server
does during it.) Later boots are free —
`createIndex` is idempotent for an index that already exists, and this driver
already relies on that.

⚠️ **The builds are SERIALIZED, so the times ADD.** `syncCollectionSchema` awaits
`createIndex` once per index in a sequential loop, and `syncSchemasBatch` awaits
`syncSchema` once per object the same way. Startup is extended by the SUM of
every build, not by the slowest one. This is the figure to plan the maintenance
window around.

**How much.** Measured on this tree: **65 lookup fields carrying `reference`
across the 52 exported platform objects** — every one gains an index, and this is
the floor, not the total. Add the objects of any enabled plugin (`plugin-security`
14 lookup fields, `plugin-approvals` 13, `plugin-audit` 6, `plugin-sharing` 4) and
one index per lookup field on your own authored objects.

Per index, as an order-of-magnitude planning figure and **not** a benchmark — get
your own numbers before sizing a window, because they depend entirely on document
count, storage and cache:

| collection size | build time, one index | index storage added |
|---|---|---|
| empty / a few thousand docs | effectively instant (metadata-only) | negligible |
| ~1M docs | seconds | tens of MB |
| ~10M docs | ~a minute | hundreds of MB |
| ~100M docs | tens of minutes | a few GB |

Most `sys_*` collections are small and will finish instantly. Budget for the ones
that accumulate: `sys_metadata_history`, `sys_metadata_audit`, `sys_notification`,
`sys_email`, `sys_session`, and any audit-log object. Count them first
(`db.<collection>.estimatedDocumentCount()`) and multiply by the lookup fields
each carries; watch a live boot with `db.currentOp()`.

**Build feature — hybrid builds, unconditionally.** `@objectstack/driver-mongodb`
depends on `mongodb@^7.5.0`, whose own compatibility statement is "the driver
currently supports 4.2+ servers". MongoDB 4.2 is exactly the release that made
index builds hybrid, so **every server version this driver can connect to builds
these indexes with the hybrid builder**: an exclusive lock is taken only briefly
at the start and end of each build, and the collection accepts reads AND writes
throughout the rest of it. This is not a full write stall. Two caveats that
remain: the brief exclusive lock at each end is real, and on a replica set the
build runs on every member.

**To take the cost outside the boot window,** create the indexes ahead of the
upgrade, with the names and specs the driver uses — `{ name: 'idx_FIELD_lookup' }`
over `{ FIELD: 1 }`, no `unique` and no `sparse`. The driver's `createIndex` is
then a no-op and startup is unaffected. Matching the options matters: a
pre-existing index of the same name with different options raises
`IndexOptionsConflict`, which this driver deliberately swallows and skips, leaving
your index in place unchanged.

37 changes: 24 additions & 13 deletions
content/docs/protocol/objectql/types.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -731,23 +731,34 @@ contacts:

<Callout type="warn">
A relationship field authored with `reference:` gets **no database-level
`FOREIGN KEY` constraint** and **no MongoDB join index**. Both gaps have one
root cause: the driver branches that would have built them were gated on
`reference_to` — a key the spec REFUSES (`FieldSchema` answers
`unrecognized_keys` for it, on any field type) and one that `reference` never
populates. `master_detail` / `tree` did not reach either branch at all.
`FOREIGN KEY` constraint** on any driver. It does get a **MongoDB join index**.
Both branches were once gated on `reference_to` — a key the spec REFUSES
(`FieldSchema` answers `unrecognized_keys` for it, on any field type) and one
that `reference` never populates — so neither fired for authored metadata; the
two were then settled in opposite directions, on their own merits.

- **SQL:** the `FOREIGN KEY` DDL is retired (#11567). A field that still carries
 `reference_to` when it reaches DDL is refused at the driver's door, in the
 schema's own words (`400 VALIDATION_ERROR`), instead of silently changing the
 physical schema.
- **MongoDB:** the field-level join index `idx_FIELD_lookup` is gated on that
 same refused key, so a canonically-spelled `reference` lookup is **not
 indexed**. A `user` field still is — that arm needs no relationship key, which
 is why the feature looked healthy. `reference_to` is refused at this driver's
 door too, with the same verdict (#13222); whether a canonical `reference`
 lookup should start building the index is tracked separately, because it
 changes boot behaviour for deployments that already hold large collections.
 physical schema. Declare an index in `indexes[]` if you want one on the
 foreign-key column.
- **MongoDB:** a `lookup` field declaring `reference:` gets the field-level join
 index `idx_FIELD_lookup` (#13222). A `user` field gets one too — that arm
 needs no relationship key. `reference_to` is refused at this driver's door as
 well, with the same verdict as the spec's.

 ⚠️ **Upgrading an existing MongoDB deployment:** this index is newer than the
 driver, so the first `syncSchema` after the upgrade BUILDS it across
 collections that already hold data — a real one-off IO and time cost on large
 collections, and index storage that persists. The driver's `CHANGELOG.md`
 entry for the release that added it carries the operational detail. Later
 boots are no-ops: `createIndex` is idempotent for an index that already
 exists.

A `lookup` that omits `reference:` gets no join index — there is no declared
target to join to. `master_detail` and `tree` reach neither branch: they get no
`FOREIGN KEY` and no join index. Declare an `indexes[]` entry if a
`master_detail` child is queried by parent often enough to need one.

Referential integrity is enforced by the **engine** instead: `deleteBehavior` is
applied on delete, which is what produces the `409 DELETE_RESTRICTED` above.
Expand Down

14 changes: 10 additions & 4 deletions
packages/drivers/driver-mongodb/README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -163,9 +163,14 @@ object's `indexes[]` — the one surface an index is declared on (a field-level
`indexed` flag is not a `FieldSchema` key and never built an index,
#2377 / #6810).

⚠️ Lookup fields are **not** indexed today. The lookup arm gates on
`reference_to`, a spelling `FieldSchema` refuses, so a canonically-spelled
`reference` lookup gets no join index — see #13222, which owns that fix.
A `lookup` field that declares `reference` also indexes itself, as
`idx_FIELD_lookup`, for join performance; so does a `user` field. A `lookup`
with no `reference` does not — there is no declared target to join to.

⚠️ **Upgrading:** this join index is newer than the driver. The first
`syncSchema` after the upgrade builds it across collections that already hold
data, which costs real IO and time on large ones. See `CHANGELOG.md` for the
operational detail. Later boots are no-ops.

```typescript
await driver.syncSchema('account', {
Expand All@@ -177,7 +182,8 @@ await driver.syncSchema('account', {
 },
 indexes: [{ fields: ['email'] }],
});
// Creates: idx_id_unique, idx_name_unique, idx_email
// Creates: idx_id_unique, idx_created_at, idx_updated_at,
// idx_name_unique, idx_company_id_lookup, idx_email
```

### Aggregation
Expand Down

89 changes: 61 additions & 28 deletions
packages/drivers/driver-mongodb/src/mongodb-13222-reference-to-refusal.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,13 +19,14 @@
// module between two suites that pin OPPOSITE halves of one arm would couple
// them for no gain.
//
// ⛔ NOT pinned here: whether a canonically-spelled `reference` lookup should
// GET `idx_FIELD_lookup`. That is part (2) of #13222 — a separate, still-open
// ruling (it is a boot-time behaviour change for existing deployments: index
// builds on large collections). The last case below is this PR's own NO-CHANGE
// control for it, and is expected to flip in the PR that takes part (2),
// alongside `mongodb-schema-declared-indexes.test.ts`'s #12252 pin, which owns
// the fact.
// Part (2) of #13222 has since been ruled and taken: the join-index arm now
// gates on the canonical `reference`, so a lookup an author can really publish
// GETS `idx_FIELD_lookup`. The last case below was this file's NO-CHANGE
// control for that question and has flipped accordingly, in the same stroke as
// `mongodb-schema-declared-indexes.test.ts`'s #12252 pin and the real-server
// pin in `mongodb-driver.test.ts`. It still belongs here: it is what proves the
// door and the arm read DIFFERENT keys and disagree about them on purpose —
// `reference_to` refused, `reference` indexed, from one call.

import { describe, it, expect } from 'vitest';
import type { Db } from 'mongodb';
Expand DownExpand Up@@ -187,40 +188,72 @@ describe('#13222 part (1) — driver-mongodb refuses `reference_to` at the schem
 expect(names(created)).toEqual(['idx_id_unique', 'idx_created_at', 'idx_updated_at']);
 });

 it('leaves the join-index arm exactly as it was — part (2) is NOT taken here', async () => {
 // ⚠️ THE NO-CHANGE CONTROL for this PR, and load-bearing in both directions.
 it('refuses `reference_to` and INDEXES `reference` — two keys, two answers, one call', async () => {
 // ⚠️ THE PAIRED CONTROL for the door, and load-bearing in both directions.
 //
 // Positive half: a `user` field still gets `idx_owner_id_lookup`, which
 // proves the arm still executes and that the harness is wired to something —
 // without it the negative half below would pass just as happily against a
 // function that created no indexes at all.
 // The door refuses one relationship spelling; the arm indexes the other. A
 // suite that only ever proves the refusal cannot tell "the door works" from
 // "syncCollectionSchema throws on everything", so the two answers are taken
 // from ONE call here on purpose.
 //
 // Negative half: a canonically-spelled `reference` lookup still gets NO join
 // index. That is the divergence #12252 pinned and part (2) of #13222 owns.
 // ⛔ This case records what the driver DOES, not what it should do: the door
 // added in this PR makes the arm's `field.reference_to` conjunct unreachable
 // but deliberately does not delete it, because deleting it would start
 // building indexes on existing deployments' large collections — an unruled
 // behaviour change. When part (2) lands, this case is expected to flip to
 // `toContain`, in the same stroke as the #12252 pin in
 // `mongodb-schema-declared-indexes.test.ts`.
 // `user` half: still `idx_owner_id_lookup`, and still the unconditional
 // disjunct — it proves the arm executed and names the shape the assertion
 // below is spelled in, so neither half can pass vacuously.
 //
 // Bound through a variable rather than written inline: the driver's own
 // `FieldDef` declares no `reference` key, so a fresh object literal carrying
 // it trips TypeScript's excess-property check.
const canonicalLookup = { type: 'lookup', reference: 'company' };

 // `reference` half: FLIPPED by part (2) of #13222. This case used to assert
 // the canonical lookup got NO index — the divergence #12252 pinned, which
 // held because the arm gated on `reference_to`, a key this very door
// refuses, making the conjunct unreachable for every input. The arm now
 // gates on `reference`, so the lookup is indexed.
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: canonicalLookup, owner_id: { type: 'user' } },
 fields: {
 company_id: { type: 'lookup', reference: 'company' },
 owner_id: { type: 'user' },
 },
 });

 // Exact set, in creation order — closes the vacuity routes a `toContain`
 // pair leaves open: an index appearing under another name, the two lookup
 // indexes swapping fields, or a stray fourth index nobody declared.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_company_id_lookup',
 'idx_owner_id_lookup',
 ]);
 });

 it('does not index a `lookup` that declares no target — `reference` is read for truth, not presence', async () => {
 // Measured on `FieldSchema` built from this tree: `{ type: 'lookup' }` with
 // no `reference`, and `{ type: 'lookup', reference: '' }`, BOTH parse
 // successfully — the spec's prose calls `reference` required for these
 // types, but the schema does not enforce it. So this is a shape an author
 // can really publish, not a hypothetical, and the arm has to answer for it.
 //
 // It answers by declining: `idx_FIELD_lookup` exists to serve a join, and a
 // lookup with no declared target has no join to serve — the index would
 // cost every write and buy no read. This is why the arm gates on
 // truthiness and not on `!== undefined` like the door above does; the two
 // predicates differ deliberately, because they are answering different
 // questions about different keys.
 for (const target of [undefined, '']) {
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: { type: 'lookup', reference: target }, owner_id: { type: 'user' } },
 });

 // The `user` control fires, so the arm ran and the zero below is a real
 // zero rather than a harness that called nothing.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_owner_id_lookup',
 ]);
 }
 });
});

42 changes: 21 additions & 21 deletions
packages/drivers/driver-mongodb/src/mongodb-driver.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -362,31 +362,31 @@ describe.skipIf(!sharedMongod)('MongoDBDriver', () => {
 expect(indexNames).toContain('idx_name_unique');
 expect(indexNames).toContain('idx_email');
 /**
 * ⚠️ [#12252] DIVERGENCE PINNED, DISPOSITION OPEN (#13222) — a
 * canonically-spelled lookup gets NO join index here.
 * ⚠️ DIVERGENCE RETIRED (#13222) — a canonically-spelled lookup gets its
 * join index here, verified against a REAL server rather than a recorder.
 *
 * This fixture used to spell the field `reference_to: 'company'` and
 * assert `idx_company_id_lookup` was CREATED. `reference_to` is a key
 * `FieldSchema` REFUSES (`unrecognized_keys`), so the object it described
 * was one no author could publish — and the assertion passed only because
 * this fixture was the sole thing in the tree reaching the lookup arm of
 * `mongodb-schema.ts`, which gates on `field.reference_to` and reads no
 * other relationship key.
 * This fixture once spelled the field `reference_to: 'company'`. That is
 * a key `FieldSchema` REFUSES (`unrecognized_keys`), so the object it
 * described was one no author could publish — and the assertion passed
 * only because this fixture was the sole thing in the tree reaching the
 * lookup arm of `mongodb-schema.ts`, which gated on `field.reference_to`
 * and read no other relationship key. Correcting the spelling (#13224)
 * therefore did not leave the outcome alone: the index disappeared, and
 * the line was inverted to record that, with the disposition left open.
 *
 * So correcting the spelling does not leave the outcome alone. Measured
 * differentially against the real `syncCollectionSchema`,
 * `idx_company_id_lookup` is the ONE index that disappears; a
 * `type: 'user'` field still gets its index, so the gate is live rather
 * than dead code. The consequence in production is that EVERY authored
 * lookup on MongoDB is unindexed — 57/57 relationship fields across the
 * 44 exported platform objects (#13222).
 * #13222 settled the disposition in two parts: `reference_to` is now
 * refused at the driver's door, and the arm gates on the canonical
 * `reference`. So the line is inverted back — by a ruling, not by a
 * re-baseline.
 *
 * ⛔ This records what the driver DOES, not what it SHOULD do. Whether
 * the lookup arm learns to read `reference` is #13222's to settle, ⛔ not
 * this pin's — when it lands, this line flips back to `toContain`
 * deliberately rather than the divergence reopening in silence.
 * ⚠️ This copy is the one that runs against real MongoDB, which is what
 * it is FOR: it proves the server actually materialized the index under
 * that name, not merely that the driver asked for it. The recorder-driven
 * twin in `mongodb-schema-declared-indexes.test.ts` is the copy that runs
 * on every ordinary CI lane — this suite is `describe.skipIf(!sharedMongod)`.
 * TWIN PIN — edit either, edit both.
 */
 expect(indexNames).not.toContain('idx_company_id_lookup');
 expect(indexNames).toContain('idx_company_id_lookup');
 });

 it('should be idempotent (safe to call multiple times)', async () => {
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
feat(driver-mongodb): index lookup joins off the canonical `reference` key by claude[bot] · Pull Request #13633 · objectstack-ai/objectstack · GitHub
Skip to content
feat(driver-mongodb): index lookup joins off the canonical reference key#13633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

zhuangjianguo
merged 1 commit into

from

Aug 31, 2026

+298

−129

 Merged


feat(driver-mongodb): index lookup joins off the canonical reference key
#13633
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


90 changes: 90 additions & 0 deletions
.changeset/mongodb-lookup-index-canonical-reference.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
---
"@objectstack/driver-mongodb": minor
---

feat(driver-mongodb): index `lookup` joins off the canonical `reference` key (#13222)

`syncCollectionSchema`'s field-level join-index arm gated on `field.reference_to`.
That is a REJECTED ALIAS — `FieldSchema` answers `unrecognized_keys` for it on any
field type — and this driver's own schema door refuses it outright. So the arm's
`lookup` conjunct could not be satisfied by any input at all, and no authored
lookup had ever been indexed on MongoDB. The arm now reads `reference`, the only
relationship spelling the spec declares.

Measured as a complete case split over the key's value domain rather than a
sample: every `reference_to` value except `undefined` is refused at the door, and
`undefined` is falsy, so the old conjunct was unreachable for every possible
input. The `user` disjunct was unaffected and is why the feature looked healthy —
it needs no relationship key, so `idx_owner_lookup`-shaped indexes were always
created.

The refusal door itself is unchanged: predicate, `VALIDATION_ERROR`/400 envelope,
placement and instruction are all as they were. Only the tail of its message
moved, because it told the reader that renaming the key would not by itself get
the field an index — true when written, false now.

A `lookup` that declares no `reference` is still not indexed. Measured on
`FieldSchema`: `{ type: 'lookup' }` and `{ type: 'lookup', reference: '' }` both
parse successfully — the spec's prose calls `reference` required for these types
but the schema does not enforce it — so this is a real authorable shape, and an
index for a join with no declared target would cost every write and buy no read.
`master_detail` and `tree` are unchanged: they reach this arm on neither spelling.

## ⚠️ OPERATORS — this changes boot behaviour on existing deployments

**What it costs.** The first `syncSchema` after this upgrade CREATES these
indexes on collections that already hold data. Each one is an awaited
`createIndex`: a full collection scan plus an external sort of the extracted
keys, followed by permanent index storage and a small write amplification on
every subsequent insert/update of the indexed field. ("Awaited" describes the
driver, not the server — see the build-feature note below for what the server
does during it.) Later boots are free —
`createIndex` is idempotent for an index that already exists, and this driver
already relies on that.

⚠️ **The builds are SERIALIZED, so the times ADD.** `syncCollectionSchema` awaits
`createIndex` once per index in a sequential loop, and `syncSchemasBatch` awaits
`syncSchema` once per object the same way. Startup is extended by the SUM of
every build, not by the slowest one. This is the figure to plan the maintenance
window around.

**How much.** Measured on this tree: **65 lookup fields carrying `reference`
across the 52 exported platform objects** — every one gains an index, and this is
the floor, not the total. Add the objects of any enabled plugin (`plugin-security`
14 lookup fields, `plugin-approvals` 13, `plugin-audit` 6, `plugin-sharing` 4) and
one index per lookup field on your own authored objects.

Per index, as an order-of-magnitude planning figure and **not** a benchmark — get
your own numbers before sizing a window, because they depend entirely on document
count, storage and cache:

| collection size | build time, one index | index storage added |
|---|---|---|
| empty / a few thousand docs | effectively instant (metadata-only) | negligible |
| ~1M docs | seconds | tens of MB |
| ~10M docs | ~a minute | hundreds of MB |
| ~100M docs | tens of minutes | a few GB |

Most `sys_*` collections are small and will finish instantly. Budget for the ones
that accumulate: `sys_metadata_history`, `sys_metadata_audit`, `sys_notification`,
`sys_email`, `sys_session`, and any audit-log object. Count them first
(`db.<collection>.estimatedDocumentCount()`) and multiply by the lookup fields
each carries; watch a live boot with `db.currentOp()`.

**Build feature — hybrid builds, unconditionally.** `@objectstack/driver-mongodb`
depends on `mongodb@^7.5.0`, whose own compatibility statement is "the driver
currently supports 4.2+ servers". MongoDB 4.2 is exactly the release that made
index builds hybrid, so **every server version this driver can connect to builds
these indexes with the hybrid builder**: an exclusive lock is taken only briefly
at the start and end of each build, and the collection accepts reads AND writes
throughout the rest of it. This is not a full write stall. Two caveats that
remain: the brief exclusive lock at each end is real, and on a replica set the
build runs on every member.

**To take the cost outside the boot window,** create the indexes ahead of the
upgrade, with the names and specs the driver uses — `{ name: 'idx_FIELD_lookup' }`
over `{ FIELD: 1 }`, no `unique` and no `sparse`. The driver's `createIndex` is
then a no-op and startup is unaffected. Matching the options matters: a
pre-existing index of the same name with different options raises
`IndexOptionsConflict`, which this driver deliberately swallows and skips, leaving
your index in place unchanged.

37 changes: 24 additions & 13 deletions
content/docs/protocol/objectql/types.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -731,23 +731,34 @@ contacts:

<Callout type="warn">
A relationship field authored with `reference:` gets **no database-level
`FOREIGN KEY` constraint** and **no MongoDB join index**. Both gaps have one
root cause: the driver branches that would have built them were gated on
`reference_to` — a key the spec REFUSES (`FieldSchema` answers
`unrecognized_keys` for it, on any field type) and one that `reference` never
populates. `master_detail` / `tree` did not reach either branch at all.
`FOREIGN KEY` constraint** on any driver. It does get a **MongoDB join index**.
Both branches were once gated on `reference_to` — a key the spec REFUSES
(`FieldSchema` answers `unrecognized_keys` for it, on any field type) and one
that `reference` never populates — so neither fired for authored metadata; the
two were then settled in opposite directions, on their own merits.

- **SQL:** the `FOREIGN KEY` DDL is retired (#11567). A field that still carries
 `reference_to` when it reaches DDL is refused at the driver's door, in the
 schema's own words (`400 VALIDATION_ERROR`), instead of silently changing the
 physical schema.
- **MongoDB:** the field-level join index `idx_FIELD_lookup` is gated on that
 same refused key, so a canonically-spelled `reference` lookup is **not
 indexed**. A `user` field still is — that arm needs no relationship key, which
 is why the feature looked healthy. `reference_to` is refused at this driver's
 door too, with the same verdict (#13222); whether a canonical `reference`
 lookup should start building the index is tracked separately, because it
 changes boot behaviour for deployments that already hold large collections.
 physical schema. Declare an index in `indexes[]` if you want one on the
 foreign-key column.
- **MongoDB:** a `lookup` field declaring `reference:` gets the field-level join
 index `idx_FIELD_lookup` (#13222). A `user` field gets one too — that arm
 needs no relationship key. `reference_to` is refused at this driver's door as
 well, with the same verdict as the spec's.

 ⚠️ **Upgrading an existing MongoDB deployment:** this index is newer than the
 driver, so the first `syncSchema` after the upgrade BUILDS it across
 collections that already hold data — a real one-off IO and time cost on large
 collections, and index storage that persists. The driver's `CHANGELOG.md`
 entry for the release that added it carries the operational detail. Later
 boots are no-ops: `createIndex` is idempotent for an index that already
 exists.

A `lookup` that omits `reference:` gets no join index — there is no declared
target to join to. `master_detail` and `tree` reach neither branch: they get no
`FOREIGN KEY` and no join index. Declare an `indexes[]` entry if a
`master_detail` child is queried by parent often enough to need one.

Referential integrity is enforced by the **engine** instead: `deleteBehavior` is
applied on delete, which is what produces the `409 DELETE_RESTRICTED` above.
Expand Down

14 changes: 10 additions & 4 deletions
packages/drivers/driver-mongodb/README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -163,9 +163,14 @@ object's `indexes[]` — the one surface an index is declared on (a field-level
`indexed` flag is not a `FieldSchema` key and never built an index,
#2377 / #6810).

⚠️ Lookup fields are **not** indexed today. The lookup arm gates on
`reference_to`, a spelling `FieldSchema` refuses, so a canonically-spelled
`reference` lookup gets no join index — see #13222, which owns that fix.
A `lookup` field that declares `reference` also indexes itself, as
`idx_FIELD_lookup`, for join performance; so does a `user` field. A `lookup`
with no `reference` does not — there is no declared target to join to.

⚠️ **Upgrading:** this join index is newer than the driver. The first
`syncSchema` after the upgrade builds it across collections that already hold
data, which costs real IO and time on large ones. See `CHANGELOG.md` for the
operational detail. Later boots are no-ops.

```typescript
await driver.syncSchema('account', {
Expand All@@ -177,7 +182,8 @@ await driver.syncSchema('account', {
 },
 indexes: [{ fields: ['email'] }],
});
// Creates: idx_id_unique, idx_name_unique, idx_email
// Creates: idx_id_unique, idx_created_at, idx_updated_at,
// idx_name_unique, idx_company_id_lookup, idx_email
```

### Aggregation
Expand Down

89 changes: 61 additions & 28 deletions
packages/drivers/driver-mongodb/src/mongodb-13222-reference-to-refusal.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,13 +19,14 @@
// module between two suites that pin OPPOSITE halves of one arm would couple
// them for no gain.
//
// ⛔ NOT pinned here: whether a canonically-spelled `reference` lookup should
// GET `idx_FIELD_lookup`. That is part (2) of #13222 — a separate, still-open
// ruling (it is a boot-time behaviour change for existing deployments: index
// builds on large collections). The last case below is this PR's own NO-CHANGE
// control for it, and is expected to flip in the PR that takes part (2),
// alongside `mongodb-schema-declared-indexes.test.ts`'s #12252 pin, which owns
// the fact.
// Part (2) of #13222 has since been ruled and taken: the join-index arm now
// gates on the canonical `reference`, so a lookup an author can really publish
// GETS `idx_FIELD_lookup`. The last case below was this file's NO-CHANGE
// control for that question and has flipped accordingly, in the same stroke as
// `mongodb-schema-declared-indexes.test.ts`'s #12252 pin and the real-server
// pin in `mongodb-driver.test.ts`. It still belongs here: it is what proves the
// door and the arm read DIFFERENT keys and disagree about them on purpose —
// `reference_to` refused, `reference` indexed, from one call.

import { describe, it, expect } from 'vitest';
import type { Db } from 'mongodb';
Expand DownExpand Up@@ -187,40 +188,72 @@ describe('#13222 part (1) — driver-mongodb refuses `reference_to` at the schem
 expect(names(created)).toEqual(['idx_id_unique', 'idx_created_at', 'idx_updated_at']);
 });

 it('leaves the join-index arm exactly as it was — part (2) is NOT taken here', async () => {
 // ⚠️ THE NO-CHANGE CONTROL for this PR, and load-bearing in both directions.
 it('refuses `reference_to` and INDEXES `reference` — two keys, two answers, one call', async () => {
 // ⚠️ THE PAIRED CONTROL for the door, and load-bearing in both directions.
 //
 // Positive half: a `user` field still gets `idx_owner_id_lookup`, which
 // proves the arm still executes and that the harness is wired to something —
 // without it the negative half below would pass just as happily against a
 // function that created no indexes at all.
 // The door refuses one relationship spelling; the arm indexes the other. A
 // suite that only ever proves the refusal cannot tell "the door works" from
 // "syncCollectionSchema throws on everything", so the two answers are taken
 // from ONE call here on purpose.
 //
 // Negative half: a canonically-spelled `reference` lookup still gets NO join
 // index. That is the divergence #12252 pinned and part (2) of #13222 owns.
 // ⛔ This case records what the driver DOES, not what it should do: the door
 // added in this PR makes the arm's `field.reference_to` conjunct unreachable
 // but deliberately does not delete it, because deleting it would start
 // building indexes on existing deployments' large collections — an unruled
 // behaviour change. When part (2) lands, this case is expected to flip to
 // `toContain`, in the same stroke as the #12252 pin in
 // `mongodb-schema-declared-indexes.test.ts`.
 // `user` half: still `idx_owner_id_lookup`, and still the unconditional
 // disjunct — it proves the arm executed and names the shape the assertion
 // below is spelled in, so neither half can pass vacuously.
 //
 // Bound through a variable rather than written inline: the driver's own
 // `FieldDef` declares no `reference` key, so a fresh object literal carrying
 // it trips TypeScript's excess-property check.
const canonicalLookup = { type: 'lookup', reference: 'company' };

 // `reference` half: FLIPPED by part (2) of #13222. This case used to assert
 // the canonical lookup got NO index — the divergence #12252 pinned, which
 // held because the arm gated on `reference_to`, a key this very door
// refuses, making the conjunct unreachable for every input. The arm now
 // gates on `reference`, so the lookup is indexed.
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: canonicalLookup, owner_id: { type: 'user' } },
 fields: {
 company_id: { type: 'lookup', reference: 'company' },
 owner_id: { type: 'user' },
 },
 });

 // Exact set, in creation order — closes the vacuity routes a `toContain`
 // pair leaves open: an index appearing under another name, the two lookup
 // indexes swapping fields, or a stray fourth index nobody declared.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_company_id_lookup',
 'idx_owner_id_lookup',
 ]);
 });

 it('does not index a `lookup` that declares no target — `reference` is read for truth, not presence', async () => {
 // Measured on `FieldSchema` built from this tree: `{ type: 'lookup' }` with
 // no `reference`, and `{ type: 'lookup', reference: '' }`, BOTH parse
 // successfully — the spec's prose calls `reference` required for these
 // types, but the schema does not enforce it. So this is a shape an author
 // can really publish, not a hypothetical, and the arm has to answer for it.
 //
 // It answers by declining: `idx_FIELD_lookup` exists to serve a join, and a
 // lookup with no declared target has no join to serve — the index would
 // cost every write and buy no read. This is why the arm gates on
 // truthiness and not on `!== undefined` like the door above does; the two
 // predicates differ deliberately, because they are answering different
 // questions about different keys.
 for (const target of [undefined, '']) {
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: { type: 'lookup', reference: target }, owner_id: { type: 'user' } },
 });

 // The `user` control fires, so the arm ran and the zero below is a real
 // zero rather than a harness that called nothing.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_owner_id_lookup',
 ]);
 }
 });
});

42 changes: 21 additions & 21 deletions
packages/drivers/driver-mongodb/src/mongodb-driver.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -362,31 +362,31 @@ describe.skipIf(!sharedMongod)('MongoDBDriver', () => {
 expect(indexNames).toContain('idx_name_unique');
 expect(indexNames).toContain('idx_email');
 /**
 * ⚠️ [#12252] DIVERGENCE PINNED, DISPOSITION OPEN (#13222) — a
 * canonically-spelled lookup gets NO join index here.
 * ⚠️ DIVERGENCE RETIRED (#13222) — a canonically-spelled lookup gets its
 * join index here, verified against a REAL server rather than a recorder.
 *
 * This fixture used to spell the field `reference_to: 'company'` and
 * assert `idx_company_id_lookup` was CREATED. `reference_to` is a key
 * `FieldSchema` REFUSES (`unrecognized_keys`), so the object it described
 * was one no author could publish — and the assertion passed only because
 * this fixture was the sole thing in the tree reaching the lookup arm of
 * `mongodb-schema.ts`, which gates on `field.reference_to` and reads no
 * other relationship key.
 * This fixture once spelled the field `reference_to: 'company'`. That is
 * a key `FieldSchema` REFUSES (`unrecognized_keys`), so the object it
 * described was one no author could publish — and the assertion passed
 * only because this fixture was the sole thing in the tree reaching the
 * lookup arm of `mongodb-schema.ts`, which gated on `field.reference_to`
 * and read no other relationship key. Correcting the spelling (#13224)
 * therefore did not leave the outcome alone: the index disappeared, and
 * the line was inverted to record that, with the disposition left open.
 *
 * So correcting the spelling does not leave the outcome alone. Measured
 * differentially against the real `syncCollectionSchema`,
 * `idx_company_id_lookup` is the ONE index that disappears; a
 * `type: 'user'` field still gets its index, so the gate is live rather
 * than dead code. The consequence in production is that EVERY authored
 * lookup on MongoDB is unindexed — 57/57 relationship fields across the
 * 44 exported platform objects (#13222).
 * #13222 settled the disposition in two parts: `reference_to` is now
 * refused at the driver's door, and the arm gates on the canonical
 * `reference`. So the line is inverted back — by a ruling, not by a
 * re-baseline.
 *
 * ⛔ This records what the driver DOES, not what it SHOULD do. Whether
 * the lookup arm learns to read `reference` is #13222's to settle, ⛔ not
 * this pin's — when it lands, this line flips back to `toContain`
 * deliberately rather than the divergence reopening in silence.
 * ⚠️ This copy is the one that runs against real MongoDB, which is what
 * it is FOR: it proves the server actually materialized the index under
 * that name, not merely that the driver asked for it. The recorder-driven
 * twin in `mongodb-schema-declared-indexes.test.ts` is the copy that runs
 * on every ordinary CI lane — this suite is `describe.skipIf(!sharedMongod)`.
 * TWIN PIN — edit either, edit both.
 */
 expect(indexNames).not.toContain('idx_company_id_lookup');
 expect(indexNames).toContain('idx_company_id_lookup');
 });

 it('should be idempotent (safe to call multiple times)', async () => {
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
feat(driver-mongodb): index lookup joins off the canonical `reference` key by claude[bot] · Pull Request #13633 · objectstack-ai/objectstack · GitHub
Skip to content
feat(driver-mongodb): index lookup joins off the canonical reference key#13633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

zhuangjianguo
merged 1 commit into

from

Aug 31, 2026

+298

−129

 Merged


feat(driver-mongodb): index lookup joins off the canonical reference key
#13633
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


90 changes: 90 additions & 0 deletions
.changeset/mongodb-lookup-index-canonical-reference.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
---
"@objectstack/driver-mongodb": minor
---

feat(driver-mongodb): index `lookup` joins off the canonical `reference` key (#13222)

`syncCollectionSchema`'s field-level join-index arm gated on `field.reference_to`.
That is a REJECTED ALIAS — `FieldSchema` answers `unrecognized_keys` for it on any
field type — and this driver's own schema door refuses it outright. So the arm's
`lookup` conjunct could not be satisfied by any input at all, and no authored
lookup had ever been indexed on MongoDB. The arm now reads `reference`, the only
relationship spelling the spec declares.

Measured as a complete case split over the key's value domain rather than a
sample: every `reference_to` value except `undefined` is refused at the door, and
`undefined` is falsy, so the old conjunct was unreachable for every possible
input. The `user` disjunct was unaffected and is why the feature looked healthy —
it needs no relationship key, so `idx_owner_lookup`-shaped indexes were always
created.

The refusal door itself is unchanged: predicate, `VALIDATION_ERROR`/400 envelope,
placement and instruction are all as they were. Only the tail of its message
moved, because it told the reader that renaming the key would not by itself get
the field an index — true when written, false now.

A `lookup` that declares no `reference` is still not indexed. Measured on
`FieldSchema`: `{ type: 'lookup' }` and `{ type: 'lookup', reference: '' }` both
parse successfully — the spec's prose calls `reference` required for these types
but the schema does not enforce it — so this is a real authorable shape, and an
index for a join with no declared target would cost every write and buy no read.
`master_detail` and `tree` are unchanged: they reach this arm on neither spelling.

## ⚠️ OPERATORS — this changes boot behaviour on existing deployments

**What it costs.** The first `syncSchema` after this upgrade CREATES these
indexes on collections that already hold data. Each one is an awaited
`createIndex`: a full collection scan plus an external sort of the extracted
keys, followed by permanent index storage and a small write amplification on
every subsequent insert/update of the indexed field. ("Awaited" describes the
driver, not the server — see the build-feature note below for what the server
does during it.) Later boots are free —
`createIndex` is idempotent for an index that already exists, and this driver
already relies on that.

⚠️ **The builds are SERIALIZED, so the times ADD.** `syncCollectionSchema` awaits
`createIndex` once per index in a sequential loop, and `syncSchemasBatch` awaits
`syncSchema` once per object the same way. Startup is extended by the SUM of
every build, not by the slowest one. This is the figure to plan the maintenance
window around.

**How much.** Measured on this tree: **65 lookup fields carrying `reference`
across the 52 exported platform objects** — every one gains an index, and this is
the floor, not the total. Add the objects of any enabled plugin (`plugin-security`
14 lookup fields, `plugin-approvals` 13, `plugin-audit` 6, `plugin-sharing` 4) and
one index per lookup field on your own authored objects.

Per index, as an order-of-magnitude planning figure and **not** a benchmark — get
your own numbers before sizing a window, because they depend entirely on document
count, storage and cache:

| collection size | build time, one index | index storage added |
|---|---|---|
| empty / a few thousand docs | effectively instant (metadata-only) | negligible |
| ~1M docs | seconds | tens of MB |
| ~10M docs | ~a minute | hundreds of MB |
| ~100M docs | tens of minutes | a few GB |

Most `sys_*` collections are small and will finish instantly. Budget for the ones
that accumulate: `sys_metadata_history`, `sys_metadata_audit`, `sys_notification`,
`sys_email`, `sys_session`, and any audit-log object. Count them first
(`db.<collection>.estimatedDocumentCount()`) and multiply by the lookup fields
each carries; watch a live boot with `db.currentOp()`.

**Build feature — hybrid builds, unconditionally.** `@objectstack/driver-mongodb`
depends on `mongodb@^7.5.0`, whose own compatibility statement is "the driver
currently supports 4.2+ servers". MongoDB 4.2 is exactly the release that made
index builds hybrid, so **every server version this driver can connect to builds
these indexes with the hybrid builder**: an exclusive lock is taken only briefly
at the start and end of each build, and the collection accepts reads AND writes
throughout the rest of it. This is not a full write stall. Two caveats that
remain: the brief exclusive lock at each end is real, and on a replica set the
build runs on every member.

**To take the cost outside the boot window,** create the indexes ahead of the
upgrade, with the names and specs the driver uses — `{ name: 'idx_FIELD_lookup' }`
over `{ FIELD: 1 }`, no `unique` and no `sparse`. The driver's `createIndex` is
then a no-op and startup is unaffected. Matching the options matters: a
pre-existing index of the same name with different options raises
`IndexOptionsConflict`, which this driver deliberately swallows and skips, leaving
your index in place unchanged.

37 changes: 24 additions & 13 deletions
content/docs/protocol/objectql/types.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -731,23 +731,34 @@ contacts:

<Callout type="warn">
A relationship field authored with `reference:` gets **no database-level
`FOREIGN KEY` constraint** and **no MongoDB join index**. Both gaps have one
root cause: the driver branches that would have built them were gated on
`reference_to` — a key the spec REFUSES (`FieldSchema` answers
`unrecognized_keys` for it, on any field type) and one that `reference` never
populates. `master_detail` / `tree` did not reach either branch at all.
`FOREIGN KEY` constraint** on any driver. It does get a **MongoDB join index**.
Both branches were once gated on `reference_to` — a key the spec REFUSES
(`FieldSchema` answers `unrecognized_keys` for it, on any field type) and one
that `reference` never populates — so neither fired for authored metadata; the
two were then settled in opposite directions, on their own merits.

- **SQL:** the `FOREIGN KEY` DDL is retired (#11567). A field that still carries
 `reference_to` when it reaches DDL is refused at the driver's door, in the
 schema's own words (`400 VALIDATION_ERROR`), instead of silently changing the
 physical schema.
- **MongoDB:** the field-level join index `idx_FIELD_lookup` is gated on that
 same refused key, so a canonically-spelled `reference` lookup is **not
 indexed**. A `user` field still is — that arm needs no relationship key, which
 is why the feature looked healthy. `reference_to` is refused at this driver's
 door too, with the same verdict (#13222); whether a canonical `reference`
 lookup should start building the index is tracked separately, because it
 changes boot behaviour for deployments that already hold large collections.
 physical schema. Declare an index in `indexes[]` if you want one on the
 foreign-key column.
- **MongoDB:** a `lookup` field declaring `reference:` gets the field-level join
 index `idx_FIELD_lookup` (#13222). A `user` field gets one too — that arm
 needs no relationship key. `reference_to` is refused at this driver's door as
 well, with the same verdict as the spec's.

 ⚠️ **Upgrading an existing MongoDB deployment:** this index is newer than the
 driver, so the first `syncSchema` after the upgrade BUILDS it across
 collections that already hold data — a real one-off IO and time cost on large
 collections, and index storage that persists. The driver's `CHANGELOG.md`
 entry for the release that added it carries the operational detail. Later
 boots are no-ops: `createIndex` is idempotent for an index that already
 exists.

A `lookup` that omits `reference:` gets no join index — there is no declared
target to join to. `master_detail` and `tree` reach neither branch: they get no
`FOREIGN KEY` and no join index. Declare an `indexes[]` entry if a
`master_detail` child is queried by parent often enough to need one.

Referential integrity is enforced by the **engine** instead: `deleteBehavior` is
applied on delete, which is what produces the `409 DELETE_RESTRICTED` above.
Expand Down

14 changes: 10 additions & 4 deletions
packages/drivers/driver-mongodb/README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -163,9 +163,14 @@ object's `indexes[]` — the one surface an index is declared on (a field-level
`indexed` flag is not a `FieldSchema` key and never built an index,
#2377 / #6810).

⚠️ Lookup fields are **not** indexed today. The lookup arm gates on
`reference_to`, a spelling `FieldSchema` refuses, so a canonically-spelled
`reference` lookup gets no join index — see #13222, which owns that fix.
A `lookup` field that declares `reference` also indexes itself, as
`idx_FIELD_lookup`, for join performance; so does a `user` field. A `lookup`
with no `reference` does not — there is no declared target to join to.

⚠️ **Upgrading:** this join index is newer than the driver. The first
`syncSchema` after the upgrade builds it across collections that already hold
data, which costs real IO and time on large ones. See `CHANGELOG.md` for the
operational detail. Later boots are no-ops.

```typescript
await driver.syncSchema('account', {
Expand All@@ -177,7 +182,8 @@ await driver.syncSchema('account', {
 },
 indexes: [{ fields: ['email'] }],
});
// Creates: idx_id_unique, idx_name_unique, idx_email
// Creates: idx_id_unique, idx_created_at, idx_updated_at,
// idx_name_unique, idx_company_id_lookup, idx_email
```

### Aggregation
Expand Down

89 changes: 61 additions & 28 deletions
packages/drivers/driver-mongodb/src/mongodb-13222-reference-to-refusal.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,13 +19,14 @@
// module between two suites that pin OPPOSITE halves of one arm would couple
// them for no gain.
//
// ⛔ NOT pinned here: whether a canonically-spelled `reference` lookup should
// GET `idx_FIELD_lookup`. That is part (2) of #13222 — a separate, still-open
// ruling (it is a boot-time behaviour change for existing deployments: index
// builds on large collections). The last case below is this PR's own NO-CHANGE
// control for it, and is expected to flip in the PR that takes part (2),
// alongside `mongodb-schema-declared-indexes.test.ts`'s #12252 pin, which owns
// the fact.
// Part (2) of #13222 has since been ruled and taken: the join-index arm now
// gates on the canonical `reference`, so a lookup an author can really publish
// GETS `idx_FIELD_lookup`. The last case below was this file's NO-CHANGE
// control for that question and has flipped accordingly, in the same stroke as
// `mongodb-schema-declared-indexes.test.ts`'s #12252 pin and the real-server
// pin in `mongodb-driver.test.ts`. It still belongs here: it is what proves the
// door and the arm read DIFFERENT keys and disagree about them on purpose —
// `reference_to` refused, `reference` indexed, from one call.

import { describe, it, expect } from 'vitest';
import type { Db } from 'mongodb';
Expand DownExpand Up@@ -187,40 +188,72 @@ describe('#13222 part (1) — driver-mongodb refuses `reference_to` at the schem
 expect(names(created)).toEqual(['idx_id_unique', 'idx_created_at', 'idx_updated_at']);
 });

 it('leaves the join-index arm exactly as it was — part (2) is NOT taken here', async () => {
 // ⚠️ THE NO-CHANGE CONTROL for this PR, and load-bearing in both directions.
 it('refuses `reference_to` and INDEXES `reference` — two keys, two answers, one call', async () => {
 // ⚠️ THE PAIRED CONTROL for the door, and load-bearing in both directions.
 //
 // Positive half: a `user` field still gets `idx_owner_id_lookup`, which
 // proves the arm still executes and that the harness is wired to something —
 // without it the negative half below would pass just as happily against a
 // function that created no indexes at all.
 // The door refuses one relationship spelling; the arm indexes the other. A
 // suite that only ever proves the refusal cannot tell "the door works" from
 // "syncCollectionSchema throws on everything", so the two answers are taken
 // from ONE call here on purpose.
 //
 // Negative half: a canonically-spelled `reference` lookup still gets NO join
 // index. That is the divergence #12252 pinned and part (2) of #13222 owns.
 // ⛔ This case records what the driver DOES, not what it should do: the door
 // added in this PR makes the arm's `field.reference_to` conjunct unreachable
 // but deliberately does not delete it, because deleting it would start
 // building indexes on existing deployments' large collections — an unruled
 // behaviour change. When part (2) lands, this case is expected to flip to
 // `toContain`, in the same stroke as the #12252 pin in
 // `mongodb-schema-declared-indexes.test.ts`.
 // `user` half: still `idx_owner_id_lookup`, and still the unconditional
 // disjunct — it proves the arm executed and names the shape the assertion
 // below is spelled in, so neither half can pass vacuously.
 //
 // Bound through a variable rather than written inline: the driver's own
 // `FieldDef` declares no `reference` key, so a fresh object literal carrying
 // it trips TypeScript's excess-property check.
const canonicalLookup = { type: 'lookup', reference: 'company' };

 // `reference` half: FLIPPED by part (2) of #13222. This case used to assert
 // the canonical lookup got NO index — the divergence #12252 pinned, which
 // held because the arm gated on `reference_to`, a key this very door
// refuses, making the conjunct unreachable for every input. The arm now
 // gates on `reference`, so the lookup is indexed.
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: canonicalLookup, owner_id: { type: 'user' } },
 fields: {
 company_id: { type: 'lookup', reference: 'company' },
 owner_id: { type: 'user' },
 },
 });

 // Exact set, in creation order — closes the vacuity routes a `toContain`
 // pair leaves open: an index appearing under another name, the two lookup
 // indexes swapping fields, or a stray fourth index nobody declared.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_company_id_lookup',
 'idx_owner_id_lookup',
 ]);
 });

 it('does not index a `lookup` that declares no target — `reference` is read for truth, not presence', async () => {
 // Measured on `FieldSchema` built from this tree: `{ type: 'lookup' }` with
 // no `reference`, and `{ type: 'lookup', reference: '' }`, BOTH parse
 // successfully — the spec's prose calls `reference` required for these
 // types, but the schema does not enforce it. So this is a shape an author
 // can really publish, not a hypothetical, and the arm has to answer for it.
 //
 // It answers by declining: `idx_FIELD_lookup` exists to serve a join, and a
 // lookup with no declared target has no join to serve — the index would
 // cost every write and buy no read. This is why the arm gates on
 // truthiness and not on `!== undefined` like the door above does; the two
 // predicates differ deliberately, because they are answering different
 // questions about different keys.
 for (const target of [undefined, '']) {
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: { type: 'lookup', reference: target }, owner_id: { type: 'user' } },
 });

 // The `user` control fires, so the arm ran and the zero below is a real
 // zero rather than a harness that called nothing.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_owner_id_lookup',
 ]);
 }
 });
});

42 changes: 21 additions & 21 deletions
packages/drivers/driver-mongodb/src/mongodb-driver.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -362,31 +362,31 @@ describe.skipIf(!sharedMongod)('MongoDBDriver', () => {
 expect(indexNames).toContain('idx_name_unique');
 expect(indexNames).toContain('idx_email');
 /**
 * ⚠️ [#12252] DIVERGENCE PINNED, DISPOSITION OPEN (#13222) — a
 * canonically-spelled lookup gets NO join index here.
 * ⚠️ DIVERGENCE RETIRED (#13222) — a canonically-spelled lookup gets its
 * join index here, verified against a REAL server rather than a recorder.
 *
 * This fixture used to spell the field `reference_to: 'company'` and
 * assert `idx_company_id_lookup` was CREATED. `reference_to` is a key
 * `FieldSchema` REFUSES (`unrecognized_keys`), so the object it described
 * was one no author could publish — and the assertion passed only because
 * this fixture was the sole thing in the tree reaching the lookup arm of
 * `mongodb-schema.ts`, which gates on `field.reference_to` and reads no
 * other relationship key.
 * This fixture once spelled the field `reference_to: 'company'`. That is
 * a key `FieldSchema` REFUSES (`unrecognized_keys`), so the object it
 * described was one no author could publish — and the assertion passed
 * only because this fixture was the sole thing in the tree reaching the
 * lookup arm of `mongodb-schema.ts`, which gated on `field.reference_to`
 * and read no other relationship key. Correcting the spelling (#13224)
 * therefore did not leave the outcome alone: the index disappeared, and
 * the line was inverted to record that, with the disposition left open.
 *
 * So correcting the spelling does not leave the outcome alone. Measured
 * differentially against the real `syncCollectionSchema`,
 * `idx_company_id_lookup` is the ONE index that disappears; a
 * `type: 'user'` field still gets its index, so the gate is live rather
 * than dead code. The consequence in production is that EVERY authored
 * lookup on MongoDB is unindexed — 57/57 relationship fields across the
 * 44 exported platform objects (#13222).
 * #13222 settled the disposition in two parts: `reference_to` is now
 * refused at the driver's door, and the arm gates on the canonical
 * `reference`. So the line is inverted back — by a ruling, not by a
 * re-baseline.
 *
 * ⛔ This records what the driver DOES, not what it SHOULD do. Whether
 * the lookup arm learns to read `reference` is #13222's to settle, ⛔ not
 * this pin's — when it lands, this line flips back to `toContain`
 * deliberately rather than the divergence reopening in silence.
 * ⚠️ This copy is the one that runs against real MongoDB, which is what
 * it is FOR: it proves the server actually materialized the index under
 * that name, not merely that the driver asked for it. The recorder-driven
 * twin in `mongodb-schema-declared-indexes.test.ts` is the copy that runs
 * on every ordinary CI lane — this suite is `describe.skipIf(!sharedMongod)`.
 * TWIN PIN — edit either, edit both.
 */
 expect(indexNames).not.toContain('idx_company_id_lookup');
 expect(indexNames).toContain('idx_company_id_lookup');
 });

 it('should be idempotent (safe to call multiple times)', async () => {
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
feat(driver-mongodb): index lookup joins off the canonical `reference` key by claude[bot] · Pull Request #13633 · objectstack-ai/objectstack · GitHub
Skip to content
feat(driver-mongodb): index lookup joins off the canonical reference key#13633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

zhuangjianguo
merged 1 commit into

from

Aug 31, 2026

+298

−129

 Merged


feat(driver-mongodb): index lookup joins off the canonical reference key
#13633
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


90 changes: 90 additions & 0 deletions
.changeset/mongodb-lookup-index-canonical-reference.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
---
"@objectstack/driver-mongodb": minor
---

feat(driver-mongodb): index `lookup` joins off the canonical `reference` key (#13222)

`syncCollectionSchema`'s field-level join-index arm gated on `field.reference_to`.
That is a REJECTED ALIAS — `FieldSchema` answers `unrecognized_keys` for it on any
field type — and this driver's own schema door refuses it outright. So the arm's
`lookup` conjunct could not be satisfied by any input at all, and no authored
lookup had ever been indexed on MongoDB. The arm now reads `reference`, the only
relationship spelling the spec declares.

Measured as a complete case split over the key's value domain rather than a
sample: every `reference_to` value except `undefined` is refused at the door, and
`undefined` is falsy, so the old conjunct was unreachable for every possible
input. The `user` disjunct was unaffected and is why the feature looked healthy —
it needs no relationship key, so `idx_owner_lookup`-shaped indexes were always
created.

The refusal door itself is unchanged: predicate, `VALIDATION_ERROR`/400 envelope,
placement and instruction are all as they were. Only the tail of its message
moved, because it told the reader that renaming the key would not by itself get
the field an index — true when written, false now.

A `lookup` that declares no `reference` is still not indexed. Measured on
`FieldSchema`: `{ type: 'lookup' }` and `{ type: 'lookup', reference: '' }` both
parse successfully — the spec's prose calls `reference` required for these types
but the schema does not enforce it — so this is a real authorable shape, and an
index for a join with no declared target would cost every write and buy no read.
`master_detail` and `tree` are unchanged: they reach this arm on neither spelling.

## ⚠️ OPERATORS — this changes boot behaviour on existing deployments

**What it costs.** The first `syncSchema` after this upgrade CREATES these
indexes on collections that already hold data. Each one is an awaited
`createIndex`: a full collection scan plus an external sort of the extracted
keys, followed by permanent index storage and a small write amplification on
every subsequent insert/update of the indexed field. ("Awaited" describes the
driver, not the server — see the build-feature note below for what the server
does during it.) Later boots are free —
`createIndex` is idempotent for an index that already exists, and this driver
already relies on that.

⚠️ **The builds are SERIALIZED, so the times ADD.** `syncCollectionSchema` awaits
`createIndex` once per index in a sequential loop, and `syncSchemasBatch` awaits
`syncSchema` once per object the same way. Startup is extended by the SUM of
every build, not by the slowest one. This is the figure to plan the maintenance
window around.

**How much.** Measured on this tree: **65 lookup fields carrying `reference`
across the 52 exported platform objects** — every one gains an index, and this is
the floor, not the total. Add the objects of any enabled plugin (`plugin-security`
14 lookup fields, `plugin-approvals` 13, `plugin-audit` 6, `plugin-sharing` 4) and
one index per lookup field on your own authored objects.

Per index, as an order-of-magnitude planning figure and **not** a benchmark — get
your own numbers before sizing a window, because they depend entirely on document
count, storage and cache:

| collection size | build time, one index | index storage added |
|---|---|---|
| empty / a few thousand docs | effectively instant (metadata-only) | negligible |
| ~1M docs | seconds | tens of MB |
| ~10M docs | ~a minute | hundreds of MB |
| ~100M docs | tens of minutes | a few GB |

Most `sys_*` collections are small and will finish instantly. Budget for the ones
that accumulate: `sys_metadata_history`, `sys_metadata_audit`, `sys_notification`,
`sys_email`, `sys_session`, and any audit-log object. Count them first
(`db.<collection>.estimatedDocumentCount()`) and multiply by the lookup fields
each carries; watch a live boot with `db.currentOp()`.

**Build feature — hybrid builds, unconditionally.** `@objectstack/driver-mongodb`
depends on `mongodb@^7.5.0`, whose own compatibility statement is "the driver
currently supports 4.2+ servers". MongoDB 4.2 is exactly the release that made
index builds hybrid, so **every server version this driver can connect to builds
these indexes with the hybrid builder**: an exclusive lock is taken only briefly
at the start and end of each build, and the collection accepts reads AND writes
throughout the rest of it. This is not a full write stall. Two caveats that
remain: the brief exclusive lock at each end is real, and on a replica set the
build runs on every member.

**To take the cost outside the boot window,** create the indexes ahead of the
upgrade, with the names and specs the driver uses — `{ name: 'idx_FIELD_lookup' }`
over `{ FIELD: 1 }`, no `unique` and no `sparse`. The driver's `createIndex` is
then a no-op and startup is unaffected. Matching the options matters: a
pre-existing index of the same name with different options raises
`IndexOptionsConflict`, which this driver deliberately swallows and skips, leaving
your index in place unchanged.

37 changes: 24 additions & 13 deletions
content/docs/protocol/objectql/types.mdx

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -731,23 +731,34 @@ contacts:

<Callout type="warn">
A relationship field authored with `reference:` gets **no database-level
`FOREIGN KEY` constraint** and **no MongoDB join index**. Both gaps have one
root cause: the driver branches that would have built them were gated on
`reference_to` — a key the spec REFUSES (`FieldSchema` answers
`unrecognized_keys` for it, on any field type) and one that `reference` never
populates. `master_detail` / `tree` did not reach either branch at all.
`FOREIGN KEY` constraint** on any driver. It does get a **MongoDB join index**.
Both branches were once gated on `reference_to` — a key the spec REFUSES
(`FieldSchema` answers `unrecognized_keys` for it, on any field type) and one
that `reference` never populates — so neither fired for authored metadata; the
two were then settled in opposite directions, on their own merits.

- **SQL:** the `FOREIGN KEY` DDL is retired (#11567). A field that still carries
 `reference_to` when it reaches DDL is refused at the driver's door, in the
 schema's own words (`400 VALIDATION_ERROR`), instead of silently changing the
 physical schema.
- **MongoDB:** the field-level join index `idx_FIELD_lookup` is gated on that
 same refused key, so a canonically-spelled `reference` lookup is **not
 indexed**. A `user` field still is — that arm needs no relationship key, which
 is why the feature looked healthy. `reference_to` is refused at this driver's
 door too, with the same verdict (#13222); whether a canonical `reference`
 lookup should start building the index is tracked separately, because it
 changes boot behaviour for deployments that already hold large collections.
 physical schema. Declare an index in `indexes[]` if you want one on the
 foreign-key column.
- **MongoDB:** a `lookup` field declaring `reference:` gets the field-level join
 index `idx_FIELD_lookup` (#13222). A `user` field gets one too — that arm
 needs no relationship key. `reference_to` is refused at this driver's door as
 well, with the same verdict as the spec's.

 ⚠️ **Upgrading an existing MongoDB deployment:** this index is newer than the
 driver, so the first `syncSchema` after the upgrade BUILDS it across
 collections that already hold data — a real one-off IO and time cost on large
 collections, and index storage that persists. The driver's `CHANGELOG.md`
 entry for the release that added it carries the operational detail. Later
 boots are no-ops: `createIndex` is idempotent for an index that already
 exists.

A `lookup` that omits `reference:` gets no join index — there is no declared
target to join to. `master_detail` and `tree` reach neither branch: they get no
`FOREIGN KEY` and no join index. Declare an `indexes[]` entry if a
`master_detail` child is queried by parent often enough to need one.

Referential integrity is enforced by the **engine** instead: `deleteBehavior` is
applied on delete, which is what produces the `409 DELETE_RESTRICTED` above.
Expand Down

14 changes: 10 additions & 4 deletions
packages/drivers/driver-mongodb/README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -163,9 +163,14 @@ object's `indexes[]` — the one surface an index is declared on (a field-level
`indexed` flag is not a `FieldSchema` key and never built an index,
#2377 / #6810).

⚠️ Lookup fields are **not** indexed today. The lookup arm gates on
`reference_to`, a spelling `FieldSchema` refuses, so a canonically-spelled
`reference` lookup gets no join index — see #13222, which owns that fix.
A `lookup` field that declares `reference` also indexes itself, as
`idx_FIELD_lookup`, for join performance; so does a `user` field. A `lookup`
with no `reference` does not — there is no declared target to join to.

⚠️ **Upgrading:** this join index is newer than the driver. The first
`syncSchema` after the upgrade builds it across collections that already hold
data, which costs real IO and time on large ones. See `CHANGELOG.md` for the
operational detail. Later boots are no-ops.

```typescript
await driver.syncSchema('account', {
Expand All@@ -177,7 +182,8 @@ await driver.syncSchema('account', {
 },
 indexes: [{ fields: ['email'] }],
});
// Creates: idx_id_unique, idx_name_unique, idx_email
// Creates: idx_id_unique, idx_created_at, idx_updated_at,
// idx_name_unique, idx_company_id_lookup, idx_email
```

### Aggregation
Expand Down

89 changes: 61 additions & 28 deletions
packages/drivers/driver-mongodb/src/mongodb-13222-reference-to-refusal.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,13 +19,14 @@
// module between two suites that pin OPPOSITE halves of one arm would couple
// them for no gain.
//
// ⛔ NOT pinned here: whether a canonically-spelled `reference` lookup should
// GET `idx_FIELD_lookup`. That is part (2) of #13222 — a separate, still-open
// ruling (it is a boot-time behaviour change for existing deployments: index
// builds on large collections). The last case below is this PR's own NO-CHANGE
// control for it, and is expected to flip in the PR that takes part (2),
// alongside `mongodb-schema-declared-indexes.test.ts`'s #12252 pin, which owns
// the fact.
// Part (2) of #13222 has since been ruled and taken: the join-index arm now
// gates on the canonical `reference`, so a lookup an author can really publish
// GETS `idx_FIELD_lookup`. The last case below was this file's NO-CHANGE
// control for that question and has flipped accordingly, in the same stroke as
// `mongodb-schema-declared-indexes.test.ts`'s #12252 pin and the real-server
// pin in `mongodb-driver.test.ts`. It still belongs here: it is what proves the
// door and the arm read DIFFERENT keys and disagree about them on purpose —
// `reference_to` refused, `reference` indexed, from one call.

import { describe, it, expect } from 'vitest';
import type { Db } from 'mongodb';
Expand DownExpand Up@@ -187,40 +188,72 @@ describe('#13222 part (1) — driver-mongodb refuses `reference_to` at the schem
 expect(names(created)).toEqual(['idx_id_unique', 'idx_created_at', 'idx_updated_at']);
 });

 it('leaves the join-index arm exactly as it was — part (2) is NOT taken here', async () => {
 // ⚠️ THE NO-CHANGE CONTROL for this PR, and load-bearing in both directions.
 it('refuses `reference_to` and INDEXES `reference` — two keys, two answers, one call', async () => {
 // ⚠️ THE PAIRED CONTROL for the door, and load-bearing in both directions.
 //
 // Positive half: a `user` field still gets `idx_owner_id_lookup`, which
 // proves the arm still executes and that the harness is wired to something —
 // without it the negative half below would pass just as happily against a
 // function that created no indexes at all.
 // The door refuses one relationship spelling; the arm indexes the other. A
 // suite that only ever proves the refusal cannot tell "the door works" from
 // "syncCollectionSchema throws on everything", so the two answers are taken
 // from ONE call here on purpose.
 //
 // Negative half: a canonically-spelled `reference` lookup still gets NO join
 // index. That is the divergence #12252 pinned and part (2) of #13222 owns.
 // ⛔ This case records what the driver DOES, not what it should do: the door
 // added in this PR makes the arm's `field.reference_to` conjunct unreachable
 // but deliberately does not delete it, because deleting it would start
 // building indexes on existing deployments' large collections — an unruled
 // behaviour change. When part (2) lands, this case is expected to flip to
 // `toContain`, in the same stroke as the #12252 pin in
 // `mongodb-schema-declared-indexes.test.ts`.
 // `user` half: still `idx_owner_id_lookup`, and still the unconditional
 // disjunct — it proves the arm executed and names the shape the assertion
 // below is spelled in, so neither half can pass vacuously.
 //
 // Bound through a variable rather than written inline: the driver's own
 // `FieldDef` declares no `reference` key, so a fresh object literal carrying
 // it trips TypeScript's excess-property check.
const canonicalLookup = { type: 'lookup', reference: 'company' };

 // `reference` half: FLIPPED by part (2) of #13222. This case used to assert
 // the canonical lookup got NO index — the divergence #12252 pinned, which
 // held because the arm gated on `reference_to`, a key this very door
// refuses, making the conjunct unreachable for every input. The arm now
 // gates on `reference`, so the lookup is indexed.
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: canonicalLookup, owner_id: { type: 'user' } },
 fields: {
 company_id: { type: 'lookup', reference: 'company' },
 owner_id: { type: 'user' },
 },
 });

 // Exact set, in creation order — closes the vacuity routes a `toContain`
 // pair leaves open: an index appearing under another name, the two lookup
 // indexes swapping fields, or a stray fourth index nobody declared.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_company_id_lookup',
 'idx_owner_id_lookup',
 ]);
 });

 it('does not index a `lookup` that declares no target — `reference` is read for truth, not presence', async () => {
 // Measured on `FieldSchema` built from this tree: `{ type: 'lookup' }` with
 // no `reference`, and `{ type: 'lookup', reference: '' }`, BOTH parse
 // successfully — the spec's prose calls `reference` required for these
 // types, but the schema does not enforce it. So this is a shape an author
 // can really publish, not a hypothetical, and the arm has to answer for it.
 //
 // It answers by declining: `idx_FIELD_lookup` exists to serve a join, and a
 // lookup with no declared target has no join to serve — the index would
 // cost every write and buy no read. This is why the arm gates on
 // truthiness and not on `!== undefined` like the door above does; the two
 // predicates differ deliberately, because they are answering different
 // questions about different keys.
 for (const target of [undefined, '']) {
 const { db, created } = fakeDb();
 await syncCollectionSchema(db, 'lead', {
 name: 'lead',
 fields: { company_id: { type: 'lookup', reference: target }, owner_id: { type: 'user' } },
 });

 // The `user` control fires, so the arm ran and the zero below is a real
 // zero rather than a harness that called nothing.
 expect(names(created)).toEqual([
 'idx_id_unique',
 'idx_created_at',
 'idx_updated_at',
 'idx_owner_id_lookup',
 ]);
 }
 });
});

42 changes: 21 additions & 21 deletions
packages/drivers/driver-mongodb/src/mongodb-driver.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -362,31 +362,31 @@ describe.skipIf(!sharedMongod)('MongoDBDriver', () => {
 expect(indexNames).toContain('idx_name_unique');
 expect(indexNames).toContain('idx_email');
 /**
 * ⚠️ [#12252] DIVERGENCE PINNED, DISPOSITION OPEN (#13222) — a
 * canonically-spelled lookup gets NO join index here.
 * ⚠️ DIVERGENCE RETIRED (#13222) — a canonically-spelled lookup gets its
 * join index here, verified against a REAL server rather than a recorder.
 *
 * This fixture used to spell the field `reference_to: 'company'` and
 * assert `idx_company_id_lookup` was CREATED. `reference_to` is a key
 * `FieldSchema` REFUSES (`unrecognized_keys`), so the object it described
 * was one no author could publish — and the assertion passed only because
 * this fixture was the sole thing in the tree reaching the lookup arm of
 * `mongodb-schema.ts`, which gates on `field.reference_to` and reads no
 * other relationship key.
 * This fixture once spelled the field `reference_to: 'company'`. That is
 * a key `FieldSchema` REFUSES (`unrecognized_keys`), so the object it
 * described was one no author could publish — and the assertion passed
 * only because this fixture was the sole thing in the tree reaching the
 * lookup arm of `mongodb-schema.ts`, which gated on `field.reference_to`
 * and read no other relationship key. Correcting the spelling (#13224)
 * therefore did not leave the outcome alone: the index disappeared, and
 * the line was inverted to record that, with the disposition left open.
 *
 * So correcting the spelling does not leave the outcome alone. Measured
 * differentially against the real `syncCollectionSchema`,
 * `idx_company_id_lookup` is the ONE index that disappears; a
 * `type: 'user'` field still gets its index, so the gate is live rather
 * than dead code. The consequence in production is that EVERY authored
 * lookup on MongoDB is unindexed — 57/57 relationship fields across the
 * 44 exported platform objects (#13222).
 * #13222 settled the disposition in two parts: `reference_to` is now
 * refused at the driver's door, and the arm gates on the canonical
 * `reference`. So the line is inverted back — by a ruling, not by a
 * re-baseline.
 *
 * ⛔ This records what the driver DOES, not what it SHOULD do. Whether
 * the lookup arm learns to read `reference` is #13222's to settle, ⛔ not
 * this pin's — when it lands, this line flips back to `toContain`
 * deliberately rather than the divergence reopening in silence.
 * ⚠️ This copy is the one that runs against real MongoDB, which is what
 * it is FOR: it proves the server actually materialized the index under
 * that name, not merely that the driver asked for it. The recorder-driven
 * twin in `mongodb-schema-declared-indexes.test.ts` is the copy that runs
 * on every ordinary CI lane — this suite is `describe.skipIf(!sharedMongod)`.
 * TWIN PIN — edit either, edit both.
 */
 expect(indexNames).not.toContain('idx_company_id_lookup');
 expect(indexNames).toContain('idx_company_id_lookup');
 });

 it('should be idempotent (safe to call multiple times)', async () => {
Expand Down
Loading
Loading