diff --git a/.changeset/exists-has-value-three-exits.md b/.changeset/exists-has-value-three-exits.md new file mode 100644 index 0000000000..50788f7fd2 --- /dev/null +++ b/.changeset/exists-has-value-three-exits.md @@ -0,0 +1,66 @@ +--- +"@objectstack/driver-memory": minor +"@objectstack/driver-mongodb": minor +--- + +fix(drivers): `$exists` means HAS A VALUE on the live mingo path, the analytics face and `translateFilter` (#13195) + +The platform's settled semantic is that `$exists` means "the field has a value" +(`!= null`), never key-presence — #5298 leg ③ / #5369, landed in PR #5962. Three +exits still read key-presence; the maintainer ruled on 2026-08-30 that all three +align. They now do: + +- `driver-memory`'s **live mingo query path** (`InMemoryDriver.find()`) — the + operator went to mingo under its own name, and mingo tests key presence; +- `driver-memory`'s **analytics execution face** — it built its own + `{$exists: }`, so it inherited key-presence independently; +- `driver-mongodb`'s **`translateFilter`** — it passed the operator through, and + MongoDB's `$exists` is key-presence at the wire level. + +Nothing was invented. All three lower to `{$ne: null}` / `{$eq: null}` — the +spelling the same files already emit for `$null` — which answers has-value on +**both** readings of "no value": a stored `null` and an absent key. + +**Grading, argued from what was measured rather than from custom.** This is +`minor`, not `patch`, and the sibling card #13166 is why the distinction is +worth stating: that one graded `patch` on the explicit ground that +`InMemoryDriver.find()` was unaffected and only the non-exported reference +matcher moved. Here the opposite is true — the live query path callers actually +reach changes on **two published drivers**, on a filter operator in the public +Filter Protocol. Measured on a 3-row fixture where one row stores `name: null`: + +| filter | before | after | +|:--|:--|:--| +| `{name: {$exists: true}}` | `['1','2','3']` | `['1','2']` | +| `{name: {$exists: false}}` | `[]` | `['3']` | +| `{$not: {name: {$exists: true}}}` | `[]` | `['3']` | + +The middle row is the harm the ruling's record calls the hardest live one: a +caller asking for the rows with **no value** got an empty result — silent +absence, with nothing to narrow — on three of the four exits. A caller who was +getting nothing starts getting rows, which is a behaviour change however welcome +it is. + +The **key-absent** reading is unchanged on every exit, by construction and by +test: `{$ne: null}` already answers has-value there, so the column that agreed +with the ruling before still agrees. It is kept in the suites as the control +that the alignment moved only what it was meant to. + +**One thing the ruled lowering needed that the ruling did not name.** `{$ne: +null}` / `{$eq: null}` reuse keys an author can write on the same field, so +`{name: {$exists: true, $ne: 'b'}}` would assign `$ne` twice into one object and +one of the two constraints would vanish — with *which* one decided by the +author's key order. Measured unguarded: that filter answered `['1','3']` and its +key-swapped twin answered `['1','2']`, where the reference matcher says `['1']` +for both. Four composed cells that agreed with the reference matcher on `main` +would have started disagreeing. So a lowered `$exists` whose key is already +taken is promoted to its own `$and` branch instead of merged; a free key still +merges inline. Both key orders now emit one document, and every composed cell +measured agrees with the reference matcher — including two that did **not** +agree before this change. + +⛔ Not included, deliberately: no `FILTER_LOGIC_CASES` enrolment and no +`packages/spec` edit (the backends had to move first — that is the card's own +step 4, and it is the next card), and nothing retires or discourages `$exists` +in favour of `$null`. Whether one predicate should keep two authorable spellings +is the consumer census, #13492. diff --git a/packages/drivers/driver-memory/src/memory-analytics-echo-operator-coverage.test.ts b/packages/drivers/driver-memory/src/memory-analytics-echo-operator-coverage.test.ts index 8ce667c278..92b2d2c185 100644 --- a/packages/drivers/driver-memory/src/memory-analytics-echo-operator-coverage.test.ts +++ b/packages/drivers/driver-memory/src/memory-analytics-echo-operator-coverage.test.ts @@ -74,7 +74,7 @@ * |:--|:--|:--|:--| * | `{name: {$in: [a, b]}}` | both rows | `name = a` — one row | `name IN (a, b)` | * | `{name: {$nin: [a]}}` | the other four | `name = a` — the **complement** | `(name IS NULL OR name NOT IN (a))` | - * | `{name: {$exists: true}}` | four rows | `name = 1` — **no** rows | `name IS NOT NULL` | + * | `{name: {$exists: true}}` | five rows | `name = 1` — **no** rows | `name IS NOT NULL` | * * # Reverse verification * @@ -347,22 +347,41 @@ describe('[#7117] the analytics echo renders the query it describes', () => { }); /** - * The one cell where SQL cannot say what mingo says, asserted as an - * INEQUALITY so it cannot be closed in silence. + * [#13195] The one cell where SQL could not say what mingo said — CLOSED, + * and closed from the mingo side. * - * mingo's `$exists` tests KEY PRESENCE; a relational column always has it. - * A row storing an explicit `null` therefore satisfies `$exists: true` on - * `query()` and fails `IS NOT NULL` in the echo. `IS NOT NULL` is - * nonetheless the spelling both of this repo's other SQL lowerings use - * (`read-scope-sql.ts`'s `$exists` arm; `driver-sql`'s "a present field is a - * non-null column in SQL"), and it is a far smaller gap than the `name = 1` - * it replaces, which matched nothing at all. + * This assertion used to be an INEQUALITY, kept so the split could not be + * repaired in silence. The split was: mingo's `$exists` tests KEY PRESENCE + * and a relational column always has a key, so row 6 — which stores an + * explicit `null` — satisfied `$exists: true` on `query()` and failed + * `IS NOT NULL` in the echo. The chart and the statement drawn beside it + * answered the same query differently. + * + * The maintainer ruled on 2026-08-30 that `$exists` means HAS A VALUE + * (`!= null`) on every exit — #5298 leg 3 / #5369, shipped in PR #5962 and + * until then unmet on this face. `IS NOT NULL` was already the ruled + * answer, so the ECHO half is untouched and the executed half moved to meet + * it: `CUBE_OPERATOR_TO_MONGO_PREDICATE`'s `set` row now emits + * `{$ne: null}` / `{$eq: null}` instead of `{$exists: }`. + * + * ⛔ It was not re-baselined onto whatever the new pipeline printed. The + * target is the ECHO's pre-existing row set, which this note named as the + * ruled answer before the repair existed, and the `$exists` entry in the + * enumeration below is no longer skipped — it is asserted by the same rule + * as every other operator. */ - it('documents the one `$exists` cell SQL cannot translate exactly', async () => { + it('the `$exists` cell SQL could not translate exactly is now exact', async () => { const where = { name: { $exists: true } }; - // Row 6 stores an explicit `null`: present to mingo, NULL to SQL. - expect(await executedIds(where)).toEqual(['1', '2', '3', '4', '5', '6']); + // Row 6 stores an explicit `null`: no longer a value to either engine. + expect(await executedIds(where)).toEqual(['1', '2', '3', '4', '5']); expect(await echoIds(where)).toEqual(['1', '2', '3', '4', '5']); + expect(await executedIds(where)).toEqual(await echoIds(where)); + + // The other direction, which the old split hid entirely: asking for the + // rows with NO value used to return none of them on the executed side. + const none = { name: { $exists: false } }; + expect(await executedIds(none)).toEqual(['6']); + expect(await echoIds(none)).toEqual(['6']); }); }); @@ -432,11 +451,10 @@ describe('[#7117] the analytics echo renders the query it describes', () => { it(`${op}: running the echo returns exactly the rows the query returns`, async () => { const executed = await executedIds(where); const echoed = await echoIds(where); - if (op === '$exists') { - // The documented residue above — asserted there, skipped here so this - // loop stays a statement about every OTHER operator. - return; - } + // [#13195] `$exists` used to return early here — the documented residue + // above was asserted there and skipped in this loop, so the loop was a + // statement about every OTHER operator. The residue is gone, the skip + // with it, and this loop is now total over the face's vocabulary. expect(echoed, `${op}: the echo describes a different row set`).toEqual(executed); }); diff --git a/packages/drivers/driver-memory/src/memory-analytics.ts b/packages/drivers/driver-memory/src/memory-analytics.ts index 2898b48618..8fe02ea749 100644 --- a/packages/drivers/driver-memory/src/memory-analytics.ts +++ b/packages/drivers/driver-memory/src/memory-analytics.ts @@ -262,9 +262,18 @@ const CUBE_OPERATOR_TO_MONGO_PREDICATE: Readonly ({ $not: { $regex: substring(raw[0]) } }), - // A presence flag, not a comparand. The `raw.length === 0` arm keeps the old - // call site's reading of a valueless `set` ("does it exist" → true). - set: ({ raw }) => ({ $exists: raw.length > 0 ? Boolean(raw[0]) : true }), + // [#13195] A presence flag, not a comparand — and "present" means HAS A + // VALUE (`!= null`), never key presence: #5298 leg 3 / #5369, landed in PR + // #5962, ruled onto this face 2026-08-30. It used to emit `{$exists: }` + // and hand it to mingo, which reads key presence, so this exit EXECUTED the + // key-presence answer while {@link CUBE_OPERATOR_TO_SQL_PREDICATE} ECHOED + // `IS NOT NULL` beside it — the rows a chart was drawn from and the statement + // shown next to it answered the same query differently. The two now agree, + // and the residue that disagreement left in + // `memory-analytics-echo-operator-coverage.test.ts` is gone rather than + // documented. The `raw.length === 0` arm keeps the old call site's reading of + // a valueless `set` ("does it exist" → true). + set: ({ raw }) => ((raw.length === 0 || Boolean(raw[0])) ? { $ne: null } : { $eq: null }), }); /** @@ -381,18 +390,29 @@ function globSubstringPattern(value: unknown): string { * the new operator's SQL spelling. The totality is proven rather than defended, * so no future operator can silently render as an equality nobody wrote. * - * # The one cell where SQL cannot say what mingo says + * # The `set` cell — once the one place SQL could not say what mingo said * * `set` renders `IS NOT NULL` / `IS NULL` — the spelling this repo's other two * SQL lowerings already use (`read-scope-sql.ts`'s `$exists` arm, `driver-sql`'s - * "a present field is a non-null column in SQL"). It is not an exact - * translation, and cannot be: mingo's `$exists` tests KEY PRESENCE, which a - * relational column always has. A row storing an explicit `null` therefore - * satisfies `$exists: true` on `query()` and fails `IS NOT NULL` in the echo. - * That residue is inherent to describing a document store in SQL, it is pinned - * as an explicit inequality in `memory-analytics-echo-operator-coverage.test.ts` - * so it cannot be "fixed" in silence, and it is a far smaller gap than the - * `name = 1` it replaces — which matched nothing at all. + * "a present field is a non-null column in SQL"). This row is UNCHANGED, and it + * is worth saying why it is now an exact translation rather than a documented + * residue. + * + * It used to be inexact in one direction only: the mingo twin emitted + * `{$exists: }`, mingo reads that as KEY PRESENCE, and a relational + * column always has a key — so a row storing an explicit `null` satisfied + * `$exists: true` on `query()` and failed `IS NOT NULL` in the echo. The chart + * and the statement drawn beside it answered the same query differently, and + * the gap was pinned as an explicit INEQUALITY in + * `memory-analytics-echo-operator-coverage.test.ts` so it could not be closed + * in silence. + * + * [#13195] It was not closed in silence: the maintainer ruled on 2026-08-30 + * that `$exists` means HAS A VALUE (`!= null`) on every exit — #5298 leg 3 / + * #5369, shipped in PR #5962 and until then still unmet here — so the mingo + * twin now emits `{$ne: null}` / `{$eq: null}`. SQL's `IS NOT NULL` was already + * the ruled answer; it is the OTHER exit that moved to meet it. The inequality + * pin is now an equality, and this face no longer contradicts itself. */ const CUBE_OPERATOR_TO_SQL_PREDICATE: Readonly> = Object.freeze({ // [#5373] A null comparand is a NULLNESS test, not a comparison. SQL's diff --git a/packages/drivers/driver-memory/src/memory-driver-document-not.test.ts b/packages/drivers/driver-memory/src/memory-driver-document-not.test.ts index 2687f90764..d1acccd782 100644 --- a/packages/drivers/driver-memory/src/memory-driver-document-not.test.ts +++ b/packages/drivers/driver-memory/src/memory-driver-document-not.test.ts @@ -208,7 +208,11 @@ describe('[#5324] InMemoryDriver.find compiles a document-level $not', () => { * * `$exists` REFERENCE is correct. `$exists` means "has a value" * (#5298 ③ / #5369, PR #5962), so mingo's key-presence - * reading is the divergent one. STILL OPEN — #13195. + * reading was the divergent one. CLOSED by #13195: the live + * path stopped handing `$exists` to mingo under its own name + * and lowers it to `{$ne: null}` / `{$eq: null}` — the + * spelling `$null` in the same method already used — so the + * two faces agree. Ruled 2026-08-30. * `$nin` LIVE was correct. Negative operators MATCH no-value rows — * #5146, extended by #5298, re-affirmed 2026-08-10 — so a * missing key satisfying `$nin` is the affirmed answer, and @@ -235,26 +239,46 @@ describe('[#5324] InMemoryDriver.find compiles a document-level $not', () => { * matcher began printing — the target was the live path's pre-existing * answer, named as correct in this very note before the fix existed. * - * ⛔ The `$exists` row is untouched and stays a pinned divergence. It is a - * different cell with a different backend list (`driver-mongodb` reads - * key-presence too), and it belongs to #13195. What that row still shows is - * why this pin exists — this package answers with two faces, so a statement - * like "driver-memory already reads has-value" is true of the reference - * matcher and FALSE of the live query path users actually reach. + * ⚠️ [#13195, ruled 2026-08-30] The third cell has now converged too, and by + * the same discipline: the target was the REFERENCE column, which this note + * named correct before the fix existed, not whatever the live path began + * printing. `driver-mongodb` — which read key-presence for its own, + * wire-level reason — moved in the same change, so the statement this row + * used to disprove is finally true of the whole package AND of the other + * document-shaped backend. + * + * ⛔ What the row still shows, and why the pin stays: this package answers + * with two faces. "driver-memory reads has-value" was true of the reference + * matcher and FALSE of the live query path users actually reach, for the + * three months between #5962 and #13195. Asserting the two columns against + * each other — rather than each against a literal — is what makes a future + * one-sided edit fail here. */ - describe('[#5299] the settled cells, live vs reference — $nin / $notContains converged (#13166), $exists still open (#13195)', () => { + describe('[#5299] the settled cells, live vs reference — $nin / $notContains converged (#13166), $exists converged (#13195)', () => { const liveVsReference = async (where: unknown) => ({ live: await idsFrom(nulled, where), reference: NULLED.filter((r) => match(r, where)).map((r) => r.id), }); - it('$exists on a present-but-null field: mingo says "the key is there", the matcher says "no value"', async () => { + it('$exists on a present-but-null field: the two faces now AGREE (#13195)', async () => { + // Was `live: ['1','2','3','4']` — mingo said "the key is there" while the + // matcher said "no value". The REFERENCE column is unchanged, and it is + // the column this note already named correct. expect(await liveVsReference({ stage: { $exists: true } })).toEqual({ - live: ['1', '2', '3', '4'], + live: ['1', '2'], reference: ['1', '2'], }); }); + it('$exists: false on a present-but-null field: the two faces agree there too (#13195)', async () => { + // The direction the old pin never recorded, and the worse one: the live + // path returned NOTHING for the query asking for the rows with no value. + expect(await liveVsReference({ stage: { $exists: false } })).toEqual({ + live: ['3', '4'], + reference: ['3', '4'], + }); + }); + it('$nin on an ABSENT field: the two faces now AGREE (#13166)', async () => { // Was `reference: ['2']` — the matcher's `value === undefined` guard // short-circuited before the `$nin` arm ran. The LIVE column is unchanged, diff --git a/packages/drivers/driver-memory/src/memory-driver.ts b/packages/drivers/driver-memory/src/memory-driver.ts index ae365253c8..46f55f505b 100644 --- a/packages/drivers/driver-memory/src/memory-driver.ts +++ b/packages/drivers/driver-memory/src/memory-driver.ts @@ -1130,6 +1130,15 @@ export class InMemoryDriver implements IDataDriver { continue; } const normalized = this.normalizeFieldOperators(value, this.temporalKind(object, key), key, here); + // [#13195] A lowered `$exists` whose mingo key was already taken by a + // sibling operator on the same field. It cannot be merged without one + // of the two constraints silently overwriting the other, so it becomes + // its own `$and` branch — see the merge in normalizeFieldOperators(). + if (normalized._presenceAnd) { + const presence: Record = normalized._presenceAnd; + delete normalized._presenceAnd; + extraAndConditions.push({ [key]: presence }); + } // Handle multiple regex conditions on the same field (e.g. $startsWith + $endsWith) if (normalized._multiRegex) { const regexConditions: Record[] = normalized._multiRegex; @@ -1176,6 +1185,8 @@ export class InMemoryDriver implements IDataDriver { const store = (v: any) => coerceTemporalValue(v, kind); const result: Record = {}; const regexConditions: Record[] = []; + /** [#13195] `$exists`, lowered — see the merge at the end of this method. */ + let presence: Record | undefined; for (const op of Object.keys(ops)) { const val = ops[op]; @@ -1288,9 +1299,27 @@ export class InMemoryDriver implements IDataDriver { case '$in': case '$nin': result[op] = store(val); break; - // Evaluated by mingo under the same name. `$exists` is a presence - // predicate, not a comparand, so it does not take the field's storage - // form (#4047). + // [#13195] `$exists` means "the field HAS A VALUE" (`!= null`), never + // key presence — #5298 leg 3 / #5369, landed in PR #5962, and ruled + // onto this exit by the maintainer on 2026-08-30. + // + // It used to be `result[op] = val`: the operator went to mingo under + // its own name, and mingo evaluates `$exists` as KEY PRESENCE. So a row + // storing an explicit `null` — how a SQL NULL round-trips into a record + // — satisfied `$exists: true` here while the reference matcher one file + // over said it had no value, and `$exists: false` returned NOTHING at + // all where the caller asked for the rows with no value. Silent + // absence, not visible surplus, which is the trade + // `filter-logic-conformance.ts` calls the worse one. + // + // The lowering is not an invention: it is the spelling `$null` already + // uses fifteen lines up, and mingo answers it has-value on BOTH + // readings of "no value" — a stored `null` and an ABSENT KEY — so the + // key-absent column, which already agreed with the ruling, is unmoved. + // + // Value comparisons take the field's storage form (#4047); this one is + // a presence predicate, so its `null` is written literally, exactly as + // the `$null` arm writes its own. // // [#5702] `$regex` and `$options` were passed through here too, on the // same line, for the same "not a comparand" reason. Both are RETIRED @@ -1299,7 +1328,9 @@ export class InMemoryDriver implements IDataDriver { // evaluation arm for a refused operator is exactly what let this // driver's two faces answer one `$regex` differently for so long. case '$exists': - result[op] = val; + // Collected, not assigned: the lowering below has to know whether the + // key it wants is already spoken for. See the merge at the end. + presence = val === true ? { $ne: null } : { $eq: null }; break; default: // [#5324] Was `result[op] = val` — a GENERIC passthrough that handed @@ -1311,6 +1342,39 @@ export class InMemoryDriver implements IDataDriver { } } + // [#13195] Merge the lowered `$exists`, and do NOT let it clobber a sibling. + // + // The lowering the ruling prescribes reuses `$ne` / `$eq` — mingo keys an + // AUTHOR can also write on the same field. `{name: {$exists: true, $ne: + // 'b'}}` would therefore assign `$ne` twice into one object literal, and + // whichever ran last would win: one of the two constraints vanishes, and + // WHICH one depends on the author's key order. Measured on this fixture + // before the guard existed: `{$exists: true, $ne: 'b'}` answered + // `['1','3']` and the key-swapped `{$ne: 'b', $exists: true}` answered + // `['1','2']` — one predicate, two row sets — while the reference matcher + // said `['1']` for both. Four cells that AGREED with the reference matcher + // before the alignment disagreed after it. + // + // So when the key is free the predicate merges inline (the common case — + // `$exists` alone on a field), and when it is taken the field is promoted + // to its own `$and` branch, where both constraints survive. `_presenceAnd` + // is an internal sentinel consumed by normalizeFilterCondition(), the same + // shape `_multiRegex` below uses for the same reason. + // + // ⚠️ Scope: this guards the operator this card moved, and only it. The + // identical clobber is reachable today through `$null`, `$between` and + // `$notContains`, which lower to `$ne`/`$eq`, `$gte`/`$lte`/`$lt` and + // `$not` respectively — measured, pre-existing, and filed separately rather + // than half-fixed here. + if (presence) { + const presenceKey = Object.keys(presence)[0]!; + if (Object.prototype.hasOwnProperty.call(result, presenceKey)) { + result._presenceAnd = presence; + } else { + Object.assign(result, presence); + } + } + // Merge regex conditions: single → inline, multiple → wrap with $and if (regexConditions.length === 1) { Object.assign(result, regexConditions[0]); diff --git a/packages/drivers/driver-memory/src/memory-exists-has-value-faces.test.ts b/packages/drivers/driver-memory/src/memory-exists-has-value-faces.test.ts index 16f5be0981..2bdf8e7918 100644 --- a/packages/drivers/driver-memory/src/memory-exists-has-value-faces.test.ts +++ b/packages/drivers/driver-memory/src/memory-exists-has-value-faces.test.ts @@ -12,25 +12,32 @@ * and this package's reference matcher (`memory-matcher.ts`, pinned by * `memory-matcher-not-null-safe.test.ts`). * - * ## What this file is, and what it is NOT + * ## What this file is — and what it WAS * - * It is a PIN OF THE MEASURED PRESENT, taken because the card that records this - * divergence records it as reading rather than execution and its own header - * says so ("Read, not re-measured"). Executing it found the reading OPTIMISTIC - * by a factor of three, exactly as `aggregation-conformance.ts`'s DEBT note - * predicts a read row will be. + * It began as a PIN OF THE MEASURED PRESENT, taken because the card recorded + * this divergence as reading rather than execution and its own header said so + * ("Read, not re-measured"). Executing it found the reading OPTIMISTIC by a + * factor of three, exactly as `aggregation-conformance.ts`'s DEBT note predicts + * a read row will be: one recorded column on two surfaces became three + * divergent cells across four exits. * - * It is NOT the ruling's enforcement, and it is NOT a decision. The direction - * for `driver-mongodb` is unsettled (its `$exists` is key-presence at the wire - * level), and the `FILTER_LOGIC_CASES` enrolment that would enforce the ruling - * cannot land before the backends move — the DEBT ledger in - * `scripts/check-driver-conformance.mjs` is per (driver x case-set), so a row - * added ahead of a backend is only "a gate that reports a known red" (#5903). + * ⚖️ RULED 2026-08-30. The maintainer adopted option A — all three lagging + * exits align to the settled semantic, `$exists` = HAS A VALUE. `driver-mongodb` + * needed no invention: the same translator already emits `{$ne: null}` / + * `{$eq: null}` for `$null`, and a real mongod 8.2.6 was measured compliant on + * both readings. So this file is no longer a pin of a divergence; it is the + * ENFORCEMENT of the ruling, INVERTED IN PLACE per its own former instruction. + * Nothing was deleted and nothing was re-baselined onto whatever the new code + * happened to print: every expectation below was already carrying the ruling's + * answer beside the measured one, and the flip is that named answer. * - * ⚠️ WHEN THE DIRECTION IS DECIDED, INVERT THESE IN PLACE. Do not delete them - * and do not re-baseline them to whatever the new output happens to be: each - * divergent expectation below names the ruling's answer beside the measured - * one, so flipping it is a one-line edit that stays reviewable. + * ⛔ Still NOT done here, and deliberately: the `FILTER_LOGIC_CASES` enrolment + * that would score this cell on the conformance gate. The DEBT ledger in + * `scripts/check-driver-conformance.mjs` is per (driver x case-set), so a row + * cannot be added one driver at a time; and `packages/spec` was fenced for the + * dispatch that landed this. The backends have now moved, which is the + * precondition the card's step 4 names — the enrolment is the next card, not a + * rider on this one. * * ## Why the fixture has TWO readings and why that is the load-bearing part * @@ -39,23 +46,31 @@ * - `name: null` — how a SQL NULL round-trips into a record; * - the key ABSENT — what a partial write leaves. * - * Measured: EVERY divergent cell in this file is on the `name: null` reading, - * and the key-absent reading agrees with the ruling on all four exits. That is - * the opposite shape from the neighbouring #13166 cell, where `$notContains` - * diverged on both readings and `$nin` on the absent one only. Consequence - * worth stating loudly: a fixture that spells "no value" as an ABSENT KEY - * measures ZERO of this divergence. Keep both columns. + * Measured: EVERY divergent cell in this file was on the `name: null` reading, + * and the key-absent reading agreed with the ruling on all four exits already. + * That is the opposite shape from the neighbouring #13166 cell, where + * `$notContains` diverged on both readings and `$nin` on the absent one only. + * Consequence worth stating loudly, and the reason both columns survive the + * flip: a fixture that spells "no value" as an ABSENT KEY measured ZERO of this + * divergence, and now measures ZERO of the repair. The key-absent column is + * kept as the CONTROL that the alignment moved only what it was meant to — + * `{$ne: null}` answers has-value on both readings, so those rows must be + * exactly as they were before. * * ## Why `$exists: false` is here and not only `$exists: true` * - * The recorded table carries one column, `$exists: true` on a null value, whose - * divergence is SURPLUS — a row the author can see and narrow. `$exists: false` - * diverges in the opposite direction, and it is the worse one: the row with no - * value is DROPPED from the query that asks for rows with no value, so the - * caller sees an empty result and nothing to narrow. `filter-logic-conformance.ts` - * makes exactly that trade its reason for keeping the include direction on - * `$ne` / `$nin` ("silent absence for visible surplus"); this cell sits on the - * wrong side of it and was invisible while only `$exists: true` was recorded. + * The recorded table carried one column, `$exists: true` on a null value, whose + * divergence was SURPLUS — a row the author can see and narrow. `$exists: false` + * diverged in the opposite direction, and it was the worse one: the row with no + * value was DROPPED from the query that asks for rows with no value, so the + * caller saw an empty result and had nothing to narrow. + * `filter-logic-conformance.ts` makes exactly that trade its reason for keeping + * the include direction on `$ne` / `$nin` ("silent absence for visible + * surplus"); this cell sat on the wrong side of it and was invisible while only + * `$exists: true` was recorded. It is the harm the ruling's record calls the + * hardest live one, and closing it is why `$exists: false` and + * `$not {$exists: true}` are both asserted below rather than one standing in + * for the other. */ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; @@ -131,7 +146,7 @@ async function analytics( } describe('[#13195] `$exists` on a row with NO VALUE — the two readings, the four exits', () => { - describe('the key-absent reading: every exit already answers the ruling', () => { + describe('the key-absent reading: the CONTROL — it answered the ruling before and must not move', () => { it('`$exists: true` excludes the no-key row everywhere', async () => { expect(await liveIds(MISSING, { name: { $exists: true } })).toEqual(['1', '2']); expect(matcherIds(MISSING, { name: { $exists: true } })).toEqual(['1', '2']); @@ -150,40 +165,68 @@ describe('[#13195] `$exists` on a row with NO VALUE — the two readings, the fo }); }); - describe('the `name: null` reading: the reference matcher answers the ruling, the other exits do not', () => { + describe('the `name: null` reading: every exit now answers the ruling (#13195, ruled 2026-08-30)', () => { it('the reference matcher reads HAS-VALUE — the ruling, shipped by #5962', () => { expect(matcherIds(NULLED, { name: { $exists: true } })).toEqual(['1', '2']); expect(matcherIds(NULLED, { name: { $exists: false } })).toEqual(['3']); expect(matcherIds(NULLED, { $not: { name: { $exists: true } } })).toEqual(['3']); }); - it('DIVERGENT — the live mingo path reads KEY-PRESENCE: `$exists: true` keeps the null row', async () => { - // Ruling: ['1','2']. Measured: ['1','2','3'] — mingo tests key presence. - expect(await liveIds(NULLED, { name: { $exists: true } })).toEqual(['1', '2', '3']); + it('the live mingo path reads HAS-VALUE: `$exists: true` drops the null row', async () => { + // Was ['1','2','3'] — the operator went to mingo under its own name and + // mingo tests KEY PRESENCE. It now lowers to `{$ne: null}`, the spelling + // `$null` in the same method already used. The ruling's answer, named in + // this line's own comment before the flip existed. + expect(await liveIds(NULLED, { name: { $exists: true } })).toEqual(['1', '2']); }); - it('DIVERGENT and WORSE — `$exists: false` on the live path returns NOTHING', async () => { - // Ruling: ['3']. Measured: [] — the row with no value is dropped from the - // query that asks for rows with no value. Silent absence, not surplus. - expect(await liveIds(NULLED, { name: { $exists: false } })).toEqual([]); - expect(await liveIds(NULLED, { $not: { name: { $exists: true } } })).toEqual([]); + it('THE HARDEST LIVE HARM, CLOSED — `$exists: false` returns the no-value row', async () => { + // Was [] on BOTH spellings: the row with no value was dropped from the + // query that asks for rows with no value, so the caller got silent + // absence and nothing to narrow. The ruling's record names this the + // hardest live harm — a caller wanting no-value rows got an empty result + // on three of the four exits. + expect(await liveIds(NULLED, { name: { $exists: false } })).toEqual(['3']); + expect(await liveIds(NULLED, { $not: { name: { $exists: true } } })).toEqual(['3']); }); - it('DIVERGENT — the analytics face EXECUTES the live mingo key-presence answer', async () => { - expect((await analytics(NULLED, { name: { $exists: true } })).executed).toEqual(['1', '2', '3']); - expect((await analytics(NULLED, { name: { $exists: false } })).executed).toEqual([]); + it('`$exists: false` and `$not {$exists: true}` AGREE — asserted as an equality', async () => { + // The two spellings diverged from the ruling together, and the record + // requires that they now agree with EACH OTHER as well as with the + // ruling. Asserted directly so a future change that moves only one of + // them cannot pass by moving both expectations independently. + const direct = await liveIds(NULLED, { name: { $exists: false } }); + const negated = await liveIds(NULLED, { $not: { name: { $exists: true } } }); + expect(direct).toEqual(negated); + expect(direct).toEqual(matcherIds(NULLED, { name: { $exists: false } })); + + // And on the other reading of "no value", where they already agreed. + const directMissing = await liveIds(MISSING, { name: { $exists: false } }); + const negatedMissing = await liveIds(MISSING, { $not: { name: { $exists: true } } }); + expect(directMissing).toEqual(negatedMissing); + expect(directMissing).toEqual(['3']); }); - it('the analytics face ECHOES the has-value answer — so it disagrees with ITSELF', async () => { + it('the analytics face EXECUTES the has-value answer', async () => { + // Was ['1','2','3'] and [] — this face built its own `{$exists: }` + // and handed it to mingo, so it inherited key-presence independently of + // the live path above. + expect((await analytics(NULLED, { name: { $exists: true } })).executed).toEqual(['1', '2']); + expect((await analytics(NULLED, { name: { $exists: false } })).executed).toEqual(['3']); + }); + + it('the analytics face no longer disagrees with ITSELF — echo and rows now match', async () => { // The statement drawn beside the chart says `IS NOT NULL` / `IS NULL`, - // which is the ruling; the rows the chart is drawn FROM say key-presence. - // Asserted as an INEQUALITY as well, so it cannot be closed in silence. + // which was already the ruling; the rows the chart was drawn FROM said + // key-presence. This assertion was an INEQUALITY, kept so the split could + // not be closed in silence. It was not closed in silence — it is an + // EQUALITY now, and the echo half is unchanged. const t = await analytics(NULLED, { name: { $exists: true } }); const f = await analytics(NULLED, { name: { $exists: false } }); expect(t.sql).toContain('name IS NOT NULL'); expect(f.sql).toContain('name IS NULL'); - expect(t.executed).not.toEqual(matcherIds(NULLED, { name: { $exists: true } })); - expect(f.executed).not.toEqual(matcherIds(NULLED, { name: { $exists: false } })); + expect(t.executed).toEqual(matcherIds(NULLED, { name: { $exists: true } })); + expect(f.executed).toEqual(matcherIds(NULLED, { name: { $exists: false } })); }); }); @@ -211,4 +254,62 @@ describe('[#13195] `$exists` on a row with NO VALUE — the two readings, the fo expect((await analytics(NULLED, { name: { $eq: 'beta' } })).executed).toEqual(['2']); }); }); + + /** + * [#13195] `$exists` SHARING a field constraint with another operator. + * + * Not in the recorded table, not in the card, and not a cell the ruling + * names — it is a consequence of the lowering the ruling prescribes, found by + * measuring it. `{$ne: null}` / `{$eq: null}` reuse mingo keys an AUTHOR can + * write on the same field, so a naive merge assigns `$ne` twice into one + * object and one of the two constraints silently disappears — and WHICH one + * depends on the author's key order. + * + * Measured with the lowering in place and the promotion guard removed, on the + * NULLED fixture: `{$exists: true, $ne: 'beta'}` answered `['1','3']`, + * the key-swapped `{$ne: 'beta', $exists: true}` answered `['1','2']`, and + * `{$exists: false, $eq: 'alpha-one'}` answered `['1']` — a row that HAS a + * value returned by a filter demanding it have none. The reference matcher, + * which loops the operators and cannot clobber, said `['1']`, `['1']` and + * `[]`. + * + * Four of those six cells AGREED with the reference matcher on `origin/main` + * before the alignment, so shipping the merge unguarded would have traded a + * fixed single-operator cell for a broken composed one — the one-driver, + * two-faces shape this card exists to remove. The reference matcher is the + * ORACLE here, exactly as it is for the single-operator cells above. + */ + describe('composed constraints — the lowering must not clobber a sibling operator', () => { + const COMPOSED: Array<[string, unknown]> = [ + ['$exists:true beside $ne', { name: { $exists: true, $ne: 'beta' } }], + ['$ne beside $exists:true (keys swapped)', { name: { $ne: 'beta', $exists: true } }], + ['$exists:false beside $eq', { name: { $exists: false, $eq: 'alpha-one' } }], + ['$exists:true beside $eq', { name: { $exists: true, $eq: 'alpha-one' } }], + ['$exists:true beside $contains', { name: { $exists: true, $contains: 'alpha' } }], + ]; + + for (const [label, where] of COMPOSED) { + it(`${label}: the live path answers what the reference matcher answers`, async () => { + for (const rows of [NULLED, MISSING]) { + expect(await liveIds(rows, where), label).toEqual(matcherIds(rows, where)); + } + }); + } + + it('the two key orders of one predicate answer identically', async () => { + for (const rows of [NULLED, MISSING]) { + expect(await liveIds(rows, { name: { $exists: true, $ne: 'beta' } })).toEqual( + await liveIds(rows, { name: { $ne: 'beta', $exists: true } }), + ); + } + }); + + it('CONTROL — the composed predicate really is narrower than either half', async () => { + // Without this the block above could pass on a fixture where the two + // constraints happen to select the same rows, certifying nothing. + expect(await liveIds(NULLED, { name: { $exists: true, $ne: 'beta' } })).toEqual(['1']); + expect(await liveIds(NULLED, { name: { $exists: true } })).toEqual(['1', '2']); + expect(await liveIds(NULLED, { name: { $ne: 'beta' } })).toEqual(['1', '3']); + }); + }); }); diff --git a/packages/drivers/driver-memory/src/memory-own-key-undefined.test.ts b/packages/drivers/driver-memory/src/memory-own-key-undefined.test.ts index 89d8e5edb9..d5062075fb 100644 --- a/packages/drivers/driver-memory/src/memory-own-key-undefined.test.ts +++ b/packages/drivers/driver-memory/src/memory-own-key-undefined.test.ts @@ -200,8 +200,17 @@ describe('#9276 an own key holding `undefined` is not a row state this driver em // Measured identical on `origin/main` before the repair. expect(await ids({ status: { $null: true } })).toEqual(['a1', 'a2', 'a3']); expect(await ids({ status: { $null: false } })).toEqual(['a4']); - expect(await ids({ status: { $exists: true } })).toEqual(['a3', 'a4']); - expect(await ids({ status: { $exists: false } })).toEqual(['a1', 'a2']); + // [#13195] `$exists` was ['a3','a4'] / ['a1','a2'] — it read KEY PRESENCE, + // so a3 (which stores an explicit `null`) counted as existing. Ruled + // 2026-08-30: `$exists` means HAS A VALUE (`!= null`), #5298 leg 3 / #5369. + // ⚠️ The consequence is visible right here and is deliberate, not + // incidental: the two lines now answer exactly as the `$null` lines above + // do, inverted. Whether one predicate should keep two authorable spellings + // is NOT settled by that ruling — it is the consumer census, #13492. + expect(await ids({ status: { $exists: true } })).toEqual(['a4']); + expect(await ids({ status: { $exists: false } })).toEqual(['a1', 'a2', 'a3']); + expect(await ids({ status: { $exists: true } })).toEqual(await ids({ status: { $null: false } })); + expect(await ids({ status: { $exists: false } })).toEqual(await ids({ status: { $null: true } })); expect(await ids({ status: 'draft' })).toEqual(['a4']); }); diff --git a/packages/drivers/driver-mongodb/src/mongodb-exists-has-value-translation.test.ts b/packages/drivers/driver-mongodb/src/mongodb-exists-has-value-translation.test.ts index e2d3a6790e..0687ccd9de 100644 --- a/packages/drivers/driver-mongodb/src/mongodb-exists-has-value-translation.test.ts +++ b/packages/drivers/driver-mongodb/src/mongodb-exists-has-value-translation.test.ts @@ -4,19 +4,27 @@ * [#13195] What `translateFilter` emits for `$exists`, and what MongoDB makes * of it on a row with NO VALUE — measured, not read. * - * ## The ruling, and why this driver is the hard half + * ## The ruling, and why this driver was thought to be the hard half * * The platform ruling is that `$exists` means "has a value" (`!= null`), never * key-presence (#5298 leg 3 / #5369, landed in PR #5962). MongoDB's `$exists` * is key-presence at the wire level, so this driver cannot satisfy the ruling - * by passing the operator through under its own name: it would have to emit - * something other than `{$exists: }`. That is a direction nobody has - * ruled on, so this file DECIDES NOTHING. It pins the present. + * by passing the operator through under its own name: it has to emit something + * other than `{$exists: }`. * - * ⚠️ WHEN THE DIRECTION IS DECIDED, INVERT THESE IN PLACE — do not delete them, - * and do not re-baseline them to whatever the new emitter happens to produce. - * Every divergent expectation names the ruling's answer beside the measured - * one, so the flip is a reviewable one-line edit. + * ⚖️ RULED 2026-08-30, option A. The something-other is not an invention and + * cost nothing to find: this same translator ALREADY emits `{$ne: null}` / + * `{$eq: null}` for `$null`, and the block at the bottom of this file measured + * that spelling answering the ruling on both readings — which is what turned + * the open question from "can MongoDB express has-value" (a capability + * question, answered YES) into "should `$exists` and `$null` stay two + * spellings of one predicate" (a vocabulary question, and NOT decided here — + * it is the consumer census, #13492). + * + * So this file has been INVERTED IN PLACE per its own former instruction. + * Nothing was deleted, and nothing was re-baselined onto whatever the new + * emitter happened to produce: every divergent expectation already named the + * ruling's answer beside the measured one, and the flip is that named answer. * * ## What was measured, and against what * @@ -48,10 +56,13 @@ * ## Why the fixture carries TWO readings of "no value" * * `name: null` (how a SQL NULL round-trips) and the key ABSENT (what a partial - * write leaves) reach different rules above. Measured: every divergent cell is - * on the `name: null` reading, and the key-absent reading already answers the - * ruling. A fixture that spells "no value" as an absent key therefore measures - * NONE of this. Keep both columns. + * write leaves) reach different rules above. Measured: every divergent cell was + * on the `name: null` reading, and the key-absent reading already answered the + * ruling. A fixture that spells "no value" as an absent key therefore measured + * NONE of this — and now measures none of the repair, which is exactly why that + * column is kept: it is the CONTROL that the emitter moved only what it was + * meant to. Rule 2 above is what makes `{$ne: null}` answer has-value on the + * absent reading too, so those rows must read the same before and after. */ import { describe, it, expect } from 'vitest'; @@ -82,6 +93,13 @@ function matchMongoDoc(row: Record, doc: Record>).some((b) => matchMongoDoc(row, b))) return false; continue; } + // [#13195] Modelled because the emitter now produces it: a lowered + // `$exists` whose key is already taken by a sibling operator is promoted to + // its own branch rather than merged over the sibling. + if (field === '$and') { + if (!(cond as Array>).every((b) => matchMongoDoc(row, b))) return false; + continue; + } if (field.startsWith('$')) throw new UnsupportedShape(`unsupported top-level operator '${field}'`); const present = Object.prototype.hasOwnProperty.call(row, field); // Rule 2: equality treats a MISSING field as `null`. `$exists` (rule 1) does not. @@ -119,16 +137,30 @@ const ids = (rows: Array>, authorable: unknown): string[ describe('[#13195] `$exists` translation and its answer on a no-value row', () => { describe('the emitted document — the machine-checkable half, no semantics needed', () => { - it('passes `$exists` through under its own name, both ways', () => { - expect(translateFilter({ name: { $exists: true } })).toEqual({ name: { $exists: true } }); - expect(translateFilter({ name: { $exists: false } })).toEqual({ name: { $exists: false } }); + it('lowers `$exists` to the nullness test, both ways', () => { + // Was `{name: {$exists: }}` — the operator passed through under its + // own name, which is key-presence at the wire level. + expect(translateFilter({ name: { $exists: true } })).toEqual({ name: { $ne: null } }); + expect(translateFilter({ name: { $exists: false } })).toEqual({ name: { $eq: null } }); + }); + + it('the emitted document is EXACTLY what `$null` emits — the same spelling, not a lookalike', () => { + // The point the ruling turned on: no invention. Asserted as an equality + // between the two translations rather than by restating the literal, so a + // change to one that is not made to the other fails here. + expect(translateFilter({ name: { $exists: true } })).toEqual( + translateFilter({ name: { $null: false } }), + ); + expect(translateFilter({ name: { $exists: false } })).toEqual( + translateFilter({ name: { $null: true } }), + ); }); - it('a document-level `$not` leaves as `$nor`, still wrapping a bare `$exists`', () => { + it('a document-level `$not` leaves as `$nor`, now wrapping the nullness test', () => { // MongoDB has no document-level `$not`, so the negation has to change shape; - // the operator inside it does not. + // the operator inside it is the lowered one. expect(translateFilter({ $not: { name: { $exists: true } } } as never)).toEqual({ - $nor: [{ name: { $exists: true } }], + $nor: [{ name: { $ne: null } }], }); }); }); @@ -141,27 +173,91 @@ describe('[#13195] `$exists` translation and its answer on a no-value row', () = }); }); - describe('the `name: null` reading: DIVERGENT on every cell', () => { - it('`$exists: true` keeps the null row — key-presence, where the ruling says has-value', () => { - // Ruling: ['1','2']. Measured: ['1','2','3']. - expect(ids(NULLED, { name: { $exists: true } })).toEqual(['1', '2', '3']); + describe('the `name: null` reading: the cells that diverged, now answering the ruling', () => { + it('`$exists: true` drops the null row — has-value, the ruling', () => { + // Was ['1','2','3'] — `{$exists: true}` is key-presence and a stored + // `null` is present. The ruling's answer, named in this line's own + // comment before the flip existed. + expect(ids(NULLED, { name: { $exists: true } })).toEqual(['1', '2']); + }); + + it('THE HARDEST LIVE HARM, CLOSED — `$exists: false` returns the no-value row', () => { + // Was [] on both spellings. Silent absence, not visible surplus: the row + // with no value was dropped from the query asking for rows with no + // value, so the caller had nothing to narrow. + expect(ids(NULLED, { name: { $exists: false } })).toEqual(['3']); + expect(ids(NULLED, { $not: { name: { $exists: true } } })).toEqual(['3']); + }); + + it('`$exists: false` and `$not {$exists: true}` AGREE — asserted as an equality', () => { + // The record requires the two spellings to agree with EACH OTHER as well + // as with the ruling, so a future change that moves only one of them + // cannot pass by moving both expectations independently. + expect(ids(NULLED, { name: { $exists: false } })).toEqual( + ids(NULLED, { $not: { name: { $exists: true } } }), + ); + expect(ids(MISSING, { name: { $exists: false } })).toEqual( + ids(MISSING, { $not: { name: { $exists: true } } }), + ); + }); + }); + + /** + * [#13195] `$exists` SHARING a field constraint with another operator. + * + * Not a cell the card or the ruling names — it is a consequence of the + * prescribed lowering, found by measuring it. `{$ne: null}` / `{$eq: null}` + * reuse MongoDB keys an AUTHOR can write on the same field, so a naive merge + * assigns `$ne` twice into one document and one of the two constraints + * silently disappears, with WHICH one decided by the author's key order. + * + * Measured with the lowering in place and the promotion guard removed: + * `{name: {$exists: true, $ne: 'beta'}}` emitted `{name: {$ne: 'beta'}}` and + * the key-swapped spelling emitted `{name: {$ne: null}}` — one predicate, two + * documents, neither carrying both constraints. The emitter promotes the + * lowered predicate to its own `$and` branch instead when the key is taken. + */ + describe('composed constraints — the lowering must not clobber a sibling operator', () => { + it('both constraints survive, and the two key orders emit the SAME document', () => { + const a = translateFilter({ name: { $exists: true, $ne: 'beta' } } as never); + const b = translateFilter({ name: { $ne: 'beta', $exists: true } } as never); + expect(a).toEqual(b); + expect(a).toEqual({ $and: [{ name: { $ne: 'beta' } }, { name: { $ne: null } }] }); + }); + + it('a free key still merges inline — the promotion is not blanket', () => { + expect(translateFilter({ name: { $exists: true, $eq: 'alpha-one' } } as never)).toEqual({ + name: { $eq: 'alpha-one', $ne: null }, + }); + }); + + it('the composed predicate answers what each half requires, and is narrower than either', () => { + expect(ids(NULLED, { name: { $exists: true, $ne: 'beta' } })).toEqual(['1']); + expect(ids(MISSING, { name: { $exists: true, $ne: 'beta' } })).toEqual(['1']); + // CONTROL — without these the block could pass on a fixture where the two + // constraints happen to select the same rows, certifying nothing. + expect(ids(NULLED, { name: { $exists: true } })).toEqual(['1', '2']); + expect(ids(NULLED, { name: { $ne: 'beta' } })).toEqual(['1', '3']); }); - it('`$exists: false` returns NOTHING — the worse direction, and unrecorded until now', () => { - // Ruling: ['3']. Measured: []. Silent absence, not visible surplus: the - // row with no value is dropped from the query asking for rows with no - // value, so the caller has nothing to narrow. - expect(ids(NULLED, { name: { $exists: false } })).toEqual([]); - expect(ids(NULLED, { $not: { name: { $exists: true } } })).toEqual([]); + it('`$exists: false` beside an `$eq` keeps the contradiction rather than dropping a half', () => { + // Unguarded this emitted `{name: {$eq: 'alpha-one'}}` and returned row 1 — + // a row that HAS a value, answering a filter that demands it have none. + expect(ids(NULLED, { name: { $exists: false, $eq: 'alpha-one' } })).toEqual([]); }); }); describe('the has-value spelling this translator ALREADY emits — measured, not proposed', () => { /** - * Stated because the cost of the open direction is otherwise guessed. It is - * NOT a recommendation and NOT a decision: what a has-value `$exists` should - * compile to, and whether it should become an exact synonym for the negation - * of `$null`, is the fork this card hands to the decision box. + * The block that decided the ruling. It measured that the cost of option A + * was zero invention — this translator already emitted a has-value spelling + * — which is why the maintainer's 2026-08-30 ruling could adopt it. + * + * ⚠️ It is kept, unchanged, and it is now also a CONTROL: the `$exists` + * cells above must equal these, and the `$null` cells here must not have + * moved to meet them. Whether two authorable spellings for one predicate + * should survive at all is NOT settled by that ruling — it is the consumer + * census, #13492, and this file decides nothing about it. */ it('`$null` lowers to `$ne: null` / `$eq: null`, which answer has-value on BOTH readings', () => { expect(translateFilter({ name: { $null: false } })).toEqual({ name: { $ne: null } }); diff --git a/packages/drivers/driver-mongodb/src/mongodb-filter.test.ts b/packages/drivers/driver-mongodb/src/mongodb-filter.test.ts index 1f97428d9c..1031555ace 100644 --- a/packages/drivers/driver-mongodb/src/mongodb-filter.test.ts +++ b/packages/drivers/driver-mongodb/src/mongodb-filter.test.ts @@ -156,9 +156,13 @@ describe('MongoDB Filter Translator', () => { }); }); - it('translates $exists', () => { + it('translates $exists to the nullness test — has-value, not key-presence', () => { + // [#13195, ruled 2026-08-30] Was `{avatar: {$exists: true}}`, a + // passthrough, which is key-presence at the wire level. `$exists` means + // HAS A VALUE (`!= null`) — #5298 leg 3 / #5369 — and the lowering is the + // one the `$null` arm in the same file already emits. expect(translateFilter({ avatar: { $exists: true } })).toEqual({ - avatar: { $exists: true }, + avatar: { $ne: null }, }); }); diff --git a/packages/drivers/driver-mongodb/src/mongodb-filter.ts b/packages/drivers/driver-mongodb/src/mongodb-filter.ts index 9f065be8a9..8163e2f76b 100644 --- a/packages/drivers/driver-mongodb/src/mongodb-filter.ts +++ b/packages/drivers/driver-mongodb/src/mongodb-filter.ts @@ -662,7 +662,17 @@ function translateCondition( const objValue = value as Record; const hasOps = Object.keys(objValue).some((k) => k.startsWith('$')); if (hasOps) { - mongoFilter[key] = translateFieldOperators(objValue, temporalKind?.(key), key, `${path}.${key}`); + const translated = translateFieldOperators(objValue, temporalKind?.(key), key, `${path}.${key}`); + // [#13195] A lowered `$exists` whose MongoDB key was already taken + // by a sibling operator on the same field. Merging it would drop one + // of the two constraints silently, so it becomes its own `$and` + // branch — see the merge in translateFieldOperators(). + const presenceAnd = translated._presenceAnd as Record | undefined; + if (presenceAnd) { + delete translated._presenceAnd; + andClauses.push({ [key]: presenceAnd } as Filter); + } + mongoFilter[key] = translated; } else { // Nested object — treat as exact match mongoFilter[key] = value; @@ -711,6 +721,8 @@ function translateFieldOperators( ): Record { const result: Record = {}; const store = (v: unknown) => coerceTemporalValue(v, kind); + /** [#13195] `$exists`, lowered — see the merge at the end of this function. */ + let presence: Record | undefined; for (const [op, value] of Object.entries(ops)) { switch (op) { @@ -725,10 +737,31 @@ function translateFieldOperators( result[op] = store(value); break; - // Value-independent — a presence predicate takes a boolean, not a - // comparand, so it is never coerced. + // [#13195] Value-independent — a presence predicate takes a boolean, not + // a comparand, so it is never coerced. And "present" means the field HAS + // A VALUE (`!= null`), never key presence: #5298 leg 3 / #5369, landed in + // PR #5962, ruled onto this driver by the maintainer on 2026-08-30. + // + // It used to be `result[op] = value` — the operator passed through under + // its own name. MongoDB's `$exists` IS key presence at the wire level, so + // that passthrough is exactly what made this backend answer a stored + // `null` differently from `driver-memory`'s reference matcher: `$exists: + // true` kept the no-value row, and `$exists: false` returned NOTHING + // where the caller asked for the rows with no value. + // + // Nothing is invented to satisfy the ruling. `{$ne: null}` / `{$eq: null}` + // is the spelling the `$null` arm below already emits, and the rule that + // makes it answer has-value is stated in this package already + // (`mongodb-driver.ts`, beside the `$null` lowering): MongoDB matches a + // MISSING field and a stored `null` identically under equality, while + // `$exists` does not. So both readings of "no value" — a stored `null` + // and an absent key — answer the ruling, and the key-absent column, which + // already agreed, is unmoved. Measured on a real mongod 8.2.6 while this + // cell was pinned. case '$exists': - result[op] = value; + // Collected, not assigned: the merge at the end of this function has to + // know whether the key this lowers to is already spoken for. + presence = value === true ? { $ne: null } : { $eq: null }; break; case '$lte': { @@ -902,6 +935,34 @@ function translateFieldOperators( } } + // [#13195] Merge the lowered `$exists`, and do NOT let it clobber a sibling. + // + // The lowering reuses `$ne` / `$eq` — MongoDB keys an AUTHOR can also write + // on the same field. `{name: {$exists: true, $ne: 'b'}}` would assign `$ne` + // twice into one object, and whichever ran last would win: one constraint + // vanishes, and WHICH one depends on the author's key order. Measured before + // this guard existed: that filter emitted `{name: {$ne: 'b'}}` and the + // key-swapped `{name: {$ne: 'b', $exists: true}}` emitted `{name: {$ne: + // null}}` — one predicate, two different documents, neither carrying both + // constraints. + // + // Free key → merge inline (the common case, `$exists` alone on a field). + // Taken → hand the caller a `_presenceAnd` sentinel, which `translateCondition` + // lifts into its `$and` list, where both constraints survive. + // + // ⚠️ Scope: this guards the operator #13195 moved, and only it. The identical + // clobber is reachable today through `$null` (`$eq`/`$ne`) and `$between` + // (`$gte`/`$lte`/`$lt`) — measured, pre-existing, filed separately rather + // than half-fixed here. + if (presence) { + const presenceKey = Object.keys(presence)[0]!; + if (Object.prototype.hasOwnProperty.call(result, presenceKey)) { + result._presenceAnd = presence; + } else { + Object.assign(result, presence); + } + } + return result; } diff --git a/packages/drivers/driver-mongodb/src/mongodb-null-comparand-refusal.test.ts b/packages/drivers/driver-mongodb/src/mongodb-null-comparand-refusal.test.ts index d3869dfe9f..6d568a4045 100644 --- a/packages/drivers/driver-mongodb/src/mongodb-null-comparand-refusal.test.ts +++ b/packages/drivers/driver-mongodb/src/mongodb-null-comparand-refusal.test.ts @@ -139,7 +139,10 @@ describe('[#5347] driver-mongodb refuses a non-boolean $null comparand', () => { expect(translateFilter({ stage: 'won' })).toEqual({ stage: 'won' }); expect(translateFilter({ stage: { $in: ['won'] } })).toEqual({ stage: { $in: ['won'] } }); expect(translateFilter({ score: { $between: [1, 2] } })).toEqual({ score: { $gte: 1, $lte: 2 } }); - expect(translateFilter({ stage: { $exists: true } })).toEqual({ stage: { $exists: true } }); + // [#13195, ruled 2026-08-30] `$exists` lowers to the nullness test now — + // has-value, not key-presence. The point of this line is unchanged: the + // operator is still ACCEPTED here, and only `$null`'s comparand is refused. + expect(translateFilter({ stage: { $exists: true } })).toEqual({ stage: { $ne: null } }); expect(translateFilter({})).toEqual({}); });