feat(spec): refuse null members in list-comparand positions — $in/$nin members, $between bounds (#13357)

.changeset/null-list-comparand-refused.md

@@ -0,0 +1,44 @@
+---
+"@objectstack/spec": minor
+---
+
+feat(spec): refuse `null` in list-comparand positions — `$in` / `$nin` members and `$between` bounds (#13357, #13495)
+
+**BREAKING** accept-set narrowing on the filter contract, shipped as `minor`
+under the repo's launch-window convention for breaking changes. Maintainer
+ruling 2026-08-31 (option C): the contract refuses the shape loudly at the
+validation entrance, and the cross-backend divergence it used to reach becomes
+constructively unreachable — ⛔ no cross-backend alignment (#5299 stays
+declined), and the reference matcher's own answers for these shapes are sealed
+behind the refusal, not repaired.
+
+What is refused, and where:
+
+- **Runtime door** (`assertListComparandShapes`, run inside `parseFilterAST`
+ and at the engine seam on every verb): a `null` member of `$in` / `$nin`,
+ and a `null` `$between` bound, are refused with the platform envelope
+ (`INVALID_FILTER` / 400). Previously the shape reached the backends, where
+ the SQL family, the mingo path and the reference matcher answered it three
+ ways — the matcher even disagreed with itself across the two readings of
+ "no value" (#13357's table).
+- **Schema door** (`SetOperatorSchema` / `FieldOperatorsSchema`): a `null`
+ member is refused at parse time with a pointed message, the same
+ check-not-type-change mechanism as the #7596 `{ $field }` member refusal.
+ A `null` `$between` endpoint never parsed (the endpoint union is
+ `number | Date | string`); it now gets the pointed message instead of zod's
+ generic union text.
+
+The refusal text prescribes the ruling's explicit spelling: "one of […] OR has
+no value" is `{"$or": [{"$in": […]}, {"$null": true}]}`, and `{"$null": false}`
+is the has-a-value half. The carve-out is null-shaped and nothing wider:
+`$in: []` / `$nin: []` stay the declared predicates they are, every non-null
+member type keeps parsing (#5041's and #5234's member questions stand
+untouched), and `$eq: null` is a separate surface (#13494, ruled separable).
+
+**Migration.** A filter refused by the new checks was already answered
+inconsistently across backends, so it had no portable meaning to preserve.
+Spell the intent explicitly: `$or: [{$in: […]}, {$null: true}]` for
+"one of […] or empty", `{$null: false}` (or `$and` with it) for the
+has-a-value direction, and `$gte` / `$lte` for a half-open range.
+
+<!-- adr-0087: not-required (no-migration-prescription) A validity narrowing over existing keys: no key is removed, renamed or re-shaped, so there is no tombstone and nothing mechanical for `objectstack migrate meta` to rewrite. The refusal reaches an affected author at the parse/query site carrying the remedy; which explicit spelling matches the author's intent ($or with $null:true, $null:false, or a half-open range) is an authoring decision no migration entry can perform — and the ruling's evidence base measured zero authored occurrences of the refused shape. -->

content/docs/references/data/filter.mdx

@@ -123,8 +123,8 @@ Type: `[FilterArray](#filterarray)[]`
 
 | Property | Type | Required | Description |
 | :--- | :--- | :--- | :--- |
-| **$in** | `any[]` | optional | Membership list. Members are literal values of any type the column stores. A `{ $field }` reference is NOT a member shape: no backend resolves one inside a list — put it in a scalar comparison ($eq/$ne/$gt/$gte/$lt/$lte) instead. |
-| **$nin** | `any[]` | optional | Membership list. Members are literal values of any type the column stores. A `{ $field }` reference is NOT a member shape: no backend resolves one inside a list — put it in a scalar comparison ($eq/$ne/$gt/$gte/$lt/$lte) instead. |
+| **$in** | `any[]` | optional | Membership list. Members are literal values of any type the column stores. A `{ $field }` reference is NOT a member shape: no backend resolves one inside a list — put it in a scalar comparison ($eq/$ne/$gt/$gte/$lt/$lte) instead. null is NOT a member shape either: state absence explicitly with the null predicate — "one of […] OR has no value" is `{ "$or": [{ "$in": […] }, { "$null": true }] }`. |
+| **$nin** | `any[]` | optional | Membership list. Members are literal values of any type the column stores. A `{ $field }` reference is NOT a member shape: no backend resolves one inside a list — put it in a scalar comparison ($eq/$ne/$gt/$gte/$lt/$lte) instead. null is NOT a member shape either: state absence explicitly with the null predicate — "one of […] OR has no value" is `{ "$or": [{ "$in": […] }, { "$null": true }] }`. |
 
 
 ---

packages/drivers/driver-memory/src/memory-null-list-member-unreachable.test.ts

@@ -0,0 +1,112 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#13357] Ruling point 3's NEGATIVE pin, matcher side: a refused null list
+ * member cannot reach this package's reference matcher.
+ *
+ * # What was ruled (2026-08-31, option C)
+ *
+ * #13357 measured that the reference matcher answers `$in: [null]` /
+ * `$nin: [null]` differently across the two readings of "no value" (a stored
+ * `null` vs an absent key) while `$null` / `$ne: null` agree — and that the
+ * SQL family answers the same filters a third way. The ruling REFUSES the
+ * shape at the contract's validation entrance (`@objectstack/spec`,
+ * `assertListComparandShapes`, run inside `parseFilterAST` and at the engine
+ * seam) instead of aligning the backends: the divergence becomes
+ * constructively unreachable, ⛔ deliberately not repaired (「⛔ 不单独修一个
+ * 到不了的路径」), so NOTHING in this file asserts what the matcher would
+ * have answered. `memory-matcher-null-value-and-comparand.test.ts` keeps
+ * those arms deliberately absent for the same reason.
+ *
+ * # What this file pins, and its honest boundary
+ *
+ * A direct caller of this driver — this repo's own conformance suites, an
+ * embedder — compiles its filter with `parseFilterAST` and hands the result
+ * over (`filter-comparand-shape.ts`'s #9228 section is the ruling that put
+ * the gate on that face for exactly this caller). This file drives that
+ * pipeline end to end and pins that for every refused shape it ABORTS at the
+ * compile face, on BOTH fixture readings, before any row is consulted: the
+ * evaluation step is provably never reached because the compile step throws.
+ * The engine half (every verb, driver-call witness) is pinned in
+ * `@objectstack/objectql`'s `engine-filter-array-lowering.test.ts`; the
+ * wire/protocol face runs the same `parseFilterAST`.
+ *
+ * The boundary, stated rather than hidden: `match()` and
+ * `InMemoryDriver.find()` remain plain library functions — a caller that
+ * skips the compile face meets only this package's own
+ * `assertFilterConditionShape`, which is deliberately NOT extended to the
+ * null-member rule (⛔ 不做跨后端对齐工程). That boundary is the same one
+ * every #5869 refusal has had since #9228, and it is not widened here.
+ */
+
+import { describe, it, expect } from 'vitest';
+import { parseFilterAST } from '@objectstack/spec/data';
+
+import { match } from './memory-matcher.js';
+
+type Refusal = Error & { code?: string; status?: number };
+
+/** The card's own fixture, in both readings of "no value" (#13357). */
+const NULLED_ROWS: Array<Record<string, unknown>> = [
+ { id: '1', name: 'a' },
+ { id: '3', name: null },
+];
+const MISSING_ROWS: Array<Record<string, unknown>> = [
+ { id: '1', name: 'a' },
+ { id: '3' },
+];
+
+/**
+ * The direct-caller pipeline, exactly as the module note describes it: compile
+ * first, evaluate second. The refusal has to land in step one — if compile
+ * returns, the matcher HAS been reached and the pin below fails on the
+ * sentinel rather than on a missing throw.
+ */
+function compileThenMatch(rows: Array<Record<string, unknown>>, where: unknown): string[] {
+ const condition = parseFilterAST(where);
+ return rows.filter((row) => match(row, condition)).map((row) => String(row.id));
+}
+
+const refusalOf = (run: () => unknown): Refusal => {
+ try {
+ run();
+ } catch (e) {
+ return e as Refusal;
+ }
+ throw new Error('expected the compile face to refuse this filter, but it returned');
+};
+
+describe('[#13357] a refused null list member cannot reach the matcher (ruled 2026-08-31)', () => {
+ it.each([
+ ['$in: [null]', { name: { $in: [null] } }],
+ ['$nin: [null]', { name: { $nin: [null] } }],
+ ['$between: [null, null]', { name: { $between: [null, null] } }],
+ ['$between: [null, max]', { name: { $between: [null, 'z'] } }],
+ ['$between: [min, null]', { name: { $between: ['a', null] } }],
+ ])('%s aborts at the compile face on BOTH readings of "no value"', (_label, where) => {
+ // Record-independent by construction — the compile face never sees a row —
+ // so the two readings that split the matcher (#13357's table) cannot even
+ // be posed. Driving both anyway is the point of the pin: neither fixture
+ // gets an answer, so there is no divergence left to observe.
+ for (const rows of [NULLED_ROWS, MISSING_ROWS]) {
+ const err = refusalOf(() => compileThenMatch(rows, where));
+ expect(err.code, _label).toBe('INVALID_FILTER');
+ expect(err.status, _label).toBe(400);
+ }
+ });
+
+ it('the pipeline itself is real — a legal list compiles and the matcher answers', () => {
+ // Positive control: without it, the refusals above would also "pass" if
+ // compileThenMatch were broken outright.
+ expect(compileThenMatch(NULLED_ROWS, { name: { $in: ['a'] } })).toEqual(['1']);
+ expect(compileThenMatch(MISSING_ROWS, { name: { $nin: ['a'] } })).toEqual(['3']);
+ });
+
+ it('an EMPTY list still passes the same face — the refusal is null-shaped, not list-shaped', () => {
+ // `$in: []` / `$nin: []` are declared predicates ("matches nothing" /
+ // "matches everything") and PR #13630 pins them downstream; the carve-out
+ // must not catch them.
+ expect(compileThenMatch(NULLED_ROWS, { name: { $in: [] } })).toEqual([]);
+ expect(compileThenMatch(NULLED_ROWS, { name: { $nin: [] } })).toEqual(['1', '3']);
+ });
+});

packages/objectql/src/engine-filter-array-lowering.test.ts

@@ -483,6 +483,69 @@ describe('Door 2 lowers FilterArray to FilterCondition before the driver (#5158)
  expect(err.message).toMatch(/where\.amount\.\$between/);
  });
 
+ // ── [#13357] the null carve-out, ruled 2026-08-31: refused at this seam, ──
+ // ── so the drivers' three-way null-member divergence is UNREACHABLE ──────
+ //
+ // Ruling point 3's negative pin, engine half: the witness is the recording
+ // driver's call log, not the thrown envelope alone — an envelope asserted
+ // without the log would also pass if a DRIVER had thrown it, i.e. if the
+ // refusal had not preceded the dispatch, which is the one thing these
+ // assertions exist to prove. The compile-face half (`parseFilterAST`, both
+ // input forms) is pinned in `@objectstack/spec`'s
+ // `filter-comparand-shape.test.ts`; the matcher-side statement lives in
+ // driver-memory's `memory-null-list-member-unreachable.test.ts`. ⛔ Nothing
+ // here asserts what any backend WOULD have answered — the divergence is
+ // sealed, not reconciled (#5299 stays declined).
+
+ it.each([
+ ['$in: [null]', { stage: { $in: [null] } }],
+ ['$nin: [null]', { stage: { $nin: [null] } }],
+ ['$between: [null, null]', { amount: { $between: [null, null] } }],
+ ['$between: [null, max]', { amount: { $between: [null, 20] } }],
+ ])('a null list member is refused on EVERY verb before any driver call — %s', async (_l, where) => {
+ await expect(engine.find('deal', { where }))
+ .rejects.toMatchObject({ status: 400, code: 'INVALID_FILTER' });
+ await expect(engine.findOne('deal', { where }))
+ .rejects.toMatchObject({ status: 400, code: 'INVALID_FILTER' });
+ await expect(engine.count('deal', { where } as unknown as EngineCountOptions))
+ .rejects.toMatchObject({ status: 400, code: 'INVALID_FILTER' });
+ await expect(engine.aggregate('deal', {
+ where: where as unknown as EngineAggregateOptions['where'],
+ groupBy: ['stage'],
+ aggregations: [{ function: 'count', field: 'id', alias: 'n' }],
+ })).rejects.toMatchObject({ status: 400, code: 'INVALID_FILTER' });
+ await expect(engine.update('deal', { amount: 1 }, { where, multi: true } as any))
+ .rejects.toMatchObject({ status: 400, code: 'INVALID_FILTER' });
+ await expect(engine.delete('deal', { where, multi: true } as any))
+ .rejects.toMatchObject({ status: 400, code: 'INVALID_FILTER' });
+ // The negative half: refused BEFORE the store — no read, no write, no row
+ // moved. (The count() control below adds its own read, so it runs after.)
+ expect(reads).toHaveLength(0);
+ expect(writes).toHaveLength(0);
+ expect(await engine.count('deal')).toBe(3);
+ });
+
+ it('the null-member refusal is not vacuous — the same list WITHOUT null reaches the driver', async () => {
+ // Positive control for the zero-call reading above: one member removed,
+ // same operator, same field, and the dispatch happens.
+ const rows = await engine.find('deal', { where: { stage: { $in: ['won'] } } });
+ expect(reads).toHaveLength(1);
+ expect(lastWhere()).toEqual({ stage: { $in: ['won'] } });
+ expect(rows.map((r: any) => r.id).sort()).toEqual(['d1', 'd3']);
+ });
+
+ it('a nested null member is refused at its own path, engine prefix and all', async () => {
+ const err = await engine.find(
+ 'deal',
+ { where: { $or: [{ stage: { $nin: [null] } }] } },
+ ).then(() => null, (e: any) => e);
+ expect(err?.status).toBe(400);
+ expect(err?.code).toBe('INVALID_FILTER');
+ expect(err.message).toMatch(/^find\('deal'\): /);
+ expect(err.message).toContain('where.$or[0].stage.$nin[0]');
+ expect(reads).toHaveLength(0);
+ });
+
  // ── what must KEEP working: the declared list shapes ───────────────────
 
  it('a proper list comparand still reaches the driver untouched', async () => {

packages/spec/src/data/filter-comparand-shape.test.ts

@@ -98,6 +98,68 @@ describe('the list-comparand shape door (#5869) runs inside parseFilterAST (#922
  expect(err.status).toBe(400);
  });
 
+ // ── the null carve-out, ruled 2026-08-31 (#13357; $between is #13495) ──
+
+ it.each([
+ ['$in, lowered array form', [['stage', 'in', [null]]]],
+ ['$in, object passthrough', { stage: { $in: [null] } }],
+ ['$nin, lowered array form', [['stage', 'not_in', [null]]]],
+ ['$nin, object passthrough', { stage: { $nin: [null] } }],
+ ['$in with a real neighbour', { stage: { $in: ['won', null] } }],
+ ])('refuses a null list MEMBER — %s', (_label, where) => {
+ const err = refusalOf(() => parseFilterAST(where));
+ expect(err.code).toBe(StandardErrorCode.enum.INVALID_FILTER);
+ expect(err.status).toBe(400);
+ });
+
+ it.each([
+ ['[null, null]', { at: { $between: [null, null] } }],
+ ['[null, max]', { at: { $between: [null, '2026-07-15'] } }],
+ ['[min, null]', { at: { $between: ['2026-07-01', null] } }],
+ ])('refuses a null $between BOUND — %s', (_label, where) => {
+ const err = refusalOf(() => parseFilterAST(where));
+ expect(err.code).toBe(StandardErrorCode.enum.INVALID_FILTER);
+ expect(err.status).toBe(400);
+ });
+
+ it('the null-member refusal prescribes the ruling\'s explicit spelling', () => {
+ // 2026-08-31: 「等于 X 或为空」的合法拼法是显式的 $or + $null — the
+ // refusal must spell it out, in both halves, and still name operator,
+ // field, position and authoring spellings (the #5346/#5348 contract).
+ const err = refusalOf(() => parseFilterAST({ stage: { $nin: [null] } }));
+ expect(err.message).toMatch(/^Operator "\$nin" on field "stage"/);
+ expect(err.message).toContain('where.stage.$nin[0]');
+ expect(err.message).toContain('{"$or": [{"stage": {"$in": […]}}, {"stage": {"$null": true}}]}');
+ expect(err.message).toContain('{"$null": false}');
+ expect(err.message).toMatch(/Authoring spellings: nin, not_in, notin/);
+ expect(err.message).toMatch(/UNFILTERED result set/);
+ });
+
+ it('the null-bound refusal points at the offending index and the working alternatives', () => {
+ const err = refusalOf(() => parseFilterAST({ at: { $between: ['2026-07-01', null] } }));
+ expect(err.message).toMatch(/^Operator "\$between" on field "at" requires two non-null bounds/);
+ expect(err.message).toContain('where.at.$between[1]');
+ expect(err.message).toContain('"$gte"/"$lte"');
+ expect(err.message).toContain('{"at": {"$null": true}}');
+ expect(err.message).toMatch(/UNFILTERED result set/);
+ });
+
+ it('a null member is refused at its own path inside $and / $or / $not too', () => {
+ expect(refusalOf(() => parseFilterAST({ $not: { stage: { $in: [null] } } })).message)
+ .toContain('where.$not.stage.$in[0]');
+ expect(refusalOf(() => parseFilterAST({ $or: [{ stage: { $nin: [null] } }] })).message)
+ .toContain('where.$or[0].stage.$nin[0]');
+ });
+
+ it('refuses ONLY null — falsy and empty-ish members are values, not absence', () => {
+ // The carve-out is null-shaped and nothing wider: #5041's and #5234's
+ // member questions stand untouched, and every falsy VALUE keeps working.
+ expect(parseFilterAST({ n: { $in: [0, false, ''] } })).toEqual({ n: { $in: [0, false, ''] } });
+ expect(parseFilterAST({ n: { $nin: [0, false, ''] } })).toEqual({ n: { $nin: [0, false, ''] } });
+ expect(parseFilterAST({ at: { $between: ['', ''] } })).toEqual({ at: { $between: ['', ''] } });
+ expect(parseFilterAST({ n: { $between: [0, 0] } })).toEqual({ n: { $between: [0, 0] } });
+ });
+
  // ── the wording contract (#5346 / #5348), unchanged by the move ────────
 
  it('names the operator, the field, what arrived, where, and the fix', () => {
@@ -134,6 +196,13 @@ describe('the list-comparand shape door (#5869) runs inside parseFilterAST (#922
  [['stage', 'not_in', 'won']],
  [['stage', 'in', 'won']],
  [['amount', 'between', 5]],
+ // The 2026-08-31 null carve-out (#13357/#13495): the prescribed $or +
+ // $null spelling makes these the LONGEST messages this door assembles,
+ // so they live inside the same unrelaxed bound.
+ { stage: { $in: [null] } },
+ { stage: { $nin: [null] } },
+ { close_date: { $between: [null, null] } },
+ { close_date: { $between: ['2026-07-01', null] } },
  ]) {
  const err = refusalOf(() => parseFilterAST(where, "find('deal')"));
  expect(err.message.length, JSON.stringify(where)).toBeLessThan(500);