From 6a65d142dc4f47a4ebcab437f0d45c946dba0c3e Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 17:42:28 +0000 Subject: [PATCH 1/4] feat(spec,drivers,objectql,analytics,formula): $icontains reaches every JS evaluation face (#6520) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #5702 implemented `$icontains` on the SQL family and correctly left the spec's `FILTER_OPERATORS` alone. This adds the operator to that array and gives every remaining evaluation face an arm, in ONE change, because those two steps cannot be separated: the array is a runtime allowlist that `driver-memory`'s shape gate derives from, so admitting the name while the matcher had no arm turns a loud refusal into a silently dropped predicate — measured on #5701's branch, and a dropped predicate WIDENS, which on an RLS read scope is a permission bypass (#3948). Faces changed, under the sanctioned one-off exception to the #5499 freeze (maintainer ruling 2026-08-08, semantic parity only): - spec: `FILTER_OPERATORS` admits `$icontains`; the vocabulary pin's diff set goes empty; new shared fold `foldAsciiCase` / `asciiCaseInsensitiveContains` / `asciiCaseInsensitiveRegexSource` - driver-memory: query path, reference matcher and analytics (cube) face - driver-mongodb: an ASCII-only `$regex`, never `$options: 'i'` - objectql `having`, `@objectstack/formula` `matchesFilterCondition` - service-analytics: the normalizer plus all three SQL compilers The fold is ASCII-only by ruling (#4706 Q1 = A) and lives in ONE definition, because both obvious per-package spellings are wrong the same way: `toLowerCase()` folds all of Unicode, and so does a `RegExp` with the `i` flag. The pattern-binding faces emit one `[Aa]` class per ASCII letter and pass no flags. Comparand rules match driver-sql's: literal, and refused when empty or non-string. The formula face keeps its silent fail-closed `false` for unknown operators — decided, not inherited, and documented on `matches-filter.ts`: it governs a write-side check where an unevaluable condition denies rather than widens, and callers rely on it being a total predicate. No operator the spec DECLARES is answered that way any more, which was the #6993 census's measured defect. Both `FILTER_TEXT_CASES` DEBT rows survive, now naming one open requirement each (the `$contains` family's Unicode fold, #6682) instead of two. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PiRUoQkTSBBmpyXBY3cVn2 --- .changeset/icontains-reaches-every-js-face.md | 64 ++++++ .../docs/protocol/objectql/query-syntax.mdx | 26 ++- content/docs/references/data/filter.mdx | 2 +- .../driver-memory/src/filter-refusal.ts | 57 ++++- .../driver-memory/src/memory-analytics.ts | 36 +++ .../driver-memory/src/memory-driver.ts | 23 +- .../src/memory-icontains.test.ts | 164 +++++++++++++ .../driver-memory/src/memory-matcher.ts | 21 +- .../driver-mongodb/src/mongodb-filter.ts | 60 +++++ .../src/mongodb-icontains.test.ts | 127 +++++++++++ ...te-transport-node-operator-refusal.test.ts | 5 +- .../src/matches-filter-icontains.test.ts | 114 ++++++++++ packages/formula/src/matches-filter.ts | 67 +++++- packages/objectql/src/having-filter.ts | 26 ++- .../objectql/src/having-icontains.test.ts | 83 +++++++ .../like-metacharacter-escape.test.ts | 59 +++-- .../objectql-echo-operator-coverage.test.ts | 15 +- .../service-analytics/src/like-pattern.ts | 54 ++++- .../service-analytics/src/read-scope-sql.ts | 27 ++- .../src/strategies/filter-normalizer.ts | 4 + .../src/strategies/native-sql-strategy.ts | 15 +- .../src/strategies/objectql-strategy.ts | 17 +- packages/spec/export-origins/data.json | 3 + .../spec/src/data/filter-ascii-fold.test.ts | 112 +++++++++ .../data/filter-operator-vocabulary.test.ts | 52 +++-- packages/spec/src/data/filter.test.ts | 2 + packages/spec/src/data/filter.zod.ts | 215 ++++++++++++++---- scripts/check-driver-conformance.mjs | 44 ++-- 28 files changed, 1350 insertions(+), 144 deletions(-) create mode 100644 .changeset/icontains-reaches-every-js-face.md create mode 100644 packages/drivers/driver-memory/src/memory-icontains.test.ts create mode 100644 packages/drivers/driver-mongodb/src/mongodb-icontains.test.ts create mode 100644 packages/formula/src/matches-filter-icontains.test.ts create mode 100644 packages/objectql/src/having-icontains.test.ts create mode 100644 packages/spec/src/data/filter-ascii-fold.test.ts diff --git a/.changeset/icontains-reaches-every-js-face.md b/.changeset/icontains-reaches-every-js-face.md new file mode 100644 index 0000000000..07563b82c3 --- /dev/null +++ b/.changeset/icontains-reaches-every-js-face.md @@ -0,0 +1,64 @@ +--- +"@objectstack/spec": minor +"@objectstack/driver-memory": minor +"@objectstack/driver-mongodb": minor +"@objectstack/objectql": minor +"@objectstack/formula": minor +"@objectstack/service-analytics": minor +--- + +feat(spec,drivers,objectql,analytics,formula): `$icontains` reaches every JS evaluation face (#6520) + +The other half of #5702. That change implemented `$icontains` on the SQL family +and correctly left the spec's `FILTER_OPERATORS` alone; this one adds the +operator to that array and gives every remaining evaluation face an arm, in ONE +change, because those two steps cannot be separated. + +**Why one PR.** `FILTER_OPERATORS` is not a word list, it is a runtime allowlist: +`driver-memory`'s shape gate derives from it, and its matcher's `default:` arm +assumes the gate already refused anything unimplemented. Measured on a branch +that added the name early (#5701): the gate stopped refusing, the matcher fell +through, and `match({ name: 'zzz' }, { name: { $icontains: 'acme' } })` returned +`true` — the predicate silently dropped, every row matched. A dropped predicate +does not narrow a query, it WIDENS it, and on an RLS read scope that is a +permission bypass rather than a degraded feature (#3948). So the word list +travels with the evaluators or not at all. + +**What now answers it**, all folding the same domain: `driver-memory` (query +path, reference matcher, and the analytics/cube face), `driver-mongodb`, +`objectql`'s `having`, `@objectstack/formula`'s `matchesFilterCondition` (the RLS +write-side `check`), and `service-analytics`' three SQL compilers (the RLS +lowering, the native-SQL strategy, and the `/analytics/sql` echo). + +**The fold is ASCII-only, and that is the contract, not an implementation +detail** (#4706 Q1 = A). `$icontains: 'café'` does not match `CAFÉ`. Every face +reads one shared definition — `foldAsciiCase` / +`asciiCaseInsensitiveContains` / `asciiCaseInsensitiveRegexSource`, new exports +on `@objectstack/spec/data` — because the two obvious per-package spellings are +both wrong in the same direction: `toLowerCase()` folds the whole Unicode range, +and so does a `RegExp` built with the `i` flag. SQLite folds ASCII only and three +of the five drivers are SQLite underneath, so a Unicode fold on a JS face would +re-open exactly the divergence the ruling closed. The pattern-binding faces +(mingo, mongo) therefore emit one `[Aa]` character class per ASCII letter and +pass NO flags; mongo's `$icontains` is the one arm in its family that does not +set `$options: 'i'`. + +The comparand keeps the rules its SQL twin has: matched LITERALLY (`%`, `_` and +regex metacharacters are ordinary characters), and refused when empty or +non-string — an empty comparand matches every row, which is a predicate that +constrains nothing. + +**User-visible effect.** A filter using `$icontains` now behaves the same on the +in-memory double and on SQL, so an app whose tests run on one and whose +production runs the other stops getting two answers from one filter. Downstream, +#5814 (better-auth `Where.mode: 'insensitive'`) no longer hits a 400 on the +memory double. + +Not changed, and still tracked: the `$contains` family still folds Unicode on +`driver-memory`'s query path and `driver-mongodb` (#6682) — both remain DEBT rows +in `scripts/check-driver-conformance.mjs`, now naming one open requirement each +instead of two. `formula`'s unknown-operator posture stays a silent, fail-closed +`false` (it governs a write-side check, where an unevaluable condition denies +rather than widens); the decision and its limits are documented on +`matches-filter.ts`, and no operator the spec DECLARES is answered that way any +more. diff --git a/content/docs/protocol/objectql/query-syntax.mdx b/content/docs/protocol/objectql/query-syntax.mdx index d284878771..e1a112dff7 100644 --- a/content/docs/protocol/objectql/query-syntax.mdx +++ b/content/docs/protocol/objectql/query-syntax.mdx @@ -291,15 +291,23 @@ not `LIKE` wildcards, and `.` / `*` / `+` are ordinary characters, not regex metacharacters — `{ name: { $icontains: 'a.b' } }` matches `a.b` and not `axb`. - **Status:** the case rules above are the protocol's declaration as of - `@objectstack/spec` 18. The backend lowerings that deliver them — making SQLite's - and turso's `LIKE` case-exact, dropping MongoDB's hardcoded `$options: 'i'`, and - implementing `$icontains` everywhere — are tracked by - [#5702](https://github.com/objectstack-ai/objectstack/issues/5702). Until it - lands, a backend that has not been aligned refuses `$icontains` outright rather - than answering it approximately, and `$contains` still follows its dialect. The - shared standard both halves are measured against is `FILTER_TEXT_CASES` - (`@objectstack/spec/data`). + **Status:** `$icontains` is implemented on **every** backend and every evaluation + face, and folds the same ASCII domain on all of them — + [#5702](https://github.com/objectstack-ai/objectstack/issues/5702) did the SQL + family and [#6520](https://github.com/objectstack-ai/objectstack/issues/6520) did + the rest (the in-memory driver's query, matcher and analytics faces, MongoDB, + ObjectQL's `having`, the RLS write-side `check`, and the analytics SQL + compilers). A filter using it means the same thing whether your tests run on the + in-memory double or your production runs SQL. + + One half of the case rules above is still landing: `$contains` / + `$startsWith` / `$endsWith` / `$notContains` are case-**sensitive** by ruling and + are so on the SQL family, but the in-memory driver's query path and MongoDB still + fold them over the whole Unicode range — + [#6682](https://github.com/objectstack-ai/objectstack/issues/6682). Until that + lands, prefer `$icontains` when you *want* a fold rather than relying on + `$contains` being loose on those two backends. The shared standard both halves + are measured against is `FILTER_TEXT_CASES` (`@objectstack/spec/data`). ### `$regex` — removed diff --git a/content/docs/references/data/filter.mdx b/content/docs/references/data/filter.mdx index 865e57234d..991f370cef 100644 --- a/content/docs/references/data/filter.mdx +++ b/content/docs/references/data/filter.mdx @@ -147,7 +147,7 @@ Type: `[FilterArray](#filterarray)[]` | **$notContains** | `string` | optional | | | **$startsWith** | `string` | optional | | | **$endsWith** | `string` | optional | | -| **$icontains** | `string` | optional | Contains substring, ignoring case — but ONLY ASCII case (A-Z against a-z). Every other character compares literally, so "café" does NOT match "CAFÉ" and "москва" does not match "МОСКВА". The domain is ASCII because that is the one fold all five backends can deliver: SQLite (and therefore turso and sqlite-wasm) folds ASCII only, so a Unicode promise here would be a guarantee three of the five could not keep. The comparand is matched LITERALLY — "%", "_" and regex metacharacters are ordinary characters, not wildcards. Case-SENSITIVE containment is $contains. [#5701 declared it; #5702 lowered it on the SQL family (driver-sql, driver-sqlite-wasm, driver-turso on both transports). driver-memory and driver-mongodb still REFUSE it with INVALID_FILTER / 400, so a filter using it is not portable across backends yet — #6520.] | +| **$icontains** | `string` | optional | Contains substring, ignoring case — but ONLY ASCII case (A-Z against a-z). Every other character compares literally, so "café" does NOT match "CAFÉ" and "москва" does not match "МОСКВА". The domain is ASCII because that is the one fold all five backends can deliver: SQLite (and therefore turso and sqlite-wasm) folds ASCII only, so a Unicode promise here would be a guarantee three of the five could not keep. The comparand is matched LITERALLY — "%", "_" and regex metacharacters are ordinary characters, not wildcards. Case-SENSITIVE containment is $contains. [#5701 declared it; #5702 lowered it on the SQL family (driver-sql, driver-sqlite-wasm, driver-turso on both transports); #6520 lowered it on every JS evaluation face, so it is portable across every backend the platform ships.] | --- diff --git a/packages/drivers/driver-memory/src/filter-refusal.ts b/packages/drivers/driver-memory/src/filter-refusal.ts index ad20c6dc7c..b80f211293 100644 --- a/packages/drivers/driver-memory/src/filter-refusal.ts +++ b/packages/drivers/driver-memory/src/filter-refusal.ts @@ -163,12 +163,24 @@ export function emptyFieldConstraintError(field: string, path: string): Error { * they are refused here like any other undeclared operator, with the spec's * prescription attached (see {@link retiredFilterOperatorError}). * - * Note what this does NOT do: it does not add `$icontains`. That name is - * declared by `StringOperatorSchema` but deliberately absent from - * `FILTER_OPERATORS` (#5701), and this set is derived, so this driver refuses it - * — fail-closed, an unimplemented capability rather than a silent widening. The - * `$icontains` implementation for the JS faces is #5499-frozen; see the - * `driver-memory` row of `scripts/check-driver-conformance.mjs`. + * ## [#6520] `$icontains` arrives here by DERIVATION, and that is the risk + * + * This set is `FILTER_OPERATORS` itself, so #6520 adding `$icontains` to the + * spec admitted it here with no edit to this file. That is the property #5701 + * measured and warned about: while the matcher had no arm, admission alone + * turned a loud refusal into `match({ name: 'zzz' }, { name: { $icontains: + * 'acme' } }) === true` — the predicate dropped, every row matched, which on an + * RLS read scope is a permission bypass rather than a degraded filter (#3948). + * + * So the arms and the word list HAD to land in one PR, and #6520 did that: + * `memory-matcher.ts` and `memory-driver.ts` both carry a `$icontains` case, and + * `memory-analytics.ts` lowers it too. Re-verified by deleting the matcher's arm + * on the #6520 branch — with the name admitted, the reference matcher answered + * EVERY row, which is the measurement, not a prediction. + * + * The lesson for the next operator is the ordering rather than this name: an + * entry in `FILTER_OPERATORS` is a claim that this driver evaluates it, and this + * file will make that claim on the spec's behalf whether or not it is true. * * Everything else is refused. That includes the mingo operators this driver used * to hand through by accident (`$elemMatch`, `$size`, `$type`, `$mod`, `$where`, @@ -624,6 +636,13 @@ function assertFieldConstraintShape( if (op === '$null' && typeof spec[op] !== 'boolean') { throw nonBooleanNullComparandError(field, spec[op], `${path}.$null`); } + // [#6520] `$icontains`' comparand is a NON-EMPTY string, the third + // comparand-shape rule and the twin of `driver-sql`'s + // `icontainsComparandError` — deliberately the same two rejections in one + // check, because they are one mistake at one position. + if (op === '$icontains' && (typeof spec[op] !== 'string' || spec[op] === '')) { + throw icontainsComparandError(field, spec[op], `${path}.$icontains`); + } } // [#5702] The `$options`-without-`$regex` companion check that stood here is // GONE. It was needed while `$options` was an allowlisted MODIFIER — a key the @@ -633,6 +652,32 @@ function assertFieldConstraintShape( // {@link retiredFilterOperatorError}. } +/** + * [#6520] `$icontains` received a comparand that is not a non-empty string. + * + * Word for word `driver-sql`'s `icontainsComparandError`, and deliberately so: + * #3948 made the backends agree that an uncompilable filter is a refusal rather + * than a silent match-everything, and a suite that swaps this driver for SQL has + * to see the same refusal for the same input. Two rejections, one constructor, + * because they are one mistake at the comparand position: + * + * - **non-string** — `StringOperatorSchema` declares `$icontains: z.string()`, + * so coercing `42` to `"42"` would answer a query nobody wrote; + * - **empty string** — every row contains the empty substring, so the predicate + * constrains nothing. A dropped predicate WIDENS a result set, and on an RLS + * read scope that is a permission bypass rather than a degraded filter. + */ +function icontainsComparandError(field: string, value: unknown, path: string): Error { + const shown = typeof value === 'string' ? `""` : JSON.stringify(value) ?? String(value); + return unsupportedFilterError( + `Operator "$icontains" on field "${field}" at ${path} requires a NON-EMPTY string comparand, ` + + `received ${shown}. "$icontains" is a case-insensitive LITERAL substring search, so its ` + + `comparand is the text to look for — an empty one matches every row (a predicate that ` + + `constrains nothing), and a non-string one would have to be coerced into text this query ` + + `never asked for.`, + ); +} + /** * [#5328] `$between`'s comparand: a two-element `[min, max]` array. * diff --git a/packages/drivers/driver-memory/src/memory-analytics.ts b/packages/drivers/driver-memory/src/memory-analytics.ts index de9e8b5840..776eacdd06 100644 --- a/packages/drivers/driver-memory/src/memory-analytics.ts +++ b/packages/drivers/driver-memory/src/memory-analytics.ts @@ -2,6 +2,8 @@ import type { IAnalyticsService, AnalyticsResult, CubeMeta } from '@objectstack/spec/contracts'; import type { Cube, AnalyticsQuery } from '@objectstack/spec/data'; +// [#6520] `$icontains`' ASCII-only fold, from the spec's one definition. +import { asciiCaseInsensitiveRegexSource } from '@objectstack/spec/data'; import type { InMemoryDriver } from './memory-driver.js'; import { Logger, createLogger, nextUtcCalendarDay } from '@objectstack/core'; import { @@ -52,6 +54,13 @@ const MONGO_TO_CUBE_OPERATOR = Object.freeze({ $nin: 'notIn', $contains: 'contains', $notContains: 'notContains', + // [#6520] This face lowers `$icontains` too, so the analytics/cube surface + // answers it like the driver's other two. Leaving it out would have been a + // LOUD refusal (`uncompilableFieldOperatorError` — "declared, but this face + // cannot compile it"), not a silent drop; it is added because the cube + // pipeline can express it, and one driver answering one operator two ways by + // entry point is the divergence class #5374 closed for `$contains`. + $icontains: 'icontains', $exists: 'set', } as const); @@ -148,6 +157,18 @@ interface MongoPredicateInput { * DRIVER's own rule (`filterSubstringPattern`) rather than re-derived here. */ readonly substring: (value: unknown) => RegExp; + /** + * [#6520] A comparand as an ASCII-case-insensitive literal-substring pattern — + * `$icontains`' fold, which is NOT {@link substring}'s. + * + * The two are deliberately separate functions rather than one with a flag. + * `substring` folds the whole Unicode range (the driver's `i` flag), which is + * the open defect #6682 tracks for the `$contains` family on this face; this + * one folds `A-Z` only, which is what the protocol says `$icontains` means + * (#4706 Q1 = A). Collapsing them would silently give one of the two operators + * the other's answer. + */ + readonly asciiSubstring: (value: unknown) => RegExp; } type MongoPredicateBuilder = (input: MongoPredicateInput) => Record; @@ -224,6 +245,11 @@ const CUBE_OPERATOR_TO_MONGO_PREDICATE: Readonly ({ $nin: [...comparands] }), // A pattern, not a comparand: `raw`, and the driver's own substring rule. contains: ({ raw, substring }) => ({ $regex: substring(raw[0]) }), + // [#6520] The case-INSENSITIVE twin, folding ASCII and nothing else. It takes + // `asciiSubstring`, not `substring`: the neighbour above folds Unicode, so + // reusing it here would answer `CAFÉ` for `café` on this face while the SQL + // family answered no rows — the divergence #6520 closed. + icontains: ({ raw, asciiSubstring }) => ({ $regex: asciiSubstring(raw[0]) }), // The fix this issue is about. `{$not: }` constrains nothing; the // negation has to wrap a pattern, which is exactly what the live query path // builds for `$notContains` (`memory-driver.ts` `normalizeFieldOperators`). @@ -320,6 +346,9 @@ export class MemoryAnalyticsService implements IAnalyticsService { comparands: this.comparandsFor(cube, filter.member, filter.values), raw: filter.values, substring: (value) => this.driver.filterSubstringPattern(value), + // [#6520] `$icontains`' fold, from the spec's shared definition rather + // than from the driver's Unicode-folding `filterSubstringPattern`. + asciiSubstring: (value) => new RegExp(asciiCaseInsensitiveRegexSource(String(value))), }); } if (Object.keys(matchStage).length > 0) { @@ -929,6 +958,13 @@ export class MemoryAnalyticsService implements IAnalyticsService { 'notEquals': '!=', 'contains': 'LIKE', 'notContains': 'NOT LIKE', + // [#6520] Needed because the `|| '='` fallback below is not a default, it + // is a wrong ANSWER: without this row `icontains` would render as `=`, an + // EQUALITY, in a statement offered to the author as a description of a + // containment query. `LIKE` is also the semantically right construct here + // — this exit emits SQLite-shaped SQL, and SQLite's `LIKE` folds ASCII + // only, which is exactly `$icontains`' domain (#4706 Q1 = A). + 'icontains': 'LIKE', 'gt': '>', 'gte': '>=', 'lt': '<', diff --git a/packages/drivers/driver-memory/src/memory-driver.ts b/packages/drivers/driver-memory/src/memory-driver.ts index c175776ec7..5177d9ceee 100644 --- a/packages/drivers/driver-memory/src/memory-driver.ts +++ b/packages/drivers/driver-memory/src/memory-driver.ts @@ -1,7 +1,11 @@ // Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. import type { DriverOptions } from '@objectstack/spec/data'; -import { canonicalAstOperator } from '@objectstack/spec/data'; +// [#6520] `asciiCaseInsensitiveRegexSource` is `$icontains`' fold, defined once +// in the spec: this face hands a PATTERN to mingo rather than comparing two +// strings, so the fold has to live in the pattern source. See its docblock for +// why an `i` flag is the wrong tool. +import { canonicalAstOperator, asciiCaseInsensitiveRegexSource } from '@objectstack/spec/data'; import type { DriverQuery, IDataDriver } from '@objectstack/spec/contracts'; import { Logger, createLogger, nextUtcCalendarDay } from '@objectstack/core'; import { Query, Aggregator } from 'mingo'; @@ -965,6 +969,23 @@ export class InMemoryDriver implements IDataDriver { case '$endsWith': regexConditions.push({ $regex: new RegExp(`${this.escapeRegex(val)}$`, 'i') }); break; + // [#6520] `$icontains` — case-insensitive over ASCII and NOTHING else. + // + // Note what this arm does NOT do, because every neighbour above does it: + // it never passes the `i` flag. That flag is the FULL Unicode fold, so + // it would match `CAFÉ` against `café` — the answer the SQL family + // cannot give (SQLite folds ASCII only) and therefore the one the + // protocol forbids (#4706 Q1 = A). The fold instead lives in the pattern + // SOURCE, one `[Aa]` class per ASCII letter, built by the spec's shared + // `asciiCaseInsensitiveRegexSource` — the same source `driver-mongodb` + // binds, so the two document-shaped faces fold identically. + // + // The neighbours' `i` flags are NOT a precedent to copy here: they are + // the `$contains` family folding Unicode, which is the open defect #6682 + // tracks on this face, not the behaviour to extend. + case '$icontains': + regexConditions.push({ $regex: new RegExp(asciiCaseInsensitiveRegexSource(val)) }); + break; case '$between': { // [#5328] The arm used to be CONDITIONAL — a comparand that was not a // two-element array skipped it and wrote nothing, so the field diff --git a/packages/drivers/driver-memory/src/memory-icontains.test.ts b/packages/drivers/driver-memory/src/memory-icontains.test.ts new file mode 100644 index 0000000000..28e2f92fa7 --- /dev/null +++ b/packages/drivers/driver-memory/src/memory-icontains.test.ts @@ -0,0 +1,164 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#6520] `$icontains` on ALL THREE of this package's filter faces. + * + * The operator was refused here until #6520 — not by an arm that said so, but by + * derivation: `SUPPORTED_FIELD_OPERATORS` is the spec's `FILTER_OPERATORS`, and + * that array deliberately omitted the name while no face could evaluate it. + * Admitting it and writing the arms had to happen in ONE PR, because admission + * alone flips this driver from a loud refusal to a SILENT WIDENING: #5701 + * measured `match({ name: 'zzz' }, { name: { $icontains: 'acme' } })` returning + * `true` on a branch that added the word early — the predicate dropped, every + * row matched, which on an RLS read scope is a permission bypass (#3948). + * + * ## Why all three faces are in one file + * + * Because this package's recurring defect is not "a face is wrong", it is "the + * faces disagree" — #5374 (`$contains` two ways), #5324/#5328 (a malformed + * `$between`: no rows on one face, EVERY row on another), #5347 (`$null` three + * ways). A per-face file would let one arm rot without the others noticing. Each + * case below therefore runs the same filter through the live query path, the + * reference matcher and the analytics face and demands ONE answer. + * + * ## Why this file drives the ROWS and spells its own cases + * + * `check-driver-conformance.mjs` judges coverage by whether a package names the + * shared text case-set's marker export. Naming it here would flip this driver's + * cell to "covered" while requirement 2 of that case-set (the `$contains` family + * folding Unicode on this driver) is still open — and a ledger entry for a + * covered cell fails the gate's RECONCILED invariant. So this file drives + * `FILTER_TEXT_ROWS`, the fixture, and writes out the `$icontains` cases it is + * entitled to answer. The DEBT row stays until #6682 closes the other half; see + * that row's `why`, which now names one open requirement instead of two. + */ + +import { describe, it, expect, beforeEach } from 'vitest'; +import { FILTER_TEXT_ROWS } from '@objectstack/spec/data'; +import { InMemoryDriver } from './memory-driver.js'; +import { match } from './memory-matcher.js'; + +const TABLE = 'text_rows'; + +/** The conformance fixture's rows, as this driver stores them. */ +const ROWS = FILTER_TEXT_ROWS.map((r) => ({ ...r })); + +async function seed(): Promise { + const driver = new InMemoryDriver(); + for (const row of ROWS) await driver.create(TABLE, { ...row }); + return driver; +} + +/** Ids the LIVE QUERY PATH returns (mingo), ascending. */ +const queryIds = async (driver: InMemoryDriver, where: unknown): Promise => + (await driver.find(TABLE, { where: where as any })) + .map((r: any) => String(r.id)) + .sort((a, b) => a.localeCompare(b)); + +/** Ids the REFERENCE MATCHER returns, ascending. */ +const matcherIds = (where: unknown): string[] => + ROWS.filter((r) => match(r, where)).map((r) => r.id).sort((a, b) => a.localeCompare(b)); + +describe('[#6520] $icontains — the accept table, on every face', () => { + let driver: InMemoryDriver; + beforeEach(async () => { driver = await seed(); }); + + /** + * The measured table from the issue, plus the boundary that makes it a + * contract rather than a preference. `expected` is the SAME answer the SQL + * family gives for these rows (`FILTER_TEXT_CASES`' first four rows) — the + * whole point being that a filter must not change meaning when an app's tests + * run on this double and production runs SQL. + */ + const CASES: Array<[string, unknown, string[]]> = [ + ['an upper-case row from a lower-case comparand', { name: { $icontains: 'acme' } }, ['1', '2']], + ['a lower-case row from an upper-case comparand', { name: { $icontains: 'ACME' } }, ['1', '2']], + // The ASCII boundary. A Unicode-folding face (`toLowerCase()`, or a RegExp + // with the `i` flag) answers ['3','4'] to BOTH of these and is wrong twice. + ['ASCII-ONLY: a lower-case non-ASCII comparand misses its upper-case row', { name: { $icontains: 'café' } }, ['4']], + ['ASCII-ONLY: an upper-case non-ASCII comparand misses its lower-case row', { name: { $icontains: 'CAFÉ' } }, ['3']], + // The comparand is LITERAL — the `$regex` defect #4706 retired, restated. + ['% is a literal character, not a wildcard', { name: { $icontains: '100%' } }, ['5']], + ['_ is a literal character, not a wildcard', { name: { $icontains: 'a_b' } }, ['7']], + ['. is a literal character, not a regex metacharacter', { name: { $icontains: 'a.b' } }, ['9']], + ]; + + for (const [label, where, expected] of CASES) { + it(`query path: ${label}`, async () => { + expect(await queryIds(driver, where)).toEqual(expected); + }); + + it(`reference matcher: ${label}`, () => { + expect(matcherIds(where)).toEqual(expected); + }); + } + + it('the two faces agree on every case — the divergence class #5374 closed', async () => { + for (const [label, where] of CASES) { + expect(await queryIds(driver, where), label).toEqual(matcherIds(where)); + } + }); + + /** + * The #3948 pin, stated as a row COUNT rather than as a throw. Every case + * above returns a strict subset of the fixture; a face that dropped the + * predicate would return all nine and still "work". + */ + it('never answers every row — a dropped predicate WIDENS', async () => { + for (const [label, where] of CASES) { + expect((await queryIds(driver, where)).length, label).toBeLessThan(ROWS.length); + expect(matcherIds(where).length, label).toBeLessThan(ROWS.length); + } + }); + + /** + * `$icontains` is the case-INSENSITIVE twin, so its case-EXACT sibling must + * NOT have moved. This is the row that would catch an implementation that + * "fixed" the fold by making `$contains` insensitive too — and note the + * matcher is the face that answers `$contains` case-exactly today (the query + * path still folds Unicode there, which is #6682, not this PR). + */ + it('leaves $contains case-SENSITIVE on the reference matcher', () => { + expect(matcherIds({ name: { $contains: 'acme' } })).toEqual(['2']); + expect(matcherIds({ name: { $contains: 'ACME' } })).toEqual(['1']); + }); +}); + +describe('[#6520] $icontains comparand refusals, in the ADR-0112 envelope', () => { + let driver: InMemoryDriver; + beforeEach(async () => { driver = await seed(); }); + + /** + * `code` AND `status`, never a bare `toThrow()`. A rejection test that only + * asserts "something threw" carries one bit where the defect has two, and this + * driver's own history is the argument: #5324 spent an issue routing uncoded + * engine errors back into the ADR-0112 envelope, and a throw-only assertion + * cannot tell a correct refusal from an uncoded one. + */ + const refusal = async (where: unknown): Promise => + driver.find(TABLE, { where: where as any }).then(() => null, (e: any) => e); + + it('refuses an EMPTY comparand — it would match every row', async () => { + const err = await refusal({ name: { $icontains: '' } }); + expect(err).toBeInstanceOf(Error); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('$icontains'); + expect(err.message).toContain('NON-EMPTY'); + }); + + it('refuses a NON-STRING comparand rather than coercing it', async () => { + const err = await refusal({ name: { $icontains: 42 } }); + expect(err).toBeInstanceOf(Error); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('$icontains'); + }); + + it('refuses on the REFERENCE MATCHER too — one gate, three faces', () => { + const err = (() => { try { match(ROWS[0], { name: { $icontains: '' } }); return null; } catch (e) { return e as any; } })(); + expect(err).toBeInstanceOf(Error); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + }); +}); diff --git a/packages/drivers/driver-memory/src/memory-matcher.ts b/packages/drivers/driver-memory/src/memory-matcher.ts index 835ec57b79..7ad3c9a159 100644 --- a/packages/drivers/driver-memory/src/memory-matcher.ts +++ b/packages/drivers/driver-memory/src/memory-matcher.ts @@ -19,7 +19,10 @@ // [#5659] The Filter Protocol's boolean identity reduction, shared with // driver-sql, driver-mongodb and the flow linter. See `evaluate` for why a // record-at-a-time matcher consults a record-INDEPENDENT verdict first. -import { reduceFilterVerdict } from '@objectstack/spec/data'; +// [#6520] `$icontains`' ASCII-only fold, defined once in the spec and shared by +// every JS evaluation face — see `foldAsciiCase`'s docblock for why it is not +// re-implemented per package. +import { reduceFilterVerdict, asciiCaseInsensitiveContains } from '@objectstack/spec/data'; import { assertFilterConditionShape } from './filter-refusal.js'; @@ -218,8 +221,20 @@ function checkCondition(value: any, condition: any): boolean { case '$startsWith': if (typeof value !== 'string' || !value.startsWith(target)) return false; break; - case '$endsWith': - if (typeof value !== 'string' || !value.endsWith(target)) return false; + case '$endsWith': + if (typeof value !== 'string' || !value.endsWith(target)) return false; + break; + // [#6520] The case-INSENSITIVE twin of `$contains`. ASCII case only, + // and the fold runs on BOTH sides — `asciiCaseInsensitiveContains` + // is the spec's own function, shared with the five other JS faces so + // the fold cannot drift per package. + // + // NOT `toLowerCase()`: that folds the whole Unicode range, so `CAFÉ` + // would match `café` on this face and not on the SQL family, which is + // the divergence #6520 closed rather than a nicety (#4706 Q1 = A). + case '$icontains': + if (typeof value !== 'string' || typeof target !== 'string' + || !asciiCaseInsensitiveContains(value, target)) return false; break; case '$null': // $null: true → value must be null/undefined; $null: false → value must not be null/undefined diff --git a/packages/drivers/driver-mongodb/src/mongodb-filter.ts b/packages/drivers/driver-mongodb/src/mongodb-filter.ts index ad2d5e9473..5b11827d45 100644 --- a/packages/drivers/driver-mongodb/src/mongodb-filter.ts +++ b/packages/drivers/driver-mongodb/src/mongodb-filter.ts @@ -40,6 +40,11 @@ import { // prints, read from the spec so this driver's sentence about `$regex` cannot // drift from the four other refusal sites' (#5701). import { RETIRED_FILTER_OPERATORS } from '@objectstack/spec/data'; +// [#6520] `$icontains`' ASCII-only fold, defined once in the spec. This driver +// hands a PATTERN to MongoDB rather than comparing strings, so the fold has to +// live in the pattern source — see its docblock for why `$options: 'i'` is the +// wrong tool. +import { asciiCaseInsensitiveRegexSource } from '@objectstack/spec/data'; import { coerceTemporalValue, type TemporalFieldKind, @@ -228,6 +233,22 @@ function classifyFilterKey(key: string, value: unknown, here: string): FilterVer throw nonBooleanNullComparandError(key, value.$null, `${here}.$null`); } + // [#6520] `$icontains`' comparand is a NON-EMPTY string, gated on the WALK for + // the same reason `$null` is one paragraph up: a gate in the emitter fires or + // not depending on whether a boolean identity settled the enclosing node + // first, so `{ $or: [ {}, { name: { $icontains: '' } } ] }` would refuse or + // not depending on its siblings. The condition and the message are + // `driver-sql`'s `icontainsComparandError`, word for word — an empty + // comparand matches every row, and a predicate that constrains nothing WIDENS + // (#3948). + if ( + isFilterNode(value) && + Object.prototype.hasOwnProperty.call(value, '$icontains') && + (typeof value.$icontains !== 'string' || value.$icontains === '') + ) { + throw icontainsComparandError(key, value.$icontains, `${here}.$icontains`); + } + // A field key always contributes a predicate. This stays `'clause'` even for // `{ field: {} }` (a field constrained by zero operators), which this // translator emits as `{ field: {} }` — an exact-match on an empty document. @@ -310,6 +331,28 @@ function nonBooleanNullComparandError(field: string, value: unknown, path: strin ); } +/** + * [#6520] `$icontains` received a comparand that is not a non-empty string. + * + * The twin of `driver-sql`'s and `driver-memory`'s constructor of the same name, + * word for word: #3948 made the backends agree that an uncompilable filter is a + * loud refusal rather than a silent match-everything, and a suite that swaps one + * driver for another has to read one sentence in one envelope. Two rejections in + * one constructor because they are one mistake at one position — a non-string + * would have to be coerced into text the query never asked for, and an empty one + * matches every row. + */ +function icontainsComparandError(field: string, value: unknown, path: string): Error { + const shown = typeof value === 'string' ? `""` : JSON.stringify(value) ?? String(value); + return unsupportedFilterError( + `Operator "$icontains" on field "${field}" at ${path} requires a NON-EMPTY string comparand, ` + + `received ${shown}. "$icontains" is a case-insensitive LITERAL substring search, so its ` + + `comparand is the text to look for — an empty one matches every row (a predicate that ` + + `constrains nothing), and a non-string one would have to be coerced into text this query ` + + `never asked for.`, + ); +} + /** A short type name for an operand a filter refusal has to describe. */ function describeFilterOperand(value: unknown): string { if (value === null) return 'null'; @@ -546,6 +589,23 @@ function translateFieldOperators( result.$options = 'i'; break; + // [#6520] `$icontains` — case-insensitive over ASCII and nothing else. + // + // The one arm in this family that does NOT set `$options: 'i'`, and the + // omission is the whole implementation. Mongo's `i` flag folds the full + // Unicode range, so it would match `CAFÉ` against `café` — the answer + // SQLite cannot give and therefore the one the protocol forbids (#4706 + // Q1 = A). The fold lives in the pattern instead, one `[Aa]` class per + // ASCII letter, from the spec's shared `asciiCaseInsensitiveRegexSource` + // — the same source `driver-memory`'s mingo path binds. + // + // Its four neighbours above ARE `$options: 'i'`, and that is not a + // precedent to copy: it is them folding Unicode for the case-SENSITIVE + // `$contains` family, the open defect #6682 tracks on this driver. + case '$icontains': + result.$regex = asciiCaseInsensitiveRegexSource(String(value)); + break; + // Range operator → $gte + upper bound (half-open on a bare-day max, // inheriting `$lte`'s whole-day rule — #4042) case '$between': diff --git a/packages/drivers/driver-mongodb/src/mongodb-icontains.test.ts b/packages/drivers/driver-mongodb/src/mongodb-icontains.test.ts new file mode 100644 index 0000000000..bd68346a32 --- /dev/null +++ b/packages/drivers/driver-mongodb/src/mongodb-icontains.test.ts @@ -0,0 +1,127 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#6520] `$icontains` on driver-mongodb — server-free, over the documents + * `translateFilter` emits. + * + * Server-free is not a convenience here, it is what makes the cell testable at + * all: this package is in the #5499 frozen family and its real-mongod suites are + * OPT-IN (`OS_TEST_MONGODB_MEMORY_SERVER_ENABLED=1`), so a suite that needed a + * server would not run in CI. The pattern this translator emits is the whole + * behaviour — MongoDB's own `$regex` semantics are not under test — which is the + * same judgement `mongodb-filter-logic-translation.test.ts` makes for the logic + * case-set. The emitted patterns are additionally EXECUTED as JS `RegExp`s + * below, which is the same regex engine flavour and turns "the pattern looks + * right" into "the pattern selects the right rows". + * + * This file deliberately drives the shared text fixture's ROWS rather than + * naming that case-set's marker export: doing the latter would flip this + * driver's conformance cell to covered while requirement 2 (the `$contains` + * family's hardcoded `$options: 'i'`) is still open — #6682 — and an entry for a + * covered cell fails the gate's RECONCILED invariant. + */ + +import { describe, it, expect } from 'vitest'; +import { FILTER_TEXT_ROWS } from '@objectstack/spec/data'; +import { translateFilter } from './mongodb-filter.js'; + +/** The pattern this translator emits for one field's `$icontains`. */ +const patternFor = (comparand: string): RegExp => { + const out = translateFilter({ name: { $icontains: comparand } }) as any; + const spec = out.name; + expect(spec, 'no predicate was emitted for $icontains').toBeDefined(); + expect(spec.$options, '$options must NOT be set — it folds the whole Unicode range').toBeUndefined(); + return new RegExp(spec.$regex); +}; + +/** Ids of the conformance rows the emitted pattern selects. */ +const selects = (comparand: string): string[] => + FILTER_TEXT_ROWS.filter((r) => patternFor(comparand).test(r.name)) + .map((r) => r.id) + .sort((a, b) => a.localeCompare(b)); + +describe('[#6520] $icontains translates to an ASCII-only case-insensitive $regex', () => { + it('emits a $regex with NO $options — the fold lives in the pattern', () => { + const out = translateFilter({ name: { $icontains: 'acme' } }) as any; + expect(out).toEqual({ name: { $regex: '[Aa][Cc][Mm][Ee]' } }); + }); + + /** + * The contrast that makes the row above meaningful: the four case-EXACT + * operators still carry the hardcoded `$options: 'i'` this driver has always + * had. That is a DEFECT (#6682 — those four must be case-sensitive), pinned + * here as the current state so that "fixed the family" and "broke + * `$icontains`" cannot be confused for one another. + */ + it('does not disturb the $contains family — still `$options: i` (the open #6682 defect)', () => { + expect(translateFilter({ name: { $contains: 'acme' } })).toEqual({ + name: { $regex: 'acme', $options: 'i' }, + }); + }); + + const CASES: Array<[string, string, string[]]> = [ + ['an upper-case row from a lower-case comparand', 'acme', ['1', '2']], + ['a lower-case row from an upper-case comparand', 'ACME', ['1', '2']], + ['ASCII-ONLY: `café` does not match `CAFÉ`', 'café', ['4']], + ['ASCII-ONLY: `CAFÉ` does not match `café`', 'CAFÉ', ['3']], + ['% is literal, not a wildcard', '100%', ['5']], + ['_ is literal, not a wildcard', 'a_b', ['7']], + ['. is literal, not a regex metacharacter', 'a.b', ['9']], + ]; + + for (const [label, comparand, expected] of CASES) { + it(`selects the right rows: ${label}`, () => { + expect(selects(comparand)).toEqual(expected); + }); + } + + it('never selects every row — a dropped predicate WIDENS (#3948)', () => { + for (const [label, comparand] of CASES) { + expect(selects(comparand).length, label).toBeLessThan(FILTER_TEXT_ROWS.length); + } + }); +}); + +describe('[#6520] $icontains comparand refusals', () => { + const refusalOf = (filter: unknown): any => { + try { + translateFilter(filter as any); + throw new Error('expected a refusal, got a translation'); + } catch (e: any) { + return e; + } + }; + + /** + * `code` AND `status`, not a bare `toThrow()`. This driver is the exact reason + * that bar exists: its `default:` arm threw a bare `new Error` with no `code` + * and no `status` until #5702, so a throw-only assertion was green against the + * very shape #5702 fixed. + */ + for (const [label, comparand] of [ + ['an EMPTY comparand — it would match every row', ''], + ['a NON-STRING comparand — coercion would answer a query nobody wrote', 42], + ] as const) { + it(`refuses ${label}`, () => { + const err = refusalOf({ name: { $icontains: comparand } }); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('$icontains'); + expect(err.message).toContain('NON-EMPTY'); + }); + } + + /** + * The gate sits on the validating WALK, not in the emitter, so a sibling that + * settles the enclosing node by boolean identity cannot skip it. Without that + * placement this filter would reduce to TRUE on its first disjunct and the + * empty comparand would never be judged — the evaluation-order dependence + * #5368 exists to rule out. + */ + it('refuses under an $or whose first branch already settles the node', () => { + const err = refusalOf({ $or: [{}, { name: { $icontains: '' } }] }); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('$icontains'); + }); +}); diff --git a/packages/drivers/driver-turso/src/remote-transport-node-operator-refusal.test.ts b/packages/drivers/driver-turso/src/remote-transport-node-operator-refusal.test.ts index a4c4e163cf..024fad98cd 100644 --- a/packages/drivers/driver-turso/src/remote-transport-node-operator-refusal.test.ts +++ b/packages/drivers/driver-turso/src/remote-transport-node-operator-refusal.test.ts @@ -149,8 +149,9 @@ const UNDECLARED: Array<[label: string, where: unknown, key: string, path: strin * and every one of them compiled to a predicate on a column named after the * operator. `$between` is included even though this transport never compiles it * (TursoDriver lowers it first) — misplaced is misplaced — and `$icontains` - * because this transport compiles it (#5702) even though `FILTER_OPERATORS` - * does not list it yet. + * because this transport compiles it (#5702). [#6520] `FILTER_OPERATORS` lists + * `$icontains` now too, so that clause's "even though" is history: the word + * list and every evaluator agree. * * [#5702] `$regex` LEFT this table for the undeclared one above: it is retired, * so it is not a field operator at any level and its author must not be told to diff --git a/packages/formula/src/matches-filter-icontains.test.ts b/packages/formula/src/matches-filter-icontains.test.ts new file mode 100644 index 0000000000..a7ba2f624e --- /dev/null +++ b/packages/formula/src/matches-filter-icontains.test.ts @@ -0,0 +1,114 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#6520] `$icontains` on `matchesFilterCondition` — the RLS write-side `check`. + * + * This face is the one where the fold's DOMAIN is a security property rather + * than a nicety. `matchesFilterCondition` evaluates the post-image of a write + * against the same predicate `read-scope-sql.ts` compiles to SQL for the read + * side. If this evaluator folded Unicode (the obvious `toLowerCase()`) while the + * read scope folded ASCII, one policy would ALLOW a write here that the read + * scope then hides — one rule, two row sets, which is #3948 reached through case + * folding rather than through a dropped predicate. + * + * The census note on #6520 (#6993) measured this face answering `$icontains` + * with a silent `false` before this change — not a refusal, an ANSWER of "no". + * That is what these cases replace. + */ + +import { describe, it, expect } from 'vitest'; +import { FILTER_TEXT_ROWS } from '@objectstack/spec/data'; +import { matchesFilterCondition } from './matches-filter.js'; + +const ids = (filter: unknown): string[] => + FILTER_TEXT_ROWS.filter((r) => matchesFilterCondition({ ...r }, filter as any)) + .map((r) => r.id) + .sort((a, b) => a.localeCompare(b)); + +describe('[#6520] matchesFilterCondition evaluates $icontains, ASCII fold only', () => { + const CASES: Array<[string, unknown, string[]]> = [ + ['an upper-case row from a lower-case comparand', { name: { $icontains: 'acme' } }, ['1', '2']], + ['a lower-case row from an upper-case comparand', { name: { $icontains: 'ACME' } }, ['1', '2']], + ['ASCII-ONLY: `café` does not match `CAFÉ`', { name: { $icontains: 'café' } }, ['4']], + ['ASCII-ONLY: `CAFÉ` does not match `café`', { name: { $icontains: 'CAFÉ' } }, ['3']], + ['% is literal', { name: { $icontains: '100%' } }, ['5']], + ['_ is literal', { name: { $icontains: 'a_b' } }, ['7']], + ['. is literal, not a regex metacharacter', { name: { $icontains: 'a.b' } }, ['9']], + ]; + + for (const [label, filter, expected] of CASES) { + it(label, () => { expect(ids(filter)).toEqual(expected); }); + } + + it('is no longer the silent `false` the #6993 census measured', () => { + // The exact call from that census note, which returned `false` before. + expect(matchesFilterCondition({ name: 'ACME CORP' }, { name: { $icontains: 'acme' } } as any)) + .toBe(true); + }); + + it('answers FALSE for a genuine non-match, not merely for an unknown operator', () => { + expect(matchesFilterCondition({ name: 'zzz' }, { name: { $icontains: 'acme' } } as any)) + .toBe(false); + }); + + it('leaves $contains case-SENSITIVE', () => { + expect(ids({ name: { $contains: 'acme' } })).toEqual(['2']); + }); + + it('composes under $or / $not, and a missing column does not satisfy it', () => { + expect(ids({ $or: [{ name: { $icontains: 'ACME' } }, { name: { $icontains: 'a.b' } }] })) + .toEqual(['1', '2', '9']); + expect(matchesFilterCondition({ other: 'x' }, { name: { $icontains: 'acme' } } as any)) + .toBe(false); + }); + + /** + * Fail-closed at the comparand, matching the drivers' refusal condition in + * VERDICT even though this face expresses it as a denial rather than a throw + * (see the module header's posture section). An empty comparand would + * otherwise satisfy every post-image, which on a `check` clause means a rule + * that permits every write. + */ + it('DENIES an empty comparand rather than matching every record', () => { + expect(matchesFilterCondition({ name: 'anything' }, { name: { $icontains: '' } } as any)) + .toBe(false); + }); + + it('DENIES a non-string comparand rather than coercing it', () => { + expect(matchesFilterCondition({ name: '42' }, { name: { $icontains: 42 } } as any)) + .toBe(false); + }); +}); + +describe('[#6520] the unknown-operator posture, pinned as DECIDED', () => { + /** + * The decision recorded on `matches-filter.ts`'s header: unknown operators + * keep the silent `false` on this face, because it governs a WRITE-side check + * (an unevaluable condition DENIES, it does not widen) and because callers + * such as `plugin-security`'s explain engine rely on this function being a + * TOTAL per-record predicate. + * + * Pinned so the posture cannot drift unnoticed in either direction: an + * upgrade to throwing should have to edit this test and say so. + */ + it('answers an unknown operator with `false`, and does not throw', () => { + expect(() => matchesFilterCondition({ name: 'x' }, { name: { $sounds_like: 'x' } } as any)) + .not.toThrow(); + expect(matchesFilterCondition({ name: 'x' }, { name: { $sounds_like: 'x' } } as any)) + .toBe(false); + }); + + it('answers a RETIRED spelling the same way — the residue #6520 left open', () => { + // The other five JS faces throw INVALID_FILTER naming `$icontains` here. + // This face denies silently. Recorded as the current, deliberate state, not + // endorsed: see the module header's closing paragraph. + expect(matchesFilterCondition({ name: 'acme' }, { name: { $regex: 'ac.*' } } as any)) + .toBe(false); + }); + + it('still THROWS for the one shape #5240 ruled refused', () => { + // The posture is "silent false for vocabulary, throw for the ruled shape" — + // this case is what keeps the two from being confused for one another. + expect(() => matchesFilterCondition({ name: 'x' }, { name: {} } as any)).toThrow(); + }); +}); diff --git a/packages/formula/src/matches-filter.ts b/packages/formula/src/matches-filter.ts index 4ecf5ba1a5..e8e6259318 100644 --- a/packages/formula/src/matches-filter.ts +++ b/packages/formula/src/matches-filter.ts @@ -15,6 +15,40 @@ * satisfy — returns `false` (the write is denied), never `true`. The operator * vocabulary mirrors `read-scope-sql.ts` so the in-memory and SQL backends agree. * + * ## The unknown-operator posture: silent `false`, DECIDED not inherited (#6520) + * + * The #6993 census measured that this face answers an operator it does not know + * with a silent `false` — no throw, no `code`, no message — where the other five + * JS evaluation faces (`driver-memory`'s three surfaces, `driver-mongodb`, + * objectql's `having`) all REFUSE with `INVALID_FILTER` / 400. #6520 was asked + * to decide whether to keep that or upgrade it, and KEPT it. The reasons, in the + * order they carry weight: + * + * 1. **The direction of the error is opposite here.** Those five faces compile + * READ predicates, where dropping a constraint WIDENS the result set — on an + * RLS read scope that is a permission bypass (#3948), so they must be loud. + * This one evaluates a WRITE-side `check`: an unevaluable condition denies the + * write. Silence costs a diagnostic, not a boundary. + * 2. **Callers depend on this being TOTAL.** `plugin-security`'s `explain-engine` + * calls it per record to answer "does THIS row satisfy the filter?", and + * several driver doubles use it as a list filter. Throwing turns a per-record + * verdict into an aborted operation for those callers — a real behaviour + * change on the read/explain path, which is not what a `$icontains` parity PR + * should be deciding. + * 3. **The measured defect is gone without it.** The census's actual complaint + * was that a spec-DECLARED operator (`$icontains`) got the silent `false`. + * Every operator in `FILTER_OPERATORS` now has an arm in {@link evalOp}, so + * the silent answer is reachable only for a name the protocol does not + * declare or has retired. + * + * What stays open, deliberately and on the record: a RETIRED spelling + * (`$regex` / `$options`) still gets the silent `false` here while the other five + * faces print `RETIRED_FILTER_OPERATORS`' prescription naming `$icontains`. That + * is the residue of #4706's second indictment on this face. It is a narrower + * question than the one this section answers and it changes an accept/reject + * surface, so #6520 left it to the maintainer rather than folding it into a + * parity PR. + * * ONE shape is refused instead of answered (#5240): `{ field: {} }`, a field * constrained by zero operators, throws `INVALID_FILTER` rather than returning * `false`. It is the shape the four backends could not agree on, so no answer @@ -26,7 +60,11 @@ */ import type { FilterCondition } from '@objectstack/spec/data'; -import { nextUtcCalendarDay, utcInstantMs } from '@objectstack/spec/data'; +// [#6520] `asciiCaseInsensitiveContains` is `$icontains`' fold, defined once in +// the spec and shared by every JS evaluation face — so a `check` evaluated here +// and the same predicate compiled to SQL by `read-scope-sql.ts` fold the same +// domain. +import { nextUtcCalendarDay, utcInstantMs, asciiCaseInsensitiveContains } from '@objectstack/spec/data'; import { StandardErrorCode } from '@objectstack/spec/api'; /** @@ -180,6 +218,21 @@ function evalOp(actual: unknown, op: string, raw: unknown, record: Record a >= b) && lteBound(actual, v[1]); case '$contains': return typeof actual === 'string' && typeof v === 'string' && actual.includes(v); + /** + * [#6520] `$contains`' case-INSENSITIVE twin, folding ASCII case and nothing + * else — `asciiCaseInsensitiveContains` is the spec's shared definition, the + * same one `driver-memory`'s matcher and objectql's `having` call. + * + * NOT `actual.toLowerCase().includes(v.toLowerCase())`, which is the obvious + * line and the wrong one: it folds the whole Unicode range, so an RLS + * `check` written with `$icontains` would ALLOW a write here that the read + * scope's SQL — folding ASCII only — then hides. One predicate, two answers, + * across the write gate and the read gate, is the #3948 shape reached + * through case folding (#4706 Q1 = A). + */ + case '$icontains': + return typeof actual === 'string' && typeof v === 'string' && v !== '' + && asciiCaseInsensitiveContains(actual, v); case '$notContains': return !(typeof actual === 'string' && typeof v === 'string' && actual.includes(v)); case '$startsWith': return typeof actual === 'string' && typeof v === 'string' && actual.startsWith(v); case '$endsWith': return typeof actual === 'string' && typeof v === 'string' && actual.endsWith(v); @@ -208,6 +261,18 @@ function evalOp(actual: unknown, op: string, raw: unknown, record: Record ({ ...r, n: Number(r.id) })); + +const ids = (having: unknown): string[] => + applyHaving(ROWS, having as any).map((r) => String(r.id)).sort((a, b) => a.localeCompare(b)); + +describe('[#6520] HAVING evaluates $icontains, folding ASCII case only', () => { + const CASES: Array<[string, unknown, string[]]> = [ + ['an upper-case row from a lower-case comparand', { name: { $icontains: 'acme' } }, ['1', '2']], + ['a lower-case row from an upper-case comparand', { name: { $icontains: 'ACME' } }, ['1', '2']], + ['ASCII-ONLY: `café` does not match `CAFÉ`', { name: { $icontains: 'café' } }, ['4']], + ['ASCII-ONLY: `CAFÉ` does not match `café`', { name: { $icontains: 'CAFÉ' } }, ['3']], + ['% is literal', { name: { $icontains: '100%' } }, ['5']], + ['_ is literal', { name: { $icontains: 'a_b' } }, ['7']], + ['. is literal, not a regex metacharacter', { name: { $icontains: 'a.b' } }, ['9']], + ]; + + for (const [label, having, expected] of CASES) { + it(label, () => { expect(ids(having)).toEqual(expected); }); + } + + it('never returns every row — an ignored operator returns UNFILTERED aggregates', () => { + // The failure this face refuses unknown operators to avoid, stated as a + // count: a `$icontains` that fell through would leave the aggregate + // unfiltered, and a chart drawn over it looks like a working chart. + for (const [label, having] of CASES) { + expect(ids(having).length, label).toBeLessThan(ROWS.length); + } + }); + + it('leaves $contains case-SENSITIVE — the twin did not absorb its sibling', () => { + expect(ids({ name: { $contains: 'acme' } })).toEqual(['2']); + expect(ids({ name: { $contains: 'ACME' } })).toEqual(['1']); + }); + + it('composes under $not / $and like any other operator', () => { + expect(ids({ $not: { name: { $icontains: 'acme' } } })) + .toEqual(['3', '4', '5', '6', '7', '8', '9']); + expect(ids({ $and: [{ name: { $icontains: 'ACME' } }, { n: { $gte: 2 } }] })).toEqual(['2']); + }); + + it('a non-string column does not satisfy it — same guard as $contains', () => { + expect(applyHaving([{ id: 'x', name: 42 }], { name: { $icontains: '4' } } as any)).toEqual([]); + }); + + /** + * The refusal wording, kept as a pin because this face's error is the author's + * whole diagnostic: an operator listed as supported must APPEAR in the + * "supports:" list a refusal prints, or the list sends its reader looking for + * a name that is not there. + */ + it('names $icontains in the supported set a refusal prints', () => { + const err = (() => { + try { applyHaving(ROWS, { name: { $sounds_like: 'acme' } } as any); return null; } + catch (e) { return e as Error; } + })(); + expect(err).toBeInstanceOf(Error); + expect(err!.message).toContain('$icontains'); + }); +}); diff --git a/packages/services/service-analytics/src/__tests__/like-metacharacter-escape.test.ts b/packages/services/service-analytics/src/__tests__/like-metacharacter-escape.test.ts index cbb768f39e..040b4a36c3 100644 --- a/packages/services/service-analytics/src/__tests__/like-metacharacter-escape.test.ts +++ b/packages/services/service-analytics/src/__tests__/like-metacharacter-escape.test.ts @@ -591,20 +591,53 @@ describe('[#5567] analytics LIKE compilers escape their comparand', () => { }); /** - * `$icontains` is UNIMPLEMENTED here, and fail-closed at both doors — the - * state #6518 leaves it in deliberately, because adding it belongs with - * #6520 (the vocabulary, driver-memory and analytics together) rather than - * to a driver-lane case-folding fix. Pinned so "unimplemented" cannot - * quietly become "dropped": a dropped predicate WIDENS. + * [#6520] This case used to pin `$icontains` as REFUSED at both doors — the + * state #6518 deliberately left it in, because implementing it belonged with + * the vocabulary admission rather than with a driver-lane case-folding fix. + * #6520 did that, so the case is REPLACED rather than re-spelled: an + * assertion that the operator throws would now be pinning a refusal that no + * longer exists, and it would keep passing for the wrong reason if the arm + * were deleted in one compiler but not the other. + * + * What it pins instead is the property the refusal was standing in for — + * that the predicate is EMITTED, folded on BOTH sides, and folded with the + * construct the ruling names. */ - it('$icontains is REFUSED, not silently dropped, at both doors', async () => { - expect(() => - compileScopedFilterToSql({ name: { $icontains: 'admin' } } as FilterCondition, 'person'), - ).toThrow(/\$icontains/); - - await expect( - new NativeSQLStrategy().generateSql(query({ name: { $icontains: 'admin' } }), nativeCtx), - ).rejects.toThrow(/\$icontains/); + it('$icontains compiles at both doors, folding ASCII on both sides', async () => { + const scoped = compileScopedFilterToSql( + { name: { $icontains: 'admin' } } as FilterCondition, 'person', + ); + const native = await new NativeSQLStrategy().generateSql( + query({ name: { $icontains: 'admin' } }), nativeCtx, + ); + + for (const [label, sql] of [['scoped', scoped.sql], ['native', native.sql]] as const) { + // BOTH sides: folding only the comparand matches just the rows that were + // already lower-case — a wrong row set that looks like a working filter. + expect(sql.match(/translate\(/g) ?? [], label).toHaveLength(2); + expect(sql, label).toContain('LIKE'); + expect(sql, label).toContain("'ABCDEFGHIJKLMNOPQRSTUVWXYZ'"); + // NOT `LOWER()`: Postgres folds Unicode with it, and the contract is + // ASCII-only (#4706 Q1 = A). This is the same assertion the neighbouring + // Postgres-shape case makes, aimed at the one operator that folds. + expect(sql, label).not.toMatch(/LOWER\s*\(|ILIKE/i); + } + // The comparand is still ESCAPED and its ESCAPE character still bound — + // the fold rides ON TOP of the literal-comparand rule, it does not replace + // it (this file's whole subject). + expect(scoped.params).toEqual(['%admin%', '\\']); + expect(native.params).toEqual(['%admin%', '\\']); + }); + + /** + * [#6520] The comparand rule survives the new arm: `$icontains` is a + * LITERAL substring search, so a `%` in the comparand is a percent sign. + */ + it('$icontains escapes LIKE metacharacters like its case-exact twin', () => { + const { params } = compileScopedFilterToSql( + { name: { $icontains: '100%' } } as FilterCondition, 'person', + ); + expect(params).toEqual(['%100\\%%', '\\']); }); }); }); diff --git a/packages/services/service-analytics/src/__tests__/objectql-echo-operator-coverage.test.ts b/packages/services/service-analytics/src/__tests__/objectql-echo-operator-coverage.test.ts index 5b70d9c0f1..44ba5a52f6 100644 --- a/packages/services/service-analytics/src/__tests__/objectql-echo-operator-coverage.test.ts +++ b/packages/services/service-analytics/src/__tests__/objectql-echo-operator-coverage.test.ts @@ -31,7 +31,7 @@ * refuses an operator it cannot map (`Unsupported filter operator …`), so the * leaf operators that can ever reach this compiler are exactly what * {@link fieldLeaves} emits for the spec's `FILTER_OPERATORS` — a finite, - * enumerable set. Driving all fifteen authorable spellings through the echo + * enumerable set. Driving all sixteen authorable spellings through the echo (#6520 added `$icontains`) * turns "the two tables drifted" from something a reader has to notice into a * failing test, which is what #4128 asked for and did not get for this third * compiler. @@ -125,6 +125,12 @@ const OPERATOR_CASES: Record = { $nin: { stage: { $nin: ['won'] } }, $between: { amount: { $between: [10, 20] } }, $contains: { stage: { $contains: 'o' } }, + // [#6520] Upper-case on purpose: the fixture's `stage` values are lower-case, + // so a comparand that only matches once the ASCII fold RUNS is the one that + // tells a working fold from an absent one. A case-exact rendering returns no + // rows here, which the row-result assertions below read as a wrong answer + // rather than as a passing count. + $icontains: { stage: { $icontains: 'O' } }, $notContains: { stage: { $notContains: 'o' } }, $startsWith: { stage: { $startsWith: 'w' } }, $endsWith: { stage: { $endsWith: 'n' } }, @@ -367,12 +373,15 @@ describe('[#5333] `/analytics/sql` echo — every authorable operator renders a }); it('does not throw for anything the normalizer can actually emit', () => { - // The leaf operators `fieldLeaves` produces: `MONGO_TO_CUBE_OP`'s twelve, + // The leaf operators `fieldLeaves` produces: `MONGO_TO_CUBE_OP`'s thirteen, // plus `set` / `notSet` (the null predicates, `$eq: null` and a bare - // `null`) — `$between` lowers to `gte` / `lte`, already in the twelve. + // `null`) — `$between` lowers to `gte` / `lte`, already in the thirteen. const EMITTABLE = [ 'equals', 'notEquals', 'gt', 'gte', 'lt', 'lte', 'in', 'notIn', 'contains', 'notContains', 'startsWith', 'endsWith', + // [#6520] `$icontains`' leaf. It renders through the same LIKE row as + // `contains`, with the ASCII fold wrapped around both sides. + 'icontains', 'set', 'notSet', ]; for (const operator of EMITTABLE) { diff --git a/packages/services/service-analytics/src/like-pattern.ts b/packages/services/service-analytics/src/like-pattern.ts index 371e069064..1f49953851 100644 --- a/packages/services/service-analytics/src/like-pattern.ts +++ b/packages/services/service-analytics/src/like-pattern.ts @@ -116,13 +116,11 @@ * Postgres-shaped, and that the family is compiled case-EXACT — so this * paragraph goes red rather than merely stale. * - * `$icontains` is a separate matter and is NOT implemented here at all: this - * package has zero references to it, and both doors refuse an unknown operator - * outright (`filter-normalizer.ts`'s `fieldLeaves` throws - * `invalidFilterError`, `read-scope-sql.ts`'s `compileOperator` throws - * `readScopeCompileError` from its `default:` arm). Unimplemented and - * fail-closed, which is the correct state until #6520 settles the vocabulary - * across the frozen backends too. + * `$icontains` IS implemented here since #6520, and it is a separate construct + * rather than a flag on the family above: it folds ASCII case on BOTH sides via + * {@link asciiLowerSqlExpr}, while the `$contains` family stays case-EXACT. The + * two must not be collapsed — a shared "case-insensitive" path would give the + * `$contains` family the fold the ruling took away from it. * * ## `String(value)` is safe here because nothing unrenderable reaches it (#5234) * @@ -181,3 +179,45 @@ export function likePattern(shape: LikeShape, value: unknown): string { const escaped = escapeLikePattern(value); return shape === 'starts' ? `${escaped}%` : shape === 'ends' ? `%${escaped}` : `%${escaped}%`; } + +/** `A`..`Z` and the `a`..`z` they fold onto — the #4706 Q1 = A domain, as data. */ +const ASCII_UPPER_LETTERS = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'; +const ASCII_LOWER_LETTERS = 'abcdefghijklmnopqrstuvwxyz'; + +/** + * [#6520] Wrap a SQL expression in `$icontains`' ASCII-ONLY case fold. + * + * Character for character `driver-sql`'s postgres arm + * (`textMatchPredicate` → `translate(expr, 'ABC…', 'abc…')`), and shared by this + * package's three compilers for the same reason `likePattern` is: the escaping + * and the fold interact, and a second copy of either is how two compilers of one + * filter tree start describing different queries (#5333). + * + * ## Why `translate()` and not `LOWER()` + * + * Postgres' `LOWER()` is locale-aware — it folds `É` to `é`, and the contract + * says it must not (#4706 Q1 = A, because SQLite folds ASCII only and three of + * the five drivers are SQLite underneath). `translate()` with an explicit + * 26-character domain folds exactly `A-Z` and leaves every other code point + * alone, so it is the fold the ruling names rather than the one the database + * happens to offer. + * + * ## The dialect this assumes, stated so it can go red rather than stale + * + * Postgres, like everything else these three compilers emit — the claim this + * file's header already makes and `__tests__/like-metacharacter-escape.test.ts` + * pins. `translate()` is Postgres/Oracle; SQLite has no such function. So the + * warning in the header applies to this helper WORD FOR WORD: if these compilers + * ever emit for SQLite or MySQL, this expression does not merely over-match, it + * fails to parse. The remedy is the per-dialect construct table `driver-sql`'s + * `textMatchPredicate` already carries — nested `REPLACE` for the dialects + * without `translate` — not a quiet fallback to `LOWER()`, which would silently + * restore the Unicode fold this function exists to avoid. + * + * The caller must apply it to BOTH sides of the comparison. Folding only the + * comparand compares a folded needle against a raw column and matches just the + * rows that were already lower-case. + */ +export function asciiLowerSqlExpr(expr: string): string { + return `translate(${expr}, '${ASCII_UPPER_LETTERS}', '${ASCII_LOWER_LETTERS}')`; +} diff --git a/packages/services/service-analytics/src/read-scope-sql.ts b/packages/services/service-analytics/src/read-scope-sql.ts index 21d3c75763..ddedbcefd1 100644 --- a/packages/services/service-analytics/src/read-scope-sql.ts +++ b/packages/services/service-analytics/src/read-scope-sql.ts @@ -2,7 +2,7 @@ import type { FilterCondition } from '@objectstack/spec/data'; import type { RegisteredErrorCode } from '@objectstack/spec/api'; -import { likePattern, LIKE_ESCAPE_CHAR } from './like-pattern.js'; +import { likePattern, LIKE_ESCAPE_CHAR, asciiLowerSqlExpr } from './like-pattern.js'; import { isBindableComparand, isRenderableTextComparand, @@ -785,6 +785,31 @@ function compileOperator(col: string, op: string, val: unknown, field: string, p // [#5234] …and it must be a value `String()` can render, which is asserted // BEFORE `likePattern` sees it — see {@link assertRenderableText}. case '$contains': assertRenderableText(op, field, val); return `${col} LIKE ${bindLike(params, likePattern('contains', val))}`; + /** + * [#6520] `$icontains` on the READ-SCOPE lowering — the one compiler in this + * package where a wrong answer is an ADR-0021 scope over-reach rather than a + * loose chart filter, which is why the fold is the spec's ruled one and not + * `LOWER()`. + * + * `assertRenderableText` first, exactly as its case-exact twin above: the + * comparand has to be something `String()` renders faithfully before a + * pattern is built from it (#5234). + * + * The fold wraps BOTH the column and the bound pattern. Folding one side + * only would compare a folded needle against a raw column — matching just + * the rows already lower-case — and on a read scope that is a row set the + * policy author never wrote, in the narrowing direction here but in the + * WIDENING direction under a `$not`. + */ + case '$icontains': { + assertRenderableText(op, field, val); + // The two binds are spelled out rather than taken from `bindLike`, because + // only the PATTERN placeholder is folded and the `ESCAPE` one must not be. + // Left-to-right, so the values land in `params` in placeholder order — + // the ordering invariant `bindLike`'s own comment states. + const patternRef = asciiLowerSqlExpr(bind(params, likePattern('contains', val))); + return `${asciiLowerSqlExpr(col)} LIKE ${patternRef} ESCAPE ${bind(params, LIKE_ESCAPE_CHAR)}`; + } // [#5298] NULL-safe: `NOT LIKE` is UNKNOWN for a NULL column, and "does not // contain" is true of a value that is not there. case '$notContains': assertRenderableText(op, field, val); return nullSafeNegative(col, `${col} NOT LIKE ${bindLike(params, likePattern('contains', val))}`); diff --git a/packages/services/service-analytics/src/strategies/filter-normalizer.ts b/packages/services/service-analytics/src/strategies/filter-normalizer.ts index dd65bf602c..1ae9b5c2ab 100644 --- a/packages/services/service-analytics/src/strategies/filter-normalizer.ts +++ b/packages/services/service-analytics/src/strategies/filter-normalizer.ts @@ -414,6 +414,10 @@ const MONGO_TO_CUBE_OP: Record = { $notContains: 'notContains', $startsWith: 'startsWith', $endsWith: 'endsWith', + // [#6520] The case-INSENSITIVE twin, ASCII fold only. A separate cube operator + // rather than a flag on `contains`, because the two compile to different SQL + // and one name would make the renderers guess which was meant. + $icontains: 'icontains', }; /** diff --git a/packages/services/service-analytics/src/strategies/native-sql-strategy.ts b/packages/services/service-analytics/src/strategies/native-sql-strategy.ts index 0a287fc680..1e5afffd40 100644 --- a/packages/services/service-analytics/src/strategies/native-sql-strategy.ts +++ b/packages/services/service-analytics/src/strategies/native-sql-strategy.ts @@ -12,7 +12,7 @@ import { } from './filter-normalizer.js'; import { compileScopedFilterToSql } from '../read-scope-sql.js'; import { datasetInvalidError, invalidMemberError } from '../dataset-refusal.js'; -import { likePattern, LIKE_ESCAPE_CHAR, type LikeShape } from '../like-pattern.js'; +import { likePattern, LIKE_ESCAPE_CHAR, asciiLowerSqlExpr, type LikeShape } from '../like-pattern.js'; import { nextUtcCalendarDay } from '@objectstack/core'; /** @@ -717,6 +717,9 @@ export class NativeSQLStrategy implements AnalyticsStrategy { equals: '=', notEquals: '!=', gt: '>', gte: '>=', lt: '<', lte: '<=', contains: 'LIKE', notContains: 'NOT LIKE', startsWith: 'LIKE', endsWith: 'LIKE', + // [#6520] `$icontains` — `LIKE` like its neighbours; what separates it is + // the ASCII fold applied below, not the keyword. + icontains: 'LIKE', }; /** * Where each string operator puts the wildcard. [#5567] The pattern itself is @@ -729,6 +732,9 @@ export class NativeSQLStrategy implements AnalyticsStrategy { const likeShape: Record = { contains: 'contains', notContains: 'contains', startsWith: 'starts', endsWith: 'ends', + // [#6520] Same wildcard placement as `contains`; the case fold is what + // differs, and it is applied to both sides of the comparison below. + icontains: 'contains', }; // Null predicates and the LIKE family read the column as stored — the former @@ -759,6 +765,13 @@ export class NativeSQLStrategy implements AnalyticsStrategy { params.push(likePattern(shape, values[0])); const patternRef = `$${params.length}`; params.push(LIKE_ESCAPE_CHAR); + // [#6520] `$icontains` folds ASCII case on BOTH sides. Only this operator + // folds: the rest of the family is case-EXACT by ruling (#4706 Q2 = A), + // and `objectql-strategy.ts`'s echo of this statement carries the same + // `fold` flag on the same single row so the two keep describing one query. + if (operator === 'icontains') { + return `${asciiLowerSqlExpr(rawCol)} ${sqlOp} ${asciiLowerSqlExpr(patternRef)} ESCAPE $${params.length}`; + } return `${rawCol} ${sqlOp} ${patternRef} ESCAPE $${params.length}`; } diff --git a/packages/services/service-analytics/src/strategies/objectql-strategy.ts b/packages/services/service-analytics/src/strategies/objectql-strategy.ts index e33e0d62d3..475111713b 100644 --- a/packages/services/service-analytics/src/strategies/objectql-strategy.ts +++ b/packages/services/service-analytics/src/strategies/objectql-strategy.ts @@ -12,7 +12,7 @@ import { } from './filter-normalizer.js'; import { compileScopedFilterToSql } from '../read-scope-sql.js'; import { invalidMemberError } from '../dataset-refusal.js'; -import { likePattern, LIKE_ESCAPE_CHAR, type LikeShape } from '../like-pattern.js'; +import { likePattern, LIKE_ESCAPE_CHAR, asciiLowerSqlExpr, type LikeShape } from '../like-pattern.js'; import { nextUtcCalendarDay } from '@objectstack/core'; import { rebucketCrossObject, @@ -45,11 +45,16 @@ const SCALAR_SQL_OPS: Record = { * than the query it claims to reproduce whenever the comparand carried a `_` or * `%` — the #3601 / #3602 / #3650 failure this render block exists to prevent. */ -const LIKE_SQL_OPS: Record = { +const LIKE_SQL_OPS: Record = { contains: { sql: 'LIKE', shape: 'contains' }, notContains: { sql: 'NOT LIKE', shape: 'contains' }, startsWith: { sql: 'LIKE', shape: 'starts' }, endsWith: { sql: 'LIKE', shape: 'ends' }, + // [#6520] `$icontains`: the same escaped pattern and bound `ESCAPE` as its + // four case-EXACT neighbours, with `fold` adding the ASCII-only case fold to + // both sides of the comparison. The flag is on this row alone — the family + // above it is case-sensitive by ruling (#4706 Q2 = A). + icontains: { sql: 'LIKE', shape: 'contains', fold: true }, }; /** One cross-object grouping dimension planned for FK-expand (#3654). */ @@ -704,7 +709,13 @@ export class ObjectQLStrategy implements AnalyticsStrategy { params.push(likePattern(like.shape, values[0])); const patternRef = `$${params.length}`; params.push(LIKE_ESCAPE_CHAR); - return `${col} ${like.sql} ${patternRef} ESCAPE $${params.length}`; + // [#6520] The fold, when the operator carries one, wraps BOTH sides: + // folding only the comparand compares a folded needle against a raw column + // and returns just the rows that were already lower-case — a wrong row set + // that looks like a working predicate. + const lhs = like.fold ? asciiLowerSqlExpr(col) : col; + const rhs = like.fold ? asciiLowerSqlExpr(patternRef) : patternRef; + return `${lhs} ${like.sql} ${rhs} ESCAPE $${params.length}`; } const op = SCALAR_SQL_OPS[operator]; diff --git a/packages/spec/export-origins/data.json b/packages/spec/export-origins/data.json index 2227529703..5728df488b 100644 --- a/packages/spec/export-origins/data.json +++ b/packages/spec/export-origins/data.json @@ -585,6 +585,8 @@ "ValueForm": "src/data/field-value.zod.ts#ValueForm (type)", "ValueShapeFieldDef": "src/data/field-value.zod.ts#ValueShapeFieldDef (interface)", "ZeroLimitConformanceCase": "src/data/pagination-conformance.ts#ZeroLimitConformanceCase (interface)", + "asciiCaseInsensitiveContains": "src/data/filter.zod.ts#asciiCaseInsensitiveContains (function)", + "asciiCaseInsensitiveRegexSource": "src/data/filter.zod.ts#asciiCaseInsensitiveRegexSource (function)", "canonicalAstOperator": "src/data/filter.zod.ts#canonicalAstOperator (function)", "canonicalizeSqlType": "src/data/type-compat.ts#canonicalizeSqlType (function)", "classifyFilterToken": "src/data/context-tokens.zod.ts#classifyFilterToken (function)", @@ -604,6 +606,7 @@ "effectiveOperationsArray": "src/data/api-derivation.ts#effectiveOperationsArray (function)", "emptyGroupValueFor": "src/data/aggregation-policy.ts#emptyGroupValueFor (function)", "fieldForm": "src/data/field.form.ts#fieldForm (const)", + "foldAsciiCase": "src/data/filter.zod.ts#foldAsciiCase (function)", "foldQueryAliasSlots": "src/data/data-engine.zod.ts#foldQueryAliasSlots (function)", "formatUnknownAuthoringKey": "src/data/authoring-key-lint.ts#formatUnknownAuthoringKey (function)", "getDriverConfigJsonSchemaById": "src/data/driver/config-registry.zod.ts#getDriverConfigJsonSchemaById (function)", diff --git a/packages/spec/src/data/filter-ascii-fold.test.ts b/packages/spec/src/data/filter-ascii-fold.test.ts new file mode 100644 index 0000000000..61982d5a19 --- /dev/null +++ b/packages/spec/src/data/filter-ascii-fold.test.ts @@ -0,0 +1,112 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#6520] The ASCII fold every JS `$icontains` face shares. + * + * These three functions are the reason #6520 could give six evaluation faces one + * answer instead of six near-copies, so they are pinned HERE rather than once + * per consumer: a fold that drifts in one package is the #3948 shape reached + * through the comparison instead of the vocabulary, and the per-face suites + * would each go red on their own schedule. + * + * The rows deliberately reuse `FILTER_TEXT_ROWS`' pairs (`ACME Corp`/`acme + * corp`, `CAFÉ`/`café`) — the conformance standard's own fixture — so a change + * that satisfies this file and fails the standard cannot happen quietly. Note + * this file must NOT import `FILTER_TEXT_CASES`: that marker is what + * `check-driver-conformance.mjs` counts as coverage, and spec is not a driver. + */ + +import { describe, it, expect } from 'vitest'; +import { + foldAsciiCase, + asciiCaseInsensitiveContains, + asciiCaseInsensitiveRegexSource, +} from './filter.zod'; + +describe('foldAsciiCase', () => { + it('folds A-Z and nothing else', () => { + expect(foldAsciiCase('ACME Corp')).toBe('acme corp'); + expect(foldAsciiCase('acme corp')).toBe('acme corp'); + }); + + it('leaves non-ASCII letters ALONE — the #4706 Q1 boundary', () => { + // The whole contract in one line: `toLowerCase()` answers 'café' here, and + // that is the answer three of the five backends cannot deliver. + expect(foldAsciiCase('CAFÉ')).toBe('cafÉ'); + expect(foldAsciiCase('МОСКВА')).toBe('МОСКВА'); + expect('CAFÉ'.toLowerCase()).toBe('café'); // the trap, pinned as a contrast + }); + + it('is idempotent and leaves digits, punctuation and spaces untouched', () => { + expect(foldAsciiCase(foldAsciiCase('A1_b%.'))).toBe('a1_b%.'); + expect(foldAsciiCase('')).toBe(''); + }); +}); + +describe('asciiCaseInsensitiveContains', () => { + it('folds BOTH sides — an upper-case row from a lower-case comparand', () => { + // Folding only the comparand matches just the already-lower-case row, which + // is the mistake FILTER_TEXT_CASES' first row exists to catch. + expect(asciiCaseInsensitiveContains('ACME Corp', 'acme')).toBe(true); + expect(asciiCaseInsensitiveContains('acme corp', 'ACME')).toBe(true); + }); + + it('does NOT fold non-ASCII case', () => { + expect(asciiCaseInsensitiveContains('CAFÉ', 'café')).toBe(false); + expect(asciiCaseInsensitiveContains('café', 'CAFÉ')).toBe(false); + // ASCII letters in the same string still fold — the fold is per character. + expect(asciiCaseInsensitiveContains('CAFÉ', 'CAFÉ')).toBe(true); + }); + + it('compares the comparand LITERALLY', () => { + expect(asciiCaseInsensitiveContains('a.b', 'a.b')).toBe(true); + expect(asciiCaseInsensitiveContains('axb', 'a.b')).toBe(false); + expect(asciiCaseInsensitiveContains('a_b', 'a_b')).toBe(true); + expect(asciiCaseInsensitiveContains('axb', 'a_b')).toBe(false); + }); +}); + +describe('asciiCaseInsensitiveRegexSource', () => { + /** How every pattern face uses it: NO flags — the source carries the fold. */ + const matches = (haystack: string, comparand: string): boolean => + new RegExp(asciiCaseInsensitiveRegexSource(comparand)).test(haystack); + + it('folds ASCII case in both directions without an `i` flag', () => { + expect(matches('ACME Corp', 'acme')).toBe(true); + expect(matches('acme corp', 'ACME')).toBe(true); + expect(asciiCaseInsensitiveRegexSource('aZ')).toBe('[Aa][Zz]'); + }); + + it('does NOT fold non-ASCII — the `i` flag would, which is why there is none', () => { + expect(matches('CAFÉ', 'café')).toBe(false); + expect(matches('café', 'CAFÉ')).toBe(false); + // The trap, pinned: the same comparand WITH the flag folds É and is wrong. + expect(new RegExp(asciiCaseInsensitiveRegexSource('café'), 'i').test('CAFÉ')).toBe(true); + }); + + it('escapes every regex metacharacter — the comparand is text, not a pattern', () => { + expect(matches('a.b', 'a.b')).toBe(true); + expect(matches('axb', 'a.b')).toBe(false); + expect(matches('100% match', '100%')).toBe(true); + expect(matches('a+b', 'a+b')).toBe(true); + expect(matches('a(b)c', 'a(b)c')).toBe(true); + expect(matches('a[b]c', 'a[b]c')).toBe(true); + expect(matches('a\\b', 'a\\b')).toBe(true); + expect(matches('a$b^c', 'a$b^c')).toBe(true); + expect(matches('a|b', 'a|b')).toBe(true); + expect(matches('a{2}b', 'a{2}b')).toBe(true); + expect(matches('a?b', 'a?b')).toBe(true); + expect(matches('a*b', 'a*b')).toBe(true); + }); + + it('produces a source that is a VALID regex for every ASCII byte', () => { + // A metacharacter this function forgot to escape shows up as a thrown + // SyntaxError rather than as a subtly wrong row set, so the sweep is worth + // more than a hand-listed table. + for (let code = 0x20; code < 0x7f; code++) { + const ch = String.fromCharCode(code); + expect(() => new RegExp(asciiCaseInsensitiveRegexSource(ch)), `char ${code}`).not.toThrow(); + expect(new RegExp(asciiCaseInsensitiveRegexSource(ch)).test(ch), `char ${code}`).toBe(true); + } + }); +}); diff --git a/packages/spec/src/data/filter-operator-vocabulary.test.ts b/packages/spec/src/data/filter-operator-vocabulary.test.ts index 98d6c8bd3a..e5228d97ff 100644 --- a/packages/spec/src/data/filter-operator-vocabulary.test.ts +++ b/packages/spec/src/data/filter-operator-vocabulary.test.ts @@ -13,20 +13,19 @@ * gate and `service-analytics`' coverage test DERIVE from. An entry here is a * claim that backends implement the operator. * - * `$icontains` is declared and still not enforced (#5701 is the contract half - * of the #4706 ruling; #5702 wrote the lowerings for the SQL family only, and - * #6520 is what the remaining JS faces wait on). Measured on the branch that - * added it to `FILTER_OPERATORS` early: driver-memory's gate stopped refusing - * it and `match({ name: 'zzz' }, { name: { $icontains: 'acme' } })` returned - * `true` — the predicate silently dropped, every row matched. That is the - * widening #3948 is about, so the staging is not a stylistic choice, and - * #5702 landing did NOT clear it: the array is read by the two faces that - * still refuse the operator, not by the three that answer it. + * The two surfaces AGREE today, and the staging that made them disagree is + * over: #6520 added `$icontains` to `FILTER_OPERATORS` in the same PR that gave + * every JS evaluation face an arm, which is the only order in which that name + * could be added at all. Measured on the branch that added it EARLY (#5701): + * driver-memory's gate stopped refusing it and + * `match({ name: 'zzz' }, { name: { $icontains: 'acme' } })` returned `true` — + * the predicate silently dropped, every row matched. That is the widening #3948 + * is about, and it is why the empty set below is a result rather than a default. * - * The pin below is deliberately an EQUALITY, not a subset check, so it fails in - * both directions: a second staged operator added without recording it fails - * here, and so does clearing `$icontains` in #6520 — which is the point. The - * failure message is the instruction. + * The pin is deliberately an EQUALITY, not a subset check, so it fails in both + * directions: an operator declared but not enforced fails here (stage it + * knowingly, by adding it to this list with an issue), and so does one enforced + * without being declared. The failure message is the instruction. */ import { describe, it, expect } from 'vitest'; @@ -48,16 +47,17 @@ describe('the declaration surface and the enforcement surface', () => { expect( stagedOnly, - 'FieldOperatorsSchema and FILTER_OPERATORS differ by something other than the recorded ' - + 'staging. If you are ADDING an operator: declare it in FieldOperatorsSchema only, and ' - + 'add it here plus a note on FILTER_OPERATORS saying which issue implements it — an ' - + 'operator in FILTER_OPERATORS with no backend arm makes driver-memory accept it and ' - + "silently DROP the predicate (measured, #5701). If you are CLEARING one because you " - + 'just implemented it on EVERY face (for `$icontains` that is #6520 — #5702 did the SQL ' - + 'family and correctly left the staging in place): remove it from this list AND delete ' - + 'the staging paragraph on FILTER_OPERATORS, which is now describing something that is ' - + 'no longer true.', - ).toEqual(['$icontains']); + 'FieldOperatorsSchema declares an operator FILTER_OPERATORS does not enforce, and nothing ' + + 'records the staging. That is legal but never silent: an operator in FILTER_OPERATORS ' + + 'with no backend arm makes driver-memory accept it and silently DROP the predicate ' + + '(measured, #5701 — a dropped predicate WIDENS, which on an RLS read scope is #3948), ' + + 'so declaring ahead of the arms is the correct staging. To stage one: declare it in ' + + 'FieldOperatorsSchema, add it to the array THIS assertion compares against, and note on ' + + 'FILTER_OPERATORS which issue implements it. To clear one: implement it on EVERY face ' + + 'in ONE PR — spec word list, driver-memory (query path, reference matcher, analytics ' + + 'face), driver-mongodb, service-analytics (3 compilers), objectql `having`, formula — ' + + 'then empty this list. #6520 is the worked example of the clearing direction.', + ).toEqual([]); }); it('has no operator enforced that is not declared', () => { @@ -95,8 +95,10 @@ describe('RETIRED_FILTER_OPERATORS', () => { it('never points at an operator the protocol no longer has', () => { // The `authoring-key-lint.test.ts` rule, applied to operators: a guidance // table whose prescriptions name something undeclared is advice that sends - // an author into a second error. Note the check is against the DECLARATION - // surface, because `$icontains` is deliberately not in FILTER_OPERATORS yet. + // an author into a second error. The check is against the DECLARATION + // surface, which since #6520 is the same set as the enforcement surface — + // it stays written this way because a prescription must name something an + // author may WRITE, and that is what FieldOperatorsSchema answers. const declared = new Set(declaredKeys()); for (const [op, guidance] of entries) { if (guidance.to === undefined) continue; diff --git a/packages/spec/src/data/filter.test.ts b/packages/spec/src/data/filter.test.ts index aee78f7c99..fd5ca8e8d0 100644 --- a/packages/spec/src/data/filter.test.ts +++ b/packages/spec/src/data/filter.test.ts @@ -744,6 +744,8 @@ describe('Filter Operator Constants', () => { expect(FILTER_OPERATORS).toContain('$contains'); expect(FILTER_OPERATORS).toContain('$startsWith'); expect(FILTER_OPERATORS).toContain('$endsWith'); + // [#6520] Enforced, not merely declared, since every JS face got an arm. + expect(FILTER_OPERATORS).toContain('$icontains'); expect(FILTER_OPERATORS).toContain('$null'); expect(FILTER_OPERATORS).toContain('$exists'); }); diff --git a/packages/spec/src/data/filter.zod.ts b/packages/spec/src/data/filter.zod.ts index 9aaf470974..81088eb512 100644 --- a/packages/spec/src/data/filter.zod.ts +++ b/packages/spec/src/data/filter.zod.ts @@ -379,39 +379,39 @@ export const RangeOperatorSchema = lazySchema(() => z.object({ * `$contains`. An application whose users search non-ASCII text should not read * `$icontains` as "accent- and case-blind search" — it is not one. * - * ### Implementation status — answered by the SQL family, refused by the rest + * ### Implementation status — EVERY face answers it (#6520) * - * #5701 shipped this declaration deliberately ahead of every runtime, and - * #5702 (closed 2026-08-08, retuned by #6518) landed the lowerings on the SQL - * family. Measured per backend, by running `{ name: { $icontains: 'acme' } }` - * against a fixture holding BOTH `acme corp` and `ACME CORP` — not by grepping - * for a case arm, which is blind to the face that inherits its compiler and so - * undercounts: + * #5701 shipped this declaration deliberately ahead of every runtime, #5702 + * (closed 2026-08-08, retuned by #6518) landed the lowerings on the SQL family, + * and #6520 closed the remainder in one PR. Measured per face, by running + * `{ name: { $icontains: 'acme' } }` against a fixture holding BOTH `acme corp` + * and `ACME CORP` — not by grepping for a case arm, which is blind to the face + * that inherits its compiler and so undercounts: * - * | driver | `$icontains` | how it gets there | + * | face | `$icontains` | how it gets there | * |---|---|---| * | `driver-sql` | ANSWERS both rows | its own `case '$icontains'`, folding through the same emitter that carries the escaping | * | `driver-sqlite-wasm` | ANSWERS both rows | INHERITED — `SqliteWasmDriver extends SqlDriver`; this package carries no text case arm of its own, on a different ENGINE | * | `driver-turso` | ANSWERS both rows, on BOTH transports | local inherits `SqlDriver`; the remote transport compiles independently and has its own arm | - * | `driver-memory` | REFUSES — `INVALID_FILTER` / 400 | no arm; its `SUPPORTED_FIELD_OPERATORS` derives from {@link FILTER_OPERATORS}, which deliberately omits it | - * | `driver-mongodb` | REFUSES — `INVALID_FILTER` / 400 | no arm; falls to its translator's `default:` | - * - * The other JS evaluators sit on the refusing side too: objectql's `having` - * face records the omission in its own source, and `formula`'s `matchesFilter` - * has no `$icontains` arm. - * - * **So the sentence an author needs is no longer "no backend answers this".** - * It is: `$icontains` is EXECUTABLE on the SQL family and refused — loudly, - * fail-closed, never silently — everywhere else, so a filter that uses it is - * not portable across backends today. An app whose tests run on the in-memory - * double and whose production runs SQL gets two different answers from one - * filter: that divergence, and the remaining implementations, are #6520. - * - * **The vocabulary gate below is still closed, and #5702 is no longer what it - * waits for.** `$icontains` stays out of {@link FILTER_OPERATORS} on purpose — - * that array is a runtime allowlist, and listing an operator the in-memory - * `match()` cannot evaluate makes it answer `true` for a NON-match (measured; - * see that array's docblock). It joins when the JS faces get arms, in #6520. + * | `driver-memory` — query path, reference matcher, analytics face | ANSWERS both rows | #6520; the pattern faces take {@link asciiCaseInsensitiveRegexSource}, the matcher {@link asciiCaseInsensitiveContains} | + * | `driver-mongodb` | ANSWERS both rows | #6520; an ASCII-only `$regex`, never `$options: 'i'` | + * | objectql `having` | ANSWERS both rows | #6520; {@link asciiCaseInsensitiveContains} over the aggregated row | + * | `formula` `matchesFilterCondition` | ANSWERS both rows | #6520; the same helper, on the RLS write-side `check` | + * | `service-analytics` (3 compilers) | ANSWERS both rows | #6520; an ASCII fold rendered into SQL on both sides of the `LIKE` | + * + * **So the sentence an author needs is no longer "not portable".** `$icontains` + * is executable on every backend and every evaluation face the platform ships, + * and it folds the SAME domain on all of them. The divergence #6520 was filed + * over — an app whose tests run on the in-memory double and whose production + * runs SQL getting two answers from one filter — is closed. + * + * One posture note that is NOT about `$icontains` and is easy to misread from + * this table: `formula`'s evaluator answers an operator it does not know with a + * silent `false` (fail-closed — it governs a WRITE-side `check`, so an + * unevaluable condition DENIES), where the other JS faces throw + * `INVALID_FILTER`. That difference is deliberate and documented on + * `matches-filter.ts`; what #6520 changed is that no operator this array + * DECLARES is answered that way any more. * * The `$contains`-family alignment the ruling above requires is likewise part * done rather than pending: #6518 made that family case-EXACT across the SQL @@ -420,14 +420,15 @@ export const RangeOperatorSchema = lazySchema(() => z.object({ * * `FILTER_TEXT_CASES` (`filter-text-conformance.ts`) is the standard that * measures all of the above, and the driver-conformance ledger still carries a - * DEBT row for each of the two backends left, so what is open stays counted + * DEBT row for `driver-memory` and `driver-mongodb` — on requirement 2 alone + * now, since #6520 closed requirement 1 on both — so what is open stays counted * rather than assumed. * * @see FILTER_TEXT_CASES — the conformance standard for every operator here. * @see RETIRED_FILTER_OPERATORS — why `$regex` is not in this list. * @see https://github.com/objectstack-ai/objectstack/issues/4706 (the ruling) * @see https://github.com/objectstack-ai/objectstack/issues/5702 (the SQL family — landed) - * @see https://github.com/objectstack-ai/objectstack/issues/6520 (the JS faces — open) + * @see https://github.com/objectstack-ai/objectstack/issues/6520 (the JS faces — landed) */ export const StringOperatorSchema = lazySchema(() => z.object({ /** Contains substring, CASE-SENSITIVELY - SQL: LIKE %?% (case-exact) */ @@ -456,12 +457,120 @@ export const StringOperatorSchema = lazySchema(() => z.object({ + 'LITERALLY — "%", "_" and regex metacharacters are ordinary characters, not ' + 'wildcards. Case-SENSITIVE containment is $contains. [#5701 declared it; #5702 ' + 'lowered it on the SQL family (driver-sql, driver-sqlite-wasm, driver-turso on ' - + 'both transports). driver-memory and driver-mongodb still REFUSE it with ' - + 'INVALID_FILTER / 400, so a filter using it is not portable across backends yet ' - + '— #6520.]' + + 'both transports); #6520 lowered it on every JS evaluation face, so it is ' + + 'portable across every backend the platform ships.]' ), })); +// ============================================================================ +// The ASCII fold — ONE implementation for every JS evaluation face (#6520) +// ============================================================================ + +/** `A`..`Z`, and the `a`..`z` they fold onto. The domain #4706 Q1 = A pinned. */ +const ASCII_UPPER_FIRST = 0x41; // 'A' +const ASCII_UPPER_LAST = 0x5a; // 'Z' +const ASCII_CASE_DELTA = 0x20; // 'a' - 'A' + +/** + * [#6520] Fold a string's ASCII case, and NOTHING else — the `$icontains` + * comparison domain, as a function. + * + * ## Why this is in the spec and not four times in four packages + * + * `$icontains` has six JS evaluation faces (`driver-memory`'s query path, + * reference matcher and analytics face, `driver-mongodb`, objectql's `having`, + * `@objectstack/formula`'s `matchesFilterCondition`) plus three SQL compilers in + * `service-analytics`. Every one of them needs the same fold, and this repo has + * already measured what happens when such a rule is written out per package: + * *"a list written out here would agree with the spec on the day it was typed + * and never again"* (`driver-memory/src/filter-refusal.ts`, on the operator + * vocabulary) — the #3948 shape, reached through the fold instead of the word + * list. One definition means a fold that is wrong is wrong everywhere at once, + * which is the only way six faces can be held to one answer. + * + * ## Why not `toLowerCase()` + * + * `String.prototype.toLowerCase()` is the FULL Unicode fold: it maps `É` to `é` + * and `МОСКВА` to `москва`. The contract says those must NOT match (#4706 Q1 = + * A), and the reason is not preference — SQLite has no ICU in this repo's build, + * so its `lower()` folds ASCII only, and three of the five drivers are SQLite + * underneath. A Unicode promise here would be one the SQL family could not keep, + * so a JS face that reaches for `toLowerCase()` does not merely differ from the + * ruling: it re-opens the divergence the ruling closed. `FILTER_TEXT_CASES`' + * `CAFÉ` / `café` pair is the row that catches it. + * + * The same trap wears a second disguise on the regex-evaluating faces: a + * `RegExp` built with the `i` flag ALSO folds the whole Unicode range. Those + * faces take {@link asciiCaseInsensitiveRegexSource}, not an `i` flag. + * + * @see asciiCaseInsensitiveContains — the containment test built on this. + * @see https://github.com/objectstack-ai/objectstack/issues/4706 (the ruling) + * @see https://github.com/objectstack-ai/objectstack/issues/6520 (the JS faces) + */ +export function foldAsciiCase(value: string): string { + let out = ''; + for (let i = 0; i < value.length; i++) { + const code = value.charCodeAt(i); + out += code >= ASCII_UPPER_FIRST && code <= ASCII_UPPER_LAST + ? String.fromCharCode(code + ASCII_CASE_DELTA) + : value[i]; + } + return out; +} + +/** + * [#6520] Does `haystack` contain `needle`, ignoring ASCII case only? + * + * The `$icontains` predicate for every face that can compare two JS strings + * directly — the reference matcher, objectql's `having`, `formula`. The fold + * runs on BOTH sides, which is the half that is easy to get wrong: folding only + * the comparand compares a folded needle against a raw haystack and matches just + * the rows that were already lower-case. `FILTER_TEXT_CASES`' first row (an + * upper-case row from a lower-case comparand) is the one that catches it, and + * `driver-sql`'s emitter carries the same note over its own two-sided fold. + */ +export function asciiCaseInsensitiveContains(haystack: string, needle: string): boolean { + return foldAsciiCase(haystack).includes(foldAsciiCase(needle)); +} + +/** + * [#6520] `comparand` as a regular-expression SOURCE that matches it LITERALLY, + * ignoring ASCII case only. + * + * For the faces that cannot fold the stored value because they hand a pattern to + * an engine rather than comparing two strings: `driver-memory`'s mingo query + * path and its analytics face, and `driver-mongodb`'s translator. Neither can + * apply {@link foldAsciiCase} to the column, so the fold has to live in the + * PATTERN — each ASCII letter becomes the two-member character class `[Aa]`, + * which folds that position on both sides without touching any other character. + * + * Two properties, and the second is the one an `i` flag would destroy: + * + * - **The comparand is LITERAL.** Every regex metacharacter is escaped, so + * `a.b` matches `a.b` and not `axb` — the `$regex` defect #4706 retired the + * operator over, restated as a requirement (`FILTER_TEXT_CASES`' `.` row). + * - **The fold is ASCII-ONLY.** `É` is not an ASCII letter, so it is emitted as + * itself and compares literally. A `new RegExp(source, 'i')` would fold it and + * fail the `CAFÉ` row — so callers pass NO flags. The returned source is + * already case-insensitive exactly where the contract says it should be. + */ +export function asciiCaseInsensitiveRegexSource(comparand: string): string { + let out = ''; + for (let i = 0; i < comparand.length; i++) { + const ch = comparand[i]; + const code = comparand.charCodeAt(i); + if (code >= ASCII_UPPER_FIRST && code <= ASCII_UPPER_LAST) { + out += `[${ch}${String.fromCharCode(code + ASCII_CASE_DELTA)}]`; + } else if (code >= ASCII_UPPER_FIRST + ASCII_CASE_DELTA && code <= ASCII_UPPER_LAST + ASCII_CASE_DELTA) { + out += `[${String.fromCharCode(code - ASCII_CASE_DELTA)}${ch}]`; + } else { + // Escape every regex metacharacter — the comparand is text, not a pattern. + out += /[\\^$.*+?()[\]{}|/]/.test(ch) ? `\\${ch}` : ch; + } + } + return out; +} + // ============================================================================ // 3.5 Special Operators // ============================================================================ @@ -1332,23 +1441,29 @@ export const FilterArraySchema: z.ZodType = z.lazy(() * not narrow a query, it WIDENS it, and on an RLS read scope that is a * permission bypass rather than a degraded feature (#3948). * - * ## `$icontains` is DECLARED but deliberately NOT here yet - * - * {@link StringOperatorSchema}, {@link FieldOperatorsSchema} and {@link Filter} - * declare `$icontains` (#5701, the contract half of the #4706 ruling). This - * array does not, and the difference is deliberate rather than an oversight: - * those three are declaration and TYPE surfaces with no runtime allowlist - * reader (verified — `NormalizedFilterSchema` is their only consumer, and - * nothing parses a filter through it at runtime), so declaring there is inert. - * Adding it HERE would flip driver-memory from a loud refusal to the silent - * widening measured above, on a face that still cannot answer the operator. - * - * **`$icontains` joins this array in the PR that gives the JS faces an arm - * (#6520), not before** — #5702 implemented the SQL family and correctly did - * NOT add it here, because the array is read by the faces that still refuse. - * `filter-operator-vocabulary.test.ts` pins the difference between - * the two surfaces at exactly `{ $icontains }`, so this staging cannot silently - * grow a second member, and clearing it is what makes that pin fail. + * ## `$icontains` JOINED this array in #6520 — the staging is over + * + * It was declared by {@link StringOperatorSchema} and deliberately absent here + * from #5701 until #6520, and that gap was the mechanism described above rather + * than an oversight: adding the name while `driver-memory`'s matcher had no arm + * flipped a loud refusal into the silent widening measured above. + * + * The gap closed the only way it could — **in ONE PR with the arms**, which is + * the constraint the #6520 ruling made binding (maintainer, 2026-08-08: the word + * list must not land ahead of the evaluators). #6520 gave every JS evaluation + * face an ASCII fold ({@link foldAsciiCase}) in the same commit that added the + * name here: `driver-memory` (query path, reference matcher and analytics face), + * `driver-mongodb`, objectql's `having`, `@objectstack/formula`, and + * `service-analytics`' three SQL compilers. So the claim this array makes — + * *backends implement this operator* — is true of `$icontains` for the first + * time. + * + * What that means for the NEXT operator is unchanged, and is the reason the + * measurement above is kept: stage it in {@link FieldOperatorsSchema} alone, + * record it in `filter-operator-vocabulary.test.ts`, and add it here only when + * every face can answer it. That pin now asserts an EMPTY difference between the + * declaration and enforcement surfaces, so a name added to one and not the other + * fails in either direction. * * Retired operators (`$regex`, `$options`) are not here either, and never were. * Their prescriptions live in {@link RETIRED_FILTER_OPERATORS}. @@ -1361,7 +1476,7 @@ export const FILTER_OPERATORS = [ // Set & Range '$in', '$nin', '$between', // String - '$contains', '$notContains', '$startsWith', '$endsWith', + '$contains', '$notContains', '$startsWith', '$endsWith', '$icontains', // Special '$null', '$exists', ] as const; diff --git a/scripts/check-driver-conformance.mjs b/scripts/check-driver-conformance.mjs index 4a5e3552b1..9b8efcd906 100644 --- a/scripts/check-driver-conformance.mjs +++ b/scripts/check-driver-conformance.mjs @@ -293,14 +293,19 @@ const CASE_SETS = [ // // What the case-set demands, and where each requirement stands: // -// 1. `$icontains` — a NEW operator (ASCII-only case fold). **DONE on the SQL -// family** (#5702, retuned by #6518): every SQL face folds through the same -// emitter that carries the escaping, and the fold is now ASCII-only on -// Postgres and MySQL too rather than only on SQLite. Still REFUSED -// (fail-closed, an unimplemented capability rather than a live defect) on -// driver-memory and driver-mongodb, which are the #5499 frozen family — -// tracked as #6520, which also explains why the spec's `FILTER_OPERATORS` -// cannot take `$icontains` until those two have arms. +// 1. `$icontains` — a NEW operator (ASCII-only case fold). **DONE +// EVERYWHERE** (#5702 + #6518 on the SQL family; #6520 on the rest). #6520 +// lifted the #5499 freeze for this operator as a sanctioned one-off +// (maintainer ruling, 2026-08-08, semantic parity only) and gave every +// remaining face an arm in ONE PR with the spec word-list admission — +// driver-memory's three surfaces, driver-mongodb, objectql's `having`, +// formula, and service-analytics' three compilers. The ordering was the +// constraint, not the code: `FILTER_OPERATORS` is what driver-memory's +// shape gate derives from, so admitting the name a PR earlier would have +// turned this driver's loud refusal into a silently dropped predicate +// (#5701 measured it; #3948 is what a dropped predicate is on a read +// scope). Both rows below therefore keep their DEBT entry for +// requirement 2 ALONE. // 2. `$contains` / `$startsWith` / `$endsWith` / `$notContains` must be // case-SENSITIVE (#4706 Q2 = A, superseding `filter.zod.ts`'s former // "Case sensitivity should be handled at backend level"). **DONE on the SQL @@ -372,11 +377,16 @@ const LEDGER = [ + 'counts as a backend) uses String.prototype.includes and is case-SENSITIVE — i.e. this package ' + 'answers one `$contains` two ways today, the divergence class #5374 fixed between the other two ' + 'faces. Whichever suite clears this cell has to pick one and align both: tracked as #6682, which is ' - + 'the successor #6518 left behind for exactly this pair of packages. `$icontains` is still refused on ' - + "both faces (`SUPPORTED_FIELD_OPERATORS` derives from the spec's FILTER_OPERATORS, which deliberately " - + 'does not carry it yet) — unimplemented but fail-closed, requirement 1 open here and tracked as ' - + '#6520. BOTH successors have to land before this row can go: coverage is judged by importing the ' - + 'whole case-set, so a cell that answers one requirement and not the other must not import it.', + + 'the successor #6518 left behind for exactly this pair of packages. Requirement 1 is DONE here ' + + 'since #6520: all THREE of this package\'s faces answer `$icontains` with the spec\'s shared ' + + 'ASCII-only fold — the query path and the analytics face bind a pattern from ' + + '`asciiCaseInsensitiveRegexSource` (no `i` flag, which folds Unicode), the reference matcher calls ' + + '`asciiCaseInsensitiveContains`, and the NON-EMPTY-string comparand rule is driver-sql\'s word for ' + + 'word. So this row now carries ONE open requirement, not two. It still cannot go: coverage is ' + + 'judged by importing the WHOLE case-set, so a cell answering one requirement and not the other must ' + + 'not import it — #6520\'s suites drive `FILTER_TEXT_ROWS` and spell their own `$icontains` cases ' + + 'rather than naming the marker, which would flip this cell to covered and fail RECONCILED while ' + + 'requirement 2 is open.', issue: 'https://github.com/objectstack-ai/objectstack/issues/6682', }, { @@ -396,7 +406,13 @@ const LEDGER = [ + '`translateFieldOperators` lowers `$contains`/`$startsWith`/`$endsWith`/`$notContains` to `$regex` ' + 'with a HARDCODED `$options: "i"` — tracked as #6682, the successor #6518 left behind for this pair ' + 'of frozen packages. `escapeRegex` does escape metacharacters, so the literal-comparand cases hold. ' - + '`$icontains` is still refused (#6520), and BOTH successors have to land before this row can go: ' + + 'Requirement 1 is DONE here since #6520: `translateFieldOperators` has a `$icontains` arm, and it is ' + + 'the ONE arm in that family that does not set `$options: "i"` — the fold lives in the pattern ' + + '(`asciiCaseInsensitiveRegexSource`, one `[Aa]` class per ASCII letter), because mongo\'s `i` flag ' + + 'folds the whole Unicode range and would fail the CAFÉ rows. The non-empty-string comparand rule ' + + 'sits on the validating WALK beside `$null`\'s, not in the emitter, so it cannot be skipped by a ' + + 'boolean identity settling the enclosing node. So this row now carries ONE open requirement, not ' + + 'two. It still cannot go: ' + 'coverage is judged by importing the whole case-set, so a half-answered cell must not import it. Note ' + 'this package is in the #5499 frozen family: its real-mongod suites are opt-in, so whatever clears ' + 'this cell needs a server-free half like `mongodb-filter-logic-translation.test.ts` has.', From 72e77e5611af77c149edc41f6d638d009d0181a5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 17:48:00 +0000 Subject: [PATCH 2/4] docs(spec): the text conformance header records $icontains as ANSWERED on memory + mongodb (#6520) The status section still said those two REFUSE it, which this PR made false. It also now says why each still carries a DEBT row rather than importing the table: coverage is judged by IMPORT, and requirement 2 (the $contains family's Unicode fold, #6682) is still open on both. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PiRUoQkTSBBmpyXBY3cVn2 --- .../spec/src/data/filter-text-conformance.ts | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/packages/spec/src/data/filter-text-conformance.ts b/packages/spec/src/data/filter-text-conformance.ts index 1e89bd1b5f..6e13478812 100644 --- a/packages/spec/src/data/filter-text-conformance.ts +++ b/packages/spec/src/data/filter-text-conformance.ts @@ -49,12 +49,15 @@ * `driver-turso` (both transports) answer `$icontains` and answer the * `$contains` family case-exactly; their suites import this whole table, * which is what `scripts/check-driver-conformance.mjs` counts as coverage. - * - `driver-memory` and `driver-mongodb` refuse `$icontains` with - * `INVALID_FILTER` / 400 (#6520) and still fold the `$contains` family over - * the whole Unicode range (#6682). Each carries a measured DEBT row in that - * same gate's ledger — the ledger is RECONCILED against the imports on - * every run, so read the open set THERE rather than trusting a count - * written in prose here. + * - `driver-memory` and `driver-mongodb` ANSWER `$icontains` since #6520 — on + * all three of driver-memory's faces — with the same ASCII-only fold, so the + * first four rows of this table are satisfied everywhere. They still fold the + * `$contains` family over the whole Unicode range (#6682), which is why each + * still carries a measured DEBT row in that same gate's ledger rather than + * importing this table: coverage is judged by IMPORT, and a cell that answers + * one requirement and not the other must not claim the whole set. The ledger + * is RECONCILED against the imports on every run, so read the open set THERE + * rather than trusting a count written in prose here. * * Rule 2 above still governs the open cells: the rows join a driver's suite * in the PR that closes its gap, not before. @@ -80,7 +83,7 @@ * @see https://github.com/objectstack-ai/objectstack/issues/4706 (the ruling) * @see https://github.com/objectstack-ai/objectstack/issues/5701 (this table) * @see https://github.com/objectstack-ai/objectstack/issues/5702 (the SQL family — landed) - * @see https://github.com/objectstack-ai/objectstack/issues/6520 ($icontains on the JS faces — open) + * @see https://github.com/objectstack-ai/objectstack/issues/6520 ($icontains on the JS faces — landed) * @see https://github.com/objectstack-ai/objectstack/issues/6682 (the $contains family on memory + mongodb — open) */ From 17f4ae0aad1c3852c9192a2f966665ca0816b643 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 18:03:17 +0000 Subject: [PATCH 3/4] test(driver-memory): the analytics face's declared-operator probe covers $icontains (#6520) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `memory-driver-filter-logic-conformance.test.ts` keys its probe map by the face's OWN declared vocabulary and fails when a declared operator has no probe — so adding `$icontains` to `MONGO_TO_CUBE_OPERATOR` correctly turned it red. That gate is the reason #5374 cannot recur, and it fired exactly as designed. The comparand is UPPER-case ('BET') against a lower-case fixture row ('beta'), so the probe only selects a row once the ASCII fold actually runs. A lower-case comparand would have been discriminating too and would have agreed across the two faces for free even if neither folded — the free-agreement certification that block's own header warns against. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PiRUoQkTSBBmpyXBY3cVn2 --- .../src/memory-driver-filter-logic-conformance.test.ts | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/packages/drivers/driver-memory/src/memory-driver-filter-logic-conformance.test.ts b/packages/drivers/driver-memory/src/memory-driver-filter-logic-conformance.test.ts index 03f77f5611..4e7c55552e 100644 --- a/packages/drivers/driver-memory/src/memory-driver-filter-logic-conformance.test.ts +++ b/packages/drivers/driver-memory/src/memory-driver-filter-logic-conformance.test.ts @@ -587,6 +587,12 @@ const DECLARED_OPERATOR_PROBES: Record = { $nin: { code: { $nin: ['100'] } } as FilterCondition, $contains: { name: { $contains: 'et' } } as FilterCondition, $notContains: { name: { $notContains: 'et' } } as FilterCondition, + // [#6520] The comparand is deliberately UPPER-case against a lower-case + // fixture (`beta`), so the probe only selects row 2 once the ASCII fold + // actually runs. `'et'` would have been discriminating too, and would have + // agreed for free on a face that never folded — which is the certification + // this block's own header warns against. + $icontains: { name: { $icontains: 'BET' } } as FilterCondition, $exists: { closed_at: { $exists: false } } as FilterCondition, }; From cf3da3d3812ac8ad01f1ba464ea2455d9ca9f4de Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 18:26:39 +0000 Subject: [PATCH 4/4] chore(spec): regenerate the api-surface snapshot for the three new fold exports (#6520) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `check:api-surface` is a SEPARATE snapshot from `check:export-origins`, and this PR regenerated only the latter. Both track the public surface and both must be regenerated when it grows: export-origins records WHICH DECLARATION each name resolves to, api-surface records the NAME SET plus factory signatures. The delta is exactly the three additive exports on `./data` — `foldAsciiCase`, `asciiCaseInsensitiveContains`, `asciiCaseInsensitiveRegexSource` — reported by the gate as "0 breaking (removed/narrowed), 3 added". Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PiRUoQkTSBBmpyXBY3cVn2 --- packages/spec/api-surface/data.json | 3 +++ 1 file changed, 3 insertions(+) diff --git a/packages/spec/api-surface/data.json b/packages/spec/api-surface/data.json index 61609f5ae1..b4296e98c0 100644 --- a/packages/spec/api-surface/data.json +++ b/packages/spec/api-surface/data.json @@ -585,6 +585,8 @@ "ValueForm (type)", "ValueShapeFieldDef (interface)", "ZeroLimitConformanceCase (interface)", + "asciiCaseInsensitiveContains (function)", + "asciiCaseInsensitiveRegexSource (function)", "canonicalAstOperator (function)", "canonicalizeSqlType (function)", "classifyFilterToken (function)", @@ -604,6 +606,7 @@ "effectiveOperationsArray (function)", "emptyGroupValueFor (function)", "fieldForm (const)", + "foldAsciiCase (function)", "foldQueryAliasSlots (function)", "formatUnknownAuthoringKey (function)", "getDriverConfigJsonSchemaById (function)",