feat(spec,drivers,objectql,analytics,formula): $icontains reaches every JS evaluation face (#6520)

.changeset/icontains-reaches-every-js-face.md

@@ -0,0 +1,64 @@
+---
+"@objectstack/spec": minor
+"@objectstack/driver-memory": minor
+"@objectstack/driver-mongodb": minor
+"@objectstack/objectql": minor
+"@objectstack/formula": minor
+"@objectstack/service-analytics": minor
+---
+
+feat(spec,drivers,objectql,analytics,formula): `$icontains` reaches every JS evaluation face (#6520)
+
+The other half of #5702. That change implemented `$icontains` on the SQL family
+and correctly left the spec's `FILTER_OPERATORS` alone; this one adds the
+operator to that array and gives every remaining evaluation face an arm, in ONE
+change, because those two steps cannot be separated.
+
+**Why one PR.** `FILTER_OPERATORS` is not a word list, it is a runtime allowlist:
+`driver-memory`'s shape gate derives from it, and its matcher's `default:` arm
+assumes the gate already refused anything unimplemented. Measured on a branch
+that added the name early (#5701): the gate stopped refusing, the matcher fell
+through, and `match({ name: 'zzz' }, { name: { $icontains: 'acme' } })` returned
+`true` — the predicate silently dropped, every row matched. A dropped predicate
+does not narrow a query, it WIDENS it, and on an RLS read scope that is a
+permission bypass rather than a degraded feature (#3948). So the word list
+travels with the evaluators or not at all.
+
+**What now answers it**, all folding the same domain: `driver-memory` (query
+path, reference matcher, and the analytics/cube face), `driver-mongodb`,
+`objectql`'s `having`, `@objectstack/formula`'s `matchesFilterCondition` (the RLS
+write-side `check`), and `service-analytics`' three SQL compilers (the RLS
+lowering, the native-SQL strategy, and the `/analytics/sql` echo).
+
+**The fold is ASCII-only, and that is the contract, not an implementation
+detail** (#4706 Q1 = A). `$icontains: 'café'` does not match `CAFÉ`. Every face
+reads one shared definition — `foldAsciiCase` /
+`asciiCaseInsensitiveContains` / `asciiCaseInsensitiveRegexSource`, new exports
+on `@objectstack/spec/data` — because the two obvious per-package spellings are
+both wrong in the same direction: `toLowerCase()` folds the whole Unicode range,
+and so does a `RegExp` built with the `i` flag. SQLite folds ASCII only and three
+of the five drivers are SQLite underneath, so a Unicode fold on a JS face would
+re-open exactly the divergence the ruling closed. The pattern-binding faces
+(mingo, mongo) therefore emit one `[Aa]` character class per ASCII letter and
+pass NO flags; mongo's `$icontains` is the one arm in its family that does not
+set `$options: 'i'`.
+
+The comparand keeps the rules its SQL twin has: matched LITERALLY (`%`, `_` and
+regex metacharacters are ordinary characters), and refused when empty or
+non-string — an empty comparand matches every row, which is a predicate that
+constrains nothing.
+
+**User-visible effect.** A filter using `$icontains` now behaves the same on the
+in-memory double and on SQL, so an app whose tests run on one and whose
+production runs the other stops getting two answers from one filter. Downstream,
+#5814 (better-auth `Where.mode: 'insensitive'`) no longer hits a 400 on the
+memory double.
+
+Not changed, and still tracked: the `$contains` family still folds Unicode on
+`driver-memory`'s query path and `driver-mongodb` (#6682) — both remain DEBT rows
+in `scripts/check-driver-conformance.mjs`, now naming one open requirement each
+instead of two. `formula`'s unknown-operator posture stays a silent, fail-closed
+`false` (it governs a write-side check, where an unevaluable condition denies
+rather than widens); the decision and its limits are documented on
+`matches-filter.ts`, and no operator the spec DECLARES is answered that way any
+more.

content/docs/protocol/objectql/query-syntax.mdx

@@ -291,15 +291,23 @@ not `LIKE` wildcards, and `.` / `*` / `+` are ordinary characters, not regex
 metacharacters — `{ name: { $icontains: 'a.b' } }` matches `a.b` and not `axb`.
 
 <Callout type="info">
- **Status:** the case rules above are the protocol's declaration as of
- `@objectstack/spec` 18. The backend lowerings that deliver them — making SQLite's
- and turso's `LIKE` case-exact, dropping MongoDB's hardcoded `$options: 'i'`, and
- implementing `$icontains` everywhere — are tracked by
- [#5702](https://github.com/objectstack-ai/objectstack/issues/5702). Until it
- lands, a backend that has not been aligned refuses `$icontains` outright rather
- than answering it approximately, and `$contains` still follows its dialect. The
- shared standard both halves are measured against is `FILTER_TEXT_CASES`
- (`@objectstack/spec/data`).
+ **Status:** `$icontains` is implemented on **every** backend and every evaluation
+ face, and folds the same ASCII domain on all of them —
+ [#5702](https://github.com/objectstack-ai/objectstack/issues/5702) did the SQL
+ family and [#6520](https://github.com/objectstack-ai/objectstack/issues/6520) did
+ the rest (the in-memory driver's query, matcher and analytics faces, MongoDB,
+ ObjectQL's `having`, the RLS write-side `check`, and the analytics SQL
+ compilers). A filter using it means the same thing whether your tests run on the
+ in-memory double or your production runs SQL.
+
+ One half of the case rules above is still landing: `$contains` /
+ `$startsWith` / `$endsWith` / `$notContains` are case-**sensitive** by ruling and
+ are so on the SQL family, but the in-memory driver's query path and MongoDB still
+ fold them over the whole Unicode range —
+ [#6682](https://github.com/objectstack-ai/objectstack/issues/6682). Until that
+ lands, prefer `$icontains` when you *want* a fold rather than relying on
+ `$contains` being loose on those two backends. The shared standard both halves
+ are measured against is `FILTER_TEXT_CASES` (`@objectstack/spec/data`).
 </Callout>
 
 ### `$regex` — removed

content/docs/references/data/filter.mdx

@@ -147,7 +147,7 @@ Type: `[FilterArray](#filterarray)[]`
 | **$notContains** | `string` | optional | |
 | **$startsWith** | `string` | optional | |
 | **$endsWith** | `string` | optional | |
-| **$icontains** | `string` | optional | Contains substring, ignoring case — but ONLY ASCII case (A-Z against a-z). Every other character compares literally, so "café" does NOT match "CAFÉ" and "москва" does not match "МОСКВА". The domain is ASCII because that is the one fold all five backends can deliver: SQLite (and therefore turso and sqlite-wasm) folds ASCII only, so a Unicode promise here would be a guarantee three of the five could not keep. The comparand is matched LITERALLY — "%", "_" and regex metacharacters are ordinary characters, not wildcards. Case-SENSITIVE containment is $contains. [#5701 declared it; #5702 lowered it on the SQL family (driver-sql, driver-sqlite-wasm, driver-turso on both transports). driver-memory and driver-mongodb still REFUSE it with INVALID_FILTER / 400, so a filter using it is not portable across backends yet — #6520.] |
+| **$icontains** | `string` | optional | Contains substring, ignoring case — but ONLY ASCII case (A-Z against a-z). Every other character compares literally, so "café" does NOT match "CAFÉ" and "москва" does not match "МОСКВА". The domain is ASCII because that is the one fold all five backends can deliver: SQLite (and therefore turso and sqlite-wasm) folds ASCII only, so a Unicode promise here would be a guarantee three of the five could not keep. The comparand is matched LITERALLY — "%", "_" and regex metacharacters are ordinary characters, not wildcards. Case-SENSITIVE containment is $contains. [#5701 declared it; #5702 lowered it on the SQL family (driver-sql, driver-sqlite-wasm, driver-turso on both transports); #6520 lowered it on every JS evaluation face, so it is portable across every backend the platform ships.] |
 
 
 ---

packages/drivers/driver-memory/src/filter-refusal.ts

@@ -163,12 +163,24 @@ export function emptyFieldConstraintError(field: string, path: string): Error {
  * they are refused here like any other undeclared operator, with the spec's
  * prescription attached (see {@link retiredFilterOperatorError}).
  *
- * Note what this does NOT do: it does not add `$icontains`. That name is
- * declared by `StringOperatorSchema` but deliberately absent from
- * `FILTER_OPERATORS` (#5701), and this set is derived, so this driver refuses it
- * — fail-closed, an unimplemented capability rather than a silent widening. The
- * `$icontains` implementation for the JS faces is #5499-frozen; see the
- * `driver-memory` row of `scripts/check-driver-conformance.mjs`.
+ * ## [#6520] `$icontains` arrives here by DERIVATION, and that is the risk
+ *
+ * This set is `FILTER_OPERATORS` itself, so #6520 adding `$icontains` to the
+ * spec admitted it here with no edit to this file. That is the property #5701
+ * measured and warned about: while the matcher had no arm, admission alone
+ * turned a loud refusal into `match({ name: 'zzz' }, { name: { $icontains:
+ * 'acme' } }) === true` — the predicate dropped, every row matched, which on an
+ * RLS read scope is a permission bypass rather than a degraded filter (#3948).
+ *
+ * So the arms and the word list HAD to land in one PR, and #6520 did that:
+ * `memory-matcher.ts` and `memory-driver.ts` both carry a `$icontains` case, and
+ * `memory-analytics.ts` lowers it too. Re-verified by deleting the matcher's arm
+ * on the #6520 branch — with the name admitted, the reference matcher answered
+ * EVERY row, which is the measurement, not a prediction.
+ *
+ * The lesson for the next operator is the ordering rather than this name: an
+ * entry in `FILTER_OPERATORS` is a claim that this driver evaluates it, and this
+ * file will make that claim on the spec's behalf whether or not it is true.
  *
  * Everything else is refused. That includes the mingo operators this driver used
  * to hand through by accident (`$elemMatch`, `$size`, `$type`, `$mod`, `$where`,
@@ -624,6 +636,13 @@ function assertFieldConstraintShape(
  if (op === '$null' && typeof spec[op] !== 'boolean') {
  throw nonBooleanNullComparandError(field, spec[op], `${path}.$null`);
  }
+ // [#6520] `$icontains`' comparand is a NON-EMPTY string, the third
+ // comparand-shape rule and the twin of `driver-sql`'s
+ // `icontainsComparandError` — deliberately the same two rejections in one
+ // check, because they are one mistake at one position.
+ if (op === '$icontains' && (typeof spec[op] !== 'string' || spec[op] === '')) {
+ throw icontainsComparandError(field, spec[op], `${path}.$icontains`);
+ }
  }
  // [#5702] The `$options`-without-`$regex` companion check that stood here is
  // GONE. It was needed while `$options` was an allowlisted MODIFIER — a key the
@@ -633,6 +652,32 @@ function assertFieldConstraintShape(
  // {@link retiredFilterOperatorError}.
 }
 
+/**
+ * [#6520] `$icontains` received a comparand that is not a non-empty string.
+ *
+ * Word for word `driver-sql`'s `icontainsComparandError`, and deliberately so:
+ * #3948 made the backends agree that an uncompilable filter is a refusal rather
+ * than a silent match-everything, and a suite that swaps this driver for SQL has
+ * to see the same refusal for the same input. Two rejections, one constructor,
+ * because they are one mistake at the comparand position:
+ *
+ * - **non-string** — `StringOperatorSchema` declares `$icontains: z.string()`,
+ * so coercing `42` to `"42"` would answer a query nobody wrote;
+ * - **empty string** — every row contains the empty substring, so the predicate
+ * constrains nothing. A dropped predicate WIDENS a result set, and on an RLS
+ * read scope that is a permission bypass rather than a degraded filter.
+ */
+function icontainsComparandError(field: string, value: unknown, path: string): Error {
+ const shown = typeof value === 'string' ? `""` : JSON.stringify(value) ?? String(value);
+ return unsupportedFilterError(
+ `Operator "$icontains" on field "${field}" at ${path} requires a NON-EMPTY string comparand, ` +
+ `received ${shown}. "$icontains" is a case-insensitive LITERAL substring search, so its ` +
+ `comparand is the text to look for — an empty one matches every row (a predicate that ` +
+ `constrains nothing), and a non-string one would have to be coerced into text this query ` +
+ `never asked for.`,
+ );
+}
+
 /**
  * [#5328] `$between`'s comparand: a two-element `[min, max]` array.
  *

packages/drivers/driver-memory/src/memory-analytics.ts

@@ -2,6 +2,8 @@
 
 import type { IAnalyticsService, AnalyticsResult, CubeMeta } from '@objectstack/spec/contracts';
 import type { Cube, AnalyticsQuery } from '@objectstack/spec/data';
+// [#6520] `$icontains`' ASCII-only fold, from the spec's one definition.
+import { asciiCaseInsensitiveRegexSource } from '@objectstack/spec/data';
 import type { InMemoryDriver } from './memory-driver.js';
 import { Logger, createLogger, nextUtcCalendarDay } from '@objectstack/core';
 import {
@@ -52,6 +54,13 @@ const MONGO_TO_CUBE_OPERATOR = Object.freeze({
  $nin: 'notIn',
  $contains: 'contains',
  $notContains: 'notContains',
+ // [#6520] This face lowers `$icontains` too, so the analytics/cube surface
+ // answers it like the driver's other two. Leaving it out would have been a
+ // LOUD refusal (`uncompilableFieldOperatorError` — "declared, but this face
+ // cannot compile it"), not a silent drop; it is added because the cube
+ // pipeline can express it, and one driver answering one operator two ways by
+ // entry point is the divergence class #5374 closed for `$contains`.
+ $icontains: 'icontains',
  $exists: 'set',
 } as const);
 
@@ -148,6 +157,18 @@ interface MongoPredicateInput {
  * DRIVER's own rule (`filterSubstringPattern`) rather than re-derived here.
  */
  readonly substring: (value: unknown) => RegExp;
+ /**
+ * [#6520] A comparand as an ASCII-case-insensitive literal-substring pattern —
+ * `$icontains`' fold, which is NOT {@link substring}'s.
+ *
+ * The two are deliberately separate functions rather than one with a flag.
+ * `substring` folds the whole Unicode range (the driver's `i` flag), which is
+ * the open defect #6682 tracks for the `$contains` family on this face; this
+ * one folds `A-Z` only, which is what the protocol says `$icontains` means
+ * (#4706 Q1 = A). Collapsing them would silently give one of the two operators
+ * the other's answer.
+ */
+ readonly asciiSubstring: (value: unknown) => RegExp;
 }
 
 type MongoPredicateBuilder = (input: MongoPredicateInput) => Record<string, unknown>;
@@ -224,6 +245,11 @@ const CUBE_OPERATOR_TO_MONGO_PREDICATE: Readonly<Record<CubeOperator, MongoPredi
  notIn: ({ comparands }) => ({ $nin: [...comparands] }),
  // A pattern, not a comparand: `raw`, and the driver's own substring rule.
  contains: ({ raw, substring }) => ({ $regex: substring(raw[0]) }),
+ // [#6520] The case-INSENSITIVE twin, folding ASCII and nothing else. It takes
+ // `asciiSubstring`, not `substring`: the neighbour above folds Unicode, so
+ // reusing it here would answer `CAFÉ` for `café` on this face while the SQL
+ // family answered no rows — the divergence #6520 closed.
+ icontains: ({ raw, asciiSubstring }) => ({ $regex: asciiSubstring(raw[0]) }),
  // The fix this issue is about. `{$not: <scalar>}` constrains nothing; the
  // negation has to wrap a pattern, which is exactly what the live query path
  // builds for `$notContains` (`memory-driver.ts` `normalizeFieldOperators`).
@@ -320,6 +346,9 @@ export class MemoryAnalyticsService implements IAnalyticsService {
  comparands: this.comparandsFor(cube, filter.member, filter.values),
  raw: filter.values,
  substring: (value) => this.driver.filterSubstringPattern(value),
+ // [#6520] `$icontains`' fold, from the spec's shared definition rather
+ // than from the driver's Unicode-folding `filterSubstringPattern`.
+ asciiSubstring: (value) => new RegExp(asciiCaseInsensitiveRegexSource(String(value))),
  });
  }
  if (Object.keys(matchStage).length > 0) {
@@ -929,6 +958,13 @@ export class MemoryAnalyticsService implements IAnalyticsService {
  'notEquals': '!=',
  'contains': 'LIKE',
  'notContains': 'NOT LIKE',
+ // [#6520] Needed because the `|| '='` fallback below is not a default, it
+ // is a wrong ANSWER: without this row `icontains` would render as `=`, an
+ // EQUALITY, in a statement offered to the author as a description of a
+ // containment query. `LIKE` is also the semantically right construct here
+ // — this exit emits SQLite-shaped SQL, and SQLite's `LIKE` folds ASCII
+ // only, which is exactly `$icontains`' domain (#4706 Q1 = A).
+ 'icontains': 'LIKE',
  'gt': '>',
  'gte': '>=',
  'lt': '<',

packages/drivers/driver-memory/src/memory-driver-filter-logic-conformance.test.ts

@@ -587,6 +587,12 @@ const DECLARED_OPERATOR_PROBES: Record<string, FilterCondition> = {
  $nin: { code: { $nin: ['100'] } } as FilterCondition,
  $contains: { name: { $contains: 'et' } } as FilterCondition,
  $notContains: { name: { $notContains: 'et' } } as FilterCondition,
+ // [#6520] The comparand is deliberately UPPER-case against a lower-case
+ // fixture (`beta`), so the probe only selects row 2 once the ASCII fold
+ // actually runs. `'et'` would have been discriminating too, and would have
+ // agreed for free on a face that never folded — which is the certification
+ // this block's own header warns against.
+ $icontains: { name: { $icontains: 'BET' } } as FilterCondition,
  $exists: { closed_at: { $exists: false } } as FilterCondition,
 };
 

packages/drivers/driver-memory/src/memory-driver.ts

@@ -1,7 +1,11 @@
 // Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
 
 import type { DriverOptions } from '@objectstack/spec/data';
-import { canonicalAstOperator } from '@objectstack/spec/data';
+// [#6520] `asciiCaseInsensitiveRegexSource` is `$icontains`' fold, defined once
+// in the spec: this face hands a PATTERN to mingo rather than comparing two
+// strings, so the fold has to live in the pattern source. See its docblock for
+// why an `i` flag is the wrong tool.
+import { canonicalAstOperator, asciiCaseInsensitiveRegexSource } from '@objectstack/spec/data';
 import type { DriverQuery, IDataDriver } from '@objectstack/spec/contracts';
 import { Logger, createLogger, nextUtcCalendarDay } from '@objectstack/core';
 import { Query, Aggregator } from 'mingo';
@@ -965,6 +969,23 @@ export class InMemoryDriver implements IDataDriver {
  case '$endsWith':
  regexConditions.push({ $regex: new RegExp(`${this.escapeRegex(val)}$`, 'i') });
  break;
+ // [#6520] `$icontains` — case-insensitive over ASCII and NOTHING else.
+ //
+ // Note what this arm does NOT do, because every neighbour above does it:
+ // it never passes the `i` flag. That flag is the FULL Unicode fold, so
+ // it would match `CAFÉ` against `café` — the answer the SQL family
+ // cannot give (SQLite folds ASCII only) and therefore the one the
+ // protocol forbids (#4706 Q1 = A). The fold instead lives in the pattern
+ // SOURCE, one `[Aa]` class per ASCII letter, built by the spec's shared
+ // `asciiCaseInsensitiveRegexSource` — the same source `driver-mongodb`
+ // binds, so the two document-shaped faces fold identically.
+ //
+ // The neighbours' `i` flags are NOT a precedent to copy here: they are
+ // the `$contains` family folding Unicode, which is the open defect #6682
+ // tracks on this face, not the behaviour to extend.
+ case '$icontains':
+ regexConditions.push({ $regex: new RegExp(asciiCaseInsensitiveRegexSource(val)) });
+ break;
  case '$between': {
  // [#5328] The arm used to be CONDITIONAL — a comparand that was not a
  // two-element array skipped it and wrote nothing, so the field