fix(drivers): a declared field written as an explicit undefined is indistinguishable from one never written

.changeset/driver-own-key-undefined-normalisation.md

@@ -0,0 +1,63 @@
+---
+"@objectstack/driver-memory": minor
+"@objectstack/driver-mongodb": minor
+---
+
+fix(drivers): a declared field written as an explicit `undefined` is indistinguishable from one never written (#9276)
+
+A row has exactly two states to say about a field, each with a defined meaning:
+**the key is absent** ("no value was ever written") or **the key holds a
+value**. An own key holding `undefined` is neither. Only a JS-backed driver can
+emit it — a SQL NULL arrives as `null`, which is a value — and every consumer
+downstream had to invent a reading of it. Measured on `origin/main`, they did
+not agree: `has(record.f)` on the real `@objectstack/formula` CEL engine reads
+it as ABSENT, `materializeDeclaredFields` reads it as ABSENT by documented
+design, and a bare `f in row` reads it as PRESENT.
+
+Both JS-backed drivers were measured separately, and they did **not** match:
+
+- **`driver-memory`** preserved the own key holding `undefined` through
+ `create` and handed it back from `find`. Its own projection path and its own
+ matcher already read the shape as absent (`projectFields` skips `undefined`
+ values, `{f: {$exists: true}}` excluded it, `{f: {$null: true}}` included it)
+ — so the returned row was the only surface in the driver still claiming the
+ key was present, and the same stored row answered `'f' in row` differently
+ depending on whether a projection was requested.
+- **`driver-mongodb`** SPLIT. `create()` returns the object it built in
+ process, so the field came back as an own key holding `undefined`; but the
+ MongoClient default is `ignoreUndefined: false` and this driver sets no
+ override, so BSON stored `null` for that same field and a subsequent `find()`
+ answered `null` — a value. One write, two answers, from one driver.
+
+Both drivers now drop own keys holding `undefined` on the way into storage, so
+a declared field written as `undefined` and one never written are the same row:
+deep-equal, same own keys, same answer to every presence test. `null` is
+untouched and stays a value.
+
+Fixed at the producer rather than at each consumer: converging one consumer
+resolves one seam, but the next consumer that reasons about key presence
+re-acquires the problem.
+
+**Behaviour that changes, precisely.** What these two packages RETURN for one
+input class, and what `driver-mongodb` STORES for it. A caller passing an
+explicitly-`undefined` property to `create`/`bulkCreate`/`update`/`updateMany`
+(or seeding `initialData`) no longer sees that key in the returned row, and no
+`null` is written for it in MongoDB. `undefined` does not survive JSON, so this
+shape cannot arrive over the wire — reaching it requires in-process code.
+
+**What does NOT change.** No accept set moves: no schema, refine, validator or
+public type is touched, nothing that parsed before is refused now, and no
+exported name is added, removed or moved. Filter results are unchanged in both
+drivers — measured identical before and after for `$null` / `$exists` /
+equality on `driver-memory`, and on `driver-mongodb` `$null: true` lowers to
+`$eq: null` and `$null: false` to `$ne: null`, which MongoDB matches
+identically against a missing field and a stored `null`.
+
+Scope on `driver-mongodb` is the INSERT doors and the values returned.
+`$set`-shaped patches are deliberately untouched: changing them would answer
+"what does a patch carrying `undefined` mean — clear the field, or leave the
+prior value standing" which is a storage-contract question, not this repair's
+to settle. On `driver-memory` the normalisation is applied POST-merge for the
+same reason — it keeps today's answer (every measured consumer read the merged
+own-key-`undefined` as "absent", and the row now says absent outright) rather
+than silently turning such a patch into a no-op.

packages/drivers/driver-memory/src/memory-driver.ts

@@ -95,6 +95,65 @@ export interface InMemoryDriverConfig {
  };
 }
 
+/**
+ * Drop every own key whose value is `undefined` (#9276).
+ *
+ * ## The rule, and why the driver owes it
+ *
+ * A row has exactly two states to say about a field, and each has a defined
+ * meaning: **the key is absent** ("no value was ever written") or **the key
+ * holds a value**. An own key holding `undefined` is NEITHER, so every consumer
+ * downstream has to pick a reading of it, and measured on `origin/main` they do
+ * not agree — `has(record.f)` on the real `@objectstack/formula` CEL engine and
+ * `materializeDeclaredFields` both read it as ABSENT, while a bare `f in row`
+ * reads it as PRESENT. That disagreement is the whole cost: a third state that
+ * nothing declares, that no consumer can resolve locally, and that only a
+ * JS-backed driver can even emit (a SQL NULL arrives as `null`, which is a
+ * value).
+ *
+ * This driver ALREADY holds "a value of `undefined` means the key is not
+ * emitted" in two of its own places, which is why normalising here is
+ * convergence rather than a new rule:
+ *
+ * - `projectFields` skips `undefined` values, so the same stored row answered
+ * `'status' in row === false` under a projection and `true` without one;
+ * - the matcher reads it as absent — measured, `{ status: { $exists: true } }`
+ * excludes it and `{ status: { $null: true } }` includes it, exactly as for
+ * a row that never carried the key at all.
+ *
+ * So the returned row was the only surface still claiming the key was present.
+ *
+ * ## Where it is applied, and what that preserves
+ *
+ * On the way INTO the backing table (see {@link InMemoryDriver.toStoredRecord}
+ * and the `initialData` seeding door), which is post-merge on the update path.
+ * That placement is load-bearing: `update(id, { f: undefined })` today merges
+ * an own key holding `undefined` over the stored value, and every measured
+ * consumer reads the result as "the field is absent". Dropping the key AFTER
+ * the merge keeps that reading byte for byte; dropping it BEFORE would make the
+ * same call a no-op that leaves the prior value standing, which is a different
+ * answer to "what does a patch carrying `undefined` mean" — a storage-contract
+ * question this normalisation deliberately does not reopen.
+ *
+ * Returns the input unchanged (same reference) when there is nothing to drop,
+ * so the common case allocates nothing — the same convention
+ * {@link InMemoryDriver.toStorageForms} follows.
+ *
+ * `@objectstack/driver-mongodb` carries a structural twin of this function on
+ * its insert doors, for the same reason its `toStorageForms` is a twin rather
+ * than an import: the two driver packages share no code. This doc comment is
+ * the canonical statement of the rule; that copy defers to it.
+ */
+function withoutUndefinedOwnKeys<T extends Record<string, any>>(record: T): T {
+ let out: Record<string, any> | undefined;
+ for (const key of Object.keys(record)) {
+ if (record[key] !== undefined) continue;
+ out ??= { ...record };
+ delete out[key];
+ }
+ return (out as T) ?? record;
+}
+
 /**
  * Snapshot for in-memory transactions.
  */
@@ -252,7 +311,7 @@ export class InMemoryDriver implements IDataDriver {
  const table = this.getTable(objectName);
  for (const record of records) {
  const id = (record as any).id || this.generateId(objectName);
- table.push({ ...record, id });
+ table.push(withoutUndefinedOwnKeys({ ...record, id }));
  }
  }
  this.logger.info('InMemory Database Connected with initial data', {
@@ -399,7 +458,7 @@ export class InMemoryDriver implements IDataDriver {
 
  const table = this.getTable(object);
 
- const newRecord = this.toStorageForms(object, {
+ const newRecord = this.toStoredRecord(object, {
  id: data.id || this.generateId(object),
  ...data,
  created_at: data.created_at || new Date().toISOString(),
@@ -426,7 +485,7 @@ export class InMemoryDriver implements IDataDriver {
  return null;
  }
 
- const updatedRecord = this.toStorageForms(object, {
+ const updatedRecord = this.toStoredRecord(object, {
  ...table[index],
  ...data,
  id: table[index].id, // Preserve original ID
@@ -525,7 +584,7 @@ export class InMemoryDriver implements IDataDriver {
  for (const record of targetRecords) {
  const index = table.findIndex(r => r.id === record.id);
  if (index !== -1) {
- const updated = this.toStorageForms(object, {
+ const updated = this.toStoredRecord(object, {
  ...table[index],
  ...data,
  updated_at: new Date().toISOString()
@@ -1467,6 +1526,18 @@ export class InMemoryDriver implements IDataDriver {
  return new RegExp(this.escapeRegex(value as string));
  }
 
+ /**
+ * The form a record takes in the backing table: no own key holding
+ * `undefined`, then every declared temporal field in its storage form.
+ *
+ * Every write door goes through here rather than through
+ * {@link toStorageForms} directly, so the two normalisations cannot drift
+ * apart door by door.
+ */
+ private toStoredRecord<T extends Record<string, any>>(object: string, record: T): T {
+ return this.toStorageForms(object, withoutUndefinedOwnKeys(record));
+ }
+
  /**
  * Put every declared temporal field of a record into its storage form — the
  * write half of the convention the filter path reads against. Returns the