From 3b093495771a9ef21b21c3fac5c2b756712009db Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 04:40:48 +0000 Subject: [PATCH 1/4] fix(objectql)!: engine.find/findOne refuse a dotted projection instead of widening to every field (#7589) The head-only known.has(head) filter kept dotted entries on the strength of a comment claiming the engine resolves them via populate; #7601 measured no populate step exists. A dotted entry is now 400 INVALID_FIELD at the engine boundary; the unknown-plain-column tolerance is explicitly kept. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01RDTnVvsgA6cUZ4xFVtPZRy --- packages/objectql/src/engine.ts | 163 ++++++++++++++++++++++++++++---- 1 file changed, 142 insertions(+), 21 deletions(-) diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index 56dcab0eec..e2ddec9d7b 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -785,13 +785,16 @@ function planFormulaProjection( * PAGE. It would pass every small-result-set test and be wrong the moment * pagination is involved. * - * SCOPE — deliberately the third verdict only. `unknown` and `dotted` names are - * NOT judged here: the ingress gate's precedence is `unknown` > `dotted` > - * unmaterializable (#4226 / #4256 / #6994), and the engine has always tolerated - * an unknown projection name by design (the `SELECT *` tolerance a few lines - * below). Widening this door to those two is a separate posture change on two - * more axes, not a free extension of this one — so a dotted path keeps reaching - * the driver exactly as before, including one whose head is a formula field. + * SCOPE — deliberately the third verdict only, on the SORT axis. `unknown` and + * `dotted` SORT names are NOT judged here: the ingress gate's precedence is + * `unknown` > `dotted` > unmaterializable (#4226 / #4256 / #6994), and widening + * this door to those two is a separate posture change on two more axes, not a + * free extension of this one — so a dotted SORT path keeps reaching the driver + * exactly as before, including one whose head is a formula field. On the + * PROJECTION axis the engine still tolerates an unknown PLAIN name by design + * (the `SELECT *` tolerance a few lines below, kept by the #7589 ruling), but + * a dotted PROJECTION entry is refused since #7589 — + * {@link assertProjectionHasNoDottedPaths}, directly below. * * A registry-less host (`schema` undefined) returns early, exactly as the * ingress gate returns early when `resolveQueryFields` cannot answer: a door @@ -844,6 +847,107 @@ function assertOrderByIsMaterializable( throw err; } +/** + * [#7589] A DOTTED projection entry — refused on the engine's own public + * boundary, the second half of #7532's ingress refusal. + * + * #7532 (PR #7588) closed this at `assertProjectionFieldsExist` + * (`400 INVALID_FIELD`), which covers everything reaching `findData`: the REST + * list route, `POST /data/:object/query`, the export route and the RPC + * dispatcher. It could not cover a caller that reaches {@link ObjectQL.find} / + * {@link ObjectQL.findOne} DIRECTLY — and that half was measured, not assumed + * (#7589): a flow-authored `get_record` node's `fields: ['name','account.name']` + * parses (`GetRecordConfigSchema` restricts nothing), travels verbatim into + * `data.find(...)`, cleared the head-only filter below on its head segment + * (`account` IS a field), and reached the driver as a projection column — where + * SQL renders `"account"."name"` against a table that was never joined, the DB + * answers `no such column`, and the #3821 recovery ladder retries `select('*')`. + * The caller asked to narrow and silently received EVERY field, byte-identical + * to no projection at all. A saved report's `query.fields` reaches this the + * same way (`plugin-reports` forwards it verbatim), as does every hook and + * internal caller. + * + * WHY A REFUSAL: ruled 2026-08-12 on #7589 (adopting the drivers seat's + * Option B) — a dotted entry the engine cannot resolve is refused loudly at + * this one site, covering every caller that reaches the engine. The head-only + * check this replaces was justified by a comment claiming the engine resolves + * relationship paths "via populate"; #7601 measured that NO populate step + * exists — the comment was the last place in the repo asserting dotted-path + * resolution does (after PR #7617) — so what is removed here is not a working + * feature but a path to widening, kept alive by a false premise. Both the typo + * (`titel.name`) and the genuine traversal intent (`account.name`) eat this + * refusal: nothing resolves either, they are not separable at this door, and + * the alternative is the over-return above (#5918's precedent, same as the + * ingress ruling). + * + * SCOPE — the dotted leg ONLY. The unknown-PLAIN-column tolerance a few lines + * below each call site is explicitly KEPT (same #7589 ruling): an unknown + * plain column is simply absent from each row, the "no records exist" failure + * that tolerance prevents is real, and it backstops registry-less hosts. A + * dotted path differs in kind — it is a projection no driver can structurally + * apply, and answering it with every column points away from both FLS and data + * minimisation. The two facts get two verdicts. + * + * A registry-less host (`schema.fields` undefined) returns early, exactly as + * the ingress gate returns early when `resolveQueryFields` cannot answer: a + * door that cannot see the field map must not invent a verdict about it. For + * that host the driver-side #3821 ladder remains the documented backstop + * (deliberately untouched — a driver-side carve-out is ruled measured-need + * only). + * + * The wording deliberately shares its core sentence and remedies with the + * ingress door's dotted refusal — one vocabulary across the doors, so a caller + * refused at the REST boundary and a caller refused here are not sent two + * different ways. Duplicated rather than imported because `metadata-protocol` + * is assembled FROM an engine, so the engine cannot import from it without + * inverting the layering; the agreement pin in + * `query-expression-conformance.test.ts` is what keeps the duplication honest + * (same mechanism as the sort axis' three-door remedy pin). + */ +function assertProjectionHasNoDottedPaths( + object: string, + operation: 'find' | 'findOne', + schema: any, + fields: unknown, +): void { + if (!Array.isArray(fields) || fields.length === 0) return; + if (!schema?.fields) return; + const dotted = fields.filter( + (f): f is string => typeof f === 'string' && f.includes('.')); + if (dotted.length === 0) return; + const first = dotted[0]; + const head = first.split('.')[0]; + const headDef: any = (schema.fields as any)[head]; + const crossesRelation = headDef != null && REFERENCE_VALUE_TYPES.has(headDef.type); + const err: any = new Error( + (crossesRelation + ? `ObjectQL.${operation}('${object}') projects '${first}', which follows the relationship ` + + `'${head}' into another object — 'fields' reaches only columns of '${object}' itself` + : `ObjectQL.${operation}('${object}') projects '${first}', a dotted path — 'fields' reaches ` + + `only whole columns of '${object}', not values inside them`) + + (dotted.length > 1 ? ` (also: ${dotted.slice(1).join(', ')})` : '') + + '. No driver resolves it: the path reaches the driver as a column name, matches no ' + + 'column, and the projection falls back to EVERY field — a narrower request answered ' + + 'with a wider response.' + + (crossesRelation + ? ` Read the related record with 'expand' (\`{ expand: { ${head}: { object: '', ` + + `fields: [''] } } }\` to choose its columns), or denormalise the value onto ` + + `'${object}' (a stored field, written when the source changes) and name that.` + : ` Name the whole column ('${head}') and read into its value in the caller.`), + ); + // `INVALID_FIELD`, not a new code, and 400 rather than 500 — the same + // reasoning `assertOrderByIsMaterializable` records for `INVALID_SORT`: one + // condition ("this projection was not applied as written") keeps ONE wire + // code however the caller reached it, so a host surfacing engine errors over + // HTTP answers the same envelope on both doors. + err.status = 400; + err.code = 'INVALID_FIELD'; + err.field = first; + err.fields = dotted; + err.object = object; + throw err; +} + /** * Evaluate formula virtual fields against the raw rows a driver handed back — * the read path (`find` / `findOne`) and, since #5504, the write path's @@ -7140,16 +7244,29 @@ export class ObjectQL implements IObjectQLEngine { // dropped. `fillQueryAstDefaults` has already normalised `orderBy` into // `SortNode[]`, so the names read here are the ones the driver would get. assertOrderByIsMaterializable(object, 'find', _findSchema, ast.orderBy); + // [#7589] The projection's DOTTED leg, judged on the caller's own + // spellings BEFORE the formula planner rewrites the projection: a dotted + // entry is structurally unresolvable (no populate step exists — #7601) + // and is refused loudly instead of riding its head segment into the + // driver, where the #3821 ladder answered it with EVERY field. + assertProjectionHasNoDottedPaths(object, 'find', _findSchema, ast.fields); const _findFormula = planFormulaProjection(_findSchema, ast.fields); if (_findFormula.projected) ast.fields = _findFormula.projected; - // Drop any requested field that doesn't exist on the schema. Without - // this, drivers (notably SqlDriver) emit `SELECT unknown_col FROM ...` - // which the DB rejects ("no such column") — and SqlDriver swallows - // that error and returns `[]`, making a frontend bug (e.g. a generic - // view requesting `name`/`due_date` on every object) look like "no - // records exist". Silently filtering matches the existing OData + // Drop any requested PLAIN field that doesn't exist on the schema. + // Without this, drivers (notably SqlDriver) emit `SELECT unknown_col + // FROM ...` which the DB rejects ("no such column") — and SqlDriver + // swallows that error and returns `[]`, making a frontend bug (e.g. a + // generic view requesting `name`/`due_date` on every object) look like + // "no records exist". Silently filtering matches the existing OData // tolerance and Salesforce/Postgres behavior of `SELECT *` semantics. + // + // [#7589] This tolerance is for unknown PLAIN columns ONLY, and it is + // KEPT deliberately (ruled 2026-08-12): the "no records exist" failure it + // prevents is real, and the driver-side half of the same tolerance + // backstops registry-less hosts. A structurally unresolvable (dotted) + // projection is a different fact and no longer reaches this filter via + // the engine — `assertProjectionHasNoDottedPaths` above refused it. if (_findSchema?.fields && Array.isArray(ast.fields) && ast.fields.length > 0) { const known = new Set(Object.keys(_findSchema.fields)); // Always allow the primary key + audit columns even if not present in @@ -7158,12 +7275,9 @@ export class ObjectQL implements IObjectQLEngine { known.add('id'); known.add('created_at'); known.add('updated_at'); - const filtered = ast.fields.filter(f => { - // Keep relationship paths like `owner.name` — the engine will - // resolve those via populate; only validate top-level segment. - const head = f.split('.')[0]; - return known.has(head); - }); + // Whole names, no head-splitting: only plain entries reach here (the + // dotted refusal above fired on anything carrying a '.'). + const filtered = ast.fields.filter(f => known.has(f)); // Guard against an empty projection — fall back to `*` so the // request still returns rows. An empty SELECT list would either // 400 in Postgres or silently project nothing. @@ -7297,12 +7411,19 @@ export class ObjectQL implements IObjectQLEngine { // dropped sort does not merely reorder the answer, it returns a DIFFERENT // record, and the one it returns looks exactly as legitimate. assertOrderByIsMaterializable(objectName, 'findOne', _findOneSchema, ast.orderBy); + // [#7589] Same dotted-projection refusal as `find`, same position: on the + // caller's own spellings, before the formula planner rewrites them. The + // measured flow chain (`get_record` → `data.findOne`) reaches THIS verb + // whenever `limit` is absent or 1, so a hole here would be the same hole. + assertProjectionHasNoDottedPaths(objectName, 'findOne', _findOneSchema, ast.fields); // [#7642] Caller's own projection, before planning rewrites it — see `find`. const _findOneRequestedFields = Array.isArray(ast.fields) ? [...ast.fields] : undefined; const _findOneFormula = planFormulaProjection(_findOneSchema, ast.fields); if (_findOneFormula.projected) ast.fields = _findOneFormula.projected; - // Drop unknown fields — see equivalent block in `find()` for rationale. + // Drop unknown PLAIN fields — see the equivalent block in `find()` for + // the rationale, and for why this tolerance is plain-columns-only ([#7589] + // refused any dotted entry above, so none reaches this filter). if (_findOneSchema?.fields && Array.isArray(ast.fields) && ast.fields.length > 0) { const known = new Set(Object.keys(_findOneSchema.fields)); // Always allow the primary key + audit columns even if not present @@ -7310,7 +7431,7 @@ export class ObjectQL implements IObjectQLEngine { known.add('id'); known.add('created_at'); known.add('updated_at'); - const filtered = ast.fields.filter(f => known.has(f.split('.')[0])); + const filtered = ast.fields.filter(f => known.has(f)); ast.fields = filtered.length > 0 ? filtered : undefined; } From 1621a712555d76b8f9fc141d51837e3a485c9058 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 04:55:08 +0000 Subject: [PATCH 2/4] test(objectql): pin the #7589 engine-door dotted-projection refusal, the kept plain tolerance, and the cross-door wording agreement Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01RDTnVvsgA6cUZ4xFVtPZRy --- .../src/query-expression-conformance.test.ts | 209 ++++++++++++++++++ 1 file changed, 209 insertions(+) diff --git a/packages/objectql/src/query-expression-conformance.test.ts b/packages/objectql/src/query-expression-conformance.test.ts index 08c0785296..f3e39939f2 100644 --- a/packages/objectql/src/query-expression-conformance.test.ts +++ b/packages/objectql/src/query-expression-conformance.test.ts @@ -1050,6 +1050,215 @@ describe('#4226 — sort / select / expand on the list path (real ObjectQL engin expect(Object.keys(r.records[0]).sort()).toEqual(['created_at', 'id', 'owner_id']); }); + // ───────────────────────────────────────────────────────────── + // [#7589] DOTTED PROJECTION at the ENGINE door — #7532's second + // half, same shape as #7095's sort refusal one section up. The + // ingress pins above cannot cover a caller that reaches + // `engine.find` / `engine.findOne` DIRECTLY (flows' `get_record`, + // saved reports, hooks, registry-less hosts), which is the exact + // caller set the drivers seat measured widening end-to-end. + // Controls first, then the rejections, then the KEPT tolerance. + // ───────────────────────────────────────────────────────────── + + it('[#7589 CONTROL] a plain projection through `engine.find` narrows to exactly the named columns', async () => { + // FIRST, and as a key-SET equality: an over-return defect passes any + // assertion written as "does not contain X", so the whole point of + // this axis has to be pinned as an equality or it pins nothing. + const rows: any[] = await engine.find('showcase_task', { fields: ['title', 'status'] }); + expect(rows).toHaveLength(5); + for (const r of rows) expect(Object.keys(r).sort()).toEqual(['status', 'title']); + }); + + it('[#7589 CONTROL] the audit-column allowance survives the filter rewrite', async () => { + // `id`/`created_at`/`updated_at` are force-allowed even when absent + // from schema.fields — the filter now matches WHOLE names, and this + // pins that dropping the head-split did not drop the allowance. + const rows: any[] = await engine.find('showcase_task', { fields: ['id', 'title', 'created_at'] }); + expect(Object.keys(rows[0]).sort()).toEqual(['created_at', 'id', 'title']); + }); + + it('`engine.find` REFUSES a dotted projection instead of widening to every field', async () => { + // The measured chain's exact call shape: `crud-nodes.ts` `get_record` + // hands flow-authored config to `data.find(objectName, { where, + // fields, limit, context })` with NO ingress gate in between. Before + // #7589 this answered 200 with EVERY column, byte-identical to no + // projection at all (`account.name` cleared the head-only filter on + // its head segment; the driver matched no column; the #3821 ladder + // retried `select('*')`). + await expect(engine.find('showcase_task', { + where: { status: 'open' }, + fields: ['title', 'project_id.name'], + limit: 10, + })).rejects.toMatchObject({ + // ADR-0112 envelope — a rejection case asserts code AND status, + // never merely that something was thrown. + status: 400, + code: 'INVALID_FIELD', + field: 'project_id.name', + object: 'showcase_task', + }); + }); + + it('`engine.findOne` refuses it too — `get_record` without `limit > 1` reaches THIS verb', async () => { + // `where` is present so this is the projection verdict and not + // `requireFindOnePredicate` answering first. + await expect(engine.findOne('showcase_task', { + where: { status: 'open' }, + fields: ['title', 'project_id.name'], + })).rejects.toMatchObject({ + status: 400, + code: 'INVALID_FIELD', + field: 'project_id.name', + object: 'showcase_task', + }); + }); + + it('a projection that is ONLY a dotted path is refused — the empty-fallback widening shape', async () => { + // The worst composition: every entry unresolvable used to EMPTY the + // projection, and the empty-projection guard fell back to `*`. + await expect(engine.find('showcase_task', { fields: ['project_id.name'] })) + .rejects.toMatchObject({ status: 400, code: 'INVALID_FIELD', field: 'project_id.name' }); + }); + + it('a dotted path whose head is a formula field is refused on the same terms', async () => { + // `assertOrderByIsMaterializable`'s scope note used to record that a + // dotted path "keeps reaching the driver … including one whose head is + // a formula field". On the PROJECTION axis that is no longer true, and + // this pin is what keeps the two docblocks honest. + await expect(engine.find('showcase_task', { fields: ['sort_key.length'] })) + .rejects.toMatchObject({ status: 400, code: 'INVALID_FIELD', field: 'sort_key.length' }); + }); + + it('an unknown HEAD is refused as dotted too — the engine door has no unknown-name verdict to defer to', async () => { + // DIFFERENT from the ingress door, deliberately: ingress refuses + // unknown plain names, so its precedence is `unknown` > `dotted`. The + // engine TOLERATES unknown plain names (the kept #3821-family + // tolerance below) — so at this door the dotted verdict is the only + // refusal there is, and `no_such.name` eats it: a dotted entry with an + // unknown head is still a projection nothing resolves, and dropping it + // silently is how the only-dotted shape above widened. + await expect(engine.find('showcase_task', { fields: ['no_such.name'] })) + .rejects.toMatchObject({ status: 400, code: 'INVALID_FIELD', field: 'no_such.name' }); + }); + + it('the engine refusal names the entry point, the relationship, `expand` and the stored-field remedy', async () => { + const err: any = await engine + .find('showcase_task', { fields: ['title', 'project_id.name'] }) + .then(() => null, (e: unknown) => e); + expect(err).toBeTruthy(); + expect(err.status).toBe(400); + expect(err.code).toBe('INVALID_FIELD'); + // It must name the entry point, or a caller who never wrote a query + // parameter cannot tell which door refused them (#7095's rule). + expect(err.message).toMatch(/ObjectQL\.find\('showcase_task'\)/); + expect(err.message).toMatch(/follows the relationship 'project_id'/); + // The two remedies, in the ingress door's vocabulary: `expand` is the + // sanctioned door for related data on this axis, and the denormalise + // prescription is the same STORED-field wording every other refusal + // on these axes uses (#6924 / #6673). + expect(err.message).toMatch(/expand/); + expect(err.message).toMatch(/a stored field, written when the source changes/); + }); + + it('a dotted path under a non-reference head gets the other message', async () => { + // `title` holds text — "follows the relationship" would be a lie and + // `expand` the wrong prescription. + await expect(engine.find('showcase_task', { fields: ['title.length'] })) + .rejects.toMatchObject({ status: 400, code: 'INVALID_FIELD', field: 'title.length' }); + await expect(engine.find('showcase_task', { fields: ['title.length'] })) + .rejects.toThrow(/whole columns/); + }); + + it('the ingress door and the engine door agree word-for-word on the dotted verdict', async () => { + // Pins the AGREEMENT itself, same mechanism as the sort axis' remedy + // pin above: `metadata-protocol` is assembled FROM an engine, so the + // engine cannot import the wording without inverting the layering — + // the prose is duplicated, and this is what keeps the duplication + // honest. Goes red if either door's core sentence or remedy is + // reworded without the other. + const core = /No driver resolves it: the path reaches the driver as a column name, matches no column, and the projection falls back to EVERY field/; + const remedy = /denormalise the value onto 'showcase_task' \(a stored field, written when the source changes\)/; + const ingress: any = await protocol + .findData({ object: 'showcase_task', query: { fields: ['project_id.name'] } }) + .then(() => null, (e: unknown) => e); + const direct: any = await engine + .find('showcase_task', { fields: ['project_id.name'] }) + .then(() => null, (e: unknown) => e); + expect(ingress.message).toMatch(core); + expect(direct.message).toMatch(core); + expect(ingress.message).toMatch(remedy); + expect(direct.message).toMatch(remedy); + }); + + // ───────────────────────────────────────────────────────────── + // [#7589] THE KEPT TOLERANCE — the ruling's explicit carve-out, + // pinned as behaviour so a future tighten cannot ride in on this + // card's precedent without meeting its own ruling. + // ───────────────────────────────────────────────────────────── + + it('[#7589 GUARD] an unknown PLAIN column is still dropped silently — mixed projection', async () => { + // Ruled KEPT 2026-08-12: an unknown plain column is simply absent + // from each row; refusing it re-opens the "no records exist" failure + // #3821 closed. The row set is unchanged and the known column + // narrows. + const rows: any[] = await engine.find('showcase_task', { fields: ['title', 'no_such_field'] }); + expect(rows).toHaveLength(5); + for (const r of rows) expect(Object.keys(r).sort()).toEqual(['title']); + }); + + it('[#7589 GUARD] a projection of ONLY unknown plain columns still falls back to every field', async () => { + // The `SELECT *` fallback itself, pinned as KEPT for the plain case: + // this is the documented tolerance the ruling preserves, not a defect + // this card missed. (The DOTTED route into this same fallback is what + // was closed above.) + const rows: any[] = await engine.find('showcase_task', { fields: ['no_such_field'] }); + expect(rows).toHaveLength(5); + expect(Object.keys(rows[0])).toEqual(expect.arrayContaining(['id', 'title', 'status'])); + }); + + it('[#7589 GUARD] `engine.findOne` keeps the same plain tolerance', async () => { + const row: any = await engine.findOne('showcase_task', { + where: { title: 'A' }, fields: ['title', 'no_such_field'], + }); + expect(row).toBeTruthy(); + expect(Object.keys(row).sort()).toEqual(['title']); + }); + + it('[#7589 GUARD] a registry-less object gets NO verdict — the door cannot see the field map', async () => { + // An object the registry does not know: the engine has no field map, + // so it must not invent a dotted verdict about it (same early-return + // the ingress gate makes when `resolveQueryFields` cannot answer). + // For that host the driver-side #3821 ladder is the documented + // backstop — which is exactly why the ruling KEEPS the ladder. + await expect(engine.find('unregistered_thing', { fields: ['a.b'] })) + .resolves.toEqual([]); + }); + + it('an `expand` sub-read raises the refusal, which the expand backstop downgrades to a warning', async () => { + // MEASURED, same as the sort axis' pin above: a nested `fields` is + // forwarded into `expandRelatedRecords`' own `this.find(...)`, where + // the [#7589] refusal fires — inside the pre-existing graceful- + // degradation `catch` ("if expand fails, keep original IDs"), which + // swallows every expand failure, this one included. Outcome improves + // from SILENT (a widened sub-read) to OBSERVABLE (a warning carrying + // the field name and the fix) — but it is not a refusal, and this pin + // says so rather than implying #7589 closed it. Reversing that catch + // is the #3821-family decision the sort axis' pin already defers. + const rows: any = await engine.find('showcase_task', { + expand: { parent_id: { fields: ['project_id.name'] } as any }, + }); + expect(rows).toHaveLength(5); + expect(rows.filter((r: any) => typeof r.parent_id === 'object' && r.parent_id !== null)).toHaveLength(0); + expect(rows.some((r: any) => r.parent_id === 't_A')).toBe(true); + // CONTROL — a PLAIN nested projection in the same position still + // expands, so the assertion above is about the refusal and not about + // expand being broken for every nested `fields`. + const ok: any = await engine.find('showcase_task', { + expand: { parent_id: { fields: ['title'] } as any }, + }); + expect(ok.some((r: any) => typeof r.parent_id === 'object' && r.parent_id !== null)).toBe(true); + }); + // ───────────────────────────────────────────────────────────── // EXPAND — control group, then rejected // ───────────────────────────────────────────────────────────── From 482f917ae823bf1dd0cf4fd4a5002a267e47410f Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 05:35:51 +0000 Subject: [PATCH 3/4] chore(spec): register the engine-dotted-projection-refused ADR-0087 semantic entry + changeset (#7589) One entry file under entries/semantic/, registry region + spec-changes.json + upgrade guide regenerated. Breaking changeset per the #7095 precedent: same door (engine public API), same class (silent degradation becomes a refusal). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01RDTnVvsgA6cUZ4xFVtPZRy --- .../engine-dotted-projection-refused.md | 55 ++++++++++++++++ docs/protocol-upgrade-guide.md | 7 +++ packages/spec/spec-changes.json | 14 +++++ .../17.engine-dotted-projection-refused.ts | 62 +++++++++++++++++++ packages/spec/src/migrations/registry.ts | 58 +++++++++++++++++ 5 files changed, 196 insertions(+) create mode 100644 .changeset/engine-dotted-projection-refused.md create mode 100644 packages/spec/src/migrations/entries/semantic/17.engine-dotted-projection-refused.ts diff --git a/.changeset/engine-dotted-projection-refused.md b/.changeset/engine-dotted-projection-refused.md new file mode 100644 index 0000000000..010f7fa4be --- /dev/null +++ b/.changeset/engine-dotted-projection-refused.md @@ -0,0 +1,55 @@ +--- +"@objectstack/objectql": minor +--- + + + +fix(objectql)!: `engine.find` / `engine.findOne` refuse a dotted projection instead of widening the response to every field (#7589) + +`engine.find()` and `engine.findOne()` are a **public API**, and a `fields` +entry carrying a dotted path (`['name', 'account.name']`) — which used to +answer 200 with **every** column, byte-identical to no projection at all — +now **throws `400 INVALID_FIELD`**. + +#7532 (PR #7588) closed this at the REST ingress +(`assertProjectionFieldsExist`), covering everything that reaches `findData`. +A caller reaching the engine directly passed through none of it, and that +caller set was measured, not assumed (#7589): a flow `get_record` node's +authored `fields: ['name', 'account.name']` parses (`GetRecordConfigSchema` +restricts nothing), travels verbatim into `data.find(...)` / +`data.findOne(...)`, cleared the engine's head-only projection filter on its +head segment (`account` IS a field), and reached the driver as a projection +column — where SQL renders `"account"."name"` against a table that was never +joined, the DB answers `no such column`, and the driver's #3821 recovery +ladder retries `select('*')`. The caller asked to narrow and silently +received everything, pointing away from both FLS and data minimisation. A +saved report's `query.fields` (`plugin-reports` forwards it verbatim) reached +it the same way. + +The head-only check was justified by its own comment: "the engine will +resolve those via populate". **No populate step exists** — #7601 measured it, +and this comment was the last place in the repo asserting dotted-path +resolution does. The comment and the check it explained are gone together; +what is removed is not a working feature but a path to widening, kept alive +by a false premise. + +**FROM → TO**: a direct-engine caller (flow `get_record` `fields`, saved +report `query.fields`, hook code) projecting `account.name` reads the related +record with `expand` (`{ expand: { account: { object: '', fields: +['name'] } } }`) while keeping the reference column itself in `fields`, or +denormalises the value onto the queried object (a stored field, written when +the source changes) and names that. A plain reference column (`fields: +['account']`) still projects. + +**Deliberately KEPT** (same ruling, 2026-08-12): the unknown-PLAIN-column +tolerance — an unknown plain name is still dropped silently and an +all-unknown projection still falls back to `*`, because the "no records +exist" failure that tolerance prevents is real. A registry-less host (no +field map) gets **no** verdict, exactly as the ingress gate returns early +there; for that host the driver-side #3821 ladder remains the documented +backstop, and a driver-side carve-out stays measured-need only. One path is +observable rather than refused: a dotted `fields` inside a nested `expand` +raises this refusal inside `expandRelatedRecords`' pre-existing +graceful-degradation `catch`, so it logs a warning naming the field and the +fix and retains the raw foreign keys — the same posture the sort axis (#7095) +records for the same catch. diff --git a/docs/protocol-upgrade-guide.md b/docs/protocol-upgrade-guide.md index a1d8e2f64b..64395e91d6 100644 --- a/docs/protocol-upgrade-guide.md +++ b/docs/protocol-upgrade-guide.md @@ -391,6 +391,13 @@ What makes this one cheaper to meet than its two siblings, and worth saying beca - **`driver-sql-distinct-bare-filter-typed`** — `SqlDriver.distinct() third argument — any value` → a bare FilterCondition (@objectstack/spec/data) — the same value find() carries under query.where, never a query envelope - Why not automatic: This entry records a TYPE being added, not a surface being withdrawn, and it says so up front because the distinction decides who has to do anything. `distinct` is not declared on `IDataDriver`, so #5181 / #6075 never reached it and it kept `filters?: any` while its body said something far more specific — `applyFilters(builder, filters)` is handed the ARGUMENT ITSELF, never a `.where` off it. ⚠️ RUNTIME BEHAVIOUR IS UNCHANGED by this entry's change: not one statement moved, so no upgrade breaks at run time and nothing that answered correctly stops. What the annotation removes is a compile-time hole, measured rather than assumed: a truthy NON-OBJECT third argument — `distinct('orders', 'product', 'completed')` — used to type-check and resolve the UNFILTERED set, because `applyFilters` emits no predicate at all for a truthy non-object, non-array filter. A call meaning "which products among completed orders" answered with EVERY product, silently. That spelling is now TS2345 at the call site. This is a driver CALL ARGUMENT — code, never stack metadata — so there is no source for the D2 chain to rewrite and deliberately no schema tombstone, the disposition `data-driver-find-stream-retired` (#4484), `storage-service-list-retired` (#5540), `actor-user-roles-to-positions` (#6011) and `driver-aggregate-undeclared-key-aliases-removed` (#6321) already carry. ⚠️ It differs from those four in ONE measured way a reader should not have to infer: because nothing changed at run time, an untyped JS caller is not affected BY THE UPGRADE at all. The entry is here for a different reason — such a caller is exactly the one tsc can never reach, and the silent widening above is a defect they may ALREADY be sitting on, before and after this major. The generated upgrade guide is the only channel that reaches them, which is why the fix is written down rather than left to the compiler. ⛔ The reverse mismatch is NOT closed and no type can close it: `FilterCondition` is an open map (`[key: string]: any`) because a filter key IS a field name, so a query envelope `{ object, where }` is structurally a valid filter — one constraining columns named `object` and `where` — and so is a FilterArray. Both reach `distinct` type-checked and are refused at run time, loudly, with INVALID_FILTER / 400. `driver-memory`'s opposite half — where the BARE spelling returns the unfiltered set in silence — stays open under the #5499 freeze (#6320). ADR-0087, #6320. - Done when: No caller passes a non-object to `distinct()`'s third argument. A scalar there is now a compile error (`TS2345: Argument of type 'string' is not assignable to parameter of type 'FilterCondition'`); rewrite it as the bare filter it was always meant to be — `'completed'` becomes `{ status: 'completed' }`. ⚠️ That is NOT an equivalent rewrite: the old spelling returned the UNFILTERED set, so the answer changes once fixed, and the changed answer is the one the call always meant. An untyped JS caller gets no compile error and no behaviour change — for them this entry is the only notice that the spelling never filtered anything. A query envelope or a FilterArray in that slot still compiles and is rejected at run time with INVALID_FILTER / 400. +- **`engine-dotted-projection-refused`** — `engine.find(object, { fields }) and engine.findOne(object, { fields }) carrying a dotted entry (`account.name`) — the direct engine path, not the REST ingress` → read the related record with `expand` (`{ expand: { account: { object: '', fields: ['name'] } } }`), keeping the reference column itself in `fields` — the relation is carried by that column and projecting it away leaves expansion nothing to resolve (#7537); or denormalise the value onto the queried object (a stored field, written when the source changes) and name that — the same remedy the REST ingress has prescribed since #7532, and the sort axis since #6924 + - Why not automatic: #7532 (PR #7588) closed the PROJECTION axis' dotted leg at the REST ingress (`assertProjectionFieldsExist`, `400 INVALID_FIELD`), which covers everything reaching `findData`. A caller reaching `engine.find()` / `engine.findOne()` DIRECTLY passed through none of it, and that caller set was measured, not assumed (#7589): a flow `get_record` node's authored `fields: ['name', 'account.name']` parses (`GetRecordConfigSchema` restricts nothing), travels verbatim into `data.find(...)`, cleared the engine's head-only projection filter on its head segment (`account` IS a field), and reached the driver as a projection column — where SQL renders `"account"."name"` against a table that was never joined, the DB answers `no such column`, and the driver's #3821 recovery ladder retries `select('*')`. The caller asked to narrow and silently received EVERY field, byte-identical to no projection at all, pointing away from both FLS and data minimisation. + +Ruled 2026-08-12 on #7589 (Option B): a dotted entry the engine cannot resolve is refused loudly at the engine's own head-only projection filter, covering every caller that reaches the engine. The check it replaces was justified by a comment claiming the engine resolves relationship paths "via populate"; #7601 measured that NO populate step exists — after PR #7617 that comment was the last place in the repo asserting dotted-path resolution does — so what was removed is not a working feature but a path to widening, kept alive by a false premise. The unknown-PLAIN-column tolerance is explicitly KEPT by the same ruling (an unknown plain name still drops silently; an all-unknown projection still falls back to `*`), a registry-less host gets no verdict (the driver-side #3821 ladder remains its documented backstop, and a driver-side carve-out is measured-need only), and a dotted `fields` inside a nested `expand` degrades to an observable warning rather than a refusal — `expandRelatedRecords`' pre-existing graceful-degradation `catch` swallows every expand failure, the same posture the sort axis (#7095) records for the same catch. + +This is a CODE-path API, not stored metadata, so — like `engine-find-formula-order-by-refused` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and the ledger entry is the notification channel. No mechanical rewrite exists: the platform cannot decide between `expand` and denormalisation for the caller, and it must not resolve the path itself — no driver ever did, and inventing a join here is a feature decision, not a migration. #7589, #7532, #7601, #3821, #5918, ADR-0112. + - Done when: No `engine.find` / `engine.findOne` call site passes a dotted `fields` entry, no flow `get_record` config authors one, and no saved report's `query.fields` names one — grep flow definitions and report definitions for a `fields` entry containing a `.`, and rewrite each to `expand` (keeping the reference column projected) or to a denormalised stored column. Reads complete with no `INVALID_FIELD` whose message says "follows the relationship" or "a dotted path", and no "Failed to expand relationship field" warning whose error text does. - **`engine-find-formula-order-by-refused`** — `engine.find(object, { orderBy }) and engine.findOne(object, { orderBy }) naming a `formula` field — the direct engine path, not the REST ingress` → denormalise the value onto the object (a stored field, written when the source changes) and sort by that — the same remedy the REST ingress has prescribed since #6924 / #6994; a `summary` field is unaffected and still sorts, because it gets a real maintained column - Why not automatic: #4226 / #4256 / #6994 closed the SORT axis at the REST ingress (`assertSortFieldsExist`, `400 INVALID_SORT`), which covers everything reaching `findData`: the list route, `POST /data/:object/query`, the export route and the RPC dispatcher. A caller reaching `engine.find()` / `engine.findOne()` DIRECTLY passed through none of it, and a `formula` ORDER BY there was dropped in silence. Measured on a real driver: `asc` and `desc` came back BYTE-IDENTICAL, in insertion order, under a success, with the rows carrying the very values they were asked to be ordered by. No column exists to order by (a formula is computed on read, so no driver materialises one), so the ORDER BY reached the driver, found nothing, and the unknown-column backstop returned the rows unordered. diff --git a/packages/spec/spec-changes.json b/packages/spec/spec-changes.json index a8f7c4e8ff..6d7df31901 100644 --- a/packages/spec/spec-changes.json +++ b/packages/spec/spec-changes.json @@ -671,6 +671,13 @@ "toMajor": 17, "rationale": "This entry records a TYPE being added, not a surface being withdrawn, and it says so up front because the distinction decides who has to do anything. `distinct` is not declared on `IDataDriver`, so #5181 / #6075 never reached it and it kept `filters?: any` while its body said something far more specific — `applyFilters(builder, filters)` is handed the ARGUMENT ITSELF, never a `.where` off it. ⚠️ RUNTIME BEHAVIOUR IS UNCHANGED by this entry's change: not one statement moved, so no upgrade breaks at run time and nothing that answered correctly stops. What the annotation removes is a compile-time hole, measured rather than assumed: a truthy NON-OBJECT third argument — `distinct('orders', 'product', 'completed')` — used to type-check and resolve the UNFILTERED set, because `applyFilters` emits no predicate at all for a truthy non-object, non-array filter. A call meaning \"which products among completed orders\" answered with EVERY product, silently. That spelling is now TS2345 at the call site. This is a driver CALL ARGUMENT — code, never stack metadata — so there is no source for the D2 chain to rewrite and deliberately no schema tombstone, the disposition `data-driver-find-stream-retired` (#4484), `storage-service-list-retired` (#5540), `actor-user-roles-to-positions` (#6011) and `driver-aggregate-undeclared-key-aliases-removed` (#6321) already carry. ⚠️ It differs from those four in ONE measured way a reader should not have to infer: because nothing changed at run time, an untyped JS caller is not affected BY THE UPGRADE at all. The entry is here for a different reason — such a caller is exactly the one tsc can never reach, and the silent widening above is a defect they may ALREADY be sitting on, before and after this major. The generated upgrade guide is the only channel that reaches them, which is why the fix is written down rather than left to the compiler. ⛔ The reverse mismatch is NOT closed and no type can close it: `FilterCondition` is an open map (`[key: string]: any`) because a filter key IS a field name, so a query envelope `{ object, where }` is structurally a valid filter — one constraining columns named `object` and `where` — and so is a FilterArray. Both reach `distinct` type-checked and are refused at run time, loudly, with INVALID_FILTER / 400. `driver-memory`'s opposite half — where the BARE spelling returns the unfiltered set in silence — stays open under the #5499 freeze (#6320). ADR-0087, #6320." }, + { + "surface": "engine.find(object, { fields }) and engine.findOne(object, { fields }) carrying a dotted entry (`account.name`) — the direct engine path, not the REST ingress", + "replacement": "read the related record with `expand` (`{ expand: { account: { object: '', fields: ['name'] } } }`), keeping the reference column itself in `fields` — the relation is carried by that column and projecting it away leaves expansion nothing to resolve (#7537); or denormalise the value onto the queried object (a stored field, written when the source changes) and name that — the same remedy the REST ingress has prescribed since #7532, and the sort axis since #6924", + "migrationId": "engine-dotted-projection-refused", + "toMajor": 17, + "rationale": "#7532 (PR #7588) closed the PROJECTION axis' dotted leg at the REST ingress (`assertProjectionFieldsExist`, `400 INVALID_FIELD`), which covers everything reaching `findData`. A caller reaching `engine.find()` / `engine.findOne()` DIRECTLY passed through none of it, and that caller set was measured, not assumed (#7589): a flow `get_record` node's authored `fields: ['name', 'account.name']` parses (`GetRecordConfigSchema` restricts nothing), travels verbatim into `data.find(...)`, cleared the engine's head-only projection filter on its head segment (`account` IS a field), and reached the driver as a projection column — where SQL renders `\"account\".\"name\"` against a table that was never joined, the DB answers `no such column`, and the driver's #3821 recovery ladder retries `select('*')`. The caller asked to narrow and silently received EVERY field, byte-identical to no projection at all, pointing away from both FLS and data minimisation.\n\nRuled 2026-08-12 on #7589 (Option B): a dotted entry the engine cannot resolve is refused loudly at the engine's own head-only projection filter, covering every caller that reaches the engine. The check it replaces was justified by a comment claiming the engine resolves relationship paths \"via populate\"; #7601 measured that NO populate step exists — after PR #7617 that comment was the last place in the repo asserting dotted-path resolution does — so what was removed is not a working feature but a path to widening, kept alive by a false premise. The unknown-PLAIN-column tolerance is explicitly KEPT by the same ruling (an unknown plain name still drops silently; an all-unknown projection still falls back to `*`), a registry-less host gets no verdict (the driver-side #3821 ladder remains its documented backstop, and a driver-side carve-out is measured-need only), and a dotted `fields` inside a nested `expand` degrades to an observable warning rather than a refusal — `expandRelatedRecords`' pre-existing graceful-degradation `catch` swallows every expand failure, the same posture the sort axis (#7095) records for the same catch.\n\nThis is a CODE-path API, not stored metadata, so — like `engine-find-formula-order-by-refused` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and the ledger entry is the notification channel. No mechanical rewrite exists: the platform cannot decide between `expand` and denormalisation for the caller, and it must not resolve the path itself — no driver ever did, and inventing a join here is a feature decision, not a migration. #7589, #7532, #7601, #3821, #5918, ADR-0112." + }, { "surface": "engine.find(object, { orderBy }) and engine.findOne(object, { orderBy }) naming a `formula` field — the direct engine path, not the REST ingress", "replacement": "denormalise the value onto the object (a stored field, written when the source changes) and sort by that — the same remedy the REST ingress has prescribed since #6924 / #6994; a `summary` field is unaffected and still sorts, because it gets a real maintained column", @@ -1673,6 +1680,13 @@ "toMajor": 17, "rationale": "This entry records a TYPE being added, not a surface being withdrawn, and it says so up front because the distinction decides who has to do anything. `distinct` is not declared on `IDataDriver`, so #5181 / #6075 never reached it and it kept `filters?: any` while its body said something far more specific — `applyFilters(builder, filters)` is handed the ARGUMENT ITSELF, never a `.where` off it. ⚠️ RUNTIME BEHAVIOUR IS UNCHANGED by this entry's change: not one statement moved, so no upgrade breaks at run time and nothing that answered correctly stops. What the annotation removes is a compile-time hole, measured rather than assumed: a truthy NON-OBJECT third argument — `distinct('orders', 'product', 'completed')` — used to type-check and resolve the UNFILTERED set, because `applyFilters` emits no predicate at all for a truthy non-object, non-array filter. A call meaning \"which products among completed orders\" answered with EVERY product, silently. That spelling is now TS2345 at the call site. This is a driver CALL ARGUMENT — code, never stack metadata — so there is no source for the D2 chain to rewrite and deliberately no schema tombstone, the disposition `data-driver-find-stream-retired` (#4484), `storage-service-list-retired` (#5540), `actor-user-roles-to-positions` (#6011) and `driver-aggregate-undeclared-key-aliases-removed` (#6321) already carry. ⚠️ It differs from those four in ONE measured way a reader should not have to infer: because nothing changed at run time, an untyped JS caller is not affected BY THE UPGRADE at all. The entry is here for a different reason — such a caller is exactly the one tsc can never reach, and the silent widening above is a defect they may ALREADY be sitting on, before and after this major. The generated upgrade guide is the only channel that reaches them, which is why the fix is written down rather than left to the compiler. ⛔ The reverse mismatch is NOT closed and no type can close it: `FilterCondition` is an open map (`[key: string]: any`) because a filter key IS a field name, so a query envelope `{ object, where }` is structurally a valid filter — one constraining columns named `object` and `where` — and so is a FilterArray. Both reach `distinct` type-checked and are refused at run time, loudly, with INVALID_FILTER / 400. `driver-memory`'s opposite half — where the BARE spelling returns the unfiltered set in silence — stays open under the #5499 freeze (#6320). ADR-0087, #6320." }, + { + "surface": "engine.find(object, { fields }) and engine.findOne(object, { fields }) carrying a dotted entry (`account.name`) — the direct engine path, not the REST ingress", + "replacement": "read the related record with `expand` (`{ expand: { account: { object: '', fields: ['name'] } } }`), keeping the reference column itself in `fields` — the relation is carried by that column and projecting it away leaves expansion nothing to resolve (#7537); or denormalise the value onto the queried object (a stored field, written when the source changes) and name that — the same remedy the REST ingress has prescribed since #7532, and the sort axis since #6924", + "migrationId": "engine-dotted-projection-refused", + "toMajor": 17, + "rationale": "#7532 (PR #7588) closed the PROJECTION axis' dotted leg at the REST ingress (`assertProjectionFieldsExist`, `400 INVALID_FIELD`), which covers everything reaching `findData`. A caller reaching `engine.find()` / `engine.findOne()` DIRECTLY passed through none of it, and that caller set was measured, not assumed (#7589): a flow `get_record` node's authored `fields: ['name', 'account.name']` parses (`GetRecordConfigSchema` restricts nothing), travels verbatim into `data.find(...)`, cleared the engine's head-only projection filter on its head segment (`account` IS a field), and reached the driver as a projection column — where SQL renders `\"account\".\"name\"` against a table that was never joined, the DB answers `no such column`, and the driver's #3821 recovery ladder retries `select('*')`. The caller asked to narrow and silently received EVERY field, byte-identical to no projection at all, pointing away from both FLS and data minimisation.\n\nRuled 2026-08-12 on #7589 (Option B): a dotted entry the engine cannot resolve is refused loudly at the engine's own head-only projection filter, covering every caller that reaches the engine. The check it replaces was justified by a comment claiming the engine resolves relationship paths \"via populate\"; #7601 measured that NO populate step exists — after PR #7617 that comment was the last place in the repo asserting dotted-path resolution does — so what was removed is not a working feature but a path to widening, kept alive by a false premise. The unknown-PLAIN-column tolerance is explicitly KEPT by the same ruling (an unknown plain name still drops silently; an all-unknown projection still falls back to `*`), a registry-less host gets no verdict (the driver-side #3821 ladder remains its documented backstop, and a driver-side carve-out is measured-need only), and a dotted `fields` inside a nested `expand` degrades to an observable warning rather than a refusal — `expandRelatedRecords`' pre-existing graceful-degradation `catch` swallows every expand failure, the same posture the sort axis (#7095) records for the same catch.\n\nThis is a CODE-path API, not stored metadata, so — like `engine-find-formula-order-by-refused` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and the ledger entry is the notification channel. No mechanical rewrite exists: the platform cannot decide between `expand` and denormalisation for the caller, and it must not resolve the path itself — no driver ever did, and inventing a join here is a feature decision, not a migration. #7589, #7532, #7601, #3821, #5918, ADR-0112." + }, { "surface": "engine.find(object, { orderBy }) and engine.findOne(object, { orderBy }) naming a `formula` field — the direct engine path, not the REST ingress", "replacement": "denormalise the value onto the object (a stored field, written when the source changes) and sort by that — the same remedy the REST ingress has prescribed since #6924 / #6994; a `summary` field is unaffected and still sorts, because it gets a real maintained column", diff --git a/packages/spec/src/migrations/entries/semantic/17.engine-dotted-projection-refused.ts b/packages/spec/src/migrations/entries/semantic/17.engine-dotted-projection-refused.ts new file mode 100644 index 0000000000..183b6789e2 --- /dev/null +++ b/packages/spec/src/migrations/entries/semantic/17.engine-dotted-projection-refused.ts @@ -0,0 +1,62 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import type { SemanticMigration } from '../../types.js'; + +export const entry: SemanticMigration = { + id: 'engine-dotted-projection-refused', + surface: + 'engine.find(object, { fields }) and engine.findOne(object, { fields }) carrying a ' + + 'dotted entry (`account.name`) — the direct engine path, not the REST ingress', + replacement: + "read the related record with `expand` (`{ expand: { account: { object: '', " + + "fields: ['name'] } } }`), keeping the reference column itself in `fields` — the " + + 'relation is carried by that column and projecting it away leaves expansion nothing ' + + 'to resolve (#7537); or denormalise the value onto the queried object (a stored ' + + 'field, written when the source changes) and name that — the same remedy the REST ' + + 'ingress has prescribed since #7532, and the sort axis since #6924', + reason: + "#7532 (PR #7588) closed the PROJECTION axis' dotted leg at the REST ingress " + + '(`assertProjectionFieldsExist`, `400 INVALID_FIELD`), which covers everything ' + + 'reaching `findData`. A caller reaching `engine.find()` / `engine.findOne()` ' + + 'DIRECTLY passed through none of it, and that caller set was measured, not assumed ' + + "(#7589): a flow `get_record` node's authored `fields: ['name', 'account.name']` " + + 'parses (`GetRecordConfigSchema` restricts nothing), travels verbatim into ' + + "`data.find(...)`, cleared the engine's head-only projection filter on its head " + + 'segment (`account` IS a field), and reached the driver as a projection column — ' + + 'where SQL renders `"account"."name"` against a table that was never joined, the ' + + "DB answers `no such column`, and the driver's #3821 recovery ladder retries " + + "`select('*')`. The caller asked to narrow and silently received EVERY field, " + + 'byte-identical to no projection at all, pointing away from both FLS and data ' + + 'minimisation.\n\n' + + 'Ruled 2026-08-12 on #7589 (Option B): a dotted entry the engine cannot resolve is ' + + "refused loudly at the engine's own head-only projection filter, covering every " + + 'caller that reaches the engine. The check it replaces was justified by a comment ' + + 'claiming the engine resolves relationship paths "via populate"; #7601 measured ' + + 'that NO populate step exists — after PR #7617 that comment was the last place in ' + + 'the repo asserting dotted-path resolution does — so what was removed is not a ' + + 'working feature but a path to widening, kept alive by a false premise. The ' + + 'unknown-PLAIN-column tolerance is explicitly KEPT by the same ruling (an unknown ' + + 'plain name still drops silently; an all-unknown projection still falls back to ' + + '`*`), a registry-less host gets no verdict (the driver-side #3821 ladder remains ' + + 'its documented backstop, and a driver-side carve-out is measured-need only), and ' + + 'a dotted `fields` inside a nested `expand` degrades to an observable warning ' + + "rather than a refusal — `expandRelatedRecords`' pre-existing graceful-degradation " + + '`catch` swallows every expand failure, the same posture the sort axis (#7095) ' + + 'records for the same catch.\n\n' + + 'This is a CODE-path API, not stored metadata, so — like ' + + '`engine-find-formula-order-by-refused` at this step — there is no `sys_metadata` ' + + 'row for the D2 chain to rewrite and the ledger entry is the notification channel. ' + + 'No mechanical rewrite exists: the platform cannot decide between `expand` and ' + + 'denormalisation for the caller, and it must not resolve the path itself — no ' + + 'driver ever did, and inventing a join here is a feature decision, not a ' + + 'migration. #7589, #7532, #7601, #3821, #5918, ADR-0112.', + acceptanceCriteria: + 'No `engine.find` / `engine.findOne` call site passes a dotted `fields` entry, no ' + + "flow `get_record` config authors one, and no saved report's " + + '`query.fields` names one — grep flow definitions and report definitions for a ' + + '`fields` entry containing a `.`, and rewrite each to `expand` (keeping the ' + + 'reference column projected) or to a denormalised stored column. Reads complete ' + + 'with no `INVALID_FIELD` whose message says "follows the relationship" or "a ' + + 'dotted path", and no "Failed to expand relationship field" warning whose error ' + + 'text does.', +}; diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index 6fa8341efb..813af2f059 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -2602,6 +2602,64 @@ const step17: MigrationStep = { + 'envelope or a FilterArray in that slot still compiles and is rejected at run time ' + 'with INVALID_FILTER / 400.', }, + { + id: 'engine-dotted-projection-refused', + surface: + 'engine.find(object, { fields }) and engine.findOne(object, { fields }) carrying a ' + + 'dotted entry (`account.name`) — the direct engine path, not the REST ingress', + replacement: + "read the related record with `expand` (`{ expand: { account: { object: '', " + + "fields: ['name'] } } }`), keeping the reference column itself in `fields` — the " + + 'relation is carried by that column and projecting it away leaves expansion nothing ' + + 'to resolve (#7537); or denormalise the value onto the queried object (a stored ' + + 'field, written when the source changes) and name that — the same remedy the REST ' + + 'ingress has prescribed since #7532, and the sort axis since #6924', + reason: + "#7532 (PR #7588) closed the PROJECTION axis' dotted leg at the REST ingress " + + '(`assertProjectionFieldsExist`, `400 INVALID_FIELD`), which covers everything ' + + 'reaching `findData`. A caller reaching `engine.find()` / `engine.findOne()` ' + + 'DIRECTLY passed through none of it, and that caller set was measured, not assumed ' + + "(#7589): a flow `get_record` node's authored `fields: ['name', 'account.name']` " + + 'parses (`GetRecordConfigSchema` restricts nothing), travels verbatim into ' + + "`data.find(...)`, cleared the engine's head-only projection filter on its head " + + 'segment (`account` IS a field), and reached the driver as a projection column — ' + + 'where SQL renders `"account"."name"` against a table that was never joined, the ' + + "DB answers `no such column`, and the driver's #3821 recovery ladder retries " + + "`select('*')`. The caller asked to narrow and silently received EVERY field, " + + 'byte-identical to no projection at all, pointing away from both FLS and data ' + + 'minimisation.\n\n' + + 'Ruled 2026-08-12 on #7589 (Option B): a dotted entry the engine cannot resolve is ' + + "refused loudly at the engine's own head-only projection filter, covering every " + + 'caller that reaches the engine. The check it replaces was justified by a comment ' + + 'claiming the engine resolves relationship paths "via populate"; #7601 measured ' + + 'that NO populate step exists — after PR #7617 that comment was the last place in ' + + 'the repo asserting dotted-path resolution does — so what was removed is not a ' + + 'working feature but a path to widening, kept alive by a false premise. The ' + + 'unknown-PLAIN-column tolerance is explicitly KEPT by the same ruling (an unknown ' + + 'plain name still drops silently; an all-unknown projection still falls back to ' + + '`*`), a registry-less host gets no verdict (the driver-side #3821 ladder remains ' + + 'its documented backstop, and a driver-side carve-out is measured-need only), and ' + + 'a dotted `fields` inside a nested `expand` degrades to an observable warning ' + + "rather than a refusal — `expandRelatedRecords`' pre-existing graceful-degradation " + + '`catch` swallows every expand failure, the same posture the sort axis (#7095) ' + + 'records for the same catch.\n\n' + + 'This is a CODE-path API, not stored metadata, so — like ' + + '`engine-find-formula-order-by-refused` at this step — there is no `sys_metadata` ' + + 'row for the D2 chain to rewrite and the ledger entry is the notification channel. ' + + 'No mechanical rewrite exists: the platform cannot decide between `expand` and ' + + 'denormalisation for the caller, and it must not resolve the path itself — no ' + + 'driver ever did, and inventing a join here is a feature decision, not a ' + + 'migration. #7589, #7532, #7601, #3821, #5918, ADR-0112.', + acceptanceCriteria: + 'No `engine.find` / `engine.findOne` call site passes a dotted `fields` entry, no ' + + "flow `get_record` config authors one, and no saved report's " + + '`query.fields` names one — grep flow definitions and report definitions for a ' + + '`fields` entry containing a `.`, and rewrite each to `expand` (keeping the ' + + 'reference column projected) or to a denormalised stored column. Reads complete ' + + 'with no `INVALID_FIELD` whose message says "follows the relationship" or "a ' + + 'dotted path", and no "Failed to expand relationship field" warning whose error ' + + 'text does.', + }, { id: 'engine-find-formula-order-by-refused', surface: From 5065b357ee12c31cd14fcaf6af098a20782ff4fd Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 05:42:02 +0000 Subject: [PATCH 4/4] docs(objectql): teach expand, not a dotted projection, for reading a lookup's related column (#7589) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit types.mdx's lookup query example was the populate premise itself — the spelling both doors now refuse. Replaced with the expand form, reference column kept projected (#7537), refusal + remedy stated inline. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01RDTnVvsgA6cUZ4xFVtPZRy --- content/docs/protocol/objectql/types.mdx | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/content/docs/protocol/objectql/types.mdx b/content/docs/protocol/objectql/types.mdx index 0c322a0cfa..5af6347961 100644 --- a/content/docs/protocol/objectql/types.mdx +++ b/content/docs/protocol/objectql/types.mdx @@ -605,11 +605,18 @@ account_id: **Storage:** Stores `id` of referenced record -**Query behavior:** +**Query behavior:** `expand` is the door for related data. A dotted `fields` +entry (`'account_id.company_name'`) is **refused** (`400 INVALID_FIELD`) — no +driver resolves one, at the REST ingress since #7532 and on direct +`engine.find` / `engine.findOne` calls since #7589. Keep the reference column +itself in the projection: the relation is carried by `account_id`, and +projecting it away leaves the expansion nothing to resolve. + ```typescript -// Expand the account lookup +// Read a column of the related account const opportunities = await engine.find('opportunity', { - fields: ['name', 'account.company_name'] // Expands account + fields: ['name', 'account_id'], + expand: { account_id: { object: 'account', fields: ['company_name'] } }, }); ```