diff --git a/.changeset/formula-filter-refusal-adr-0087-entry.md b/.changeset/formula-filter-refusal-adr-0087-entry.md new file mode 100644 index 0000000000..8d3d34f9a5 --- /dev/null +++ b/.changeset/formula-filter-refusal-adr-0087-entry.md @@ -0,0 +1,32 @@ +--- +"@objectstack/spec": patch +--- + + + +docs(spec): register the FILTER-axis formula refusal in the ADR-0087 ledger (#8370) + +The refusal itself shipped in 17.0.0 (#8296 / PR #8369): a `where` naming a +`formula` field is `400 INVALID_FIELD` at both doors — the REST ingress +(`assertFilterFieldsExist`) and the engine's own filter seam +(`assertFilterIsMaterializable`), which saved reports, flows and dashboard +widgets reach directly. It shipped with **no** ADR-0087 semantic entry, so +`objectstack migrate meta`, `spec-changes.json` and the generated upgrade guide +said nothing about it. + +Its SORT-axis twin (#7095, `engine-find-formula-order-by-refused`) carries one, +for the identical shape. This adds the FILTER-axis sibling — +`engine-find-formula-filter-refused` under protocol 17 — and regenerates the two +projections of the registry. + +For a code-path API there is no `sys_metadata` row for the D2 chain to rewrite +and no mechanical rewrite in either direction (the platform cannot invent the +stored column, and it must not filter post-hoc — `driver.find` has already +applied `limit` / `offset`, so a post-hoc predicate would filter an arbitrary +PAGE), which makes the ledger entry the only notification channel this class +has. The remedy it prescribes is the one the sort and search axes already +prescribe, in the same words: denormalise the value onto a stored field written +when the source changes, and filter that. `summary` and `autonumber` fields need +no action — both get real maintained columns and filter correctly. + +No behaviour changes: registration and regenerated artifacts only. diff --git a/docs/protocol-upgrade-guide.md b/docs/protocol-upgrade-guide.md index 61b6d8eda8..5f6f550386 100644 --- a/docs/protocol-upgrade-guide.md +++ b/docs/protocol-upgrade-guide.md @@ -410,6 +410,19 @@ Ruled 2026-08-12 on #7589 (Option B): a dotted entry the engine cannot resolve i This is a CODE-path API, not stored metadata, so — like `engine-find-formula-order-by-refused` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and the ledger entry is the notification channel. No mechanical rewrite exists: the platform cannot decide between `expand` and denormalisation for the caller, and it must not resolve the path itself — no driver ever did, and inventing a join here is a feature decision, not a migration. #7589, #7532, #7601, #3821, #5918, ADR-0112. - Done when: No `engine.find` / `engine.findOne` call site passes a dotted `fields` entry, no flow `get_record` config authors one, and no saved report's `query.fields` names one — grep flow definitions and report definitions for a `fields` entry containing a `.`, and rewrite each to `expand` (keeping the reference column projected) or to a denormalised stored column. Reads complete with no `INVALID_FIELD` whose message says "follows the relationship" or "a dotted path", and no "Failed to expand relationship field" warning whose error text does. +- **`engine-find-formula-filter-refused`** — `a `where` / filter naming a `formula` field — at BOTH doors: the REST ingress (`assertFilterFieldsExist`, covering everything that reaches `findData`) and the engine seam itself (`engine.find` / `findOne` / `count` / `aggregate` / `update` / `delete`), which saved reports, flows and dashboard widgets reach directly` → denormalise the value onto the object (a stored field, written when the source changes) and filter that — deliberately the same remedy, in the same words, the SORT axis prescribes (#6924 / #6994 / #7095) and the SEARCH axis has prescribed since #6674; `summary` and `autonumber` fields need NO action, because both get real maintained columns and filter correctly + - Why not automatic: `formula` is the one field type no driver materialises a column for, and FILTER was the last of the three query axes still fail-open on it: SORT refuses it (#6994 at the ingress, #7095 at the engine) and SEARCH refuses it by name (#6674), while a `where` on a `formula` field cleared every gate precisely BECAUSE the object declares the field, reached a driver with no column behind it, and answered 200 with zero rows. Measured on a real `ObjectQL` with `is_open` a `formula` over the stored `status` column: `where {is_open: true}` and `where {is_open: false}` each returned 0 rows with NO error, while the controls `where {status: 'open'}` returned 4 rows and `where {subtask_total: 5}` (a `summary`, which HAS a column) returned 1 row. + +BOTH directions are wrong and the `false` one is the dangerous one: the same predicate against a STORED boolean returns every matching row, so a filter meaning "not yet done" silently became "no records at all" — a row SET changed under a 200, which no amount of inspecting the response can reveal, and the formula READS correctly in that very same response, so the field is visibly populated and simultaneously unfilterable. That is strictly worse than the sort axis it mirrors: a refused sort returns the same rows in a different order, a refused filter changes which rows exist. + +Both doors now refuse it with `400 INVALID_FIELD` (#8296 / PR #8369), naming the offending key path and carrying the remedy sentence — the ingress gate (`assertFilterFieldsExist`, `@objectstack/metadata-protocol`) for everything reaching `findData`, and `assertFilterIsMaterializable` (`@objectstack/objectql`, `filter-comparand-shape.ts`) at the engine's own filter seam, which every caller-supplied `where` passes through whichever verb it arrived by. Both judge the field by the SAME `@objectstack/spec/data` predicate the SEARCH axis uses (`isVirtualSearchField` / `SEARCH_VIRTUAL_TYPES`, which holds `formula` and nothing else), so gate and drivers cannot disagree about which types have a column: a gate widened to the spec's `COMPUTED_VALUE_TYPES` (the WRITE contract) would refuse two working types. DOTTED filter paths are deliberately not judged on this axis at either door. + +This is a CODE-path API, not stored metadata, so — like `engine-find-formula-order-by-refused` and `engine-dotted-projection-refused` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and this ledger entry is the notification channel. No mechanical rewrite exists in either direction: the platform cannot invent the stored column the remedy prescribes, and it must not filter post-hoc instead — `driver.find` has already applied `limit` / `offset`, so a predicate applied after the formulas are evaluated would filter an ARBITRARY PAGE, which looks correct on small result sets and is wrong the moment pagination is involved. + +AUTHOR-REACHABLE SURFACES are why this is not merely a code-side note. A saved report's `query.filter` (`sys_saved_report`) is forwarded VERBATIM into `engine.find` by `plugin-reports` (`report-service.ts`, `where: q.filter`), bypassing the ingress gate entirely; flow node `config.filter` and dashboard widget filters are author-written the same way. A report or flow authored to filter on a formula field used to run and quietly return the wrong row set; it now fails loudly, with the remedy in the message. + +Registered on the inherited ruling of #7095 ("register it anyway"), re-affirmed for this axis at triage on 2026-08-13 (#8370): the shape is identical to the sort axis and the consequence here is larger. #8296, #8370, #7095, #6994, #6924, #6674, ADR-0112. + - Done when: No filter names a `formula` field on any surface — grep your saved report definitions (`sys_saved_report.query.filter`), flow node `config.filter`, dashboard widget filters and view filters for a filtered field whose object declares it as a `formula`, and denormalise each onto a stored column written when the source changes. A `summary` / `autonumber` field needs no action: both have real maintained columns and filter correctly. Reads complete with no `INVALID_FIELD` naming a virtual `formula` field in a filter, at either door. - **`engine-find-formula-order-by-refused`** — `engine.find(object, { orderBy }) and engine.findOne(object, { orderBy }) naming a `formula` field — the direct engine path, not the REST ingress` → denormalise the value onto the object (a stored field, written when the source changes) and sort by that — the same remedy the REST ingress has prescribed since #6924 / #6994; a `summary` field is unaffected and still sorts, because it gets a real maintained column - Why not automatic: #4226 / #4256 / #6994 closed the SORT axis at the REST ingress (`assertSortFieldsExist`, `400 INVALID_SORT`), which covers everything reaching `findData`: the list route, `POST /data/:object/query`, the export route and the RPC dispatcher. A caller reaching `engine.find()` / `engine.findOne()` DIRECTLY passed through none of it, and a `formula` ORDER BY there was dropped in silence. Measured on a real driver: `asc` and `desc` came back BYTE-IDENTICAL, in insertion order, under a success, with the rows carrying the very values they were asked to be ordered by. No column exists to order by (a formula is computed on read, so no driver materialises one), so the ORDER BY reached the driver, found nothing, and the unknown-column backstop returned the rows unordered. diff --git a/packages/spec/spec-changes.json b/packages/spec/spec-changes.json index 23e1993aa4..994fbf3105 100644 --- a/packages/spec/spec-changes.json +++ b/packages/spec/spec-changes.json @@ -705,6 +705,13 @@ "toMajor": 17, "rationale": "#7532 (PR #7588) closed the PROJECTION axis' dotted leg at the REST ingress (`assertProjectionFieldsExist`, `400 INVALID_FIELD`), which covers everything reaching `findData`. A caller reaching `engine.find()` / `engine.findOne()` DIRECTLY passed through none of it, and that caller set was measured, not assumed (#7589): a flow `get_record` node's authored `fields: ['name', 'account.name']` parses (`GetRecordConfigSchema` restricts nothing), travels verbatim into `data.find(...)`, cleared the engine's head-only projection filter on its head segment (`account` IS a field), and reached the driver as a projection column — where SQL renders `\"account\".\"name\"` against a table that was never joined, the DB answers `no such column`, and the driver's #3821 recovery ladder retries `select('*')`. The caller asked to narrow and silently received EVERY field, byte-identical to no projection at all, pointing away from both FLS and data minimisation.\n\nRuled 2026-08-12 on #7589 (Option B): a dotted entry the engine cannot resolve is refused loudly at the engine's own head-only projection filter, covering every caller that reaches the engine. The check it replaces was justified by a comment claiming the engine resolves relationship paths \"via populate\"; #7601 measured that NO populate step exists — after PR #7617 that comment was the last place in the repo asserting dotted-path resolution does — so what was removed is not a working feature but a path to widening, kept alive by a false premise. The unknown-PLAIN-column tolerance is explicitly KEPT by the same ruling (an unknown plain name still drops silently; an all-unknown projection still falls back to `*`), a registry-less host gets no verdict (the driver-side #3821 ladder remains its documented backstop, and a driver-side carve-out is measured-need only), and a dotted `fields` inside a nested `expand` degrades to an observable warning rather than a refusal — `expandRelatedRecords`' pre-existing graceful-degradation `catch` swallows every expand failure, the same posture the sort axis (#7095) records for the same catch.\n\nThis is a CODE-path API, not stored metadata, so — like `engine-find-formula-order-by-refused` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and the ledger entry is the notification channel. No mechanical rewrite exists: the platform cannot decide between `expand` and denormalisation for the caller, and it must not resolve the path itself — no driver ever did, and inventing a join here is a feature decision, not a migration. #7589, #7532, #7601, #3821, #5918, ADR-0112." }, + { + "surface": "a `where` / filter naming a `formula` field — at BOTH doors: the REST ingress (`assertFilterFieldsExist`, covering everything that reaches `findData`) and the engine seam itself (`engine.find` / `findOne` / `count` / `aggregate` / `update` / `delete`), which saved reports, flows and dashboard widgets reach directly", + "replacement": "denormalise the value onto the object (a stored field, written when the source changes) and filter that — deliberately the same remedy, in the same words, the SORT axis prescribes (#6924 / #6994 / #7095) and the SEARCH axis has prescribed since #6674; `summary` and `autonumber` fields need NO action, because both get real maintained columns and filter correctly", + "migrationId": "engine-find-formula-filter-refused", + "toMajor": 17, + "rationale": "`formula` is the one field type no driver materialises a column for, and FILTER was the last of the three query axes still fail-open on it: SORT refuses it (#6994 at the ingress, #7095 at the engine) and SEARCH refuses it by name (#6674), while a `where` on a `formula` field cleared every gate precisely BECAUSE the object declares the field, reached a driver with no column behind it, and answered 200 with zero rows. Measured on a real `ObjectQL` with `is_open` a `formula` over the stored `status` column: `where {is_open: true}` and `where {is_open: false}` each returned 0 rows with NO error, while the controls `where {status: 'open'}` returned 4 rows and `where {subtask_total: 5}` (a `summary`, which HAS a column) returned 1 row.\n\nBOTH directions are wrong and the `false` one is the dangerous one: the same predicate against a STORED boolean returns every matching row, so a filter meaning \"not yet done\" silently became \"no records at all\" — a row SET changed under a 200, which no amount of inspecting the response can reveal, and the formula READS correctly in that very same response, so the field is visibly populated and simultaneously unfilterable. That is strictly worse than the sort axis it mirrors: a refused sort returns the same rows in a different order, a refused filter changes which rows exist.\n\nBoth doors now refuse it with `400 INVALID_FIELD` (#8296 / PR #8369), naming the offending key path and carrying the remedy sentence — the ingress gate (`assertFilterFieldsExist`, `@objectstack/metadata-protocol`) for everything reaching `findData`, and `assertFilterIsMaterializable` (`@objectstack/objectql`, `filter-comparand-shape.ts`) at the engine's own filter seam, which every caller-supplied `where` passes through whichever verb it arrived by. Both judge the field by the SAME `@objectstack/spec/data` predicate the SEARCH axis uses (`isVirtualSearchField` / `SEARCH_VIRTUAL_TYPES`, which holds `formula` and nothing else), so gate and drivers cannot disagree about which types have a column: a gate widened to the spec's `COMPUTED_VALUE_TYPES` (the WRITE contract) would refuse two working types. DOTTED filter paths are deliberately not judged on this axis at either door.\n\nThis is a CODE-path API, not stored metadata, so — like `engine-find-formula-order-by-refused` and `engine-dotted-projection-refused` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and this ledger entry is the notification channel. No mechanical rewrite exists in either direction: the platform cannot invent the stored column the remedy prescribes, and it must not filter post-hoc instead — `driver.find` has already applied `limit` / `offset`, so a predicate applied after the formulas are evaluated would filter an ARBITRARY PAGE, which looks correct on small result sets and is wrong the moment pagination is involved.\n\nAUTHOR-REACHABLE SURFACES are why this is not merely a code-side note. A saved report's `query.filter` (`sys_saved_report`) is forwarded VERBATIM into `engine.find` by `plugin-reports` (`report-service.ts`, `where: q.filter`), bypassing the ingress gate entirely; flow node `config.filter` and dashboard widget filters are author-written the same way. A report or flow authored to filter on a formula field used to run and quietly return the wrong row set; it now fails loudly, with the remedy in the message.\n\nRegistered on the inherited ruling of #7095 (\"register it anyway\"), re-affirmed for this axis at triage on 2026-08-13 (#8370): the shape is identical to the sort axis and the consequence here is larger. #8296, #8370, #7095, #6994, #6924, #6674, ADR-0112." + }, { "surface": "engine.find(object, { orderBy }) and engine.findOne(object, { orderBy }) naming a `formula` field — the direct engine path, not the REST ingress", "replacement": "denormalise the value onto the object (a stored field, written when the source changes) and sort by that — the same remedy the REST ingress has prescribed since #6924 / #6994; a `summary` field is unaffected and still sorts, because it gets a real maintained column", @@ -1755,6 +1762,13 @@ "toMajor": 17, "rationale": "#7532 (PR #7588) closed the PROJECTION axis' dotted leg at the REST ingress (`assertProjectionFieldsExist`, `400 INVALID_FIELD`), which covers everything reaching `findData`. A caller reaching `engine.find()` / `engine.findOne()` DIRECTLY passed through none of it, and that caller set was measured, not assumed (#7589): a flow `get_record` node's authored `fields: ['name', 'account.name']` parses (`GetRecordConfigSchema` restricts nothing), travels verbatim into `data.find(...)`, cleared the engine's head-only projection filter on its head segment (`account` IS a field), and reached the driver as a projection column — where SQL renders `\"account\".\"name\"` against a table that was never joined, the DB answers `no such column`, and the driver's #3821 recovery ladder retries `select('*')`. The caller asked to narrow and silently received EVERY field, byte-identical to no projection at all, pointing away from both FLS and data minimisation.\n\nRuled 2026-08-12 on #7589 (Option B): a dotted entry the engine cannot resolve is refused loudly at the engine's own head-only projection filter, covering every caller that reaches the engine. The check it replaces was justified by a comment claiming the engine resolves relationship paths \"via populate\"; #7601 measured that NO populate step exists — after PR #7617 that comment was the last place in the repo asserting dotted-path resolution does — so what was removed is not a working feature but a path to widening, kept alive by a false premise. The unknown-PLAIN-column tolerance is explicitly KEPT by the same ruling (an unknown plain name still drops silently; an all-unknown projection still falls back to `*`), a registry-less host gets no verdict (the driver-side #3821 ladder remains its documented backstop, and a driver-side carve-out is measured-need only), and a dotted `fields` inside a nested `expand` degrades to an observable warning rather than a refusal — `expandRelatedRecords`' pre-existing graceful-degradation `catch` swallows every expand failure, the same posture the sort axis (#7095) records for the same catch.\n\nThis is a CODE-path API, not stored metadata, so — like `engine-find-formula-order-by-refused` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and the ledger entry is the notification channel. No mechanical rewrite exists: the platform cannot decide between `expand` and denormalisation for the caller, and it must not resolve the path itself — no driver ever did, and inventing a join here is a feature decision, not a migration. #7589, #7532, #7601, #3821, #5918, ADR-0112." }, + { + "surface": "a `where` / filter naming a `formula` field — at BOTH doors: the REST ingress (`assertFilterFieldsExist`, covering everything that reaches `findData`) and the engine seam itself (`engine.find` / `findOne` / `count` / `aggregate` / `update` / `delete`), which saved reports, flows and dashboard widgets reach directly", + "replacement": "denormalise the value onto the object (a stored field, written when the source changes) and filter that — deliberately the same remedy, in the same words, the SORT axis prescribes (#6924 / #6994 / #7095) and the SEARCH axis has prescribed since #6674; `summary` and `autonumber` fields need NO action, because both get real maintained columns and filter correctly", + "migrationId": "engine-find-formula-filter-refused", + "toMajor": 17, + "rationale": "`formula` is the one field type no driver materialises a column for, and FILTER was the last of the three query axes still fail-open on it: SORT refuses it (#6994 at the ingress, #7095 at the engine) and SEARCH refuses it by name (#6674), while a `where` on a `formula` field cleared every gate precisely BECAUSE the object declares the field, reached a driver with no column behind it, and answered 200 with zero rows. Measured on a real `ObjectQL` with `is_open` a `formula` over the stored `status` column: `where {is_open: true}` and `where {is_open: false}` each returned 0 rows with NO error, while the controls `where {status: 'open'}` returned 4 rows and `where {subtask_total: 5}` (a `summary`, which HAS a column) returned 1 row.\n\nBOTH directions are wrong and the `false` one is the dangerous one: the same predicate against a STORED boolean returns every matching row, so a filter meaning \"not yet done\" silently became \"no records at all\" — a row SET changed under a 200, which no amount of inspecting the response can reveal, and the formula READS correctly in that very same response, so the field is visibly populated and simultaneously unfilterable. That is strictly worse than the sort axis it mirrors: a refused sort returns the same rows in a different order, a refused filter changes which rows exist.\n\nBoth doors now refuse it with `400 INVALID_FIELD` (#8296 / PR #8369), naming the offending key path and carrying the remedy sentence — the ingress gate (`assertFilterFieldsExist`, `@objectstack/metadata-protocol`) for everything reaching `findData`, and `assertFilterIsMaterializable` (`@objectstack/objectql`, `filter-comparand-shape.ts`) at the engine's own filter seam, which every caller-supplied `where` passes through whichever verb it arrived by. Both judge the field by the SAME `@objectstack/spec/data` predicate the SEARCH axis uses (`isVirtualSearchField` / `SEARCH_VIRTUAL_TYPES`, which holds `formula` and nothing else), so gate and drivers cannot disagree about which types have a column: a gate widened to the spec's `COMPUTED_VALUE_TYPES` (the WRITE contract) would refuse two working types. DOTTED filter paths are deliberately not judged on this axis at either door.\n\nThis is a CODE-path API, not stored metadata, so — like `engine-find-formula-order-by-refused` and `engine-dotted-projection-refused` at this step — there is no `sys_metadata` row for the D2 chain to rewrite and this ledger entry is the notification channel. No mechanical rewrite exists in either direction: the platform cannot invent the stored column the remedy prescribes, and it must not filter post-hoc instead — `driver.find` has already applied `limit` / `offset`, so a predicate applied after the formulas are evaluated would filter an ARBITRARY PAGE, which looks correct on small result sets and is wrong the moment pagination is involved.\n\nAUTHOR-REACHABLE SURFACES are why this is not merely a code-side note. A saved report's `query.filter` (`sys_saved_report`) is forwarded VERBATIM into `engine.find` by `plugin-reports` (`report-service.ts`, `where: q.filter`), bypassing the ingress gate entirely; flow node `config.filter` and dashboard widget filters are author-written the same way. A report or flow authored to filter on a formula field used to run and quietly return the wrong row set; it now fails loudly, with the remedy in the message.\n\nRegistered on the inherited ruling of #7095 (\"register it anyway\"), re-affirmed for this axis at triage on 2026-08-13 (#8370): the shape is identical to the sort axis and the consequence here is larger. #8296, #8370, #7095, #6994, #6924, #6674, ADR-0112." + }, { "surface": "engine.find(object, { orderBy }) and engine.findOne(object, { orderBy }) naming a `formula` field — the direct engine path, not the REST ingress", "replacement": "denormalise the value onto the object (a stored field, written when the source changes) and sort by that — the same remedy the REST ingress has prescribed since #6924 / #6994; a `summary` field is unaffected and still sorts, because it gets a real maintained column", diff --git a/packages/spec/src/migrations/entries/semantic/17.engine-find-formula-filter-refused.ts b/packages/spec/src/migrations/entries/semantic/17.engine-find-formula-filter-refused.ts new file mode 100644 index 0000000000..dbbe33bc4e --- /dev/null +++ b/packages/spec/src/migrations/entries/semantic/17.engine-find-formula-filter-refused.ts @@ -0,0 +1,77 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import type { SemanticMigration } from '../../types.js'; + +export const entry: SemanticMigration = { + id: 'engine-find-formula-filter-refused', + surface: + 'a `where` / filter naming a `formula` field — at BOTH doors: the REST ingress ' + + '(`assertFilterFieldsExist`, covering everything that reaches `findData`) and the ' + + 'engine seam itself (`engine.find` / `findOne` / `count` / `aggregate` / `update` / ' + + '`delete`), which saved reports, flows and dashboard widgets reach directly', + replacement: + 'denormalise the value onto the object (a stored field, written when the source ' + + 'changes) and filter that — deliberately the same remedy, in the same words, the ' + + 'SORT axis prescribes (#6924 / #6994 / #7095) and the SEARCH axis has prescribed ' + + 'since #6674; `summary` and `autonumber` fields need NO action, because both get ' + + 'real maintained columns and filter correctly', + reason: + '`formula` is the one field type no driver materialises a column for, and FILTER was ' + + 'the last of the three query axes still fail-open on it: SORT refuses it (#6994 at ' + + 'the ingress, #7095 at the engine) and SEARCH refuses it by name (#6674), while a ' + + '`where` on a `formula` field cleared every gate precisely BECAUSE the object ' + + 'declares the field, reached a driver with no column behind it, and answered 200 ' + + 'with zero rows. Measured on a real `ObjectQL` with `is_open` a `formula` over the ' + + "stored `status` column: `where {is_open: true}` and `where {is_open: false}` each " + + "returned 0 rows with NO error, while the controls `where {status: 'open'}` returned " + + '4 rows and `where {subtask_total: 5}` (a `summary`, which HAS a column) returned 1 ' + + 'row.\n\n' + + 'BOTH directions are wrong and the `false` one is the dangerous one: the same ' + + 'predicate against a STORED boolean returns every matching row, so a filter meaning ' + + '"not yet done" silently became "no records at all" — a row SET changed under a 200, ' + + 'which no amount of inspecting the response can reveal, and the formula READS ' + + 'correctly in that very same response, so the field is visibly populated and ' + + 'simultaneously unfilterable. That is strictly worse than the sort axis it mirrors: ' + + 'a refused sort returns the same rows in a different order, a refused filter changes ' + + 'which rows exist.\n\n' + + 'Both doors now refuse it with `400 INVALID_FIELD` (#8296 / PR #8369), naming the ' + + 'offending key path and carrying the remedy sentence — the ingress gate ' + + '(`assertFilterFieldsExist`, `@objectstack/metadata-protocol`) for everything ' + + 'reaching `findData`, and `assertFilterIsMaterializable` ' + + "(`@objectstack/objectql`, `filter-comparand-shape.ts`) at the engine's own filter " + + 'seam, which every caller-supplied `where` passes through whichever verb it arrived ' + + 'by. Both judge the field by the SAME `@objectstack/spec/data` predicate the SEARCH ' + + 'axis uses (`isVirtualSearchField` / `SEARCH_VIRTUAL_TYPES`, which holds `formula` ' + + 'and nothing else), so gate and drivers cannot disagree about which types have a ' + + 'column: a gate widened to the spec\'s `COMPUTED_VALUE_TYPES` (the WRITE contract) ' + + 'would refuse two working types. DOTTED filter paths are deliberately not judged on ' + + 'this axis at either door.\n\n' + + 'This is a CODE-path API, not stored metadata, so — like ' + + '`engine-find-formula-order-by-refused` and `engine-dotted-projection-refused` at ' + + 'this step — there is no `sys_metadata` row for the D2 chain to rewrite and this ' + + 'ledger entry is the notification channel. No mechanical rewrite exists in either ' + + 'direction: the platform cannot invent the stored column the remedy prescribes, and ' + + 'it must not filter post-hoc instead — `driver.find` has already applied `limit` / ' + + '`offset`, so a predicate applied after the formulas are evaluated would filter an ' + + 'ARBITRARY PAGE, which looks correct on small result sets and is wrong the moment ' + + 'pagination is involved.\n\n' + + 'AUTHOR-REACHABLE SURFACES are why this is not merely a code-side note. A saved ' + + "report's `query.filter` (`sys_saved_report`) is forwarded VERBATIM into " + + '`engine.find` by `plugin-reports` (`report-service.ts`, `where: q.filter`), ' + + 'bypassing the ingress gate entirely; flow node `config.filter` and dashboard widget ' + + 'filters are author-written the same way. A report or flow authored to filter on a ' + + 'formula field used to run and quietly return the wrong row set; it now fails ' + + 'loudly, with the remedy in the message.\n\n' + + 'Registered on the inherited ruling of #7095 ("register it anyway"), re-affirmed for ' + + 'this axis at triage on 2026-08-13 (#8370): the shape is identical to the sort axis ' + + 'and the consequence here is larger. #8296, #8370, #7095, #6994, #6924, #6674, ' + + 'ADR-0112.', + acceptanceCriteria: + 'No filter names a `formula` field on any surface — grep your saved report definitions ' + + "(`sys_saved_report.query.filter`), flow node `config.filter`, dashboard widget " + + 'filters and view filters for a filtered field whose object declares it as a ' + + '`formula`, and denormalise each onto a stored column written when the source ' + + 'changes. A `summary` / `autonumber` field needs no action: both have real ' + + 'maintained columns and filter correctly. Reads complete with no `INVALID_FIELD` ' + + 'naming a virtual `formula` field in a filter, at either door.', +}; diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index 2b846b422f..4cf108d0f8 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -2816,6 +2816,79 @@ const step17: MigrationStep = { + 'dotted path", and no "Failed to expand relationship field" warning whose error ' + 'text does.', }, + { + id: 'engine-find-formula-filter-refused', + surface: + 'a `where` / filter naming a `formula` field — at BOTH doors: the REST ingress ' + + '(`assertFilterFieldsExist`, covering everything that reaches `findData`) and the ' + + 'engine seam itself (`engine.find` / `findOne` / `count` / `aggregate` / `update` / ' + + '`delete`), which saved reports, flows and dashboard widgets reach directly', + replacement: + 'denormalise the value onto the object (a stored field, written when the source ' + + 'changes) and filter that — deliberately the same remedy, in the same words, the ' + + 'SORT axis prescribes (#6924 / #6994 / #7095) and the SEARCH axis has prescribed ' + + 'since #6674; `summary` and `autonumber` fields need NO action, because both get ' + + 'real maintained columns and filter correctly', + reason: + '`formula` is the one field type no driver materialises a column for, and FILTER was ' + + 'the last of the three query axes still fail-open on it: SORT refuses it (#6994 at ' + + 'the ingress, #7095 at the engine) and SEARCH refuses it by name (#6674), while a ' + + '`where` on a `formula` field cleared every gate precisely BECAUSE the object ' + + 'declares the field, reached a driver with no column behind it, and answered 200 ' + + 'with zero rows. Measured on a real `ObjectQL` with `is_open` a `formula` over the ' + + "stored `status` column: `where {is_open: true}` and `where {is_open: false}` each " + + "returned 0 rows with NO error, while the controls `where {status: 'open'}` returned " + + '4 rows and `where {subtask_total: 5}` (a `summary`, which HAS a column) returned 1 ' + + 'row.\n\n' + + 'BOTH directions are wrong and the `false` one is the dangerous one: the same ' + + 'predicate against a STORED boolean returns every matching row, so a filter meaning ' + + '"not yet done" silently became "no records at all" — a row SET changed under a 200, ' + + 'which no amount of inspecting the response can reveal, and the formula READS ' + + 'correctly in that very same response, so the field is visibly populated and ' + + 'simultaneously unfilterable. That is strictly worse than the sort axis it mirrors: ' + + 'a refused sort returns the same rows in a different order, a refused filter changes ' + + 'which rows exist.\n\n' + + 'Both doors now refuse it with `400 INVALID_FIELD` (#8296 / PR #8369), naming the ' + + 'offending key path and carrying the remedy sentence — the ingress gate ' + + '(`assertFilterFieldsExist`, `@objectstack/metadata-protocol`) for everything ' + + 'reaching `findData`, and `assertFilterIsMaterializable` ' + + "(`@objectstack/objectql`, `filter-comparand-shape.ts`) at the engine's own filter " + + 'seam, which every caller-supplied `where` passes through whichever verb it arrived ' + + 'by. Both judge the field by the SAME `@objectstack/spec/data` predicate the SEARCH ' + + 'axis uses (`isVirtualSearchField` / `SEARCH_VIRTUAL_TYPES`, which holds `formula` ' + + 'and nothing else), so gate and drivers cannot disagree about which types have a ' + + 'column: a gate widened to the spec\'s `COMPUTED_VALUE_TYPES` (the WRITE contract) ' + + 'would refuse two working types. DOTTED filter paths are deliberately not judged on ' + + 'this axis at either door.\n\n' + + 'This is a CODE-path API, not stored metadata, so — like ' + + '`engine-find-formula-order-by-refused` and `engine-dotted-projection-refused` at ' + + 'this step — there is no `sys_metadata` row for the D2 chain to rewrite and this ' + + 'ledger entry is the notification channel. No mechanical rewrite exists in either ' + + 'direction: the platform cannot invent the stored column the remedy prescribes, and ' + + 'it must not filter post-hoc instead — `driver.find` has already applied `limit` / ' + + '`offset`, so a predicate applied after the formulas are evaluated would filter an ' + + 'ARBITRARY PAGE, which looks correct on small result sets and is wrong the moment ' + + 'pagination is involved.\n\n' + + 'AUTHOR-REACHABLE SURFACES are why this is not merely a code-side note. A saved ' + + "report's `query.filter` (`sys_saved_report`) is forwarded VERBATIM into " + + '`engine.find` by `plugin-reports` (`report-service.ts`, `where: q.filter`), ' + + 'bypassing the ingress gate entirely; flow node `config.filter` and dashboard widget ' + + 'filters are author-written the same way. A report or flow authored to filter on a ' + + 'formula field used to run and quietly return the wrong row set; it now fails ' + + 'loudly, with the remedy in the message.\n\n' + + 'Registered on the inherited ruling of #7095 ("register it anyway"), re-affirmed for ' + + 'this axis at triage on 2026-08-13 (#8370): the shape is identical to the sort axis ' + + 'and the consequence here is larger. #8296, #8370, #7095, #6994, #6924, #6674, ' + + 'ADR-0112.', + acceptanceCriteria: + 'No filter names a `formula` field on any surface — grep your saved report definitions ' + + "(`sys_saved_report.query.filter`), flow node `config.filter`, dashboard widget " + + 'filters and view filters for a filtered field whose object declares it as a ' + + '`formula`, and denormalise each onto a stored column written when the source ' + + 'changes. A `summary` / `autonumber` field needs no action: both have real ' + + 'maintained columns and filter correctly. Reads complete with no `INVALID_FIELD` ' + + 'naming a virtual `formula` field in a filter, at either door.', + }, { id: 'engine-find-formula-order-by-refused', surface: