Skip to content

The FILTER axis has no unmaterializable verdict: a where on a virtual formula field returns 0 rows silently, while sort and search refuse the same field with a 400 #8296

Description

@os-zhuang

Found while implementing #7226 (PR #8295), where this gap is what made that card's defect invisible. Filed unassigned; the fix is a platform change in packages/metadata-protocol, out of scope for an examples-only PR.

The asymmetry

formula is the one field type no driver materialises a column for. Three query axes can name a field; two of them refuse a formula loudly, one accepts it and answers nothing:

axisgateverdict for a formula field
SORTassertSortFieldsExist + assertOrderByIsMaterializable400 INVALID_SORT, with a denormalize-onto-a-stored-field remedy (#6994, #7095)
SEARCHassertSearchFieldsAreSearchablerefused by name, with a mirror-onto-a-stored-text-field remedy (#6674)
FILTERassertFilterFieldsExistaccepted — 200, 0 rows, no error

assertFilterFieldsExist (packages/metadata-protocol/src/protocol.ts) computes exactly one verdict:

const unknown = names.filter((f) => !gate.known.has(f.split('.')[0]));
if (unknown.length === 0) return;

A formula field IS known, so it passes the door and reaches the driver, which has no column for it.

Measured

Real ObjectQL over @objectstack/driver-sqlite-wasm, one object, two rows — is_completed a formula whose source is record.status == "completed", status a stored text column:

where { is_completed: true } -> 0 rows, no error
where { is_completed: false } -> 0 rows, no error
CONTROL where { status: 'completed' } -> 1 row
CONTROL where { status: { $ne: 'completed' } } -> 1 row

The formula READS correctly on the same rows in the same call (applyFormulaPlan hydrates it), which is what makes this hard to spot: the field is visibly populated in the response and simultaneously unfilterable.

packages/spec/src/data/search-fields.ts already records the same measurement for the $contains case ("0 rows on driver-memory ... 0 rows with NO error on driver-sql/better-sqlite3"), so the storage fact is established repo-side; only the filter axis never grew a door for it.

Why it matters

Both directions are wrong, and the false direction is the dangerous one:

  • { formula_field: true } returns nothing — a surface that looks merely empty.
  • { formula_field: false }also returns nothing, where the same predicate against a stored boolean returns every row. A filter meaning "not yet done" silently becomes "no records at all".

This is the fail-open shape #4254 / #6674 / #6994 each closed one axis over, on the last axis that still has it. It is also a live authoring trap: #7226's app-todo had eight filters on that exact predicate, and the natural repair for that card — deriving the flags as formula fields — would have silently emptied a list view, a scheduled reminder flow and two reports, with no error anywhere. The fix there had to route around the platform rather than use it.

Shape of the fix, if wanted

Give assertFilterFieldsExist the third verdict its two neighbours already have, splitting unknown from virtual, judged by the same @objectstack/spec/data predicate the search axis uses (isVirtualSearchField / SEARCH_VIRTUAL_TYPES) so gate and drivers cannot disagree about which types have a column. The remedy sentence should match the wording the sort and search refusals already share ("denormalise onto a stored field ... and filter that"), per the one-vocabulary-across-doors discipline recorded on assertSortFieldsExist.

Two things worth deciding rather than assuming:

  1. Ingress-only, or the engine door too?engine.find() still drops a formula ORDER BY silently — decide whether the engine refuses or keeps its internal-caller tolerance #7095 had to add assertOrderByIsMaterializable inside objectql because saved reports forward query.orderBy straight into engine.find, bypassing the REST ingress. Filters are forwarded the same way from reports, flows and dashboards, so an ingress-only fix would leave the author-reachable half open — the same lesson, one axis over.
  2. Blast radius. Unlike sort, a filter changes the row SET, so a refusal turns today's silent-zero surfaces into loud 4xx. That is the correct direction (ADR-0032, no silent failure), but it will surface every existing app that has one of these, which is a migration question rather than a pure bug fix.

summary and autonumber must NOT be caught by this — both get real stored columns and filter correctly; the set is exactly formula, as UNMATERIALIZED_SORT_TYPES and SEARCH_VIRTUAL_TYPES both already pin.


Generated by Claude Code

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions