diff --git a/content/docs/api/data-api.mdx b/content/docs/api/data-api.mdx index b6f4bf522b..26e4e1394b 100644 --- a/content/docs/api/data-api.mdx +++ b/content/docs/api/data-api.mdx @@ -128,6 +128,7 @@ corrupts something the earlier axes do not: | `?search=alpha&searchFields=title` | scans only `title` | | `?search=alpha&searchFields=no_such_field` | `400 INVALID_FIELD` | | `?search=alpha&searchFields=amount` | `400 INVALID_FIELD` — real field, but not searchable | +| `?search=alpha&searchFields=project_id.name` | `400 INVALID_FIELD` — search scans this object's own columns; mirror the related title instead (see below) | | `groupBy: ["status"]` | one bucket per status value | | `groupBy: ["no_such_field"]` | `400 INVALID_FIELD` | | `aggregations: [{function:"sum", field:"amount", alias:"total"}]` | the real total | @@ -143,6 +144,18 @@ corrupts something the earlier axes do not: `searchableFields`), and a `searchableFields` entry that names no field (a stale declaration — the bug is on the object, and clients that echo the declaration verbatim are told so). + + A **dotted path** (`project_id.name`) is the typo case with its own hint: + `search` scans this object's own columns, so a related record's column can + never be a search target, and the search axis does not resolve traversal the + way `$select` / `$orderby` / `$filter` do. To search by a related record's + title, **mirror** that title into a stored field on this object and declare + *that* field searchable — a task list searched by project name carries a + `project_name` text column on `task`, maintained on write and listed in + `task.searchableFields`. It must be a stored field: a `formula` field is + virtual, so no driver has a column for `$contains` to scan. Cross-object + search paths are rejected by design, not pending — see + [Schema Design → Searching by a related record's title](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value). - **`groupBy`** — an unknown column projected `null` for every row, so all rows fell into **one bucket** whose count is the true row count: structurally perfect, indistinguishable from a column that really holds a diff --git a/content/docs/data-modeling/queries.mdx b/content/docs/data-modeling/queries.mdx index 0d6f8b63da..f6980a387d 100644 --- a/content/docs/data-modeling/queries.mdx +++ b/content/docs/data-modeling/queries.mdx @@ -513,6 +513,30 @@ that exists to narrow a search, silently widening it. Internal callers reaching `engine.find()` directly are unaffected. +### Searching by a related record's title — mirror the value + +`search` scans **the queried object's own columns**. A dotted path +(`project_id.name`) is not a search target — unlike `fields` / `sort` / `filters`, +the search axis does not resolve traversal, and a dotted entry is refused, not +silently dropped: + +```text +Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which +columns 'search' scans, so a name the object does not declare cannot narrow +anything — and the engine used to drop it and scan the default columns instead, +answering a NARROWER search with a WIDER one. 'search' scans this object's own +columns; a related record's column cannot be a search target. +``` + +The answer is a **mirror field**: copy the related record's title into a stored +field on this object and declare *that* field searchable. A task list searched by +project name gets a `project_name` text column on `task`, maintained on write and +listed in `task.searchableFields`. It has to be a **stored** field — a `formula` +field is virtual, so no driver has a column for `$contains` to scan. Cross-object +search paths are rejected by design, not pending. Full recipe (the hooks that keep +the mirror fresh, and the lint wording) in [Schema Design → Searching by a related +record's title](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value). + ### Pinyin recall (Chinese deployments) When pinyin search is enabled (`OS_SEARCH_PINYIN_ENABLED` — auto-on when the stack's diff --git a/content/docs/data-modeling/schema-design.mdx b/content/docs/data-modeling/schema-design.mdx index c58c713fdf..a40f4c3d1d 100644 --- a/content/docs/data-modeling/schema-design.mdx +++ b/content/docs/data-modeling/schema-design.mdx @@ -103,6 +103,95 @@ Queries then pass a top-level `$search` parameter to match across `searchableFie `$searchFields`. When `searchableFields` is unset, search falls back to the name/title field plus short-text fields. +#### Searching by a related record's title — mirror the value + +`$search` scans **the queried object's own columns**. A dotted path such as +`project_id.name` is not a search target: unlike `$select` / `$orderby` / +`$filter`, the search axis does not resolve traversal, and a dotted entry is +refused rather than silently dropped (#4254). That refusal is deliberate, not a +missing feature — cross-object search paths are rejected by design. + +The declarative answer is a **mirror field**: copy the related record's title +into a stored field on *this* object, and make that field the search target. +To let users search a task list by project name: + +```typescript +// `project_name` is a stored, denormalized copy of the parent's title. +{ + name: 'task', + enable: { searchable: true }, + fields: { + name: { type: 'text', required: true }, + project_id: { type: 'lookup', reference: 'project' }, + project_name: { type: 'text', label: 'Project Name' }, // ← the mirror + }, + searchableFields: ['name', 'project_name'], +} +``` + +`?search=apollo` now expands to `name $contains 'apollo' OR project_name +$contains 'apollo'` — one single-table scan, on every driver, with no traversal. +If the object declares no `searchableFields` at all, a `text` mirror is picked up +by the auto-default anyway; declare the set explicitly when you want to pin it. + + +**The mirror must be a stored field — a `formula` field does not work.** A +`formula` field is *virtual*: no driver materializes a column for it, so a +`$contains` predicate against one has nothing to scan (the SQL driver would emit +a `WHERE` over a column that does not exist). A CEL formula also only reads this +record's own fields (`record.`), so it cannot fetch the related title in +the first place. Nothing catches the mistake for you — `searchableFields` admits +any field the object declares, so a formula entry passes both lint and the +ingress gate and then just never matches. + + +**Keeping the mirror fresh.** A mirror is denormalized data, only as current as +whatever maintains it. Two write paths have to be covered: + +| When | What maintains the mirror | +|:-----|:--------------------------| +| A task is created, or re-pointed at another project | `beforeInsert` / `beforeUpdate` hook on `task` — read the parent's `name` for the incoming `project_id` and stamp `project_name` | +| A project is renamed | `afterUpdate` hook on `project` — re-stamp `project_name` on that project's tasks | + +Rows written by a path that bypasses hooks (bulk import, direct SQL) need a +one-off backfill. See [Hooks](/docs/automation/hooks) for the hook shapes. + +**The errors you get if you try the dotted path.** Both the lint and the runtime +send you to the same fix, so either message is greppable back to this section. + +`os validate` reports `searchable-field-unknown`: + +```text +searchableFields entry "project_id.name" is not a field on object "task". The +declaration is stale: searching it can never match, and the engine silently +drops it — leaving a narrower search than declared, or the auto-default set once +every entry is dropped. + +hint: 'search' scans this object's own columns, so a related record's column +cannot be a search target — expand the relation and search the related object, +or copy the value onto a formula field here. Clients echo this declaration +verbatim as the '$searchFields' override, so a stale entry becomes a 400 +INVALID_FIELD on list search (#4254), not just a quietly narrowed one. +``` + +(That hint's "text/formula" family of wording is loose — only the **stored** +half works; see the callout above.) + +A request that sends the dotted path is `400 INVALID_FIELD`: + +```text +Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which +columns 'search' scans, so a name the object does not declare cannot narrow +anything — and the engine used to drop it and scan the default columns instead, +answering a NARROWER search with a WIDER one. 'search' scans this object's own +columns; a related record's column cannot be a search target. +``` + +If the dotted path is in the object's own `searchableFields` (so clients echo it +back verbatim), the same 400 arrives under its stale-declaration wording +instead: `Field 'project_id.name' on object 'task' is declared in +'searchableFields' but does not exist.` + --- ## Field Types & Configuration diff --git a/content/docs/protocol/objectql/query-syntax.mdx b/content/docs/protocol/objectql/query-syntax.mdx index e577c066c7..a0d35c3894 100644 --- a/content/docs/protocol/objectql/query-syntax.mdx +++ b/content/docs/protocol/objectql/query-syntax.mdx @@ -887,6 +887,32 @@ a user typing `acme` does not find `ACME Corp`. Only `select` / `status` option `[EXPERIMENTAL — not enforced]` markers (#4286): the schema accepts them, the expansion ignores them. +#### Searching by a related record's title — mirror the value + +Search targets are **this object's own columns**. A dotted path is not one of +them: `searchFields: ['project_id.name']` is refused at the ingress rather than +dropped, because the search axis does not resolve traversal the way `fields`, +`sort` and `filters` do: + +```text +Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which +columns 'search' scans, so a name the object does not declare cannot narrow +anything — and the engine used to drop it and scan the default columns instead, +answering a NARROWER search with a WIDER one. 'search' scans this object's own +columns; a related record's column cannot be a search target. +``` + +The declarative answer is a **mirror field**: copy the related record's title +into a stored field on this object and declare *that* field searchable — a task +list searched by project name carries a `project_name` text column on `task`, +maintained on write and listed in `task.searchableFields`, so the expansion stays +a single-table `$or` of `$contains`. The mirror must be **stored**: a `formula` +field is virtual, no driver materializes a column for it, and a `$contains` +against one has nothing to scan. Cross-object search paths are rejected by +design, not pending — see [Schema Design → Searching by a related record's +title](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value) +for the maintenance hooks. + ### Joins — removed (#4286) `query.joins` was **removed in `@objectstack/spec` 17** (#4286, ADR-0049 diff --git a/content/docs/ui/views.mdx b/content/docs/ui/views.mdx index a0821feca7..1ed5b4ac28 100644 --- a/content/docs/ui/views.mdx +++ b/content/docs/ui/views.mdx @@ -103,7 +103,7 @@ A List View controls how a collection of records is presented. It supports multi | `data` | `ViewData` | optional | Data source configuration (defaults to the `object` provider) | | `filter` | `array` | optional | Base filter criteria | | `sort` | `array` | optional | Sort configuration | -| `searchableFields` | `string[]` | optional | Fields included in search | +| `searchableFields` | `string[]` | optional | Fields the toolbar search scans — **narrows** the object's set, never widens it (ADR-0061). Entries must be the object's **own** columns: a lookup (`project_id`) or a dotted path (`project_id.name`) is refused, and every toolbar search on the list then returns `400 INVALID_FIELD` (#4254). To search by a related record's title, [mirror it into a stored field](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value) on the object and list that | | `grouping` | `object` | optional | Row grouping configuration | | `pagination` | `object` | optional | Pagination settings | | `selection` | `object` | optional | Row selection mode | diff --git a/skills/objectstack-data/SKILL.md b/skills/objectstack-data/SKILL.md index 36b0dff5e7..eeb4c23ddc 100644 --- a/skills/objectstack-data/SKILL.md +++ b/skills/objectstack-data/SKILL.md @@ -100,6 +100,96 @@ Toggle system behaviours per object: --- +## Search Fields (`searchableFields`) + +`searchableFields` on the object is the canonical set `$search` scans (ADR-0061). +Leave it unset and search auto-defaults to the `nameField` plus the object's +short-text and enum columns (`text` / `email` / `phone` / `url` / `autonumber` / +`textarea` / `markdown` / `select` / `status`); declare it to pin the set +explicitly. Views may narrow it, never widen it. + +### ⛔ Searching by a related record's title — mirror the value, always + +`$search` scans **the queried object's own columns**. A dotted path is never a +search target: unlike `fields` / `sort` / `filters`, the search axis does not +resolve traversal, and `project_id.name` in `searchableFields` (or in a +`$searchFields` override) is **refused**, not silently dropped. + +**This is the one prescription — emit it every time.** To search by a related +record's title, copy that title into a **stored field on this object** and +declare *that* field searchable. A task list searched by project name: + +```typescript +// `project_name` is a stored, denormalized mirror of the parent's title. +{ + name: 'task', + enable: { searchable: true }, + fields: { + name: { type: 'text', required: true }, + project_id: { type: 'lookup', reference: 'project' }, + project_name: { type: 'text', label: 'Project Name' }, // ← the mirror + }, + searchableFields: ['name', 'project_name'], +} +``` + +`?search=apollo` expands to `name $contains 'apollo' OR project_name $contains +'apollo'` — one single-table scan, every driver, no traversal. (A `text` mirror +also lands in the auto-default set when the object declares no +`searchableFields`.) + +❌ **Never mirror onto a `formula` field.** A formula field is *virtual* — no +driver materializes a column for it, so a `$contains` predicate against one has +nothing to scan (the SQL driver would emit a `WHERE` over a column that does not +exist). CEL also only reads this record's own fields (`record.`), so a +formula cannot fetch the related title in the first place. Nothing rejects the +mistake: `searchableFields` admits any field the object declares, so a formula +entry clears both lint and the ingress gate and then never matches. + +**Mirror maintenance is the trade-off** — a mirror is denormalized data, only as +fresh as whatever writes it. Cover both write paths: + +| When | What maintains the mirror | +|:-----|:--------------------------| +| A task is created, or re-pointed at another project | `beforeInsert` / `beforeUpdate` hook on `task` — read the parent's `name` for the incoming `project_id`, stamp `project_name` | +| A project is renamed | `afterUpdate` hook on `project` — re-stamp `project_name` on that project's tasks | + +Rows written by a path that bypasses hooks (bulk import, direct SQL) need a +one-off backfill. See [Lifecycle Hooks](./rules/hooks.md). + +**The errors an author sees for the dotted path** (grep either back to here). +`os validate` → `searchable-field-unknown`: + +```text +searchableFields entry "project_id.name" is not a field on object "task". The +declaration is stale: searching it can never match, and the engine silently +drops it — leaving a narrower search than declared, or the auto-default set once +every entry is dropped. + +hint: 'search' scans this object's own columns, so a related record's column +cannot be a search target — expand the relation and search the related object, +or copy the value onto a formula field here. Clients echo this declaration +verbatim as the '$searchFields' override, so a stale entry becomes a 400 +INVALID_FIELD on list search (#4254), not just a quietly narrowed one. +``` + +(The hint's "formula field" wording is loose — only a **stored** mirror works.) + +A request carrying the dotted path is `400 INVALID_FIELD`: + +```text +Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which +columns 'search' scans, so a name the object does not declare cannot narrow +anything — and the engine used to drop it and scan the default columns instead, +answering a NARROWER search with a WIDER one. 'search' scans this object's own +columns; a related record's column cannot be a search target. +``` + +Cross-object search paths are rejected by design, not pending. Do not invent a +per-project convention for this — the mirror field is the answer. + +--- + ## Field Groups (MVP) Organize fields into logical groups (e.g., "Contact Information", "Billing", diff --git a/skills/objectstack-query/SKILL.md b/skills/objectstack-query/SKILL.md index 072d5d1a09..68e9347e5d 100644 --- a/skills/objectstack-query/SKILL.md +++ b/skills/objectstack-query/SKILL.md @@ -499,6 +499,56 @@ fields match by option *label*, mapped to stored values. Omit `fields` to search the object's declared `searchableFields` (or an auto-default of name/title + short-text fields), resolved server-side. +`fields` can only **narrow** that set, never widen it: over the REST/protocol +ingress a name outside it is `400 INVALID_FIELD` (#4254), not a silent +fall-back to the full scan. + +### ⛔ Searching by a related record's title — mirror the value, always + +`search` scans **the queried object's own columns**. A dotted path is never a +search target: unlike `fields` (projection) / `sort` / `filters`, the search axis +does not resolve traversal, so `searchFields: ['project_id.name']` is **refused**: + +```text +Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which +columns 'search' scans, so a name the object does not declare cannot narrow +anything — and the engine used to drop it and scan the default columns instead, +answering a NARROWER search with a WIDER one. 'search' scans this object's own +columns; a related record's column cannot be a search target. +``` + +**This is the one prescription — emit it every time.** Copy the related record's +title into a **stored field on the queried object** and search that field. A task +list searched by project name gets a `project_name` text column on `task`, +maintained on write and listed in `task.searchableFields`: + +```typescript +{ + object: 'task', + search: { query: 'apollo', fields: ['name', 'project_name'] }, + limit: 20, +} +// Expands to a single-table scan — no traversal, every driver: +// { $and: [{ $or: [ +// { name: { $contains: 'apollo' } }, +// { project_name: { $contains: 'apollo' } }, +// ]}]} +``` + +❌ The mirror must be a **stored** field — a `formula` field is virtual, no +driver materializes a column for it, so a `$contains` predicate against one has +nothing to scan. Nothing rejects the mistake for you: `searchableFields` admits +any field the object declares, so a formula entry clears both lint and the +ingress gate and then never matches. The trade-off is mirror maintenance — hooks +on both write paths (child re-parented, parent renamed) plus a backfill for rows +written around the hooks. + +Cross-object search paths are rejected by design, not pending. Modelling side of +this (the field, the hooks, the lint wording): **objectstack-data → Search Fields +(`searchableFields`)**. To *filter* by a related record's column — a different +axis — use a [nested relation filter](#nested-relation-filters); to *display* it, +use [`expand`](#expand-related-records). + > ⚠️ **`[EXPERIMENTAL — not enforced]` (#4286):** `fuzzy`, `boost`, > `operator`, `minScore`, `language`, and `highlight` validate against the > schema but are never read — their `.describe()` markers now say so. Terms @@ -535,6 +585,7 @@ auto-default of name/title + short-text fields), resolved server-side. |:---------|:----| | Load lookup fields for display | `expand` | | Filter parent by child conditions | Nested relation filter | +| **Keyword-search by a related record's title** | **Mirror the title into a stored field on this object and search that** — `search` never traverses (see **Full-Text Search** above) | | Simple parent→child navigation | `expand` | | Paginate/sort a parent's related records | Query the related object directly | | Analytical queries across objects | Report/dashboard metadata, or separate queries combined in app code (`joins` was removed in #4286 — see above) |