From 9c180688a3f5b7a738c089b64eb9e8780d16eb24 Mon Sep 17 00:00:00 2001 From: os-zhuang Date: Sat, 8 Aug 2026 09:46:15 +0000 Subject: [PATCH] docs(search): make the mirror field the one prescription for related-title search MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `$search` scans the queried object's own columns; a dotted path (`project_id.name`) in `searchableFields` / `$searchFields` is refused by the #4254 ingress gate and by `validate-searchable-fields`. Both errors already pointed at "mirror the related title onto a local field", but nothing in the authoring corpus said it, so every project invented its own convention. Writes ONE prescription, told the same way in every place authors and AI look: to search by a related record's title, mirror that title into a STORED text field on this object and declare that field searchable. - skills/objectstack-data — new `## Search Fields (searchableFields)` section with the worked `task` → `project_name` example, the maintenance hooks, and both error messages quoted verbatim so either is greppable back to the fix. - skills/objectstack-query — mirror-field subsection under Full-Text Search (query-side shape + expansion), plus a row in the cross-object tool table. - content/docs/data-modeling/schema-design — canonical long form. - content/docs/data-modeling/queries, protocol/objectql/query-syntax, api/data-api, ui/views — short form pointing at the canonical section. Records one correctness point the existing hints get loose: the mirror must be a STORED field. A `formula` field is virtual (no driver materializes a column), so a `$contains` predicate against one has nothing to scan, and CEL only reads `record.` so it cannot fetch the related title anyway — yet `searchableFields` admits any declared field, so the mistake clears both lint and the ingress gate and then silently never matches. Cross-object search paths stay rejected by design; routes 2/3 are not written in as roadmap. Fixes #4840 --- content/docs/api/data-api.mdx | 13 +++ content/docs/data-modeling/queries.mdx | 24 +++++ content/docs/data-modeling/schema-design.mdx | 89 ++++++++++++++++++ .../docs/protocol/objectql/query-syntax.mdx | 26 ++++++ content/docs/ui/views.mdx | 2 +- skills/objectstack-data/SKILL.md | 90 +++++++++++++++++++ skills/objectstack-query/SKILL.md | 51 +++++++++++ 7 files changed, 294 insertions(+), 1 deletion(-) diff --git a/content/docs/api/data-api.mdx b/content/docs/api/data-api.mdx index b6f4bf522b..26e4e1394b 100644 --- a/content/docs/api/data-api.mdx +++ b/content/docs/api/data-api.mdx @@ -128,6 +128,7 @@ corrupts something the earlier axes do not: | `?search=alpha&searchFields=title` | scans only `title` | | `?search=alpha&searchFields=no_such_field` | `400 INVALID_FIELD` | | `?search=alpha&searchFields=amount` | `400 INVALID_FIELD` — real field, but not searchable | +| `?search=alpha&searchFields=project_id.name` | `400 INVALID_FIELD` — search scans this object's own columns; mirror the related title instead (see below) | | `groupBy: ["status"]` | one bucket per status value | | `groupBy: ["no_such_field"]` | `400 INVALID_FIELD` | | `aggregations: [{function:"sum", field:"amount", alias:"total"}]` | the real total | @@ -143,6 +144,18 @@ corrupts something the earlier axes do not: `searchableFields`), and a `searchableFields` entry that names no field (a stale declaration — the bug is on the object, and clients that echo the declaration verbatim are told so). + + A **dotted path** (`project_id.name`) is the typo case with its own hint: + `search` scans this object's own columns, so a related record's column can + never be a search target, and the search axis does not resolve traversal the + way `$select` / `$orderby` / `$filter` do. To search by a related record's + title, **mirror** that title into a stored field on this object and declare + *that* field searchable — a task list searched by project name carries a + `project_name` text column on `task`, maintained on write and listed in + `task.searchableFields`. It must be a stored field: a `formula` field is + virtual, so no driver has a column for `$contains` to scan. Cross-object + search paths are rejected by design, not pending — see + [Schema Design → Searching by a related record's title](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value). - **`groupBy`** — an unknown column projected `null` for every row, so all rows fell into **one bucket** whose count is the true row count: structurally perfect, indistinguishable from a column that really holds a diff --git a/content/docs/data-modeling/queries.mdx b/content/docs/data-modeling/queries.mdx index 0d6f8b63da..f6980a387d 100644 --- a/content/docs/data-modeling/queries.mdx +++ b/content/docs/data-modeling/queries.mdx @@ -513,6 +513,30 @@ that exists to narrow a search, silently widening it. Internal callers reaching `engine.find()` directly are unaffected. +### Searching by a related record's title — mirror the value + +`search` scans **the queried object's own columns**. A dotted path +(`project_id.name`) is not a search target — unlike `fields` / `sort` / `filters`, +the search axis does not resolve traversal, and a dotted entry is refused, not +silently dropped: + +```text +Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which +columns 'search' scans, so a name the object does not declare cannot narrow +anything — and the engine used to drop it and scan the default columns instead, +answering a NARROWER search with a WIDER one. 'search' scans this object's own +columns; a related record's column cannot be a search target. +``` + +The answer is a **mirror field**: copy the related record's title into a stored +field on this object and declare *that* field searchable. A task list searched by +project name gets a `project_name` text column on `task`, maintained on write and +listed in `task.searchableFields`. It has to be a **stored** field — a `formula` +field is virtual, so no driver has a column for `$contains` to scan. Cross-object +search paths are rejected by design, not pending. Full recipe (the hooks that keep +the mirror fresh, and the lint wording) in [Schema Design → Searching by a related +record's title](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value). + ### Pinyin recall (Chinese deployments) When pinyin search is enabled (`OS_SEARCH_PINYIN_ENABLED` — auto-on when the stack's diff --git a/content/docs/data-modeling/schema-design.mdx b/content/docs/data-modeling/schema-design.mdx index c58c713fdf..a40f4c3d1d 100644 --- a/content/docs/data-modeling/schema-design.mdx +++ b/content/docs/data-modeling/schema-design.mdx @@ -103,6 +103,95 @@ Queries then pass a top-level `$search` parameter to match across `searchableFie `$searchFields`. When `searchableFields` is unset, search falls back to the name/title field plus short-text fields. +#### Searching by a related record's title — mirror the value + +`$search` scans **the queried object's own columns**. A dotted path such as +`project_id.name` is not a search target: unlike `$select` / `$orderby` / +`$filter`, the search axis does not resolve traversal, and a dotted entry is +refused rather than silently dropped (#4254). That refusal is deliberate, not a +missing feature — cross-object search paths are rejected by design. + +The declarative answer is a **mirror field**: copy the related record's title +into a stored field on *this* object, and make that field the search target. +To let users search a task list by project name: + +```typescript +// `project_name` is a stored, denormalized copy of the parent's title. +{ + name: 'task', + enable: { searchable: true }, + fields: { + name: { type: 'text', required: true }, + project_id: { type: 'lookup', reference: 'project' }, + project_name: { type: 'text', label: 'Project Name' }, // ← the mirror + }, + searchableFields: ['name', 'project_name'], +} +``` + +`?search=apollo` now expands to `name $contains 'apollo' OR project_name +$contains 'apollo'` — one single-table scan, on every driver, with no traversal. +If the object declares no `searchableFields` at all, a `text` mirror is picked up +by the auto-default anyway; declare the set explicitly when you want to pin it. + + +**The mirror must be a stored field — a `formula` field does not work.** A +`formula` field is *virtual*: no driver materializes a column for it, so a +`$contains` predicate against one has nothing to scan (the SQL driver would emit +a `WHERE` over a column that does not exist). A CEL formula also only reads this +record's own fields (`record.`), so it cannot fetch the related title in +the first place. Nothing catches the mistake for you — `searchableFields` admits +any field the object declares, so a formula entry passes both lint and the +ingress gate and then just never matches. + + +**Keeping the mirror fresh.** A mirror is denormalized data, only as current as +whatever maintains it. Two write paths have to be covered: + +| When | What maintains the mirror | +|:-----|:--------------------------| +| A task is created, or re-pointed at another project | `beforeInsert` / `beforeUpdate` hook on `task` — read the parent's `name` for the incoming `project_id` and stamp `project_name` | +| A project is renamed | `afterUpdate` hook on `project` — re-stamp `project_name` on that project's tasks | + +Rows written by a path that bypasses hooks (bulk import, direct SQL) need a +one-off backfill. See [Hooks](/docs/automation/hooks) for the hook shapes. + +**The errors you get if you try the dotted path.** Both the lint and the runtime +send you to the same fix, so either message is greppable back to this section. + +`os validate` reports `searchable-field-unknown`: + +```text +searchableFields entry "project_id.name" is not a field on object "task". The +declaration is stale: searching it can never match, and the engine silently +drops it — leaving a narrower search than declared, or the auto-default set once +every entry is dropped. + +hint: 'search' scans this object's own columns, so a related record's column +cannot be a search target — expand the relation and search the related object, +or copy the value onto a formula field here. Clients echo this declaration +verbatim as the '$searchFields' override, so a stale entry becomes a 400 +INVALID_FIELD on list search (#4254), not just a quietly narrowed one. +``` + +(That hint's "text/formula" family of wording is loose — only the **stored** +half works; see the callout above.) + +A request that sends the dotted path is `400 INVALID_FIELD`: + +```text +Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which +columns 'search' scans, so a name the object does not declare cannot narrow +anything — and the engine used to drop it and scan the default columns instead, +answering a NARROWER search with a WIDER one. 'search' scans this object's own +columns; a related record's column cannot be a search target. +``` + +If the dotted path is in the object's own `searchableFields` (so clients echo it +back verbatim), the same 400 arrives under its stale-declaration wording +instead: `Field 'project_id.name' on object 'task' is declared in +'searchableFields' but does not exist.` + --- ## Field Types & Configuration diff --git a/content/docs/protocol/objectql/query-syntax.mdx b/content/docs/protocol/objectql/query-syntax.mdx index e577c066c7..a0d35c3894 100644 --- a/content/docs/protocol/objectql/query-syntax.mdx +++ b/content/docs/protocol/objectql/query-syntax.mdx @@ -887,6 +887,32 @@ a user typing `acme` does not find `ACME Corp`. Only `select` / `status` option `[EXPERIMENTAL — not enforced]` markers (#4286): the schema accepts them, the expansion ignores them. +#### Searching by a related record's title — mirror the value + +Search targets are **this object's own columns**. A dotted path is not one of +them: `searchFields: ['project_id.name']` is refused at the ingress rather than +dropped, because the search axis does not resolve traversal the way `fields`, +`sort` and `filters` do: + +```text +Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which +columns 'search' scans, so a name the object does not declare cannot narrow +anything — and the engine used to drop it and scan the default columns instead, +answering a NARROWER search with a WIDER one. 'search' scans this object's own +columns; a related record's column cannot be a search target. +``` + +The declarative answer is a **mirror field**: copy the related record's title +into a stored field on this object and declare *that* field searchable — a task +list searched by project name carries a `project_name` text column on `task`, +maintained on write and listed in `task.searchableFields`, so the expansion stays +a single-table `$or` of `$contains`. The mirror must be **stored**: a `formula` +field is virtual, no driver materializes a column for it, and a `$contains` +against one has nothing to scan. Cross-object search paths are rejected by +design, not pending — see [Schema Design → Searching by a related record's +title](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value) +for the maintenance hooks. + ### Joins — removed (#4286) `query.joins` was **removed in `@objectstack/spec` 17** (#4286, ADR-0049 diff --git a/content/docs/ui/views.mdx b/content/docs/ui/views.mdx index a0821feca7..1ed5b4ac28 100644 --- a/content/docs/ui/views.mdx +++ b/content/docs/ui/views.mdx @@ -103,7 +103,7 @@ A List View controls how a collection of records is presented. It supports multi | `data` | `ViewData` | optional | Data source configuration (defaults to the `object` provider) | | `filter` | `array` | optional | Base filter criteria | | `sort` | `array` | optional | Sort configuration | -| `searchableFields` | `string[]` | optional | Fields included in search | +| `searchableFields` | `string[]` | optional | Fields the toolbar search scans — **narrows** the object's set, never widens it (ADR-0061). Entries must be the object's **own** columns: a lookup (`project_id`) or a dotted path (`project_id.name`) is refused, and every toolbar search on the list then returns `400 INVALID_FIELD` (#4254). To search by a related record's title, [mirror it into a stored field](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value) on the object and list that | | `grouping` | `object` | optional | Row grouping configuration | | `pagination` | `object` | optional | Pagination settings | | `selection` | `object` | optional | Row selection mode | diff --git a/skills/objectstack-data/SKILL.md b/skills/objectstack-data/SKILL.md index 36b0dff5e7..eeb4c23ddc 100644 --- a/skills/objectstack-data/SKILL.md +++ b/skills/objectstack-data/SKILL.md @@ -100,6 +100,96 @@ Toggle system behaviours per object: --- +## Search Fields (`searchableFields`) + +`searchableFields` on the object is the canonical set `$search` scans (ADR-0061). +Leave it unset and search auto-defaults to the `nameField` plus the object's +short-text and enum columns (`text` / `email` / `phone` / `url` / `autonumber` / +`textarea` / `markdown` / `select` / `status`); declare it to pin the set +explicitly. Views may narrow it, never widen it. + +### ⛔ Searching by a related record's title — mirror the value, always + +`$search` scans **the queried object's own columns**. A dotted path is never a +search target: unlike `fields` / `sort` / `filters`, the search axis does not +resolve traversal, and `project_id.name` in `searchableFields` (or in a +`$searchFields` override) is **refused**, not silently dropped. + +**This is the one prescription — emit it every time.** To search by a related +record's title, copy that title into a **stored field on this object** and +declare *that* field searchable. A task list searched by project name: + +```typescript +// `project_name` is a stored, denormalized mirror of the parent's title. +{ + name: 'task', + enable: { searchable: true }, + fields: { + name: { type: 'text', required: true }, + project_id: { type: 'lookup', reference: 'project' }, + project_name: { type: 'text', label: 'Project Name' }, // ← the mirror + }, + searchableFields: ['name', 'project_name'], +} +``` + +`?search=apollo` expands to `name $contains 'apollo' OR project_name $contains +'apollo'` — one single-table scan, every driver, no traversal. (A `text` mirror +also lands in the auto-default set when the object declares no +`searchableFields`.) + +❌ **Never mirror onto a `formula` field.** A formula field is *virtual* — no +driver materializes a column for it, so a `$contains` predicate against one has +nothing to scan (the SQL driver would emit a `WHERE` over a column that does not +exist). CEL also only reads this record's own fields (`record.`), so a +formula cannot fetch the related title in the first place. Nothing rejects the +mistake: `searchableFields` admits any field the object declares, so a formula +entry clears both lint and the ingress gate and then never matches. + +**Mirror maintenance is the trade-off** — a mirror is denormalized data, only as +fresh as whatever writes it. Cover both write paths: + +| When | What maintains the mirror | +|:-----|:--------------------------| +| A task is created, or re-pointed at another project | `beforeInsert` / `beforeUpdate` hook on `task` — read the parent's `name` for the incoming `project_id`, stamp `project_name` | +| A project is renamed | `afterUpdate` hook on `project` — re-stamp `project_name` on that project's tasks | + +Rows written by a path that bypasses hooks (bulk import, direct SQL) need a +one-off backfill. See [Lifecycle Hooks](./rules/hooks.md). + +**The errors an author sees for the dotted path** (grep either back to here). +`os validate` → `searchable-field-unknown`: + +```text +searchableFields entry "project_id.name" is not a field on object "task". The +declaration is stale: searching it can never match, and the engine silently +drops it — leaving a narrower search than declared, or the auto-default set once +every entry is dropped. + +hint: 'search' scans this object's own columns, so a related record's column +cannot be a search target — expand the relation and search the related object, +or copy the value onto a formula field here. Clients echo this declaration +verbatim as the '$searchFields' override, so a stale entry becomes a 400 +INVALID_FIELD on list search (#4254), not just a quietly narrowed one. +``` + +(The hint's "formula field" wording is loose — only a **stored** mirror works.) + +A request carrying the dotted path is `400 INVALID_FIELD`: + +```text +Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which +columns 'search' scans, so a name the object does not declare cannot narrow +anything — and the engine used to drop it and scan the default columns instead, +answering a NARROWER search with a WIDER one. 'search' scans this object's own +columns; a related record's column cannot be a search target. +``` + +Cross-object search paths are rejected by design, not pending. Do not invent a +per-project convention for this — the mirror field is the answer. + +--- + ## Field Groups (MVP) Organize fields into logical groups (e.g., "Contact Information", "Billing", diff --git a/skills/objectstack-query/SKILL.md b/skills/objectstack-query/SKILL.md index 072d5d1a09..68e9347e5d 100644 --- a/skills/objectstack-query/SKILL.md +++ b/skills/objectstack-query/SKILL.md @@ -499,6 +499,56 @@ fields match by option *label*, mapped to stored values. Omit `fields` to search the object's declared `searchableFields` (or an auto-default of name/title + short-text fields), resolved server-side. +`fields` can only **narrow** that set, never widen it: over the REST/protocol +ingress a name outside it is `400 INVALID_FIELD` (#4254), not a silent +fall-back to the full scan. + +### ⛔ Searching by a related record's title — mirror the value, always + +`search` scans **the queried object's own columns**. A dotted path is never a +search target: unlike `fields` (projection) / `sort` / `filters`, the search axis +does not resolve traversal, so `searchFields: ['project_id.name']` is **refused**: + +```text +Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which +columns 'search' scans, so a name the object does not declare cannot narrow +anything — and the engine used to drop it and scan the default columns instead, +answering a NARROWER search with a WIDER one. 'search' scans this object's own +columns; a related record's column cannot be a search target. +``` + +**This is the one prescription — emit it every time.** Copy the related record's +title into a **stored field on the queried object** and search that field. A task +list searched by project name gets a `project_name` text column on `task`, +maintained on write and listed in `task.searchableFields`: + +```typescript +{ + object: 'task', + search: { query: 'apollo', fields: ['name', 'project_name'] }, + limit: 20, +} +// Expands to a single-table scan — no traversal, every driver: +// { $and: [{ $or: [ +// { name: { $contains: 'apollo' } }, +// { project_name: { $contains: 'apollo' } }, +// ]}]} +``` + +❌ The mirror must be a **stored** field — a `formula` field is virtual, no +driver materializes a column for it, so a `$contains` predicate against one has +nothing to scan. Nothing rejects the mistake for you: `searchableFields` admits +any field the object declares, so a formula entry clears both lint and the +ingress gate and then never matches. The trade-off is mirror maintenance — hooks +on both write paths (child re-parented, parent renamed) plus a backfill for rows +written around the hooks. + +Cross-object search paths are rejected by design, not pending. Modelling side of +this (the field, the hooks, the lint wording): **objectstack-data → Search Fields +(`searchableFields`)**. To *filter* by a related record's column — a different +axis — use a [nested relation filter](#nested-relation-filters); to *display* it, +use [`expand`](#expand-related-records). + > ⚠️ **`[EXPERIMENTAL — not enforced]` (#4286):** `fuzzy`, `boost`, > `operator`, `minScore`, `language`, and `highlight` validate against the > schema but are never read — their `.describe()` markers now say so. Terms @@ -535,6 +585,7 @@ auto-default of name/title + short-text fields), resolved server-side. |:---------|:----| | Load lookup fields for display | `expand` | | Filter parent by child conditions | Nested relation filter | +| **Keyword-search by a related record's title** | **Mirror the title into a stored field on this object and search that** — `search` never traverses (see **Full-Text Search** above) | | Simple parent→child navigation | `expand` | | Paginate/sort a parent's related records | Query the related object directly | | Analytical queries across objects | Report/dashboard metadata, or separate queries combined in app code (`joins` was removed in #4286 — see above) |