Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions content/docs/api/data-api.mdx
Original file line numberDiff line numberDiff line change
Expand Up@@ -128,6 +128,7 @@ corrupts something the earlier axes do not:
| `?search=alpha&searchFields=title` | scans only `title` |
| `?search=alpha&searchFields=no_such_field` | `400 INVALID_FIELD` |
| `?search=alpha&searchFields=amount` | `400 INVALID_FIELD` — real field, but not searchable |
| `?search=alpha&searchFields=project_id.name` | `400 INVALID_FIELD` — search scans this object's own columns; mirror the related title instead (see below) |
| `groupBy: ["status"]` | one bucket per status value |
| `groupBy: ["no_such_field"]` | `400 INVALID_FIELD` |
| `aggregations: [{function:"sum", field:"amount", alias:"total"}]` | the real total |
Expand All@@ -143,6 +144,18 @@ corrupts something the earlier axes do not:
`searchableFields`), and a `searchableFields` entry that names no field (a
stale declaration — the bug is on the object, and clients that echo the
declaration verbatim are told so).

A **dotted path** (`project_id.name`) is the typo case with its own hint:
`search` scans this object's own columns, so a related record's column can
never be a search target, and the search axis does not resolve traversal the
way `$select` / `$orderby` / `$filter` do. To search by a related record's
title, **mirror** that title into a stored field on this object and declare
*that* field searchable — a task list searched by project name carries a
`project_name` text column on `task`, maintained on write and listed in
`task.searchableFields`. It must be a stored field: a `formula` field is
virtual, so no driver has a column for `$contains` to scan. Cross-object
search paths are rejected by design, not pending — see
[Schema Design → Searching by a related record's title](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value).
- **`groupBy`** — an unknown column projected `null` for every row, so all
rows fell into **one bucket** whose count is the true row count:
structurally perfect, indistinguishable from a column that really holds a
Expand Down
24 changes: 24 additions & 0 deletions content/docs/data-modeling/queries.mdx
Original file line numberDiff line numberDiff line change
Expand Up@@ -513,6 +513,30 @@ that exists to narrow a search, silently widening it. Internal callers reaching
`engine.find()` directly are unaffected.
</Callout>

### Searching by a related record's title — mirror the value

`search` scans **the queried object's own columns**. A dotted path
(`project_id.name`) is not a search target — unlike `fields` / `sort` / `filters`,
the search axis does not resolve traversal, and a dotted entry is refused, not
silently dropped:

```text
Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which
columns 'search' scans, so a name the object does not declare cannot narrow
anything — and the engine used to drop it and scan the default columns instead,
answering a NARROWER search with a WIDER one. 'search' scans this object's own
columns; a related record's column cannot be a search target.
```

The answer is a **mirror field**: copy the related record's title into a stored
field on this object and declare *that* field searchable. A task list searched by
project name gets a `project_name` text column on `task`, maintained on write and
listed in `task.searchableFields`. It has to be a **stored** field — a `formula`
field is virtual, so no driver has a column for `$contains` to scan. Cross-object
search paths are rejected by design, not pending. Full recipe (the hooks that keep
the mirror fresh, and the lint wording) in [Schema Design → Searching by a related
record's title](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value).

### Pinyin recall (Chinese deployments)

When pinyin search is enabled (`OS_SEARCH_PINYIN_ENABLED` — auto-on when the stack's
Expand Down
89 changes: 89 additions & 0 deletions content/docs/data-modeling/schema-design.mdx
Original file line numberDiff line numberDiff line change
Expand Up@@ -103,6 +103,95 @@ Queries then pass a top-level `$search` parameter to match across `searchableFie
`$searchFields`. When `searchableFields` is unset, search falls back to the
name/title field plus short-text fields.

#### Searching by a related record's title — mirror the value

`$search` scans **the queried object's own columns**. A dotted path such as
`project_id.name` is not a search target: unlike `$select` / `$orderby` /
`$filter`, the search axis does not resolve traversal, and a dotted entry is
refused rather than silently dropped (#4254). That refusal is deliberate, not a
missing feature — cross-object search paths are rejected by design.

The declarative answer is a **mirror field**: copy the related record's title
into a stored field on *this* object, and make that field the search target.
To let users search a task list by project name:

```typescript
// `project_name` is a stored, denormalized copy of the parent's title.
{
name: 'task',
enable: { searchable: true },
fields: {
name: { type: 'text', required: true },
project_id: { type: 'lookup', reference: 'project' },
project_name: { type: 'text', label: 'Project Name' }, // ← the mirror
},
searchableFields: ['name', 'project_name'],
}
```

`?search=apollo` now expands to `name $contains 'apollo' OR project_name
$contains 'apollo'` — one single-table scan, on every driver, with no traversal.
If the object declares no `searchableFields` at all, a `text` mirror is picked up
by the auto-default anyway; declare the set explicitly when you want to pin it.

<Callout type="warn">
**The mirror must be a stored field — a `formula` field does not work.** A
`formula` field is *virtual*: no driver materializes a column for it, so a
`$contains` predicate against one has nothing to scan (the SQL driver would emit
a `WHERE` over a column that does not exist). A CEL formula also only reads this
record's own fields (`record.<field>`), so it cannot fetch the related title in
the first place. Nothing catches the mistake for you — `searchableFields` admits
any field the object declares, so a formula entry passes both lint and the
ingress gate and then just never matches.
</Callout>

**Keeping the mirror fresh.** A mirror is denormalized data, only as current as
whatever maintains it. Two write paths have to be covered:

| When | What maintains the mirror |
|:-----|:--------------------------|
| A task is created, or re-pointed at another project | `beforeInsert` / `beforeUpdate` hook on `task` — read the parent's `name` for the incoming `project_id` and stamp `project_name` |
| A project is renamed | `afterUpdate` hook on `project` — re-stamp `project_name` on that project's tasks |

Rows written by a path that bypasses hooks (bulk import, direct SQL) need a
one-off backfill. See [Hooks](/docs/automation/hooks) for the hook shapes.

**The errors you get if you try the dotted path.** Both the lint and the runtime
send you to the same fix, so either message is greppable back to this section.

`os validate` reports `searchable-field-unknown`:

```text
searchableFields entry "project_id.name" is not a field on object "task". The
declaration is stale: searching it can never match, and the engine silently
drops it — leaving a narrower search than declared, or the auto-default set once
every entry is dropped.

hint: 'search' scans this object's own columns, so a related record's column
cannot be a search target — expand the relation and search the related object,
or copy the value onto a formula field here. Clients echo this declaration
verbatim as the '$searchFields' override, so a stale entry becomes a 400
INVALID_FIELD on list search (#4254), not just a quietly narrowed one.
```

(That hint's "text/formula" family of wording is loose — only the **stored**
half works; see the callout above.)

A request that sends the dotted path is `400 INVALID_FIELD`:

```text
Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which
columns 'search' scans, so a name the object does not declare cannot narrow
anything — and the engine used to drop it and scan the default columns instead,
answering a NARROWER search with a WIDER one. 'search' scans this object's own
columns; a related record's column cannot be a search target.
```

If the dotted path is in the object's own `searchableFields` (so clients echo it
back verbatim), the same 400 arrives under its stale-declaration wording
instead: `Field 'project_id.name' on object 'task' is declared in
'searchableFields' but does not exist.`

---

## Field Types & Configuration
Expand Down
26 changes: 26 additions & 0 deletions content/docs/protocol/objectql/query-syntax.mdx
Original file line numberDiff line numberDiff line change
Expand Up@@ -887,6 +887,32 @@ a user typing `acme` does not find `ACME Corp`. Only `select` / `status` option
`[EXPERIMENTAL — not enforced]` markers (#4286): the schema accepts them, the
expansion ignores them.

#### Searching by a related record's title — mirror the value

Search targets are **this object's own columns**. A dotted path is not one of
them: `searchFields: ['project_id.name']` is refused at the ingress rather than
dropped, because the search axis does not resolve traversal the way `fields`,
`sort` and `filters` do:

```text
Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which
columns 'search' scans, so a name the object does not declare cannot narrow
anything — and the engine used to drop it and scan the default columns instead,
answering a NARROWER search with a WIDER one. 'search' scans this object's own
columns; a related record's column cannot be a search target.
```

The declarative answer is a **mirror field**: copy the related record's title
into a stored field on this object and declare *that* field searchable — a task
list searched by project name carries a `project_name` text column on `task`,
maintained on write and listed in `task.searchableFields`, so the expansion stays
a single-table `$or` of `$contains`. The mirror must be **stored**: a `formula`
field is virtual, no driver materializes a column for it, and a `$contains`
against one has nothing to scan. Cross-object search paths are rejected by
design, not pending — see [Schema Design → Searching by a related record's
title](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value)
for the maintenance hooks.

### Joins — removed (#4286)

`query.joins` was **removed in `@objectstack/spec` 17** (#4286, ADR-0049
Expand Down
2 changes: 1 addition & 1 deletion content/docs/ui/views.mdx
Original file line numberDiff line numberDiff line change
Expand Up@@ -103,7 +103,7 @@ A List View controls how a collection of records is presented. It supports multi
| `data` | `ViewData` | optional | Data source configuration (defaults to the `object` provider) |
| `filter` | `array` | optional | Base filter criteria |
| `sort` | `array` | optional | Sort configuration |
| `searchableFields` | `string[]` | optional | Fields included in search |
| `searchableFields` | `string[]` | optional | Fields the toolbar search scans — **narrows** the object's set, never widens it (ADR-0061). Entries must be the object's **own** columns: a lookup (`project_id`) or a dotted path (`project_id.name`) is refused, and every toolbar search on the list then returns `400 INVALID_FIELD` (#4254). To search by a related record's title, [mirror it into a stored field](/docs/data-modeling/schema-design#searching-by-a-related-records-title--mirror-the-value) on the object and list that |
| `grouping` | `object` | optional | Row grouping configuration |
| `pagination` | `object` | optional | Pagination settings |
| `selection` | `object` | optional | Row selection mode |
Expand Down
90 changes: 90 additions & 0 deletions skills/objectstack-data/SKILL.md
Original file line numberDiff line numberDiff line change
Expand Up@@ -100,6 +100,96 @@ Toggle system behaviours per object:

---

## Search Fields (`searchableFields`)

`searchableFields` on the object is the canonical set `$search` scans (ADR-0061).
Leave it unset and search auto-defaults to the `nameField` plus the object's
short-text and enum columns (`text` / `email` / `phone` / `url` / `autonumber` /
`textarea` / `markdown` / `select` / `status`); declare it to pin the set
explicitly. Views may narrow it, never widen it.

### ⛔ Searching by a related record's title — mirror the value, always

`$search` scans **the queried object's own columns**. A dotted path is never a
search target: unlike `fields` / `sort` / `filters`, the search axis does not
resolve traversal, and `project_id.name` in `searchableFields` (or in a
`$searchFields` override) is **refused**, not silently dropped.

**This is the one prescription — emit it every time.** To search by a related
record's title, copy that title into a **stored field on this object** and
declare *that* field searchable. A task list searched by project name:

```typescript
// `project_name` is a stored, denormalized mirror of the parent's title.
{
name: 'task',
enable: { searchable: true },
fields: {
name: { type: 'text', required: true },
project_id: { type: 'lookup', reference: 'project' },
project_name: { type: 'text', label: 'Project Name' }, // ← the mirror
},
searchableFields: ['name', 'project_name'],
}
```

`?search=apollo` expands to `name $contains 'apollo' OR project_name $contains
'apollo'` — one single-table scan, every driver, no traversal. (A `text` mirror
also lands in the auto-default set when the object declares no
`searchableFields`.)

❌ **Never mirror onto a `formula` field.** A formula field is *virtual* — no
driver materializes a column for it, so a `$contains` predicate against one has
nothing to scan (the SQL driver would emit a `WHERE` over a column that does not
exist). CEL also only reads this record's own fields (`record.<field>`), so a
formula cannot fetch the related title in the first place. Nothing rejects the
mistake: `searchableFields` admits any field the object declares, so a formula
entry clears both lint and the ingress gate and then never matches.

**Mirror maintenance is the trade-off** — a mirror is denormalized data, only as
fresh as whatever writes it. Cover both write paths:

| When | What maintains the mirror |
|:-----|:--------------------------|
| A task is created, or re-pointed at another project | `beforeInsert` / `beforeUpdate` hook on `task` — read the parent's `name` for the incoming `project_id`, stamp `project_name` |
| A project is renamed | `afterUpdate` hook on `project` — re-stamp `project_name` on that project's tasks |

Rows written by a path that bypasses hooks (bulk import, direct SQL) need a
one-off backfill. See [Lifecycle Hooks](./rules/hooks.md).

**The errors an author sees for the dotted path** (grep either back to here).
`os validate` → `searchable-field-unknown`:

```text
searchableFields entry "project_id.name" is not a field on object "task". The
declaration is stale: searching it can never match, and the engine silently
drops it — leaving a narrower search than declared, or the auto-default set once
every entry is dropped.

hint: 'search' scans this object's own columns, so a related record's column
cannot be a search target — expand the relation and search the related object,
or copy the value onto a formula field here. Clients echo this declaration
verbatim as the '$searchFields' override, so a stale entry becomes a 400
INVALID_FIELD on list search (#4254), not just a quietly narrowed one.
```

(The hint's "formula field" wording is loose — only a **stored** mirror works.)

A request carrying the dotted path is `400 INVALID_FIELD`:

```text
Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which
columns 'search' scans, so a name the object does not declare cannot narrow
anything — and the engine used to drop it and scan the default columns instead,
answering a NARROWER search with a WIDER one. 'search' scans this object's own
columns; a related record's column cannot be a search target.
```

Cross-object search paths are rejected by design, not pending. Do not invent a
per-project convention for this — the mirror field is the answer.

---

## Field Groups (MVP)

Organize fields into logical groups (e.g., "Contact Information", "Billing",
Expand Down
51 changes: 51 additions & 0 deletions skills/objectstack-query/SKILL.md
Original file line numberDiff line numberDiff line change
Expand Up@@ -499,6 +499,56 @@ fields match by option *label*, mapped to stored values.
Omit `fields` to search the object's declared `searchableFields` (or an
auto-default of name/title + short-text fields), resolved server-side.

`fields` can only **narrow** that set, never widen it: over the REST/protocol
ingress a name outside it is `400 INVALID_FIELD` (#4254), not a silent
fall-back to the full scan.

### ⛔ Searching by a related record's title — mirror the value, always

`search` scans **the queried object's own columns**. A dotted path is never a
search target: unlike `fields` (projection) / `sort` / `filters`, the search axis
does not resolve traversal, so `searchFields: ['project_id.name']` is **refused**:

```text
Unknown field 'project_id.name' on object 'task'. '$searchFields' narrows which
columns 'search' scans, so a name the object does not declare cannot narrow
anything — and the engine used to drop it and scan the default columns instead,
answering a NARROWER search with a WIDER one. 'search' scans this object's own
columns; a related record's column cannot be a search target.
```

**This is the one prescription — emit it every time.** Copy the related record's
title into a **stored field on the queried object** and search that field. A task
list searched by project name gets a `project_name` text column on `task`,
maintained on write and listed in `task.searchableFields`:

```typescript
{
object: 'task',
search: { query: 'apollo', fields: ['name', 'project_name'] },
limit: 20,
}
// Expands to a single-table scan — no traversal, every driver:
// { $and: [{ $or: [
// { name: { $contains: 'apollo' } },
// { project_name: { $contains: 'apollo' } },
// ]}]}
```

❌ The mirror must be a **stored** field — a `formula` field is virtual, no
driver materializes a column for it, so a `$contains` predicate against one has
nothing to scan. Nothing rejects the mistake for you: `searchableFields` admits
any field the object declares, so a formula entry clears both lint and the
ingress gate and then never matches. The trade-off is mirror maintenance — hooks
on both write paths (child re-parented, parent renamed) plus a backfill for rows
written around the hooks.

Cross-object search paths are rejected by design, not pending. Modelling side of
this (the field, the hooks, the lint wording): **objectstack-data → Search Fields
(`searchableFields`)**. To *filter* by a related record's column — a different
axis — use a [nested relation filter](#nested-relation-filters); to *display* it,
use [`expand`](#expand-related-records).

> ⚠️ **`[EXPERIMENTAL — not enforced]` (#4286):** `fuzzy`, `boost`,
> `operator`, `minScore`, `language`, and `highlight` validate against the
> schema but are never read — their `.describe()` markers now say so. Terms
Expand DownExpand Up@@ -535,6 +585,7 @@ auto-default of name/title + short-text fields), resolved server-side.
|:---------|:----|
| Load lookup fields for display | `expand` |
| Filter parent by child conditions | Nested relation filter |
| **Keyword-search by a related record's title** | **Mirror the title into a stored field on this object and search that** — `search` never traverses (see **Full-Text Search** above) |
| Simple parent→child navigation | `expand` |
| Paginate/sort a parent's related records | Query the related object directly |
| Analytical queries across objects | Report/dashboard metadata, or separate queries combined in app code (`joins` was removed in #4286 — see above) |
Expand Down
Loading