From 92d8d5e929b764755998b0c07a1bb212cfdf014a Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 18:42:47 +0000 Subject: [PATCH] fix(meta): /meta/object/:name reports the __search companion, agreeing with the list read (#8038) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two `/meta` reads of an object answered "does this object have a `__search` column?" two different ways, split cleanly by PROVENANCE. Measured end-to-end on the showcase, booted from a compiled artifact with `OS_SEARCH_PINYIN_ENABLED=true` (69 objects served): all 22 companion-bearing package objects carried it on `GET /meta/object` and were served WITHOUT it by `GET /meta/object/:name` and by `?layers=true`'s `effective`; all 45 companion-bearing platform objects carried it on both. Nothing about an object caused this; where its by-name read was ANSWERED FROM did. The companion is provisioned at the SchemaRegistry's object-materialization seam, so the list read — composed from `listItems('object')` — serves a materialized body, while the by-name read consults the `metadata` SERVICE first and on an artifact-booted deployment that service holds the author's declaration, captured before materialization. Fixed the way #6562 ruled this same seam (Option B: the read serves the effective runtime schema and the minority converges on the registry-backed majority) — at the read exits, from the registry that made the deployment-gated provisioning decision, so the pass and the decision cannot disagree. The `?layers=true` `code` and `overlay` layers stay byte-verbatim. The read adds a real field declaration, so the write path takes it back off again, as #6562's strip does for the injected columns: a runtime-created object's stored row went from `fields: [name]` to `fields: [__search, name]` on a single GET -> PUT before this. The strip is exact — only an entry byte-identical to what the provisioning seam would stamp is removed, recomputed from that function rather than transcribed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013bJJCxvDx9Z8uJQG5CikMj --- .../meta-object-search-companion-agreement.md | 67 ++++ packages/metadata-protocol/src/protocol.ts | 106 ++++- ...-object-search-companion-roundtrip.test.ts | 196 ++++++++++ packages/objectql/src/registry.ts | 92 ++++- ...-object-search-companion-agreement.test.ts | 366 ++++++++++++++++++ 5 files changed, 820 insertions(+), 7 deletions(-) create mode 100644 .changeset/meta-object-search-companion-agreement.md create mode 100644 packages/objectql/src/meta-object-search-companion-roundtrip.test.ts create mode 100644 packages/rest/src/meta-object-search-companion-agreement.test.ts diff --git a/.changeset/meta-object-search-companion-agreement.md b/.changeset/meta-object-search-companion-agreement.md new file mode 100644 index 0000000000..896a18b606 --- /dev/null +++ b/.changeset/meta-object-search-companion-agreement.md @@ -0,0 +1,67 @@ +--- +"@objectstack/objectql": patch +"@objectstack/metadata-protocol": patch +--- + +fix(meta): `/meta/object/:name` reports the `__search` companion column, agreeing with `GET /meta/object` (#8038) + +The two `/meta` reads of an object answered the "does this object have a +`__search` column?" question two different ways, split cleanly by PROVENANCE. +Measured end-to-end on the showcase, booted from a compiled artifact with +`OS_SEARCH_PINYIN_ENABLED=true` (69 objects served): + +- **22 package objects** carried the companion on `GET /meta/object` and were + served **without** it by `GET /meta/object/:name` — all 22 of them, and by + `?layers=true`'s `effective` layer too. +- **45 platform objects** carried it on both routes and always had. + +(The two objects with no title-eligible field — `showcase_project_membership` +and `sys_session` — have no companion on either route, correctly.) + +Nothing about an object caused this; where its by-name read was ANSWERED FROM +did. The companion is provisioned at the SchemaRegistry's +object-materialization seam, so `GET /meta/object` — composed from +`listItems('object')` — serves a materialized body. The by-name read consults +the `metadata` SERVICE first, and on a deployment booted from a compiled +artifact (`artifactSource`: every sealed/served runtime, and `objectstack +serve`) that service holds the author's DECLARATION, captured before +materialization. Platform objects are registered straight into the registry, so +their by-name read never meets that copy and agreed all along. Which side a +caller lands on is invisible in the response. + +This is the third thing to arrive through this exact gap, and it is fixed the +way the second one was ruled: #7556 folded the missing `objectExtensions`, and +#6562 ruled (maintainer, 2026-08-08, Option B) that a `/meta` object read serves +the **effective runtime schema** and the minority path converges on the +registry-backed majority — that is what `governServedItem` already does for the +injected system columns (`created_at`, `owner_id`, `organization_id`). The +companion is the same kind of thing coming through the same door, so it +converges at the same read exits, from the same authority: the registry that +made the provisioning decision. It is deployment-gated +(`OS_SEARCH_PINYIN_ENABLED`, or an explicit `searchCompanion` option), and the +gate is read off that registry rather than re-derived from the environment, so +the pass and the decision cannot disagree. + +**This is a payload change for every consumer of these routes.** Objects served +by the by-name read on an artifact-booted deployment now carry one additional +hidden field declaration — `__search` (`hidden`, `system`, `readonly`, +`searchable: false`) — where they previously did not, matching what the list +read has always served for the same object. Nothing is removed, and the +`?layers=true` `code` and `overlay` layers stay byte-verbatim: they are what the +package shipped and what the tenant customised, and the convergence deliberately +lands only on read exits and on `effective` (#6562 ruling constraint 1). + +Unrelated to #7642, which strips `__search` from RECORD bodies on the data path. +That is row values; this is the schema description, where the companion's +presence is the documented shape — #7561 exists precisely because `/meta` +re-parses the served object body and the stamp had to be spec-valid there. + +**Write path.** The read adds a real field declaration, so the write path takes +it back off again, exactly as #6562's `stripInjectedSystemColumns` does for the +injected columns: without it the ordinary GET → edit → PUT stored the platform's +own column as a tenant customisation. Measured on the runtime-created object +path — the write door type `object` has open by default — the stored row went +from `fields: [name]` to `fields: [__search, name]` on a single round-trip. The +strip is exact: only an entry byte-identical to what the provisioning seam would +stamp is removed, recomputed from that function rather than transcribed, so a +body carrying anything else under that name keeps it. diff --git a/packages/metadata-protocol/src/protocol.ts b/packages/metadata-protocol/src/protocol.ts index 544a80b7e0..bf5dfc76b1 100644 --- a/packages/metadata-protocol/src/protocol.ts +++ b/packages/metadata-protocol/src/protocol.ts @@ -4146,6 +4146,100 @@ export class ObjectStackProtocolImplementation implements } } + /** + * [#8038] The `__search` half of {@link governServedItem}'s presence + * convergence — the third stamp in the same family, and the one the + * module-level function cannot carry on its own. + * + * `applyInjectedSystemColumns` (#6562) converges the columns whose + * membership is a pure function of the document + * (`resolveInjectedSystemColumns`), so it needs nothing but the body. The + * search companion is DEPLOYMENT-gated: `SchemaRegistry` provisions it at + * the object-materialization seam only when its own `searchCompanion` flag + * is on, and that flag is `options.searchCompanion ?? + * resolveSearchPinyinEnabled()` — a host may set it explicitly. So the + * answer has to come from the registry that made the decision, which is why + * this is a method here and a `provisionSearchCompanionOnto` there, exactly + * as {@link foldObjectExtendersFromRegistry} is a method here and a + * `foldObjectExtendersOnto` there (#7556). + * + * Applied AFTER `governServedItem` at each exit, because the registry + * provisions the companion after `applySystemFields` too: the source field + * is resolved by `resolveDisplayField` over the POST-injection field set, + * so injecting first is what makes this pass and the registry's own answer + * the same answer rather than two independent guesses. + * + * No-op for every type but `object`, and — via the registry — for a + * deployment with companions off, an object with no eligible display field, + * and a body that already carries the column (every registry-backed read, + * which is the majority path this converges the minority onto). + */ + private provisionSearchCompanionFromRegistry(type: string, body: T): T { + if (canonicalMetaType(type) !== 'object') return body; + if (body === null || typeof body !== 'object') return body; + const registry = (this.engine as any)?.registry; + // Partial registry doubles in tests predate this method; a host that + // cannot answer the gate answers exactly as it did before. + if (!registry || typeof registry.provisionSearchCompanionOnto !== 'function') return body; + try { + return registry.provisionSearchCompanionOnto(body) as T; + } catch { + // A read over an in-memory flag and the body's own fields; a failure + // here must not turn a served schema into a 5xx. + return body; + } + } + + /** + * {@link governServedItem} plus the deployment-gated half it cannot reach + * ({@link provisionSearchCompanionFromRegistry}) — the whole of what a + * `/meta` READ EXIT owes an object document. Every call site of the free + * function that is a read exit goes through this instead; the one call site + * that is not — `getMetaItemLayered`'s `code` / `overlay` layers, which are + * deliberately raw — never called it in the first place (#6562 ruling + * constraint 1, #7556's boundary). + */ + private governServedObject(type: string, item: T): T { + return this.provisionSearchCompanionFromRegistry(type, governServedItem(type, item)); + } + + /** + * [#8038] The write-side counterpart of + * {@link provisionSearchCompanionFromRegistry}, owed for the reason + * {@link stripServedSystemColumns} is owed one field family over: the write + * path persists the request body verbatim (ADR-0005 §Validation), so a + * document this service's read added the companion to would otherwise be + * handed straight back and stored carrying it. + * + * Measured on the runtime-created object path — the write door type + * `object` has open by default, an artifact-backed object refusing the save + * outright with `NOT_OVERRIDABLE` — the stored row went from + * `fields: [name]` to `fields: [__search, name]` on a single GET → PUT + * before this was added. + */ + private stripSearchCompanionFromRegistry(type: string, item: T): T { + if (canonicalMetaType(type) !== 'object') return item; + if (item === null || typeof item !== 'object') return item; + const registry = (this.engine as any)?.registry; + if (!registry || typeof registry.stripProvisionedSearchCompanionFrom !== 'function') return item; + try { + return registry.stripProvisionedSearchCompanionFrom(item) as T; + } catch { + // A read over the body's own fields; a failure here must not turn a + // save into a 5xx. + return item; + } + } + + /** + * {@link stripServedSystemColumns} plus the companion half + * ({@link stripSearchCompanionFromRegistry}) — the whole of what the write + * path owes {@link governServedObject}. + */ + private stripServedObjectColumns(type: string, item: T): T { + return this.stripSearchCompanionFromRegistry(type, stripServedSystemColumns(type, item)); + } + /** * [#5840] Read ONE item from the `metadata` service, keeping the ADR-0110 * D3 verdict instead of flattening it into `undefined`. @@ -4610,7 +4704,7 @@ export class ObjectStackProtocolImplementation implements // is the other exit a client reads field metadata from, and // an overlay row wins over the (already-governed) registry // entry in the merge above, so it carries the same lie. - return governServedItem(request.type, mergeArtifactProtection(it, a)) as any; + return this.governServedObject(request.type, mergeArtifactProtection(it, a)) as any; }), ), }; @@ -4674,7 +4768,7 @@ export class ObjectStackProtocolImplementation implements return { type: request.type, name: request.name, - item: decorateMetadataItem(request.type, governServedItem(request.type, draftItem)), + item: decorateMetadataItem(request.type, this.governServedObject(request.type, draftItem)), }; } } catch (error) { @@ -4768,7 +4862,7 @@ export class ObjectStackProtocolImplementation implements return { type: request.type, name: request.name, - item: decorateMetadataItem(request.type, governServedItem(request.type, item)), + item: decorateMetadataItem(request.type, this.governServedObject(request.type, item)), }; } @@ -4893,7 +4987,7 @@ export class ObjectStackProtocolImplementation implements const artifactItem = this.lookupArtifactItem(request.type, request.name, request.packageId); let decorated = decorateMetadataItem( request.type, - governServedItem(request.type, mergeArtifactProtection(item, artifactItem)), + this.governServedObject(request.type, mergeArtifactProtection(item, artifactItem)), ); // ADR-0047 — list views additionally get reference-integrity // diagnostics (userFilters/tabs fields must exist on the source @@ -5185,7 +5279,7 @@ export class ObjectStackProtocolImplementation implements const effectiveBase: unknown | null = overlay !== null ? this.foldObjectExtendersFromRegistry(request.type, request.name, overlay) : code; - const effective: unknown | null = governServedItem(request.type, effectiveBase); + const effective: unknown | null = this.governServedObject(request.type, effectiveBase); const _diagnostics = effective !== null && effective !== undefined @@ -10026,7 +10120,7 @@ export class ObjectStackProtocolImplementation implements // schema gate, the authoring gate and the persisted body all still see // one document. See {@link stripServedSystemColumns} for why this is a // separate strip from the decoration list and not another entry in it. - request.item = stripServedSystemColumns(request.type, request.item); + request.item = this.stripServedObjectColumns(request.type, request.item); // Per-item lifecycle (ADR-0005 §"Drafts"). Default is `'publish'` // (legacy semantics — save goes straight live) to keep callers // that predate the draft/publish split working. Studio's diff --git a/packages/objectql/src/meta-object-search-companion-roundtrip.test.ts b/packages/objectql/src/meta-object-search-companion-roundtrip.test.ts new file mode 100644 index 0000000000..e4b80b07b3 --- /dev/null +++ b/packages/objectql/src/meta-object-search-companion-roundtrip.test.ts @@ -0,0 +1,196 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#8038] The write path takes back exactly what the converged read added + * (#4326) — the `__search` half. + * + * `packages/rest/src/meta-object-search-companion-agreement.test.ts` pins the + * READ: the by-name and list reads of an object agree about the hidden + * `__search` companion column, because both now converge on the registry's + * materialized schema. That convergence adds a real, spec-valid field + * declaration to bodies that did not carry one, and the write path persists a + * request body VERBATIM by design (ADR-0005 §Validation) — so without a strip + * counterpart the ordinary Studio GET → edit → PUT bakes the platform's own + * column into `sys_metadata.metadata`, into its checksum, and into every + * history diff. #6562 owed `applyInjectedSystemColumns` exactly this and says + * so in `governServedItem`'s docstring; this is the same debt for the same + * reason, one column over. + * + * Measured before the strip existed, on the host below: the stored row went + * from `fields: [name]` to `fields: [__search, name]` on a single round-trip. + * + * ## Why the object here is RUNTIME-CREATED, and why this file is in objectql + * + * Type `object` carries `allowOrgOverride: false`, so a save over an + * artifact-backed object is (correctly) refused with `NOT_OVERRIDABLE` and a + * test written that way would only measure the refusal. The write door that IS + * open by default is the runtime-created path (`allowRuntimeCreate` — the + * AI-authoring surface), which is what this seeds; the same body is reachable + * on any deployment listing `object` in `OS_METADATA_WRITABLE`. And the file + * lives here for the reason `protocol-meta-effective-schema.test.ts` does: the + * claim is about the REAL `SchemaRegistry` provisioning and the REAL protocol + * write agreeing, and only this package has both — `@objectstack/objectql` + * depends on `@objectstack/metadata-protocol`, never the reverse. + */ + +import { describe, it, expect } from 'vitest'; +import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; +// [#5619] The producer's OWN write-verb dispatch decisions, so the fake engine +// below cannot accept a call ObjectQL refuses. +import { assertEngineDeleteDispatch, assertEngineUpdateDispatch } from '@objectstack/metadata-core'; +import { SchemaRegistry } from './registry.js'; +import { SEARCH_COMPANION_FIELD } from './search-companion.js'; + +interface Row { + id: string; + type: string; + name: string; + organization_id: string | null; + package_id: string | null; + state: string; + metadata: string; + checksum?: string; + version?: number; +} + +/** A title-eligible field, so the materialization seam provisions a companion. */ +const AUTHORED = { + name: 'crm_lead', + label: 'Lead', + fields: { name: { name: 'name', label: 'Name', type: 'text' } }, +}; + +const clone = (v: T): T => JSON.parse(JSON.stringify(v)) as T; + +function matches(r: Row, where: Record): boolean { + for (const [k, v] of Object.entries(where)) { + if (v === undefined) continue; + if ((r as any)[k] !== v) return false; + } + return true; +} + +function keyOf(w: Record) { + return `${w.type}|${w.name}|${w.organization_id ?? '__env__'}|${w.state ?? 'active'}|${w.package_id ?? '__nopkg__'}`; +} + +/** A `sys_metadata` store over a REAL {@link SchemaRegistry}. */ +function makeHost() { + const registry = new SchemaRegistry({ multiTenant: false, searchCompanion: true } as never); + const rows = new Map(); + let nextId = 0; + const findRow = (w: Record) => { + for (const [k, r] of rows) if (matches(r, w)) return { key: k, row: r }; + return null; + }; + const engine: any = { + registry, + async findOne(_t: string, o: { where: Record }) { + return findRow(o.where)?.row ?? null; + }, + async find(_t: string, o: { where: Record }) { + return Array.from(rows.values()).filter((r) => matches(r, o.where)); + }, + async insert(table: string, data: Record) { + if (table !== 'sys_metadata') return { id: 'side_table' }; + const row = { id: `r_${++nextId}`, ...(data as any) } as Row; + rows.set(keyOf(data), row); + return { id: row.id }; + }, + async update(table: string, data: Record, o: { where: Record }) { + assertEngineUpdateDispatch(data, o); + if (table !== 'sys_metadata') return { id: null }; + const found = findRow(o.where); + if (!found) return { id: null }; + const merged = { ...found.row, ...(data as any) }; + rows.delete(found.key); + rows.set(keyOf(merged), merged); + return { id: found.row.id }; + }, + async delete(_t: string, o?: Record) { + assertEngineDeleteDispatch(o); + return { deleted: 0 }; + }, + async transaction(cb: (c: any, i: { owned: boolean }) => Promise): Promise { + return cb(undefined, { owned: true }); + }, + async syncObjectSchema() { /* no DDL in this stub */ }, + async count() { return 0; }, + async aggregate() { return []; }, + }; + const protocol = new ObjectStackProtocolImplementation(engine as never, () => new Map() as never); + const storedBody = () => { + const row = Array.from(rows.values()).find((r) => r.name === AUTHORED.name); + return row ? (JSON.parse(row.metadata) as Record) : undefined; + }; + return { protocol, rows, registry, storedBody }; +} + +async function seed() { + const host = makeHost(); + await host.protocol.saveMetaItem({ + type: 'object', name: AUTHORED.name, item: clone(AUTHORED), + } as never); + return host; +} + +describe('[#8038] the write path takes back the `__search` the read added (#4326)', () => { + it('GET → PUT the whole served document stores no companion column', async () => { + const host = await seed(); + const firstStored = host.storedBody()!; + // Precondition: the author's own row never carried the column. + expect(Object.keys(firstStored.fields)).toEqual(['name']); + + // What a client actually holds: the SERVED document, companion included. + const served: any = (await host.protocol.getMetaItem({ + type: 'object', name: AUTHORED.name, + })).item; + expect(Object.keys(served.fields)).toContain(SEARCH_COMPANION_FIELD); + + // Edit one label and PUT the whole thing back, as the designer does. + await host.protocol.saveMetaItem({ + type: 'object', name: AUTHORED.name, item: { ...served, label: 'Lead (edited)' }, + } as never); + + const stored = host.storedBody()!; + expect(Object.keys(stored.fields)).toEqual(['name']); + expect(stored.label).toBe('Lead (edited)'); + // Everything except the edited key is byte-identical to the first save. + expect({ ...stored, label: firstStored.label }).toEqual(firstStored); + }); + + it('a round-trip with NO edit leaves the stored body byte-identical', async () => { + const host = await seed(); + const firstStored = host.storedBody()!; + + const served: any = (await host.protocol.getMetaItem({ + type: 'object', name: AUTHORED.name, + })).item; + await host.protocol.saveMetaItem({ + type: 'object', name: AUTHORED.name, item: served, + } as never); + + expect(host.storedBody()).toEqual(firstStored); + }); + + it('strips ONLY the platform’s own definition, never a body carrying something else', async () => { + const host = await seed(); + + // Same field NAME, a definition the provisioning seam would never + // stamp. The strip is exact — recomputed from `provisionSearchCompanion` + // — so this survives, and a future drift between the two is a failure + // here rather than a silently discarded declaration. + const foreign = { type: 'text', label: 'Not the platform stamp' }; + await host.protocol.saveMetaItem({ + type: 'object', + name: AUTHORED.name, + item: { + ...clone(AUTHORED), + fields: { ...clone(AUTHORED.fields), [SEARCH_COMPANION_FIELD]: foreign }, + }, + } as never); + + const stored = host.storedBody()!; + expect(stored.fields[SEARCH_COMPANION_FIELD]).toEqual(foreign); + }); +}); diff --git a/packages/objectql/src/registry.ts b/packages/objectql/src/registry.ts index 46c9a9904a..6c4d6d0999 100644 --- a/packages/objectql/src/registry.ts +++ b/packages/objectql/src/registry.ts @@ -20,7 +20,7 @@ import { import { SystemFieldName } from '@objectstack/spec/system'; import { resolveTenancyPosture, resolveSearchPinyinEnabled } from '@objectstack/types'; import { postureEnforcesWall } from '@objectstack/spec/security'; -import { provisionSearchCompanion } from './search-companion.js'; +import { provisionSearchCompanion, SEARCH_COMPANION_FIELD } from './search-companion.js'; import { ObjectStackManifest, ManifestSchema, InstalledPackage, InstalledPackageSchema, checkFieldCompleteness } from '@objectstack/spec/kernel'; import { AppSchema } from '@objectstack/spec/ui'; import { applyProtection } from '@objectstack/spec/shared'; @@ -1575,6 +1575,96 @@ export class SchemaRegistry { ) as unknown as T; } + /** + * [#8038] Apply THIS registry's `__search` companion-column decision to a + * base body the CALLER supplies — the deployment-gated provisioning step + * {@link registerObject} runs on every base layer it materializes, exposed + * for a body that did not come from this registry. + * + * Same reason {@link foldObjectExtendersOnto} is public API rather than the + * protocol reaching for the contributor list: `GET /meta/object/:name` + * reaches an object body through a source this registry never sees — the + * copy `MetadataPlugin` registers into the `metadata` SERVICE when a + * deployment ingests a compiled artifact. That copy is the author's + * DECLARATION, captured before materialization, so it carries no companion + * column, while `GET /meta/object` reads `resolveObject` and carries one. + * Measured on the showcase (#8038): all 22 companion-bearing package objects + * were served WITHOUT `__search` by the by-name read and WITH it by the list + * read, while every platform object — registered straight into this registry, + * so never routed through the service — agreed on both. + * + * ⛔ The gate is the registry's own `searchCompanion` field and NOT a fresh + * `resolveSearchPinyinEnabled()` call. That field is `options.searchCompanion + * ?? resolveSearchPinyinEnabled()`, so a host which passes it explicitly — + * every test that pins this behaviour, and any embedder that overrides the + * env default — would otherwise get an answer re-derived from the + * environment instead of from the registry that actually provisioned (or + * declined to provision) the column. That is the #6562 failure mode + * restated: an injection pass and an injection table free to disagree. + * + * Returns `base` by reference when the deployment has companions off, when + * the object has no eligible display field, and when the column is already + * present — {@link provisionSearchCompanion} is pure and idempotent — so a + * registry-sourced body (provisioned at registration) pays a comparison and + * no copy, and a caller may apply it unconditionally. + */ + provisionSearchCompanionOnto(base: T): T { + if (base === null || typeof base !== 'object') return base; + if (!this.searchCompanion) return base; + return provisionSearchCompanion(base as never) as unknown as T; + } + + /** + * [#8038] The WRITE-side counterpart of {@link provisionSearchCompanionOnto}: + * take the companion column back off a body on its way IN, so a document the + * read served and a client handed straight back still persists byte-identical + * (#4326). + * + * Owed for exactly the reason `stripInjectedSystemColumns` is owed to + * `applyInjectedSystemColumns` (#6562): the write path persists the request + * body verbatim by design (ADR-0005 §Validation), so anything the READ adds + * must come off again or it is baked into `sys_metadata.metadata`, into its + * checksum, and into every history diff. Measured before this was added: a + * runtime-created object stored `fields: [name]`, and one GET → PUT + * round-trip through the converged read turned that into + * `fields: [__search, name]` — a column no author wrote, in the row that + * records what they customised. + * + * ⛔ Deliberately NOT gated on `searchCompanion`. The read that added the + * column and the write that hands it back are different requests, and a + * deployment may have flipped `OS_SEARCH_PINYIN_ENABLED` off in between; a + * gated strip would then bake in exactly the body the gate exists to + * prevent. What bounds this instead is EXACTNESS, the same discipline + * `isInjectedDefinition` applies: the entry is removed only when it is + * byte-identical to the definition {@link provisionSearchCompanion} would + * stamp — recomputed from that one function rather than transcribed, so the + * two cannot drift — and a body carrying anything else under the name keeps + * it. + * + * Returns `base` by reference when there is nothing to remove. Pure and total. + */ + stripProvisionedSearchCompanionFrom(base: T): T { + if (base === null || typeof base !== 'object') return base; + const fields = (base as { fields?: unknown }).fields; + if (!fields || typeof fields !== 'object' || Array.isArray(fields)) return base; + const present = (fields as Record)[SEARCH_COMPANION_FIELD]; + if (present === undefined) return base; + + // Re-stamp a copy WITHOUT the column and compare: the canonical definition + // comes from the provisioning function itself, and an object the seam would + // not provision for yields no stamp and therefore no removal. + const withoutCompanion = { ...(fields as Record) }; + delete withoutCompanion[SEARCH_COMPANION_FIELD]; + const restamped = provisionSearchCompanion( + { ...(base as Record), fields: withoutCompanion } as never, + ) as unknown as { fields?: Record }; + const canonical = restamped.fields?.[SEARCH_COMPANION_FIELD]; + if (canonical === undefined) return base; + if (stableStringify(present) !== stableStringify(canonical)) return base; + + return { ...(base as Record), fields: withoutCompanion } as unknown as T; + } + /** * [#8027] Remove from `base` the `validations` / `indexes` entries the * `extend` contributors are about to contribute — the half of diff --git a/packages/rest/src/meta-object-search-companion-agreement.test.ts b/packages/rest/src/meta-object-search-companion-agreement.test.ts new file mode 100644 index 0000000000..aa4c3bb90c --- /dev/null +++ b/packages/rest/src/meta-object-search-companion-agreement.test.ts @@ -0,0 +1,366 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. +// +// [#8038] The two producers of "what fields does this object have" must agree +// about the hidden `__search` companion column too. +// +// ## The defect, measured on `origin/main` +// +// The `__search` companion (#2486) is provisioned at the SchemaRegistry's +// object-materialization seam — `registerObject` runs `provisionSearchCompanion` +// on every base layer when the deployment has companions on. `GET /meta/object` +// composes its objects from `SchemaRegistry.listItems('object')`, so it serves +// the materialized body. `GET /meta/object/:name` consults the `metadata` +// SERVICE first, and on a deployment booted from a compiled artifact +// (`artifactSource` — every sealed/served runtime, and `objectstack serve`) that +// service holds the author's DECLARATION, captured before materialization. So +// the by-name read served a body with no companion column. +// +// Measured end-to-end on the showcase, booted from a build-shaped artifact with +// `OS_SEARCH_PINYIN_ENABLED=true` (69 objects served): +// +// - 23 package objects. 22 carry a companion on the list read; ALL 22 were +// served WITHOUT it by the by-name read. (The 23rd, +// `showcase_project_membership`, has no title-eligible field, so nothing +// provisions a companion for it on either route.) +// - 46 platform objects. 45 carry a companion, and every one of them agreed on +// both routes — they are registered straight into the registry, so the +// by-name read falls through to it and never meets the artifact copy. +// +// That clean partition is the whole tell: it is not a per-object accident, it is +// PROVENANCE. Whether an object's by-name read is answered by the artifact copy +// or by the registry is invisible to the caller. +// +// ## Which route is right +// +// The by-name one is wrong, on the settled reading of this exact seam. #6562 +// asked the same question about the platform's injected system columns — +// `created_at`, `owner_id`, `organization_id` — which the stored layers also +// reported as ABSENT, and the maintainer ruled (2026-08-08, Option B) that the +// read serves the EFFECTIVE runtime schema and the overlay-backed minority +// converges on the registry-backed majority. `__search` is the same kind of +// thing arriving through the same gap: a column the platform provisions, that +// the driver's `syncSchema` materializes (ADR-0045), and that the majority path +// already serves on 45 platform objects and on every list read. +// +// It is NOT the #7642 question. That issue strips `__search` from RECORD bodies +// on the data path, and says so in its own scope note — "deliberately this one +// column, not hidden system columns as a class" — about row VALUES. This is the +// schema description, where the companion's presence is the documented shape: +// #7561 exists precisely because `/meta` re-parses the served object body and +// the companion's stamp had to be spec-valid there. +// +// ## WHAT THIS FILE ASSERTS, and why it is shaped this way +// +// NOT "the by-name route returns `__search`". That passes again the day someone +// re-adds the column at the route, which is the same defect one layer over. It +// asserts AGREEMENT — the by-name read and the list read expose the same field +// set — with both sides MEASURED from the real producers in one test, over a +// real `RestServer` on a real `ObjectStackProtocolImplementation` on a real +// `SchemaRegistry`. +// +// Agreement ALONE is not enough, and #8045 is why: when both routes agree on a +// WRONG body an agreement pin is green throughout the defect. So every host also +// pins what they agree ON — the registry's own resolved schema, which is the +// contract both routes are reporting — and the anti-vacuity hosts below pin that +// the companion is genuinely DISCRIMINATED rather than always-present. + +import { describe, it, expect, vi } from 'vitest'; +import { SchemaRegistry } from '@objectstack/objectql'; +import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; +import { RestServer } from './rest-server.js'; + +/** The hidden companion column, spelled out rather than imported — see below. */ +const SEARCH_COMPANION = '__search'; + +/** + * An object with a title-eligible field, so the materialization seam provisions + * a companion for it. Stands for the 22. + */ +const TITLED_DECLARATION = { + name: 'showcase_account', + label: 'Account', + fields: { + name: { name: 'name', label: 'Name', type: 'text' }, + industry: { name: 'industry', label: 'Industry', type: 'text' }, + }, +}; + +/** + * A junction object with NO title-eligible field (`lookup` and `number` are both + * outside `TITLE_ELIGIBLE_TYPES`), so `resolveSearchCompanionSources` finds no + * source and NOTHING provisions a companion — on either route, on any host. + * Stands for `showcase_project_membership` and `sys_session` in the measurement + * above, and it is one of this file's two anti-vacuity cases. + */ +const UNTITLED_DECLARATION = { + name: 'showcase_project_membership', + label: 'Project Membership', + fields: { + project: { name: 'project', label: 'Project', type: 'lookup', reference_to: 'showcase_project' }, + seats: { name: 'seats', label: 'Seats', type: 'number' }, + }, +}; + +const clone = (v: T): T => JSON.parse(JSON.stringify(v)) as T; + +function createMockServer() { + return { + get: vi.fn(), post: vi.fn(), put: vi.fn(), delete: vi.fn(), patch: vi.fn(), + use: vi.fn(), + listen: vi.fn().mockResolvedValue(undefined), + close: vi.fn().mockResolvedValue(undefined), + }; +} + +/** + * Field names off a served object body, tolerant of both shapes the wire uses + * (`fields` as a record, or as an array of `{name}`), so the comparison is about + * WHICH fields are served rather than about how they are spelled. + */ +function fieldNamesOf(item: unknown): string[] { + const fields = (item as { fields?: unknown } | null | undefined)?.fields; + if (!fields) return []; + const names = Array.isArray(fields) + ? (fields as Array<{ name?: unknown }>).map((f) => String(f?.name)) + : Object.keys(fields as Record); + return [...names].sort(); +} + +/** + * How this host's `metadata` service was populated — the ONE axis that makes the + * two routes disagree, and the reason the hosts below are not variations on a + * theme. Same axis #7556 turned, because it is the same seam. + */ +type ServiceMode = + /** Artifact ingest: the service holds the author's declaration, pre-materialization. THE 22. */ + | 'artifact' + /** In-process boot: `bridgeObjectsToMetadataService` seeds the service from the MATERIALIZED registry. */ + | 'bridged' + /** No `metadata` service at all — the read falls through to the registry. THE PLATFORM SIDE. */ + | 'absent'; + +interface Host { + /** Field names `GET /meta/object` serves for the object. */ + listed: string[]; + /** Field names `GET /meta/object/:name` serves. */ + byName: string[]; + /** Field names the `effective` layer of `?layers=true` serves. */ + layerEffective: string[]; + /** What the `metadata` service itself holds — the base the by-name read starts from. */ + serviceBody: string[]; + /** The registry's own materialized schema — what both routes are supposed to be reporting. */ + registryResolved: string[]; +} + +/** + * Boot a REST server over the REAL protocol over a REAL registry, and read BOTH + * routes off it. The registry is always the real one, so the provisioning under + * test is the shipped provisioning and not a re-description of it. + */ +async function measure(opts: { + serviceMode: ServiceMode; + /** The deployment gate (`OS_SEARCH_PINYIN_ENABLED`, resolved once per registry). */ + searchCompanion?: boolean; + /** Register the junction object with no title-eligible field instead. */ + untitled?: boolean; + /** + * Seed a `sys_metadata` customisation row carrying the author's body — the + * THIRD link of the chain, and the one #6562 was raised on. ADR-0005 + * §Validation persists a written body verbatim, so this layer has been + * through no materialization either, and BOTH routes read it: the by-name + * read takes it in step 1, and the list read merges it over the registry + * entry. It is the only host on which the list exit does any work. + */ + overlayRow?: boolean; +}): Promise { + const declaration = opts.untitled ? UNTITLED_DECLARATION : TITLED_DECLARATION; + const objectName = declaration.name; + + const registry = new SchemaRegistry({ + multiTenant: false, + searchCompanion: opts.searchCompanion !== false, + } as never); + registry.registerObject(clone(declaration) as never, 'showcase', undefined, 'own'); + + // An overlay row stores the author's declaration — pre-materialization, + // exactly like the artifact copy, and for the same reason. + const overlayRows = opts.overlayRow + ? [{ + id: 'row_1', + type: 'object', + name: objectName, + organization_id: null, + package_id: null, + state: 'active', + metadata: JSON.stringify(clone(declaration)), + }] + : []; + const matchesRow = (where: Record | undefined) => + overlayRows.filter((r) => { + for (const [k, v] of Object.entries(where ?? {})) { + if (v === undefined) continue; + if ((r as Record)[k] !== v) return false; + } + return true; + }); + + const engine = { + registry, + find: async (_t: string, o?: { where?: Record }) => matchesRow(o?.where), + findOne: async (_t: string, o?: { where?: Record }) => + matchesRow(o?.where)[0], + }; + + const services = new Map(); + if (opts.serviceMode !== 'absent') { + // 'artifact' registers the AUTHOR'S DECLARATION (what a compiled + // artifact's `objects` collection holds — no companion, because nothing + // has materialized it yet); 'bridged' registers what the registry + // resolved. This single difference is the whole reproduction. + const body = opts.serviceMode === 'artifact' + ? clone(declaration) + : registry.getObject(objectName); + services.set('metadata', { + get: async (type: string, name: string) => + (type === 'object' || type === 'objects') && name === objectName + ? clone(body) + : undefined, + }); + } + + const protocol = new ObjectStackProtocolImplementation( + engine as never, + () => services as Map, + ); + + const rest = new RestServer( + createMockServer() as never, + protocol as never, + { api: { requireAuth: false } } as never, + ); + (rest as unknown as { resolveExecCtx: () => Promise }).resolveExecCtx = + async () => ({ userId: 'test-user' }); + rest.registerRoutes(); + const routes = rest.getRouteManager(); + + const run = async (path: string, params: Record, query: Record) => { + const entry = routes.get('GET', path); + if (!entry) throw new Error(`route not registered: ${path}`); + let body: unknown; + const res = { + status: () => res, + header: () => res, + json: (b: unknown) => { body = b; }, + send: (b: unknown) => { body = b; }, + } as unknown as Parameters[1]; + await entry.handler( + { params, query, headers: {}, method: 'GET', path } as unknown as Parameters[0], + res, + ); + return body as Record | undefined; + }; + + const listBody = await run('/api/v1/meta/:type', { type: 'object' }, {}); + const listItems = (Array.isArray(listBody) + ? listBody + : ((listBody?.items ?? listBody?.data ?? []) as unknown[])) as Array<{ name?: string }>; + const listed = listItems.find((o) => o?.name === objectName); + + const singleBody = await run('/api/v1/meta/:type/:name', { type: 'object', name: objectName }, {}); + const layeredBody = await run( + '/api/v1/meta/:type/:name', { type: 'object', name: objectName }, { layers: 'true' }, + ); + + const metadataService = services.get('metadata') as + { get(t: string, n: string): Promise } | undefined; + + return { + listed: fieldNamesOf(listed), + byName: fieldNamesOf(singleBody?.item), + layerEffective: fieldNamesOf(layeredBody?.effective), + serviceBody: metadataService + ? fieldNamesOf(await metadataService.get('object', objectName)) + : [], + registryResolved: fieldNamesOf(registry.getObject(objectName)), + }; +} + +describe('[#8038] the by-name and list reads agree about the `__search` companion', () => { + it('agrees on an ARTIFACT-ingested host — the 22 package objects, where the by-name read used to drop the column', async () => { + const host = await measure({ serviceMode: 'artifact' }); + + // The hosts are genuinely different, measured rather than asserted: the + // service's copy really does lack the column, so agreement here cannot + // be reached by this host having had nothing to converge. + expect(host.serviceBody).not.toContain(SEARCH_COMPANION); + // …and the list read really does serve it, so agreement cannot be + // reached by the list read quietly losing it too. + expect(host.listed).toContain(SEARCH_COMPANION); + + // The pin: whatever the list read serves, the by-name read serves. + expect(host.byName).toEqual(host.listed); + // …and what they agree ON is the registry's materialized schema, so the + // agreement cannot be satisfied by both routes drifting together (#8045). + expect(host.byName).toEqual(host.registryResolved); + // `?layers=true`'s `effective` is documented as "what `getMetaItem` + // would return" (#4513/#8027), so it converges with them or that + // sentence stops being true. + expect(host.layerEffective).toEqual(host.listed); + }); + + it('agrees on a REGISTRY-ONLY host — the platform objects, which already agreed', async () => { + const host = await measure({ serviceMode: 'absent' }); + + expect(host.listed).toContain(SEARCH_COMPANION); + expect(host.byName).toEqual(host.listed); + expect(host.byName).toEqual(host.registryResolved); + }); + + it('agrees on a BRIDGED in-process host — the boot that always happened to work', async () => { + const host = await measure({ serviceMode: 'bridged' }); + + expect(host.serviceBody).toContain(SEARCH_COMPANION); + expect(host.byName).toEqual(host.listed); + expect(host.byName).toEqual(host.registryResolved); + }); + + it('agrees on an OVERLAY-backed host — the third link, and the only one the LIST exit converges', async () => { + // The customisation row is the base layer for BOTH routes here, so this + // is the one host where the list read is not simply reporting the + // registry: revert the list exit's convergence and this case alone goes + // red, in the `listed` half rather than the `byName` half. + const host = await measure({ serviceMode: 'absent', overlayRow: true }); + + expect(host.listed).toContain(SEARCH_COMPANION); + expect(host.byName).toEqual(host.listed); + expect(host.byName).toEqual(host.registryResolved); + }); + + // ── Anti-vacuity ──────────────────────────────────────────────────────── + // Both routes agreeing is only worth something if the column is genuinely + // discriminated. These two hosts prove it is: the convergence adds the + // companion where the registry provisions one and NOWHERE else, so a fix + // that simply stamped `__search` onto every served object fails here. + + it('serves no companion on either route when the DEPLOYMENT has companions off', async () => { + const host = await measure({ serviceMode: 'artifact', searchCompanion: false }); + + expect(host.registryResolved).not.toContain(SEARCH_COMPANION); + expect(host.listed).not.toContain(SEARCH_COMPANION); + expect(host.byName).not.toContain(SEARCH_COMPANION); + expect(host.byName).toEqual(host.listed); + expect(host.byName).toEqual(host.registryResolved); + }); + + it('serves no companion on either route for an object with NO title-eligible field', async () => { + const host = await measure({ serviceMode: 'artifact', untitled: true }); + + // Companions are ON for this deployment — the gate is not what makes + // this case empty; the object having no source field is. + expect(host.registryResolved).not.toContain(SEARCH_COMPANION); + expect(host.listed).not.toContain(SEARCH_COMPANION); + expect(host.byName).not.toContain(SEARCH_COMPANION); + expect(host.byName).toEqual(host.listed); + expect(host.byName).toEqual(host.registryResolved); + }); +});