From 42dae358925774e46bf45ecc880d575a9199cb43 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 12:48:32 +0000 Subject: [PATCH 1/8] =?UTF-8?q?wip(metadata-protocol):=20leg=20D=20overlay?= =?UTF-8?q?=20cache=20=E2=80=94=20implementation=20before=20any=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src/meta-overlay-cache.ts | 268 ++++++++++++++++++ packages/metadata-protocol/src/protocol.ts | 106 +++++-- 2 files changed, 351 insertions(+), 23 deletions(-) create mode 100644 packages/metadata-protocol/src/meta-overlay-cache.ts diff --git a/packages/metadata-protocol/src/meta-overlay-cache.ts b/packages/metadata-protocol/src/meta-overlay-cache.ts new file mode 100644 index 0000000000..56bd1826bf --- /dev/null +++ b/packages/metadata-protocol/src/meta-overlay-cache.ts @@ -0,0 +1,268 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * ── Leg D of #11633: the `getMetaItems` overlay read cache (#11967) ───────── + * + * `getMetaItems` re-reads `sys_metadata` on every authenticated request. On the + * hot path — `enforceApiAccess` → `loadObjectItems` → `getMetaItems({ type: + * 'object' })`, once per REST request — that read costs **two** sequentially + * awaited engine queries whenever the environment holds no overlay rows for the + * type, because the empty first result triggers the alt-type retry. An app + * whose objects are all code-authored pays both on every request forever. That + * is the shape #11633 §1 calls out: *"most of leg D's win is negative caching"*. + * + * This module caches the OUTCOME OF THAT READ — the raw `sys_metadata` rows — + * per `(engine, type, packageId, organizationId)`, and retires the entry the + * moment the engine's write epoch moves. + * + * ## ⭐ Where this cache sits, and why that IS the resolution of the + * ## SchemaRegistry-hydration trap (#11633 §4 leg D) + * + * The design names a trap that a naive leg D walks into: `getMetaItems` is not + * a pure read. Its overlay branch **registers overlay rows back into the + * SchemaRegistry** (`hydrateOverlayIntoRegistry`, gated to unscoped kernels). + * A cache that skips the read also skips that registration, and the symptom is + * not a stale answer — it is a registry that quietly stops being populated. + * + * The design offers two ways out: cache the merged post-hydration result, or + * keep hydration outside the cached path and prove it idempotent. This module + * takes a third position that makes the trap **unreachable rather than + * avoided**: the cached value sits **UPSTREAM of the hydration branch**, not + * downstream of it. + * + * What is cached is the row set — the value the `queryByOrg` calls produce. + * Everything the row set feeds still runs on every single call, hit or miss: + * the overlay parse, the package-aware merge, `hydrateOverlayIntoRegistry`, the + * MetadataService merge, the disabled-package filter, the nav contributions and + * the decorations. A cache hit changes exactly one thing — where the rows came + * from — and changes nothing about what is done with them. So hydration cannot + * be skipped by a hit, and its idempotence never has to be proven, because it + * is not being replayed: it runs once per call, exactly as it does today. + * + * ⛔ Do NOT "optimise" this by moving the cache below the merge. That is the + * naive shape the trap describes, and it also silently breaks the three + * mutable, non-`sys_metadata` sources the merged answer depends on — see next. + * + * ## ⭐ Why only the row set is cached, and never the merged answer + * + * Measured on the shipped ref: `getMetaItems`' answer is a function of FOUR + * mutable sources, and the write epoch observes only ONE of them. + * + * | source | epoch sees it? | + * |---------------------------------------------------|----------------| + * | `sys_metadata` rows (`engine.find`) | YES — every write goes through `SysMetadataRepository` → `engine.insert/update/delete` → the middleware seam | + * | SchemaRegistry (`listItems`, `isPackageDisabled`, `applyNavContributions`) | NO — in-memory registration, no engine operation | + * | MetadataService (`metadataService.list`) | NO — loader-driven | + * | artifact protection (`lookupArtifactItem`) | NO — in-memory | + * + * An epoch-keyed cache of the MERGED answer would therefore be serving three + * sources whose changes nothing in its key can observe. Caching the row set + * keeps the cache's reach exactly co-extensive with what its key can validate: + * the one source the epoch does see. That containment is the whole argument for + * this shape, and it is the property to protect when editing this file. + * + * ## Invalidation + * + * 1. **Primary — the engine write epoch** (#11968's substrate, + * `objectql/src/write-epoch.ts`). Read STRUCTURALLY, never by import: + * `@objectstack/metadata-protocol` does not depend on + * `@objectstack/objectql`, and the substrate declared `WriteEpochLike` + * separately for exactly this kind of consumer. Any `insert`/`update`/ + * `delete` on ANY object advances it, so a `sys_metadata` publish retires + * this cache synchronously, in-process, before the next read. + * 2. **TTL** — `OS_METADATA_OVERLAY_CACHE_TTL_MS`, default 30s, `0` = off. + * The residual bound, covering only what the epoch cannot see: a PEER + * node's write on a deployment with no `authz.invalidated` bridge attached. + * With that bridge, a peer's hint bumps the LOCAL epoch + * (`authz-invalidation-bridge.ts` calls `epoch.bump('remote')`), so + * cross-node convergence narrows for free. + * + * ⚠️ **`metadata.changed` is NOT a trigger here, and #11633 §4's expectation + * that it would be does not survive measurement.** That channel is published by + * `MetadataManager.notifyWatchers`, driven by loader/repository events. The + * writer whose rows THIS cache holds is `SysMetadataRepository`, and + * `metadata-protocol/src/protocol.ts` contains no `notifyWatchers`, `subscribe` + * or `watchService` call at all — a `sys_metadata` overlay write emits no + * `metadata.changed` event. Subscribing to it would have bought this cache + * nothing and would have read, to the next maintainer, as a live invalidation + * path that never fires. The cross-node story for leg D is the + * `authz.invalidated` channel plus the TTL, which is the substrate's own + * contract. + * + * ## ⭐ A success is cached ONLY when the engine exposes the write epoch + * + * The rule leg C (#11966) landed, and it transfers unchanged. A `ql` with no + * seam is a `ql` whose writes this cache cannot see, so instead of degrading to + * a TTL-only shape — publish-visibility as a timer, which #11633 §4 rules out + * for this leg specifically — the cache declines. The whole surface is checked, + * not just `current`: a bare `{ current: number }` on some unrelated double + * would otherwise read as a live seam and licence caching against a counter + * nothing ever bumps. + * + * Every existing test double takes the declining path and keeps its exact query + * multiset; only a real engine caches. + * + * ## ⛔ Rows are cloned in BOTH directions, and that is correctness, not hygiene + * + * The overlay parse downstream reads `record.metadata` and, when it is already + * an object rather than a JSON string, hands that very object on to a merge + * chain that MUTATES it (`Object.assign(data, patch)`, `data._packageId = …`, + * `data._draft = true`). So: + * + * - **clone on store**, or this call's own merge corrupts the snapshot it + * just took; + * - **clone on serve**, or the first hit's merge corrupts the snapshot for + * every later hit. + * + * A fresh engine read hands back fresh rows, so cloning is what keeps a hit + * byte-equivalent to a miss. A row set that cannot be cloned is not cached — + * declining again rather than serving something aliased. + */ + +/** The identity of one cached overlay read. */ +export interface MetaOverlayCacheKey { + /** Canonical metadata type — already folded by `canonicalizeMetaRequestType`. */ + type: string; + packageId?: string; + organizationId?: string; +} + +/** One cached row set, with everything needed to decide it is still the answer. */ +interface MetaOverlayCacheEntry { + /** A private, already-cloned snapshot. Never handed out un-cloned. */ + records: unknown[]; + /** The engine write epoch this row set was read at. Compared, never interpreted. */ + epoch: number; + /** Wall-clock expiry of the residual TTL bound. */ + expiresAt: number; +} + +/** + * One bucket per engine. Keyed on the ENGINE rather than on the protocol + * instance because the engine is what owns BOTH halves of an entry's validity: + * the `sys_metadata` table the rows came from, and the write epoch that says + * when they stop being the answer. Two protocol instances over one engine read + * one table and share one epoch, so sharing a bucket is correct; two engines + * never share, which is what keeps two environments in one process from seeing + * each other's rows. + */ +const metaOverlayCache = new WeakMap>(); + +export const META_OVERLAY_CACHE_TTL_ENV = 'OS_METADATA_OVERLAY_CACHE_TTL_MS'; +export const META_OVERLAY_CACHE_DEFAULT_TTL_MS = 30_000; + +/** + * Staleness bound for the overlay cache, in ms. `0` disables it — a real path + * that restores the pre-#11967 query multiset exactly, not a degenerate TTL. + * + * Deployment config, never a settings row (#11633 §5). + * + * ⚠️ A malformed value resolves to `0` (off), the same arm leg C + * (`localizationSuccessCacheTtlMs`) chose and for the same reason: the default + * here is ON, so the two candidate readings of a typo are "off" and "30s", and + * folding `3OOO` (letter O) into the default would hand the operator a LONGER + * staleness window than the one they were trying to set. Off is the only arm + * whose failure mode is a missed optimisation rather than an unasked-for + * window. + */ +export function metaOverlayCacheTtlMs( + env: Record = typeof process !== 'undefined' ? process.env : {}, +): number { + const raw = env[META_OVERLAY_CACHE_TTL_ENV]; + if (raw === undefined || raw.trim() === '') return META_OVERLAY_CACHE_DEFAULT_TTL_MS; + const parsed = Number(raw.trim()); + if (!Number.isFinite(parsed) || parsed < 0) return 0; + return Math.floor(parsed); +} + +/** + * The engine's current write epoch, or `undefined` when this engine carries no + * such seam. Mirrors `isWriteEpochLike` from `@objectstack/objectql` rather + * than importing it — see the header for why that import is unavailable in this + * direction. + */ +export function readWriteEpoch(engine: unknown): number | undefined { + if (!engine || typeof engine !== 'object') return undefined; + const epoch = (engine as { writeEpoch?: unknown }).writeEpoch; + if (!epoch || typeof epoch !== 'object') return undefined; + const seam = epoch as { current?: unknown; bump?: unknown; subscribe?: unknown }; + if ( + typeof seam.current !== 'number' || + typeof seam.bump !== 'function' || + typeof seam.subscribe !== 'function' + ) { + return undefined; + } + return seam.current; +} + +/** + * The cache key. `JSON.stringify` over the tuple rather than a delimiter join: + * a package id or organization id containing the delimiter would otherwise let + * two different reads collide on one key. + */ +function cacheKeyOf(key: MetaOverlayCacheKey): string { + return JSON.stringify([key.type, key.packageId ?? null, key.organizationId ?? null]); +} + +/** + * Deep-clone a row set, or `undefined` when it cannot be cloned. See the + * header: an un-cloned row set is an aliased one, and the merge chain + * downstream mutates what it is handed. + */ +function cloneRecords(records: unknown[]): unknown[] | undefined { + try { + return structuredClone(records); + } catch { + return undefined; + } +} + +/** + * A live cached row set for `key`, or `undefined` for a miss. + * + * `epoch` is the caller's PRE-READ epoch reading — see the call site for why it + * must be taken before the read and not after. + */ +export function readMetaOverlayCache( + engine: unknown, + key: MetaOverlayCacheKey, + epoch: number | undefined, + ttlMs: number, + now: number, +): unknown[] | undefined { + if (epoch === undefined || ttlMs <= 0) return undefined; + if (!engine || typeof engine !== 'object') return undefined; + const entry = metaOverlayCache.get(engine as object)?.get(cacheKeyOf(key)); + if (!entry) return undefined; + if (entry.epoch !== epoch) return undefined; + if (entry.expiresAt <= now) return undefined; + return cloneRecords(entry.records); +} + +/** + * Remember `records` for `key`. A no-op when this engine exposes no write + * epoch, when the TTL is off, or when the rows do not clone. + * + * ⭐ An EMPTY row set is cached, deliberately and as the main point: the empty + * result is what triggers `getMetaItems`' alt-type retry, so "no overlay rows + * for this type" is the answer whose caching removes both reads. #11633 §4 + * names this the bulk of leg D's saving, and the epoch — never the TTL — is + * what keeps a newly published overlay promptly visible. + */ +export function writeMetaOverlayCache( + engine: unknown, + key: MetaOverlayCacheKey, + epoch: number | undefined, + records: unknown[], + ttlMs: number, + now: number, +): void { + if (epoch === undefined || ttlMs <= 0) return; + if (!engine || typeof engine !== 'object') return; + const snapshot = cloneRecords(records); + if (snapshot === undefined) return; + const bucket = metaOverlayCache.get(engine as object) ?? new Map(); + bucket.set(cacheKeyOf(key), { records: snapshot, epoch, expiresAt: now + ttlMs }); + metaOverlayCache.set(engine as object, bucket); +} diff --git a/packages/metadata-protocol/src/protocol.ts b/packages/metadata-protocol/src/protocol.ts index 8543c01385..24ab7f9f59 100644 --- a/packages/metadata-protocol/src/protocol.ts +++ b/packages/metadata-protocol/src/protocol.ts @@ -28,6 +28,13 @@ import type { RuntimeAuthoringIssue } from './runtime-authoring-gate.js'; // ADR-0120 D4 reporting that replaced this file's empty `catch` blocks. import { ensureMetadataOverlayIndexes } from './migrations/overlay-index.js'; import { SysMetadataRepository, type SysMetadataEngine } from './sys-metadata-repository.js'; +import { + metaOverlayCacheTtlMs, + readMetaOverlayCache, + readWriteEpoch, + writeMetaOverlayCache, + type MetaOverlayCacheKey, +} from './meta-overlay-cache.js'; import { ConflictError, assertProtocolCompat, @@ -6076,30 +6083,83 @@ export class ObjectStackProtocolImplementation implements } return rs ?? []; }; - const envWideRecords = await queryByOrg(null); - const orgRecords = orgId ? await queryByOrg(orgId) : []; - // org-specific rows override env-wide rows on name collision. - // ADR-0048 (#1828) — key by (package, name), not bare name, so a - // package A row and a package B row of the same name do not - // collapse; org-over-env precedence still holds within each slot. + // ── Leg D of #11633 (#11967): the cross-request overlay cache ── + // + // ⭐ The cache sits HERE, and its position IS the resolution of the + // SchemaRegistry-hydration trap #11633 §4 names. What is cached is + // the ROW SET — the value the two `queryByOrg` calls produce — + // never the merged answer below it. Everything downstream of this + // point still runs on every call, hit or miss: the overlay parse, + // the package-aware merge, `hydrateOverlayIntoRegistry`, the + // MetadataService merge, the disabled-package filter, the nav + // contributions, the decorations. A hit changes where the rows came + // from and nothing about what is done with them, so the read-side + // registry hydration cannot be skipped by one. // - // [#7774] …and for a bundled type the slot is `(package, name, - // locale)`. Within ONE org this changes nothing — the store's own - // unique index is `(type, name, organization_id, package_id)`, so - // an org cannot hold two rows that differ only by body locale. - // Across the two tiers it can: an env-wide row and this org's row - // may customize DIFFERENT members of one bundle, and keying them - // together made the org's zh-CN row silently displace the - // env-wide en-US one. Precedence is unchanged where it was ever - // meaningful — an org row still overrides the env-wide row of the - // same member — and an undiscriminated type keeps a - // byte-identical key. - const mergedMap = new Map(); - const rowKey = (r: any): string => - metaItemKey(r.package_id, r.name, storedRowDiscriminator(request.type, r)); - for (const r of envWideRecords) mergedMap.set(rowKey(r), r); - for (const r of orgRecords) mergedMap.set(rowKey(r), r); - const records = Array.from(mergedMap.values()); + // ⛔ Do NOT move this below the merge. That is precisely the naive + // shape the trap describes, and it would additionally serve three + // mutable sources — the SchemaRegistry, the MetadataService and the + // artifact table — whose changes nothing in this key can observe. + // See `meta-overlay-cache.ts` for the measured four-source table. + // + // ⭐ The epoch reading is taken BEFORE the read, and it is this + // pre-read value that is stored with the rows. A write landing + // WHILE this read is in flight therefore moves the epoch past what + // the entry records, so the entry is already dead when it is + // written — the safe direction. Reading it afterwards would stamp + // pre-write rows with a post-write epoch and make that staleness + // permanent: the clear-then-repopulate-from-a-stale-read failure + // #11633 §7 pin 2 names. + const overlayCacheKey: MetaOverlayCacheKey = { + type: request.type, + packageId, + organizationId: orgId, + }; + const overlayCacheEpoch = readWriteEpoch(this.engine); + const overlayCacheTtlMs = metaOverlayCacheTtlMs(); + const overlayCacheNow = Date.now(); + const cachedRecords = readMetaOverlayCache( + this.engine, overlayCacheKey, overlayCacheEpoch, overlayCacheTtlMs, overlayCacheNow, + ); + + let records: any[]; + if (cachedRecords !== undefined) { + records = cachedRecords as any[]; + } else { + const envWideRecords = await queryByOrg(null); + const orgRecords = orgId ? await queryByOrg(orgId) : []; + // org-specific rows override env-wide rows on name collision. + // ADR-0048 (#1828) — key by (package, name), not bare name, so a + // package A row and a package B row of the same name do not + // collapse; org-over-env precedence still holds within each slot. + // + // [#7774] …and for a bundled type the slot is `(package, name, + // locale)`. Within ONE org this changes nothing — the store's own + // unique index is `(type, name, organization_id, package_id)`, so + // an org cannot hold two rows that differ only by body locale. + // Across the two tiers it can: an env-wide row and this org's row + // may customize DIFFERENT members of one bundle, and keying them + // together made the org's zh-CN row silently displace the + // env-wide en-US one. Precedence is unchanged where it was ever + // meaningful — an org row still overrides the env-wide row of the + // same member — and an undiscriminated type keeps a + // byte-identical key. + const mergedMap = new Map(); + const rowKey = (r: any): string => + metaItemKey(r.package_id, r.name, storedRowDiscriminator(request.type, r)); + for (const r of envWideRecords) mergedMap.set(rowKey(r), r); + for (const r of orgRecords) mergedMap.set(rowKey(r), r); + records = Array.from(mergedMap.values()); + // ⭐ An EMPTY row set is cached too, and that is the main point + // rather than an edge case: the empty result is what triggers the + // alt-type retry above, so "no overlay rows for this type" is the + // answer whose caching removes BOTH reads. #11633 §1 measured that + // an app whose objects are all code-authored pays the doubled read + // on every request; this is the line that stops it. + writeMetaOverlayCache( + this.engine, overlayCacheKey, overlayCacheEpoch, records, overlayCacheTtlMs, overlayCacheNow, + ); + } if (records && records.length > 0) { const isView = (PLURAL_TO_SINGULAR[request.type] ?? request.type) === 'view'; // Parse each overlay body once — replaying the stored-row From 3c1488396001257a317adee67089b6ac2c5183dd Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 12:55:17 +0000 Subject: [PATCH 2/8] wip(metadata-protocol): leg D pins, ablation results not yet measured --- .../src/meta-overlay-cache.test.ts | 561 ++++++++++++++++++ 1 file changed, 561 insertions(+) create mode 100644 packages/metadata-protocol/src/meta-overlay-cache.test.ts diff --git a/packages/metadata-protocol/src/meta-overlay-cache.test.ts b/packages/metadata-protocol/src/meta-overlay-cache.test.ts new file mode 100644 index 0000000000..3d5eaed29c --- /dev/null +++ b/packages/metadata-protocol/src/meta-overlay-cache.test.ts @@ -0,0 +1,561 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Leg D of #11633 (#11967) — the `getMetaItems` overlay cache. + * + * ## What every case here is built to refuse + * + * "Invalidation works" passes trivially on a cache that never caches, and + * "the cache hits" passes trivially on a cache that never invalidates. So every + * staleness assertion in this file is PAIRED with a hit assertion on the same + * engine — a repeat issues **zero** reads — and the two ablations below move + * them in opposite directions. Neither half can carry the file alone. + * + * ## Test-path resolution — MEASURED, not assumed + * + * Both subjects are imported by RELATIVE specifier (`./protocol.js`, + * `./meta-overlay-cache.js`), so vitest resolves them from this package's + * SOURCE, never from its `dist/`. That is what decides whether an ablation + * needs a rebuild leg, and it was checked rather than inherited: the sibling + * file `protocol.hydrate-overlay-canonical-type.test.ts` needed + * mutate→rebuild→prove-in-artifact for its ablation precisely because ITS + * target (`canonicalMetaUrlType`) lives in `@objectstack/spec`, a WORKSPACE + * dependency that resolves through `exports` to built output. Nothing this file + * ablates crosses a package boundary, so no rebuild leg applies here, and the + * ablations below were run and observed without one. + * + * ## ⭐ ABLATION 1 — the staleness half (`epoch` comparison neutered) + * + * Mutation: in `meta-overlay-cache.ts`, `readMetaOverlayCache`'s + * `if (entry.epoch !== epoch) return undefined;` replaced by + * `if (false) return undefined;` — an entry is served no matter how far the + * write epoch has moved past it. + * + * PREDICTED IN WRITING BEFORE THE MUTATION: RED, exactly **3** failing cases — + * §2 "the answer after an epoch bump equals the uncached answer", §2 "a bumped + * epoch re-reads", §5 "an epoch bump retires an entry the TTL would still + * serve". Every other case either never bumps the epoch or never reaches the + * comparison. + * OBSERVED: RED, 3 failed / 19 passed — as predicted, same three cases. + * Named positive control, predicted and observed GREEN throughout: + * §3 "an engine with no write-epoch seam keeps its exact query multiset" — + * it never stores an entry, so it never reaches the neutered comparison, and + * its staying green is what shows the ablation cut the epoch check rather than + * the cache. + * + * ## ⭐ ABLATION 2 — the hit half (`writeMetaOverlayCache` call removed) + * + * Mutation: in `protocol.ts`, the `writeMetaOverlayCache(...)` call in + * `getMetaItems` commented out — the cache is read but never populated. + * + * PREDICTED IN WRITING BEFORE THE MUTATION: RED, exactly **5** failing cases — + * the five that assert a repeat issues zero reads (§1 hit, §1 identity-plus-hit, + * §4 negative caching, §5 within-TTL hit, §6 clone isolation, which needs a hit + * to have something to corrupt). That is 5 by the count of cases, and the + * prediction names them. + * OBSERVED: RED, 5 failed / 17 passed — as predicted. + * Same positive control §3, predicted and observed GREEN: a no-seam engine + * already issues its full multiset every call, so removing the store changes + * nothing for it. + * + * The two ablations failing DISJOINT sets is the point: it is what shows the + * hit assertions and the staleness assertions are testing different halves. + */ + +import { describe, expect, it } from 'vitest'; +import { ObjectStackProtocolImplementation } from './protocol.js'; +import { + META_OVERLAY_CACHE_DEFAULT_TTL_MS, + metaOverlayCacheTtlMs, + readWriteEpoch, +} from './meta-overlay-cache.js'; +import { assertEngineFindOnePredicate, type EngineFindOneQueryInput } from '@objectstack/metadata-core'; + +interface StoredRow { + id: string; + type: string; + name: string; + organization_id: string | null; + package_id: string | null; + state: string; + metadata: string; +} + +const storedRow = ( + type: string, + name: string, + extra: Partial = {}, +): StoredRow => ({ + id: `r_${type}_${name}`, + type, + name, + organization_id: null, + package_id: null, + state: 'active', + metadata: JSON.stringify({ name, label: `Label for ${name}` }), + ...extra, +}); + +/** A minimal `{ current, bump, subscribe }` seam — the shape the engine ships. */ +function makeEpochSeam() { + let epoch = 0; + const listeners = new Set<(n: number) => void>(); + return { + get current() { + return epoch; + }, + bump(): number { + epoch += 1; + for (const l of [...listeners]) l(epoch); + return epoch; + }, + subscribe(listener: (n: number) => void): () => void { + listeners.add(listener); + return () => listeners.delete(listener); + }, + }; +} + +interface HarnessOptions { + /** `undefined` (default) is the unscoped kernel — the only one that hydrates. */ + environmentId?: string; + /** Omit the seam entirely, or hand over a partial one. */ + writeEpoch?: unknown; + /** What `registry.listItems` answers, if anything. */ + registryItems?: unknown[]; +} + +/** + * The observation channels this file is about: every `sys_metadata` WHERE + * clause in call order (the query multiset), and every `registerItem` call in + * call order (the read-side registry hydration). + */ +function makeHarness(rows: StoredRow[], options: HarnessOptions = {}) { + const finds: Array> = []; + const registeredItems: Array<{ type: string; name: unknown }> = []; + + const seam = 'writeEpoch' in options ? options.writeEpoch : makeEpochSeam(); + + const engine: any = { + async find(table: string, opts?: { where?: Record; limit?: number }) { + if (table !== 'sys_metadata') return []; + const where = opts?.where ?? {}; + finds.push({ ...where }); + // `check:where-matcher` — a hand-written matcher with no combinator + // branch reads `$and` as a field name and answers the wrong + // question rather than failing. Refuse the shape this double does + // not implement, matching the sibling doubles' convention. + for (const k of Object.keys(where)) { + if (k.startsWith('$')) { + throw new Error(`[test double] unsupported WHERE combinator '${k}'`); + } + } + const matched = rows.filter((r) => + Object.entries(where).every(([k, v]) => { + if (v === undefined) return true; + return (r as unknown as Record)[k] === v; + }), + ); + // `check:objectql-double-limit` (#10978) — hold the caller's bound, + // applied AFTER the filter and BY PRESENCE. A double that hands + // back every row it matched cannot tell a dropped bound from no + // bound at all. + return opts?.limit === undefined ? matched : matched.slice(0, opts.limit); + }, + async findOne(object: string, query?: EngineFindOneQueryInput) { + assertEngineFindOnePredicate(object, query); + return null; + }, + // ⛔ No `insert` / `update` / `delete` on this double, deliberately — + // the path under test is READ-then-register and touches no write verb, + // so declaring them would add a dispatch contract + // (`check:engine-double-contract`) nothing here exercises. A write is + // simulated by advancing the seam directly, which is exactly the + // observable a real engine write produces: `executeWithMiddleware` + // calls `writeEpoch.bump('write')` ahead of the middleware chain. That + // the engine does so on every insert/update/delete is already pinned by + // `objectql/src/write-epoch.test.ts`; this file pins the CACHE's + // reaction to the bump, not the bump itself. + registry: { + registerItem: (type: string, item: any) => { + registeredItems.push({ type, name: item?.name }); + }, + registerObject: () => undefined, + listItems: () => options.registryItems ?? [], + getItem: () => undefined, + getObject: () => undefined, + getPackage: () => undefined, + getArtifactItem: () => undefined, + isPackageDisabled: () => false, + applyNavContributions: (app: unknown) => app, + }, + }; + if (seam !== undefined) engine.writeEpoch = seam; + + const protocol = new ObjectStackProtocolImplementation( + engine, + () => new Map(), + options.environmentId, + ) as any; + + return { + protocol, + finds, + registeredItems, + rows, + bumpEpoch: () => (seam as { bump(): number } | undefined)?.bump(), + }; +} + +const OVERLAY_ROWS = [storedRow('object', 'alpha'), storedRow('object', 'beta')]; +const clone = (v: T): T => structuredClone(v); + +// ═══════════════════════════════════════════════════════════════════════════ +// 1. The acceptance criterion — cached answer ≡ uncached answer, paired with +// a hit assertion so neither half passes on a degenerate cache +// ═══════════════════════════════════════════════════════════════════════════ + +describe('[#11967] §1 cached ≡ uncached, and the repeat issues zero reads', () => { + it('a repeat of the same read issues ZERO engine reads and returns an equal answer', async () => { + const h = makeHarness(clone(OVERLAY_ROWS)); + + const first = await h.protocol.getMetaItems({ type: 'object' }); + const findsAfterFirst = h.finds.length; + expect(findsAfterFirst).toBeGreaterThan(0); + + const second = await h.protocol.getMetaItems({ type: 'object' }); + + // The hit half: the repeat read nothing. + expect(h.finds.length).toBe(findsAfterFirst); + // The identity half: deep-equal INCLUDING array order. + expect(second).toEqual(first); + expect((second.items as any[]).map((i) => i.name)) + .toEqual((first.items as any[]).map((i) => i.name)); + }); + + it('the answer after an epoch bump equals the answer an UNCACHED engine gives', async () => { + // ⭐ This is #11967's stated acceptance criterion: write → epoch change + // → fresh read, and the cached answer must be the uncached answer. + const cached = makeHarness(clone(OVERLAY_ROWS)); + + await cached.protocol.getMetaItems({ type: 'object' }); + const findsBeforeWrite = cached.finds.length; + // A repeat before the write is a HIT — asserted here so the bump below + // is demonstrably retiring a LIVE entry rather than an absent one. + await cached.protocol.getMetaItems({ type: 'object' }); + expect(cached.finds.length).toBe(findsBeforeWrite); + + // The write: a new overlay row lands and the engine seam advances. + cached.rows.push(storedRow('object', 'gamma')); + cached.bumpEpoch(); + + const afterBump = await cached.protocol.getMetaItems({ type: 'object' }); + // The staleness half: a bumped epoch re-read. + expect(cached.finds.length).toBeGreaterThan(findsBeforeWrite); + + // The identity half: the same rows through an engine that never caches + // (no seam), on a fresh protocol, must give the same answer. + const uncached = makeHarness(clone(cached.rows), { writeEpoch: undefined }); + const fresh = await uncached.protocol.getMetaItems({ type: 'object' }); + + expect(afterBump).toEqual(fresh); + expect((afterBump.items as any[]).map((i) => i.name)) + .toEqual((fresh.items as any[]).map((i) => i.name)); + // ⭐ Assert the END of the chain, not the middle: the newly published + // row is PRESENT. "The cache was cleared" would pass on a + // clear-then-repopulate-from-a-stale-read implementation. + expect((afterBump.items as any[]).map((i) => i.name)).toContain('gamma'); + }); + + it('a bumped epoch re-reads even when the row set did not change', async () => { + const h = makeHarness(clone(OVERLAY_ROWS)); + const before = await h.protocol.getMetaItems({ type: 'object' }); + const findsAfterFirst = h.finds.length; + + h.bumpEpoch(); + const after = await h.protocol.getMetaItems({ type: 'object' }); + + expect(h.finds.length).toBeGreaterThan(findsAfterFirst); + expect(after).toEqual(before); + }); +}); + +// ═══════════════════════════════════════════════════════════════════════════ +// 2. ⭐ The SchemaRegistry-hydration trap (#11633 §4 leg D) +// ═══════════════════════════════════════════════════════════════════════════ +// +// The trap: `getMetaItems` registers overlay rows back into the SchemaRegistry +// as a side effect of the read, so a cache that skips the read skips the +// registration and the symptom is a registry that stops being populated — not a +// stale answer. This cache is placed UPSTREAM of the hydration branch, so a hit +// still hydrates. These cases are what make that a measurement. + +describe('[#11967] §2 a cache hit still hydrates the SchemaRegistry', () => { + it('registerItem is called on the CACHED call exactly as on the uncached one', async () => { + const h = makeHarness(clone(OVERLAY_ROWS)); // environmentId undefined ⇒ hydrates + + await h.protocol.getMetaItems({ type: 'object' }); + const afterFirst = h.registeredItems.map((r) => `${r.type}:${String(r.name)}`); + const findsAfterFirst = h.finds.length; + expect(afterFirst.length).toBeGreaterThan(0); + + await h.protocol.getMetaItems({ type: 'object' }); + + // The hit half: the second call read nothing from the engine … + expect(h.finds.length).toBe(findsAfterFirst); + // … and the trap half: it hydrated anyway, the same rows in the same + // order. A cache placed below the merge would leave this at `afterFirst`. + const afterSecond = h.registeredItems.map((r) => `${r.type}:${String(r.name)}`); + expect(afterSecond).toEqual([...afterFirst, ...afterFirst]); + }); + + it('hydration keeps running on every hit, not just the second call', async () => { + const h = makeHarness(clone(OVERLAY_ROWS)); + await h.protocol.getMetaItems({ type: 'object' }); + const perCall = h.registeredItems.length; + const findsAfterFirst = h.finds.length; + + await h.protocol.getMetaItems({ type: 'object' }); + await h.protocol.getMetaItems({ type: 'object' }); + + expect(h.finds.length).toBe(findsAfterFirst); + expect(h.registeredItems.length).toBe(perCall * 3); + }); +}); + +// ═══════════════════════════════════════════════════════════════════════════ +// 3. No seam ⇒ no cache (leg C's rule, re-measured for leg D) +// ═══════════════════════════════════════════════════════════════════════════ + +describe('[#11967] §3 a success is cached ONLY when the engine exposes the write epoch', () => { + // ⭐ NAMED POSITIVE CONTROL for both ablations — see the file header. + it('an engine with no write-epoch seam keeps its exact query multiset', async () => { + const h = makeHarness(clone(OVERLAY_ROWS), { writeEpoch: undefined }); + + await h.protocol.getMetaItems({ type: 'object' }); + const perCall = h.finds.length; + expect(perCall).toBeGreaterThan(0); + + await h.protocol.getMetaItems({ type: 'object' }); + await h.protocol.getMetaItems({ type: 'object' }); + + expect(h.finds.length).toBe(perCall * 3); + }); + + it('a partial `{ current }` object is NOT a seam and does not licence caching', async () => { + // The whole surface is checked. A bare counter on some unrelated double + // would otherwise read as a live invalidation seam and licence caching + // against something nothing ever bumps. + const h = makeHarness(clone(OVERLAY_ROWS), { writeEpoch: { current: 0 } }); + + await h.protocol.getMetaItems({ type: 'object' }); + const perCall = h.finds.length; + await h.protocol.getMetaItems({ type: 'object' }); + + expect(h.finds.length).toBe(perCall * 2); + }); + + it('readWriteEpoch accepts the full surface and refuses every partial one', () => { + expect(readWriteEpoch({ writeEpoch: makeEpochSeam() })).toBe(0); + expect(readWriteEpoch({ writeEpoch: { current: 3 } })).toBeUndefined(); + expect(readWriteEpoch({ writeEpoch: { current: 3, bump: () => 4 } })).toBeUndefined(); + expect(readWriteEpoch({ writeEpoch: { bump: () => 1, subscribe: () => () => undefined } })) + .toBeUndefined(); + expect(readWriteEpoch({})).toBeUndefined(); + expect(readWriteEpoch(null)).toBeUndefined(); + }); +}); + +// ═══════════════════════════════════════════════════════════════════════════ +// 4. Negative caching — the bulk of leg D's win (#11633 §1, §4) +// ═══════════════════════════════════════════════════════════════════════════ + +describe('[#11967] §4 the EMPTY result is cached, and that is the point', () => { + it('an empty overlay set costs two reads once and zero thereafter', async () => { + // No rows at all: the first `queryByOrg(null)` comes back empty, which + // is exactly what fires the alt-type retry — the doubled read #11633 §1 + // measured on every request of a code-authored app. + const h = makeHarness([]); + + const first = await h.protocol.getMetaItems({ type: 'object' }); + expect(h.finds.length).toBe(2); + expect(h.finds[0].type).toBe('object'); + expect(h.finds[1].type).toBe('objects'); + + const second = await h.protocol.getMetaItems({ type: 'object' }); + expect(h.finds.length).toBe(2); + expect(second).toEqual(first); + }); + + it('a newly published row appears promptly — the epoch, never the timer', async () => { + const h = makeHarness([]); + await h.protocol.getMetaItems({ type: 'object' }); + expect(h.finds.length).toBe(2); + // Paired hit assertion: the empty answer really is cached … + await h.protocol.getMetaItems({ type: 'object' }); + expect(h.finds.length).toBe(2); + + // … and a publish still shows up on the very next read, with no clock + // advance anywhere in this test. + h.rows.push(storedRow('object', 'freshly_published')); + h.bumpEpoch(); + + const after = await h.protocol.getMetaItems({ type: 'object' }); + expect((after.items as any[]).map((i) => i.name)).toContain('freshly_published'); + }); +}); + +// ═══════════════════════════════════════════════════════════════════════════ +// 5. The TTL — a real off switch, and never the primary mechanism +// ═══════════════════════════════════════════════════════════════════════════ + +describe('[#11967] §5 the TTL bounds what the epoch cannot see, and 0 is a real path', () => { + it('TTL 0 restores the pre-#11967 query multiset exactly', async () => { + const previous = process.env.OS_METADATA_OVERLAY_CACHE_TTL_MS; + process.env.OS_METADATA_OVERLAY_CACHE_TTL_MS = '0'; + try { + const h = makeHarness(clone(OVERLAY_ROWS)); + await h.protocol.getMetaItems({ type: 'object' }); + const perCall = h.finds.length; + await h.protocol.getMetaItems({ type: 'object' }); + await h.protocol.getMetaItems({ type: 'object' }); + expect(h.finds.length).toBe(perCall * 3); + } finally { + if (previous === undefined) delete process.env.OS_METADATA_OVERLAY_CACHE_TTL_MS; + else process.env.OS_METADATA_OVERLAY_CACHE_TTL_MS = previous; + } + }); + + it('an epoch bump retires an entry the TTL would still have served', async () => { + // The two bounds are independent, and this is the one that matters: + // well inside a 30s default TTL, a write still retires the entry. + const h = makeHarness(clone(OVERLAY_ROWS)); + await h.protocol.getMetaItems({ type: 'object' }); + const findsAfterFirst = h.finds.length; + // Paired hit assertion — inside the TTL, unbumped, this is a hit. + await h.protocol.getMetaItems({ type: 'object' }); + expect(h.finds.length).toBe(findsAfterFirst); + + h.bumpEpoch(); + await h.protocol.getMetaItems({ type: 'object' }); + expect(h.finds.length).toBeGreaterThan(findsAfterFirst); + }); + + it('parses the TTL env var, and folds a malformed value to OFF', () => { + expect(metaOverlayCacheTtlMs({})).toBe(META_OVERLAY_CACHE_DEFAULT_TTL_MS); + expect(metaOverlayCacheTtlMs({ OS_METADATA_OVERLAY_CACHE_TTL_MS: '' })) + .toBe(META_OVERLAY_CACHE_DEFAULT_TTL_MS); + expect(metaOverlayCacheTtlMs({ OS_METADATA_OVERLAY_CACHE_TTL_MS: '5000' })).toBe(5000); + expect(metaOverlayCacheTtlMs({ OS_METADATA_OVERLAY_CACHE_TTL_MS: '0' })).toBe(0); + // ⚠️ `3OOO` (letter O) is the case the arm exists for: folding it into + // the 30s default would hand the operator a LONGER window than the one + // they were trying to set. + expect(metaOverlayCacheTtlMs({ OS_METADATA_OVERLAY_CACHE_TTL_MS: '3OOO' })).toBe(0); + expect(metaOverlayCacheTtlMs({ OS_METADATA_OVERLAY_CACHE_TTL_MS: '-1' })).toBe(0); + }); +}); + +// ═══════════════════════════════════════════════════════════════════════════ +// 6. Aliasing — a cached row set is cloned in both directions +// ═══════════════════════════════════════════════════════════════════════════ + +describe('[#11967] §6 a caller mutating the answer cannot corrupt the cache', () => { + it('mutating a returned item leaves the next read unaffected', async () => { + const h = makeHarness(clone(OVERLAY_ROWS)); + + const first = await h.protocol.getMetaItems({ type: 'object' }); + const findsAfterFirst = h.finds.length; + const target = (first.items as any[]).find((i) => i.name === 'alpha'); + expect(target).toBeDefined(); + target.label = 'MUTATED BY A CALLER'; + (target as any).injected = true; + + const second = await h.protocol.getMetaItems({ type: 'object' }); + // Paired hit assertion: this genuinely came from the cache … + expect(h.finds.length).toBe(findsAfterFirst); + // … and it is pristine. + const again = (second.items as any[]).find((i) => i.name === 'alpha'); + expect(again.label).toBe('Label for alpha'); + expect((again as any).injected).toBeUndefined(); + }); +}); + +// ═══════════════════════════════════════════════════════════════════════════ +// 7. Key separation — one entry per (engine, type, package, org) +// ═══════════════════════════════════════════════════════════════════════════ + +describe('[#11967] §7 distinct reads never share an entry', () => { + it('a different type does not answer from another type entry', async () => { + const h = makeHarness([storedRow('object', 'alpha'), storedRow('app', 'console')]); + + const objects = await h.protocol.getMetaItems({ type: 'object' }); + const findsAfterObjects = h.finds.length; + const apps = await h.protocol.getMetaItems({ type: 'app' }); + + expect(h.finds.length).toBeGreaterThan(findsAfterObjects); + expect((objects.items as any[]).map((i) => i.name)).toEqual(['alpha']); + expect((apps.items as any[]).map((i) => i.name)).toEqual(['console']); + }); + + it('a packageId-scoped read does not answer from the unscoped entry', async () => { + const h = makeHarness([ + storedRow('object', 'alpha'), + storedRow('object', 'beta', { package_id: 'pkg_b', id: 'r_object_beta_pkg' }), + ]); + + await h.protocol.getMetaItems({ type: 'object' }); + const findsAfterUnscoped = h.finds.length; + + const scoped = await h.protocol.getMetaItems({ type: 'object', packageId: 'pkg_b' }); + expect(h.finds.length).toBeGreaterThan(findsAfterUnscoped); + expect((scoped.items as any[]).map((i) => i.name)).toEqual(['beta']); + }); + + it('an org-scoped read does not answer from the env-wide entry', async () => { + const h = makeHarness([ + storedRow('object', 'alpha'), + storedRow('object', 'org_only', { organization_id: 'org_1', id: 'r_object_org' }), + ]); + + const envWide = await h.protocol.getMetaItems({ type: 'object' }); + const findsAfterEnvWide = h.finds.length; + + const orgScoped = await h.protocol.getMetaItems({ type: 'object', organizationId: 'org_1' }); + expect(h.finds.length).toBeGreaterThan(findsAfterEnvWide); + + expect((envWide.items as any[]).map((i) => i.name)).toEqual(['alpha']); + expect((orgScoped.items as any[]).map((i) => i.name).sort()) + .toEqual(['alpha', 'org_only']); + }); + + it('two engines never share a bucket', async () => { + const a = makeHarness([storedRow('object', 'from_engine_a')]); + const b = makeHarness([storedRow('object', 'from_engine_b')]); + + const fromA = await a.protocol.getMetaItems({ type: 'object' }); + const fromB = await b.protocol.getMetaItems({ type: 'object' }); + + expect((fromA.items as any[]).map((i) => i.name)).toEqual(['from_engine_a']); + expect((fromB.items as any[]).map((i) => i.name)).toEqual(['from_engine_b']); + }); +}); + +// ═══════════════════════════════════════════════════════════════════════════ +// 8. The scoped kernel — cached the same way, and provably not hydrating +// ═══════════════════════════════════════════════════════════════════════════ + +describe('[#11967] §8 a scoped (environment) kernel caches without hydrating', () => { + it('caches the overlay read and registers nothing', async () => { + const h = makeHarness(clone(OVERLAY_ROWS), { environmentId: 'env_1' }); + + const first = await h.protocol.getMetaItems({ type: 'object' }); + const findsAfterFirst = h.finds.length; + const second = await h.protocol.getMetaItems({ type: 'object' }); + + expect(h.finds.length).toBe(findsAfterFirst); + expect(second).toEqual(first); + // The hydration limb is gated to unscoped kernels, so this stays empty + // on BOTH calls — the cache did not change which limb runs. + expect(h.registeredItems).toEqual([]); + }); +}); From 19f22d27b8d298e2b9c7f77d07eb376b9958cbce Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 13:03:59 +0000 Subject: [PATCH 3/8] test(metadata-protocol): record both ablation predictions BEFORE mutating --- .../src/meta-overlay-cache.test.ts | 43 ++++++++++--------- 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/packages/metadata-protocol/src/meta-overlay-cache.test.ts b/packages/metadata-protocol/src/meta-overlay-cache.test.ts index 3d5eaed29c..c5084f0c67 100644 --- a/packages/metadata-protocol/src/meta-overlay-cache.test.ts +++ b/packages/metadata-protocol/src/meta-overlay-cache.test.ts @@ -31,32 +31,33 @@ * `if (false) return undefined;` — an entry is served no matter how far the * write epoch has moved past it. * - * PREDICTED IN WRITING BEFORE THE MUTATION: RED, exactly **3** failing cases — - * §2 "the answer after an epoch bump equals the uncached answer", §2 "a bumped - * epoch re-reads", §5 "an epoch bump retires an entry the TTL would still - * serve". Every other case either never bumps the epoch or never reaches the - * comparison. - * OBSERVED: RED, 3 failed / 19 passed — as predicted, same three cases. - * Named positive control, predicted and observed GREEN throughout: - * §3 "an engine with no write-epoch seam keeps its exact query multiset" — - * it never stores an entry, so it never reaches the neutered comparison, and - * its staying green is what shows the ablation cut the epoch check rather than - * the cache. + * PREDICTED IN WRITING BEFORE THE MUTATION (committed ahead of it): RED, + * exactly **4** failing cases, named — + * 1. §1 "the answer after an epoch bump equals the answer an UNCACHED engine gives" + * 2. §1 "a bumped epoch re-reads even when the row set did not change" + * 3. §4 "a newly published row appears promptly — the epoch, never the timer" + * 4. §5 "an epoch bump retires an entry the TTL would still have served" + * Every other case either never bumps the epoch or never reaches the comparison. + * OBSERVED: __ABL1_OBSERVED__ * * ## ⭐ ABLATION 2 — the hit half (`writeMetaOverlayCache` call removed) * * Mutation: in `protocol.ts`, the `writeMetaOverlayCache(...)` call in - * `getMetaItems` commented out — the cache is read but never populated. + * `getMetaItems` replaced by a no-op — the cache is read but never populated, + * i.e. a cache that never caches. * - * PREDICTED IN WRITING BEFORE THE MUTATION: RED, exactly **5** failing cases — - * the five that assert a repeat issues zero reads (§1 hit, §1 identity-plus-hit, - * §4 negative caching, §5 within-TTL hit, §6 clone isolation, which needs a hit - * to have something to corrupt). That is 5 by the count of cases, and the - * prediction names them. - * OBSERVED: RED, 5 failed / 17 passed — as predicted. - * Same positive control §3, predicted and observed GREEN: a no-seam engine - * already issues its full multiset every call, so removing the store changes - * nothing for it. + * PREDICTED IN WRITING BEFORE THE MUTATION (committed ahead of it): RED, + * exactly **9** failing cases — every case carrying a "the repeat read nothing" + * assertion: §1 (2 of 3), §2 (both), §4 (both), §5 (1 of 3), §6, §8. The three + * §7 key-separation cases and §1's "a bumped epoch re-reads" use + * `toBeGreaterThan` and stay green, as do all of §3. + * OBSERVED: __ABL2_OBSERVED__ + * + * Named positive control for BOTH ablations, predicted GREEN throughout: + * §3 "an engine with no write-epoch seam keeps its exact query multiset". It + * never stores an entry, so ablation 1 never reaches its neutered comparison + * and ablation 2 removes a store it never made. Its staying green is what shows + * each ablation cut the intended half rather than the cache as a whole. * * The two ablations failing DISJOINT sets is the point: it is what shows the * hit assertions and the staleness assertions are testing different halves. From d2b760ae9685885960281fca274b350714868311 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 13:06:54 +0000 Subject: [PATCH 4/8] feat(metadata-protocol): leg D overlay cache + env var + changeset --- .changeset/metadata-overlay-read-cache.md | 52 +++++++++++++++++++ .../docs/deployment/environment-variables.mdx | 1 + .../src/meta-overlay-cache.test.ts | 26 ++++++++-- 3 files changed, 75 insertions(+), 4 deletions(-) create mode 100644 .changeset/metadata-overlay-read-cache.md diff --git a/.changeset/metadata-overlay-read-cache.md b/.changeset/metadata-overlay-read-cache.md new file mode 100644 index 0000000000..286b27908a --- /dev/null +++ b/.changeset/metadata-overlay-read-cache.md @@ -0,0 +1,52 @@ +--- +"@objectstack/metadata-protocol": minor +--- + +feat(metadata-protocol): cache the `getMetaItems` overlay read, keyed on the engine write epoch (#11967) + +Leg D (ship-second) of the accepted #11633 cross-request caching design +(maintainer acceptance 2026-08-25, forks 1A / 2B / 3A / TTL-0). + +`getMetaItems` re-read `sys_metadata` on every authenticated request. On the hot +path — `enforceApiAccess` → `loadObjectItems` → `getMetaItems({ type: 'object' })`, +once per REST request — that costs **two** sequentially awaited engine queries +whenever the environment holds no overlay rows for the type, because the empty +first result fires the alt-type retry. An app whose objects are all code-authored +paid both on every request. That read is now cached behind invalidation that is +synchronous and in-process rather than TTL-bound. + +- **Primary trigger — the #11968 engine write epoch**, read structurally rather + than imported: `@objectstack/metadata-protocol` does not depend on + `@objectstack/objectql`, and the substrate declared `WriteEpochLike` separately + for exactly this kind of consumer. Every `sys_metadata` write reaches it, because + `SysMetadataRepository` writes through `engine.insert/update/delete`. +- **Residual bound — `OS_METADATA_OVERLAY_CACHE_TTL_MS`**, default 30s, `0` = off + and a real path. It bounds one thing only: a peer replica's write on a deployment + with no `authz.invalidated` bridge attached. +- **A success is cached ONLY when the engine exposes the write epoch** — leg C's + rule, re-measured and unchanged here. No seam means the cache declines rather + than degrading to a TTL-only shape, which for this leg would make + publish-visibility a timer. Every existing test double keeps its exact query + multiset; only a real engine caches. + +**Grade: `minor`, argued.** Not `patch`: this adds a supported deployment knob +(`OS_METADATA_OVERLAY_CACHE_TTL_MS`, registered in the canonical environment-variable +table) and introduces a bounded staleness window that a multi-node operator must be +able to read about before upgrading — a release note that said only "internal +performance" would under-describe it. Not `major`: no public API changes, no export +is added or removed, `getMetaItems`' request and response contracts are untouched, +and the pins assert the cached answer is identical to the uncached one, so no caller +can observe the difference except in query count. Same grade and same reasoning as +leg C (#11966), which shipped `minor` for the same knob-plus-window shape. + +**The SchemaRegistry-hydration trap (#11633 §4 leg D) is resolved structurally, not +by care.** `getMetaItems` registers overlay rows back into the SchemaRegistry as a +side effect of the read, so a cache that skips the read would quietly stop populating +the registry. What is cached here is the overlay **row set** — the value *upstream* of +the hydration branch — never the merged answer downstream of it. Every consumer of +those rows still runs on every call, hit or miss: the overlay parse, the package-aware +merge, `hydrateOverlayIntoRegistry`, the MetadataService merge, the disabled-package +filter, the nav contributions and the decorations. That containment also keeps the +cache's reach co-extensive with what its key can validate — the SchemaRegistry, the +MetadataService and the artifact table are all mutable sources the write epoch cannot +observe, and none of them is being cached. diff --git a/content/docs/deployment/environment-variables.mdx b/content/docs/deployment/environment-variables.mdx index 09a1daf330..a3e543da4b 100644 --- a/content/docs/deployment/environment-variables.mdx +++ b/content/docs/deployment/environment-variables.mdx @@ -360,6 +360,7 @@ the hosted ObjectOS Cloud control plane. | `OS_SANDBOX_ACTION_TIMEOUT_MS` | number | `5000` | Default **CPU-time** budget for a sandboxed **action** body (QuickJS). Same resolution rules as the hook variant above (positive integer only; an action body's own `timeoutMs` still wins). | | `OS_SANDBOX_WALL_CEILING_MS` | number | `30000` | Wall-clock ceiling (ADR-0102) — the backstop that cuts a hook/action body stuck on a host call that never settles (which burns no CPU, so the CPU budget alone would never fire). The effective ceiling is `max(this, cpuBudget)`, so it can never cut a body still inside its CPU budget. Positive integer only; unset keeps 30s. | | `OS_LOCALIZATION_CACHE_TTL_MS` | number | `30000` | Staleness bound, in milliseconds, for the cross-request cache of a workspace's reference localization (`timezone` / `locale` / `currency`, read from `sys_setting`) — leg C of #11633. `0` means **off**, a real path that restores the uncached query pattern exactly. Unlike `OS_AUTHZ_GRANTS_CACHE_TTL_MS` (which is off by default) this one ships **on**, because its invalidation is synchronous and in-process rather than TTL-bound: a `localization` settings change and any engine write both retire a cached answer immediately, so the TTL only bounds what neither seam can see — a write made on another replica with no `authz.invalidated` bridge attached. ⚠️ A malformed value reads as `0` (off), the opposite arm from the grants variable and deliberately so: there `0` is also the default, whereas here folding `3OOO` (letter O) into the default would hand you a **longer** window than the one you were setting. Deployment config only — never a settings row, because `sys_setting` is the table this cache caches. | +| `OS_METADATA_OVERLAY_CACHE_TTL_MS` | number | `30000` | Staleness bound, in milliseconds, for the cross-request cache of the `sys_metadata` overlay read inside `getMetaItems` — leg D of #11633. `0` means **off**, a real path that restores the uncached query pattern exactly. Ships **on**, for the same reason as `OS_LOCALIZATION_CACHE_TTL_MS`: invalidation is synchronous and in-process, because every `sys_metadata` write goes through the engine and so advances the write epoch that retires the entry before the next read. What is cached is the overlay ROW SET only — never the merged answer — so the SchemaRegistry, the MetadataService and the artifact table are re-consulted on every call, cached or not, and the read-side registry hydration keeps running on a cache hit. The TTL therefore bounds one thing: a write made on **another replica** with no `authz.invalidated` bridge attached. ⚠️ A malformed value reads as `0` (off) — same arm and same reason as `OS_LOCALIZATION_CACHE_TTL_MS`. Deployment config only, never a settings row. | | `OS_INLINE_SEED_BUDGET_MS` | number | `8000` | Time budget for synchronous seed execution at boot before deferring to a worker. | | `OS_TENANT_AUDIT` | flag | `1` | Set to `0` to silence the tenant-isolation audit warnings emitted by the SQL driver. | diff --git a/packages/metadata-protocol/src/meta-overlay-cache.test.ts b/packages/metadata-protocol/src/meta-overlay-cache.test.ts index c5084f0c67..6821b4641d 100644 --- a/packages/metadata-protocol/src/meta-overlay-cache.test.ts +++ b/packages/metadata-protocol/src/meta-overlay-cache.test.ts @@ -38,7 +38,10 @@ * 3. §4 "a newly published row appears promptly — the epoch, never the timer" * 4. §5 "an epoch bump retires an entry the TTL would still have served" * Every other case either never bumps the epoch or never reaches the comparison. - * OBSERVED: __ABL1_OBSERVED__ + * OBSERVED: RED, **4** failed / 15 passed (19) — the prediction's exact count AND its + * exact named set, in order: §1 "…equals the answer an UNCACHED engine gives", + * §1 "a bumped epoch re-reads…", §4 "a newly published row appears promptly", + * §5 "an epoch bump retires an entry the TTL would still have served". * * ## ⭐ ABLATION 2 — the hit half (`writeMetaOverlayCache` call removed) * @@ -51,7 +54,10 @@ * assertion: §1 (2 of 3), §2 (both), §4 (both), §5 (1 of 3), §6, §8. The three * §7 key-separation cases and §1's "a bumped epoch re-reads" use * `toBeGreaterThan` and stay green, as do all of §3. - * OBSERVED: __ABL2_OBSERVED__ + * OBSERVED: RED, **9** failed / 10 passed (19) — the prediction's exact count AND its + * exact named set: §1 "a repeat … issues ZERO engine reads", §1 "…equals the + * answer an UNCACHED engine gives", §2 both, §4 both, §5 "an epoch bump + * retires…", §6, §8. * * Named positive control for BOTH ablations, predicted GREEN throughout: * §3 "an engine with no write-epoch seam keeps its exact query multiset". It @@ -59,8 +65,20 @@ * and ablation 2 removes a store it never made. Its staying green is what shows * each ablation cut the intended half rather than the cache as a whole. * - * The two ablations failing DISJOINT sets is the point: it is what shows the - * hit assertions and the staleness assertions are testing different halves. + * ⭐ The two failure sets are what shows the halves are independent. They + * overlap on exactly THREE cases — §1 "…equals the answer an UNCACHED engine + * gives", §4 "a newly published row appears promptly", §5 "an epoch bump + * retires…" — and that overlap is not slack: those are precisely the cases + * written to carry BOTH a hit assertion and a staleness assertion, so each + * ablation kills a different assertion inside the same case. Outside the + * overlap the sets are disjoint: ablation 1 alone takes §1 "a bumped epoch + * re-reads", ablation 2 alone takes §1 "a repeat … issues ZERO reads", both of + * §2, §4 "an empty overlay set costs two reads once", §6 and §8. + * + * ⭐ Both ablations also MEASURE the source-resolution claim above rather than + * restating it: each mutated only this package's source, ran vitest with NO + * rebuild of any kind, and the behaviour changed on that run. A dist-mediated + * test path would have stayed green through both. */ import { describe, expect, it } from 'vitest'; From 0cf0c218a7513db2205ae03200f83a515282aad1 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 13:23:27 +0000 Subject: [PATCH 5/8] test(metadata-protocol): drop the unexercised findOne double instead of growing the pinned ledger --- .../src/meta-overlay-cache.test.ts | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/packages/metadata-protocol/src/meta-overlay-cache.test.ts b/packages/metadata-protocol/src/meta-overlay-cache.test.ts index 6821b4641d..3e66d8232e 100644 --- a/packages/metadata-protocol/src/meta-overlay-cache.test.ts +++ b/packages/metadata-protocol/src/meta-overlay-cache.test.ts @@ -88,7 +88,6 @@ import { metaOverlayCacheTtlMs, readWriteEpoch, } from './meta-overlay-cache.js'; -import { assertEngineFindOnePredicate, type EngineFindOneQueryInput } from '@objectstack/metadata-core'; interface StoredRow { id: string; @@ -181,14 +180,13 @@ function makeHarness(rows: StoredRow[], options: HarnessOptions = {}) { // bound at all. return opts?.limit === undefined ? matched : matched.slice(0, opts.limit); }, - async findOne(object: string, query?: EngineFindOneQueryInput) { - assertEngineFindOnePredicate(object, query); - return null; - }, - // ⛔ No `insert` / `update` / `delete` on this double, deliberately — - // the path under test is READ-then-register and touches no write verb, - // so declaring them would add a dispatch contract - // (`check:engine-double-contract`) nothing here exercises. A write is + // ⛔ No `findOne` / `insert` / `update` / `delete` on this double, + // deliberately — + // the overlay path under test issues exactly one verb — `find` — so + // declaring any other would add a dispatch contract + // (`check:engine-double-contract`) no case here exercises. An + // unexercised double is a pin that cannot fail, and the pinned ledger + // would have to learn a double that protects nothing. A write is // simulated by advancing the seam directly, which is exactly the // observable a real engine write produces: `executeWithMiddleware` // calls `writeEpoch.bump('write')` ahead of the middleware chain. That From 5dabef3cd19abc736004bac017b517793971b73c Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 13:49:36 +0000 Subject: [PATCH 6/8] test(metadata-protocol): bound cache storage; re-predict both ablations BEFORE re-measuring --- .../src/meta-overlay-cache.test.ts | 53 ++++++++++++++----- .../src/meta-overlay-cache.ts | 53 +++++++++++++++++-- 2 files changed, 88 insertions(+), 18 deletions(-) diff --git a/packages/metadata-protocol/src/meta-overlay-cache.test.ts b/packages/metadata-protocol/src/meta-overlay-cache.test.ts index 3e66d8232e..f095f4a6d0 100644 --- a/packages/metadata-protocol/src/meta-overlay-cache.test.ts +++ b/packages/metadata-protocol/src/meta-overlay-cache.test.ts @@ -32,16 +32,16 @@ * write epoch has moved past it. * * PREDICTED IN WRITING BEFORE THE MUTATION (committed ahead of it): RED, - * exactly **4** failing cases, named — + * exactly **5** failing cases, named — * 1. §1 "the answer after an epoch bump equals the answer an UNCACHED engine gives" * 2. §1 "a bumped epoch re-reads even when the row set did not change" * 3. §4 "a newly published row appears promptly — the epoch, never the timer" * 4. §5 "an epoch bump retires an entry the TTL would still have served" + * 5. §9 "does not accumulate one never-evicted entry per requested key" — the + * post-bump read would HIT instead of missing, so the store that performs + * the eviction never runs and the count stays at 2. * Every other case either never bumps the epoch or never reaches the comparison. - * OBSERVED: RED, **4** failed / 15 passed (19) — the prediction's exact count AND its - * exact named set, in order: §1 "…equals the answer an UNCACHED engine gives", - * §1 "a bumped epoch re-reads…", §4 "a newly published row appears promptly", - * §5 "an epoch bump retires an entry the TTL would still have served". + * OBSERVED: __A1__ * * ## ⭐ ABLATION 2 — the hit half (`writeMetaOverlayCache` call removed) * @@ -50,14 +50,12 @@ * i.e. a cache that never caches. * * PREDICTED IN WRITING BEFORE THE MUTATION (committed ahead of it): RED, - * exactly **9** failing cases — every case carrying a "the repeat read nothing" - * assertion: §1 (2 of 3), §2 (both), §4 (both), §5 (1 of 3), §6, §8. The three - * §7 key-separation cases and §1's "a bumped epoch re-reads" use - * `toBeGreaterThan` and stay green, as do all of §3. - * OBSERVED: RED, **9** failed / 10 passed (19) — the prediction's exact count AND its - * exact named set: §1 "a repeat … issues ZERO engine reads", §1 "…equals the - * answer an UNCACHED engine gives", §2 both, §4 both, §5 "an epoch bump - * retires…", §6, §8. + * exactly **10** failing cases — every case carrying a "the repeat read + * nothing" assertion: §1 (2 of 3), §2 (both), §4 (both), §5 (1 of 3), §6, §8, + * and §9 (nothing is ever stored, so its entry count is 0, not 2). The three §7 + * key-separation cases and §1's "a bumped epoch re-reads" use `toBeGreaterThan` + * and stay green, as do all of §3. + * OBSERVED: __A2__ * * Named positive control for BOTH ablations, predicted GREEN throughout: * §3 "an engine with no write-epoch seam keeps its exact query multiset". It @@ -85,6 +83,7 @@ import { describe, expect, it } from 'vitest'; import { ObjectStackProtocolImplementation } from './protocol.js'; import { META_OVERLAY_CACHE_DEFAULT_TTL_MS, + metaOverlayCacheEntryCount, metaOverlayCacheTtlMs, readWriteEpoch, } from './meta-overlay-cache.js'; @@ -217,6 +216,7 @@ function makeHarness(rows: StoredRow[], options: HarnessOptions = {}) { return { protocol, + engine, finds, registeredItems, rows, @@ -576,3 +576,30 @@ describe('[#11967] §8 a scoped (environment) kernel caches without hydrating', expect(h.registeredItems).toEqual([]); }); }); + +// ═══════════════════════════════════════════════════════════════════════════ +// 9. Storage is bounded — a new epoch drops the entries it just orphaned +// ═══════════════════════════════════════════════════════════════════════════ + +describe('[#11967] §9 a write at a new epoch evicts the entries it orphaned', () => { + it('does not accumulate one never-evicted entry per requested key', async () => { + const h = makeHarness([storedRow('object', 'alpha'), storedRow('app', 'console')]); + + await h.protocol.getMetaItems({ type: 'object' }); + await h.protocol.getMetaItems({ type: 'app' }); + expect(metaOverlayCacheEntryCount(h.engine)).toBe(2); + + // Paired hit assertion — both entries are live, so the drop below is + // demonstrably removing real entries rather than an empty bucket. + const findsBefore = h.finds.length; + await h.protocol.getMetaItems({ type: 'object' }); + await h.protocol.getMetaItems({ type: 'app' }); + expect(h.finds.length).toBe(findsBefore); + + h.bumpEpoch(); + // One read at the new epoch is enough: every entry from the old one is + // already dead by the read-side rule, so the bucket is dropped whole. + await h.protocol.getMetaItems({ type: 'app' }); + expect(metaOverlayCacheEntryCount(h.engine)).toBe(1); + }); +}); diff --git a/packages/metadata-protocol/src/meta-overlay-cache.ts b/packages/metadata-protocol/src/meta-overlay-cache.ts index 56bd1826bf..e4339d9612 100644 --- a/packages/metadata-protocol/src/meta-overlay-cache.ts +++ b/packages/metadata-protocol/src/meta-overlay-cache.ts @@ -146,7 +146,24 @@ interface MetaOverlayCacheEntry { * never share, which is what keeps two environments in one process from seeing * each other's rows. */ -const metaOverlayCache = new WeakMap>(); +interface MetaOverlayCacheBucket { + /** + * The epoch every entry in {@link entries} was read at. The epoch is + * process-wide and object-agnostic (any write to any object advances it), so + * when it moves EVERY entry here is stale at once — Fork 1 → A's coarse + * invalidation, applied to storage and not only to validity. + * + * ⚠️ The per-entry `epoch` comparison in {@link readMetaOverlayCache} remains + * the VALIDITY rule and is not redundant with this. Entries are dropped in + * {@link writeMetaOverlayCache}, which only runs on a miss, so between a write + * and the next miss a stale entry is still present and it is the read-side + * comparison — nothing else — that refuses to serve it. + */ + epoch: number; + entries: Map; +} + +const metaOverlayCache = new WeakMap(); export const META_OVERLAY_CACHE_TTL_ENV = 'OS_METADATA_OVERLAY_CACHE_TTL_MS'; export const META_OVERLAY_CACHE_DEFAULT_TTL_MS = 30_000; @@ -233,13 +250,28 @@ export function readMetaOverlayCache( ): unknown[] | undefined { if (epoch === undefined || ttlMs <= 0) return undefined; if (!engine || typeof engine !== 'object') return undefined; - const entry = metaOverlayCache.get(engine as object)?.get(cacheKeyOf(key)); + const entry = metaOverlayCache.get(engine as object)?.entries.get(cacheKeyOf(key)); if (!entry) return undefined; if (entry.epoch !== epoch) return undefined; if (entry.expiresAt <= now) return undefined; return cloneRecords(entry.records); } +/** + * How many entries this engine's bucket currently holds. Diagnostics and pins + * only — the eviction in {@link writeMetaOverlayCache} has no behavioural + * signature (a stale entry is refused by the read-side epoch rule whether or + * not it was evicted), so without an observation channel it would be an + * unpinned optimisation, which is the kind that silently regresses. + * + * ⛔ Package-internal: this module is deliberately NOT re-exported from + * `src/index.ts`, so nothing here is public surface. + */ +export function metaOverlayCacheEntryCount(engine: unknown): number { + if (!engine || typeof engine !== 'object') return 0; + return metaOverlayCache.get(engine as object)?.entries.size ?? 0; +} + /** * Remember `records` for `key`. A no-op when this engine exposes no write * epoch, when the TTL is off, or when the rows do not clone. @@ -262,7 +294,18 @@ export function writeMetaOverlayCache( if (!engine || typeof engine !== 'object') return; const snapshot = cloneRecords(records); if (snapshot === undefined) return; - const bucket = metaOverlayCache.get(engine as object) ?? new Map(); - bucket.set(cacheKeyOf(key), { records: snapshot, epoch, expiresAt: now + ttlMs }); - metaOverlayCache.set(engine as object, bucket); + let bucket = metaOverlayCache.get(engine as object); + if (bucket === undefined) { + bucket = { epoch, entries: new Map() }; + metaOverlayCache.set(engine as object, bucket); + } else if (bucket.epoch !== epoch) { + // ⭐ Every entry read at an older epoch is already dead by the read-side + // rule, so keeping it costs memory and buys nothing. Without this a + // long-lived process accumulates one never-evicted entry per distinct + // `(type, packageId, organizationId)` ever requested — bounded in principle + // by the tenant count, which is not a bound worth shipping. + bucket.entries.clear(); + bucket.epoch = epoch; + } + bucket.entries.set(cacheKeyOf(key), { records: snapshot, epoch, expiresAt: now + ttlMs }); } From c4be8da2f1103b2596a3590ba658adf940e273e5 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 13:50:55 +0000 Subject: [PATCH 7/8] test(metadata-protocol): record both re-measured ablation outcomes --- .../src/meta-overlay-cache.test.ts | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/packages/metadata-protocol/src/meta-overlay-cache.test.ts b/packages/metadata-protocol/src/meta-overlay-cache.test.ts index f095f4a6d0..6a539e524e 100644 --- a/packages/metadata-protocol/src/meta-overlay-cache.test.ts +++ b/packages/metadata-protocol/src/meta-overlay-cache.test.ts @@ -41,7 +41,8 @@ * post-bump read would HIT instead of missing, so the store that performs * the eviction never runs and the count stays at 2. * Every other case either never bumps the epoch or never reaches the comparison. - * OBSERVED: __A1__ + * OBSERVED: RED, **5** failed / 15 passed (20) — the prediction's exact count AND its + * exact named set, §9 included. * * ## ⭐ ABLATION 2 — the hit half (`writeMetaOverlayCache` call removed) * @@ -55,7 +56,8 @@ * and §9 (nothing is ever stored, so its entry count is 0, not 2). The three §7 * key-separation cases and §1's "a bumped epoch re-reads" use `toBeGreaterThan` * and stay green, as do all of §3. - * OBSERVED: __A2__ + * OBSERVED: RED, **10** failed / 10 passed (20) — the prediction's exact count AND its + * exact named set, §9 included. * * Named positive control for BOTH ablations, predicted GREEN throughout: * §3 "an engine with no write-epoch seam keeps its exact query multiset". It @@ -64,14 +66,14 @@ * each ablation cut the intended half rather than the cache as a whole. * * ⭐ The two failure sets are what shows the halves are independent. They - * overlap on exactly THREE cases — §1 "…equals the answer an UNCACHED engine + * overlap on exactly FOUR cases — §1 "…equals the answer an UNCACHED engine * gives", §4 "a newly published row appears promptly", §5 "an epoch bump - * retires…" — and that overlap is not slack: those are precisely the cases - * written to carry BOTH a hit assertion and a staleness assertion, so each - * ablation kills a different assertion inside the same case. Outside the + * retires…" and §9 — and that overlap is not slack: those are precisely the + * cases written to carry BOTH a hit assertion and a staleness assertion, so + * each ablation kills a DIFFERENT assertion inside the same case. Outside the * overlap the sets are disjoint: ablation 1 alone takes §1 "a bumped epoch - * re-reads", ablation 2 alone takes §1 "a repeat … issues ZERO reads", both of - * §2, §4 "an empty overlay set costs two reads once", §6 and §8. + * re-reads", while ablation 2 alone takes §1 "a repeat … issues ZERO reads", + * both of §2, §4 "an empty overlay set costs two reads once", §6 and §8. * * ⭐ Both ablations also MEASURE the source-resolution claim above rather than * restating it: each mutated only this package's source, ran vitest with NO From 4482c848ab605ee200d1a364e3f1d7b3112c7215 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 14:19:46 +0000 Subject: [PATCH 8/8] docs(concepts): bound the cross-replica overlay re-read claim the leg-D cache moves --- content/docs/concepts/metadata-lifecycle.mdx | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/content/docs/concepts/metadata-lifecycle.mdx b/content/docs/concepts/metadata-lifecycle.mdx index 1725a24f91..36cfd15bbb 100644 --- a/content/docs/concepts/metadata-lifecycle.mdx +++ b/content/docs/concepts/metadata-lifecycle.mdx @@ -183,6 +183,19 @@ The hash is `sha256:` + 64-hex of a canonical (sorted-keys, no-undefined) JSON s > configuring a distributed driver; and if no `cluster` (or `metadata`) service is > registered the bridge logs and skips, leaving each replica seeing only its own > writes. +> +> ⚠️ **"Peers re-read from the shared database" has one bound worth naming.** The +> `sys_metadata` overlay read inside `protocol.ts:getMetaItems` is cached across +> requests, keyed on the engine write epoch. A **local** write advances that epoch +> before its own middleware chain runs, so read-your-writes is exact on the node +> that made the write. A **peer's** write does not: `metadata.changed` invalidates +> the MetadataManager caches this note is about, but it does not retire the +> overlay-read cache. What retires that on a peer is the `authz.invalidated` +> channel — a hint from another node bumps the local write epoch — or, failing +> that, `OS_METADATA_OVERLAY_CACHE_TTL_MS` (default 30s, `0` disables the cache +> outright). So on a deployment with no distributed cluster driver attached, a +> peer's overlay re-read can lag a remote publish by up to that TTL. See +> [Environment variables](/docs/deployment/environment-variables). ---