diff --git a/.changeset/adr-0104-attestation-adr-note.md b/.changeset/adr-0104-attestation-adr-note.md new file mode 100644 index 0000000000..27cba21432 --- /dev/null +++ b/.changeset/adr-0104-attestation-adr-note.md @@ -0,0 +1,4 @@ +--- +--- + +docs(adr-0104): record the fresh-datastore attestation seam as implemented — driver-reported created-vs-found table counts, not "sys_migration is missing" (which is the 16→17 trap) — and note that born-attested dogfood now exercises the #3459 collection path on every boot. Releases nothing. diff --git a/.changeset/adr-0104-fresh-datastore-attestation.md b/.changeset/adr-0104-fresh-datastore-attestation.md new file mode 100644 index 0000000000..589133e9cf --- /dev/null +++ b/.changeset/adr-0104-fresh-datastore-attestation.md @@ -0,0 +1,47 @@ +--- +"@objectstack/spec": minor +"@objectstack/platform-objects": minor +"@objectstack/objectql": minor +"@objectstack/driver-memory": minor +"@objectstack/driver-sql": minor +"@objectstack/service-storage": minor +--- + +feat(migrate): a datastore created from empty attests its data migrations at creation (#3438, ADR-0104 2026-07-30 addendum) + +Deployment-level migration flags could only be recorded by running +`os migrate`. That left a hole at the other end of a deployment's life: a +database created on a version that already ships the migrations started **lax** +and stayed lax until someone thought to run a command that, for them, converts +nothing and finds nothing. Every new deployment re-entered the warn regime, so +the warn regime would never die out — and, since #3459, every new deployment +also kept every released file forever. + +A store the platform **creates from empty** now records +`adr-0104-file-references` and `adr-0104-value-shapes` at that moment. Nothing +to run; enforcement and collection are live from the first boot. + +**This is not version-gating in disguise.** The fact recorded — no legacy value +is stored here — is *observed*: the store had no history at all. The platform +attests only what it watched itself create, and the test is deliberately +strict: every table made by this boot and **none found already present**. One +pre-existing table anywhere, one datasource that was already there, one driver +that cannot account for its schema sync — any of those and the deployment +attests nothing and produces its evidence by scan, exactly as before. "Found +empty" and "created empty" are not the same claim, and only the second is an +observation. + +**New surfaces.** `IDataDriver.getSchemaSyncStats?()` (optional, purely +observational: tables created vs found since connect — implemented by the SQL +and in-memory drivers), `engine.wasDatastoreCreatedFromEmpty()`, +`attestFreshDatastore()` in `@objectstack/platform-objects/system`, and +`VALUE_SHAPES_MIGRATION_ID` / `CREATION_ATTESTED_MIGRATION_IDS` in +`@objectstack/spec/system`. Attestation never overwrites an existing flag row +and never throws into a boot: a failure leaves the deployment lax, which a +migration run can still fix. + +**Upgrading changes nothing for an existing database.** It is non-empty when +the platform reaches it, so it is never attested — run +`os migrate files-to-references --apply` as before. Importing legacy values +into an attested deployment is rejected loudly at the write path; +`OS_ALLOW_LAX_MEDIA_VALUES=1` re-opens leniency while you diagnose. diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index 2f51c1db70..711a8cf0ed 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -675,6 +675,25 @@ delete check re-reads the flag fresh, so a later failing run stops collection without a restart. +#### A database created by this version needs no migration + +A deployment whose database the platform **creates from empty** records these +flags at that moment, so it is enforcing from its first boot and never enters +the warn regime at all. Nothing to run: the fact a migration would establish — +no legacy value is stored here — is already settled by the store having no +history. + +The platform attests this only for a store it watched itself create: every +table made by that first boot, none found already present. A database that +existed before — an upgrade, a restore, a store shared with anything else — +attests nothing and produces its evidence by running the command, because +"found empty" and "created empty" are not the same claim. + +Importing legacy values into such a deployment is rejected at the write path +rather than silently accepted. That is the intended outcome; if you must admit +them temporarily, `OS_ALLOW_LAX_MEDIA_VALUES=1` re-opens leniency, and +re-running the migration re-establishes the flag from the data itself. + ### Scaffolding | Command | Alias | Description | diff --git a/content/docs/releases/v17.mdx b/content/docs/releases/v17.mdx index 203d7d0ef0..81cd8d96f1 100644 --- a/content/docs/releases/v17.mdx +++ b/content/docs/releases/v17.mdx @@ -910,6 +910,13 @@ records** — never the platform version — is what authorises irreversible behaviour, and media value shapes enforce only once *this* deployment has verified its own migration. +**A database created by 17 attests both flags at creation** (#3438), so a new +deployment enforces from its first boot instead of waiting for someone to run a +migration that, for an empty store, does nothing. The platform attests only a +store it watched itself create — every table made by that boot, none found +already there; an upgraded or restored database attests nothing and produces its +evidence by running the command. + **Released-file collection is live behind that same flag** (#3459). On a verified deployment, a field file whose one owning record lets go — the field cleared, or the record deleted — is tombstoned into the declared 30-day grace diff --git a/docs/adr/0104-field-runtime-value-shape-contract.md b/docs/adr/0104-field-runtime-value-shape-contract.md index b733aaeb76..4057cca497 100644 --- a/docs/adr/0104-field-runtime-value-shape-contract.md +++ b/docs/adr/0104-field-runtime-value-shape-contract.md @@ -852,10 +852,29 @@ implementing PR's decision (candidates: the system seed that first creates requirement: it fires only for a store it is itself creating from empty, never for one it found. +**Seam, as implemented.** Neither candidate survived contact: *"`sys_migration` +is absent"* is precisely the 16→17 trap — an upgrading database lacks that +table too — and the schema bootstrap runs identically on both. Nothing in the +platform recorded datastore newness at all; the drivers knew it for one +statement (`hasTable` → `createTable`) and discarded it. So the observation is +now kept: each driver counts the tables it **created** against those it +**found** since connect (`IDataDriver.getSchemaSyncStats`), and a store is +attested only when the whole engine reports *created > 0 and existing = 0*. +Every uncertainty resolves to "no" — a driver that cannot report, a deferred or +skipped sync, a second datasource that was already there. Being strict here is +cheap in the only direction that matters: a wrongly-withheld attestation costs +a command someone can still run, a wrongly-granted one asserts a store's +history on no evidence. The write lands at `kernel:ready` from the service that +owns `sys_migration`, and never overwrites an existing row: a store with flag +rows is by definition not one being created. + Born-strict deployments also make the platform's own dogfood the standing canary for R2 (a codified shape stricter than some legitimate client's writes): showcase/CRM boots are fresh datastores, so they enforce from birth, -and a false rejection fails our suites before any customer sees it. +and a false rejection fails our suites before any customer sees it. Since +#3459 that canary covers the collection path too — a born-attested dogfood run +exercises release-time tombstoning and the reap guard's re-verify on every +boot, so an over-eager delete fails our suites rather than a customer's data. ### D2 action params: the evidence cannot exist, so strict is the 17.0 default diff --git a/packages/objectql/src/engine.test.ts b/packages/objectql/src/engine.test.ts index 63be689c24..9f1f22e946 100644 --- a/packages/objectql/src/engine.test.ts +++ b/packages/objectql/src/engine.test.ts @@ -2274,4 +2274,48 @@ describe('ObjectQL — file-as-reference migration flag (#3617)', () => { const flagReads = vi.mocked(driver.find).mock.calls.filter((c) => c[0] === 'sys_migration'); expect(flagReads).toHaveLength(2); }); + + // ── Was this datastore created from empty? (#3438, ADR-0104) ────── + // The one input to fresh-datastore attestation. Permission is granted on + // this answer, so every uncertainty must resolve to `false`. + describe('wasDatastoreCreatedFromEmpty', () => { + const withStats = (d: IDataDriver, stats: { created: number; existing: number } | null) => { + if (stats) (d as any).getSchemaSyncStats = () => stats; + else delete (d as any).getSchemaSyncStats; + }; + + it('is true when the driver created tables and found none', () => { + withStats(driver, { created: 12, existing: 0 }); + expect(engine.wasDatastoreCreatedFromEmpty()).toBe(true); + }); + + it('is false when even one table was already there', () => { + withStats(driver, { created: 11, existing: 1 }); + expect(engine.wasDatastoreCreatedFromEmpty()).toBe(false); + }); + + it('is false when nothing was created (sync skipped or deferred)', () => { + withStats(driver, { created: 0, existing: 0 }); + expect(engine.wasDatastoreCreatedFromEmpty()).toBe(false); + }); + + it('is false when no driver can account for its schema sync', () => { + withStats(driver, null); + expect(engine.wasDatastoreCreatedFromEmpty()).toBe(false); + }); + + it('is false when a second datasource was not created by this boot', () => { + withStats(driver, { created: 12, existing: 0 }); + const other = { + name: 'other', + connect: vi.fn().mockResolvedValue(undefined), + disconnect: vi.fn().mockResolvedValue(undefined), + capabilities: {} as any, + } as unknown as IDataDriver; + withStats(other, { created: 0, existing: 5 }); + engine.registerDriver(other); + + expect(engine.wasDatastoreCreatedFromEmpty()).toBe(false); + }); + }); }); diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index ab7d117af5..407362a2bb 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -2262,6 +2262,36 @@ export class ObjectQL implements IDataEngine { this.fileReferencesMigrationVerified = null; } + /** + * Did this process CREATE the datastore it is talking to, from empty? + * + * True only when every driver that can account for its schema sync created + * tables and found none already there. That conjunction is the whole point: + * one pre-existing table anywhere means something ran here before us, so + * this store's history is not ours to vouch for. A driver that cannot + * report (`getSchemaSyncStats` absent, deferred DDL, sync skipped) makes the + * answer no rather than maybe — the consumers of this fact grant + * permissions, so "cannot say" must read as "no". + * + * The one caller is the fresh-datastore attestation (#3438, ADR-0104's + * 2026-07-30 addendum), which records that a store born empty needs no data + * migration. Ask it only during boot: once the process starts serving, the + * counts describe a moment that has passed. + */ + wasDatastoreCreatedFromEmpty(): boolean { + let created = 0; + let existing = 0; + let reporting = 0; + for (const driver of this.drivers.values()) { + const stats = (driver as any).getSchemaSyncStats?.(); + if (!stats) continue; + reporting += 1; + created += Number(stats.created) || 0; + existing += Number(stats.existing) || 0; + } + return reporting > 0 && existing === 0 && created > 0; + } + async destroy() { this.logger.info('Destroying ObjectQL engine', { driverCount: this.drivers.size }); diff --git a/packages/platform-objects/src/system/index.ts b/packages/platform-objects/src/system/index.ts index a44e4dceae..b55bce4ceb 100644 --- a/packages/platform-objects/src/system/index.ts +++ b/packages/platform-objects/src/system/index.ts @@ -17,6 +17,9 @@ export { readDataMigrationFlag, isDataMigrationVerified, recordDataMigrationRun, + attestFreshDatastore, + CREATION_ATTESTATION_DETAIL, type MigrationFlagEngine, type DataMigrationRunOutcome, + type AttestationLogger, } from './migration-flag.js'; diff --git a/packages/platform-objects/src/system/migration-flag.test.ts b/packages/platform-objects/src/system/migration-flag.test.ts index 60d9307e22..8015ceed0f 100644 --- a/packages/platform-objects/src/system/migration-flag.test.ts +++ b/packages/platform-objects/src/system/migration-flag.test.ts @@ -1,10 +1,13 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. -import { describe, it, expect } from 'vitest'; +import { describe, it, expect, vi } from 'vitest'; +import { CREATION_ATTESTED_MIGRATION_IDS } from '@objectstack/spec/system'; import { readDataMigrationFlag, isDataMigrationVerified, recordDataMigrationRun, + attestFreshDatastore, + CREATION_ATTESTATION_DETAIL, type MigrationFlagEngine, } from './migration-flag.js'; @@ -112,3 +115,85 @@ describe('deployment-level data-migration flags (#3617)', () => { expect(await isDataMigrationVerified(engine, MIGRATION)).toBe(false); }); }); + +describe('fresh-datastore attestation (ADR-0104, 2026-07-30 addendum)', () => { + it('attests every creation-attested migration, verified and blocking-free', async () => { + const engine = fakeEngine(); + + const attested = await attestFreshDatastore(engine); + + expect(attested).toEqual([...CREATION_ATTESTED_MIGRATION_IDS]); + expect(engine.tables.sys_migration).toHaveLength(CREATION_ATTESTED_MIGRATION_IDS.length); + for (const id of CREATION_ATTESTED_MIGRATION_IDS) { + expect(await isDataMigrationVerified(engine, id)).toBe(true); + } + }); + + it('records nothing as applied — no backfill ran, and none was needed', async () => { + const engine = fakeEngine(); + + await attestFreshDatastore(engine); + + const row = engine.tables.sys_migration[0]; + expect(row.applied_at).toBeNull(); + expect(row.blocking).toBe(0); + }); + + it('marks the row so evidence-by-birth is distinguishable from evidence-by-scan', async () => { + const engine = fakeEngine(); + + await attestFreshDatastore(engine); + + expect(JSON.parse(String(engine.tables.sys_migration[0].details))).toEqual( + CREATION_ATTESTATION_DETAIL, + ); + }); + + /** + * The load-bearing safety property. An existing row means this store is not + * one being created — whatever the caller believed — so attestation must + * leave it exactly as it found it. Overwriting could only ever RAISE a gate + * the deployment's own evidence had closed. + */ + it('never overwrites an existing row, including one a failed run closed', async () => { + const engine = fakeEngine([ + { id: MIGRATION, last_run_at: 'yesterday', verified_at: null, blocking: 7 }, + ]); + + const attested = await attestFreshDatastore(engine); + + expect(attested).not.toContain(MIGRATION); + const row = engine.tables.sys_migration.find((r) => r.id === MIGRATION)!; + expect(row).toMatchObject({ verified_at: null, blocking: 7, last_run_at: 'yesterday' }); + expect(await isDataMigrationVerified(engine, MIGRATION)).toBe(false); + }); + + it('is idempotent — a second call adds nothing', async () => { + const engine = fakeEngine(); + + await attestFreshDatastore(engine); + const second = await attestFreshDatastore(engine); + + expect(second).toEqual([]); + expect(engine.tables.sys_migration).toHaveLength(CREATION_ATTESTED_MIGRATION_IDS.length); + }); + + it('does nothing when sys_migration is not registered (bare kernel)', async () => { + const engine = fakeEngine([], { registered: false }); + + expect(await attestFreshDatastore(engine)).toEqual([]); + expect(engine.tables.sys_migration).toHaveLength(0); + }); + + it('a write failure warns and leaves the deployment lax — it never throws into a boot', async () => { + const engine = fakeEngine(); + engine.insert = async () => { + throw new Error('table locked'); + }; + const logger = { info: vi.fn(), warn: vi.fn() }; + + await expect(attestFreshDatastore(engine, { logger })).resolves.toEqual([]); + expect(logger.warn).toHaveBeenCalled(); + expect(logger.info).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/platform-objects/src/system/migration-flag.ts b/packages/platform-objects/src/system/migration-flag.ts index 336a38f7ce..5fddddb21e 100644 --- a/packages/platform-objects/src/system/migration-flag.ts +++ b/packages/platform-objects/src/system/migration-flag.ts @@ -1,6 +1,7 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. import { + CREATION_ATTESTED_MIGRATION_IDS, DATA_MIGRATION_FLAG_OBJECT, isDataMigrationFlagVerified, type DataMigrationFlag, @@ -132,3 +133,92 @@ export async function recordDataMigrationRun( } return flag; } + +/** Marker written into a creation-attested row's `details`, so an operator + * reading a verified flag can tell evidence-by-scan from evidence-by-birth. */ +export const CREATION_ATTESTATION_DETAIL = { attested: 'datastore-created-empty' } as const; + +/** Optional sink for attestation trouble — best-effort, never fatal. */ +export interface AttestationLogger { + info(msg: string, meta?: unknown): void; + warn(msg: string, meta?: unknown): void; +} + +/** + * Attest, at the moment a datastore is created from empty, the migrations + * whose facts that emptiness already settles (ADR-0104, 2026-07-30 addendum). + * + * **The caller carries the whole precondition.** This function cannot check + * it: by the time anything can be queried, "created empty" and "found empty" + * look identical, and they are not the same claim — a half-initialised store, + * a misconfigured connection, or a store someone truncated all *look* empty + * while legacy values may yet arrive. Only the code performing the creation + * knows it watched the store come into being. Call this ONLY from there; + * never from a probe that concluded a store looks empty. + * + * Given that, the fact recorded is observed, not assumed — the same discipline + * the gates run on. Without it, every deployment born on a version that + * already ships these migrations would start lax and stay lax until someone + * ran a command that, for them, does nothing. + * + * **Never overwrites.** A migration id that already has a row is skipped + * untouched: a store with flag rows is by definition not one being created, + * so a write here would be evidence about the wrong database — and it could + * only ever *raise* a gate the real evidence had closed. + * + * **Best-effort, deliberately diverging from this module's "writes fail + * loudly" rule.** That rule fits the migration commands, whose entire output + * is the record. This runs inside a fresh deployment's boot, where the two + * failure directions are not symmetric: a missed attestation leaves the + * deployment lax (warnings, retained files — recoverable by running the + * migration), while a thrown error would break the boot of a brand-new + * deployment over bookkeeping. + * + * @returns the ids actually attested (absent ones only). + */ +export async function attestFreshDatastore( + engine: MigrationFlagEngine, + options: { migrationIds?: readonly string[]; logger?: AttestationLogger } = {}, +): Promise { + const ids = options.migrationIds ?? CREATION_ATTESTED_MIGRATION_IDS; + const logger = options.logger; + if (!engine.getObject(DATA_MIGRATION_FLAG_OBJECT)) return []; + + const now = new Date().toISOString(); + const attested: string[] = []; + for (const id of ids) { + try { + if (await readDataMigrationFlag(engine, id)) continue; // not ours to write + await engine.insert( + DATA_MIGRATION_FLAG_OBJECT, + { + id, + last_run_at: now, + verified_at: now, + // Nothing was applied — no backfill ran, and none was needed. + applied_at: null, + blocking: 0, + advisory: 0, + details: JSON.stringify(CREATION_ATTESTATION_DETAIL), + created_at: now, + updated_at: now, + }, + { context: { ...SYSTEM_CTX } }, + ); + attested.push(id); + } catch (err) { + logger?.warn( + `[migration] could not attest '${id}' on this new datastore ` + + `(${(err as Error)?.message ?? err}) — it stays in the lax posture until ` + + `\`os migrate\` records the flag`, + ); + } + } + if (attested.length > 0) { + logger?.info( + `[migration] new datastore attested at creation: ${attested.join(', ')} — ` + + `no legacy data can exist here, so the gated behaviour is enabled from birth`, + ); + } + return attested; +} diff --git a/packages/plugins/driver-memory/src/memory-driver.test.ts b/packages/plugins/driver-memory/src/memory-driver.test.ts index 6c0397c90c..126d3bf40b 100644 --- a/packages/plugins/driver-memory/src/memory-driver.test.ts +++ b/packages/plugins/driver-memory/src/memory-driver.test.ts @@ -790,4 +790,53 @@ describe('InMemoryDriver', () => { }); }); + // The observation behind fresh-datastore attestation (#3438, ADR-0104): + // a store where this process created every table and found none had no + // history, so it can hold no legacy value. + describe('getSchemaSyncStats', () => { + it('starts at zero — a driver that synced nothing vouches for nothing', () => { + expect(driver.getSchemaSyncStats()).toEqual({ created: 0, existing: 0 }); + }); + + it('counts each table it creates', async () => { + await driver.syncSchema('alpha', { fields: {} }); + await driver.syncSchema('beta', { fields: {} }); + + expect(driver.getSchemaSyncStats()).toEqual({ created: 2, existing: 0 }); + }); + + it('counts a table that already held rows as pre-existing', async () => { + // What a persistence adapter's restore looks like: rows land before any + // schema is declared, so the store did not start empty here. + await driver.create('restored', { id: '1' }); + + await driver.syncSchema('restored', { fields: {} }); + + expect(driver.getSchemaSyncStats()).toEqual({ created: 0, existing: 1 }); + }); + + /** + * Boot syncs the schema more than once, and objects registered later sync + * on demand — so a table we created ourselves comes back around. Counting + * that as pre-existing would make every store look like it had history and + * silently kill attestation everywhere. + */ + it('ignores a re-sync of a table it created this boot', async () => { + await driver.syncSchema('alpha', { fields: {} }); + await driver.syncSchema('alpha', { fields: {} }); + await driver.syncSchema('alpha', { fields: {} }); + + expect(driver.getSchemaSyncStats()).toEqual({ created: 1, existing: 0 }); + }); + + it('counts each pre-existing table once, however often it is synced', async () => { + await driver.create('restored', { id: '1' }); + + await driver.syncSchema('restored', { fields: {} }); + await driver.syncSchema('restored', { fields: {} }); + + expect(driver.getSchemaSyncStats()).toEqual({ created: 0, existing: 1 }); + }); + }); + }); diff --git a/packages/plugins/driver-memory/src/memory-driver.ts b/packages/plugins/driver-memory/src/memory-driver.ts index cdd52bccb7..81241528c1 100644 --- a/packages/plugins/driver-memory/src/memory-driver.ts +++ b/packages/plugins/driver-memory/src/memory-driver.ts @@ -201,6 +201,10 @@ export class InMemoryDriver implements IDataDriver { * The "Database": A map of TableName -> Array of Records */ private db: Record = {}; + /** Tables this driver created since connect — see {@link getSchemaSyncStats}. */ + private tablesCreatedHere: Set = new Set(); + /** Tables that were already populated when this driver first synced them. */ + private tablesFoundExisting: Set = new Set(); // =================================== // Lifecycle @@ -1149,7 +1153,14 @@ export class InMemoryDriver implements IDataDriver { async syncSchema(object: string, schema: any, options?: DriverOptions) { if (!this.db[object]) { this.db[object] = []; + this.tablesCreatedHere.add(object); this.logger.info('Created in-memory table', { object }); + } else if (!this.tablesCreatedHere.has(object)) { + // Present without us having made it: a persistence adapter restored it, + // or fixtures seeded it before any schema was declared. Either way the + // store did not start empty here. A re-sync of a table we created this + // boot is not that, and is ignored. See {@link getSchemaSyncStats}. + this.tablesFoundExisting.add(object); } // Learn the object's temporal fields, then converge the rows ALREADY in the // table (#4047). Both halves matter: the map is what the write and filter @@ -1177,6 +1188,17 @@ export class InMemoryDriver implements IDataDriver { } } + /** + * Tables created vs found since connect — the `IDataDriver.getSchemaSyncStats` + * contract. An in-memory store is normally born empty on every boot, so + * `existing === 0 && created > 0` is the common case here; a store a + * persistence adapter restored reports `existing > 0` and vouches for + * nothing (#3438 / ADR-0104). + */ + getSchemaSyncStats(): { created: number; existing: number } { + return { created: this.tablesCreatedHere.size, existing: this.tablesFoundExisting.size }; + } + // =================================== // Helpers // =================================== diff --git a/packages/plugins/driver-sql/src/sql-driver-schema-sync-stats.test.ts b/packages/plugins/driver-sql/src/sql-driver-schema-sync-stats.test.ts new file mode 100644 index 0000000000..bbdccfae59 --- /dev/null +++ b/packages/plugins/driver-sql/src/sql-driver-schema-sync-stats.test.ts @@ -0,0 +1,115 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * `getSchemaSyncStats` — the observation behind fresh-datastore attestation + * (#3438, ADR-0104's 2026-07-30 addendum). + * + * A deployment may record that it needs no data migration only when the + * platform watched itself create the store: every table made by this boot, + * none found already there. Nothing else in the platform knows that — the + * driver knows it for one statement (`hasTable` → `createTable`) and used to + * throw it away. + * + * The counts are per TABLE, not per sync call, because boot syncs the schema + * more than once and objects registered later sync on demand. Counting a + * re-sync of our own table as pre-existing would make every store look like it + * had history, silently disabling attestation everywhere it matters. + */ + +import { describe, it, expect } from 'vitest'; +import { SqlDriver } from '../src/index.js'; + +const sqlite = () => + new SqlDriver({ + client: 'better-sqlite3', + connection: { filename: ':memory:' }, + useNullAsDefault: true, + }); + +const OBJECTS = [ + { name: 'alpha', fields: { name: { type: 'string' } } }, + { name: 'beta', fields: { name: { type: 'string' } } }, +]; + +describe('SqlDriver.getSchemaSyncStats', () => { + it('reports nothing before any schema work — a driver that synced nothing vouches for nothing', async () => { + const driver = sqlite(); + try { + expect(driver.getSchemaSyncStats()).toEqual({ created: 0, existing: 0 }); + } finally { + await driver.disconnect(); + } + }); + + it('counts every table it creates in an empty store', async () => { + const driver = sqlite(); + try { + await driver.initObjects(OBJECTS as any); + + expect(driver.getSchemaSyncStats()).toEqual({ created: 2, existing: 0 }); + } finally { + await driver.disconnect(); + } + }); + + /** Boot's own repeated sync must not look like history. */ + it('stays "created from empty" across repeated syncs of its own tables', async () => { + const driver = sqlite(); + try { + await driver.initObjects(OBJECTS as any); + await driver.initObjects(OBJECTS as any); + await driver.syncSchema('alpha', OBJECTS[0] as any); + + expect(driver.getSchemaSyncStats()).toEqual({ created: 2, existing: 0 }); + } finally { + await driver.disconnect(); + } + }); + + /** + * The load-bearing negative: a store with tables from an earlier run. Here + * a second driver connects to the same file, which is what an upgrade looks + * like from the platform's side. + */ + it('reports pre-existing tables for a store it did not create', async () => { + const { mkdtemp } = await import('node:fs/promises'); + const { tmpdir } = await import('node:os'); + const { join } = await import('node:path'); + const dir = await mkdtemp(join(tmpdir(), 'oss-sync-stats-')); + const filename = join(dir, 'store.db'); + const opts = { client: 'better-sqlite3' as const, connection: { filename }, useNullAsDefault: true }; + + // Both pools are closed unconditionally: a knex pool outlives a failed + // assertion with a live timer, which keeps the test process from exiting + // long after the run itself has passed. + const first = new SqlDriver(opts); + try { + await first.initObjects(OBJECTS as any); + expect(first.getSchemaSyncStats()).toEqual({ created: 2, existing: 0 }); + } finally { + await first.disconnect(); + } + + // A later boot against the same store — the tables precede it. + const second = new SqlDriver(opts); + try { + await second.initObjects(OBJECTS as any); + + expect(second.getSchemaSyncStats()).toEqual({ created: 0, existing: 2 }); + } finally { + await second.disconnect(); + } + }); + + it('a store that gains an object later is still not one we found', async () => { + const driver = sqlite(); + try { + await driver.initObjects([OBJECTS[0]] as any); + await driver.syncSchema('gamma', { name: 'gamma', fields: { n: { type: 'string' } } } as any); + + expect(driver.getSchemaSyncStats()).toEqual({ created: 2, existing: 0 }); + } finally { + await driver.disconnect(); + } + }); +}); diff --git a/packages/plugins/driver-sql/src/sql-driver.ts b/packages/plugins/driver-sql/src/sql-driver.ts index 70f3e5df60..6e21d01184 100644 --- a/packages/plugins/driver-sql/src/sql-driver.ts +++ b/packages/plugins/driver-sql/src/sql-driver.ts @@ -568,6 +568,10 @@ export class SqlDriver implements IDataDriver { /** External columnMap inverse: physical remote column -> logical field (for read output remap). */ protected columnFieldByObject: Record> = {}; protected tablesWithTimestamps: Set = new Set(); + /** Tables this driver created since connect — see `getSchemaSyncStats`. */ + protected tablesCreatedHere: Set = new Set(); + /** Tables that were already present when this driver first touched them. */ + protected tablesFoundExisting: Set = new Set(); /** * Autonumber field configs per table, captured during initObjects. * @@ -2893,6 +2897,14 @@ export class SqlDriver implements IDataDriver { } let exists = await this.knex.schema.hasTable(tableName); + // Recorded BEFORE the legacy-`_id` rebuild below can flip `exists`: a + // table that was here when we arrived stays "found" even if we then drop + // and recreate it. Re-syncing a table we made ourselves this boot — boot + // runs the sync more than once, and objects registered later sync on + // demand — is not evidence of anything preceding us, so it is ignored. + // See `getSchemaSyncStats`. + if (!exists) this.tablesCreatedHere.add(tableName); + else if (!this.tablesCreatedHere.has(tableName)) this.tablesFoundExisting.add(tableName); if (exists) { const columnInfo = await this.knex(tableName).columnInfo(); @@ -3336,6 +3348,16 @@ export class SqlDriver implements IDataDriver { return this.deferredSchemaObjects.size; } + /** + * Tables this driver created vs found already present since `connect()` — + * the `IDataDriver.getSchemaSyncStats` contract. `existing === 0 && + * created > 0` is the platform's only observation that a datastore was + * empty when this process reached it (#3438 / ADR-0104). + */ + getSchemaSyncStats(): { created: number; existing: number } { + return { created: this.tablesCreatedHere.size, existing: this.tablesFoundExisting.size }; + } + /** * What the deferred sync *would* do, without doing it. * diff --git a/packages/services/service-storage/src/storage-service-plugin.test.ts b/packages/services/service-storage/src/storage-service-plugin.test.ts index 5b6027601b..307d195b33 100644 --- a/packages/services/service-storage/src/storage-service-plugin.test.ts +++ b/packages/services/service-storage/src/storage-service-plugin.test.ts @@ -370,3 +370,83 @@ describe('StorageServicePlugin: sys_file orphan lifecycle wiring (#2755)', () => await expect(ctx._flushReady()).resolves.toBeUndefined(); }); }); + +describe('StorageServicePlugin: fresh-datastore attestation (#3438, ADR-0104)', () => { + /** Engine fake with a `sys_migration` table and a settable newness verdict. */ + function engineWith(createdFromEmpty: boolean | undefined) { + const rows: Array> = []; + const engine: any = { + registerHook: () => {}, + registerMiddleware: () => {}, + getObject: (name: string) => (name === 'sys_migration' ? { name } : undefined), + find: async (_object: string, options: any) => + rows.filter((r) => options?.where?.id === undefined || r.id === options.where.id), + findOne: async () => null, + insert: async (_object: string, data: any) => { + rows.push({ ...data }); + return data; + }, + update: async () => ({}), + rows, + }; + if (createdFromEmpty !== undefined) { + engine.wasDatastoreCreatedFromEmpty = () => createdFromEmpty; + } + return engine; + } + + async function boot(engine: unknown) { + const dir = await fs.mkdtemp(join(tmpdir(), 'oss-attest-')); + const plugin = new StorageServicePlugin({ adapter: 'local', local: { rootDir: dir }, registerRoutes: false }); + const ctx = makeCtx(); + ctx.registerService('objectql', engine); + await plugin.init(ctx); + await plugin.start(ctx); + await ctx._flushReady(); + } + + it('attests a store this boot created from empty', async () => { + const engine = engineWith(true); + + await boot(engine); + + expect(engine.rows.map((r: any) => r.id).sort()).toEqual([ + 'adr-0104-file-references', + 'adr-0104-value-shapes', + ]); + for (const row of engine.rows) { + expect(row.verified_at).toBeTruthy(); + expect(row.blocking).toBe(0); + } + }); + + /** + * The property the whole design rests on: a store that was FOUND — an + * upgrade, a restart, anything with history — attests nothing and keeps + * producing its evidence by scan. + */ + it('attests nothing on a store that already existed', async () => { + const engine = engineWith(false); + + await boot(engine); + + expect(engine.rows).toHaveLength(0); + }); + + it('attests nothing when the engine cannot say (older engine)', async () => { + const engine = engineWith(undefined); + + await boot(engine); + + expect(engine.rows).toHaveLength(0); + }); + + it('a failing attestation never breaks the boot', async () => { + const engine = engineWith(true); + engine.insert = async () => { + throw new Error('disk full'); + }; + + await expect(boot(engine)).resolves.toBeUndefined(); + }); +}); diff --git a/packages/services/service-storage/src/storage-service-plugin.ts b/packages/services/service-storage/src/storage-service-plugin.ts index 24624fe867..cc2703437b 100644 --- a/packages/services/service-storage/src/storage-service-plugin.ts +++ b/packages/services/service-storage/src/storage-service-plugin.ts @@ -34,7 +34,7 @@ import { SysAttachment } from '@objectstack/platform-objects/audit'; // its first consuming domain: the ADR-0104 file-as-reference row gates this // service's released-file collection (#3459 PR-5b) and the strict media // value-shape default (#3438). -import { SysMigration, isDataMigrationVerified } from '@objectstack/platform-objects/system'; +import { SysMigration, isDataMigrationVerified, attestFreshDatastore } from '@objectstack/platform-objects/system'; import { FILE_REFERENCES_MIGRATION_ID } from '@objectstack/spec/system'; import { SwappableStorageService } from './swappable-storage-service.js'; import { @@ -333,6 +333,37 @@ export class StorageServicePlugin implements Plugin { // declaration stays safe: rows only gain reap triggers via the // hooks above, and nothing sweeps without the LifecycleService. } + + // ── Fresh-datastore attestation (#3438, ADR-0104 2026-07-30) ─── + // A store this process just created from empty can hold no legacy + // value, so the data migrations that exist to find and convert them + // are settled here before they are ever run — recorded now, while + // that emptiness is still an observed fact rather than something a + // later scan would have to infer. Without it every new deployment + // would start lax and stay lax until someone ran a command that, for + // them, does nothing: the warn regime would never die out. + // + // This service owns the call because it registers `sys_migration` + // (above) and is the platform's only holder of the flag's other + // consumers. A store that was found rather than created attests + // nothing and keeps producing evidence by scan. + if (typeof (engine as any).wasDatastoreCreatedFromEmpty === 'function') { + try { + if ((engine as any).wasDatastoreCreatedFromEmpty()) { + await attestFreshDatastore(engine as any, { logger: ctx.logger }); + // The engine memoizes the flag read on first use; this write + // may already have raced it on a fast boot. + (engine as any).invalidateDataMigrationFlags?.(); + } + } catch (err) { + // Bookkeeping must never break a new deployment's boot. Not + // attesting only leaves it lax — recoverable by running the + // migration, which for an empty store is a no-op that passes. + ctx.logger.warn( + `StorageServicePlugin: fresh-datastore attestation skipped (${(err as Error)?.message ?? err})`, + ); + } + } } // ── HTTP routes (existing behaviour) ─────────────────────────── diff --git a/packages/spec/api-surface.json b/packages/spec/api-surface.json index d424f6e66d..2ceb4f293f 100644 --- a/packages/spec/api-surface.json +++ b/packages/spec/api-surface.json @@ -700,6 +700,7 @@ "CRDTState (type)", "CRDTStateSchema (const)", "CRDTType (type)", + "CREATION_ATTESTED_MIGRATION_IDS (const)", "CacheAvalanchePrevention (type)", "CacheAvalanchePreventionSchema (const)", "CacheConfig (type)", @@ -1309,6 +1310,7 @@ "UserActivityStatus (type)", "VALIDATION_MESSAGE_FALLBACK_LOCALE (const)", "VALIDATION_MESSAGE_KEY_PREFIX (const)", + "VALUE_SHAPES_MIGRATION_ID (const)", "ValidationMessageTranslator (type)", "VectorClock (type)", "VectorClockSchema (const)", diff --git a/packages/spec/src/contracts/data-driver.ts b/packages/spec/src/contracts/data-driver.ts index da49489406..fd7a6771fe 100644 --- a/packages/spec/src/contracts/data-driver.ts +++ b/packages/spec/src/contracts/data-driver.ts @@ -218,6 +218,28 @@ export interface IDataDriver { */ registerExternalObject?(schema: unknown): void | Promise; + /** + * What this driver's schema synchronisation has DONE since `connect()`: + * how many tables it created, and how many it found already present. + * + * The pair answers a question no other surface can: **was this datastore + * empty when the process reached it?** `existing === 0 && created > 0` means + * every table here was made by this boot — nothing preceded it, so no row + * written by an earlier version can exist. That is an observation of + * creating a store, not an inference from finding one that looks empty, and + * it is what lets a fresh deployment attest data migrations it can never + * need (ADR-0104's 2026-07-30 addendum, #3438). + * + * Counts tables the driver actually inspected: deferred DDL (`os migrate + * plan`) and skipped sync leave both at zero, which reads as "cannot say" + * and keeps the conservative posture. A table that existed and was rebuilt + * counts as `existing` — it was here before us. + * + * Optional and purely observational; drivers that omit it simply cannot + * vouch for a store's newness. + */ + getSchemaSyncStats?(): { created: number; existing: number }; + /** Drop the underlying table or collection (destructive) */ dropTable(object: string, options?: DriverOptions): Promise; diff --git a/packages/spec/src/system/migration.zod.ts b/packages/spec/src/system/migration.zod.ts index 5f1845af94..1e221d6d54 100644 --- a/packages/spec/src/system/migration.zod.ts +++ b/packages/spec/src/system/migration.zod.ts @@ -137,6 +137,36 @@ export const DATA_MIGRATION_FLAG_OBJECT = 'sys_migration'; */ export const FILE_REFERENCES_MIGRATION_ID = 'adr-0104-file-references'; +/** + * Well-known migration id: ADR-0104 D1 value shapes — every stored value of + * the covered non-media classes (single-value references and structured JSON) + * parses against `valueSchemaFor(field, 'stored')`. Unlike the file migration + * there is nothing to convert: a malformed payload is an application-data fix + * only its owner can make, so the scan reports and prescribes, and `--apply`'s + * only write is this flag. Gates strict enforcement of those classes (#3438). + */ +export const VALUE_SHAPES_MIGRATION_ID = 'adr-0104-value-shapes'; + +/** + * The migrations a datastore attests at CREATION rather than by scanning. + * + * Both facts these ids stand for — no legacy file value here, no malformed + * stored value here — are true by construction of an empty store, and true + * *observably*: the creator watched it come into being with no rows at all. + * That is the same observed-transition discipline the gates run on, not + * version-gating in disguise; a store that merely *looks* empty when found + * earns nothing, because "found empty" is an inference and "created empty" is + * an observation. + * + * Without this, every deployment born on a version that already ships the + * migrations would start lax and stay lax until someone ran a command that is, + * for them, a no-op — so the warn regime would never die out. + */ +export const CREATION_ATTESTED_MIGRATION_IDS = [ + FILE_REFERENCES_MIGRATION_ID, + VALUE_SHAPES_MIGRATION_ID, +] as const; + export const DataMigrationFlagSchema = lazySchema(() => z.object({ id: z.string().describe('Migration id (e.g. adr-0104-file-references) — one row per data migration'), last_run_at: z.string().datetime().describe('When this migration last completed a gated (apply-mode) run on this deployment'),