From ff23956aa62c4976e6aefe5ffbb3bff590919c64 Mon Sep 17 00:00:00 2001 From: os-zhuang Date: Fri, 19 Jun 2026 07:27:52 +0800 Subject: [PATCH 1/3] feat(verify): extract dogfood engine into public @objectstack/verify + CLI Productize the internal dogfood regression engine as a published library and an `objectstack verify` CLI so third-party and template authors can run the same runtime proofs (auto-derived CRUD round-trip fidelity + the cross-owner RLS invariant) against their own apps. - new public @objectstack/verify: bootStack / deriveCrudCases / runCrudVerification / runRlsProofs, moved+generalized from packages/dogfood/src (tsup ESM+CJS+DTS, publishConfig public) - new `objectstack verify` oclif command (--app/--rls/--multi-tenant/--json); exits 1 on create-failed/read-failed/fidelity-gaps/rls-hole - dogfood now consumes @objectstack/verify; drop 3 redundant auto-verify tests (replaced by an `objectstack verify` CI step over the example apps); keep the hand-written golden regressions (#2018 tz, #1994 RLS, #2004 field fidelity) - fix: the harness used a port-less loopback inject origin (http://localhost) that fails better-auth's default dev trusted-origins (http://localhost:*); use a ported base so dev-admin sign-in passes in a bare node CLI and the test runner alike - add @objectstack/verify to the changeset `fixed` group; changeset for verify+cli Co-Authored-By: Claude Opus 4.8 --- .changeset/config.json | 3 +- .changeset/verify-package.md | 8 ++ .github/workflows/ci.yml | 13 +++ packages/cli/package.json | 1 + packages/cli/src/commands/verify.ts | 102 ++++++++++++++++++ packages/dogfood/README.md | 25 +++-- packages/dogfood/package.json | 15 +-- .../test/analytics-timezone.dogfood.test.ts | 6 +- .../test/auto-verify-rls.dogfood.test.ts | 57 ---------- .../dogfood/test/auto-verify.dogfood.test.ts | 54 ---------- .../test/field-zoo-roundtrip.dogfood.test.ts | 6 +- .../dogfood/test/rls-fixture.dogfood.test.ts | 12 +-- .../test/rls-multitenant.dogfood.test.ts | 8 +- packages/dogfood/test/rls-runner.test.ts | 8 +- .../test/verify-external.dogfood.test.ts | 40 ------- packages/verify/README.md | 102 ++++++++++++++++++ packages/verify/package.json | 56 ++++++++++ packages/{dogfood => verify}/src/derive.ts | 0 packages/{dogfood => verify}/src/harness.ts | 60 +++++++---- packages/verify/src/index.ts | 20 ++++ packages/{dogfood => verify}/src/rls.ts | 4 +- packages/{dogfood => verify}/src/verify.ts | 4 +- packages/verify/tsconfig.json | 9 ++ pnpm-lock.yaml | 85 +++++++++------ 24 files changed, 449 insertions(+), 249 deletions(-) create mode 100644 .changeset/verify-package.md create mode 100644 packages/cli/src/commands/verify.ts delete mode 100644 packages/dogfood/test/auto-verify-rls.dogfood.test.ts delete mode 100644 packages/dogfood/test/auto-verify.dogfood.test.ts delete mode 100644 packages/dogfood/test/verify-external.dogfood.test.ts create mode 100644 packages/verify/README.md create mode 100644 packages/verify/package.json rename packages/{dogfood => verify}/src/derive.ts (100%) rename packages/{dogfood => verify}/src/harness.ts (76%) create mode 100644 packages/verify/src/index.ts rename packages/{dogfood => verify}/src/rls.ts (98%) rename packages/{dogfood => verify}/src/verify.ts (98%) create mode 100644 packages/verify/tsconfig.json diff --git a/.changeset/config.json b/.changeset/config.json index 6a32845941..65418a2c58 100644 --- a/.changeset/config.json +++ b/.changeset/config.json @@ -80,7 +80,8 @@ "@objectstack/account", "create-objectstack", "objectstack-vscode", - "@objectstack/connector-openapi" + "@objectstack/connector-openapi", + "@objectstack/verify" ] ], "linked": [], diff --git a/.changeset/verify-package.md b/.changeset/verify-package.md new file mode 100644 index 0000000000..a4bb4b73fb --- /dev/null +++ b/.changeset/verify-package.md @@ -0,0 +1,8 @@ +--- +"@objectstack/verify": minor +"@objectstack/cli": minor +--- + +Add `@objectstack/verify` — boot any ObjectStack app in-process and verify it through the real HTTP stack: auto-derived CRUD round-trip fidelity (`runCrudVerification`) plus the cross-owner RLS invariant (`runRlsProofs`, "you can't write what you can't read"). Also adds an `objectstack verify` CLI command that runs these proofs against an app config and exits non-zero on real failures. + +Extracted from the internal dogfood regression gate so third-party and template authors can run the same runtime proofs against their own apps. The private `@objectstack/dogfood` package now consumes this library for its golden regression tests. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 08defeebce..26d9eb3c33 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -161,6 +161,19 @@ jobs: - name: Boot example apps and exercise real user flows run: pnpm turbo run test --filter=@objectstack/dogfood + # Replaces the former auto-verify dogfood tests: runs the published + # `objectstack verify` engine over each example app through the CLI — + # auto-derived CRUD round-trip fidelity + the cross-owner RLS invariant. + # Exits non-zero on a real runtime failure, so it gates like the tests did. + - name: Verify example apps via the `objectstack verify` CLI + run: | + pnpm turbo run build --filter=@objectstack/cli + for app in examples/app-crm examples/app-showcase; do + echo "::group::objectstack verify $app --rls" + OS_LOG_LEVEL=error node packages/cli/bin/run.js verify --app "$app/objectstack.config.ts" --rls + echo "::endgroup::" + done + build-core: name: Build Core needs: filter diff --git a/packages/cli/package.json b/packages/cli/package.json index 4c258f5937..d69cdd428b 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -88,6 +88,7 @@ "@objectstack/service-storage": "workspace:*", "@objectstack/spec": "workspace:*", "@objectstack/types": "workspace:*", + "@objectstack/verify": "workspace:*", "@oclif/core": "^4.11.4", "bundle-require": "^5.1.0", "chalk": "^5.6.2", diff --git a/packages/cli/src/commands/verify.ts b/packages/cli/src/commands/verify.ts new file mode 100644 index 0000000000..93f721a0d3 --- /dev/null +++ b/packages/cli/src/commands/verify.ts @@ -0,0 +1,102 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { Command, Flags } from '@oclif/core'; +import chalk from 'chalk'; +import { readEnvWithDeprecation } from '@objectstack/types'; +import { + bootStack, + runCrudVerification, + formatReport, + runRlsProofs, + formatRlsReport, + type VerifyReport, + type RlsReport, +} from '@objectstack/verify'; +import { loadConfig } from '../utils/config.js'; + +/** + * `objectstack verify` — boot the app in-process and exercise it through the + * real HTTP stack, asserting runtime behavior the static gates can't see: + * - data fidelity: author → write → read → assert, per object/field type + * - authorization (--rls): "you can't write what you can't read" (#1994 class) + * + * Exits non-zero on real failures so it drops straight into CI. + */ +export default class Verify extends Command { + static override description = + 'Boot the app in-process and verify it through the real HTTP stack (CRUD round-trip fidelity + the cross-owner RLS invariant)'; + + static override examples = [ + '<%= config.bin %> verify', + '<%= config.bin %> verify --app ./objectstack.config.ts --rls', + '<%= config.bin %> verify --rls --multi-tenant --json', + ]; + + static override flags = { + app: Flags.string({ + char: 'a', + description: 'Path to the app config (defaults to ./objectstack.config.{ts,js,mjs})', + }), + rls: Flags.boolean({ + description: 'Also run the cross-owner RLS invariant (a fresh member must not write what it cannot read)', + default: false, + }), + 'multi-tenant': Flags.boolean({ + description: 'Boot org-scoped (register plugin-org-scoping) so tenant-isolation RLS policies apply (also honors $OS_MULTI_ORG_ENABLED)', + default: false, + }), + json: Flags.boolean({ description: 'Emit the structured report as JSON', default: false }), + }; + + async run(): Promise { + const { flags } = await this.parse(Verify); + + const { config, absolutePath } = await loadConfig(flags.app); + + const multiTenant = + flags['multi-tenant'] || + String(readEnvWithDeprecation('OS_MULTI_ORG_ENABLED', 'OS_MULTI_TENANT') ?? 'false').toLowerCase() !== + 'false'; + + const stack = await bootStack(config, { multiTenant }); + + let crud: VerifyReport; + let rls: RlsReport | undefined; + try { + const adminToken = await stack.signIn(); + crud = await runCrudVerification(stack, adminToken, config); + + if (flags.rls) { + const memberToken = await stack.signUp('verify-member@objectstack.test'); + rls = await runRlsProofs(stack, adminToken, memberToken, config); + } + } finally { + await stack.stop(); + } + + // Failure contract: a "real" runtime break the app's author must see. + const hardFailures = + crud.summary.createFailed + + crud.summary.readFailed + + crud.summary.fidelityGaps + + (rls?.summary.holes ?? 0); + + if (flags.json) { + this.log(JSON.stringify({ app: crud.app, config: absolutePath, multiTenant, crud, rls, hardFailures }, null, 2)); + } else { + this.log(formatReport(crud)); + if (rls) this.log(formatRlsReport(rls)); + this.log(''); + this.log( + hardFailures > 0 + ? chalk.red(`✗ verify FAILED — ${hardFailures} runtime failure(s)`) + : chalk.green('✓ verify passed — no runtime failures'), + ); + } + + // Force process exit: the in-process stack leaves handles open (http server, + // sqlite-wasm, better-auth timers) that keep the event loop alive after + // stop(), so a bare return would hang. exit() also encodes the CI contract. + this.exit(hardFailures > 0 ? 1 : 0); + } +} diff --git a/packages/dogfood/README.md b/packages/dogfood/README.md index 77c6964b94..7c2d30d547 100644 --- a/packages/dogfood/README.md +++ b/packages/dogfood/README.md @@ -23,7 +23,13 @@ sockets, CI-stable). Tests act as a browser client would: sign in, hit ## Layout -- `src/harness.ts` — `bootDogfoodStack(config)` → `{ kernel, api, raw, signIn, apiAs, stop }`. +The in-process harness + auto-derived verifiers (`bootStack`, `runCrudVerification`, +`runRlsProofs`) now live in the published **[`@objectstack/verify`](../verify)** +package — point it at any app. This package is the framework's own consumer: it +holds the **hand-written golden tests** that pin specific historical regressions +the generic verifier cannot auto-derive. + +- depends on `@objectstack/verify` for `bootStack(config)` → `{ kernel, api, raw, signIn, signUp, apiAs, stop }`. - `test/*.dogfood.test.ts` — golden flows. Each should assert on **observable output** (a number, a bucket label, a row count), not just "no error". @@ -31,7 +37,7 @@ sockets, CI-stable). Tests act as a browser client would: sign in, hit 1. Pick a real user flow that a static test can't cover (it spans engine + service + HTTP, or depends on seeded/written data). -2. `bootDogfoodStack()`, `signIn()`, drive it via `api()/apiAs()`. +2. `bootStack()`, `signIn()`, drive it via `api()/apiAs()`. 3. Assert on the concrete result. 4. **Prove it catches the bug**: temporarily revert the relevant fix and confirm the test goes red. A green-on-the-bug test is not a gate. @@ -67,7 +73,7 @@ The binding policy — every authorable+live primitive must carry a runtime proo The capability matrix above proves *data* round-trips. The authorization dimension proves a record the caller must not touch stays untouched. The -app-agnostic invariant (`src/rls.ts`, `runRlsProofs`): +app-agnostic invariant (`runRlsProofs`, from `@objectstack/verify`): > **A user who cannot READ a record must not be able to WRITE it.** @@ -87,9 +93,10 @@ changed. Verdicts: `rls-consistent` (can't read **and** can't write — good), `rls-hole` (can't read **yet** wrote — the #1994 bug), `member-visible` (member *can* read it — inconclusive, not a cross-owner scenario). -`auto-verify-rls.dogfood.test.ts` runs this over the example apps, but they boot -**single-tenant**, where every object comes back `member-visible` — so the -by-id-write path is never actually exercised. Two ways to create real isolation: +`objectstack verify --rls` runs this over any app in CI, but the example +apps boot **single-tenant**, where every object comes back `member-visible` — so +the by-id-write path is never actually exercised. Two ways to create real +isolation, both pinned as golden tests here: ### 1. Owner-scoped fixture — `test/rls-fixture.dogfood.test.ts` (hard gate) @@ -98,11 +105,11 @@ permission set carries `RLS.ownerPolicy('rls_note', 'created_by')`. The predicat is `created_by = current_user.id` — keyed on the column the engine stamps on every record and referencing `current_user.id`, **not** `current_user.organization_id`, so it survives single-tenant policy stripping. A -fresh member genuinely can't read the admin's note. `bootDogfoodStack` takes a +fresh member genuinely can't read the admin's note. `bootStack` takes a `security:` override so the fixture's permission set is the member's fallback: ```ts -bootDogfoodStack(rlsFixtureStack, { security: rlsFixtureSecurity(ownerScopedMemberSet) }) +bootStack(rlsFixtureStack, { security: rlsFixtureSecurity(ownerScopedMemberSet) }) ``` - **Green gate** (owner policy on `all` ops) → `rls-consistent`. Safe *only* @@ -139,7 +146,7 @@ Faithful fix — boot multi-tenant so `@objectstack/plugin-org-scoping` register before `SecurityPlugin` and the `organization_id` policies apply: ```ts -bootDogfoodStack(crmStack, { multiTenant: true }) +bootStack(crmStack, { multiTenant: true }) ``` The dev admin is bound to the seeded default org; a fresh `signUp` member is not, diff --git a/packages/dogfood/package.json b/packages/dogfood/package.json index dc6bbb403f..b87afe6849 100644 --- a/packages/dogfood/package.json +++ b/packages/dogfood/package.json @@ -3,27 +3,18 @@ "version": "0.0.1", "private": true, "license": "Apache-2.0", - "description": "Dogfood regression gate — boots real example apps in-process and exercises them through the real HTTP/service stack, catching runtime regressions that static checks (build, unit tests, spec liveness) miss.", + "description": "Dogfood regression gate — hand-written golden tests that boot real example apps through @objectstack/verify's in-process HTTP stack, pinning historical runtime regressions (#2018 timezone bucketing, #1994 cross-owner RLS, #2004 field fidelity) that static checks miss.", "type": "module", "scripts": { "test": "vitest run" }, "dependencies": { - "@objectstack/core": "workspace:*", - "@objectstack/runtime": "workspace:*", + "@objectstack/verify": "workspace:*", "@objectstack/objectql": "workspace:*", "@objectstack/spec": "workspace:*", - "@objectstack/driver-sqlite-wasm": "workspace:*", - "@objectstack/plugin-hono-server": "workspace:*", - "@objectstack/rest": "workspace:*", - "@objectstack/plugin-auth": "workspace:*", "@objectstack/plugin-security": "workspace:*", - "@objectstack/service-settings": "workspace:*", - "@objectstack/service-analytics": "workspace:*", "@objectstack/example-crm": "workspace:*", - "@objectstack/example-showcase": "workspace:*", - "@objectstack/plugin-sharing": "workspace:*", - "@objectstack/plugin-org-scoping": "workspace:*" + "@objectstack/example-showcase": "workspace:*" }, "devDependencies": { "@types/node": "^25.9.3", diff --git a/packages/dogfood/test/analytics-timezone.dogfood.test.ts b/packages/dogfood/test/analytics-timezone.dogfood.test.ts index f453eb057f..7d3492be1e 100644 --- a/packages/dogfood/test/analytics-timezone.dogfood.test.ts +++ b/packages/dogfood/test/analytics-timezone.dogfood.test.ts @@ -10,7 +10,7 @@ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; import crmStack from '@objectstack/example-crm'; -import { bootDogfoodStack, type DogfoodStack } from '../src/harness.js'; +import { bootStack, type VerifyStack } from '@objectstack/verify'; // 03:00 UTC on 2024-03-01 is still 2024-02-29 (19:00) in America/Los_Angeles // (PST = UTC-8, before DST). So a *day* bucket labels this instant 2024-03-01 @@ -29,11 +29,11 @@ const leadByDay = { }; describe('dogfood: org timezone drives analytics date bucketing (#1982/#2018)', () => { - let stack: DogfoodStack; + let stack: VerifyStack; let token: string; beforeAll(async () => { - stack = await bootDogfoodStack(crmStack); + stack = await bootStack(crmStack); // Deterministic fixture: N leads pinned to the tz-boundary instant, inserted // as system so the write path's defaults/validation don't fight the setup. diff --git a/packages/dogfood/test/auto-verify-rls.dogfood.test.ts b/packages/dogfood/test/auto-verify-rls.dogfood.test.ts deleted file mode 100644 index b2a9c29a7f..0000000000 --- a/packages/dogfood/test/auto-verify-rls.dogfood.test.ts +++ /dev/null @@ -1,57 +0,0 @@ -// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. -// -// Live RLS cross-owner smoke (#1994) over the framework example apps, with a -// real second user. The runner's hole-detection logic is unit-proven in -// `rls-runner.test.ts`; this exercises it end-to-end against real apps. -// -// SCOPE: this file is the single-tenant SMOKE over real apps. Single-tenant -// strips the org `tenant_isolation` policy and a fresh member falls back to -// `member_default` (broad read), so every object reports `member-visible` and -// the by-id-write path isn't exercised HERE. That gap is now closed by two -// sibling tests, so the hard gate lives there, not here: -// • rls-fixture.dogfood.test.ts — owner-scoped fixture; green gate + -// automated red proof + a documented manual #1994 revert proof (README). -// • rls-multitenant.dogfood.test.ts — `{ multiTenant: true }`; org-scoped -// (organization_id) isolation, the model real apps like hotcrm use. -// The invariant asserted here (zero holes) still guards against a regression -// that makes a member able to mutate a record it cannot read. - -import { describe, it, expect, beforeAll, afterAll } from 'vitest'; -import crmStack from '@objectstack/example-crm'; -import showcaseStack from '@objectstack/example-showcase'; -import { bootDogfoodStack, type DogfoodStack } from '../src/harness.js'; -import { runRlsProofs, formatRlsReport, type RlsReport } from '../src/rls.js'; - -const APPS: Array<[string, unknown]> = [ - ['crm', crmStack], - ['showcase', showcaseStack], -]; - -for (const [name, config] of APPS) { - describe(`objectstack verify RLS: ${name} (#1994 cross-owner)`, () => { - let stack: DogfoodStack; - let report: RlsReport; - - beforeAll(async () => { - stack = await bootDogfoodStack(config as never); - const adminToken = await stack.signIn(); - const memberToken = await stack.signUp(`member-${name}@verify.test`); - report = await runRlsProofs(stack, adminToken, memberToken, config); - // eslint-disable-next-line no-console - console.error(formatRlsReport(report)); - }, 60_000); - - afterAll(async () => { - await stack?.stop(); - }); - - it('boots with two distinct users and runs cross-owner proofs', () => { - expect(report.summary.objects).toBeGreaterThan(0); - }); - - it('has ZERO by-id-write RLS holes (#1994 invariant)', () => { - const holes = report.results.filter((r) => r.status === 'rls-hole'); - expect(holes, formatRlsReport(report)).toHaveLength(0); - }); - }); -} diff --git a/packages/dogfood/test/auto-verify.dogfood.test.ts b/packages/dogfood/test/auto-verify.dogfood.test.ts deleted file mode 100644 index fef89decd0..0000000000 --- a/packages/dogfood/test/auto-verify.dogfood.test.ts +++ /dev/null @@ -1,54 +0,0 @@ -// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. -// -// objectstack verify — metadata-driven runtime verification, proven against the -// framework's own example apps. From each app's metadata ALONE it auto-derives a -// CRUD round-trip contract (no hand-written cases) and runs it over real HTTP. -// The same runner points at any third-party app's built artifact via -// `test/verify-external.dogfood.test.ts`. - -import { describe, it, expect, beforeAll, afterAll } from 'vitest'; -import crmStack from '@objectstack/example-crm'; -import showcaseStack from '@objectstack/example-showcase'; -import { bootDogfoodStack, type DogfoodStack } from '../src/harness.js'; -import { runCrudVerification, formatReport, type VerifyReport } from '../src/verify.js'; - -const APPS: Array<[string, unknown]> = [ - ['crm', crmStack], - ['showcase', showcaseStack], -]; - -for (const [name, config] of APPS) { - describe(`objectstack verify: ${name} (auto-derived CRUD round-trip)`, () => { - let stack: DogfoodStack; - let report: VerifyReport; - - beforeAll(async () => { - stack = await bootDogfoodStack(config as never); - const token = await stack.signIn(); - report = await runCrudVerification(stack, token, config); - // eslint-disable-next-line no-console - console.error(formatReport(report)); - }, 60_000); - - afterAll(async () => { - await stack?.stop(); - }); - - it('derives a runtime contract from metadata and boots', () => { - expect(report.summary.objects).toBeGreaterThan(0); - }); - - it('verifies objects end-to-end over real HTTP', () => { - expect(report.summary.verified).toBeGreaterThan(0); - }); - - it('has no object that fails to create or read (the hard runtime invariant)', () => { - // create/read failures = the app's metadata produces a record the runtime - // refuses or can't read back — a real platform/integration finding (vs - // `needs-fixture`, the auto-record tripping the app's own validation rules, - // and `fidelity-gaps`, type leaks tracked separately). - const hard = report.results.filter((r) => r.status === 'create-failed' || r.status === 'read-failed'); - expect(hard, JSON.stringify(hard, null, 2)).toHaveLength(0); - }); - }); -} diff --git a/packages/dogfood/test/field-zoo-roundtrip.dogfood.test.ts b/packages/dogfood/test/field-zoo-roundtrip.dogfood.test.ts index 74eeb48c35..6b7640d50c 100644 --- a/packages/dogfood/test/field-zoo-roundtrip.dogfood.test.ts +++ b/packages/dogfood/test/field-zoo-roundtrip.dogfood.test.ts @@ -14,7 +14,7 @@ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; import showcaseStack from '@objectstack/example-showcase'; import { SECRET_MASK } from '@objectstack/objectql'; -import { bootDogfoodStack, type DogfoodStack } from '../src/harness.js'; +import { bootStack, type VerifyStack } from '@objectstack/verify'; // A field-type coverage entry. `write` is the value POSTed; `expect` describes // how the value must come back. `equal` = exact (or set-equal for arrays); @@ -113,11 +113,11 @@ const MATRIX: FieldCase[] = [ ]; describe('dogfood: field-type capability matrix round-trips over HTTP (#2004)', () => { - let stack: DogfoodStack; + let stack: VerifyStack; let record: Record; beforeAll(async () => { - stack = await bootDogfoodStack(showcaseStack); + stack = await bootStack(showcaseStack); const token = await stack.signIn(); // Build the create body from every entry that carries a `write` value diff --git a/packages/dogfood/test/rls-fixture.dogfood.test.ts b/packages/dogfood/test/rls-fixture.dogfood.test.ts index 0726344fac..ed64e64e6c 100644 --- a/packages/dogfood/test/rls-fixture.dogfood.test.ts +++ b/packages/dogfood/test/rls-fixture.dogfood.test.ts @@ -19,8 +19,8 @@ // the gate can actually go red. import { describe, it, expect, beforeAll, afterAll } from 'vitest'; -import { bootDogfoodStack, type DogfoodStack } from '../src/harness.js'; -import { runRlsProofs, formatRlsReport, type RlsReport } from '../src/rls.js'; +import { bootStack, type VerifyStack } from '@objectstack/verify'; +import { runRlsProofs, formatRlsReport, type RlsReport } from '@objectstack/verify'; import { rlsFixtureStack, ownerScopedMemberSet, @@ -31,13 +31,13 @@ import { describe('objectstack verify RLS: owner-isolated fixture (#1994 hard gate)', () => { // ── GREEN: the gate that must stay consistent ────────────────────────────── describe('owner-scoped member set (all ops)', () => { - let stack: DogfoodStack; + let stack: VerifyStack; let report: RlsReport; let adminToken: string; let memberToken: string; beforeAll(async () => { - stack = await bootDogfoodStack(rlsFixtureStack, { + stack = await bootStack(rlsFixtureStack, { security: rlsFixtureSecurity(ownerScopedMemberSet), }); adminToken = await stack.signIn(); @@ -80,11 +80,11 @@ describe('objectstack verify RLS: owner-isolated fixture (#1994 hard gate)', () // ── RED: proof the gate can go red on the #1994 hole class ────────────────── describe('read-only-scoped member set (select only) — #1994 hole reproduced', () => { - let stack: DogfoodStack; + let stack: VerifyStack; let report: RlsReport; beforeAll(async () => { - stack = await bootDogfoodStack(rlsFixtureStack, { + stack = await bootStack(rlsFixtureStack, { security: rlsFixtureSecurity(readOnlyScopedMemberSet), }); const adminToken = await stack.signIn(); diff --git a/packages/dogfood/test/rls-multitenant.dogfood.test.ts b/packages/dogfood/test/rls-multitenant.dogfood.test.ts index 681a1a8b73..112643141a 100644 --- a/packages/dogfood/test/rls-multitenant.dogfood.test.ts +++ b/packages/dogfood/test/rls-multitenant.dogfood.test.ts @@ -27,15 +27,15 @@ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; import crmStack from '@objectstack/example-crm'; -import { bootDogfoodStack, type DogfoodStack } from '../src/harness.js'; -import { runRlsProofs, formatRlsReport, type RlsReport } from '../src/rls.js'; +import { bootStack, type VerifyStack } from '@objectstack/verify'; +import { runRlsProofs, formatRlsReport, type RlsReport } from '@objectstack/verify'; describe('objectstack verify RLS: CRM multi-tenant (#1994 org-scoped)', () => { - let stack: DogfoodStack; + let stack: VerifyStack; let report: RlsReport; beforeAll(async () => { - stack = await bootDogfoodStack(crmStack as never, { multiTenant: true }); + stack = await bootStack(crmStack as never, { multiTenant: true }); const adminToken = await stack.signIn(); const memberToken = await stack.signUp('member-mt@verify.test'); report = await runRlsProofs(stack, adminToken, memberToken, crmStack); diff --git a/packages/dogfood/test/rls-runner.test.ts b/packages/dogfood/test/rls-runner.test.ts index ab3843438b..6be82af6a4 100644 --- a/packages/dogfood/test/rls-runner.test.ts +++ b/packages/dogfood/test/rls-runner.test.ts @@ -7,8 +7,8 @@ // The invariant: a user who CANNOT READ a record must not be able to WRITE it. import { describe, it, expect } from 'vitest'; -import { runRlsProofs } from '../src/rls.js'; -import type { DogfoodStack } from '../src/harness.js'; +import { runRlsProofs } from '@objectstack/verify'; +import type { VerifyStack } from '@objectstack/verify'; const CONFIG = { manifest: { id: 'fixture' }, @@ -19,12 +19,12 @@ const CONFIG = { function fakeStack(opts: { memberCanRead: boolean; memberWriteMutates: boolean; // does member's PATCH actually change the row? -}): DogfoodStack { +}): VerifyStack { const store: Record = {}; const json = (body: unknown, status = 200) => new Response(JSON.stringify(body), { status, headers: { 'Content-Type': 'application/json' } }); - const apiAs: DogfoodStack['apiAs'] = async (token, method, path, body) => { + const apiAs: VerifyStack['apiAs'] = async (token, method, path, body) => { const isAdmin = token === 'admin'; const [, , object, id] = path.split('/'); // /data// if (method === 'POST') { diff --git a/packages/dogfood/test/verify-external.dogfood.test.ts b/packages/dogfood/test/verify-external.dogfood.test.ts deleted file mode 100644 index 2ab1725276..0000000000 --- a/packages/dogfood/test/verify-external.dogfood.test.ts +++ /dev/null @@ -1,40 +0,0 @@ -// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. -// -// Point the verifier at ANY app's built artifact — the embryonic -// `objectstack verify `. This is the consumer-facing use: a third-party app -// (e.g. hotcrm) runs it against its OWN metadata to learn where its declared -// behavior doesn't hold at runtime. -// -// Gated on OS_VERIFY_ARTIFACT so it never runs in framework CI (the external app -// isn't in the workspace). Run locally: -// OS_VERIFY_ARTIFACT=/abs/path/to/app/dist/objectstack.json \ -// pnpm --filter @objectstack/dogfood exec vitest run test/verify-external.dogfood.test.ts - -import { describe, it, expect, beforeAll, afterAll } from 'vitest'; -import { readFileSync, existsSync } from 'node:fs'; -import { bootDogfoodStack, type DogfoodStack } from '../src/harness.js'; -import { runCrudVerification, formatReport, type VerifyReport } from '../src/verify.js'; - -const ARTIFACT = process.env.OS_VERIFY_ARTIFACT; - -describe.skipIf(!ARTIFACT || !existsSync(ARTIFACT))('objectstack verify: external app artifact', () => { - let stack: DogfoodStack; - let report: VerifyReport; - - beforeAll(async () => { - const config = JSON.parse(readFileSync(ARTIFACT as string, 'utf8')); - stack = await bootDogfoodStack(config); - const token = await stack.signIn(); - report = await runCrudVerification(stack, token, config); - // eslint-disable-next-line no-console - console.error(formatReport(report)); - }, 120_000); - - afterAll(async () => { - await stack?.stop(); - }); - - it('boots the external app and auto-derives a runtime contract', () => { - expect(report.summary.objects).toBeGreaterThan(0); - }); -}); diff --git a/packages/verify/README.md b/packages/verify/README.md new file mode 100644 index 0000000000..984674694f --- /dev/null +++ b/packages/verify/README.md @@ -0,0 +1,102 @@ +# @objectstack/verify + +Boot any ObjectStack app **in-process** and verify it through the **real HTTP +stack** — no mocks, no ports, no sockets. Two app-agnostic proof families, both +derived from your own metadata: + +- **Data fidelity** — author one record per object covering every field type, + write it over the real REST API, read it back, assert each field round-trips + with type fidelity. +- **Authorization** — the cross-owner RLS invariant: *a user who cannot READ a + record must not be able to WRITE it.* + +## Why + +Static gates — type-check, unit tests, schema validation — verify each layer in +isolation, usually against mocks. A whole class of regressions only appears when +the **real engine + strategies + services + HTTP context run together**: a date +bucket that ignores the org timezone, a field type that persists but reads back +as the wrong shape, a by-id write that skips the row-level security filter. Each +layer is individually correct; the break is at the seams. + +`@objectstack/verify` boots the integrated stack (in-memory SQLite, the same +service plugins `objectstack dev` loads) and exercises it as a browser client +would, so those breaks are observable in CI. + +This matters most on a **metadata platform**: the risk isn't "a platform change +broke the example app" — it's "a valid primitive your app uses, but the examples +don't exercise, silently breaks at runtime." Point this at *your* app. + +> Posture: development / in-memory. The harness forces `NODE_ENV=development` to +> provision a known dev admin and uses an in-memory database. It never touches a +> real database or production data. + +## CLI (zero-config) + +```sh +# from an app directory (auto-detects objectstack.config.ts) +objectstack verify + +# explicit config + the RLS invariant + multi-tenant isolation +objectstack verify --app ./objectstack.config.ts --rls --multi-tenant +``` + +Exit code is **non-zero** on real failures (`create-failed`, `read-failed`, +`fidelity-gaps`, `rls-hole`) so it drops straight into a CI gate. Inconclusive +verdicts (`needs-fixture`, `skipped`, `member-visible`) are warnings and exit 0. + +## Programmatic (embed in your own test runner) + +```ts +import { bootStack, runCrudVerification, runRlsProofs, formatReport } from '@objectstack/verify'; +import myApp from './objectstack.config.js'; + +const stack = await bootStack(myApp); +const adminToken = await stack.signIn(); + +// Data fidelity +const report = await runCrudVerification(stack, adminToken, myApp); +console.log(formatReport(report)); +expect(report.summary.fidelityGaps).toBe(0); + +// Authorization (RLS / cross-owner): a fresh member must not write what it can't read +const memberToken = await stack.signUp('member@example.com'); +const rls = await runRlsProofs(stack, adminToken, memberToken, myApp); +expect(rls.summary.holes).toBe(0); + +await stack.stop(); +``` + +## Verdicts + +**Data fidelity** (`runCrudVerification`): + +| verdict | meaning | +| --- | --- | +| `verified` | every asserted field round-tripped | +| `fidelity-gaps` | wrote a value, read back a different shape/type **(failure)** | +| `create-failed` / `read-failed` | the write or read errored **(failure)** | +| `needs-fixture` | the app's own validation rejected the auto-derived record (supply a fixture) | +| `skipped` | object has a required field that can't be auto-synthesized (e.g. a required lookup) | + +**Authorization** (`runRlsProofs`): + +| verdict | meaning | +| --- | --- | +| `rls-consistent` | member can't read **and** can't write — good | +| `rls-hole` | member can't read **yet** wrote it by id — RLS bypass **(failure)** | +| `member-visible` | member *can* read it — not a cross-owner scenario (inconclusive) | + +`member-visible` everywhere usually means the app is single-tenant; pass +`--multi-tenant` (or `{ multiTenant: true }`) to register org-scoping so tenant +isolation policies actually apply. + +## API + +- `bootStack(config, opts?)` → `VerifyStack` (`api` / `raw` / `signIn` / `signUp` / `apiAs` / `stop`). +- `deriveCrudCases(config)` → the auto-derived round-trip cases (write one, read one, assert) for every object. +- `runCrudVerification(stack, token, config)` → `VerifyReport`; `formatReport(report)` for a log summary. +- `runRlsProofs(stack, adminToken, memberToken, config)` → `RlsReport`; `formatRlsReport(report)`. + +`bootStack` options: `admin`, `authSecret`, `security` (a custom `SecurityPlugin` +for owner-scoped fixtures), `multiTenant`. diff --git a/packages/verify/package.json b/packages/verify/package.json new file mode 100644 index 0000000000..0aa0da006e --- /dev/null +++ b/packages/verify/package.json @@ -0,0 +1,56 @@ +{ + "name": "@objectstack/verify", + "version": "0.1.0", + "license": "Apache-2.0", + "description": "Boot any ObjectStack app in-process and verify it through the real HTTP stack — auto-derived CRUD round-trip fidelity plus the cross-owner RLS invariant. Catches runtime regressions that static checks miss.", + "type": "module", + "main": "dist/index.js", + "types": "dist/index.d.ts", + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js", + "require": "./dist/index.cjs" + } + }, + "scripts": { + "build": "tsup --config ../../tsup.config.ts", + "dev": "tsc -w", + "lint": "eslint src" + }, + "dependencies": { + "@objectstack/core": "workspace:*", + "@objectstack/runtime": "workspace:*", + "@objectstack/objectql": "workspace:*", + "@objectstack/spec": "workspace:*", + "@objectstack/driver-sqlite-wasm": "workspace:*", + "@objectstack/plugin-hono-server": "workspace:*", + "@objectstack/rest": "workspace:*", + "@objectstack/plugin-auth": "workspace:*", + "@objectstack/plugin-security": "workspace:*", + "@objectstack/plugin-sharing": "workspace:*", + "@objectstack/plugin-org-scoping": "workspace:*", + "@objectstack/service-settings": "workspace:*", + "@objectstack/service-analytics": "workspace:*" + }, + "devDependencies": { + "@types/node": "^25.9.3", + "typescript": "^6.0.3" + }, + "keywords": ["objectstack", "verify", "testing", "regression", "integration"], + "author": "ObjectStack", + "repository": { + "type": "git", + "url": "https://github.com/objectstack-ai/framework.git", + "directory": "packages/verify" + }, + "homepage": "https://objectstack.ai/docs", + "bugs": "https://github.com/objectstack-ai/framework/issues", + "publishConfig": { + "access": "public" + }, + "files": ["dist", "README.md"], + "engines": { + "node": ">=18.0.0" + } +} diff --git a/packages/dogfood/src/derive.ts b/packages/verify/src/derive.ts similarity index 100% rename from packages/dogfood/src/derive.ts rename to packages/verify/src/derive.ts diff --git a/packages/dogfood/src/harness.ts b/packages/verify/src/harness.ts similarity index 76% rename from packages/dogfood/src/harness.ts rename to packages/verify/src/harness.ts index 6487ef868f..96be31bccf 100644 --- a/packages/dogfood/src/harness.ts +++ b/packages/verify/src/harness.ts @@ -1,19 +1,24 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. // -// Dogfood boot harness. +// @objectstack/verify — boot harness. // // Boots a real ObjectStack app **in-process** against an in-memory SQLite // database, wired with the same service plugins `objectstack dev` loads, and // exposes the live HTTP surface via Hono's request-injection (no port, no -// sockets — CI-stable). Tests then exercise the app exactly as a browser +// sockets — CI-stable). A verifier then exercises the app exactly as a browser // client would: sign in, hit `/api/v1/...`, assert on real responses. // -// Why this exists: the bucketing regression fixed in #2018 passed every static -// gate (build, 900+ unit tests, spec-liveness, CodeQL) because each layer was -// individually correct and individually mocked — the break only appeared when -// the real engine + strategy + settings + REST context ran together. Unit -// tests that mock the protocol/server (e.g. rest.test.ts) cannot catch that. -// This harness runs the integrated stack so they can. +// Why in-process + real HTTP: a whole class of regressions only surfaces when +// the real engine + strategies + services + REST context run together — each +// layer can be individually correct (and individually mocked in unit tests) yet +// break at the seams (e.g. timezone date-bucketing across analytics strategy, +// in-memory aggregation, and the REST execution context). This harness runs the +// integrated stack so those breaks are observable. +// +// Posture: development / in-memory. `NODE_ENV` is forced to `development` so the +// auth plugin's dev-admin bootstrap provisions a known, loginable admin (mirrors +// `objectstack dev`). This is a verification harness — it never touches a real +// database or production data. import { ObjectKernel, AppPlugin, DriverPlugin, createDispatcherPlugin } from '@objectstack/runtime'; import { ObjectQLPlugin } from '@objectstack/objectql'; @@ -34,8 +39,9 @@ interface InjectableApp { const API_PREFIX = '/api/v1'; const DEFAULT_ADMIN_EMAIL = 'admin@objectos.ai'; const DEFAULT_ADMIN_PASSWORD = 'admin123'; +const DEFAULT_AUTH_SECRET = 'objectstack-verify-secret'; -export interface DogfoodStack { +export interface VerifyStack { /** The booted kernel — for direct service calls when bypassing HTTP is intentional. */ kernel: ObjectKernel; /** Inject an HTTP request through the real Hono app (no socket). Path is relative to `/api/v1`. */ @@ -57,10 +63,12 @@ export interface DogfoodStack { export interface BootOptions { /** Override the dev admin credentials the harness signs in with. */ admin?: { email: string; password: string }; + /** Override the auth signing secret. Defaults to a fixed in-process dev secret. */ + authSecret?: string; /** * Override the SecurityPlugin instance. Pass a `new SecurityPlugin({...})` * to carry a custom `fallbackPermissionSet` / extra permission sets — this - * is how the owner-isolated RLS fixture makes a fresh member fall back to a + * is how an owner-isolated RLS fixture makes a fresh member fall back to a * permission set that carries `RLS.ownerPolicy(...)` instead of the broad-read * `member_default`. Defaults to a vanilla `new SecurityPlugin()`. */ @@ -71,23 +79,23 @@ export interface BootOptions { * the default permission sets actually apply (SecurityPlugin probes the * `org-scoping` service once at start and otherwise STRIPS them — see * `collectRLSPolicies`). This exercises the org-scoped isolation real apps - * (e.g. hotcrm) rely on, rather than the single-tenant default where every - * tenant policy is stripped and a member sees every row. Default `false`. + * rely on, rather than the single-tenant default where every tenant policy is + * stripped and a member sees every row. Default `false`. */ multiTenant?: boolean; } /** - * Boot an app config in-process and return a live dogfood stack. + * Boot an app config in-process and return a live verification stack. * * `NODE_ENV` is forced to `development` so the auth plugin's dev-admin * bootstrap provisions a known, loginable admin (mirrors `objectstack dev`). */ -export async function bootDogfoodStack( +export async function bootStack( // eslint-disable-next-line @typescript-eslint/no-explicit-any config: any, opts: BootOptions = {}, -): Promise { +): Promise { process.env.NODE_ENV = 'development'; const kernel = new ObjectKernel(); @@ -107,12 +115,12 @@ export async function bootDogfoodStack( // Service plugins `objectstack dev` auto-loads for an app of this shape. await kernel.use(new SettingsServicePlugin()); await kernel.use(new AnalyticsServicePlugin()); - await kernel.use(new AuthPlugin({ secret: 'dogfood-regression-secret' })); + await kernel.use(new AuthPlugin({ secret: opts.authSecret ?? DEFAULT_AUTH_SECRET })); // Multi-tenant: org-scoping MUST register BEFORE SecurityPlugin — the latter // probes the `org-scoping` service exactly once at start and caches it, then // keeps (vs strips) the wildcard `organization_id` RLS policies accordingly. - // Mirrors `plugin-dev`'s ordering for `OS_MULTI_ORG_ENABLED`. + // Mirrors the CLI's ordering for `OS_MULTI_ORG_ENABLED`. if (opts.multiTenant) { const { OrgScopingPlugin } = await import('@objectstack/plugin-org-scoping'); await kernel.use(new OrgScopingPlugin()); @@ -148,7 +156,15 @@ export async function bootDogfoodStack( ); const app = httpServer.getRawApp(); - const raw = (path: string, init?: RequestInit) => app.request(path, init); + // Same-origin loopback base for request-injection. A *ported* localhost origin + // matches better-auth's default dev trusted-origins set (`http://localhost:*`), + // so the in-process dev-admin sign-in passes the CSRF origin check regardless + // of runtime (a bare `node` CLI vs a test runner) or ambient CORS env. A + // path-only inject yields `http://localhost` (no port), which does NOT match + // the `:*` wildcard and gets a 403. Routing is by path; the host:port only + // shapes `new URL(request.url).origin`, which the auth layer reads. + const ORIGIN = 'http://localhost:3000'; + const raw = (path: string, init?: RequestInit) => app.request(`${ORIGIN}${path}`, init); const api = (path: string, init?: RequestInit) => raw(`${API_PREFIX}${path}`, init); const admin = opts.admin ?? { email: DEFAULT_ADMIN_EMAIL, password: DEFAULT_ADMIN_PASSWORD }; @@ -163,10 +179,10 @@ export async function bootDogfoodStack( body: JSON.stringify({ email, password }), }); if (!res.ok) { - throw new Error(`dogfood signIn failed: ${res.status} ${await res.text()}`); + throw new Error(`verify signIn failed: ${res.status} ${await res.text()}`); } const data = (await res.json()) as { token?: string }; - if (!data.token) throw new Error('dogfood signIn: no token in response'); + if (!data.token) throw new Error('verify signIn: no token in response'); return data.token; }; @@ -181,10 +197,10 @@ export async function bootDogfoodStack( body: JSON.stringify({ email, password, name: name ?? email.split('@')[0] }), }); if (!res.ok) { - throw new Error(`dogfood signUp failed: ${res.status} ${await res.text()}`); + throw new Error(`verify signUp failed: ${res.status} ${await res.text()}`); } const data = (await res.json()) as { token?: string }; - if (!data.token) throw new Error('dogfood signUp: no token in response'); + if (!data.token) throw new Error('verify signUp: no token in response'); return data.token; }; diff --git a/packages/verify/src/index.ts b/packages/verify/src/index.ts new file mode 100644 index 0000000000..64b12436eb --- /dev/null +++ b/packages/verify/src/index.ts @@ -0,0 +1,20 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. +// +// @objectstack/verify — public API. +// +// Boot any ObjectStack app in-process and verify it through the real HTTP +// stack. Two proof families, both app-agnostic (derived from your metadata): +// - data fidelity : runCrudVerification — author → write → read → assert +// - authorization : runRlsProofs — "you can't write what you can't read" + +export { bootStack } from './harness.js'; +export type { VerifyStack, BootOptions } from './harness.js'; + +export { deriveCrudCases } from './derive.js'; +export type { CrudCase, DerivedAssert, AssertKind } from './derive.js'; + +export { runCrudVerification, formatReport } from './verify.js'; +export type { VerifyReport, ObjectVerifyResult } from './verify.js'; + +export { runRlsProofs, formatRlsReport } from './rls.js'; +export type { RlsReport, RlsResult } from './rls.js'; diff --git a/packages/dogfood/src/rls.ts b/packages/verify/src/rls.ts similarity index 98% rename from packages/dogfood/src/rls.ts rename to packages/verify/src/rls.ts index 8d7c26dfbc..44bc1b435a 100644 --- a/packages/dogfood/src/rls.ts +++ b/packages/verify/src/rls.ts @@ -17,7 +17,7 @@ /* eslint-disable @typescript-eslint/no-explicit-any */ -import type { DogfoodStack } from './harness.js'; +import type { VerifyStack } from './harness.js'; import { deriveCrudCases } from './derive.js'; const PROBE_TYPES = new Set(['text', 'textarea', 'string']); @@ -36,7 +36,7 @@ export interface RlsReport { } export async function runRlsProofs( - stack: DogfoodStack, + stack: VerifyStack, adminToken: string, memberToken: string, config: any, diff --git a/packages/dogfood/src/verify.ts b/packages/verify/src/verify.ts similarity index 98% rename from packages/dogfood/src/verify.ts rename to packages/verify/src/verify.ts index c28fa5f590..c660a039dd 100644 --- a/packages/dogfood/src/verify.ts +++ b/packages/verify/src/verify.ts @@ -9,7 +9,7 @@ /* eslint-disable @typescript-eslint/no-explicit-any */ -import type { DogfoodStack } from './harness.js'; +import type { VerifyStack } from './harness.js'; import { deriveCrudCases, type CrudCase } from './derive.js'; export interface ObjectVerifyResult { @@ -51,7 +51,7 @@ function deepEqual(a: unknown, b: unknown): boolean { * never throws on a per-object failure (collects them). */ export async function runCrudVerification( - stack: DogfoodStack, + stack: VerifyStack, token: string, config: any, ): Promise { diff --git a/packages/verify/tsconfig.json b/packages/verify/tsconfig.json new file mode 100644 index 0000000000..5e3095a8a4 --- /dev/null +++ b/packages/verify/tsconfig.json @@ -0,0 +1,9 @@ +{ + "extends": "../../tsconfig.json", + "compilerOptions": { + "outDir": "./dist", + "rootDir": "./src" + }, + "include": ["src/**/*"], + "exclude": ["node_modules", "dist", "**/*.spec.ts", "**/*.test.ts"] +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index cd91c9c19e..482b586cf3 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -547,6 +547,9 @@ importers: '@objectstack/types': specifier: workspace:* version: link:../types + '@objectstack/verify': + specifier: workspace:* + version: link:../verify '@oclif/core': specifier: ^4.11.4 version: 4.11.4 @@ -823,12 +826,6 @@ importers: packages/dogfood: dependencies: - '@objectstack/core': - specifier: workspace:* - version: link:../core - '@objectstack/driver-sqlite-wasm': - specifier: workspace:* - version: link:../plugins/driver-sqlite-wasm '@objectstack/example-crm': specifier: workspace:* version: link:../../examples/app-crm @@ -838,36 +835,15 @@ importers: '@objectstack/objectql': specifier: workspace:* version: link:../objectql - '@objectstack/plugin-auth': - specifier: workspace:* - version: link:../plugins/plugin-auth - '@objectstack/plugin-hono-server': - specifier: workspace:* - version: link:../plugins/plugin-hono-server - '@objectstack/plugin-org-scoping': - specifier: workspace:* - version: link:../plugins/plugin-org-scoping '@objectstack/plugin-security': specifier: workspace:* version: link:../plugins/plugin-security - '@objectstack/plugin-sharing': - specifier: workspace:* - version: link:../plugins/plugin-sharing - '@objectstack/rest': - specifier: workspace:* - version: link:../rest - '@objectstack/runtime': - specifier: workspace:* - version: link:../runtime - '@objectstack/service-analytics': - specifier: workspace:* - version: link:../services/service-analytics - '@objectstack/service-settings': - specifier: workspace:* - version: link:../services/service-settings '@objectstack/spec': specifier: workspace:* version: link:../spec + '@objectstack/verify': + specifier: workspace:* + version: link:../verify devDependencies: '@types/node': specifier: ^25.9.3 @@ -2206,6 +2182,55 @@ importers: specifier: ^4.1.9 version: 4.1.9(@opentelemetry/api@1.9.1)(@types/node@25.9.3)(@vitest/coverage-v8@4.1.9)(happy-dom@20.10.2)(msw@2.14.6(@types/node@25.9.3)(typescript@6.0.3))(vite@8.0.16(@types/node@25.9.3)(esbuild@0.28.1)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0)) + packages/verify: + dependencies: + '@objectstack/core': + specifier: workspace:* + version: link:../core + '@objectstack/driver-sqlite-wasm': + specifier: workspace:* + version: link:../plugins/driver-sqlite-wasm + '@objectstack/objectql': + specifier: workspace:* + version: link:../objectql + '@objectstack/plugin-auth': + specifier: workspace:* + version: link:../plugins/plugin-auth + '@objectstack/plugin-hono-server': + specifier: workspace:* + version: link:../plugins/plugin-hono-server + '@objectstack/plugin-org-scoping': + specifier: workspace:* + version: link:../plugins/plugin-org-scoping + '@objectstack/plugin-security': + specifier: workspace:* + version: link:../plugins/plugin-security + '@objectstack/plugin-sharing': + specifier: workspace:* + version: link:../plugins/plugin-sharing + '@objectstack/rest': + specifier: workspace:* + version: link:../rest + '@objectstack/runtime': + specifier: workspace:* + version: link:../runtime + '@objectstack/service-analytics': + specifier: workspace:* + version: link:../services/service-analytics + '@objectstack/service-settings': + specifier: workspace:* + version: link:../services/service-settings + '@objectstack/spec': + specifier: workspace:* + version: link:../spec + devDependencies: + '@types/node': + specifier: ^25.9.3 + version: 25.9.3 + typescript: + specifier: ^6.0.3 + version: 6.0.3 + packages/vscode-objectstack: devDependencies: '@types/vscode': From 1cc6e2c803dbade2265ddc25080c505c13e92233 Mon Sep 17 00:00:00 2001 From: os-zhuang Date: Fri, 19 Jun 2026 07:48:13 +0800 Subject: [PATCH 2/3] fix(verify): isolate the RLS-proof stack; document write-transform fidelity gaps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Surfaced by running `objectstack verify` against the real third-party hotcrm app: - The CLI ran data-fidelity and RLS proofs against the SAME in-process stack, so the RLS phase's admin-creates collided with rows the fidelity phase had already written on unique-constrained fields (e.g. a unique `sku`/`account_number`) → a 409 that silently skipped those objects' authorization check. Boot a separate pristine stack for the RLS phase. hotcrm `--rls --multi-tenant` goes from 5-consistent/3-skipped(409) to 8-consistent/0-holes. - Document the known limitation that fields normalized on write (uppercase/trim hooks, canonicalizing formulas) read back as `fidelity-gaps` even when working as designed (hotcrm's `sku`/`account_number` uppercase hooks). Co-Authored-By: Claude Opus 4.8 --- packages/cli/src/commands/verify.ts | 35 ++++++++++++++++++++--------- packages/verify/README.md | 14 ++++++++++++ 2 files changed, 38 insertions(+), 11 deletions(-) diff --git a/packages/cli/src/commands/verify.ts b/packages/cli/src/commands/verify.ts index 93f721a0d3..c81c5929ca 100644 --- a/packages/cli/src/commands/verify.ts +++ b/packages/cli/src/commands/verify.ts @@ -58,20 +58,33 @@ export default class Verify extends Command { String(readEnvWithDeprecation('OS_MULTI_ORG_ENABLED', 'OS_MULTI_TENANT') ?? 'false').toLowerCase() !== 'false'; - const stack = await bootStack(config, { multiTenant }); - + // Data fidelity runs on its own pristine stack. let crud: VerifyReport; - let rls: RlsReport | undefined; - try { - const adminToken = await stack.signIn(); - crud = await runCrudVerification(stack, adminToken, config); + { + const stack = await bootStack(config, { multiTenant }); + try { + const adminToken = await stack.signIn(); + crud = await runCrudVerification(stack, adminToken, config); + } finally { + await stack.stop(); + } + } - if (flags.rls) { - const memberToken = await stack.signUp('verify-member@objectstack.test'); - rls = await runRlsProofs(stack, adminToken, memberToken, config); + // The RLS proofs run on a SEPARATE, fresh stack. Reusing the fidelity stack + // would let the RLS phase's admin-creates collide with the rows the fidelity + // phase already wrote on unique-constrained fields (e.g. a unique `sku` or + // `account_number`) — a 409 that silently skips the object instead of + // proving its authorization. + let rls: RlsReport | undefined; + if (flags.rls) { + const rlsStack = await bootStack(config, { multiTenant }); + try { + const adminToken = await rlsStack.signIn(); + const memberToken = await rlsStack.signUp('verify-member@objectstack.test'); + rls = await runRlsProofs(rlsStack, adminToken, memberToken, config); + } finally { + await rlsStack.stop(); } - } finally { - await stack.stop(); } // Failure contract: a "real" runtime break the app's author must see. diff --git a/packages/verify/README.md b/packages/verify/README.md index 984674694f..dd3669bfbb 100644 --- a/packages/verify/README.md +++ b/packages/verify/README.md @@ -100,3 +100,17 @@ isolation policies actually apply. `bootStack` options: `admin`, `authSecret`, `security` (a custom `SecurityPlugin` for owner-scoped fixtures), `multiTenant`. + +## Known limitations + +- **Intentional write-transforms read back as fidelity gaps.** The fidelity + check asserts an *exact* round-trip, so a field normalized on write — an + `uppercase`/`trim` hook, a canonicalizing formula — is reported as a + `fidelity-gaps` mismatch (e.g. `sku`: wrote `"abc-1"` → read `"ABC-1"`) and + fails the run, even though the app behaves as designed. The report shows the + exact `wrote → read` diff so it's diagnosable; letting an app declare such + fields so the verifier can allow them is a planned enhancement. +- **The auto-derived sweep is coarser than a hand-written matrix.** It exercises + one synthesized record per object and skips fields it can't synthesize + (required lookups / master-detail, media, computed). It's a broad runtime + smoke test, not a substitute for targeted golden tests of specific behavior. From f9d495fc7edcac7eac1999b7591878d93af55e15 Mon Sep 17 00:00:00 2001 From: os-zhuang Date: Fri, 19 Jun 2026 08:10:30 +0800 Subject: [PATCH 3/3] =?UTF-8?q?docs(adr):=20ADR-0054=20=E2=80=94=20record?= =?UTF-8?q?=20the=20proof=20engine's=20extraction=20into=20@objectstack/ve?= =?UTF-8?q?rify?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-0054 assigned the prove-it-runs mechanism to @objectstack/dogfood; this PR extracted that engine into the published @objectstack/verify. Append an Update addendum (and list it as a consumer) mapping the package to the ADR's phases — dogfood stays the gate (hand-written golden proofs) and now runs on the verify engine; Phase 1's field/RLS matrix is the published derive/verify/rls; Phase 3's generative pass has deriveCrudCases as its seed — and stating the honest scope of the auto-derived path (scalar round-trip + by-id RLS only) so it doesn't read as subsuming the golden proofs. Decision unchanged. Co-Authored-By: Claude Opus 4.8 --- ...54-runtime-proof-for-authorable-surface.md | 44 ++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/docs/adr/0054-runtime-proof-for-authorable-surface.md b/docs/adr/0054-runtime-proof-for-authorable-surface.md index ed1d96e94c..48844b110a 100644 --- a/docs/adr/0054-runtime-proof-for-authorable-surface.md +++ b/docs/adr/0054-runtime-proof-for-authorable-surface.md @@ -3,7 +3,7 @@ **Status**: Accepted (2026-06-18) **Deciders**: ObjectStack Protocol Architects **Builds on**: [ADR-0049](./0049-no-unenforced-security-properties.md) (enforce-or-remove gate), [ADR-0005](./0005-metadata-customization-overlay.md) (artifact vs runtime), [ADR-0053](./0053-date-and-datetime-semantics.md) (the domain of the motivating regression) -**Consumers**: `@objectstack/spec` (liveness ledger `packages/spec/liveness/.json`), the Spec Liveness Check CI gate (#1919), `@objectstack/dogfood` (the runtime gate, [#2020](https://github.com/objectstack-ai/framework/pull/2020)), spec authors, platform contributors. +**Consumers**: `@objectstack/spec` (liveness ledger `packages/spec/liveness/.json`), the Spec Liveness Check CI gate (#1919), `@objectstack/dogfood` (the runtime gate, [#2020](https://github.com/objectstack-ai/framework/pull/2020)), `@objectstack/verify` (the published proof engine + CLI, [#2041](https://github.com/objectstack-ai/framework/pull/2041)), spec authors, platform contributors. **Surfaced by**: PR [#2018](https://github.com/objectstack-ai/framework/pull/2018) — "organization timezone drives analytics date bucketing" was **green on every static gate** (build, ~900 unit tests, spec-liveness, CodeQL) yet broken end-to-end across three integration seams; and the field-type capability-matrix dogfood ([#2022](https://github.com/objectstack-ai/framework/pull/2022)), which on its first run found `rating`/`slider`/`toggle` reading back wrong-typed. --- @@ -156,3 +156,45 @@ the proof corpus stays CI-cheap as it grows. server-reachable behavior. Pure objectui/React render correctness belongs in objectui's own suite; a property whose only failure mode is client render is out of scope for this gate. + + +--- + +## Update (2026-06-19) — the proof engine is now `@objectstack/verify` + +The proof *mechanism* this ADR assigns to `@objectstack/dogfood` has since been +extracted into a published, app-agnostic package: **`@objectstack/verify`** +([#2041](https://github.com/objectstack-ai/framework/pull/2041)) — `bootStack` +(the real in-process stack via Hono request-injection), `deriveCrudCases` +(a runtime contract auto-derived from any app's metadata), `runCrudVerification` +(write → read → assert type fidelity), and `runRlsProofs` (the #1994 +"can't-write-what-you-can't-read" invariant) — plus an `objectstack verify` CLI. + +This sharpens the decision without changing it: + +- **The gate vs. the engine.** `@objectstack/dogfood` remains the *gate* — the + framework's own **hand-written golden proofs** (e.g. the #2018 tz-bucketing + test, which `derive` can never auto-generate) — and now runs *on* the + `@objectstack/verify` engine instead of carrying it. A `proof` (§1) is still a + dogfood test; what changed is that the harness underneath it is reusable. +- **Phase 1 is now a reusable matrix.** The field-type matrix (#2022) is the + published `deriveCrudCases` + `runCrudVerification`; the #1994 RLS seed is + `runRlsProofs`. They are no longer internal to dogfood. +- **Phase 3 has a concrete vehicle.** `deriveCrudCases` — metadata → synthesized + record → asserted round-trip — *is* the seed of the deferred generative pass; + Phase 3 grows it rather than starting from zero. +- **Third parties get the same gate.** Because it is published, a third-party or + template author runs the identical proofs against their own app + (`objectstack verify --rls`), extending *prove-it-runs* beyond the framework's + curated examples — the AI-authoring audience this ADR is written for. Validated + against the external `hotcrm` app and the 9-app template corpus (#2041). + +**Honest scope of the *auto-derived* path.** `@objectstack/verify`'s auto-derive +asserts only **scalar field round-trip fidelity** and the **by-id RLS invariant**, +and **skips** objects whose required fields it can't synthesize (lookups / +master-detail) and field classes it can't assert (computed/formula, flow nodes, +analytics bucketing, UI). A green `objectstack verify` therefore proves those two +dimensions over the auto-reachable subset — it does **not** subsume the +hand-written golden proofs, which is exactly why §4's gate keeps them. Closing +that gap (related-record topological synthesis; computed/flow/analytics +assertions) is the substance of Phases 2–3.