fix(cli,driver-sql): defer migrate DDL until confirmed + detect SQLite occupancy (#3917)

.changeset/migrate-occupancy-and-deferred-ddl.md

@@ -0,0 +1,37 @@
+---
+'@objectstack/driver-sql': minor
+'@objectstack/runtime': minor
+'@objectstack/cli': minor
+---
+
+`os migrate` no longer touches the database before you confirm, and refuses a
+SQLite database another process is using (#3917).
+
+**Nothing is written before the prompt.** `plan` called itself a dry run and
+`apply` gated on `[y/N]`, but both booted the full plugin set first — and boot
+schema-sync issued create-table/add-column DDL (plus the artifact's inline seed
+wrote rows) against the target database before either promise was kept.
+`SqlDriver` gains `setDeferredDdl` / `previewDeferredSchemaWork` /
+`flushDeferredSchemaDdl`: while armed, `initObjects` still registers every
+in-memory map drift detection depends on but records the physical work instead
+of performing it. Both commands boot with it armed, render the held-back work
+as a `New (additive)` section of the plan, and `apply` performs it only after
+confirmation. `os meta resync` / `os migrate files-to-references` keep the old
+behaviour — they need the tables to exist.
+
+**Occupancy check.** A live `os dev`/`os serve` holding the same SQLite file is
+the usual way a migration goes wrong: the migration is transactional and swaps
+tables inside the file, but the running server keeps prepared statements and a
+schema cookie the migration invalidates. `os migrate` now probes the target
+before booting — `PRAGMA locking_mode = EXCLUSIVE` + `BEGIN IMMEDIATE` under
+`busy_timeout = 0`, which reports `SQLITE_BUSY` when another connection is
+*attached*, not merely writing. (`wal_checkpoint(TRUNCATE)` only sees an active
+writer, and `-wal`/`-shm` presence cannot tell a live server from a crashed one;
+both are encoded as tests.) `apply` refuses with exit 1 — `error: database_busy`
+under `--json` — unless the new `--force` flag is passed; `plan` warns and
+continues, since it writes nothing either way. SQLite only: Postgres and MySQL
+take their own server-side locks.
+
+`@objectstack/runtime` also exports `resolveStandaloneDatabase()`, so a caller
+can resolve the database target with the same precedence the boot uses without
+building the stack, and `createStandaloneStack` accepts `skipSeedData`.

content/docs/deployment/cli.mdx

@@ -494,9 +494,49 @@ os migrate plan # Preview drift (no changes)
 os migrate apply # Apply safe (loosening) changes, with a confirm prompt
 os migrate apply --yes # Skip the prompt (CI / scripts)
 os migrate apply --allow-destructive --yes # Also drop orphaned columns, tighten NOT NULL, narrow types
+os migrate apply --force # Migrate even though another process is using the database
 os migrate plan --json # Machine-readable output
 ```
 
+#### Nothing is written before you confirm
+
+Both commands boot your app to read its metadata. That boot no longer touches
+the target database: the additive schema sync (create missing tables, add
+missing columns) and the artifact's inline seed data are **deferred**, not
+performed. So `plan` really is a dry run, and everything `apply` is about to do
+— additive work included — is on screen before the `[y/N]` prompt:
+
+```text
+ New (additive — created when you apply)
+ + crm_quote [create_table, 9 column(s)]
+ + crm_contact [add_columns: nickname, region]
+
+ Safe (loosening — applied without --allow-destructive)
+ ✓ crm_contact.email [relax_not_null]
+```
+
+Answering `n` leaves the database exactly as it was.
+
+#### Occupancy check (SQLite)
+
+A running `os dev` / `os serve` holding the same SQLite file open is the usual
+way a migration goes wrong: the migration itself is transactional and swaps
+tables *inside* the file, but the live server keeps prepared statements and a
+schema cookie that the migration invalidates, and its writes can collide as
+`SQLITE_BUSY`. Before booting, `os migrate` asks the database whether anyone
+else is attached (`PRAGMA locking_mode = EXCLUSIVE` under `busy_timeout = 0` —
+non-destructive, and unlike a WAL checkpoint it also sees a connection that is
+merely *open* rather than actively writing).
+
+| Command | If the database is in use |
+|---------|---------------------------|
+| `os migrate plan` | Warns and continues — a plan writes nothing either way |
+| `os migrate apply` | **Refuses** (exit 1, `error: database_busy` under `--json`). Stop the other process, or pass `--force` |
+
+The check applies to SQLite only. Postgres and MySQL take their own server-side
+locks, and a `-wal`/`-shm` left behind by a crashed process is deliberately not
+treated as occupancy on its own.
+
 | Category | Examples | Applied by |
 |----------|----------|------------|
 | `safe` | relax `NOT NULL` → nullable, widen a `varchar`, create a declared index, replace a legacy global unique with its tenant-scoped composite | `os migrate apply` (and dev auto-reconcile) |

packages/cli/package.json

@@ -100,9 +100,13 @@
  "yaml": "^2.9.0",
  "zod": "^4.4.3"
  },
+ "optionalDependencies": {
+ "better-sqlite3": "^13.0.1"
+ },
  "devDependencies": {
  "@oclif/plugin-help": "^6.2.55",
  "@oclif/plugin-plugins": "^5.4.86",
+ "@types/better-sqlite3": "^7.6.13",
  "@types/node": "^26.1.1",
  "tsup": "^8.5.1",
  "typescript": "^6.0.3",

packages/cli/src/commands/migrate/apply.ts

@@ -16,9 +16,13 @@ import {
 import {
  bootSchemaStack,
  renderPlan,
+ renderPendingSchemaWork,
  summarize,
+ summarizePendingSchemaWork,
  groupByCategory,
 } from '../../utils/schema-migrate.js';
+import { OCCUPANCY_HINT, probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js';
+import { describeOccupancy } from '../../utils/sqlite-occupancy.js';
 
 async function confirm(question: string): Promise<boolean> {
  if (!process.stdin.isTTY) return false; // non-interactive → require --yes
@@ -36,6 +40,17 @@ async function confirm(question: string): Promise<boolean> {
  * Applies safe (loosening) + needs-confirm changes by default; destructive
  * changes (drop column, tighten NOT NULL, narrow type) require
  * `--allow-destructive`.
+ *
+ * Two operational-safety guarantees, both added by #3917:
+ *
+ * 1. **Nothing is written before you say yes.** The stack boots with schema
+ * DDL deferred and the artifact seed suppressed, so the additive
+ * create-table / add-column work that used to run during boot is now part
+ * of the plan you confirm — not something that already happened by the time
+ * the prompt appeared.
+ * 2. **A database somebody else is using is not migrated by accident.** The
+ * SQLite target is probed for other attached connections before boot, and a
+ * busy database refuses without `--force`.
  */
 export default class MigrateApply extends Command {
  static override description =
@@ -45,6 +60,7 @@ export default class MigrateApply extends Command {
  '$ os migrate apply',
  '$ os migrate apply --yes',
  '$ os migrate apply --allow-destructive --yes',
+ '$ os migrate apply --force',
  '$ os migrate apply --json',
  ];
 
@@ -58,6 +74,10 @@ export default class MigrateApply extends Command {
  default: false,
  }),
  yes: Flags.boolean({ char: 'y', description: 'Skip the confirmation prompt', default: false }),
+ force: Flags.boolean({
+ description: 'Migrate even when another process is using the database (SQLite occupancy check)',
+ default: false,
+ }),
  json: Flags.boolean({ description: 'Output as JSON (implies non-interactive; requires --yes to mutate)' }),
  };
 
@@ -68,12 +88,45 @@ export default class MigrateApply extends Command {
 
  if (!flags.json) {
  printHeader('Migrate · apply');
+ printStep('Checking whether the database is in use…');
+ }
+
+ // Occupancy gate — BEFORE the stack boots, or our own pooled connections
+ // are what the probe finds (#3917).
+ const occupancy = await probeMigrationTarget(flags['database-url']);
+ if (occupancy.status === 'busy' && !flags.force) {
+ if (flags.json) {
+ await emitJson({
+ error: 'database_busy',
+ database: occupancy.filename,
+ signal: occupancy.signal,
+ detail: occupancy.detail,
+ hint: OCCUPANCY_HINT,
+ }, 0, { compact: true });
+ this.exit(1);
+ return;
+ }
+ printError(describeOccupancy(occupancy));
+ printWarning(OCCUPANCY_HINT);
+ this.exit(1);
+ return;
+ }
+ if (occupancy.status === 'busy' && flags.force && !flags.json) {
+ printWarning(`--force: ${describeOccupancy(occupancy)} Migrating anyway — the live process may see stale schema or SQLITE_BUSY.`);
+ }
+ if (occupancy.status === 'unknown' && !flags.json) {
+ printWarning(`Could not check whether the database is in use — ${occupancy.detail}`);
+ }
+
+ if (!flags.json) {
  printStep('Booting schema stack…');
  }
 
  let stack;
  try {
- stack = await bootSchemaStack({ databaseUrl: flags['database-url'] });
+ // `deferSchemaDdl` is what makes the prompt below meaningful: without it
+ // the boot has already created tables and added columns by this point.
+ stack = await bootSchemaStack({ databaseUrl: flags['database-url'], deferSchemaDdl: true });
  } catch (error: any) {
  if (flags.json) { await emitJson({ error: error.message }, 0, { compact: true }); this.exit(1); }
  printError(error.message || String(error));
@@ -90,9 +143,13 @@ export default class MigrateApply extends Command {
 
  const drift = await stack.driver.detectManagedDrift();
  const grouped = groupByCategory(drift);
+ // Additive work the boot sync was held back from doing. Not drift — it
+ // is what `initObjects` does on its own — but it IS a change to the
+ // target database, so it belongs in the plan and behind the prompt.
+ const pending = stack.pendingSchemaWork;
 
- if (drift.length === 0) {
- if (flags.json) { await emitJson({ applied: [], skipped: [], message: 'in_sync' }, 0, { compact: true }); return; }
+ if (drift.length === 0 && pending.length === 0) {
+ if (flags.json) { await emitJson({ applied: [], skipped: [], created: [], message: 'in_sync' }, 0, { compact: true }); return; }
  printSuccess('Physical schema is already in sync with metadata — nothing to apply.');
  return;
  }
@@ -104,7 +161,9 @@ export default class MigrateApply extends Command {
  if (!flags.json) {
  printInfo(`Database: ${chalk.white(stack.dbLabel)}`);
  console.log('');
+ renderPendingSchemaWork(pending);
  renderPlan(drift);
+ if (pending.length > 0) printInfo(summarizePendingSchemaWork(pending));
  printInfo(summarize(drift));
  if (deferred.length > 0) {
  printWarning(`${deferred.length} destructive change(s) will be SKIPPED (re-run with --allow-destructive to include them).`);
@@ -114,28 +173,35 @@ export default class MigrateApply extends Command {
  }
  }
 
- if (intended.length === 0) {
- if (flags.json) { await emitJson({ applied: [], skipped: deferred, message: 'nothing_safe_to_apply' }, 0, { compact: true }); return; }
+ const totalIntended = intended.length + pending.length;
+ if (totalIntended === 0) {
+ if (flags.json) { await emitJson({ applied: [], skipped: deferred, created: [], message: 'nothing_safe_to_apply' }, 0, { compact: true }); return; }
  printWarning('No changes to apply without --allow-destructive.');
  return;
  }
 
- // Confirmation gate.
+ // Confirmation gate. Nothing above this line has touched the database.
  if (!flags.yes) {
  if (flags.json || !process.stdin.isTTY) {
- if (flags.json) { await emitJson({ applied: [], skipped: drift, message: 'confirmation_required', hint: 'pass --yes' }, 0, { compact: true }); return; }
+ if (flags.json) { await emitJson({ applied: [], skipped: drift, pending, message: 'confirmation_required', hint: 'pass --yes' }, 0, { compact: true }); return; }
  printWarning('Confirmation required. Re-run with --yes to apply, or use "os migrate plan" to preview.');
  return;
  }
- const ok = await confirm(chalk.bold(`\nApply ${intended.length} change(s) to ${stack.dbLabel}? [y/N] `));
+ const ok = await confirm(chalk.bold(`\nApply ${totalIntended} change(s) to ${stack.dbLabel}? [y/N] `));
  if (!ok) { printInfo('Aborted — no changes made.'); return; }
  }
 
+ // Additive work first: a table has to exist before its columns can be
+ // reconciled. Drift was detected against the pre-flush database, and a
+ // just-created table matches metadata by construction, so the two sets
+ // never overlap.
+ const created = await stack.flushSchemaDdl();
  const { applied, skipped } = await stack.driver.applyMigrationEntries(drift, { allowDestructive });
 
  if (flags.json) {
  await emitJson({
  database: stack.dbLabel,
+ created,
  applied,
  skipped,
  duration: timer.elapsed(),
@@ -144,6 +210,9 @@ export default class MigrateApply extends Command {
  }
 
  console.log('');
+ if (created.length > 0) {
+ printSuccess(`Created/extended ${created.length} table(s): ${summarizePendingSchemaWork(created)}.`);
+ }
  printSuccess(`Applied ${applied.length} change(s).`);
  if (skipped.length > 0) {
  printWarning(`Skipped ${skipped.length} change(s) (destructive without --allow-destructive, or unsupported on this dialect).`);

packages/cli/src/commands/migrate/plan.ts

@@ -12,12 +12,27 @@ import {
  createTimer,
  emitJson,
 } from '../../utils/format.js';
-import { bootSchemaStack, renderPlan, summarize } from '../../utils/schema-migrate.js';
+import {
+ bootSchemaStack,
+ renderPlan,
+ renderPendingSchemaWork,
+ summarize,
+ summarizePendingSchemaWork,
+} from '../../utils/schema-migrate.js';
+import { probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js';
+import { describeOccupancy } from '../../utils/sqlite-occupancy.js';
 
 /**
  * `os migrate plan` — dry-run diff of metadata vs the physical database,
  * categorised safe / needs-confirm / destructive (issue #2186). Never mutates
  * the schema.
+ *
+ * "Never mutates" is enforced rather than merely documented since #3917: the
+ * stack boots with schema DDL deferred and the artifact seed suppressed, so the
+ * boot-time create-table / add-column sync that used to run before this command
+ * printed a single line is now REPORTED as pending work instead of performed.
+ * A database another process is using is reported too — as a warning, not a
+ * refusal, since a plan writes nothing either way.
  */
 export default class MigratePlan extends Command {
  static override description =
@@ -46,9 +61,16 @@ export default class MigratePlan extends Command {
  printStep('Booting schema stack…');
  }
 
+ // Probed before boot so the answer is about somebody else's connections,
+ // not our own pool.
+ const occupancy = await probeMigrationTarget(flags['database-url']);
+ if (occupancy.status === 'busy' && !flags.json) {
+ printWarning(`${describeOccupancy(occupancy)} The plan below is still accurate — nothing is written — but "os migrate apply" will refuse until it is free (or you pass --force).`);
+ }
+
  let stack;
  try {
- stack = await bootSchemaStack({ databaseUrl: flags['database-url'] });
+ stack = await bootSchemaStack({ databaseUrl: flags['database-url'], deferSchemaDdl: true });
  } catch (error: any) {
  if (flags.json) { await emitJson({ error: error.message }, 0, { compact: true }); this.exit(1); }
  printError(error.message || String(error));
@@ -64,13 +86,18 @@ export default class MigratePlan extends Command {
  }
 
  const drift = await stack.driver.detectManagedDrift();
+ const pending = stack.pendingSchemaWork;
 
  if (flags.json) {
  await emitJson({
  database: stack.dbLabel,
  managedTables: stack.managedTableCount,
  total: drift.length,
  changes: drift,
+ pending,
+ ...(occupancy.status === 'busy'
+ ? { occupancy: { status: 'busy', signal: occupancy.signal, detail: occupancy.detail } }
+ : {}),
  duration: timer.elapsed(),
  });
  return;
@@ -80,13 +107,15 @@ export default class MigratePlan extends Command {
  printInfo(`Examined ${chalk.white(String(stack.managedTableCount))} managed table(s).`);
  console.log('');
 
- if (drift.length === 0) {
+ if (drift.length === 0 && pending.length === 0) {
  printSuccess('Physical schema is in sync with metadata — nothing to migrate.');
  console.log('');
  return;
  }
 
+ renderPendingSchemaWork(pending);
  renderPlan(drift);
+ if (pending.length > 0) printInfo(summarizePendingSchemaWork(pending));
  printInfo(summarize(drift));
  console.log(chalk.dim(' Apply with: ') + chalk.white('os migrate apply') +
  chalk.dim(' (add --allow-destructive for drops / tightenings)'));