Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions .changeset/adr-0119-d2-migration-journal.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
---
"@objectstack/spec": minor
"@objectstack/platform-objects": minor
"@objectstack/core": minor
"@objectstack/metadata-protocol": patch
---

feat(core,platform-objects,spec): the ADR-0119 D2 migration-journal runner — a migration killed mid-run is resumable to completion or compensable to clean, with journal rows proving which (#4617)

**The gap D1 left open.** ADR-0119 D1 made `engine.transaction()` reachable
through the contract, which is the right answer for multi-write atomicity that
fits in one transaction. Migration-class work does not fit: a million-row
backfill cannot hold one write-lock for its duration, `driver-memory`'s
`beginTransaction` deep-clones the entire database (O(db) per begin),
`ObjectQL.transaction()` binds the **default driver only** so a multi-datasource
migration silently commits part of its work outside it, and a process **killed**
— as distinct from a thrown error — defeats in-process rollback entirely. So the
unit of atomicity is the *chunk*, and durability across chunks is a journal.

Four consumers had each converged on the same four moves — dry-run preflight,
undo journal, LIFO compensation, re-entrant forward recovery (ADR-0105 D13
promotion, ADR-0117 D8's ownership backfill, the org lifecycle transitions, and
D10 master-data distribution #4585). One copy is engineering; four is platform
debt, and the fourth author would have had to rediscover the invariant below
from scratch.

**New: `runMigrationJournal` (`@objectstack/core`).** Preflight runs every
step's read-only validator before any step writes, so a plan that would fail at
step 3 has not written step 1. Rows are chunked per the `bulk-write.ts`
discipline; each chunk's writes run inside `engine.transaction()`. On failure,
committed chunks are compensated newest-first, each in its own transaction. On
restart, a rediscovered run resumes forward from the first chunk lacking
`chunk_done`, or unwinds, per the plan's `onCrash` policy. Forward and
compensate callbacks receive an `attempt` counter; `attempt > 1` means the prior
outcome is UNKNOWN and the callback must recheck by natural key before
re-writing — the same at-least-once contract `bulk-write.ts` already documents,
reused rather than re-derived.

**The invariant that carries the design:** `chunk_done(i)` is written **inside**
the chunk's own transaction, so `done ⇔ committed` holds by construction;
`chunk_started(i)` is written autonomously **before** it. That asymmetry is what
gives `started ∧ ¬done` exactly one meaning — *the outcome is unknown* — which
is the only state a crash can leave and the only state recovery reasons about.
Making both writes symmetric would look tidier and would destroy recovery.

**New: `sys_migration_journal` (`@objectstack/platform-objects`).** Rows keyed
`(run_id, seq)` under a unique index, so a resumed run that miscomputes its next
sequence fails loudly rather than double-recording an event. Registered
unconditionally alongside `sys_migration` because recovery must be discoverable
with **zero host wiring** — a journal some kernels compose and others do not is
a journal a boot scanner cannot rely on (ADR-0078). Distinct in grain from
`sys_migration`, which holds one durable verdict per named migration; this holds
many rows per *run*. Read-only over the API; writes go through the runner in
system context.

**The runner refuses rather than degrades**, in four places: the runtime cannot
roll back; any preflight fails; the plan declares `onCrash: 'compensate'` but a
step cannot compensate; or a resume's plan hash disagrees with the journal
(resuming a changed plan would apply chunk boundaries the journal never
described). A compensation failure halts and is journalled — never swallowed —
and the run ends `failed`, not `compensated`, because a database in a state no
clean story covers must not be reported as a tidy rollback.

**`engineCanRollBack` is now shared.** The two-level probe (engine method AND
default-driver `beginTransaction`) was the same condition written twice — here
and in `batchData`'s atomic gate. It now lives in `@objectstack/core` and
`@objectstack/metadata-protocol` imports it, as a type predicate so callers do
not each re-narrow the optional member by hand. Two copies of "can this runtime
actually roll back?" drift by one clause and leave one caller believing it has
atomicity it does not have.

Boot reconciliation and `os migrate resume` land separately; `findInterruptedRuns`
is the discovery primitive they will consume, and is exported here.

**Docs:** ADR-0118 (plugin-reachable transactions) is renumbered **ADR-0119**.
It merged one day after an unrelated ADR-0118 (非用户 actor 的平台契约) and the
earlier merge holds the number; citations of "ADR-0118 D1/D2/D3/D4" written
before 2026-08-03 mean the renumbered record.
Original file line numberDiff line numberDiff line change
Expand Up@@ -3,7 +3,7 @@
"@objectstack/metadata-protocol": minor
---

feat(spec,metadata-protocol): `IObjectQLEngine.transaction` joins the slot contract, and `batchData`'s `atomic` flag becomes real — rollback or refusal, never silent best-effort (ADR-0118 D1/D4, #4612)
feat(spec,metadata-protocol): `IObjectQLEngine.transaction` joins the slot contract, and `batchData`'s `atomic` flag becomes real — rollback or refusal, never silent best-effort (ADR-0119 D1/D4, #4612)

**D1 — the contract fix.** `ObjectQL.transaction()` — ADR-0034's ambient
transaction, shipped since v8.0.0 — was reachable from plugin space only
Expand DownExpand Up@@ -53,7 +53,7 @@ If you were passing `atomic: true` and relying on partial results surviving a
failure, that was the bug — switch to `atomic: false` (or omit it) for
best-effort semantics.

ADR-0118 also rules on two items landing separately: D2 specifies a
ADR-0119 also rules on two items landing separately: D2 specifies a
framework-owned migration-journal runner for multi-step migrations too large
for one transaction, and D3 retires the declared-but-unimplemented
`IDataEngine.batch?`.
25 changes: 23 additions & 2 deletions content/docs/references/system/migration.mdx
Original file line numberDiff line numberDiff line change
Expand Up@@ -34,8 +34,8 @@ irreversibly on migrated data gate on the flag instead of the version.
## TypeScript Usage

```typescript
import { AddFieldOperation, ChangeSetSchema, CreateObjectOperation, DataMigrationFlagSchema, DeleteObjectOperation, ExecuteSqlOperation, MigrationDependencySchema, MigrationOperationSchema, ModifyFieldOperation, RemoveFieldOperation, RenameObjectOperation } from '@objectstack/spec/system';
import type { ChangeSet, DataMigrationFlag, MigrationOperation } from '@objectstack/spec/system';
import { AddFieldOperation, ChangeSetSchema, CreateObjectOperation, DataMigrationFlagSchema, DeleteObjectOperation, ExecuteSqlOperation, MigrationDependencySchema, MigrationJournalEventSchema, MigrationOperationSchema, ModifyFieldOperation, RemoveFieldOperation, RenameObjectOperation } from '@objectstack/spec/system';
import type { ChangeSet, DataMigrationFlag, MigrationJournalEvent, MigrationOperation } from '@objectstack/spec/system';

// Validate data
const result = AddFieldOperation.parse(data);
Expand DownExpand Up@@ -153,6 +153,27 @@ Dependency reference to another migration that must run first
| **package** | `string` | optional | Package that owns the dependency migration |


---

## MigrationJournalEvent

One event in a migration run journal — the durable trace that lets a killed run be resumed forward or compensated back, with rows proving which

### Properties

| Property | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| **run_id** | `string` | ✅ | Identifies one run. Rows are keyed (run_id, seq) |
| **seq** | `integer` | ✅ | Monotonic per-run sequence. Ordering authority — wall-clock timestamps can tie or skew |
| **kind** | `Enum<'run_started' \| 'chunk_started' \| 'chunk_done' \| 'compensated' \| 'run_done' \| 'run_failed'>` | ✅ | Event kind |
| **migration_id** | `string` | optional | The named migration this run belongs to, when it has one — joins to sys_migration.id |
| **plan_hash** | `string` | optional | On run_started: hash of the plan shape. A resume whose plan hash differs REFUSES rather than resuming a changed plan against an old journal |
| **chunk_index** | `integer` | optional | On chunk_started / chunk_done / compensated: the run-global chunk index |
| **attempt** | `integer` | optional | Which attempt produced this event. attempt > 1 means a prior outcome was unknown and the callback was asked to recheck by natural key |
| **detail** | `string` | optional | JSON-encoded payload — the chunk plan on run_started, the error on run_failed / a failed compensation |
| **created_at** | `string` | optional | Wall-clock stamp, for humans. Never the ordering authority — that is seq |


---

## MigrationOperation
Expand Down
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
# ADR-0118: Multi-write atomicity is reachable through the contract, `atomic` means atomic or refuses, and migrations too big for one transaction get a journal runner
# ADR-0119: Multi-write atomicity is reachable through the contract, `atomic` means atomic or refuses, and migrations too big for one transaction get a journal runner

**Status**: Accepted (2026-08-02) — D1/D4 implemented in this PR; D2 tracked in [#4617](https://github.com/objectstack-ai/objectstack/issues/4617); D3 tracked in [#4618](https://github.com/objectstack-ai/objectstack/issues/4618)
**Status**: Accepted (2026-08-02) — D1/D4 implemented in [#4623](https://github.com/objectstack-ai/objectstack/pull/4623): D1 in `packages/spec/src/contracts/objectql-engine.ts` (test `packages/objectql/src/protocol-batch-atomic.test.ts`), D4 in `packages/metadata-protocol/src/protocol.ts` (test `packages/metadata-protocol/src/protocol.batch-atomic.test.ts`). D2 tracked in [#4617](https://github.com/objectstack-ai/objectstack/issues/4617); D3 tracked in [#4618](https://github.com/objectstack-ai/objectstack/issues/4618) — neither is implemented, so this record is *not* wholly "implemented".
**Renumbered**: published for one day as ADR-0118. Renumbered to 0119 because [ADR-0118 (非用户 actor 的平台契约)](./0118-non-user-actor-contract.md) merged first (10:37 vs 12:11 on 2026-08-02) and holds the number. Citations of "ADR-0118 D1/D2/D3/D4" written before 2026-08-03 mean this record.
**Deciders**: ObjectStack Protocol Architects
**Builds on**: [ADR-0034](./0034-transactional-writes-and-ambient-transaction.md) (the ambient `AsyncLocalStorage` transaction D1 declares — this ADR adds no mechanism to it), [ADR-0067](./0067-commit-history-and-rollback-for-ai-authoring.md) (D2 — the join-don't-nest rule that makes an outer transaction the sole owner of commit/rollback), [ADR-0049](./0049-no-unenforced-security-properties.md) (enforce-or-remove — the disposition method applied to `batch?` in D3 and to the `atomic` flag in D4), [ADR-0087](./0087-metadata-protocol-upgrade-contract.md) (D3's replayable migration chain — the metadata-side analogue of the data-side runner D2 specifies), [ADR-0008](./0008-metadata-repository-and-change-log.md) (the JSONL change log — the journal shape D2 deliberately does *not* reuse), [ADR-0060](./0060-conformance-ledger-platform-pattern.md) (framework-owned ledger pattern — the precedent for `sys_migration_journal`), [ADR-0117](./0117-owning-business-unit-record-stamp.md) (D8 — backfill plus a fail-closed enable gate, the migration posture D2 and D4 both inherit), [ADR-0078](./0078-no-silently-inert-metadata.md) (no silently inert declarations — why D2 rejects a pluggable journal store)
**Consumers**: `@objectstack/spec` (`contracts/objectql-engine.ts`, `api/batch.zod.ts`), `@objectstack/metadata-protocol` (`protocol.ts`, `host-engine.ts`, `sys-metadata-repository.ts`), `@objectstack/objectql` (the implementation — unchanged by D1), `@objectstack/rest` (the `/batch` routes — unchanged), and for D2: `@objectstack/core`, `@objectstack/platform-objects`
Expand Down
5 changes: 5 additions & 0 deletions packages/core/src/index.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -30,6 +30,11 @@ export * from './utils/datetime.js';
// Export the shared batched-write helper (framework#2678)
export * from './utils/bulk-write.js';

// Export the migration-journal runner (ADR-0119 D2, #4617) — chunk-atomic
// migrations with durable recovery, plus the shared `engineCanRollBack` gate
// that `@objectstack/metadata-protocol`'s atomic `batchData` also uses.
export * from './utils/migration-journal.js';

// Export the runtime filter-placeholder resolver (framework#3582)
export * from './utils/filter-tokens.js';

Expand Down
Loading
Loading