feat(spec,metadata-protocol): ADR-0118 — transactions reach plugin space through the contract, and atomic stops lying (#4612)

.changeset/adr-0118-plugin-reachable-transactions.md

@@ -0,0 +1,59 @@
+---
+"@objectstack/spec": minor
+"@objectstack/metadata-protocol": minor
+---
+
+feat(spec,metadata-protocol): `IObjectQLEngine.transaction` joins the slot contract, and `batchData`'s `atomic` flag becomes real — rollback or refusal, never silent best-effort (ADR-0118 D1/D4, #4612)
+
+**D1 — the contract fix.** `ObjectQL.transaction()` — ADR-0034's ambient
+transaction, shipped since v8.0.0 — was reachable from plugin space only
+through `as unknown as` casts: the metadata protocol's atomic publish and its
+`transactionalBatch` discovery probe, and the sys-metadata repository's
+`withTxn`, each declared a private structural slice of an engine none of them
+import. It is now declared on `IObjectQLEngine`, required per that contract's
+own rule, with its caveats written into the TSDoc as part of the declared
+meaning rather than left to be discovered: it covers the **default driver
+only**, and when that driver has no `beginTransaction` the callback runs with
+no transaction and no rollback. `MetadataHostEngine` and the sys-metadata
+repository's engine surface now type their optional member as
+`IObjectQLEngine['transaction']`, so a narrow host surface can no longer drift
+from the real signature. Runtime `typeof === 'function'` probes stay — that is
+test-double defence the type system does not replace.
+
+**D4 — the honesty fix.** `batchData`'s `options.atomic` promised "rollback
+entire batch on any failure (transaction mode)" and delivered a `break`
+statement. Every write before the failure stayed committed, and — the part that
+did the real damage — the response reported those rows `success: true` under
+the one flag whose job is to guarantee they were undone.
+
+Now an explicitly atomic batch runs inside ONE `engine.transaction()`: the
+first failure rolls back every prior write, and the response says so
+(`succeeded: 0`, with rows marked `ROLLED_BACK:` / the causal error /
+`NOT_ATTEMPTED:`, and no row reporting success). On a runtime that cannot roll
+back — no `transaction()`, or a default driver without `beginTransaction` — an
+atomic request is **refused** with `501 NOT_IMPLEMENTED` rather than silently
+degrading, matching the cross-object `/batch` route. `atomic` takes precedence
+over `continueOnError`, whose own description already scoped it to
+`atomic=false`. In atomic mode the upsert path no longer falls back to an
+insert when its update throws: inside an aborted transaction that fallback can
+only fail with a secondary error that buries the real cause.
+
+**Aligned declaration.** `BatchOptionsSchema.atomic` declared `.default(true)`
+while no enforcement site delivered atomicity — and the REST route forwards the
+original request body rather than the parsed output, so the declared default
+never reached the loop at all. The default is now `false`: the declaration is
+aligned down to what every site already does, rather than up to what none of
+them did. Honouring the old `true` would have silently flipped the failure
+semantics of every existing batch caller and hard-failed ordinary batches on
+any driver that cannot transact. Callers who were explicitly sending
+`atomic: true` now get what they always asked for; callers sending nothing keep
+today's behaviour exactly.
+
+If you were passing `atomic: true` and relying on partial results surviving a
+failure, that was the bug — switch to `atomic: false` (or omit it) for
+best-effort semantics.
+
+ADR-0118 also rules on two items landing separately: D2 specifies a
+framework-owned migration-journal runner for multi-step migrations too large
+for one transaction, and D3 retires the declared-but-unimplemented
+`IDataEngine.batch?`.

content/docs/api/client-sdk.mdx

@@ -504,9 +504,13 @@ The `find` method accepts an options object with **canonical** (recommended) fie
 
 | Property | Type | Default | Description |
 |:---------|:-----|:--------|:------------|
-| `atomic` | `boolean` | `true` | Rollback entire batch on any failure |
+| `atomic` | `boolean` | `false` | Run the batch in one transaction and roll every write back on the first failure. Refused with `501 NOT_IMPLEMENTED` where the driver cannot roll back, rather than degrading to best-effort. Takes precedence over `continueOnError` |
 | `returnRecords` | `boolean` | `false` | Include full records in response |
-| `continueOnError` | `boolean` | `false` | Continue after errors (when atomic=false) |
+| `continueOnError` | `boolean` | `false` | Continue after errors (when atomic is false) |
+
+A rolled-back atomic batch reports `succeeded: 0`, with each row carrying
+`ROLLED_BACK:`, the causal error, or `NOT_ATTEMPTED:` — no row is reported as a
+success, because none of them survived.
 
 ---
 

content/docs/api/data-api.mdx

@@ -228,6 +228,8 @@ Execute a batch operation (create / update / upsert / delete) on multiple record
 
 **Response**: `BatchUpdateResponse` with `succeeded`, `failed`, `total`, and a per-record `results` array. Each entry in `results` has `id`, `success`, an optional `errors` array, and optional `data` (the full record, present when `returnRecords` is `true`).
 
+`options.atomic` defaults to `false` (sequential best-effort, stopping at the first failure). Set it to `true` and the whole batch runs inside one transaction: the first failure rolls back every prior write, and the response reports `succeeded: 0` with each row marked `ROLLED_BACK:`, the causal error, or `NOT_ATTEMPTED:`. A deployment whose driver cannot roll back rejects an atomic request with `501 NOT_IMPLEMENTED` instead of running it best-effort — probe `capabilities.transactionalBatch` on `/discovery` first. `atomic` takes precedence over `continueOnError`.
+
 ### `POST /data/:object/createMany`
 
 Batch create multiple records.
@@ -260,16 +262,20 @@ ignored, on this route and on `deleteMany`.
 
 Batch delete records by ID list.
 
-**Body**: `{ "ids": ["1", "2", "3"], "options": { "atomic": true } }` — `options` is
+**Body**: `{ "ids": ["1", "2", "3"], "options": { "continueOnError": true } }` — `options` is
 the same `BatchOptions` bag `/batch` takes. The body is validated against the
 contract and unknown keys are dropped: the id list is the *only* thing that
 selects rows, so no body key can widen the delete into a filter.
 
 **Response**: `BatchUpdateResponse` — one `results` entry per id. Records are
 deleted one at a time by primary key, so each honours `deleteBehavior`
-(`cascade` / `set_null` / `restrict`) on relations pointing at it. `atomic`
-(default) stops the run at the first failure; `atomic: false` with
-`continueOnError: true` processes the remaining ids and reports the failures.
+(`cascade` / `set_null` / `restrict`) on relations pointing at it. The run stops
+at the first failure; `continueOnError: true` processes the remaining ids and
+reports the failures instead.
+
+Note that on this route `atomic` only stops the run — deletes already performed
+are **not** rolled back. Unlike `/batch`, `deleteMany` has no wrapping
+transaction yet.
 
 ### Batch size
 

content/docs/api/wire-format.mdx

@@ -471,7 +471,11 @@ release group); any code reading them directly should move to `error.code`.
 
 **`POST /api/v1/data/task/batch`**
 
-Process many records of a **single** operation type in one request. The body carries one `operation` (`create`, `update`, `upsert`, or `delete`) plus a `records` array. By default (`options.atomic: true`) processing stops at the first failing record — records already written earlier in the same batch are **not** rolled back, since there is no wrapping database transaction. Set `options.atomic: false` (with `options.continueOnError: true`) to keep processing every record and collect a full partial-success report.
+Process many records of a **single** operation type in one request. The body carries one `operation` (`create`, `update`, `upsert`, or `delete`) plus a `records` array.
+
+By default (`options.atomic` omitted or `false`) processing stops at the first failing record, and records written earlier in the same batch are **not** rolled back — there is no wrapping transaction. Add `options.continueOnError: true` to keep going instead and collect a full partial-success report.
+
+Send `options.atomic: true` to run the whole batch inside one database transaction: the first failure rolls back every prior write, and the response reports zero successes with each row marked `ROLLED_BACK:`, the causal error, or `NOT_ATTEMPTED:`. A deployment whose driver cannot roll back **rejects** an atomic request with `501 NOT_IMPLEMENTED` rather than quietly running it best-effort — probe `capabilities.transactionalBatch` on `/discovery` to know in advance. `atomic` takes precedence over `continueOnError`.
 
 ### Request
 
@@ -483,7 +487,7 @@ Process many records of a **single** operation type in one request. The body car
  { "id": "tsk_01HQ4B8C0E4G6H9K3L5M", "data": { "status": "done" } }
  ],
  "options": {
- "atomic": true,
+ "atomic": false,
  "continueOnError": false
  }
 }
@@ -509,7 +513,7 @@ The response is the `BatchUpdateResponse` envelope: a top-level `success` flag p
 
 ### Partial Failure Response
 
-When `options.atomic: false` and some records fail, the failing entries carry a single `error` message string (not an array):
+When the batch is not atomic and some records fail, the failing entries carry a single `error` message string (not an array). An atomic batch never returns this shape — it either commits everything or reports every row as failed:
 
 ```json
 {

content/docs/references/api/batch.mdx

@@ -86,7 +86,7 @@ const result = BatchConfigSchema.parse(data);
 
 | Property | Type | Required | Description |
 | :--- | :--- | :--- | :--- |
-| **atomic** | `boolean` | ✅ | If true, rollback entire batch on any failure (transaction mode) |
+| **atomic** | `boolean` | ✅ | Opt-in all-or-nothing. When explicitly true the whole batch runs inside ONE engine transaction: the first failure rolls back every prior write, and the response reports zero successes with rows marked ROLLED_BACK / NOT_ATTEMPTED. A runtime that cannot roll back REFUSES the request (501 NOT_IMPLEMENTED) rather than silently degrading to best-effort — probe `capabilities.transactionalBatch` on /discovery first. Takes precedence over continueOnError. Default false: sequential best-effort. |
 | **returnRecords** | `boolean` | ✅ | If true, return full record data in response |
 | **continueOnError** | `boolean` | ✅ | If true (and atomic=false), continue processing remaining records after errors |
 | **validateOnly** | `any` | optional | [REMOVED] `options.validateOnly` was removed from BatchOptions in @objectstack/spec (#4052). It was never implemented: the batch surfaces persisted regardless, so a "dry-run" would have silently executed. There is no dry-run today — drop the key. If you need to preview a batch without writing, open an issue so it can be designed (no-commit cascade / constraint semantics) and reintroduced as a flag that actually holds. |