Add durable Yieldstar reconciliation

.changeset/reconciler.md

@@ -9,6 +9,4 @@
 "@notation/state-sqlite": minor
 ---
 
-Add the reconciler API, versioned event streams, renewable mutation leases,
-SQLite state, backend-neutral dashboard state, and compiled infrastructure
-graphs.
+Add the reconciler API, versioned event streams, renewable mutation leases, SQLite state, backend-neutral dashboard state, compiled infrastructure graphs, and durable YieldStar 0.5.0 reconciliation for Node.js runtimes.

docs/internals/reconciler.md

@@ -4,6 +4,25 @@ The reconciler runs deployment operations to transition infrastructure from its
 
 Source: `@notation/reconciler`
 
+## Durable workflow boundary
+
+`reconcileWithYieldStar` is the durable Node.js integration. It is an async generator intended to be composed inside an application-owned YieldStar workflow:
+
+```ts
+const deploy = workflow(async function* (step, event) {
+ yield* reconcileWithYieldStar(step, {
+ deploymentId: "production",
+ executionId: event.executionId,
+ resources,
+ state,
+ });
+});
+```
+
+The host owns runtime wiring and scheduling. Notation owns graph ordering, decisions, provider calls, drift reads, state persistence, and orphan lifecycle. Provider calls and state mutations are YieldStar steps, so completed calls are replayed instead of repeated after a process crash.
+
+The synchronous `Reconciler` described below remains the CLI path for this release.
+
 ## Deploy flow
 
 ```ts [packages/reconciler/src/index.ts]

docs/internals/state.md

@@ -78,6 +78,16 @@ Stores state and leases in SQLite. Select it in the CLI by setting
 const state = new SqliteStateBackend(".notation/state.db");
 ```
 
+### `YieldStarStateBackend`
+
+The durable workflow integration stores resources in YieldStar 0.5.0 stores and runs on the Node SQLite runtime. Each resource is a live store; a missing store means a missing resource.
+
+```ts
+const state = new YieldStarStateBackend(storeClient, "production");
+```
+
+YieldStar assigns a UUIDv7 `instanceId` when the store is created and increments its version on update. Conditional workflow updates and deletes compare both values, preventing a stale snapshot from modifying a deleted and recreated resource. The one-based value exposed as `StateNode.rev` is derived from the authoritative YieldStar store version.
+
 ### `StateBackend` interface
 
 All backends implement the same interface:

docs/manual/introduction.md

@@ -13,8 +13,7 @@ todoRouter.get("/todos", getTodos);
 
 Notation is a compiler, reconciler, and deployment engine.
 
-The reconciler is also available as an embedded library. A Node.js host can construct
-resources, choose a state backend, and run plan, deploy, or destroy without the CLI.
+The reconciler is also available as an embedded library. A Node.js host can construct resources and compose durable reconciliation inside its own YieldStar workflow without the CLI.
 
 The compiler runs two passes over your codebase:
 

docs/manual/reconciler.md

@@ -1,55 +1,34 @@
 # Reconciler
 
-Use the reconciler directly when a Node.js application needs to deploy resources without
-starting the Notation CLI.
-
-This complete program deploys two static sites and keeps their deployment state in
-SQLite:
+Use `reconcileWithYieldStar` when a Node.js application needs durable resource reconciliation without starting the Notation CLI. Notation supplies reconciliation decisions and resource lifecycle operations; the application owns the outer YieldStar workflow and chooses the runtime.
 
 ```ts
-import { Reconciler, createResourceRegistry } from "@notation/reconciler";
-import { SqliteStateBackend } from "@notation/state-sqlite";
-import { StaticSite } from "./static-site";
-
-const state = new SqliteStateBackend("sites.db");
-
-const resources = [
- new StaticSite({
- id: "documentation",
- config: {
- siteDirectory: "sites/docs",
- html: "<h1>Documentation</h1>\n",
- },
- }),
- new StaticSite({
- id: "status",
- config: {
- siteDirectory: "sites/status",
- html: "<h1>All systems operational</h1>\n",
- },
- }),
-];
-
-const reconciler = new Reconciler({
- state,
- registry: createResourceRegistry([StaticSite]),
+import { SqliteSchedulerClient, SqliteStoreClient, SqliteTaskQueueClient, SqliteTimersClient, createSqliteDb } from "@yieldstar/sqlite-runtime/node";
+import { YieldStarStateBackend, reconcileWithYieldStar } from "@notation/reconciler";
+import { workflow } from "yieldstar";
+
+const database = createSqliteDb({ path: ".notation/workflows.db" });
+const schedulerClient = new SqliteSchedulerClient({
+ taskQueueClient: new SqliteTaskQueueClient(database),
+ timersClient: new SqliteTimersClient(database),
+});
+const storeClient = new SqliteStoreClient({ db: database, schedulerClient });
+const state = new YieldStarStateBackend(storeClient, "my-application");
+
+export const deploy = workflow(async function* (step, event) {
+ yield* reconcileWithYieldStar(step, {
+ deploymentId: "my-application",
+ executionId: event.executionId,
+ resources,
+ state,
+ });
 });
-
-try {
- await reconciler.deploy(resources);
-} finally {
- state.close();
-}
 ```
 
-`StaticSite` contains the provider operations which create, read, update, and delete a
-site. A real provider would call its infrastructure API instead of writing local files.
+The outer workflow supplies durable step execution, timers, shared stores, waiting, and scheduling. `reconcileWithYieldStar` uses those primitives to cache completed provider calls, retry provider waiting without holding a process, persist state conditionally, delete state conditionally, and serialize deployments with `store.take`.
 
-Pass the complete desired set to `deploy`. A resource which remains in deployment state
-but is absent from that set is deleted. The explicit registry lets the reconciler find
-its delete operation.
+Each live resource is one YieldStar store. Absence is represented by no store, not a tombstone. YieldStar's UUIDv7 store `instanceId` and version are authoritative for conditional update and delete; Notation exposes the version as the resource state's `rev` for its existing state contract.
 
-Notation's state records what was deployed. It does not replace application data which
-owns the desired configuration.
+Pass the complete desired set on every invocation. Persisted resources absent from that set are deleted through the supplied resource registry.
 
-The runnable version is in `examples/reconciler`.
+The runnable Node SQLite version is in `examples/reconciler`.

docs/rfcs/reconciler.md

@@ -1,102 +1,30 @@
-# RFC: Reconciler
+# RFC: Durable YieldStar reconciliation
 
-**Status:** implemented
-**Scope:** `@notation/state`, `@notation/reconciler`
+**Status:** implemented release slice
+**Scope:** `@notation/reconciler`, YieldStar 0.5.0
 
-Notation evaluates an infrastructure program into resources, then reconciles those
-resources against recorded state. The same engine now runs behind the CLI, the dashboard,
-and direct library integrations.
+Notation describes reconciliation and resource lifecycle operations. A host-owned YieldStar workflow supplies durable execution, waiting, state, and coordination by calling `yield* reconcileWithYieldStar(step, options)`.
 
-```ts
-import { Reconciler } from "@notation/reconciler";
-import { SqliteStateBackend } from "@notation/state-sqlite";
+## Boundary
 
-const state = new SqliteStateBackend(".notation/state.db");
-const reconciler = new Reconciler({ state });
+Live resource objects remain in the workflow worker. They are not serialized into workflow parameters. This keeps provider clients and operation closures under application control while YieldStar persists step results and shared state.
 
-await reconciler.deploy(resources);
-state.close();
-```
+Provider create, update, read, and delete calls are durable steps with stable resource-scoped keys. A process crash after a completed create replays the cached result and continues at state persistence instead of creating the provider resource again. Retryable provider conditions become YieldStar delays, allowing the process to stop until the scheduler wakes the execution.
 
-The reconciler boundary consists of live resource objects, a state backend, and an event
-subscriber. Resource operations run in the host process.
+## State lifecycle
 
-## State
+`YieldStarStateBackend` stores one live resource per `notation/resource-state` store. The store ID is scoped by deployment and resource ID. Store absence is resource absence; no application tombstone is created.
 
-Each state record carries a revision. Updates and deletes can require the revision which
-the caller previously read:
+The runtime-assigned UUIDv7 `instanceId` distinguishes a deleted store from a later store created under the same logical ID. YieldStar's version is the concurrency token and is exposed as Notation's one-based `rev`. Workflow updates use `store.updateFrom` and deletes use `store.deleteFrom`, so both the instance and version must match the snapshot that informed the operation.
 
-```ts
-await state.update(resource.id, patch, resource.rev);
-```
+`values` uses YieldStar 0.5.0's merged `listStores` lifecycle API, and administrative cleanup uses `deleteStore`.
 
-A stale writer receives `RevConflict`. A missing record has revision zero, so
-`expectedRev: 0` means that the record must not exist.
+## Coordination
 
-The reconciler also takes a renewable per-resource lease before it reads a resource for
-mutation. The lease remains held across the provider operation and state write. Two
-hosts therefore cannot create or update the same resource concurrently through the same
-backend.
+Each deployment has a `notation/deployment-coordination` store. The workflow atomically claims it with `store.take`. A concurrent execution suspends as a durable waiter and is woken when the holder releases the store. The same execution can recover an acquisition across the store-commit/heap-write crash gap through YieldStar's applied-step ledger.
 
-Orphan deletion takes an additional snapshot lease. The snapshot remains stable while
-the reconciler decides which state records no longer appear in the desired graph.
+## Release boundary
 
-## Backends
+This slice delivers durable deploy reconciliation, drift handling, orphan deletion, Node SQLite execution, external state access, conditional persistence, and concurrent deployment serialization. The existing synchronous `Reconciler` remains the CLI execution path in this release.
 
-`@notation/state` provides file and memory backends. `@notation/state-sqlite` provides
-the reference database backend.
-
-Every backend implements the same contract:
-
-```ts
-interface StateBackend {
- get(id: string): Promise<StateNode | undefined>;
- has(id: string): Promise<boolean>;
- update(
- id: string,
- patch: Partial<StateNode>,
- expectedRev?: number,
- ): Promise<{ rev: number }>;
- delete(id: string, expectedRev?: number): Promise<void>;
- values(): Promise<StateNode[]>;
- lease(scope: string, ttl: number): Promise<Lease>;
-}
-```
-
-The dashboard reads this interface. It does not inspect a state file directly.
-
-## Events
-
-The reconciler accepts one subscriber:
-
-```ts
-const reconciler = new Reconciler({
- state,
- emit: async (event) => auditLog.write(event),
-});
-```
-
-`createNdjsonEventEmitter` adapts the subscriber to a versioned newline-delimited JSON
-stream. The CLI uses the same adapter for `deploy --json` and `destroy --json`.
-
-## Package boundary
-
-The CLI creates resources from compiled Notation programs, then hands those live objects
-to `Reconciler`. An application can construct the same resource classes directly.
-
-The reconciler does not serialise resource classes or execute operations in another
-process. Detached execution needs manifests, resource-reference encoding, actuator
-binding, and a runtime consumer. That work has its own RFC and release.
-
-## Acceptance
-
-The reconciler example is the compatibility test for this boundary. It must:
-
-1. Construct a resource without the CLI.
-2. Plan and deploy it through `Reconciler`.
-3. Close and reopen SQLite state.
-4. Plan and apply an update.
-5. Receive versioned events.
-6. Destroy the resource and remove its state.
-
-The example lives in `examples/reconciler` and runs without cloud credentials.
+The next stacked phase will move CLI deploy and destroy onto a resident workflow runtime, add durable destroy as a first-class workflow operation, and fan independent dependency-level resources into coordinated child executions.

examples/reconciler/README.md

@@ -1,22 +1,16 @@
-# Reconciler
+# Durable reconciler
 
-This example deploys two static sites from an ordinary Node.js program. It does not
-compile a Notation project or start the Notation CLI.
+This example deploys two static sites from an ordinary Node.js program using YieldStar 0.5.0 for durable execution, state, retries, waiting, and deployment coordination.
 
-[`src/index.ts`](./src/index.ts) is the complete program. It defines the desired
-resources inline, opens a SQLite state backend, and passes the resources directly to the
-reconciler. [`src/static-site.ts`](./src/static-site.ts) defines the local provider
-operations used to create, read, update, and delete each site.
+[`src/index.ts`](./src/index.ts) owns the outer workflow and Node SQLite runtime. It passes YieldStar's `step` context to `reconcileWithYieldStar`, while [`src/static-site.ts`](./src/static-site.ts) contains only the desired resources and provider lifecycle operations.
 
 Run it from the repository root:
 
 ```sh
 pnpm --filter reconciler-example demo
 ```
 
-The generated sites are written to `sites/`, and deployment state is stored in
-`sites.db`. Change the resource configuration and run the command again to update the
-sites. Remove a resource from the array and run it again to delete that site.
+The generated sites are written to `sites/`, and the workflow heap, resource stores, timers, and coordination state are stored in `sites.db`. Change the resource configuration and run the command again to update the sites. Remove a resource from the array and run it again to delete that site.
 
 Run the integration test with:
 

examples/reconciler/package.json

@@ -11,7 +11,10 @@
  "dependencies": {
  "@notation/reconciler": "workspace:*",
  "@notation/resource": "workspace:*",
- "@notation/state-sqlite": "workspace:*"
+ "@yieldstar/core": "0.5.0",
+ "@yieldstar/sqlite-runtime": "0.5.0",
+ "pino": "^9.9.0",
+ "yieldstar": "0.5.0"
  },
  "devDependencies": {
  "@types/node": "^22.13.4",

examples/reconciler/src/index.ts

@@ -1,8 +1,30 @@
-import { Reconciler, createResourceRegistry } from "@notation/reconciler";
-import { SqliteStateBackend } from "@notation/state-sqlite";
+import { WorkflowRunner } from "@yieldstar/core";
+import {
+ SqliteHeapClient,
+ SqliteSchedulerClient,
+ SqliteStoreClient,
+ SqliteTaskQueueClient,
+ SqliteTimersClient,
+ createSqliteDb,
+} from "@yieldstar/sqlite-runtime/node";
+import {
+ YieldStarStateBackend,
+ createResourceRegistry,
+ reconcileWithYieldStar,
+} from "@notation/reconciler";
+import pino from "pino";
+import { createWorkflowRouter, workflow } from "yieldstar";
 import { StaticSite } from "./static-site";
 
-const state = new SqliteStateBackend("sites.db");
+const logger = pino();
+const database = createSqliteDb({ path: "sites.db" });
+const taskQueueClient = new SqliteTaskQueueClient(database);
+const schedulerClient = new SqliteSchedulerClient({
+ taskQueueClient,
+ timersClient: new SqliteTimersClient(database),
+});
+const storeClient = new SqliteStoreClient({ db: database, schedulerClient });
+const state = new YieldStarStateBackend(storeClient, "static-sites");
 
 const resources = [
  new StaticSite({
@@ -21,13 +43,34 @@ const resources = [
  }),
 ];
 
-const reconciler = new Reconciler({
- state,
- registry: createResourceRegistry([StaticSite]),
+const deploy = workflow(async function* (step, event) {
+ yield* reconcileWithYieldStar(step, {
+ deploymentId: "static-sites",
+ executionId: event.executionId,
+ resources,
+ state,
+ registry: createResourceRegistry([StaticSite]),
+ });
+});
+
+const runner = new WorkflowRunner({
+ router: createWorkflowRouter({ deploy }),
+ heapClient: new SqliteHeapClient(database),
+ storeClient,
+ schedulerClient,
+ logger,
 });
 
 try {
- await reconciler.deploy(resources);
+ await runner.run(
+ {
+ workflowId: "deploy",
+ executionId: crypto.randomUUID(),
+ params: {},
+ context: new Map(),
+ },
+ logger,
+ );
 } finally {
-state.close();
+database.close();
 }

packages/reconciler/package.json

@@ -14,7 +14,12 @@
  "dependencies": {
  "@notation/resource": "workspace:*",
  "@notation/state": "workspace:*",
+ "@yieldstar/core": "0.5.0",
  "deep-object-diff": "^1.1.9",
- "yieldstar": "^0.4.6"
+ "yieldstar": "0.5.0"
+ },
+ "devDependencies": {
+ "@yieldstar/sqlite-runtime": "0.5.0",
+ "pino": "^9.9.0"
  }
 }