Run CLI workflows on Yieldstar

.changeset/reconciler.md

@@ -9,4 +9,4 @@
 "@notation/state-sqlite": minor
 ---
 
-Add the reconciler API, versioned event streams, renewable mutation leases, SQLite state, backend-neutral dashboard state, compiled infrastructure graphs, and durable YieldStar 0.5.0 reconciliation for Node.js runtimes.
+Add durable YieldStar 0.5.0 deploy and destroy workflows, a resident Node SQLite runtime for CLI execution, versioned event streams, backend-neutral dashboard state, and compiled infrastructure graphs.

docs/cli/dashboard.md

@@ -1,21 +1,13 @@
 # notation dashboard
 
 ```sh
-notation dashboard
+notation dashboard <entryPoint>
 ```
 
-Starts a local web dashboard for observing deployment state.
+Starts a local web dashboard for observing the deployment's YieldStar resource stores.
 
 ```sh
-notation dashboard
+notation dashboard infra/api.ts
 ```
 
-The dashboard uses the same state backend as deploy and destroy. Set
-`NOTATION_STATE_PATH` to select SQLite:
-
-```sh
-NOTATION_STATE_PATH=.notation/state.db notation dashboard
-```
-
-The server reads through `StateBackend`, so file and SQLite state produce the same
-dashboard payload.
+The dashboard reads `.notation/workflows.db`, the same database used by deploy, destroy, and plan. Set `NOTATION_STATE_PATH` to choose another SQLite database path.

docs/cli/deploy.md

@@ -4,40 +4,44 @@
 notation deploy <entryPoint>
 ```
 
-Compiles and deploys the stack to AWS.
+Compiles and durably deploys the stack through the resident YieldStar 0.5.0 Node runtime.
 
 ```sh
 notation deploy infra/api.ts
 ```
 
 ## Event stream
 
-`--json` writes versioned reconciler events to stdout as newline-delimited JSON. Build
-output and diagnostics move to stderr.
+`--json` writes versioned reconciler events to stdout as newline-delimited JSON. Build output, the execution ID, and diagnostics move to stderr.
 
 ```sh
 notation deploy infra/api.ts --json > deploy.ndjson
 ```
 
+## Durable execution
+
+The command prints its YieldStar execution ID before starting provider work. If the process crashes, resume the same durable heap with that ID:
+
+```sh
+notation deploy infra/api.ts --execution-id <id>
+```
+
+Do not reuse a completed execution ID for a new deploy or for destroy.
+
+Retryable provider conditions and consistency reads suspend on durable SQLite timers. The CLI stays resident until the scheduler wakes the execution and the workflow completes; completed provider calls are replayed from the heap rather than repeated.
+
 ## What happens
 
-1. **Compile** – esbuild compiles infra and runtime modules to `dist/`.
+1. **Compile** – esbuild compiles infrastructure and runtime modules to `dist/`.
 
-2. **Build resource graph** – imports the compiled output and collects the declared resources.
+2. **Build resource graph** – the worker imports the compiled output and collects declared resources.
 
-3. **Reconcile** – the reconciler compares desired state (graph) against current state (`.notation/state.json`):
- - New resources → **create**
- - Changed params → **update**
- - No changes → **noop**
- - Orphaned resources (in state but not in graph) → **delete**
+3. **Reconcile** – Notation compares desired resources with YieldStar stores, then creates, updates, recreates, or leaves each resource unchanged.
 
-4. **Topological deployment** – resources deploy in dependency order (levels). Resources at the same level deploy concurrently.
+4. **Order dependencies** – dependency levels run in topological order.
 
-5. **Drift detection** – enabled by default. Reads actual AWS state and compares against stored state. If drifted, Notation updates to match your definition.
+5. **Detect drift** – unchanged resources are read from the provider and repaired when their remote state differs.
 
-State is persisted to `.notation/state.json` after each operation. Set
-`NOTATION_STATE_PATH` to a path ending in `.db` or `.sqlite` to use SQLite:
+6. **Delete orphans** – persisted resources absent from the graph are deleted when their resource type is registered.
 
-```sh
-NOTATION_STATE_PATH=.notation/state.db notation deploy infra/api.ts
-```
+State, step results, timers, task coordination, and resource stores are persisted to `.notation/workflows.db`. Set `NOTATION_STATE_PATH` to choose another SQLite database path.

docs/cli/destroy.md

@@ -4,7 +4,7 @@
 notation destroy <entryPoint>
 ```
 
-Removes all resources in the stack. Tears down runs in reverse dependency order, so routes are removed before APIs and Lambdas before IAM roles etc.
+Compiles the application and runs durable destroy through the resident YieldStar 0.5.0 Node runtime. Resources are removed in reverse dependency order, then registered persisted orphans are removed.
 
 ```sh
 notation destroy infra/api.ts
@@ -15,3 +15,11 @@ notation destroy infra/api.ts
 ```sh
 notation destroy infra/api.ts --json > destroy.ndjson
 ```
+
+The command prints its execution ID. Resume a crashed destroy with the same ID so a provider delete that already completed is replayed instead of repeated:
+
+```sh
+notation destroy infra/api.ts --execution-id <id>
+```
+
+Retryable deletes suspend on durable SQLite timers. Resource state is removed only after the provider delete succeeds or reports that the resource is already absent.

docs/internals/reconciler.md

@@ -1,126 +1,40 @@
 # Reconciler
 
-The reconciler runs deployment operations to transition infrastructure from its current state to the state defined in the project.
-
-Source: `@notation/reconciler`
-
-## Durable workflow boundary
-
-`reconcileWithYieldStar` is the durable Node.js integration. It is an async generator intended to be composed inside an application-owned YieldStar workflow:
-
-```ts
-const deploy = workflow(async function* (step, event) {
- yield* reconcileWithYieldStar(step, {
- deploymentId: "production",
- executionId: event.executionId,
- resources,
- state,
- });
-});
-```
-
-The host owns runtime wiring and scheduling. Notation owns graph ordering, decisions, provider calls, drift reads, state persistence, and orphan lifecycle. Provider calls and state mutations are YieldStar steps, so completed calls are replayed instead of repeated after a process crash.
-
-The synchronous `Reconciler` described below remains the CLI path for this release.
+The reconciler expresses deployment and destruction as YieldStar async generators. Notation owns desired-state decisions and provider lifecycle; the caller's YieldStar runtime owns durable execution, waiting, shared state, and coordination.
 
 ## Deploy flow
 
-```ts [packages/reconciler/src/index.ts]
-const reconciler = new Reconciler({ state, registry, emit });
-await reconciler.deploy(resources, { dryRun, driftDetection });
-```
+`deployWithYieldStar` acquires the deployment coordination store, walks dependency levels in order, decides an action for every resource, executes provider calls as durable steps, persists the result in a resource store, and deletes registered orphans.
 
-The reconciler walks the resource graph and, for each resource, determines an action:
+| Condition | Decision |
+| --- | --- |
+| Not in state | **create** |
+| In state, params changed | **update** |
+| In state, params unchanged, no drift | **noop** |
+| In state, but deleted from the provider | **drift-recreate** |
+| In state, provider state differs from stored state | **drift-update** |
+| In state, not in graph | **delete** |
 
-| Condition | Decision |
-| --------------------------------------------- | ------------------ |
-| Not in state | **create** |
-| In state, params changed | **update** |
-| In state, params unchanged, no drift | **noop** |
-| In state, but deleted from AWS | **drift-recreate** |
-| In state, AWS state differs from stored state | **drift-update** |
-| In state, not in graph (orphan) | **delete** |
+Dry-run deploy performs decisions and emits lifecycle events without calling providers or mutating state.
 
-The `dryRun` flag runs the full diffing pipeline without executing any operations, so you can preview what a deploy would do.
+## Destroy flow
 
-## Topological ordering
+`destroyWithYieldStar` is a first-class durable operation. It acquires the same deployment coordination store as deploy, deletes desired resources in reverse dependency order, deletes hydratable persisted orphans, and conditionally removes each resource store only after the provider delete succeeds or reports that the resource is already absent.
 
-Resources are deployed in dependency order using `buildResourceDepthLevels()`. This function partitions the resource graph into levels – each level contains resources whose dependencies have all been satisfied by previous levels.
+Provider delete is a stable durable step. If the process crashes after the provider acknowledges deletion but before state removal, replay uses the cached delete result and continues at the conditional store delete.
 
-```
-Level 0: IAM Role, CloudWatch LogGroup
-Level 1: Lambda Function (depends on Role, LogGroup)
-Level 2: API Gateway Integration (depends on Lambda)
-Level 3: API Gateway Route (depends on Integration)
-```
+## Waiting and replay
 
-Resources within a level deploy concurrently, so independent resources like the IAM Role and LogGroup above are provisioned in parallel. Dependent resources wait for their dependencies.
+Retryable provider errors become YieldStar `RetryableError` delays. The resident Node runtime can remain idle until the SQLite timer queues a wake-up, then rebuild the resource graph and replay completed heap steps. Reads that wait for provider consistency use the same mechanism.
 
-Destroy operates in reverse order with dependents getting removed before their dependencies.
+Every provider call, event emission, state read, state write, and coordination transition has a stable step key. A resumed execution must use the same execution ID. A new deploy or destroy must use a new execution ID so its heap does not alias an earlier operation.
 
-### Cycle detection
+## State and coordination
 
-Cycle detection is built in. If resources form a circular dependency, the build fails with:
+Each resource is stored under `notation/resource-state` with a deployment-scoped ID. Conditional updates and deletes compare the snapshot's UUIDv7 `instanceId` and version, so a stale execution cannot modify a deleted and recreated store.
 
-```
-Resource dependency cycle detected
-```
-
-This catches configuration errors before any cloud operations are attempted.
-
-## Drift detection
-
-Drift detection is enabled by default. After confirming no local changes to a resource, the reconciler reads the resource's current state from AWS (via the resource's `read()` operation) and diffs it against stored state.
-
-If AWS has drifted (e.g. someone changed a Lambda timeout in the console, or an IAM policy was modified by another tool), Notation updates the resource to match the canoncial definition in the source code.
-
-Properties marked as `volatile` in the schema (like `LastModified` timestamps) are excluded from drift comparison.
+Deploy and destroy share one `notation/deployment-coordination` store per deployment. `store.take` suspends a competing execution as a durable waiter and wakes it after the holder releases.
 
 ## Events
 
-The reconciler emits events at each step of an operation's lifecycle. The default `createConsoleReconcilerSubscriber()` logs these to the console with formatted output.
-
-| Event | When |
-| ------------------------------------ | --------------------------------------------------- |
-| `reconciler.deploy.decision` | After deciding what action to take for a resource |
-| `reconciler.drift.detected` | When drift is found between stored and actual state |
-| `reconciler.operation.lifecycle` | When an operation starts, finishes, skips, or fails |
-| `reconciler.orphan-deletion.skipped` | When no registered class can delete an orphan |
-
-Lifecycle events contain the operation (`create`, `read`, `update`, or `delete`) and its
-status (`start`, `success`, `error`, `skip`, or `dry-run`). Events carry the resource ID,
-type, and relevant diff or error details.
-
-## Operations
-
-Each CRUD operation is implemented as an async generator with retry support:
-
-- **`createResourceOperation`** – creates the resource, reads back its state, persists to state backend
-- **`updateResourceOperation`** – applies the update, reads back new state, persists to state backend
-- **`deleteResourceOperation`** – deletes the resource, removes the entry from state backend
-- **`readResourceOperation`** – reads current state from the cloud provider (used for drift detection)
-
-### Retry and polling
-
-Operations support polling for eventual consistency:
-
-```ts [packages/reconciler/src/index.ts]
-{
- maxAttempts: 10,
- retryInterval: 2000,
-}
-```
-
-This handles AWS services that return success before the resource is fully available. For example, after creating an IAM Role, a Lambda function may briefly fail to deploy until the role propagates. The retry loop handles cases like this.
-
-### Operation lifecycle
-
-Each operation follows the following pattern:
-
-1. Emit `started` event
-2. Execute the cloud operation (with retries)
-3. Read back the resource state
-4. Persist to state backend
-5. Emit `completed` event (or `failed` on error)
-
-State is updated after the provider operation and read-back complete.
+The durable workflows emit `reconciler.deploy.decision`, `reconciler.drift.detected`, `reconciler.operation.lifecycle`, and `reconciler.orphan-deletion.skipped`. Lifecycle events cover create, read, update, and delete with `start`, `success`, `error`, `skip`, or `dry-run` status.