Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .changeset/reconciler.md
Original file line numberDiff line numberDiff line change
Expand Up@@ -9,4 +9,4 @@
"@notation/state-sqlite": minor
---

Add durable YieldStar 0.5.0 deploy and destroy workflows, a resident Node SQLite runtime for CLI execution, versioned event streams, backend-neutral dashboard state, and compiled infrastructure graphs.
Add durable Yieldstar 0.5.0 deploy and destroy workflows, a resident Node SQLite runtime for CLI execution, versioned event streams, backend-neutral dashboard state, and compiled infrastructure graphs.
2 changes: 1 addition & 1 deletion docs/cli/dashboard.md
Original file line numberDiff line numberDiff line change
Expand Up@@ -4,7 +4,7 @@
notation dashboard <entryPoint>
```

Starts a local web dashboard for observing the deployment's YieldStar resource stores.
Starts a local web dashboard for observing the deployment's Yieldstar resource stores.

```sh
notation dashboard infra/api.ts
Expand Down
6 changes: 3 additions & 3 deletions docs/cli/deploy.md
Original file line numberDiff line numberDiff line change
Expand Up@@ -4,7 +4,7 @@
notation deploy <entryPoint>
```

Compiles and durably deploys the stack through the resident YieldStar 0.5.0 Node runtime.
Compiles and durably deploys the stack through the resident Yieldstar 0.5.0 Node runtime.

```sh
notation deploy infra/api.ts
Expand All@@ -20,7 +20,7 @@ notation deploy infra/api.ts --json > deploy.ndjson

## Durable execution

The command prints its YieldStar execution ID before starting provider work. If the process crashes, resume the same durable heap with that ID:
The command prints its Yieldstar execution ID before starting provider work. If the process crashes, resume the same durable heap with that ID:

```sh
notation deploy infra/api.ts --execution-id <id>
Expand All@@ -36,7 +36,7 @@ Retryable provider conditions and consistency reads suspend on durable SQLite ti

2. **Build resource graph** – the worker imports the compiled output and collects declared resources.

3. **Reconcile** – Notation compares desired resources with YieldStar stores, then creates, updates, recreates, or leaves each resource unchanged.
3. **Reconcile** – Notation compares desired resources with Yieldstar stores, then creates, updates, recreates, or leaves each resource unchanged.

4. **Order dependencies** – dependency levels run in topological order.

Expand Down
2 changes: 1 addition & 1 deletion docs/cli/destroy.md
Original file line numberDiff line numberDiff line change
Expand Up@@ -4,7 +4,7 @@
notation destroy <entryPoint>
```

Compiles the application and runs durable destroy through the resident YieldStar 0.5.0 Node runtime. Resources are removed in reverse dependency order, then registered persisted orphans are removed.
Compiles the application and runs durable destroy through the resident Yieldstar 0.5.0 Node runtime. Resources are removed in reverse dependency order, then registered persisted orphans are removed.

```sh
notation destroy infra/api.ts
Expand Down
12 changes: 6 additions & 6 deletions docs/internals/reconciler.md
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
# Reconciler

The reconciler expresses deployment and destruction as YieldStar async generators. Notation owns desired-state decisions and provider lifecycle; the caller's YieldStar runtime owns durable execution, waiting, shared state, and coordination.
The reconciler expresses deployment and destruction as Yieldstar async generators. Notation owns desired-state decisions and provider lifecycle; the caller's Yieldstar runtime owns durable execution, waiting, shared state, and coordination.

## Deploy flow

`deployWithYieldStar` acquires the deployment coordination store, walks dependency levels in order, decides an action for every resource, executes provider calls as durable steps, persists the result in a resource store, and deletes registered orphans.
`deployWithYieldstar` acquires the deployment coordination store, walks dependency levels in order, decides an action for every resource, executes provider calls as durable steps, persists the result in a resource store, and deletes registered orphans.

| Condition | Decision |
| --- | --- |
Expand All@@ -19,22 +19,22 @@ Dry-run deploy performs decisions and emits lifecycle events without calling pro

## Destroy flow

`destroyWithYieldStar` is a first-class durable operation. It acquires the same deployment coordination store as deploy, deletes desired resources in reverse dependency order, deletes hydratable persisted orphans, and conditionally removes each resource store only after the provider delete succeeds or reports that the resource is already absent.
`destroyWithYieldstar` is a first-class durable operation. It acquires the same deployment coordination store as deploy, deletes desired resources in reverse dependency order, deletes hydratable persisted orphans, and conditionally removes each resource store only after the provider delete succeeds or reports that the resource is already absent.

Provider delete is a stable durable step. If the process crashes after the provider acknowledges deletion but before state removal, replay uses the cached delete result and continues at the conditional store delete.

## Waiting and replay

Retryable provider errors become YieldStar `RetryableError` delays. The resident Node runtime can remain idle until the SQLite timer queues a wake-up, then rebuild the resource graph and replay completed heap steps. Reads that wait for provider consistency use the same mechanism.
Retryable provider errors become Yieldstar `RetryableError` delays. The resident Node runtime can remain idle until the SQLite timer queues a wake-up, then rebuild the resource graph and replay completed heap steps. Reads that wait for provider consistency use the same mechanism.

Every provider call, event emission, state read, state write, and coordination transition has a stable step key. A resumed execution must use the same execution ID. A new deploy or destroy must use a new execution ID so its heap does not alias an earlier operation.

## State and coordination

Each resource is stored under `notation/resource-state` with a deployment-scoped ID. Conditional updates and deletes compare the snapshot's UUIDv7 `instanceId` and version, so a stale execution cannot modify a deleted and recreated store.

Deploy and destroy share one `notation/deployment-coordination` store per deployment. `store.take` suspends a competing execution as a durable waiter and wakes it after the holder releases.
Deploy and destroy share one `notation/deployment-coordination` store per deployment. `store.take` suspends a competing execution as a durable waiter and wakes it after the holder releases. Before suspending, the waiter emits `reconciler.coordination.waiting` naming the holding execution ID, so a wait behind a crashed execution is visible instead of silent.

## Events

The durable workflows emit `reconciler.deploy.decision`, `reconciler.drift.detected`, `reconciler.operation.lifecycle`, and `reconciler.orphan-deletion.skipped`. Lifecycle events cover create, read, update, and delete with `start`, `success`, `error`, `skip`, or `dry-run` status.
The durable workflows emit `reconciler.deploy.decision`, `reconciler.drift.detected`, `reconciler.operation.lifecycle`, `reconciler.coordination.waiting`, and `reconciler.orphan-deletion.skipped`. Lifecycle events cover create, read, update, and delete with `start`, `success`, `error`, `skip`, or `dry-run` status.
8 changes: 4 additions & 4 deletions docs/internals/state.md
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
# State

Notation CLI deploy, destroy, plan, and dashboard use YieldStar 0.5.0 stores in `.notation/workflows.db`. Override the database path with `NOTATION_STATE_PATH`.
Notation CLI deploy, destroy, plan, and dashboard use Yieldstar 0.5.0 stores in `.notation/workflows.db`. Override the database path with `NOTATION_STATE_PATH`.

Each live resource is a `notation/resource-state` store scoped by deployment and resource ID. A missing store means the resource is absent. No application tombstone is written.

```ts
const state = new YieldStarStateBackend(storeClient, "infra/api.ts");
const state = new YieldstarStateBackend(storeClient, "infra/api.ts");
```

The runtime assigns a UUIDv7 `instanceId` when a store is created and increments its version on update. Conditional workflow updates and deletes compare both values, preventing a stale snapshot from modifying a deleted and recreated resource. The one-based value exposed as `StateNode.rev` is derived from the authoritative YieldStar store version.
The runtime assigns a UUIDv7 `instanceId` when a store is created and increments its version on update. Conditional workflow updates and deletes compare both values, preventing a stale snapshot from modifying a deleted and recreated resource. The one-based value exposed as `StateNode.rev` is derived from the authoritative Yieldstar store version.

```ts
interface StateBackend {
Expand All@@ -20,6 +20,6 @@ interface StateBackend {
}
```

Coordination is not part of the state backend contract. The outer YieldStar workflow serializes deploy and destroy through a deployment coordination store and records applied store steps for crash-safe replay.
Coordination is not part of the state backend contract. The outer Yieldstar workflow serializes deploy and destroy through a deployment coordination store and records applied store steps for crash-safe replay.

`MemoryStateBackend`, `FileStateBackend`, and `SqliteStateBackend` remain data adapters for tests and embedded read/write consumers. They are not CLI execution runtimes and do not provide mutation coordination.
2 changes: 1 addition & 1 deletion docs/manual/introduction.md
Original file line numberDiff line numberDiff line change
Expand Up@@ -13,7 +13,7 @@ todoRouter.get("/todos", getTodos);

Notation is a compiler, reconciler, and deployment engine.

The reconciler is also available as an embedded library. A Node.js host can construct resources and compose durable reconciliation inside its own YieldStar workflow without the CLI.
The reconciler is also available as an embedded library. A Node.js host can construct resources and compose durable reconciliation inside its own Yieldstar workflow without the CLI.

The compiler runs two passes over your codebase:

Expand Down
16 changes: 8 additions & 8 deletions docs/manual/reconciler.md
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
# Reconciler

Use `deployWithYieldStar` and `destroyWithYieldStar` when a Node.js application needs durable resource lifecycle operations without starting the Notation CLI. Notation owns reconciliation intent, graph ordering, provider calls, and resource state; the application owns the outer YieldStar workflow and runtime.
Use `deployWithYieldstar` and `destroyWithYieldstar` when a Node.js application needs durable resource lifecycle operations without starting the Notation CLI. Notation owns reconciliation intent, graph ordering, provider calls, and resource state; the application owns the outer Yieldstar workflow and runtime.

```ts
import { SqliteSchedulerClient, SqliteStoreClient, SqliteTaskQueueClient, SqliteTimersClient, createSqliteDb } from "@yieldstar/sqlite-runtime/node";
import { YieldStarStateBackend, deployWithYieldStar, destroyWithYieldStar } from "@notation/reconciler";
import { YieldstarStateBackend, deployWithYieldstar, destroyWithYieldstar } from "@notation/reconciler";
import { workflow } from "yieldstar";

const database = createSqliteDb({ path: ".notation/workflows.db" });
Expand All@@ -13,10 +13,10 @@ const schedulerClient = new SqliteSchedulerClient({
timersClient: new SqliteTimersClient(database),
});
const storeClient = new SqliteStoreClient({ db: database, schedulerClient });
const state = new YieldStarStateBackend(storeClient, "my-application");
const state = new YieldstarStateBackend(storeClient, "my-application");

export const deploy = workflow(async function* (step, event) {
yield* deployWithYieldStar(step, {
yield* deployWithYieldstar(step, {
deploymentId: "my-application",
executionId: event.executionId,
resources,
Expand All@@ -25,7 +25,7 @@ export const deploy = workflow(async function* (step, event) {
});

export const destroy = workflow(async function* (step, event) {
yield* destroyWithYieldStar(step, {
yield* destroyWithYieldstar(step, {
deploymentId: "my-application",
executionId: event.executionId,
resources,
Expand All@@ -34,11 +34,11 @@ export const destroy = workflow(async function* (step, event) {
});
```

The outer workflow supplies durable step execution, timers, shared stores, waiting, scheduling, and coordination. Completed provider calls are replayed from the heap after a crash, retryable provider conditions suspend on a durable timer, and conditional state writes use YieldStar store identity and version.
The outer workflow supplies durable step execution, timers, shared stores, waiting, scheduling, and coordination. Completed provider calls are replayed from the heap after a crash, retryable provider conditions suspend on a durable timer, and conditional state writes use Yieldstar store identity and version.

Each live resource is one YieldStar store. Absence is represented by no store, not a tombstone. YieldStar's UUIDv7 store `instanceId` and version are authoritative for conditional update and delete; Notation exposes the version as the resource state's `rev`.
Each live resource is one Yieldstar store. Absence is represented by no store, not a tombstone. Yieldstar's UUIDv7 store `instanceId` and version are authoritative for conditional update and delete; Notation exposes the version as the resource state's `rev`.

Operations against the same `deploymentId` are serialized through a coordination store keyed by `executionId`. Resume a crashed operation with the same execution ID; use a new globally unique execution ID for every new deploy or destroy.
Operations against the same `deploymentId` are serialized through a coordination store keyed by `executionId`. Resume a crashed operation with the same execution ID; use a new globally unique execution ID for every new deploy or destroy. An execution that must wait emits a `reconciler.coordination.waiting` event naming the holder before it suspends, which also identifies a crashed holder that should be resumed instead.

Pass the complete desired set on every deployment. Persisted resources absent from that set are deleted through the supplied resource registry. Destroy removes current resources in reverse dependency order and then removes any persisted orphans that the registry can hydrate.

Expand Down
18 changes: 9 additions & 9 deletions docs/rfcs/reconciler.md
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,28 @@
# RFC: Durable YieldStar reconciliation
# RFC: Durable Yieldstar reconciliation

**Status:** implemented
**Scope:** `@notation/reconciler`, `@notation/core`, YieldStar 0.5.0
**Scope:** `@notation/reconciler`, `@notation/core`, Yieldstar 0.5.0

Notation describes reconciliation intent and resource lifecycle operations. An outer YieldStar workflow supplies durable execution, waiting, state, and coordination by composing `deployWithYieldStar` or `destroyWithYieldStar`.
Notation describes reconciliation intent and resource lifecycle operations. An outer Yieldstar workflow supplies durable execution, waiting, state, and coordination by composing `deployWithYieldstar` or `destroyWithYieldstar`.

## Boundary

Live resource objects remain in the workflow process. They are not serialized into workflow parameters. This keeps provider clients and operation closures under Notation's lifecycle control while YieldStar persists step results and shared state.
Live resource objects remain in the workflow process. They are not serialized into workflow parameters. This keeps provider clients and operation closures under Notation's lifecycle control while Yieldstar persists step results and shared state.

Provider create, update, read, and delete calls are durable steps with stable resource-scoped keys. A process crash after a completed provider call replays the cached result and continues at state persistence instead of repeating the call. Retryable provider conditions become YieldStar delays, allowing the process to wait without polling the provider continuously.
Provider create, update, read, and delete calls are durable steps with stable resource-scoped keys. A process crash after a completed provider call replays the cached result and continues at state persistence instead of repeating the call. Retryable provider conditions become Yieldstar delays, allowing the process to wait without polling the provider continuously.

## State lifecycle

`YieldStarStateBackend` stores one live resource per `notation/resource-state` store. The store ID is scoped by deployment and resource ID. Store absence is resource absence.
`YieldstarStateBackend` stores one live resource per `notation/resource-state` store. The store ID is scoped by deployment and resource ID. Store absence is resource absence.

The runtime-assigned UUIDv7 `instanceId` distinguishes a deleted store from a later store created under the same logical ID. YieldStar's version is the concurrency token and is exposed as Notation's one-based `rev`. Workflow updates use `store.updateFrom` and deletes use `store.deleteFrom`, so both the instance and version must match the snapshot that informed the operation.
The runtime-assigned UUIDv7 `instanceId` distinguishes a deleted store from a later store created under the same logical ID. Yieldstar's version is the concurrency token and is exposed as Notation's one-based `rev`. Workflow updates use `store.updateFrom` and deletes use `store.deleteFrom`, so both the instance and version must match the snapshot that informed the operation.

## Coordination

Each deployment has a `notation/deployment-coordination` store shared by deploy and destroy. The workflow atomically claims it with `store.take`. A concurrent execution suspends as a durable waiter and is woken when the holder releases the store. The same execution can recover an acquisition across the store-commit and heap-write crash gap through YieldStar's applied-step ledger.
Each deployment has a `notation/deployment-coordination` store shared by deploy and destroy. The workflow atomically claims it with `store.take`. A concurrent execution suspends as a durable waiter and is woken when the holder releases the store. The same execution can recover an acquisition across the store-commit and heap-write crash gap through Yieldstar's applied-step ledger.

## Node CLI runtime

`NodeYieldStarRuntime` wires `WorkflowRunner`, `SqliteHeapClient`, `SqliteStoreClient`, `SqliteSchedulerClient`, and `SqliteEventLoop` against one Node SQLite database. CLI deploy and destroy run through this resident runtime and wait for a workflow result across timer and store wake-ups.
`NodeYieldstarRuntime` wires `WorkflowRunner`, `SqliteHeapClient`, `SqliteStoreClient`, `SqliteSchedulerClient`, and `SqliteEventLoop` against one Node SQLite database. CLI deploy and destroy run through this resident runtime and wait for a workflow result across timer and store wake-ups.

The CLI prints a new execution ID for each operation. Re-running with `--execution-id <id>` resumes that operation from its durable heap after a process crash.
4 changes: 2 additions & 2 deletions examples/reconciler/README.md
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
# Durable reconciler

This example deploys two static sites from an ordinary Node.js program using YieldStar 0.5.0 for durable execution, state, retries, waiting, and deployment coordination.
This example deploys two static sites from an ordinary Node.js program using Yieldstar 0.5.0 for durable execution, state, retries, waiting, and deployment coordination.

[`src/index.ts`](./src/index.ts) owns the outer workflow and Node SQLite runtime. It passes YieldStar's `step` context to `deployWithYieldStar`, while [`src/static-site.ts`](./src/static-site.ts) contains only the desired resources and provider lifecycle operations.
[`src/index.ts`](./src/index.ts) owns the outer workflow and Node SQLite runtime. It passes Yieldstar's `step` context to `deployWithYieldstar`, while [`src/static-site.ts`](./src/static-site.ts) contains only the desired resources and provider lifecycle operations.

Run it from the repository root:

Expand Down
8 changes: 4 additions & 4 deletions examples/reconciler/src/index.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -8,9 +8,9 @@ import {
createSqliteDb,
} from "@yieldstar/sqlite-runtime/node";
import {
YieldStarStateBackend,
YieldstarStateBackend,
createResourceRegistry,
deployWithYieldStar,
deployWithYieldstar,
} from "@notation/reconciler";
import pino from "pino";
import { createWorkflowRouter, workflow } from "yieldstar";
Expand All@@ -24,7 +24,7 @@ const schedulerClient = new SqliteSchedulerClient({
timersClient: new SqliteTimersClient(database),
});
const storeClient = new SqliteStoreClient({ db: database, schedulerClient });
const state = new YieldStarStateBackend(storeClient, "static-sites");
const state = new YieldstarStateBackend(storeClient, "static-sites");

const resources = [
new StaticSite({
Expand All@@ -44,7 +44,7 @@ const resources = [
];

const deploy = workflow(async function* (step, event) {
yield* deployWithYieldStar(step, {
yield* deployWithYieldstar(step, {
deploymentId: "static-sites",
executionId: event.executionId,
resources,
Expand Down
Loading
Loading