Uh oh!
There was an error while loading. Please reload this page.
Move reconciliation to the durable Yieldstar runtime - #45
Merged
Conversation
Introduce the durable reconciliation modules under durable/, with the Yieldstar runtime confined to a single adapter seam. Adds @notation/utils and moves error-matcher classification next to ErrorMatcher in @notation/resource as findErrorMatcher. The legacy reconciler remains in place until the runtime cutover.
The durable planner and operations now treat an undefined read as absence, and a tagged ResourceNotReadyError as a wait: durable operations retry it, while planning reports an indeterminate node carrying the message.
Run deploy, plan, and destroy through NodeDurableRuntime, unify CLI error handling, and remove the legacy reconciler, operation workflows, and state leasing that the old runtime required.
The durable path reimplemented create, update, read and delete inline rather than using the StepRunner seam those operations were built for, leaving two copies of the same provider protocol to keep in step. Persistence becomes a driver-supplied step. In process that is a compare-and-set against the revision read before the operation; in a workflow it is a store write stamped with the step that made it, so the applied-step ledger and the state change commit together and a replay returns the recorded result instead of retrying a stale compare-and-set. Event emission becomes a step for the same reason. Scoped step keys let one operation body run at several call sites, which replaces the hand-threaded key prefixes and the parameter bags that carried them. With writes owned by the workflow, DurableStateBackend is read-only for the planner and reporting, so the revision arithmetic and the creation token that guarded create-if-absent both go. Also collapses the three copies of the reconciler event types, delegates Reconciler.plan to createPlan, and exports the durable subsystem, which no consumer could previously import.
A delay whose deadline has already passed is continued inline rather than suspended, so a one millisecond retry could resume inside the execution that scheduled it once the heap write outlived it. The tests then saw two provider calls where they expected the workflow to have parked after one. Names the interval and puts it comfortably beyond a SQLite write. The assertions are unchanged: the provider is still called once before the workflow parks, and once more after it resumes.
The hold was released in a finally, so every failure mode released it — except a crash, where the runner abandons the workflow generator without calling .throw() and the finally never runs. That split matters because yieldstar caches steps per execution: a resumed execution replays `take` from the cache without re-acquiring, and replays `release` as a cached no-op, so a resumption that followed the release path proceeds holding nothing while it mutates the deployment. Unify every failure mode on the crash path's semantics. The hold is released in exactly one place, after the body completes. An execution that will never resume now holds its deployment forever, which is why takeOverDeploymentHold lands with it: force-clearing the coordination store through storeClient.updateStore is unconditional and undocumented, so without a named, compare-holder takeover this commit would turn any transient provider error into a wedge with no runbook. The test cannot be written against a failing step: a cached StepError is rethrown before the step's function runs again, so the resumption would never reach uncached work. It fails in plain generator code between two resources instead, and asserts the holder observed from inside the second resource's create — null before this change, the execution id after it.
createResourceWorkflow, readResourceWorkflow, updateResourceWorkflow and deleteResourceWorkflow had no consumers, and each was typed `unknown`, so nothing could have called one without casting anyway. They were also the only runtime yieldstar import reachable from the base entry point, which made the durable runtime look like a dependency of the in-process driver. YieldstarApi goes with them: `export type`, so never a runtime import, and equally unused. operation.workflows.test.ts exercises the operations directly despite its name, so it keeps its coverage. After this, the only "yieldstar" imports under src are in durable/.
Both are persisted by create and update, but the schema was loose, so neither appeared in StoredResourceState and nothing checked that a durable record carried the same fields as an in-process one. The store stays loose. PersistedResourceState deliberately carries an index signature, and v.object strips unknown entries on parse, so tightening the object would silently drop forward-compatible fields at the store boundary. The schema is a floor on the record, not a description of it. Validation runs on write rather than read, so records persisted before this commit are still readable; the next write to one supplies both fields. The test fixture that seeds a record directly now supplies them.
Operations resolved two things for themselves that the driver had already
decided against: the desired params, through resource.getParams(), and
the persisted record, through a `state: Pick<State, "get">` seam that
existed for a single read in operation.read.ts.
Both are now passed in. getParams is resolved once per scheduled
reconciliation — conflict recovery included, so recovery compares the
remote against the same params the failed attempt decided on — and
persistedOutput replaces the state seam, which also deletes the fake
`state: { get: async () => stateNode }` adapter the durable driver had to
construct.
This is not a live correctness fix. getParams spreads config over
deriveParams(id, config, deps) and never reads this.output, so repeated
calls do not diverge today. The win is that deriveParams is user code:
a non-deterministic one would let the emitted decision diff and the
persisted params record disagree with nothing able to reconcile them.
Deletion keeps resolving params lazily, on the recovery path only, since
that is the sole consumer there and an orphan hydrated from state should
not be asked to derive params just to be deleted.
Three construction sites changed: reconciler.ts, durable/operations.ts
and planner.ts, which already had both the state node and the resolved
plan params in hand and was resolving params a second time inside the
drift read.
Also adds the typecheck script the package never had, which is how the
call sites still passing the long-deleted expectedRev field surfaced.
Making the package typecheck exposed two more things: the durable store
schema now requires groupId and groupType that PersistedResourceState did
not declare (declared here, since StateNode only carries them through its
index signature), and a batch of test fixtures that were never type
checked. The fixture casts are annotated: a resource declared without API
types cannot express a named schema key at all.Scoping was a durable-only wrapper that rebuilt a step object. It is now part of the StepRunner contract: scope(prefix) is the identity in process, because createStepRunner ignores keys entirely, and the prefixing wrapper in a workflow. scopeStore goes with it. A store outlives the scope that opened it, so prefixing its caller-supplied keys with whichever scope happened to open the handle was arbitrary; the three call sites now qualify their own keys, as state:persist:<id>, state:delete:<id> and the notation:coordination:* keys, which were already fully qualified. The keyless run/delay overloads are gone. Yieldstar derives a key from the call site for those, and through a scoping wrapper that call site is the wrapper's, not the caller's — so any two keyless steps reached through one scope would have collided. Nothing called them; removing them makes that structural rather than a convention. Keys cost nothing in process, where they are ignored. BLAST RADIUS. Store keys change, so an execution that was in flight before this commit and resumes after it must be drained or abandoned first. It will not refuse cleanly: plain run steps have no idempotency ledger, so a resumed execution takes a cache miss and RE-EXECUTES the provider mutations behind those keys. It then fails safe at the first conditional state write, because updateFrom/deleteFrom carry a snapshot that is now stale and abort with RevConflict — but the remote calls will already have happened. Registered store waiters are keyed by stepKey too, so a `take` suspended on the deployment hold re-registers under its new key when it next wakes. The key shapes now in use are documented at the durable entry point, where they can be read as the contract they are.
The durable driver threaded dryRun into the drift read's params, so
readResourceOperation returned {} without calling the provider. decideAction
then diffed toComparable({}) against the desired params, found every param
"missing" from the remote, and upgraded a noop into a drift-update carrying
a fabricated diff. Every dry-run test set driftDetection: false, so nothing
caught it.
The in-process driver never had this: #readForDrift deliberately omits
dryRun, and createPlan does the same. The contract is that a dry run
suppresses mutations, not reads — a dry run that cannot read the remote
cannot report drift, which is most of what it is for. So this matches the
in-process behaviour rather than guarding the read.
The new test pins both halves: the provider read happens during the dry
run, and the decision stays noop with no drift event.Conflict recovery re-reads the remote, re-decides against it, and can land on drift-update — but it emitted only deploy.decision, so the drift.detected event that the same decision produces on a first pass was missing. Recovery after a conflict is drift adoption, so it is owed. This is a visible change for anything diffing event sequences: a recovering deployment now emits reconciler.drift.detected before its deploy.decision, where before it emitted only the decision. It lands on its own so that it is not buried in the driver unification that follows.
takeOverDeploymentHold is the only exit from a hold left by an execution that will never resume, so it ships tested: it takes when the named holder still holds, and refuses — reporting the actual holder, leaving the record alone — when the hold has moved on. The key map gains the destroy-path orphan sweep, which it omitted, and a note that resource ids are spliced in unescaped. That ambiguity predates the map and is recorded rather than fixed, since changing key composition invalidates in-flight executions.
The per-resource algorithm existed twice — #deployResourceOnce plus #recoverDeployResource in process, reconcileResource in the workflow — and the copies had already drifted apart. There is now one generator, src/reconcile.ts, that owns hydration, the decision, the drift read, event emission, dispatch, recovery adoption and deletion recovery. Both drivers run it. What stays with a driver is what genuinely differs: dependency-level scheduling and concurrency (Promise.all under a mutation lease against a sequential yield* under a deployment hold), the conflict retry policy, how a state session is opened, and how a step is run. Two seams carry that split. The driver passes openSession(resource), a factory rather than an open session, because recovery re-reads and persists against what the winning writer left; the first pass opens once and a recovering pass opens again. And ResourceStateSession is a discriminated union, so `remove` exists only alongside a `node` — the precondition deleteResource expressed as an early return and destroy expressed as skip-when-absent, now expressed in the type. Keeping the writes bound to the read that produced them is what stops a node being paired with a precondition from a different read. Recovery is a recoverFrom parameter rather than a second near-copy of the algorithm. The behaviours that were easy to lose in that collapse are each still pinned by an unchanged test: a recovering noop persists the adopted record with lastOperation "drift" at the re-read revision (guarded by dryRun), a resource that cannot be read cannot be recovered, and the drift read carries no dryRun. Orphan sweeping is aligned too. In-process destroy now sweeps, which also means destroyApp no longer runs a refresh pass first — but the Reconciler it builds must be given the registry, or the sweep silently falls back to the types of the resources still declared and skips, with a warning, exactly the orphans a destroy exists to remove. registry is optional, so nothing else would have caught that. Note the ordering flip: in-process destroy used to sweep before deleting declared resources and now sweeps after, matching the durable driver and deploy's act-then-sweep shape. Public refresh is unchanged, for standalone cleanup. The #emit/#emitStep split collapses with it: OperationLifecycleEvent is a subset of ReconcilerEvent, so one scoped emit seam serves decisions, drift and operation lifecycle alike. reconciler.ts 511 -> 361 lines, durable/operations.ts 252 -> 197, with 248 shared between them.
Combines the yieldstar-node-runtime-cutover branch with the last eleven foundation commits it predated. The cutover's core, CLI, docs and state changes land as written; inside the reconciler package the foundation's architecture wins: the shared operations library and reconcile algorithm stay, and the cutover contributes the deletions of the in-process Reconciler driver and its tests. Resolutions beyond file picks: - Core and the example import the durable driver from the @notation/reconciler/durable subpath rather than a root re-export. - The legacy-state migration writes through the store client's getOrCreateStore, since DurableStateBackend is now read-only. - The dashboard declares the read-only state it actually needs.
This was referenced Jul 28, 2026
The hand-rolled standard-schema validator predated the convergence on Valibot for store schemas; this brings the one store defined outside the reconciler in line with the others, and lets the schema type the store state instead of a cast.
# Conflicts: # pnpm-lock.yaml # pnpm-workspace.yaml
The two-driver seam (reconcile.ts with OpenStateSession, EmitFromStep and the recoverFrom recovery paths) outlived its second driver: the in-process reconciler was deleted when the runtime cut over, leaving the durable driver as the sole caller and the recovery machinery unreachable. Inline the algorithm into durable/operations.ts, preserving every step key, and drop the dead recovery code. The RevConflict tests still expect conflicts to propagate, which is what the durable driver always did. Also extract withRuntime so deployApp, destroyApp and planApp share one runtime acquire/close lifecycle instead of three copies.
The runtime's event pump routed completion through a shared promise, a completed flag and three callbacks, with a dead branch for replay errors on executions the pump never feeds it. Replace it with driveToCompletion: run the trigger event, then loop the queue until the runner returns a result, deferring foreign tasks exactly as before. Errors now propagate by throwing instead of by rejecting a captured resolver. Also extract runDurableWorkflow so deployApp and destroyApp share the one-command-one-workflow wrapper instead of each building a router, and declare DurableStepRunner as an interface extending StepRunner rather than restating its members alongside a conditional-type assertion.
The legacy-state migration verified every shared record twice: once from the durable listing, then again per legacy node through a fresh store read whose mismatch branch could no longer fire. Import from the ids the first pass already verified instead, dropping a store round-trip per record. DurableStateBackend.has had no callers, so remove it.
- coordination.ts -> deployment-hold.ts; acquire/releaseDeploymentCoordination -> acquire/releaseDeploymentHold. One concept, one name: the identifiers now all say hold, and the module head records that the persisted store name, step keys, and event string keep the older coordination wording. - durable/operations.ts -> reconcile.ts, so 'operations' means only resource CRUD, as it does everywhere else in the package. - DurableOperationOptions -> DurableWorkflowOptions for the same reason. - Docs: correct the store names (resource-state, deployment-coordination) that three docs still gave with a dropped notation/ prefix, point the manual's embedded example at @notation/reconciler/durable, and document the deployment-hold escape hatch (takeOverDeploymentHold) where operators will look. Drop the Yieldstar version pin from prose that would drift; it stays in package.json and the RFC's scope line. - Say why the custom event pump exists (Yieldstar's event loop is a resident server; a command needs one execution run to completion), what initialize() is for, and that store names are persisted contract like step keys.
Nothing on this branch has shipped: @notation/reconciler and @notation/state were never published, and the released core@0.11.1 never wrote .notation/state.json. So the persisted strings, the legacy migration, and the file/SQLite state backends had no dependents. - Rename the deployment hold's persisted strings to match its name: the deployment-hold store, notation:hold:* step keys, and the reconciler.hold.waiting event. Delete the comment that apologised for the old coordination wording. - URI-encode resource ids inside step scopes and give the orphan sweep one scope (notation:orphans) in both workflows, making the key delimiter unambiguous instead of documenting the ambiguity. - Delete the legacy state.json migration, and with it initialize(), the legacyStatePath option, and FileStateBackend. - Delete the orphaned @notation/state-sqlite package. - Move the orphan-deletion-skipped event next to the other events and inline its single-use factory; name event types after their event strings (DeployDecisionEvent, DriftDetectedEvent, HoldWaitingEvent, OrphanDeletionSkippedEvent). - Drop the runtime's unused params option, its redundant per-task re-binding, and its Yieldstar-branded log and error wording. - Trim comments to what is and what binds: no history, no defences.
- Namespace every Notation-owned store name as notation/<name>: notation/resource-state and notation/deployment-hold join notation/execution-binding, so an application sharing a store client with these workflows cannot collide with them. Documented in the durable index key map and the manuals. - Rename NOTATION_STATE_PATH to NOTATION_DATABASE_PATH: it selects the whole workflows SQLite database (heap, timers, task queue, stores), not a state file, and the code's own constant already said so. - Split decideAction: the driftRead arms that could not fire (create on an absent remote, plain update) existed only because one function served two moments. decideDriftAction now takes over after a noop, where a state node necessarily exists and local params match it. - Gate the drift read on resource.read in the durable driver, as the planner already did: a resource with no read has no remote to compare, and its noop now stands without read skip/success lifecycle noise. A new test locks the shared behaviour in. - One name per concept: drop the State alias of StateBackend, the DurableDestroyOptions alias of DurableWorkflowOptions, and the unused StateBackend.has(). - Callers now pre-scope the step for reconcileResource as they already did for deleteResource; step keys are unchanged. - Rename the statePatch test helper to resourceStateRecord: it returns a full record, not a patch. - withRuntime rejects runtime plus databasePath instead of silently ignoring the path.
Wire emit back through planApp and the CLI plan command, as on main: deploy and destroy still surface reconciler events through the same seam, and plan-time drift reads call remote providers, so their lifecycle belongs in front of the user. createPlan's emit option has a caller again. Drop the example's undeclared-in-source dependencies, state the hold inventory's and waiting event's exact bindings, and let sweepOrphans document itself once.
takeOverDeploymentHold clears the hold, full stop: the unused toExecutionId transfer parameter is gone. Deployment identity now lives only on DurableStateBackend, which already carried it in its store prefix; DurableWorkflowOptions no longer repeats it and the deployment hold keys off the backend. The durable module is imported as `durable` everywhere instead of shadow-naming the reconciler package. Emitter absence is absorbed in one layer — the adapters (toEmitStep, durableEmitter) — so operations always hold an emit step and neither the planner nor emitLifecycleEvent guards again. Docs drop the sample's duplicated deploymentId and the sentences describing what state is not.
- lastOperation persists only what is written: create and update. Drift repair persists update, and delete removes the store, so the drift and delete members were unreachable; the package is unpublished, so no persisted record carries them. - Drop the unconsumed ResourceApi, StateApi and DeepObjectDiffApi exports. - StateBackend is now the read interface its consumers use; the compare-and-swap update/delete half existed only for test seeding, which MemoryStateBackend's constructor already covers. - Cover notation:orphans:list in the durable key map and give the emit entry the *: scope prefix its keys always carry. - Say "registered" where the code says resource-type-not-registered, keep "load the persisted record" distinct from the drift read, name toEmitStep/durableEmitter/createStepRunner instead of "the in-process driver", align the decision table with delete-orphan, and state the no-tombstone fact once, in docs/internals/state.md.
- Type store snapshots as PersistedResourceState and drop StoredResourceState; a satisfies assertion pins the valibot schema's output to it. - Delete the PlannerState alias; createPlan takes StateBackend. - Expose the store version directly as StateNode.version, removing the one-based rev shim; RevConflict becomes VersionConflict, its unread fields now constructor arguments that only feed the message. - Inline the vestigial resourceParams aliases in the create/update/read operations. - Rename ReadResourceParams to ResolvedResourceParams: the neutral base create and update extend, and what read takes exactly. - Rename runWithCliErrorHandling to runWithErrorHandling, matching its file name.
… crash-window contract one home per audience - removeResourceState now distinguishes a version conflict (naming the actual version) from a genuinely absent record, instead of reporting every failed deleteFrom as missing; the delete-race test locks the distinction in. - resourceType is ResourceType on all four resource-bearing reconciler events instead of string on two of them. - The crash-window/idempotency contract is stated once for CLI readers (docs/cli/deploy.md) and once for internals readers (docs/internals/reconciler.md, moved to Waiting and replay where it applies to all provider calls); destroy.md, the manual, and the RFC now reference those homes instead of restating it.
- Inline OperationEventEmitter as EmitStep<OperationLifecycleEvent>, so "emitter" names only the plain callback and the step seam is named as a step. - Rename the deploy scope key notation:resource:<id> to notation:deploy:<id>, symmetric with notation:destroy:<id>, and drop the key map's parentheticals. - Rename the scoped step key state:snapshot to persisted-record, so the state: prefix means only store-handle keys, as the key map says. - Extract applyDriftDetection: the noop-plus-drift-read gate and its driftDetection default now live once, used by both reconcileResource and createPlan; the redundant outer defaults in deployApp and planApp are gone too. - Require executionId through runtime.run, runDurableWorkflow, and deployApp/destroyApp: the outermost caller generates the resume handle and reports it, so no layer silently defaults an ID the caller can never learn. - Reorder deploy.md's "What happens": ordering precedes reconciliation.
…rant - Rename takeOverDeploymentHold to clearDeploymentHold (DeploymentHoldTakeover to DeploymentHoldClearance, taken to cleared): the operation clears a hold, it does not transfer one, and "take" already means acquire in this file. - deleteResourceOperation now treats ResourceNotFoundError from a delete handler as success: absence is delete's goal state, which makes crash-window replayed deletes idempotent as the docs already claimed. Test added. - docs/cli/deploy.md: the compiled output is imported in-process by the CLI; the worker does not exist yet.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
This PR moves deploys onto a durable workflow runtime (Yieldstar). Before, a deploy ran in memory: if the process crashed halfway, all progress was lost and state could be left inconsistent. Now every step of a deploy is saved to a local SQLite database as it completes. A crashed or interrupted deploy can be resumed and will continue from the last saved step instead of starting over.
How it works
takeOverDeploymentHold..notation/workflows.db. On first run, an existing legacy JSON state file is imported and then renamed to*.migrated.Changes by package
@notation/reconcilerdurable/module (exported as@notation/reconciler/durable) runs it as Yieldstar workflows. The old in-processReconcilerclass is deleted, along with its locking and conflict-retry code.@notation/coreNodeDurableRuntime: owns the SQLite database, the workflow event loop, and the legacy state import. The deploy/destroy/plan workflows run through it. The refresh workflow is gone (destroy now sweeps orphans itself).@notation/clideployanddestroygain--execution-id(resume a run) and--json(stream events as NDJSON).@notation/state@notation/dashboardexamples/reconcilerNotes for review
notation:resource:<id>:create:remote:attempt:0) are a public contract: changing one makes a resumed execution redo that work. The key map is documented inpackages/reconciler/src/durable/index.ts.packages/stateandstate-sqlitechanges alter semantics, not just size — worth focused review; they are easy to miss among the deletions.