fix(dag): session-following location stamps + adoption-vs-deletion fencing (#269, #270) - #272
Merged
Conversation
…onMoved (#269) Root cause (#269): the execution-location key (WorkflowTable.directory) was stamped at dag.create and never moved afterwards, while a SessionEvent.Moved only rewrote SessionTable.directory. ownsSession conjuncts the directory over ALL of a session's workflow rows, so after a move the rows disagree (the pre-move workflows keep the old stamp) and the fail-closed ownership policy leaves EVERY directory with no owner — the session's wakes wedge until restart. Probe C4 pinned the wedge rather than fixing it (v1.0.13 decision). Fix (design-gate Model a + b integration, atomic adoption): - C1: the session projector's Moved projection now re-stamps every workflow row of the moved session to the payload-sourced destination directory, inside the SAME durable publish transaction as the SessionTable update — the stamp moves WITH the session in one transaction, so there is never a mixed-stamp window (no SessionTable read, no fs I/O in the projector). - C5: DagLoop subscribes to SessionEvent.Moved. The instance whose directory no longer owns the moved session's workflows evicts its stale runtime entries (fail-closed), and the NEW owner re-forks the serialized wake drain so a wake that was wedged behind the old stamps delivers immediately (bounded time) rather than at the next idle event or restart. - C6: no-ALTER idempotent convergence migration converges pre-fix divergent stamps to the session's current directory, preserving fail-closed NULL rows. Probes (red-first; each was RED before the corresponding seam landed): - C4 flips from pinning the wedge to pinning the resolution: exactly one live owner = the session's new durable directory, no cross-directory double adoption. RED evidence: wf1 stayed "/wtA" pre-C1. - C8 pins the bounded-time resolution: a moved session's wedged wake reaches the new owner's admission seam immediately, delivered exactly once. RED evidence: the wake stayed wedged (poll timeout) pre-C5. - C9 pins the migration convergence (fail-closed NULL preserved). RED evidence: a no-op migration leaves the stale stamp divergent. Invariants preserved: fail-closed NULL, session-sourced create stamp, H1 latch, R7 negative half. Only the write-once rule is narrowed to a two-writer whitelist (create INSERT + Moved re-stamp) — encoded in the follow-up R7-ext(a) amendment.
Root cause (#270, shared territory with #269): admission was check-then-act — an instance re-read the durable row (ownsWorkflow) and then published its in-memory entry / materialized a child session. A deletion (Session.remove → FK cascade) that commits between the passed ownership check and the action slipped past it: the in-flight spawn could create a child for a workflow the cascade had already removed. SpawnReady eviction was the mitigation, not a fence. Fix (Model (b) conditional-claim fence of the accepted atomic-adoption design — status+directory conditionality, no lease column): - C2: DagStore.tryClaimAdoption — a single conditional UPDATE that matches the row only while it STILL EXISTS and is non-terminal, returning whether the admission claimed it. It converts the admission from check-then-act into act-with-condition; a committed deletion/terminal matches zero rows and the admission aborts atomically. Directory ownership is deliberately NOT re-done here: ownsWorkflow canonicalizes both sides and a SQL directory comparison would diverge (create stamps are realpathed, Moved re-stamps are not). - C3: recoverWorkflow and the WorkflowStarted handler gate runtimes.set on the claim (after the ownsWorkflow directory revalidation), so a cascade in the final window aborts the adoption before it ever publishes an entry. - C4: spawnNode re-admits atomically right before sessions.create — the fence that closes the spawn window nodeQueued's guard alone leaves open between its read and its publish. A committed deletion matches zero rows and the spawn aborts before any child session exists. Probe C7 (red-first, park-gate style): parks the window-2 claim, commits a deletion, releases, and asserts no post-deletion child survives. RED evidence (fence conditionality reverted → always-claim): a child session is created-then-cancelled for the deleted workflow, so `expect(cancelsA).toEqual([])` fails (Received +3). Restoring the conditional claim makes the claim return false on the deleted row and abort before sessions.create (GREEN). All existing DAG-LOC-01 guards stay green (24 probes).
…ext(a) (#269) Root cause (#269): the atomic-adoption redesign narrows the DAG-LOC-01 write-once directory invariant into a two-writer whitelist. R7-ext(a) previously asserted "no UPDATE writes the directory column anywhere in the dag trees," which held but left the new whitelisted writer (the Moved re-stamp) unpinned. This is an invariant UPDATE, not a weakening — mandated by #269's own acceptance ("the directory stamp must move WITH the session in one transaction"): - The ONLY sanctioned directory writers are the dag projector's WorkflowCreated INSERT (create-time stamp, onConflictDoNothing) and the session projector's SessionEvent.Moved re-stamp (payload-sourced, no SessionTable read). - R7-ext(a) keeps its negative half (no .set({directory}) in the dag trees) and gains a positive whitelist half asserting the Moved re-stamp exists in the session projector (outside the dag trees, so R7/R7-ext stay scoped correctly), pinning the #269 resolution at the static-contract level. - WorkflowTable.directory column comment documents the two-writer whitelist. RED evidence (mutation reversion — Moved re-stamp removed from the session projector): the whitelist expect fails, Expected true / Received false. No user-visible semantic change; no runtime behavior altered in this commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
What
tryClaimAdoption): status whitelist['pending','running','paused','stepping'](exactly the non-terminal WorkflowStatus set); no lease column, no fail-open gap. Gated afterownsWorkflowat both adoption sites and at spawn re-admission (fix(dag): vs-deletion adoption tail — adoption can slip past a workflow's own deletion within the revalidation window #270).Red-first evidence
M1 revert C1 → 22/2 RED on C4+R7-ext · M2 revert Moved subscription → 23/1 RED on C8 · M3 migration no-op → 23/1 RED on C9 · M4 always-claim → 23/1 RED on C7 · M5 drop spawn re-admission → 23/1 RED on C7. All restores byte-identical (sha256-verified).
Gates
Typecheck exit 0 ·
test/dag test/goal test/project721 pass / 1 skip / 0 fail (58 files) · 24/24 location probes (21 baseline + C7/C8/C9, C4 flipped to pin resolution) · migration --check exit 0 · license-scope 5 pass · lint 4849/0 (budget 4850).Review: design gate ACCEPT (combined model) + independent verify (mutation re-proofs) + standards/intent arbitration ACCEPT.
Known residual
Bounded re-stamp-vs-adoption race (
tryClaimAdoptionbumpstime_updatedwith wall clock) — documented for follow-up. Directory-authority asymmetry (create stamps realpathed, Moved re-stamps not) documented at the claim site.Closes#269, closes#270.