Skip to content

fix(dag): host-level supervision sweep settles nodes orphaned by instance teardown - #338

Merged
LeXwDeX merged 7 commits into
devfrom
fix/dag-node-supervision
Aug 18, 2026
Merged

fix(dag): host-level supervision sweep settles nodes orphaned by instance teardown#338
LeXwDeX merged 7 commits into
devfrom
fix/dag-node-supervision

Conversation

@LeXwDeX

Copy link
Copy Markdown
Owner

Incident (2026-08-18, dag_fe5feabfcae607fqVdRh47lN1B)

A per-directory instance teardown mid-run silently reaped every fiber in its scope — DagLoop subscriptions, the spawn fiber, AND the deadline watcher — while the durable node row stayed running. In a live host nothing re-arms supervision, so sg4-integrate rotted in running for 7.5h+ past its deadline with escalation_pending=0, timeout_extensions=0 (the sg4-commands sibling verdict_fail was correct engine behavior). Recovery can settle such rows, but only re-init/crash triggers it.

Fix

New host-level service DagSupervisionSweep (src/dag/runtime/supervision-sweep.ts), wired into AppLayer:

  • A repeating 60s sweep fiber forked into the LAYER scope at construction — it survives per-instance teardown and never depends on ambient InstanceRef.
  • Detects supervision death by flat-streak: timeout_extensions unchanged for frozenTicksNeeded(escalateIntervalMs) = ceil(max(interval,1s)/60s)+1 consecutive ticks, with the cadence read per-node from the persisted workflow config (defensive parseWorkflowConfig; every degrade path lands on the DEFAULT 10-min cadence — the widest guaranteed-safe window). A live watcher at ANY configured timeout (incl. 30-min verifier timeouts) always moves the counter inside the window.
  • Settle = cause-recovered best-effort child cancel (the layer context has no InstanceRef; a real cancel dies and must never block the settle — mirrors recovery.ts's optional cancelSession) + nodeFailed(..., "timeout") under the durable terminal-status guard (cross-host double settles collapse to one).
  • Every recovery site propagates interrupts (repo loop discipline) and gates the success log/streak-drop on the settle outcome; failed settles retry next tick.
  • Known adjacency (out of scope): queued/paused overdue nodes are not covered — deadline rows only exist once running.

Tests

test/dag/dag-node-supervision.test.ts — incident harness reproducing the supervision shape at 2s deadlines: healthy control, stream-hang (incident shape), dispose-instance (mechanism + sweep fallback settle), cancel-defect (production InstanceRef die never blocks the settle; mutation-verified), freeze window (live watcher never swept, production cadence relationship), plus pure unit tests for cadence derivation/degrade and window boundaries. 9/9 pass; typecheck + lint ratchet green.

Reviewed through a 6-round fixed-point loop (R1–R6); R5/R6 consecutive PASS with zero findings.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@LeXwDeX