feat(ci): stall guard gets a second signal — source-side write progress, with the CPU inversion designed out

[os-steve]

Fixes#11855

Adds a second liveness signal to scripts/run-with-stall-guard.mjs beside output flushes: the bytes every process in the wrapped group has passed to write(), read from /proc/[pid]/io (wchar). It is the same quantity the guard already trusts, sampled at the source — before any buffering layer can hide it.

--stall-minutes is untouched.

What a genuinely frozen suite looks like to this probe

Zero bytes written by any process in the group, for the whole stall window. Frozen means produced nothing, not did nothing — and specifically not "no CPU", because CPU is the one signal that does not work here.

Measured before designing anything (node 22.22.2, 2.5s sample, one process per shape):

shape	state	ΔCPU (ticks)	Δwchar (bytes)
idle hang (self-test case 4)	S	0	0
sync-spinning hang (case 5)	R	251	0
GC-thrash hang	R	272	0
healthy silent-but-working	S	1	25,542

Read the CPU column before the wchar column. The two genuine hangs peg a core, while the healthy suite the probe exists to protect is nearly idle — it is I/O-bound, it is busy writing. So direction (a) does not merely fail to separate these shapes, it separates them backwards: a CPU-confirming probe would have protected the hangs and killed the healthy suite. That is worse than the card anticipated, and it is why (a) lost on measurement rather than on taste.

wchar separates all four correctly, and it is the quantity the guard actually cares about.

Why direction (b) lost as well

The vitest reporter side-channel measures the right quantity, but it is not reachable at proportionate cost in this repo: there is no shared vitest preset. 31 packages, 14 with their own vitest.config.ts, every test script a bare vitest run, and the guard wraps pnpm turbo run test — so there is no single place to inject a reporter. (b) is a 31-file rollout through the test infrastructure of every package, for a CI-tooling card.

wchar buys the same protection with zero producer wiring, no code injected into any test process, and it is tool-agnostic — vitest, turbo, sh, and whatever replaces them. It also closes more of the class: a reporter only covers buffering layers below itself, while wchar covers every layer above the write() syscall, which is the whole pipeline.

How the inversion is designed out

The probe is a confirming signal in exactly one direction — it can only ever delay a kill, never prevent one:

• no output AND no source-side bytes → stall, killed at --stall-minutes, exactly as before. Idle and spinning hangs take this path with no added latency: they write nothing.
• no output BUT bytes still moving → the kill is deferred, and the deferral is announced loudly. A live suite hidden from the guard by a buffering layer is a bug to fix, not to tolerate, so the note names the cause and points at --log-order=stream.
• …until --stall-cap-minutes → the group is killed anyway under a distinct STALL-CAP verdict, which states that the group kept writing and names the two shapes that look like that.

The cap is the entire hazard containment. Without it, a hang that writes while wedged — a retry loop, a poller, a log spin — looks alive forever and the guard never fires, which is strictly worse than the defect being fixed: the job then sits in_progress until the job timeout, the exact state this guard exists to abolish, and no green run can tell that regression from success.

The cap ceiling is measured, not chosen. Every guard-wrapped job in ci.yml runs timeout-minutes: 30 against --stall-minutes 10, so a cap set "far above" the window would just hand the verdict back to the job timeout. The default is 2 × the window, and a cap at or below the window is refused at startup — a probe that is silently off is indistinguishable from one that works.

If /proc/[pid]/io cannot be read (not Linux, hardened permissions), the probe reports itself UNAVAILABLE and the guard behaves exactly as before this signal existed. An unreadable counter must never read as liveness.

The three required conditions, holding simultaneously

pnpm check:stall-guard — 41/41 pass (27 assertions before, +14 here):

 ✓ idle hang is declared a stall
✓ idle hang is not rescued by the liveness probe (plain verdict, no deferral)
✓ sync-spinning hang is declared a stall
✓ sync-spinning hang is NOT rescued by the probe — burning CPU is not liveness
✓ a healthy silent-but-working run is NOT killed
✓ ...and is never called a stall
✓ ...and the deferral is announced, not silent
✓ positive control: the same run IS killed with --no-liveness-probe
✓ a hang that keeps writing is still killed
✓ ...at the cap, under its own distinct STALL-CAP verdict
✓ a --stall-cap-minutes at or below --stall-minutes is refused

The positive control is what makes the healthy case evidence rather than coincidence: the same child, same window, --no-liveness-probe, must die. Without it a probe that did nothing at all would have scored identically to one that works.

Ablation — the new cases can actually fail

Two mutations, each proved on disk in both directions (injected text present / removed text absent), each node --check'd so a crash-red cannot be mistaken for an assertion-red, each restored under trap restore EXIT INT TERM, with the restored tree re-verified pristine:

mutation	predicted	observed
sourceAlive forced false (probe can never defer)	healthy case turns red	5 cases red — a healthy silent-but-working run is NOT killed — exit 75
capped forced false (hard cap removed)	writing hang never killed	3 cases red — the guard never exited — the probe inverted the defect, exit null

M2 is the load-bearing one: with the cap removed, the guard literally stops firing and the self-test times out. That is the inversion the card warned about, reproduced on demand and then closed.

Gates

Union run at fbc723061, derived live with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (never a hand-written list) — all 8 named families plus check:nul-bytes, exit codes captured before any pipe:

check:stall-guard → ✓ 41 case(s) passed — the guard fires, classifies and tears down.
check:agent-test-spelling → exit 0
check:cross-package-test-inputs→ OK: 16 package(s) read outside themselves, all declared
check:entry-guard → ✓ 154 scripts/ file(s) — every entry guard goes through invoked-as.mjs
check:parse-guard → exit 0
check:pnpm-filter-targets → ✓ 135/168 `--filter` occurrence(s) across 26 file(s) resolve
check-ci-filter-parity → OK: all 96 declared cross-package glob(s) (81 unique) are covered
check-cross-package-test-inputs→ OK: 16 package(s) read outside themselves, all declared
check-nul-bytes → check-nul-bytes: OK (scanned 6631 text file(s))

check-ci-filter-parity first exited 1 with PREREQUISITE NOT MET — the dependency yaml is not installed; it says of itself that "nothing was measured … this result says NOTHING about what it gates". Green above is after pnpm install in the worktree.

Changeset

skip-changeset: the diff is one CI-tooling script under scripts/. Nothing published, no package surface, no user-visible behaviour — the change is only visible to CI and to whoever reads a stall verdict.

---

Generated by Claude Code

[os-steve]

ACCEPT — and this is the strongest result of the shift. Flipped out of draft (scripts/** is not governed); arming once CI is green.

⭐ Both routes the card offered were falsified by measurement, and you shipped a third. The card said those two directions "survive the hazard at first look, to be measured, not assumed." You measured, and they did not survive. That is exactly what the Zone 2 fence is for, and it is worth more than an implementation of either.

The CPU finding is the part I want on the record, because it is worse than the card feared. The card worried CPU would fail to separate the shapes. Your table says it separates them backwards:

shape	ΔCPU ticks	Δwchar bytes
idle hang	0	0
sync-spinning hang	251	0
GC-thrash hang	272	0
healthy silent-but-working	1	25,542

A CPU-confirming probe would have protected the two hangs and killed the healthy suite — the inversion the card named, but arrived at from the opposite side. Route (b) losing on cost rather than correctness (no shared vitest preset; 31 packages, 14 own configs, every script a bare vitest run) is a proportionality judgment I agree with and would not have reached without the measurement.

Verified independently, not read back

I checked out your head fbc723061 and ran the self-test myself: ✓ 41 case(s) passed, exit 0. The card's three conditions hold simultaneously in my own run, and I can see all of them:

✓ idle hang is declared a stall
✓ idle hang is not rescued by the liveness probe (plain verdict, no deferral)
✓ sync-spinning hang is declared a stall
✓ sync-spinning hang is NOT rescued by the probe — burning CPU is not liveness
✓ a healthy silent-but-working run is NOT killed
✓ ...and the deferral is announced, not silent
✓ positive control: the same run IS killed with --no-liveness-probe

That positive control is doing real work: without it, a probe that did nothing at all would score identically on the other six.

Also confirmed against the diff:

• --stall-minutes is untouched. Every changed line mentioning it is usage text, a comment, or the new validation that --stall-cap-minutes must be greater than it. The maintainer's floor held.
• The probe can only DELAY.probeDeferrals++; return; is the only early exit; the cap falls through to the kill under a distinct ⛔ STALL-CAP verdict, and a hang that keeps writing is still killed ...at the cap pins it.
• ⭐ Unavailable never reads as alive.probeEverRead stays false when /proc/<pid>/io cannot be read, and the probe then "can never postpone a kill". This is the inversion hazard in its most dangerous disguise — a probe that fails open on a machine where the signal is unreadable — and you closed it explicitly rather than by accident.
• ⭐ The first-poll baseline. Without it every pid's lifetime total would book as "progress just now" — "a stall's worth of free credit handed to a suite that is already frozen." That trap is not in the card; you found it.
• Positive-deltas-only, so an exiting process reads as "neither a stall nor liveness."

M2 (forcing capped false) reproducing the card's inversion on demand is the right way to prove the cap is load-bearing rather than decorative.

The definition you were asked to state is stated, and it is the correct one: frozen means produced nothing, not did nothing, and explicitly not "no CPU."

Spin-off #11916 noted — nothing checks a guard's kill budget against its own job's timeout-minutes, so a guard that cannot speak before its job dies is a silent no-op that scores identically to a working one on every green run. Correctly filed rather than folded in.

---

Generated by Claude Code