Found while implementing #11855 (not addressed there — that card is the probe's design, this is a CI-wiring invariant and needs its own gate).
The gap
run-with-stall-guard.mjs exists so a stalled job says it stalled instead of sitting in_progress until the job timeout. That guarantee only holds while the guard's own kill budget lands before its job's timeout-minutes. Nothing checks that relationship — not check:agent-test-spelling, which already parses these command lines, and not check-ci-filter-parity. Verified: no script under scripts/ reads timeout-minutes at all.
If the budget ever exceeds the job budget, the job timeout wins, the guard never prints a verdict, and the outcome is exactly the state the guard was built to abolish. Worse, the regression is invisible on green runs — the guard is only observable when it fires, so a guard that can no longer fire in time scores identically to one that works. That is the same "no green run can distinguish the regression from success" shape #11855 had to design around.
Current values (measured on origin/main @ fd50e59e7)
| workflow | step | --stall-minutes | job timeout-minutes |
|---|
ci.yml | Test Core (test) | 10 | 30 |
ci.yml | temporal-conformance ×2 | 10 | 30 |
ci.yml | dogfood | 10 | 30 |
coverage-nightly.yml | coverage | 10 | 30 |
rerun-safety-nightly.yml | rerun-safety ×2 | 15 | 120 |
All of these are currently fine, and #11855's new --stall-cap-minutes (default 2 × the window) keeps every derived cap strictly below its job budget too — 20 < 30 and 30 < 120. The values are right; the invariant is just unenforced. It is a one-line edit in either direction to break it, and nothing would notice:
- lowering a job's
timeout-minutes (a plausible "tighten CI" change), - raising
--stall-minutes or passing an explicit --stall-cap-minutes above the job budget, - adding a new guard-wrapped step in a job with a short timeout.
Note the budget that matters is the cap, not the window, now that a deferral can extend a run — and it is really timeout-minutes minus checkout/install/build, which is a fair chunk of a 30-minute job. A gate should probably assert headroom, not just strict inequality.
Suggested shape
A scripts/check-stall-guard-budget.mjs that walks .github/workflows/**, finds every step invoking run-with-stall-guard.mjs, resolves the effective cap (explicit --stall-cap-minutes, else DEFAULT_CAP_MULTIPLE × the window, else the guard's built-in default), and reds when it is not comfortably below the enclosing job's timeout-minutes. It should carry a --self-test that drives itself red over a synthetic workflow on disk, per the repo's non-vacuity convention.
⛔ Worth stating in the gate itself: the fix for a red here is to lower the guard budget or raise the job timeout — never to delete the guard from the step.
Refs: #11855 (the probe and the cap) · #11808 (the log-order precondition) · #4250 (the real-stall family) · scripts/run-with-stall-guard.mjs header.
Generated by Claude Code
Found while implementing #11855 (not addressed there — that card is the probe's design, this is a CI-wiring invariant and needs its own gate).
The gap
run-with-stall-guard.mjsexists so a stalled job says it stalled instead of sittingin_progressuntil the job timeout. That guarantee only holds while the guard's own kill budget lands before its job'stimeout-minutes. Nothing checks that relationship — notcheck:agent-test-spelling, which already parses these command lines, and notcheck-ci-filter-parity. Verified: no script underscripts/readstimeout-minutesat all.If the budget ever exceeds the job budget, the job timeout wins, the guard never prints a verdict, and the outcome is exactly the state the guard was built to abolish. Worse, the regression is invisible on green runs — the guard is only observable when it fires, so a guard that can no longer fire in time scores identically to one that works. That is the same "no green run can distinguish the regression from success" shape #11855 had to design around.
Current values (measured on
origin/main@fd50e59e7)--stall-minutestimeout-minutesci.ymltest)ci.ymltemporal-conformance×2ci.ymldogfoodcoverage-nightly.ymlcoveragererun-safety-nightly.ymlrerun-safety×2All of these are currently fine, and #11855's new
--stall-cap-minutes(default2 ×the window) keeps every derived cap strictly below its job budget too — 20 < 30 and 30 < 120. The values are right; the invariant is just unenforced. It is a one-line edit in either direction to break it, and nothing would notice:timeout-minutes(a plausible "tighten CI" change),--stall-minutesor passing an explicit--stall-cap-minutesabove the job budget,Note the budget that matters is the cap, not the window, now that a deferral can extend a run — and it is really
timeout-minutesminus checkout/install/build, which is a fair chunk of a 30-minute job. A gate should probably assert headroom, not just strict inequality.Suggested shape
A
scripts/check-stall-guard-budget.mjsthat walks.github/workflows/**, finds every step invokingrun-with-stall-guard.mjs, resolves the effective cap (explicit--stall-cap-minutes, elseDEFAULT_CAP_MULTIPLE ×the window, else the guard's built-in default), and reds when it is not comfortably below the enclosing job'stimeout-minutes. It should carry a--self-testthat drives itself red over a synthetic workflow on disk, per the repo's non-vacuity convention.⛔ Worth stating in the gate itself: the fix for a red here is to lower the guard budget or raise the job timeout — never to delete the guard from the step.
Refs: #11855 (the probe and the cap) · #11808 (the log-order precondition) · #4250 (the real-stall family) ·
scripts/run-with-stall-guard.mjsheader.Generated by Claude Code