Skip to content

stall guard: nothing checks a guard-wrapped step's kill budget against its own job's timeout-minutes — a guard that can't speak before the job dies is a silent no-op #11916

Description

@os-steve

Found while implementing #11855 (not addressed there — that card is the probe's design, this is a CI-wiring invariant and needs its own gate).

The gap

run-with-stall-guard.mjs exists so a stalled job says it stalled instead of sitting in_progress until the job timeout. That guarantee only holds while the guard's own kill budget lands before its job's timeout-minutes. Nothing checks that relationship — not check:agent-test-spelling, which already parses these command lines, and not check-ci-filter-parity. Verified: no script under scripts/ reads timeout-minutes at all.

If the budget ever exceeds the job budget, the job timeout wins, the guard never prints a verdict, and the outcome is exactly the state the guard was built to abolish. Worse, the regression is invisible on green runs — the guard is only observable when it fires, so a guard that can no longer fire in time scores identically to one that works. That is the same "no green run can distinguish the regression from success" shape #11855 had to design around.

Current values (measured on origin/main @ fd50e59e7)

workflowstep--stall-minutesjob timeout-minutes
ci.ymlTest Core (test)1030
ci.ymltemporal-conformance ×21030
ci.ymldogfood1030
coverage-nightly.ymlcoverage1030
rerun-safety-nightly.ymlrerun-safety ×215120

All of these are currently fine, and #11855's new --stall-cap-minutes (default 2 × the window) keeps every derived cap strictly below its job budget too — 20 < 30 and 30 < 120. The values are right; the invariant is just unenforced. It is a one-line edit in either direction to break it, and nothing would notice:

  • lowering a job's timeout-minutes (a plausible "tighten CI" change),
  • raising --stall-minutes or passing an explicit --stall-cap-minutes above the job budget,
  • adding a new guard-wrapped step in a job with a short timeout.

Note the budget that matters is the cap, not the window, now that a deferral can extend a run — and it is really timeout-minutes minus checkout/install/build, which is a fair chunk of a 30-minute job. A gate should probably assert headroom, not just strict inequality.

Suggested shape

A scripts/check-stall-guard-budget.mjs that walks .github/workflows/**, finds every step invoking run-with-stall-guard.mjs, resolves the effective cap (explicit --stall-cap-minutes, else DEFAULT_CAP_MULTIPLE × the window, else the guard's built-in default), and reds when it is not comfortably below the enclosing job's timeout-minutes. It should carry a --self-test that drives itself red over a synthetic workflow on disk, per the repo's non-vacuity convention.

⛔ Worth stating in the gate itself: the fix for a red here is to lower the guard budget or raise the job timeout — never to delete the guard from the step.

Refs: #11855 (the probe and the cap) · #11808 (the log-order precondition) · #4250 (the real-stall family) · scripts/run-with-stall-guard.mjs header.


Generated by Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions