feat(scripts): check-stall-guard-budget — the guard's kill budget must clear its job's timeout-minutes by a stall window

[os-zhuang]

Fixes#11916

run-with-stall-guard.mjs exists so a stalled job says it stalled instead of sitting in_progress until the job timeout. That guarantee holds only while the guard's own kill budget lands before its job's timeout-minutes — and nothing read that relationship. Measured on f907fbe9e: git grep -l 'timeout-minutes' scripts/ returns exactly one file, run-with-stall-guard.mjs itself, and both of its hits are prose in the header explaining this very relationship. So it was documented at the one place that could not enforce it. (The card said "no script reads it at all"; that is true of the code and false of grep -l, which returns 1.)

The regression is invisible on green runs: the guard is observable only when it fires, so a budget that can no longer fire in time scores identically to one that works.

What landed

scripts/check-stall-guard-budget.mjs walks .github/workflows/**, finds every step invoking the guard, resolves the effective cap, and reds when it is not comfortably below the binding timeout-minutes. Wired as pnpm check:stall-guard-budget and run in lint.yml next to the guard's own self-test.

Measured on this branch: 28 workflow files, 51 jobs, 534 steps, 7 guard-wrapped steps, 0 violations — ci.yml x4 and coverage-nightly x1 at W=10 C=20 T=30 (slack 10, exactly one window), rerun-safety-nightly x2 at W=15 C=30 T=120 (slack 90).

The defaults are read from the guard, never copied

The window default and DEFAULT_CAP_MULTIPLE are extracted from run-with-stall-guard.mjs (comments masked, because its header quotes both in prose). A second copy of 2 and 10 in the gate would be two numbers that must agree with nobody holding them to it — the very shape this gate closes, one level up. Rename either declaration and the gate refuses instead of judging against a stale default; a self-test case drives that.

The criterion, and where its constant comes from

With W = window, C = effective cap, T = binding timeout-minutes, p = prep before the step, s = how far into the step the output froze — the guard's clock starts at the last output (silentMs = Date.now() - lastOutputAt), so the verdict lands at p + s + C.

• Tier 1 — C < T. No constant at all: at C >= T the guard cannot deliver a verdict in any scenario, not even p = s = 0 (a step that never emits a first line — a shape the guard's own self-test pins). A guaranteed silent no-op.
• Tier 2 — T - C >= W.T - C is the entire room for p + s. Requiring at least one full stall window of it says the guard must still be able to speak for a freeze that happens after the step has done some work — the mid-suite freeze it exists for.

The unit of tier 2 is the step's own declared window, not a fraction someone picked. A taste-chosen ratio becomes a constant nobody can explain, and the point of this card is to make the invariant explainable. All three one-line breakages the card named are caught, and tier 1 alone would miss the first two:

edit	tier 1	tier 2
job timeout-minutes 30 → 25	green (20 < 25)	red
--stall-minutes 10 → 12	green (24 < 30)	red
guarded step added to a 15-minute job	red	red

What the measurement said, including a premise that did not survive

The card reasoned the real budget is timeout-minutes minus checkout/install/build, "a fair chunk of a 30-minute job". That half is false, measured on GitHub's own runner timestamps for CI run 33135187774 (head f907fbe9e, the six test shards) — job start to guarded-step start, then the step:

shard	prep	guarded step (healthy)
3/6	33s	2s (turbo cache hit)
1/6	35s	14m35s
6/6	38s	6m07s
2/6	56s	4m54s
4/6	44s	2s
5/6	2m03s	2s

Prep is 33s–2m03s, at most 7% of the budget. The term that consumes it is the healthy run, up to 14m35s of the 30 — and that is exactly what no static gate can read. So the gate does not model it and encodes no number derived from it; it refuses the structural class instead.

The same measurement bounds the gate's promise, and the bound is in its header: on the ci.yml family T - C is 10 minutes while the worst shard spent 15m10s before it would have frozen, so a freeze at the very end of a healthy shard, on the deferred path only, still loses to the job timeout. The undeferred path covers it with 4.8m spare. That is a property of the checked-in values, not of this gate, and it is filed separately as #12846 — not addressed here.

The gate says how to fix a red, and how not to

Per the card's ⛔, in the failure message itself: lower the guard budget (--stall-minutes, or an explicit --stall-cap-minutes) or raise the job's timeout-minutes. Deleting the guard from the step is not a remedy — it removes the only instrument that can report a stall and converts a red gate into precisely the unlabeled in_progress-until-timeout outcome the guard abolishes.

Non-vacuity

"Every guard-wrapped step has headroom" is true of a tree with no guard-wrapped steps, and equally true of a broken selector that finds none — the success criterion and the total-failure criterion would be word for word identical. So run()refuses (exit 2, never 0) on a missing workflow directory, no workflow file, no jobs, no guard invocation at all, unreadable guard defaults, or a --stall-minutes / timeout-minutes that is not a positive number. The verdict line prints the population, and the self-test pins the sweep's hit count above zero on the fixture whose green it also asserts.

Verification

• node scripts/check-stall-guard-budget.mjs --self-test → 40 assertions over real fixture trees on disk, driving the real run(): both violation tiers red, the comment/self-test/post--- selector negatives, the guard defaults proven read (a fixture declaring multiple 3 turns the same workflow red, and the fixture's 99-in-a-comment is masked), every refusal path, and the real repository green and non-vacuous.
• Ablation on the real tree (mutation and restore both proven by state, never by exit code) at 7683f3a1f: ci.yml's test job timeout-minutes: 30 → 25 — anchor counts moved both ways (30 x5→x4, 25 x0→x1) and the blob changed (c4d019d34 → cfaed7e7d) before any reading was taken. Gate went exit 1 naming ci.yml:573 on the headroom reason. Restore proved by state: blob back to c4d019d34 = the HEAD blob, git diff HEAD empty; gate green again. No build or dist/ is involved — the gate reads workflow text.
• Gate union re-run after the final commit, at 7683f3a1f: the 27 families node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack derives for this diff, plus the two convention-triggered ones (bare-root-worklist --self-test, check:pm-dispatch-gates) and check:nul-bytes — all green. Repo-wide pnpm lint also green (59s under the shared verify lock), so no narrowing was needed.
• check:type-check-debt is NOT MEASURED: its --re-measure half refuses on an unbuilt closure (56 workspace dependencies ... have no built type entry point on disk) — a refusal to measure, not a finding. Its declaration half (check:type-check-coverage, same script) is green, and the diff cannot move the ledger: no .ts file, no tsconfig, no package typecheck script, and the root tsconfig.json sets no allowJs, so a .mjs file is in no tsc program. CI runs it after its build step regardless.

Bare-root verdict

Recorded per the #12074 obligation, measured in both directions rather than assumed: node scripts/pm/bare-root-worklist.mjs --self-test is byte-identical before and after this change — 46 live rows, 39 recorded verdicts, none stale, missing or contradicted. The new gate's population constants (WORKFLOW_DIR = '.github/workflows', GUARD_SCRIPT = 'scripts/run-with-stall-guard.mjs') both carry a separator, so extractWatchHints already sees them — confirmed by the derivation naming this card via gate source '.github/workflows' and gate script. No ledger row is owed and none was added; the map stays shrink-only and untouched.

Landing site: a new script, not an extension of check:agent-test-spelling

The card floated extending that gate, since it already parses these command lines. Measured against its source: it is a text scan with no job or step model at all — it walks source files for bare -- separators and has no notion of a job, a step, or timeout-minutes. Hosting this invariant there means building a workflow job/step model inside a gate whose header explicitly refuses to grow a second matching rule. A new script reuses shellCommands / commandWords from check-shard-attestation.mjs (the existing import edge check-required-contexts already uses) and costs one lint.yml step — cheaper and clearer.

No changeset

skip-changeset: root scripts/, package.json scripts wiring and .github/workflows/ only — nothing published changes.

---

Generated by Claude Code

[os-zhuang]

PM 复核:ACCEPT — #11916 / check-stall-guard-budget

复核对象 head 7683f3a1f,base 6f0fec3d0,3 文件 +768/−1,skip-changeset。
下面每一条都标注了是我自己测的还是转述报告——转述的不当证据用。

一、我自己复算的(不是转述)

⚠️ 空集合陷阱(我派发时点名的头号风险:一个找不到任何 guard-wrapped step 的 sweep,会完美满足「每个 step 预算都低于 job timeout」)——闭合,由我独立测出:

在 head 上普查 .github/workflows/**,run-with-stall-guard 共 13 处文本命中;逐行分解:

• 6 处不是调用 — ci.yml:529 / :938 / :1133 注释、lint.yml:3243 注释、rerun-safety-nightly.yml:45 注释、:135 一句 echo "::error::…" 文案;
• 7 处是真调用 — ci.yml:573 / :969 / :1009 / :1152、coverage-nightly.yml:59、rerun-safety-nightly.yml:99 / :130。

与 gate 报的 7 guard-wrapped step(s)逐个对上,也与报告的分档普查对上(W=10 的 5 处 = ci.yml×4 + coverage-nightly;W=15 的 2 处 = rerun-safety)。lint.yml 里 guard 自己的自测步骤不含路径字面量,不会被误计;self-test 另有两条钉住这一点(「注释形不是 site」「--self-test 包不住任何东西,不是 site」)。⇒ 命中数 > 0 是测出来的,不是收下来的。

空集合行为(读源码,非转述):scripts/check-stall-guard-budget.mjs:470-479 — sites.length === 0 ⇒ REFUSING to report a verdict,返回 EXIT_REFUSED(:149 = 2),文案写明「a green here would be indistinguishable from a broken selector」。永不静默绿。

⛔ 失败处方(我派发时的硬要求:红了修预算或 job timeout,永不删 guard)——PRESCRIPTION 在 :416-429,原文进了正文:两条补救都是改预算(降 --stall-minutes、显式 --stall-cap-minutes,或抬 job timeout-minutes),并明写 Deleting run-with-stall-guard.mjs from the step is NOT a remedy,理由是它「removes the only instrument that can report a stall at all」。要求满足。

⭐ bare-root verdict(#12074,triage 指定):我没有采信报告,而是开了两棵独立 detached worktree(base 6f0fec3d0 / head 7683f3a1f)各跑一次 scripts/pm/bare-root-worklist.mjs --self-test,diff逐字节相同:

OK self-test: 46 live row(s), 39 unreachable as spelled, 39 recorded verdict(s) — none stale, none missing, none contradicted.

⇒ 新 gate 的 repoRoot()不欠新账,报告属实。

人工地板:未触发。package.json 的 diff 只有一行脚本接线(check:stall-guard-budget);yaml ^2.9.0 在 base 的 package.json:162 就已是 root devDependency(与 typescript 同一块,两边同行号)。本 PR 没有新增任何运行时依赖。

二、更正我自己派发前的一处判断(公开纠正,别让它进 lane 记忆)

我此前认定「这是第一个需要 node_modules 的 scripts/ gate」——这是错的,错在我的量具:我用的正则锚在行首,漏掉了 () => import('yaml') 这种行中动态 import。去掉锚重测:

• base 上 scripts/ 里 import yaml 的文件已有 13 个,本 gate 是第 14 个;
• requireDependency(scripts/import-prerequisite.mjs:366,既有共享 helper)在 base 上已有 17 个使用者,本 gate 是第 18 个。

⇒ 它沿用既有惯例,不是开新口子。我原来的说法反了。

三、无法测量(⛔ 这不是缺陷)

我这个席位跑不动 gate 本体:/home/user/objectstack/node_modules 是空的(0 项),dev 的 worktree 已拆,scratch worktree 没有 install。我确实碰出了它的 PREREQUISITE 分支两次(一次误撞、一次特意复现),它按设计明说 Nothing was measured … It is NOT a finding,并给出 Fix: pnpm install,还顺手警告了管道吞掉 $? 的坑——这个分支的行为我验过,是对的。gate 挂在 lint job(install 之后),权威读数是 CI 的 Lint & Repo Gates,我不拿本地无法运行当结论。

四、认可的判断(转述+抽查)

• Zone 2 被证伪,而且证伪得比我预期硬:没用容器 wall-clock(那是争用读数,[finding] os-verify-lock serialises locked heavy jobs but NOT gate scripts — so every timing card measures a contended box while believing the lock gave it an idle one #12528),改用 GitHub runner 时间戳(run 33135187774 @ f907fbe9e)。prep 实测 33s–2m03s = 至多预算的 7%,card 说的「a fair chunk of a 30-minute job」不成立;真正吃预算的是健康运行本身(最长 14m35s)——而这恰是静态 sweep 唯一读不到的项,所以 gate 没有编进任何由它派生的常数,判据只落在 W/C/T 三个静态可读量上。这正是 Zone 2 该有的用法:机制假设被实测推翻,路线随之改,裁决不动。
• 阈值单位是该 step 自己声明的 window,不是谁挑的比例。比例阈值(20/30 = 0.67)今天就会把 card 明说「值是对的」的 ci.yml 判红。
• Tier 1 单独会漏掉 card 点名的三处一行破坏中的两处(timeout 30→25、--stall-minutes 10→12),Tier 2 两处都抓 ⇒ 第二档不是装饰。
• typeof === 'number' 而非 Number(x):Number(null) === 0 会凭空造出一个关于「零分钟预算」的自信判词。任务中途自己发现并修,两个 self-test case 钉住。
• 23/51 个 job 未声明 timeout-minutes ⇒ 解析为 GitHub 文档的 360 分钟默认并写进普查,而不是判红。不变式对它确实成立,写进普查让这个零读起来是测量。
• guard 默认值是读出来的,不是抄过来的(注释屏蔽);fixture 里两行注释形假声明(// const DEFAULT_CAP_MULTIPLE = 99;)就是这条的反证。改名 ⇒ REFUSE,不拿陈旧默认值下判。
• 落点选择有实测支撑:check-agent-test-spelling 是文本扫描、无 job/step 模型,寄生进去等于在一个明确拒绝长第二条规则的 gate 里再建工作流模型。新脚本复用 check-shard-attestation.mjs 的 shellCommands/commandWords(check-required-contexts 已在用的同一条 import 边),更便宜也更清楚。

五、记账(不影响本卡验收)

① 偏差,已自报未隐瞒 —— 判定:接受,不返工。 三个 commit 带 Part of #11916 trailer,与 squash 分支规则不符。发现时首个 commit 已推,改写需 force-push(更硬的禁令)。三个 trailer 一致为 Part of,squash 产不出规则要防的 Fixes + Part-of 矛盾形;闭环关系在 PR body 里声明了一次;CI 的 Part-of PR must not also close its card已绿。补救成本高于收益。下一班注意:trailer 要在第一个 commit 落地之前定好,推完再想就只剩更差的选项。

② #12846 已归档(finding · tooling · domain:devx,无 assignee):deferred 路径上,健康 shard 末尾的冻结仍会输给 job timeout(实测 15m10s prep+run 对 10m 的 cap headroom;undeferred 路径覆盖得住,还余 4.8m)。正确地没在本卡修——补救是工作流预算决策,gate 判不了,因为运行时长正是静态 sweep 读不到的那一项。它现在是未定级卡,走 triage,本席位不自我提级。

③ 标签自纠:创建 finding 时标签落成了 tooling/pm:queue/domain:devx(finding 缺席、pm:queue 在场,与指令相反)。dev 判定为自己那次写入的结果(同一共享身份、创建时刻、无第三方),加减式更正并读回,现为 tooling/domain:devx/finding。处理方式对:先判「是不是别人的状态」,再动手,且读回。

---

结论:ACCEPT。 待 32 项检查全绿(逐名读完 get_check_runs 每一页)后转正式并挂自动合并;Lint & Repo Gates 是本卡的关键读数。

---

Generated by Claude Code