Observed while fixing #10807. Nothing is wrong with the three self-tests themselves — this is about what the step reports when one of them fails.
The shape
.github/workflows/lint.yml's Shallow-history guard self-tests step runs three self-tests as one run: block:
- name: Shallow-history guard self-testsrun: | node scripts/pm/git-history.mjs --self-test node scripts/check-engine-split-ratio.mjs --self-test bash scripts/collect-release-notes.sh --self-test
GitHub runs a run: block under bash -e, so the first non-zero exit aborts the step and the remaining lines never execute. The step then reports a single failure, and says nothing about the two commands that were never reached.
Measured, and it is not hypothetical — it happened today
git-history.mjs --self-test was red on main from ~12:00Z until #10807 landed (a bare --since date approxidated to the current wall-clock time). It is first in that block. So for the whole of that window:
check-engine-split-ratio.mjs --self-test — never ran in CIcollect-release-notes.sh --self-test — never ran in CI
Both are green. Verified by hand at 12:12:31Z and 12:12:45Z respectively, exit 0 both. They were masked, not failing — which is the point: nothing in the step's output distinguishes "this passed" from "this was never reached", and the absence is invisible precisely when an earlier guard is broken, i.e. exactly when you would most want the others to have run.
The blast radius was every PR in the repo for the duration, because that step gates Lint & Repo Gates.
Why this is worth a card rather than a shrug
⚠️ This is the repo's standing defect class, one level up: a partial result that reads like a complete one. The other three self-tests in that step each refuse rather than report a clean sweep they cannot support — git-history.mjs in particular exists to refuse a windowed answer a shallow clone cannot see all of, and prints an empty stdout rather than a plausible number "because zero is a broken scan, not a clean repo (#4690)". The step that runs them does not hold itself to the same rule.
It also has a compounding effect worth naming: while the first guard is red, a second regression in either of the other two can land unnoticed, and will only surface once the first is fixed — at which point it looks like the fix broke something.
Direction, not a prescription
The obvious shape is tolerate-and-collect: run all three, remember the exits, fail the step at the end naming every one that failed. That is a small change to one run: block.
⚠️ Two things to weigh, which is why this is a card and not a rider:
- It is one step, not three, for a reason — three steps would be three lines of YAML and clearer output, but
check-required-contexts pins required context names and check-shard-attestation classifies jobs by their steps' run: text, so splitting a step is not always free in this repo. Whoever takes it should check both before choosing between "collect within one step" and "split into three". - Same question, wider surface: whether other multi-command
run: blocks in lint.yml and ci.yml mask later commands the same way. This one was found by accident. A census would say whether it is a one-off or a pattern — and that census is the more valuable half of this card.
⛔ Not proposed: changing any of the three self-tests. They are correct.
Refs: #10807 (the failure that exposed it) · #4690 (zero is a broken scan, not a clean repo) — filed unassigned by the domain:devx PM seat.
Generated by Claude Code
Observed while fixing #10807. Nothing is wrong with the three self-tests themselves — this is about what the step reports when one of them fails.
The shape
.github/workflows/lint.yml's Shallow-history guard self-tests step runs three self-tests as onerun:block:GitHub runs a
run:block underbash -e, so the first non-zero exit aborts the step and the remaining lines never execute. The step then reports a single failure, and says nothing about the two commands that were never reached.Measured, and it is not hypothetical — it happened today
git-history.mjs --self-testwas red onmainfrom ~12:00Z until #10807 landed (a bare--sincedate approxidated to the current wall-clock time). It is first in that block. So for the whole of that window:check-engine-split-ratio.mjs --self-test— never ran in CIcollect-release-notes.sh --self-test— never ran in CIBoth are green. Verified by hand at 12:12:31Z and 12:12:45Z respectively, exit 0 both. They were masked, not failing — which is the point: nothing in the step's output distinguishes "this passed" from "this was never reached", and the absence is invisible precisely when an earlier guard is broken, i.e. exactly when you would most want the others to have run.
The blast radius was every PR in the repo for the duration, because that step gates
Lint & Repo Gates.Why this is worth a card rather than a shrug
git-history.mjsin particular exists to refuse a windowed answer a shallow clone cannot see all of, and prints an empty stdout rather than a plausible number "because zero is a broken scan, not a clean repo (#4690)". The step that runs them does not hold itself to the same rule.It also has a compounding effect worth naming: while the first guard is red, a second regression in either of the other two can land unnoticed, and will only surface once the first is fixed — at which point it looks like the fix broke something.
Direction, not a prescription
The obvious shape is tolerate-and-collect: run all three, remember the exits, fail the step at the end naming every one that failed. That is a small change to one
run:block.check-required-contextspins required context names andcheck-shard-attestationclassifies jobs by their steps'run:text, so splitting a step is not always free in this repo. Whoever takes it should check both before choosing between "collect within one step" and "split into three".run:blocks inlint.ymlandci.ymlmask later commands the same way. This one was found by accident. A census would say whether it is a one-off or a pattern — and that census is the more valuable half of this card.⛔ Not proposed: changing any of the three self-tests. They are correct.
Refs: #10807 (the failure that exposed it) · #4690 (zero is a broken scan, not a clean repo) — filed unassigned by the
domain:devxPM seat.Generated by Claude Code