docs(bugbot): encode the three shapes of a vacuous test (backend#2265)

[LukasWodka]

Fixes tracebloc/backend#2265

Why

Three PRs merged on 2026-08-20 — #762, #763 and tracebloc/release-train#94 — shipped eleven tests that asserted the right property and proved nothing. Every one was written deliberately, reviewed, and green. That is the most frequent finding across those PRs by a wide margin — more than every production defect in them combined — and .cursor/BUGBOT.md said nothing about it.

The repo convention is that a finding recurring across PRs becomes a rule here.

What's added

Three shapes, ascending in subtlety — only the first is widely understood:

1. Unreachable fixture — the input never takes the path. A one-liner test that indents the function when the opener rule is anchored at column 0.
2. Redundant mechanisms — two code paths give the same observable for that input. A fixture with one Dockerfile cannot distinguish "the fallback demanded everything" from "the comment attributed that one file": both count 1.
3. Inert mutation — the mutation fails to express the defect. For a root Dockerfile, rel and the basename are the same string, so mutating one of two redundant arms leaves the other matching.

Shape 3 is the one worth internalising: an anchor-resolution check cannot detect it. The anchor resolves perfectly and the log is indistinguishable from real coverage. Only a surviving mutation reveals it.

Plus:

• a surviving mutation is a defect in the test, never a nuisance to annotate — and a green mutation log is evidence only if the run asserts the mutation applied
• a derived vocabulary must fail closed (the _brand_rgbs case: deleting rule 1's whole RGB arm left its test green, because the extractor fell through to the hex list)
• the early-close hazard now has a CI gate, with diagnose.sh as the worked example of an instance that is correct as a pipe
• a corollary on the scripts/lib/*.sh non-issue: they set no options but they run under both, so errexit/pipefail rules apply in full
• never resolve a thread on "the reported case now passes"

Relationship to backend#1729

#1729 already requires mutation-proving, deriving the input domain, and never testing a copy of the rule. All three were followed in these PRs and the tests were still vacuous eleven times. The existing rules say to mutation-test; they don't say what a surviving mutation means, or that a fixture can be too thin for the property to be observable. That's the gap.

Verification

Docs-only, but every factual claim was checked against the tree — which caught two errors in my own draft before commit:

• I wrote that diagnose.sh carries a # pipefail-guard: allow marker. It doesn't — I removed it once the guard learned to read set +e. Reworded to describe what the file actually does.
• I referenced make mutation-markers, which is a release-train target, not a client one. Replaced with a repo-neutral phrasing.

Writing a rule about prose that contradicts code, and nearly shipping exactly that, is not lost on me.

check-style.sh, gen-manifest --check, check-drift.sh and the early-close gate all clean. No customer or internal detail (public repo).

🤖 Generated with Claude Code

---

Note

Low Risk
Docs-only change to the Bugbot review guide; no runtime, security, or installer behavior is modified.

Overview
Updates .cursor/BUGBOT.md so reviewers treat vacuous guard tests as the default suspicion: fixtures that never hit the named path, inputs that cannot distinguish two mechanisms, and mutations that do not actually express the defect.

Also tells Bugbot to flag surviving mutations as test defects (not nuisances), derived vocabularies that fail open, and unexplained # pipefail-guard: allow markers. Clarifies that sourced scripts/lib/*.sh still run under errexit/pipefail, and that a finding must not be resolved just because the reported case now passes.

^{Reviewed by Cursor Bugbot for commit f7d2196. Bugbot is set up for automated code reviews on this repo. Configure here.}

[aptracebloc]

Reviewed. Docs-only addition to .cursor/BUGBOT.md encoding the three shapes of a vacuous guard test, plus the surviving-mutation / fail-closed-vocabulary / early-close corollaries.

Spot-checked every concrete claim against the tree and they hold: scripts/lib/diagnose.sh keeps df -h | head -20 under run_diagnose's set +e (line 96 / 42), # pipefail-guard: allow has no live usages, and scripts/tests/pipefail-early-close.sh + _brand_rgbs in check-style.bats both exist on develop with the fail-closed behavior the doc describes. Accurate and internally consistent.

— drafted with Claude Code