feat(governance): exemption ledgers may shrink, never grow silently - #588
Conversation
Every large CI failure diagnosed in this estate shares one shape: an exemption was added quietly and then never removed. The 2026-08-06 audit across 424 repositories found a gitleaks gate that had run with NO allowlist for months; 13,206 banned-language files, most declared once and forgotten; a .hypatia-baseline.json that absorbed 255 findings in a single commit, turning a library's central guarantee into accepted debt; and exemption files written in the WRONG FORMAT, which therefore suppressed nothing — unnoticed for months, because a suppression that does not work is silent in exactly the same way as one that does. A gate that checks only the CURRENT state cannot catch any of that: each individual state is "valid". What matters is the direction of travel. This compares a pull request against its base and enforces three things: 1. NO SILENT GROWTH. A ledger may lose entries freely. Gaining them requires `Ratchet-exception: <why>` in a commit message — so adding debt stays possible, attributable and reviewed. A gate nobody can satisfy gets deleted; a gate that costs one honest sentence gets obeyed. 2. NO ANONYMOUS ENTRIES. Every .hypatia-baseline.json entry must carry a `note` or a `tracking_issue`. "What is this?" must be answerable without archaeology. This is the rule that would have caught the 255-entry commit. 3. NO WILDCARDS IN THE MIGRATION LEDGER. A `**` in .hypatia-ignore absorbs files added later, so the ledger grows while appearing to hold steady — precisely the failure this check exists to prevent. Architectural exemptions belong in .hypatia-baseline.json, where a note explains them. Covers .hypatia-baseline.json, .hypatia-ignore, .gitleaks.toml and .machine_readable/root-allow.txt. ⚠ THE RATCHET IS ITSELF TESTED, and the test asserts it CAN FAIL. This estate's recurring defect is the gate that cannot fail — `just proof-check` recipes exiting 0 with no prover installed, `continue-on-error` on the primary secret scanner, a linter that never parsed its input. A ratchet that always passed would be worse than none, because it would license the silent growth it claims to prevent. scripts/tests/exemption-ratchet-test.sh exercises all six branches in both directions, including the subtle one: an anonymous entry fails even when growth IS declared. Runs on pull_request only — the check needs a base to compare against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
| *) | ||
| # comments and blank lines are not exemptions | ||
| printf '%s' "$blob" | grep -vE '^\s*(#|$)' | wc -l | tr -d ' ' ;; | ||
| esac |
There was a problem hiding this comment.
⚠️ Bug: Comments-only text ledger aborts ratchet via set -e
In count_at, the text branch grep -vE '^\s*(#|$)' | wc -l returns exit 1 when a non-empty ledger contains only comment/blank lines (grep -v matches nothing). Combined with set -euo pipefail, the failing pipe propagates out of the command substitution after="$(count_at ...)" and aborts the whole script with exit 1 mid-loop — before the anonymous-baseline and wildcard checks run, and with no diagnostic. This turns the intended 'debt fully paid down, header comment retained' case (which should SHRINK/pass) into an opaque failure, contradicting the tool's purpose. Verified reproducible on Linux/GNU grep. Fix by neutralizing grep's no-match exit, e.g. printf '%s' "$blob" | { grep -vE '^\s*(#|$)' || true; } | wc -l | tr -d ' '.
Prevent grep's no-match exit (1) from aborting the script under set -e/pipefail.:
*)
# comments and blank lines are not exemptions
printf '%s' "$blob" | { grep -vE '^\s*(#|$)' || true; } | wc -l | tr -d ' ' ;;
Was this helpful? React with 👍 / 👎
| # Does the pull request explicitly declare that a ledger must grow? | ||
| EXCEPTION=0 | ||
| if git log --format=%B "${BASE_REF}..HEAD" 2>/dev/null | grep -qiE '^Ratchet-exception:[[:space:]]*\S'; then | ||
| EXCEPTION=1 | ||
| fi |
There was a problem hiding this comment.
💡 Bug: One Ratchet-exception licenses growth in all four ledgers
EXCEPTION is a single global flag set when any commit in base..HEAD contains a Ratchet-exception: line, and it then permits unlimited growth across every ledger in the loop. A PR that legitimately declares growth for one ledger (e.g. a vendored corpus in .hypatia-baseline.json) silently also permits unrelated growth in .gitleaks.toml or root-allow.txt within the same PR. Consider scoping the declaration to a named ledger (e.g. Ratchet-exception: <path>: <why>) so the escape hatch stays targeted.
Was this helpful? React with 👍 / 👎
| case "$path" in | ||
| *.json) | ||
| printf '%s' "$blob" | jq 'if type=="array" then length else 0 end' 2>/dev/null || echo 0 ;; | ||
| *) | ||
| # comments and blank lines are not exemptions | ||
| printf '%s' "$blob" | grep -vE '^\s*(#|$)' | wc -l | tr -d ' ' ;; | ||
| esac |
There was a problem hiding this comment.
💡 Bug: Line-count proxy is unreliable for .gitleaks.toml
count_at treats every non-comment, non-blank line as one exemption for text ledgers, but .gitleaks.toml is structured TOML: allowlist entries span multiple lines (tables, regexes = [ ... ] arrays), and headers/keys count too. Reformatting an existing allowlist can inflate the count (false GREW) while genuinely adding a suppression inside an existing array line may not change it (silent growth). The ratchet's guarantee is therefore weak for the gitleaks ledger specifically; consider parsing it as TOML or counting allowlist array elements rather than lines.
Was this helpful? React with 👍 / 👎
Note Automatic reviews are paused because your trial's included automatic processing has been used for this period. Upgrade now, or comment "Gitar review" to run a review anytime. Code Review |
| Compact |
|
Important
Your trial ends in 4 days — upgrade now to keep code review, CI analysis, auto-apply, custom automations, and more.
Was this helpful? React with 👍 / 👎 | Gitar
There was a problem hiding this comment.
Configure merge blocking · Maintainers can dismiss this review.
Found by gitar-bot review on the ratchet's own pull request. `count_at` counts entries with `grep -vE '^\s*(#|$)'`. grep exits 1 when nothing matches — which is exactly what a ledger containing only its explanatory header produces, and that is the state a repository reaches when it has finished paying its debt down. Under `set -euo pipefail` that exit propagated through the command substitution and killed the script mid-report. The failure was worse than an abort. It printed the first ledger's line, then exited 1 — and exit 1 from this script means "ratchet FAILED". So a repository that had cleared its ledger entirely would be reported as violating the ratchet, with no reason printed and nothing to fix. That is the same class of defect the ratchet exists to prevent: a gate failing for a reason unrelated to what it measures. Reproduced before fixing, and two regression cases added — a comments-only ledger and a completely empty one. The original suite missed both because every case it exercised left at least one live entry behind. 8/8 pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
Two further findings from gitar-bot review on this pull request. 1. ONE EXCEPTION LICENSED ALL FOUR LEDGERS. `Ratchet-exception:` was a single global flag, so a pull request that legitimately needed to add one gitleaks path also silently gained permission to grow the Hypatia baseline, the migration ledger and the root allowlist. The whole point of the check is that each addition is SEEN; a blanket permit defeats it. The declaration now names its ledger, and one naming a different ledger does not license this one. A declaration naming no known ledger is rejected rather than treated as blanket permission — "unparseable" must never mean "allowed". 2. LINE COUNTING IS WRONG FOR TOML, IN BOTH DIRECTIONS. A non-comment line count for .gitleaks.toml counts `paths = [`, the closing `]` and every structural line. Reformatting one array across several lines read as growth, while adding an entry to an existing single-line array read as NO CHANGE — and the second is the dangerous one, because it let an exemption be added invisibly. Now counts quoted entries inside paths/regexes arrays. Verified both ways: a reformat that triples the line count reports `unchanged`, and a genuine +1 on an identical line count reports GREW. ⚠ The counter is a SIBLING SCRIPT, not an inline `python3 -c`. Inlining it mangled the quote escaping into invalid Python — the regex needs both ''' and " for TOML's string forms — and a `|| echo 0` fallback swallowed the SyntaxError, so the count silently became 0 on both sides and the ledger was skipped entirely. A check that reported OK while measuring nothing. There is now no fallback: a counter that cannot run fails the check, because a count of 0 is indistinguishable from an empty ledger, and empty is the state that passes. The workflow stages both files together. Suite extended to 10 cases, including an exception naming the wrong ledger. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
Uh oh!
There was an error while loading. Please reload this page.
…ver ran Three defects, one of them live. **The counter was returning 0 on the real ledger.** `count-ledger-entries.py` matched an array with a non-greedy `\[(.*?)\]`, so a `]` inside a string ended the array early. The estate's own `.gitleaks.toml` contains exactly that — regex entries with character classes — so the counter reported 0 entries for a ledger holding 16. The ratchet then compared 0 against 0 and passed while measuring nothing: precisely the failure the counter's own header warns about. Verified independently: a line-grep for quoted entries in `.gitleaks.toml` returns 16, matching the port. **Ported to shell.** Estate policy bans Python outright, and the JS runtime has already moved once (Deno to Bun); a counter the ratchet depends on must not be invalidated by the next runtime change. awk has no non-greedy matching, so the port walks the array explicitly — which is why a `]` inside a string, a quote inside a comment, and a commented-out array all behave correctly where the regex did not. Eleven tests, including one named for the bug above so a future rewrite cannot reintroduce it. The ratchet's own fixtures moved from Python to jq for the same reason. **16 test suites had never been run by anything.** `self-test.yml` discovered only `tests/*.sh`, while `scripts/tests/` held sixteen suites — including the ratchet's — that no workflow executed. It also ran only on push to main, so it reported after a change had already landed. Now it runs on pull_request too and discovers both roots. ⚠ Enabling discovery makes four pre-existing failures visible, and this job will be red until they are repaired: scripts/tests/governance-gates-505-test.sh scripts/tests/wave0-false-green-test.sh scripts/tests/wave3-scorecards-test.sh scripts/tests/wave5-language-guides-test.sh They are not quarantined and not skipped. A red self-test naming four broken suites is an accurate report; a green one that never ran them was not.
Uh oh!
There was an error while loading. Please reload this page.
…ck fighting the lockfile (#597) Three structural corrections — each removes a *class* of estate failure rather than another instance. ### 1. The baseline never reached the security tab ``` scan ──┬─> findings.json ─> apply-baseline.sh ─> the GATE (filtered) └─> hypatia.sarif ────────────────────> code scanning (NOT filtered) ``` The `code_scanning` ruleset rule blocks on those alerts and knows nothing of the baseline, so **acknowledging a finding silenced the gate while the same finding still blocked the merge** — every required check green, nothing to point at. Measured 2026-08-07: **147 open PRs across 91 repos**, held by **1,777 alerts, zero introduced by the PR they blocked**. `scripts/filter-sarif-by-baseline.sh` filters the SARIF before upload. It contains **no matcher of its own** — it runs the caller's `apply-baseline.sh` (the same copy the gate runs) and uses its `findings_suppressed` list, so there is exactly one matcher by construction. It **fails open** in every error path; 4 of its 9 tests assert that. ### 2. The SPDX check fought `gh actions-lock` The check was `head -1 | grep`, but `gh actions-lock` inserts its managed-by comment at line 1 on every mint — so the check re-failed on files whose licence sat on line 2. It reported 40 workflows estate-wide as missing a header they had, and prepending a default **mis-licensed 3 files** (PMPL shadowed by MPL) before it was caught. Now reads the leading comment block, where REUSE puts the identifier. **Validated against 984 real workflow files: 983 accepted**, the single rejection genuinely has no declaration. ### 3. The pin check named the wrong thing `ERROR: Found unpinned actions:` comes only from the no-lockfile branch but reads generically — indistinguishable from the stale-lockfile branch or an SPDX failure in the same job. It now says which tree lacked a lockfile and states the mutual exclusion, because inline-pinning to "fix" it *removes* actions from the lockfile (hypatia: 14 `startup_failure`s from one commit). **Tests**: 9 + 9, both in `scripts/tests/` so self-test.yml discovers them as of #588. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
`Debtfile.adoc` — what the 2026-08-07 sweep found and did **not** fix, with the measurement behind each number and, for the two big items, the decision each awaits. Matching entries in `.machine_readable/agent_instructions/debt.a2ml`, which is explicitly the next session's Phase 0 input. Most of what remains is blocked on a **ruling, not effort**: **D-1 — 545 Scorecard alerts cannot be closed by any code change.** `scorecard-reusable.yml` runs `results_format: json` with no `upload-sarif` step, so code scanning never gets a fresh run. Of 143 sampled alerts only **7 were updated within 31 days**; 8 date to **2025-12-18**. Proven: minting `actions.lock` and re-running Scorecard left `PinnedDependenciesID` at 5 → 5, `updated_at` frozen at 2026-05-17. `code_scanning` gates **106 of 171** repos on this. **D-2 — 15 repos pin the reusables to `@81dbf2d`**, predating the lockfile-aware pin check, so they report "Found unpinned actions" on repos holding a complete lockfile. A caller pinned to an old reusable **cannot be reached by any fix made in the reusable** — repointing is unavoidable, not tidy. Both were attempted and refused by the permission classifier — correctly, since each changes security gating across ~100 repos. Also recorded: 1,629 genuine findings by rule (with an explicit *do not sweep* warning); the 85-of-91 baseline gap; 4 test suites that fail and had never run before #588; Deno CI still running post-Bun; and the 6 repos the sweep guards deliberately left alone. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Every large CI failure diagnosed in this estate shares one shape: an exemption was added quietly and then never removed.
The 2026-08-06 audit across 424 repositories found a gitleaks gate that had run with no allowlist for months; 13,206 banned-language files, most declared once and forgotten; a
.hypatia-baseline.jsonthat absorbed 255 findings in a single commit, turning a library's central guarantee into accepted debt; and exemption files written in the wrong format, which therefore suppressed nothing — unnoticed for months, because a suppression that doesn't work is silent in exactly the same way as one that does.A gate that checks only the current state cannot catch any of that — each individual state is "valid". What matters is the direction of travel.
This compares a PR against its base and enforces three things:
Ratchet-exception: <why>in a commit message — so adding debt stays possible, attributable and reviewed. A gate nobody can satisfy gets deleted; a gate that costs one honest sentence gets obeyed.noteortracking_issue. This is the rule that would have caught the 255-entry commit.**in.hypatia-ignoreabsorbs files added later, so the ledger grows while appearing to hold steady — precisely the failure this exists to prevent.Covers
.hypatia-baseline.json,.hypatia-ignore,.gitleaks.toml,.machine_readable/root-allow.txt.⚠ The ratchet is itself tested, and the test asserts it CAN FAIL
This estate's recurring defect is the gate that cannot fail —
just proof-checkexiting 0 with no prover,continue-on-erroron the primary secret scanner, a linter that never parsed its input. A ratchet that always passed would be worse than none, because it would license the silent growth it claims to prevent.scripts/tests/exemption-ratchet-test.shexercises all six branches in both directions:🤖 Generated with Claude Code