Skip to content

ci(lint): run the .claude/hooks self-test matrices in the required job - #11799

Merged
os-steve merged 1 commit into
mainfrom
claude/issue-11514-hook-selftests-in-ci
Aug 24, 2026
Merged

ci(lint): run the .claude/hooks self-test matrices in the required job#11799
os-steve merged 1 commit into
mainfrom
claude/issue-11514-hook-selftests-in-ci

Conversation

@os-steve

Copy link
Copy Markdown
Collaborator

Fixes#11514

.claude/hooks/ holds the enforcement behind the two rules whose violation is most expensive in this repo — worktree-first and the stash ban — and each guard ships a hermetic self-test matrix whose header tells you to re-run it after touching the hook. Nothing ran them: grep -rn 'selftest' .github/workflows/ matched nothing. So those headers were the enforcement, and a guard that had silently stopped guarding landed green — the worse half of a broken guard, because everyone keeps behaving as though it works.

One step is added to the lint job. No hook is modified.claude/** is governed surface, this workflow is not.

Baseline on main — measured before wiring anything

The first time anything measured these. Both healthy, so this wires an alarm rather than fixing an outage:

self-testcasesexit
.claude/hooks/guard-main-checkout-bash.selftest.sh121 passed, 0 failed0
.claude/hooks/guard-shared-stash.selftest.sh32 passed, 0 failed0

Measured set, rather than the card's description: .claude/hooks/ holds two self-tests, not three. guard-main-checkout.sh — the Edit/Write half of the pair — ships no self-test at all; only the Bash half does. That gap is reported, not closed here (authoring one is a .claude/** change).

Non-vacuity — the step goes RED when a guard breaks

The deliverable, both directions, driving the real block extracted from this file under a real bash -e, the way Actions runs it. guard-main-checkout-bash.sh's sole BLOCK verdict was mutated exit 2exit 0 — the guard stops guarding — as a scratch mutation under trap … EXIT INT TERM, never committed, proven on disk by anchor counts in both directions (exit 2 1→0, exit 0 0→1, sha moved) and restored byte-identical (sha 5db89551c9090bdb before and after; git diff on .claude/ empty).

runstep exitwhat the log says
unmutated tree0discovered 2 · both PASS · all 2 ran and passed
guard mutated1FAIL .claude/hooks/guard-main-checkout-bash.selftest.sh (70 passed, 51 failed, every one want=block got=allow)

The mutated guard is the first discovered, and the second still ran and passed (32 passed, 0 failed) in that same red run — so the collector really does defeat the bash -e masking, rather than merely being written to.

Discovery is the contract, not a list

A hard-coded list would be this card's own defect one level up: add a hook with a matrix tomorrow, it silently is not run, nothing goes red. Proven with a scratch fixture (uncommitted, removed; git status on .claude/ clean afterwards):

runstep exitresult
new passing self-test added, workflow untouched0discovered 3 — picked up with no edit here
same file made to exit 11FAIL … zz-scratch-fixture.selftest.sh — it is really executed, not just listed
run where nothing is discovered1DISCOVERED NOTHING … verified nothing, which is a failure and not a pass (#4690)

That last row is the direction a glob normally fails in: a renamed or moved directory would otherwise make the step pass by running nothing — the same green line this whole family exists to distrust.

Three placement decisions, each on evidence

A step of the lint job, not a job of its own.Lint & Repo Gates is a required status context (registered in check:required-contexts). A new job publishes a context in no ruleset, so a red guard would be advisory and the merge queue would not stop for it — #5617 verbatim, the exact shape this card exists to close.

Tolerate-and-collect, not a bare loop (#10814). A run: block is bash -e, so a bare loop aborts at the first red matrix and leaves the rest unrun. ⚠️check:step-collectorscannot see this block — its population is scripts/|packages/ paths carrying --self-test, and these are .claude/hooks/*.selftest.sh. Confirmed empirically: with this step in the tree that gate still reports 2 step(s) run 2+ independent self-tests. The collector shape here is held by review, so the step carries a ⛔ note against "simplifying" it.

Dependencies verified on the runner's terms, not assumed. Needs jq and git and nothing else — no pnpm, no node, no build, no network. jq is used bare on ubuntu-latest by cut-rc.yml and release.yml already. Hermeticity was measured rather than read off the headers: each matrix builds its own git repo and linked worktree under $TMPDIR, and both were re-run from a primary checkout, a detached HEAD, and a non-repo cwd — 121/0 and 32/0 in all three. So neither this checkout's depth nor its branch is an input. ~2 s.

Gates

Union re-run after the final commit, at a05a5f961. The 13 families derived by node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (which reads the change set itself), plus check:nul-bytes: all green — check-step-collectors✓ 329 run: steps across 26 workflow(s), check-workflow-status-functionsOK (scanned 26 workflow file(s), 49 job(s)), check-nul-bytesOK (scanned 6573 text file(s)), check:required-contexts, check:aggregator-roster, check:shard-attestation, check:type-check-coverage, check:agent-test-spelling, check:node-version, check:pnpm-filter-targets, ci-failure --self-test all exit 0.

One declared narrowing.check:type-check-debt (--self-test && --re-measure) did not complete locally: its --re-measure half refuses on an unbuilt dependency closure by design (#6376, "measuring now would not fail, it would silently measure a DIFFERENT WORLD"), which is a property of a fresh worktree, not of this diff. Its --self-test half passed (182 cases), and the half that reads lint.ymlobserved() — is shared with check:type-check-coverage, which passed with this edit in place, so every assertion this gate makes about lint.yml holds. This diff changes 1 file and 0 TypeScript files, so no ledger number is reachable from it; CI builds the closure before that step and measures there.

No changeset: this PR edits one CI workflow and releases nothing — the textbook skip-changeset case by this file's own prescription.

Sequencing: #11779 also edits .github/workflows/lint.yml, at line ~1654 (after Node-version drift guard). This edit is at line ~787. Verified by content, not assumed — #11779 had not landed on main at branch time.


Generated by Claude Code

`.claude/hooks/` holds the enforcement behind worktree-first and the stash
ban, and each guard ships a hermetic self-test matrix whose header says to
re-run it after touching the hook. Nothing ran them — a grep for `selftest`
over `.github/workflows/` matched nothing — so a guard that had silently
stopped guarding landed green.
Adds one step to the `lint` job (`Lint & Repo Gates`, a required status
context) that DISCOVERS every `.claude/hooks/**/*.selftest.sh` at run time
and runs it. No hook is modified: `.claude/**` is governed surface and this
workflow is not.
- discovered, never listed, so a matrix added tomorrow is picked up with no
edit here
- an empty discovery is red, not green (#4690)
- tolerate-and-collect (#10814): a bare loop under `bash -e` would abort at
the first red matrix and leave the rest unrun, neither green nor red
- a step of the required job rather than a job of its own, so a red guard
stops the merge queue instead of publishing an advisory context (#5617)
Baseline measured on main before wiring: guard-main-checkout-bash 121/0,
guard-shared-stash 32/0. Non-vacuity: with the bash guard's sole BLOCK
verdict mutated to `exit 0` (scratch, restored byte-identical), the step
exits 1, names the failing matrix, and the second matrix still runs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ahemw8RcTgqtxrj15PEZx
@os-steveClaude

Copy link
Copy Markdown
CollaboratorAuthor

ACCEPT. Verified by content on origin/main, not by reading the report back.

Both guards are green — and that is a measurement, not a relief

guard-main-checkout-bash.selftest.sh EXIT 0 (121 passed, 0 failed), guard-shared-stash.selftest.sh EXIT 0 (32 passed, 0 failed). First time anything has measured them. I asked you to run them before wiring precisely because a pre-existing red would have outranked the card, and it is worth stating plainly that the answer came back clean.

The zeros are falsifiable because of the mutation pair, and the ambient-independence check settles my Zone 2 worry properly: both re-run with cwd = primary checkout, detached HEAD, and a non-repo dir — 121/0 and 32/0 in all three. So neither checkout depth nor branch is an input. That is measured, not read off the headers, which is what I asked for.

⭐ Your measured set correction, verified

.claude/hooks/
guard-main-checkout-bash.selftest.sh ← matrix
guard-main-checkout-bash.sh
guard-main-checkout.sh ← NO matrix
guard-shared-stash.selftest.sh ← matrix
guard-shared-stash.sh
*.selftest.sh count: 2

Two, not three.guard-main-checkout.selftest.sh does not exist. The card said "the guard-main-checkout pair" and I passed that phrasing straight through in the dispatch without checking it — CLAUDE.md describes the hooks as a pair, so the self-tests read as a pair too. They are not.

#11800 is the right filing and it matters more than it looks.guard-main-checkout.sh is the Edit/Write half — the one that blocks the first file edit outside a worktree, arguably the more load-bearing of the two — and it has zero cases while its Bash sibling has 121. After your wiring, .claude/hooks/ reads as a covered directory while that hook stays unchecked, and CLAUDE.md states it fails open by design, so a regression there is silent. Wiring a discovery loop over a set with a hole in it is exactly how a directory acquires the appearance of coverage.

Three design points I did not specify, each on evidence

(a) A step of the already-required job, not a new job."a new job publishes a context in no ruleset, so a red guard would be advisory and the merge queue would not stop for it (#5617)". My dispatch said "lint.yml family" and stopped there. A new job would have been green-looking, correctly discovered, correctly failing — and unable to block anything. That is the same defect class as the card itself, one layer up: a guard that runs and cannot enforce.

(b) Tolerate-and-collect, because a run: block is bash -e and a bare loop aborts at the first red, leaving the rest unrun (#10814). And you proved it rather than asserting it: the mutated matrix is the FIRST discovered, and in the same red run the second still ran and passed (32 passed, 0 failed). Without that, "I used a collector" is a claim about the code; with it, it is a property of the run.

(c) Empty discovery is RED (#4690). I ⛔'d hard-coded lists; you went further and closed the mirror-image hole — a renamed or moved directory can no longer make the step pass by running nothing. DISCOVERED NOTHING … verified nothing, which is a failure and not a pass is the right verdict text.

Non-vacuity

Red/green driven through the real extracted block as bash -e <file>, the invocation Actions uses. Mutation: the sole exit 2 BLOCK verdict → exit 0 (the guard stops guarding) ⇒ STEP_EXIT 1, naming the file, 70 passed, 51 failed, all 51 want=block got=allow. Proven on disk both directions by anchor counts and sha256, with the script aborting at exit 99 if any anchor failed to move — so a no-op edit could not have been read as a result. Given what a sibling dev hit an hour ago (a shell quoting error silently killing an edit while the gate read green), that abort is not paranoia.

Discovery guarantee proven with a scratch fixture: discovered 3 with the workflow untouched, and made to fail ⇒ named in the output — which proves it is executed, not merely listed. That distinction is the one a hard-coded list would have hidden.

#11801 is you reporting a gap in your own work

check-step-collectors counted 2 step-collectors while the file now has three — empirical confirmation that its population (scripts/|packages/ paths carrying --self-test) is structurally blind to .claude/hooks/*.selftest.sh. So the collector shape you argued for in (b) is currently held by a review comment rather than by the gate whose entire argument is that a static shape cannot vouch for itself.

Finding that your own new block sits outside the gate that would police it, and filing it rather than leaving the count unexplained, is the report I most want to see. Graded next round.

The declared narrowing on check:type-check-debt is disposed of with three pieces of evidence rather than one, and the second is the good one: the half that reads lint.yml is observed(), shared with check:type-check-coverage, which passed with this edit in place — so every assertion that gate makes about the workflow holds.

Flipping to ready. Arming once every check is green.


Generated by Claude Code

@os-steve
os-steve marked this pull request as ready for review August 24, 2026 17:00
@os-steve
os-steve added this pull request to the merge queueAug 24, 2026
Merged via the queue into main with commit dce6a01Aug 24, 2026
27 checks passed
@os-steve
os-steve deleted the claude/issue-11514-hook-selftests-in-ci branch August 24, 2026 18:05
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cdsize/mskip-changesetPR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nothing in CI runs the .claude/hooks/*.selftest.sh matrices — a broken PreToolUse guard lands green (objectstack half of objectui#5754)

2 participants

@os-steve@claude