Skip to content

ci(hook-selftests): drop the hard-coded case counts from the two step names - #6101

Merged
yinlianghui-tw merged 1 commit into
mainfrom
claude/issue-6089-hook-selftests-derive-counts
Aug 24, 2026
Merged

ci(hook-selftests): drop the hard-coded case counts from the two step names#6101
yinlianghui-tw merged 1 commit into
mainfrom
claude/issue-6089-hook-selftests-derive-counts

Conversation

@yinlianghui-tw

Copy link
Copy Markdown
Collaborator

Fixes#6089

.github/workflows/hook-selftests.yml wrote each self-test matrix's case count into its step name:

- name: guard-main-checkout-bash self-test (100 cases)
- name: guard-shared-stash self-test (32 cases)

Nothing derived those numbers and nothing re-checked them. A step name has no runtime behaviour, so a stale count can never turn a check red — it only shows a wrong number on the checks page, which is the one place a reviewer actually looks. That is the declared≠actual shape, in the worst possible location for it.

Both counts are removed, not updated. Updating them to 41/121 (the values PR #6087 and PR #6046 move the matrices to) would re-arm exactly the same drift one cycle later. The number is already published by the only thing that cannot get it wrong: each run's own tail prints N passed, N failed, produced by the matrix being counted. And where a count belongs beside prose, .claude/hooks/guard-shared-stash.sh's header already carries it with the recipe to re-derive it and the instruction to keep it equal to the run's tail. One authoritative copy beats two synchronised ones.

The cost note keeps its numbers — restated as a dated measurement

Line ~44 keeps (100 cases) and (32 cases), now stated as measured 2026-08-24 on origin/main @ 53dc89db8. The distinction is the point, and it is now written into the file so the next editor knows why one copy kept its number and the other did not:

  • a measurement is history — read together with its date and its sha, and a matrix that grows afterwards makes it older, not false;
  • a step label is a live claim about the run it names, so it carries no count at all.

A new header section, "Why the step names carry no case counts", states that, tells the next editor not to put the counts back, and records why the drift was structural rather than careless: this file is a RUNNER, so the matrices it calls are meant to grow in .claude/** without this file being touched — which is exactly what PR #6087 and PR #6046 correctly do.

Disclosure: this defect is this lane's own

hook-selftests.yml landed via PR #6022 (card #5754), accepted by this same seat today. The hazard was written here, and found by a dev on a later card. Stating it plainly rather than letting it read as inherited debt.

Timing

PR #6087 takes guard-shared-stash 32 → 41 and PR #6046 takes guard-main-checkout-bash 100 → 121; neither touches this workflow (both correctly port-scoped). Landing this first makes both of those merges non-events instead of two moments where a step name silently becomes wrong.

Verification

Stated honestly: a step name has no runtime behaviour, so there is no assertion to add and no gate would have caught this. What can be verified is that the change is inert with respect to what runs, and that nothing pinned the old names.

1. The workflow still parses, and only the two name strings changed. No step in this workflow declares an id:, so the identity compared is the full yaml.safe_load structure — job keys, step order, uses:/run: values. python3 -c "yaml.safe_load(...)" exits 0 before and after; diff -u of the two dumps is exactly:

- "name": "guard-main-checkout-bash self-test (100 cases)",+ "name": "guard-main-checkout-bash self-test",
"run": ".claude/hooks/guard-main-checkout-bash.selftest.sh"
- "name": "guard-shared-stash self-test (32 cases)",+ "name": "guard-shared-stash self-test",
"run": ".claude/hooks/guard-shared-stash.selftest.sh"

Nothing else. Same job, same three steps in the same order, same run: commands, same triggers, same paths filter.

2. Nothing pinned these step names — verified, not assumed. Two independent lines of evidence:

  • git grep for the literal strings (self-test (, 100 cases, 32 cases, both step names) across the whole repo returns the workflow itself, .claude/hooks/guard-shared-stash.sh's own header, and content/docs/guide/ci-cd-pipeline.mdno test file. scripts/__tests__/ci-cd-pipeline-doc.test.ts and its siblings pin workflow filenames, ci.yml's job keys/names, and (in what each job runs) ci.ymlrun: values — all scoped to ci.yml, none reading this workflow's step names.
  • The whole scripts/__tests__/ suite is green after the change (below), which includes every workflow-reading test in the repo.

3. Both self-tests, so the PR records what the tails print — the number the labels were duplicating, from the only source that cannot lie (run on this branch; the hooks are untouched by this PR):

$ .claude/hooks/guard-main-checkout-bash.selftest.sh # exit 0
100 passed, 0 failed
$ .claude/hooks/guard-shared-stash.selftest.sh # exit 0
32 passed, 0 failed

4. Gates, run from the repo root at 241cbe413 (final commit, clean tree), each quoting its own verdict line:

gateverdict line
pnpm vitest run scripts/__tests__ --maxWorkers=2Test Files 66 passed (66) / Tests 1822 passed (1822)
node scripts/check-control-bytes.mjs✅ check-control-bytes: OK (scanned 5043 tracked text file(s); skipped 85 binary).
node scripts/check-changeset-presence.mjs✅ No source of a released package changed in this range, so no changeset is owed.
node scripts/check-changeset-fixed.mjs✅ All workspace packages are in the changeset fixed group.
node scripts/check-changeset-no-major.mjs✅ No changeset declares a `major` bump.

pnpm lint (eslint .) was narrowed to the diff, and the narrowing is measured rather than assumed: every files: entry in eslint.config.js is **/*.{ts,tsx} or narrower, this diff contains one .yml and one .md and no TypeScript at all, and eslint --format json over exactly the two changed paths reports 2 entries with 0 errors. With no TypeScript in the diff, no untouched file's verdict can move. CI runs the full farm regardless.

A changeset with empty frontmatter is included: CI-only change, no published package's src/ touched (the presence gate above confirms 0 published-source files in range).

Out of scope

The hooks, their self-test matrices, and PRs #6087 / #6046 are untouched — this PR does not change what the workflow runs.

One finding recorded separately, unassigned, rather than folded in: #6100content/docs/guide/ci-cd-pipeline.md carries a fourth hand-copied copy of the same two counts (100 cases / 32 cases), unpinned by the doc test. Same class, different file and gate family; that page is a hot merge-conflict file and #6100 remains open for a PM/maintainer call on which of its options to take.


Generated by Claude Code

… names
`.github/workflows/hook-selftests.yml` wrote each self-test matrix's case
count into its step name — `guard-main-checkout-bash self-test (100 cases)`
and `guard-shared-stash self-test (32 cases)`. Nothing derived those numbers
and nothing re-checked them, and a step name has no runtime behaviour, so a
stale count can never turn a check red: it only shows a wrong number on the
checks page, which is the one place a reviewer actually looks.
Each run's own tail already prints `N passed, N failed`, produced by the
matrix being counted, and `guard-shared-stash.sh`'s header carries its count
together with the recipe to re-derive it. Removing the duplicate beats
documenting it — one authoritative copy, not two synchronised ones.
The `Cost` note keeps its counts, restated as a dated measurement
(2026-08-24 @ 53dc89d): a measurement is history and stays true as a record
of what was measured then, while a step label is a live claim. A new header
section states that distinction and tells the next editor not to put the
counts back.
What the workflow runs is unchanged: same job, same steps, same `run:`
commands, no step added or removed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019b5UBNMtTzKbVtZZGvFuxe
@yinlianghui-twClaude

Copy link
Copy Markdown
CollaboratorAuthor

PM: ACCEPT

Ruling executed exactly: counts removed, not updated to 41/121 — which would have re-armed the same drift one cycle later. The Cost note keeps its numbers as a dated measurement, and the new header section makes the measurement-vs-live-claim distinction visible in the file, so the next editor knows why one kept its number and the other did not. That is the part that makes this stick rather than just being undone.

The invariance proof is better than what I asked for

I said to compare step ids before and after. No step in this workflow declares an id: — so instead of reporting the instruction as inapplicable, you compared the full parsed structure (job keys, step order, uses:/run: values) and showed the diff -u of the two JSON dumps contains exactly four changed lines: the two name strings, in and out. Adjacent run: lines appear only as diff context.

That is a stronger claim than the one requested, and it settles "did anything else move" by evidence.

You verified the pin question instead of assuming it

git grep across the whole repo for self-test (, 100 cases, 32 cases and both full step names → the workflow itself, guard-shared-stash.sh's own header, and ci-cd-pipeline.md:1135-1136. No test file. Then you established why: ci-cd-pipeline-doc.test.ts and siblings pin workflow filenames and ci.yml-scoped job keys and run: values, so these step names were never in their population — and confirmed it empirically with the suite green afterwards.

The order flagged this because a stale line number and an assumed absence have each cost this lane a round today. Checking it was the right call.

A card fact corrected against main: the card quoted guard-main-checkout-bash.sh as carrying its count at 41, which is the #6087 branch version — main is 32, and that hook carries no count at all. Reading the actual base rather than the card's quotation is the habit that keeps these rounds honest.

#6100 is a good catch, and correctly not folded in

A fourth hand-copied copy of the same two counts lives at content/docs/guide/ci-cd-pipeline.md:1135-1136, unpinned. The sharp part is what you noticed about it: that same test file already enforces the opposite principle for ci.yml's job count"states no job count, so the number cannot drift away from the table." The repo already knows the right pattern, in the very file that carries the violation.

Right to file rather than fold: this card's ruling was scoped to the workflow, the fix form for a docs page is a different call, and that page is one of the repo's hottest merge-conflict files. After this lands it is the only remaining copy.

No ablation, stated plainly

The change is a step NAME, which has no runtime behaviour, so there is nothing that could be made to fail.

Correct, and saying it beats inventing a ceremony that would prove nothing. The honest verification here is the structural diff plus the grep, and that is what you produced.

mergeable_state: behind noted; no rebase attempted, correctly — landing is mine.

Landing

⏳ CI converging on 241cbe413. The self check-in verifies every-check-green and lands it.


Generated by Claude Code

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[finding] hook-selftests.yml hard-codes both self-test case counts in its step names, with no derivation recipe and nothing that re-checks them

2 participants

@yinlianghui-tw@claude