ci: require every self-test CI ships to be a self-test CI runs

[os-steve]

Fixes#11150

Direction 1 of the card: enumerate the matching-rule-gate family, then require --self-test in CI for every member so removing one becomes a visible act. Membership is derived and measured, not inherited from the card's candidate list — and the measurement changed the answer.

The criterion

A gate is in the family iff its clean-tree verdict is a fixed point of predicate weakening.

Green means the finding set is empty. Weakening the rule that decides membership in that set — a looser correspondence, a narrower recognizer — can only shrink the set. The empty set is the fixed point of shrinking. So the production verdict is identical before and after the rule breaks, and --self-test, which supplies an adversarial input a clean tree by construction does not contain, is the only instrument watching it.

A gate leaves the family exactly when its rule has a must-be-reached witness recorded in the tree — a ledger row that fails when the scan stops finding it, an exact-set pin, a count. That converts "no findings" into "the recorded set is exactly reached", which is an equality, and weakening an equality's measured side breaks it.

The derivation — four ablations, on 1f6d04703

One ablation per gate. Each mutation was confirmed on disk before any reading was taken (anchor count 1 ⇒ 0, injected marker 0 ⇒ 1), and each ran under a restore trap.

gate	rule ablated	production	--self-test	in family?
check-auth-mount-ledger	exact METHOD path accounting ⇒ strict-prefix credit	0 GREEN	1 RED	yes
check-error-code-casing	local-fallback recognizer ⇒ never matches	0 GREEN	1 RED	yes
check-route-envelope	MODULES[file] ⇒ basename fallback	0 GREEN	0 GREEN	yes, and uninstrumented
check-dispatcher-error-vocabulary	objlittemplate recognizer ⇒ never matches	1 RED	1 RED	no

Verdict lines quoted from the runs themselves:

• check-auth-mount-ledger production stayed exit 0 and still printed accounting is EXACT on `METHOD path` -- a prefix route is never credited to a longer sibling — the prose kept claiming the property the code no longer had. Only --self-test said a strict-prefix mount was CREDITED to its longer sibling -- the #10534 defect, reintroduced. This reproduces PR feat(gate): fail a rawApp auth mount that carries no ledger row (#10534 follow-up 4) #11149's measurement on current main, as Zone 2 asked rather than inheriting it.
• check-error-code-casing production stayed exit 0 with the recognizer that check-error-code-casing reports a clean sweep it cannot support: a lowercase code in an || fallback matches none of its four recognizers, and two live ones ship today #10658's escape needed deleted outright; --self-test failed 4 cases (expected 1 hit(s), got 0). This answers the card's explicit open question: its self-test would have caught the loosening, and nothing else would.
• check-dispatcher-error-vocabularyreddens its own production run: [stale-row] packages/runtime/src/dispatcher-error-vocabulary.ts declares 'APPROVAL_*_FAILED' at packages/rest/src/rest-server.ts (objlittemplate) but the scan no longer finds it. Its classification ledger is a must-be-reached witness, so the rule is observable on a clean tree. A candidate the card named, falsified by measurement.
• check-route-envelope moved neither run. Positive control in the same harness, same gate, different mutation: narrowing its discovery convention (entry.endsWith('-routes.ts')) reddens the production run at exit 1 — so the double-green is a property of that rule, not a broken harness. Filed separately; not touched here (strengthening-only, and it is a different card).

Why this ships a superset rather than a list of members

check-route-envelope and check-dispatcher-error-vocabulary sit on opposite sides of the criterion, and check-route-envelope sits on both sides for two different rules inside one file. Membership is a property of a rule, not of a script, and no static classifier decides it. A family enumerated by name would wire --self-test for the wrong set and then read as complete — which is the exact failure shape the card exists to close.

So the enforcement runs over the mechanically decidable superset:

every script CI runs whose code ships a --self-test must have that self-test run by CI.

Every family member is inside it by construction — a gate whose only instrument is its self-test necessarily ships one — and nothing is admitted by judgment.

What landed

scripts/check-self-test-wired.mjs (new), wired into lint.yml in the dominant SCRIPT --self-test + SCRIPT shape:

• Population, derived: a script a workflow names — directly, or through a root package.json alias a workflow names, following pnpm ALIAS chains — whose code, comments masked with the repo's own js-comment-mask, carries the literal. The mask decides in both directions: pnpm check:platform-checklist appears in lint.yml only inside a comment (maintainer-run by ruling), and counting it would fabricate a member.
• Right boundary, because this gate is itself in the family for it: --self-test-extra is not an invocation of --self-test, and script paths compare by exact equality so scripts/check-foo.mjs is never credited to scripts/check-foobar.mjs. Both directions pinned in --self-test.
• SELF_TEST_RUN_OTHERWISE, shrink-only: scripts whose self-test CI does run, by a route other than the flag. Two measured shapes — drives (a wrapper whose whole body spawns another tool's self-test) and inline (a gate that runs its own cases on every invocation). Each row carries evidence that is re-checked every run, not a sentence: a literal that must still be in the script's code, and for drives the driven tool, which must still ship a self-test. A row whose evidence is gone fails; a row whose script gets wired fails.
• Refusals, never quiet passes: empty population, missing workflow dir, unreadable ledger row, package.json with no scripts — all exit 1 naming what could not be read.
• No count is written into CI prose. The gate prints its own scope line, per the card's own warning that a number in prose goes stale in silence.

One genuine gap closed. The enumeration found five members whose --self-test no workflow ran with the flag. Measured one at a time rather than assumed, as Zone 2 asked, four turned out to be already covered by another route (recorded as evidence-checked ledger rows) and one was real:

script	disposition
scripts/render-release-coverage-anchor.mjs	wired in release-coverage-patrol.yml — genuinely unrun; its verdict() mapping decides did-not-run / findings / clean, and a broken mapping renders a wrong report at exit 0, including the reading #4690 forbids
scripts/pm/check-dispatch-gates.mjs	ledger drives — its whole body spawns scripts/pm/dispatch-gates.mjs --self-test
scripts/docs-audit/check-affected-docs.mjs	ledger drives — its whole body spawns scripts/docs-audit/affected-docs.mjs --self-test
scripts/check-comment-mask-corpus.mjs	ledger inline — main() runs the comparator's 12 cases before the sweep
scripts/check-test-completeness.mjs	ledger inline — selfTest({ quiet: true }) on every invocation, a placement its header records as deliberate

Live verdict on this branch:

✓ check-self-test-wired: every one of the 127 script(s) CI runs that ship a `--self-test` has that self-test run by CI.
scope: 172 file(s) under scripts/, 136 carrying `--self-test` in code (comments masked); 127 of those are run by 27 workflow(s); 123 have their self-test run through the flag, 4 through a recorded route.

Reverse verification — the teeth, demonstrated

From the committed state, node scripts/check-step-collectors.mjs --self-test was deleted from its lint.yml step (run-line anchor 1 ⇒ 0, marker 0 ⇒ 1, restore trap armed):

gate exit=1
check-self-test-wired: 1 finding(s)
[self-test-not-run] scripts/check-step-collectors.mjs

Restored, git diff --stat empty. Removing a --self-test invocation from CI is now a visible act, which is what direction 1 asked for.

Direction 2 and 3

Direction 2 (a convention pin asserting a self-test contains a case the production run cannot produce) is not delivered: the classifier it needs is precisely the one the measurement above shows is not statically decidable — two rules in one file land on opposite sides. Triage marked it optional; the superset covers every member it would have covered, without the classifier. Direction 3 is not delivered alone anywhere: every statement here is attached to a mechanism that fails.

Gates run — all on fbe397b22, the final commit

Derived live with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no hand-written path list; the script took the change set from the merge base itself — 3 path(s) vs merge base 1f6d04703, three-dot semantics). It reported 24 matched families and already discovers this PR's own gate (node scripts/check-self-test-wired.mjs [lint.yml]). Each verdict below is the gate's own printed line, captured before any pipe.

check:nul-bytes exit=0 check-nul-bytes: OK (scanned 6637 text file(s) ... no NUL bytes)
check:agent-test-spelling exit=0 ✓ 0 violations — 365 file(s) · 3617 bare `--` token(s)
check:entry-guard exit=0 ✓ 155 scripts/ file(s) — every entry guard goes through invoked-as.mjs
check:parse-guard exit=0 ✓ 154 scripts/ file(s) — every TypeScript parse goes through ts-parse.mjs
check:node-version exit=0 OK (33 setup-node step(s) across 27 workflow(s), all on Node 22)
check:pnpm-acquisition exit=0 OK -- every job that runs pnpm acquires it first
check:pnpm-filter-targets exit=0 ✓ 135/168 `--filter` occurrence(s) across 26 file(s) resolve
check:required-contexts exit=0 ✓ 6 required context name(s) pinned across 2 workflow(s)
check:shard-attestation exit=0 ✓ 2 aggregate gate(s) count 3 declared leg(s)
check:workflow-status-functions exit=0 OK (scanned 27 workflow file(s), 50 job(s))
check:cross-package-test-inputs exit=0 All 109 self-test cases passed.
check:type-check-coverage exit=0 OK — 65/78 workspace packages type-checked
check-aggregator-roster exit=0 ✓ 3 aggregator(s) across 2 workflow(s); roster == needs: in both directions
check-ci-filter-parity exit=0
check-cross-package-test-inputs exit=0
check-release-section-coverage exit=0 (advisory)
check-required-contexts exit=0 ✓ 6 required context name(s) pinned
check-shard-attestation exit=0
check-step-collectors --self-test exit=0 ✓ 120 assertions, 3 live block(s) driven under a real `bash -e`
check-step-collectors exit=0 ✓ 341 `run:` steps across 27 workflow(s); 3 step(s) run 2+ independent self-tests, all through a collector
check-whole-set-label-write exit=0
docs-audit/check-drift-comment exit=0 ✓ 56 cases pass across 5 fixture diff(s)
pm/ci-failure --self-test exit=0
render-anchor --self-test exit=0 OK render-release-coverage-anchor --self-test: 10 assertions pass
check-self-test-wired --self-test exit=0 4 live ledger row(s) verified, plus the comment mask, the right boundary, alias resolution and both audit directions
check-self-test-wired exit=0 ✓ every one of the 127 script(s) ... has that self-test run by CI

check-step-collectors is the one worth reading twice: its static half refuses a run: block that self-tests two or more distinct scripts without a collector. Both blocks added here name one script each — SCRIPT --self-test + SCRIPT, the tolerated shape where the abort is the point — and the gate confirms the repo-wide count is unchanged at 3 collector steps.

Two declared narrowings

1. pnpm lint was run narrowed, and the narrowing is measured rather than assumed. (a) Population read from eslint.config.mjs itself via isPathIgnored / calculateConfigForFile: 1 of the 3 changed paths is in eslint's population — the two workflow files report ignored=true with zero configured rules — and that path has exactly 2 active rules (no-restricted-imports, comment-swallow/no-code-inside-block-comment). (b) Count read from --format json: results.length = 1, errors 0, warnings 0. (c) Invariance for untouched files is the config's own recorded property, not my inference: eslint.config.mjs states it "never enables type-aware linting (no parserOptions.project, no typed @typescript-eslint rules) for ANY file", so no cross-file program exists for this diff to move. Positive control in the same run, on the same real file path: two planted violations (code inside a block comment; a bare fs import) each produced exactly 1 error where the real file produced 0 — a control that reported nothing would have made the zero unfalsifiable.
2. check:type-check-debt was not run. Its --re-measure leg refuses without the built workspace closure (56 workspace dependenc(ies) ... have no built type entry point on disk) and asks for a full turbo run build over every package — heavy work behind a contended shared lock. Its sibling check:type-check-coverage passes here, and this diff adds no package, no tsconfig, and no typecheck script, so it cannot move the debt reading. CI builds the closure before that step, as lint.yml does.

skip-changeset

Nothing user-visible ships: one new CI gate script, one new lint.yml step, one --self-test leg added to a patrol workflow. No package source, no public surface, nothing to publish.

Co-tenancy

.github/workflows/lint.yml is also touched by PR #11908. Checked against current origin/main (bb62d4d08): no commit between the merge base and origin/main touches either workflow file, and git merge-tree reports 0 conflicted hunks — this branch is not dirty on that file, so no step was resolved away and no before/after step count is in question. #11908 remains open and absorbs the merge.

---

Generated by Claude Code

[os-steve]

ACCEPT. Flipping out of draft (scripts/** + workflows are not governed); arming once green.

⭐ The card asked for a family. The measurement says a family cannot exist — and that is the result

Triage scoped this as "enumerate the matching-rule-gate family … measure membership rather than assume." You measured, and found something better than a list: two rules inside one file land on opposite sides.check-route-envelope is in the family for one rule and instrumented for another; check-dispatcher-error-vocabulary — a named candidate — is out, because its stale-row ledger reddens the production run.

So no static classifier decides membership, and enforcement over a mechanically decidable superset is the only sound shape. That is precisely the failure my ruling 4 named — "a family enumerated by guess wires --self-test for the wrong set and then reads as complete" — reached from the other direction: not a wrong guess, but a proof that guessing cannot work here.

The criterion earns its keep: a gate is in the family iff its clean-tree verdict is a fixed point of predicate weakening (green ⇒ empty finding set; weakening only shrinks it; empty is the fixed point), and it leaves the family exactly when the rule has a must-be-reached witness that turns emptiness into an equality. That is a real definition, not a heuristic.

Verified against the branch, not read back

claim	check	result
scope	git diff --stat	✅ 3 files, 661/−1
ledger is shrink-only, 4 rows	the source	✅ ⛔ SHRINK-ONLY at :124, exactly 4 rows, each carrying evidence rather than a bare name
patrol wiring is real	the workflow diff	✅ render-release-coverage-anchor self-test added, with its rationale
⚠️lint.yml co-tenancy	- name: step count	✅ 183 → 184 — one added, nothing dropped

That last one is the check I asked for specifically, because a bad lint.yml resolution silently drops a step. Going further and proving it with git merge-tree (0 conflicted hunks) and by confirming no commit between the merge base and origin/main touches either workflow is the stronger form — it shows the branch never went dirty, rather than that you cleaned up after it did.

Three things I want on the record

Ablation B and its control C. B loosened MODULES[file] and moved neither production nor --self-test; C, a different mutation on the same gate, reddened production. Without C, B's double-green is indistinguishable from a broken harness. That pairing is what makes #11920 a finding rather than a guess.

Ablation A answered the card's own open question — only check-error-code-casing's self-test catches the loosening, so it is in the family. The card asked; you measured.

The discarded readings. A syntax-error no-op ablation, and a positive control on a synthetic path that produced 0 and was non-discriminating — both discarded rather than reported. Discarding a control that cannot discriminate is the difference between a measurement and a coincidence, and this is the third time tonight a dev has caught themselves that way.

Two notes, neither blocking

⚠️ You used a whole-set label write because the additive endpoint was unavailable ($GITHUB_TOKEN → 401 from this seat). That is the exact surface of open findings #11881 and #10778 — a whole-set PUT can silently drop a label another actor added between your read and your write. You declared it and did it as a read-then-union, which is the right mitigation, and the label set reads correctly now. Recording it because the hazard is real, not because this instance went wrong.

Direction 2 correctly not delivered: the classifier it needs is the one your own measurement shows is undecidable. Saying so beats shipping a weaker version of it.

Spin-off #11920 noted — check-route-envelope's declared-vs-discovered correspondence has no instrument at all.

---

Generated by Claude Code