Skip to content

ci(lint): run the skill-frame freshness gate's self-test half in CI (#8110) - #8171

Merged
hotlong merged 1 commit into
mainfrom
claude/issue-8110-skill-frame-selftest-ci
Aug 12, 2026
Merged

ci(lint): run the skill-frame freshness gate's self-test half in CI (#8110)#8171
hotlong merged 1 commit into
mainfrom
claude/issue-8110-skill-frame-selftest-ci

Conversation

@hotlong

Copy link
Copy Markdown
Contributor

Fixes#8110

One step added to lint.yml's typecheck job, immediately after check:skill-frame-sync:

- name: Self-test the skill-frame freshness gate (self-test half only, never the scan)run: node scripts/check-skill-frame-freshness.mjs --self-test

Invoked with node, deliberately notpnpm check:skill-frame-freshness — that npm script is --self-test && the scan, and the scan half is the CI-hostile one the script's own header argues against (tautological green on a merge ref, false red on any PR that legitimately edits the decision frame). Wiring the npm script would recreate exactly the problem this card exists to route around. No scan-half wiring was added, here or anywhere.

The step shape has precedent in the same job: the changeset-family self-tests (#6509) are already in lint.yml as "the SELF-TEST halves only — the real scans stay in pr-automation.yml's changeset-check, and the split is the whole point of this step". This is the second instance of that split, so the step name says self-test half only and the comment block says why, in order that the omission reads as a decision rather than an oversight.

Measurements

The dispatching PM held four hypotheses weakly and asked for readings rather than agreement. All four were measured. Three held; the fourth needs a correction to how it was stated.

A. Hermetic — confirmed, and no checkout change is needed. The self-test manufactures its specimen from the working tree's own frame documents and drives it through temp git repos whose origin points at a path that does not exist, so the offline rung is exercised by real code without touching the network. Measured in a git clone --depth 1 of the branch, which carries one commit and norefs/remotes/origin/main at all: 12/12 cases green. It is also moot for the placement chosen — the typecheck job already checks out with fetch-depth: 0 for the authorable-surface deletion gate — but the depth-1 reading is the one that proves the step would survive a future job that does not.

B. Cheap — confirmed. ~1.3s in the depth-1 clone, ~2.0s in a full worktree, wall-clock, including node start-up. It needs no node_modules (the script imports only node builtins and check-skill-frame-sync.mjs), so it is cheaper than any of its neighbours, all of which pay a pnpm script hop.

C. Non-empty family — confirmed, with exactly one other member, filed as #8170 and not fixed here per the dispatch ruling. check:dev-prereqs is the only other gate matching the criterion (a scan half that cannot run in CI, keeping a CI-safe --self-test half out with it): its self-test is green in 0.12s on an unbuilt checkout with no dependencies installed, while its scan is hard-red there ("The workspace is not built — 67 of 67 workspace packages..."). Enumeration method, and the two rejected candidates with reasons, are written up in #8170 so the negative half of the survey is reviewable rather than asserted. One methodological note worth carrying: workflow YAML comments must be stripped before matching, or check:platform-checklist lands in the "wired" column on the strength of a lint.yml comment explaining why it is deliberately not wired.

D. Frame-editing PRs stay green — confirmed for wording, but the claim as stated was too broad; here is the sharper version. A structure-preserving edit does not redden it: rewording an axis body in .claude/agents/os-dev.md, and rewording axis prose across all three frame files at once, both left the self-test andcheck:skill-frame-sync green. What does redden it is a structural change — an axis added or removed, or a declaring sentence reworded past the shared anchors. Measured that case too: rewording os-dev.md's declaring sentence gives

Error: self-test fixture could not be derived from .claude/agents/os-dev.md (internal-dev):
the REAL document does not parse, so there is nothing to demote — this is a
`pnpm check:skill-frame-sync` failure first

and, in the same tree, node scripts/check-skill-frame-sync.mjs exits 1 with the same diagnosis. So the new step adds no failure class the step directly above it does not already have: any input that reddens it already reddens check:skill-frame-sync, which has been required in this job since #5865, and the self-test's own message names that gate as the thing to fix first. That is the honest form of D's premise — not "nothing can redden it", but "nothing can redden it alone".

The one real cost is on a PR that deliberately restructures the frame (adds a fourth axis, drops one): it must update the derivation in check-skill-frame-freshness.mjs in the same PR. That PR was already obliged to update check-skill-frame-sync.mjs's anchors, so the obligation is not new — the CI step only makes it arrive before merge instead of after.

Verification

  • node scripts/check-skill-frame-freshness.mjs --self-test — 12/12 cases pass, on the worktree and in the depth-1 clone.
  • Gates re-derived against the actual changed path (node scripts/pm/dispatch-gates.mjs .github/workflows/lint.yml) and run: check:changeset-gate-self-tests, check:node-version, check:nul-bytes (7422 files scanned, 0 control bytes), check:required-contexts, check:shard-attestation, check:type-check-coverage, check:workflow-status-functions, node scripts/check-changeset-no-major.mjs — all green.
  • check:type-check-debt was not run: its --re-measure half refuses to run without the full workspace built ("55 workspace dependencies of the ledgered packages have no built type entry point on disk"), and building 77 packages to re-measure TypeScript debt for a change that touches no TypeScript source is not proportionate. Its --self-test half is shared with check:type-check-coverage, which passed. CI runs it against a built tree.
  • The workflow file parses and the step lands where intended (41 steps in typecheck; the new one runs node scripts/check-skill-frame-freshness.mjs --self-test).

Changeset

None, by route 2 — skip-changeset, applied to this PR. lint.yml's own prose prescribes it: "this PR edits a CI-internal script is the textbook skip-changeset case — such a PR releases nothing". Nothing user-visible ships from a workflow step. Note the interaction the same comment block documents: a skip-changeset PR is exempted from changeset-check wholesale, which is exactly why the changeset-family self-tests were moved into lint.yml unconditionally — and the same reasoning is what puts this step there rather than anywhere gated.


Generated by Claude Code

…8110)
`check:skill-frame-freshness` is `--self-test && the scan`, and only the SCAN
half is CI-hostile: on a fresh merge ref "is this tree current with origin/main"
is a tautological green, and on any PR that legitimately edits the decision
frame it is a false red. That reasoning — the script's own header — was keeping
the OTHER half out of CI too, and the price is measured on #8024: the self-test
fixture had stranded twice, neither time noticed, while the gate sat in the gate
list reading as coverage and protecting nothing.
Add one step to lint.yml's `typecheck` job, next to `check:skill-frame-sync`,
running only `node scripts/check-skill-frame-freshness.mjs --self-test`. Invoked
with `node`, deliberately not `pnpm check:skill-frame-freshness`, which would
drag the scan half in with it. Same step shape as the changeset-family
self-tests already in this job (#6509): self-test halves in lint.yml, the
CI-hostile scan left where it belongs.
Measured before wiring, in a `--depth 1` clone with no remote-tracking ref:
green, ~1.3s, no network and no history — the specimen is manufactured from the
tree's own frame documents and driven through temp git repos whose `origin`
points at a path that does not exist. Adversarially: rewording axis prose across
all three frame files keeps it green; a structural change (an axis dropped, or a
declaring sentence reworded past the shared anchors) reddens it AND reddens
`check:skill-frame-sync` above with the same diagnosis, so this step adds no
failure class the neighbouring step does not already have.
Claude-Session: https://claude.ai/code/session_01P6QzoCR4U6RKxdbCHMJp3V
Co-authored-by: Claude <noreply@anthropic.com>
@vercel

vercelBot commented Aug 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
objectstackIgnoredIgnoredAug 12, 2026 6:36pm

Request Review

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cdsize/sskip-changesetPR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

check:skill-frame-freshness --self-test runs nowhere in CI, so a broken fixture is invisible until a human runs the gate

2 participants

@hotlong@claude