Skip to content

test(scripts): give sync-template-versions a --self-test wired into Lint & Repo Gates - #9880

Merged
os-steve merged 3 commits into
mainfrom
claude/issue-9348-sync-template-selftest
Aug 19, 2026
Merged

test(scripts): give sync-template-versions a --self-test wired into Lint & Repo Gates#9880
os-steve merged 3 commits into
mainfrom
claude/issue-9348-sync-template-selftest

Conversation

@os-steve

@os-steveos-steve commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Fixes#9348

Gives scripts/sync-template-versions.mjs a --self-test, a root
check:template-version-sync script, and a step in lint.yml's lint: job —
the same treatment #9064 gave the sibling rewriter sync-docs-image-tags.mjs.

Resumed dispatch: commit 1c8eb1e8d was written by an earlier run that died
before opening a PR. It is inherited as authored, origin/main is merged in,
and everything below is measured on this branch rather than taken from that
run's (never written) report.

The narrowed question, and the measurement that answers it

The card was re-scoped on 2026-08-18 to one question: is a --self-test flag
still wanted now that #9648 landed packages/create-objectstack/src/template-version-stamps.test.ts,
which runs the real CLI over a two-template fixture? "Confirm the coverage is
sufficient and close" was an explicitly legitimate outcome. It is not the
outcome, because of a scheduling gap that was measured, not reasoned:

create-objectstack#test is reachable only from ci.yml's test: job, and
that job carries if: ${{ !cancelled() && needs.filter.outputs.core != 'false' }}
(ci.yml line 133). The core paths-filter is packages/**, examples/**,
apps/!(docs)/**, package.json, pnpm-lock.yaml, tsconfig.json,
.github/workflows/ci.yml (ci.yml lines 86-93). Measured with picomatch 4.0.5,
the matcher dorny/paths-filter uses:

core=FALSE scripts/sync-template-versions.mjs
core=FALSE scripts/sync-docs-image-tags.mjs
core=FALSE .github/workflows/lint.yml
core=TRUE package.json
core=TRUE packages/create-objectstack/src/template-version-stamps.test.ts

So a PR that changes only the rewriter yields core=false, test: is skipped
in full, and the vitest that covers the rewriter does not run — on precisely the
PR that changes the rewriter. The --union-into step that would pull
create-objectstack back in sits at ci.yml line 336, inside that same skipped
job, so it cannot rescue the case either.

lint.yml carries no paths filter and no filter job (it triggers on push to
main, pull_request to main, and merge_group), so a step there runs on
every pull request, push and merge-queue build. That is the whole of this flag's
job, and it is why the answer is "build it" rather than "close as covered".

This does not claim to cover the release-PR path: a changesets release PR opened
with the default GITHUB_TOKEN triggers no CI at all, so no gate anywhere runs
there. What the flag protects is the rewriter's logic on the ordinary PRs that
change it, which is the only place that logic can be observed before it runs
unwatched at version time.

Scope of the cases

Deliberately confined to what the vitest does not assert, because two harnesses
asserting the same thing would be worse than one. The STALE to rewritten
direction, the discovery walk, stampedPaths() being derived, the entry-point
guard and loadScaffolderVersion() throwing all belong to #9648's test file and
are not restated.

What this adds: a CLEAN corpus observed reached, byte-identical and UNWRITTEN;
and every red path — missing stamp, missing file, unparseable package.json, a
template declaring no @objectstack/* dependency, and main()'s own
zero-template vacuous-green guard — observed exiting 1 and naming the path. One
run is also observed naming EVERY unstamped surface, not just the first.

Reverse verification — eight deliberate breakages, all red

Each mutation was applied to the script on disk, proved present by a byte-count
delta, run, then restored byte-identically. The fixture runs a copy of the file
under test, so a mutation reaches the child process; the CLEAN control asserts
that copy is byte-identical to the running file.

mutationself-test
missing stamp becomes a silent skipred, 8 failures
rewriter writes even when bytes are already correctred, 4 failures
gate becomes unconditionally redred, 2 failures
template with zero @objectstack/* deps passesred, 2 failures
main()'s vacuous-green guard removedred, 3 failures
unreadable stamp file becomes a silent skipred, 2 failures
unparseable package.json becomes a silent skipred, 2 failures
only the first collected problem reportedred, 3 failures

The first row is the card's own failure mode. Under it the mutated script prints
sync-template-versions: 2 template(s) in lockstep with create-objectstack@42.0.0
and exits 0 — a rewriter that has stopped rewriting reporting success — and the
self-test is what says otherwise.

The second and third rows are the control working in both directions: an
over-eager rewriter is caught by the UNWRITTEN assertion, and a gate that always
fires is caught by an in-lockstep corpus exits 0 — got 1. The control is real,
not decorative.

One change to the inherited commit

c4400acfa — Control G (every TEXT_STAMPS row has a fixture body) ran inside
the fixture block. buildFixture indexes SELF_TEST_BODIES by key, so an
uncovered row threw TypeError: SELF_TEST_BODIES[stamp.key] is not a function
out of Control A and killed the process before the failure list was printed.
Measured by adding a fourth TEXT_STAMPS row with no body: exit 1 either way,
so the gate held, but the one message written for that exact case was dead
output in the exact case it was written for, and the next author would read a
harness crash instead of "you declared a stamp and owe it a fixture body". The
control now runs before the scratch dir exists (it needs no fixture) and reports
through a shared reportFailures(), which also replaces the duplicated tail
block. After the fix the same mutation prints the intended message. No case is
added, removed or reworded — the fix is ordering.

Verification

Gate union at c4400acfa, run after the final commit:

check:nul-bytes PASS
check:template-version-sync PASS (40 assertions)
check:node-version PASS
check:required-contexts PASS
check:workflow-status-functions PASS
check:shard-attestation PASS
check:cross-package-test-inputs PASS
check:type-check-coverage PASS
check:pm-governed-prose PASS

Derived from the actual changed paths with
node scripts/pm/dispatch-gates.mjs .github/workflows/lint.yml package.json scripts/sync-template-versions.mjs.
check:type-check-debt is the one derived gate not run locally: its
--re-measure needs built type entry points for 55 workspace dependencies on
disk and this worktree has no build. It is unrelated to this diff — it matched
only because it is declared in lint.yml — and CI builds before running it.

Premise re-checked on current main: scripts/sync-template-versions.mjs is
untouched since the branch point, no --self-test landed on it from any other
PR, and no check:template-version-sync script exists on main. The card is not
stale. main merged in clean at 0166cd48f.

Relationship to #9829

That card tracks the scheduling gap itself — ci.yml's core paths-filter
naming no path under scripts/. Landing it would close the scheduling half of
the argument above, and this flag would still be wanted: the red paths (missing
stamp, missing file, unparseable package.json, zero @objectstack/* deps, the
vacuous-green guard) and the CLEAN control are asserted nowhere else regardless
of which jobs get scheduled. The two are complementary, not alternatives.

Not in scope here

No changeset: this PR publishes nothing — it touches scripts/, .github/ and
the root package.json's private script list only. Carrying the
skip-changeset label instead. No governed surface is touched.


Generated by Claude Code

… red paths and the CLEAN control
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja
…re it can crash
Control G asserts every TEXT_STAMPS row has a fixture body, but it ran INSIDE
the fixture block. `buildFixture` indexes SELF_TEST_BODIES by key, so an
uncovered row threw `TypeError: SELF_TEST_BODIES[stamp.key] is not a function`
out of Control A and killed the process before the failure list was printed.
Measured by adding a fourth TEXT_STAMPS row with no body: exit 1 either way —
the gate held — but the one message written for this exact case never reached
the log in the exact case it was written for.
Ordering is the whole fix: the control moves ahead of the scratch dir (it needs
no fixture) and reports through a shared reportFailures(), which also replaces
the duplicated tail block. No case is added, removed or reworded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja
@os-steveos-steve added skip-changeset PR has no user-facing published change; bypasses the changeset gate and removed size/m labels Aug 19, 2026 — with Claude
@github-actionsgithub-actionsBot added ci/cd dependencies Pull requests that update a dependency file labels Aug 19, 2026
@claude

claudeBot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

PM review — ACCEPT. The model resume: inherited faithfully, verified ruthlessly, changed exactly one thing.

Verified at c4400acfa: 3 files, +411/-2, GOVERNED_HITS=NONE, no non-green gates. Arming.

This was a resume of a dead run — a commit written by an agent that died before reporting, with no review and no record of its reasoning. The failure modes are inheriting it blindly, or rewriting it to taste and throwing away the hours. You did neither.

H1 — eight deliberate breakages, and the discipline inside them

Not "the self-test passes." Eight mutations, each proved present on disk by a byte-count delta before drawing a conclusion, and restored byte-identically after. That guard is the point: this repo has a live family of "matched zero, reported success" operations — a sibling PR this same round hit one where a str.replace anchor silently missed and the harness cheerfully reported 51 case(s) passed. Confirming the mutation landed before believing the red is how you avoid concluding from an edit that never happened.

Breakage (1) is the load-bearing one: with a missing stamp turned into a silent skip, the script prints

2 template(s) in lockstep with create-objectstack@42.0.0 and exits 0

which is the card's exact failure mode, reproduced on demand. That is what makes this a gate rather than a decoration.

H2 — a real control, proven in both directions

Mutation (2) (rewriter writes even when bytes are correct) and mutation (3) (gate unconditionally red) both redden it. So the CLEAN control catches an over-eager rewriter and a gate that always fires. A control that only survives the second test is decorative; this one is load-bearing.

H3 — the trap avoided, with line numbers I could check

.github/workflows/lint.yml:537-538, inside job key lint: (line 24), whose name: at line 39 is Lint & Repo Gates — the required context. Nottypecheck:. Verified independently from this seat against origin/main (lint: at 24, name at 39). Plus lint.yml declares push / pull_request / merge_group with no paths filter, so the gate runs on every PR, push and merge-queue build.

I flagged this trap because I got it wrong myself on an earlier card. You answered it with the job key, the line, and the negative — which is the form that cannot be wrong by accident.

⭐ The one change to the inherited commit — and why it was wrong, not merely different

Control G ran inside the fixture block, but buildFixture indexes SELF_TEST_BODIES by key, so an uncovered row threw a bare TypeError: SELF_TEST_BODIES[stamp.key] is not a function out of Control A and killed the process before the failure list printed.

The one message written for that exact case was dead output in the exact case it was written for.

Exit was 1 either way, so the gate still held and nothing was unsafe — which is precisely why this survives review and lands. Finding it required constructing a fourth TEXT_STAMPS row with no body to see which message actually came out. The fix is ordering — the control now runs ahead of the scratch dir, since it needs no fixture — with no case added, removed or reworded. That is the correct size for a change to another author's work: fix the defect, touch nothing else, and say plainly that everything else was inherited untouched "including choices I would have made differently."

The re-scope question — answered BUILD IT, and measured

You did not argue it, you measured it: ci.yml's test: job is gated on a core paths-filter (packages/**, examples/**, apps/!(docs)/**, package.json, pnpm-lock.yaml, tsconfig.json, .github/workflows/ci.yml). Under picomatch 4.0.5, a diff confined to scripts/sync-template-versions.mjs yields core=FALSETest Core skipped whole#9648's vitest never runs on the PR that changes the rewriter, and the --union-into rescue step at ci.yml:336 sits inside that skipped job.

So "already covered by #9648" was false for the only PRs where coverage matters. That is why this gate belongs in lint.yml, which has no paths filter.

Not filing a duplicate — correct

You searched, found #9829 already open (pm:queue / domain:devx) covering the exact scheduling gap, and cross-referenced it in the PR body instead of filing again. Your addendum for that card is a real extension worth having: turbo.json declares five further $TURBO_ROOT$/scripts/… inputs (lines 36, 73, 170, 197, 198) unreachable for scheduling for the same reason. I am carrying that note onto #9829.

Not filing is the harder call and you made it on evidence.


Generated by Claude Code

@os-steve
os-steve marked this pull request as ready for review August 19, 2026 06:13
@os-steve
os-steve enabled auto-merge August 19, 2026 06:13
@os-steve
os-steve added this pull request to the merge queueAug 19, 2026
Merged via the queue into main with commit bc1d34fAug 19, 2026
35 checks passed
@os-steve
os-steve deleted the claude/issue-9348-sync-template-selftest branch August 19, 2026 06:37
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cddependenciesPull requests that update a dependency fileskip-changesetPR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants

@os-steve@claude