Skip to content

CI: run the generator and assert its output matches what is committed - #30

Merged
emmanuelgjr merged 2 commits into
mainfrom
ci/generator-reproducibility-guard
Aug 28, 2026
Merged

CI: run the generator and assert its output matches what is committed#30
emmanuelgjr merged 2 commits into
mainfrom
ci/generator-reproducibility-guard

Conversation

@emmanuelgjr

Copy link
Copy Markdown
Contributor

Why

generate.js sat broken on main for several commits (fixed in #28) and nothing caught it — because no job ran it. The Content Validation workflow runs validate.js and stats:check, neither of which loads the generator. A file that produces every entry in data/entries/ and both webapp data bundles had no CI coverage at all.

What

A Generator reproducibility job that runs node scripts/generate.js and then asserts:

git diff --exit-code -- data/entries docs/data.js docs/incidents.js

That catches three failures the existing jobs cannot:

  1. the generator failing to load or throw at all (the Fix: generate.js failed to run — AST catalogue block inserted three times #28 case);
  2. a hand-edit to a generated file, which would be silently overwritten on the next run;
  3. a source change that was never regenerated, so data/entries/ and the webapp disagree with the Markdown.

Also

The path filters still listed three source directories. Nothing under ast-top10/ triggered validation, and a change to scripts/generate.js itself triggered nothing. Both are added.

Verification

Run locally against the current tree: generator runs clean, git diff --exit-code over those paths is empty — the guard passes on main as it stands, so it gates future drift rather than flagging existing state.

🤖 Generated with Claude Code

emmanuelgjr and others added 2 commits August 28, 2026 17:00
generate.js sat broken on main for several commits (#28) and nothing
caught it, because no job ran it: the Content Validation workflow runs
validate.js and stats:check, neither of which loads the generator. A
file that produces every entry in data/entries/ and both webapp data
bundles had no coverage at all.

Adds a "Generator reproducibility" job that runs generate.js and then
asserts `git diff --exit-code` over data/entries, docs/data.js and
docs/incidents.js. That catches three things the previous jobs could
not: the generator failing to load, a hand-edit to a generated file,
and a source change that was never regenerated.

Also brings the path filters up to date — they still listed three source
directories, so nothing under ast-top10/ triggered validation, and a
change to generate.js itself triggered nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without this the workflow cannot verify its own edits — the new
Generator reproducibility job did not run on the PR that added it,
because .github/ was not in the path filters.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@emmanuelgjr
emmanuelgjr merged commit 8ab2097 into main Aug 28, 2026
7 checks passed
@emmanuelgjr
emmanuelgjr deleted the ci/generator-reproducibility-guard branch August 28, 2026 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant