Check a driftNote on every run, and say what the audit does not evaluate

[ptr727]

Resolves the audit-tooling half of #563, filed from ptr727/Blog after its re-integration. All four of that issue's asks, plus a defect in one of the fixes it proposes.

The gate that suppressed the check

spec/audit.py wrapped the whole freshness check in if not findings:. The rationale was sound in isolation, in that a clean repo has no outstanding work for a pending-marker note to describe, but the consequence is that any repo carrying one finding it cannot clear has its entire driftNotes list exempted. Blog carries exactly that, the carried: AGENTS.md references the template repo finding tracked in #552, so its notes were never checked however clean the rest of the audit ran. The repo with open findings is where a stale note is most likely, which is the inverse of what the gate produced.

Both shapes are now evaluated on every run, worded by context rather than suppressed:

Note	Audit clean	Findings open
prose marker (pending, still, ...)	contradicted outright, as before	raised as which of the open findings it means
names a check id	surfaced for a hand decision	surfaced for a hand decision

Measured live, this is one added advisory across the fleet. Of 53 notes on 22 repos, one carries a marker (MediaTools, pending fleet-wide ratification), and it now reads while 25 finding(s) are open - confirm it describes one of them rather than closed work. Under the old gate it was silent.

The check-id matcher, and why it is not anchored

#563 proposes \(([a-z]+\.[a-z.]+)\)$. That pattern matches neither of the two notes it was written for, because both end the sentence after the paren:

... so it cannot be moved or diffed against upstream (hugo.vendored.provenance).

Run over the whole registry it matches zero notes on zero repos, which is indistinguishable from a fleet carrying no such note. That is the silent-narrowing shape GOVERNANCE.md "Verification Discipline" names: a pattern that matches less still exits zero.

CHECK_ID_RE is therefore unanchored, and the self-test covers the trailing-period case, the mid-sentence case, an id absent from the catalog, an id whose type the repo does not declare, and a parenthesized version string as the false positive the shape has to exclude. Watched failing: re-anchoring the pattern to $ turns five cases red and the suite reports SELFTEST FAIL.

What the audit will not claim

Resolving the id is as far as a tool can honestly go here, because spec/audit.py never reads spec/project-types.json at all. So the audit checks that the id exists and that the repo declares its type, then hands the check itself to the auditor. The finding clears when the note is deleted, which is the retirement path #560 promised, made visible rather than left to a matcher that could never fire.

That same fact is now stated where an agent reads a run rather than only in the issue: AUDIT.md section 4, the console line printed for a clean repo, the --issue body preamble, and OPERATIONS.md. Adding a check to project-types.json changes what an auditor must judge and changes no tool's output, and silence from a tool that was never looking reads exactly like a pass.

Blog's two notes are dropped

Both deviations closed in ptr727/Blog#30 and are on Blog's ground-truth main (2b132e4), verified by reading that branch rather than trusting the issue:

• hugo.vendored.provenance - themes/README.md records the upstream repository, commit 154d006e, its upstream date, git describe, the license location, and both local edits.
• hugo.generator.pinned - the version and SHA256 are declared once at .github/actions/install-hugo/action.yml:26-27, and validate-task.yml:76 and deploy-site-task.yml:79 both consume that composite action.

Verification

Gate	Result
spec/audit.py --selftest	PASS, and FAIL on the anchored matcher
spec/validate.py	22 cataloged, 0 backlog, clean
spec/audit.py Blog MediaTools (live)	behaves as described above
markdownlint-cli2	44 files, 0 issues
cspell (README, HISTORY)	0 issues
editorconfig-checker	clean
scripts/prose_lint.py	no new violations on any touched file, net -2

Refs rather than Closes, since a closing keyword cannot fire from a develop-targeted pull request. #563 is closed by hand with evidence once this merges.

Filed by an agent in ptr727/Blog, resolved here.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR adjusts the fleet audit tooling so driftNotes freshness is checked on every run (even when other findings exist), adds support for driftNotes that name a parenthesized check id, and clarifies (in tooling output + docs) what spec/audit.py does not evaluate.

Changes:

• Refactors driftNote freshness into driftnote_findings() and runs it unconditionally, including check-id extraction via CHECK_ID_RE.
• Loads spec/project-types.json to resolve whether a driftNote’s named check id exists and whether the repo declares the owning type.
• Removes two now-stale Hugo driftNotes from the Blog registry entry and updates messaging/docs to emphasize that per-type checks are judged by hand.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
spec/audit.py	Adds check-id parsing + resolution for driftNotes, ungates driftNote freshness checks, and updates console/issue text about non-mechanized per-type checks.
registry/repos.json	Drops two retired Hugo driftNotes from the Blog entry.
OPERATIONS.md	Clarifies that a clean spec/audit.py run is not evidence about per-type checks in spec/project-types.json.
AUDIT.md	Expands the guidance that section-4 checks are hand-judged and explains driftNote freshness behavior.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (5)

spec/audit.py:1281

• The console clean-status line uses "no per-type check in spec/project-types.json is run here". Since cross-cutting checks also live in spec/project-types.json, this is ambiguous and can read as if cross-cutting checks are evaluated; the intent elsewhere is that none of them are.

 print(" clean (deterministic checks only; no per-type check in spec/project-types.json is run here - AUDIT.md section 4 is judged by hand)")

spec/audit.py:1174

• This issue preamble says "No per-type check in spec/project-types.json is run here". Because spec/project-types.json also contains cross-cutting checks, this wording can be misread as implying cross-cutting checks are run; the audit runs none of them.

 f"This lists what the audit mechanically detects. No per-type check in `spec/project-types.json` is run here, "

spec/audit.py:162

• The driftNote finding message says "no per-type check is mechanized", but driftNotes can also name cross-cutting check ids (from the same spec/project-types.json catalog). Using "per-type" here can be misread as implying cross-cutting checks are evaluated, even though the audit evaluates none of them.

This issue also appears in the following locations of the same file:

• line 1174
• line 1281

 out.append(("DRIFT", f"registry: driftNote names check '{cid}', which this audit does not evaluate (no per-type check is mechanized, AUDIT.md section 4) - judge it by hand and delete the note once it passes: {quoted}"))

OPERATIONS.md:46

• This paragraph says "No per-type check in spec/project-types.json runs here". Because cross-cutting checks also live in spec/project-types.json, this phrasing can be read as if cross-cutting checks do run; the intent is that none of them do.

Findings are a point-in-time snapshot read live over the API. Re-run before acting on one, and quote the run stamp in any issue derived from it. The deterministic subset lives here, and the full letter-and-intent verdict is [AUDIT.md](./AUDIT.md). No per-type check in `spec/project-types.json` runs here, so a clean run is not evidence about any of them.

AUDIT.md:136

• The driftNotes section describes a check-id note as "a parenthesized id" but the example shows only the bare id in backticks. Since the matcher requires parentheses, the example should include them to avoid suggesting that hugo.generator.pinned (without parens) will be detected.

**Reconcile `driftNotes` in the same pass.** A registry `driftNote` records a *current* deviation from the baseline. Once the deviation is resolved the note is deleted, not left describing finished work, since hand-maintained prose drifts silently otherwise. `spec/audit.py` flags two shapes of note, neither of them gated on the rest of the audit being clean. A note asserting outstanding work in prose ("pending", "not yet", "missing", "behind", ...) is contradicted outright by a clean audit, and where findings are open it is raised as a question of which one it means, because gating the check on a clean audit meant one standing finding a repo could not clear exempted its whole note list, and the repo carrying open findings is where a stale note is most likely. A note naming the check that would retire it, as a parenthesized id (`hugo.generator.pinned`), is the mechanically checkable shape and is surfaced on **every** run: the audit resolves the id against the catalog and confirms the repo declares its type, then hands the check itself to the auditor, since section 4 above is judged by hand. **So a note naming a check id is retired by a person, not by a run.** Write it that way anyway. The id says exactly what would close the note, and the surfaced finding puts that decision in front of whoever runs the audit rather than leaving the note to sit until someone rereads it.

[ptr727]

Answering round 2's five suppressed findings here, since a suppressed finding has no thread to pair an answer to. All five accepted, fixed in 792d7c6. Four are one class, and chasing it found a false claim of mine underneath.

The class: "per-type" read as excluding cross-cutting

spec/audit.py:1281 - The console clean-status line uses "no per-type check in spec/project-types.json is run here". Since cross-cutting checks also live in spec/project-types.json, this is ambiguous and can read as if cross-cutting checks are evaluated; the intent elsewhere is that none of them are.

spec/audit.py:1174 - This issue preamble says "No per-type check in spec/project-types.json is run here". Because spec/project-types.json also contains cross-cutting checks, this wording can be misread as implying cross-cutting checks are run; the audit runs none of them.

spec/audit.py:162 - The driftNote finding message says "no per-type check is mechanized", but driftNotes can also name cross-cutting check ids (from the same spec/project-types.json catalog). Using "per-type" here can be misread as implying cross-cutting checks are evaluated, even though the audit evaluates none of them.

OPERATIONS.md:46 - This paragraph says "No per-type check in spec/project-types.json runs here". Because cross-cutting checks also live in spec/project-types.json, this phrasing can be read as if cross-cutting checks do run; the intent is that none of them do.

The ambiguity is real and all four sites are fixed. But the closing clause each finding shares, that the audit evaluates none of the cross-cutting checks either, is not correct, and taking it as the fix would have made things worse. I checked the catalog against the runner rather than adopting it:

crossCutting check	Deterministic counterpart in spec/audit.py
branch.both-protected	branch facts and protection
branch.ruleset.develop, branch.ruleset.main	normalized ruleset diff
setup.secrets.present, setup.secrets.noforbidden	secret names against spec/secrets.json
setup.dependabot.ecosystems	ecosystem coverage
setup.driftnotes.current	the freshness check this PR rewrites
parity.cspell	the workspace-duplicates-the-word-list check
recurring.norepoxref	carried-file template reference
readme.sections	per-scope section presence

So roughly a third of section 4 is mechanized, and my own sentence in AUDIT.md said "Every check below is judged by hand, and none of them is mechanized." That was the actual defect here, and the four "per-type" strings were the accurate part. Widening them to cover all checks, which is the reading the comments invite, would have entrenched a false claim in four more places.

Fixed in the other direction. The line falls between the two halves of the catalog, not around it: no check belonging to a project type is evaluated anywhere, while the deterministic subset lands on several cross-cutting checks without being organized by them. AUDIT.md section 4 now states that split and names which cross-cutting ground the subset covers, the four sites say "project-type" and name the partial cross-cutting coverage, and the driftNote finding now says the audit does not evaluate the named check by id, which holds whichever half the id came from. That last one matters for the exact case raised: a note may name setup.driftnotes.current, and the self-test covers a cross-cutting id.

The instance

AUDIT.md:136 - The driftNotes section describes a check-id note as "a parenthesized id" but the example shows only the bare id in backticks. Since the matcher requires parentheses, the example should include them to avoid suggesting that hugo.generator.pinned (without parens) will be detected.

Accepted, and worth more than a formatting nit given what this PR is about: an example that does not match the matcher is how the anchored-pattern defect got written in the first place. The example is now (hugo.generator.pinned) and says the bare id is not detected.

Gates after the fix: --selftest PASS, validate.py clean, markdownlint 44 files 0 issues, editorconfig clean, prose_lint 0 on both touched docs, live audit.py Blog renders the new strings correctly.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

spec/audit.py:879

• audit_repo() now always calls driftnote_findings(), which assumes spec includes the loaded project-types catalog at spec["types"]. _selftest() (and any other minimal-spec callsites) pass a reduced spec dict, so this can raise KeyError even when the repo entry has no driftNotes to check. Guard the call so driftNotes logic only runs when driftNotes exist, and emit a clear ERROR if the catalog is missing when needed.

 findings.extend(driftnote_findings(entry, spec, len(findings)))

spec/audit.py:1175

• The issue preamble sentence ends with "AUDIT.md section 4's", which reads like a possessive and is grammatically unclear. Since this text is user-facing in generated issues, it would be clearer to say the verdict "lives in" (or "is in") AUDIT.md section 4.

 f"This lists what the audit mechanically detects. No check belonging to a project type in `spec/project-types.json` is run "
f"here, and the cross-cutting dimensions are covered only in part, so the full letter and intent verdict is AUDIT.md section 4's.")

[ptr727]

Round 3's two suppressed findings, answered here since neither has a thread. Both accepted, fixed in d5cb17a. The earlier five are answered above and unchanged.

The catalog dependency

spec/audit.py:879 - audit_repo() now always calls driftnote_findings(), which assumes spec includes the loaded project-types catalog at spec["types"]. _selftest() (and any other minimal-spec callsites) pass a reduced spec dict, so this can raise KeyError even when the repo entry has no driftNotes to check. Guard the call so driftNotes logic only runs when driftNotes exist, and emit a clear ERROR if the catalog is missing when needed.

Measured rather than taken on the description, and the two halves land differently.

The reachability half does not hold. check_id_owner is called only from inside the loop over driftNotes, so an entry with no notes never touches the catalog. Driven directly:

reduced spec + a check-id note -> KeyError: 'types'
reduced spec + no driftNotes -> []

The existing _selftest fixture is the second row, which is why the suite passed throughout. No live caller produces the first, since main() always loads the catalog.

The message half is a real defect, so it is fixed. It now raises naming spec/project-types.json, and a self-test case drives that path so the behavior is asserted rather than assumed.

Deliberately still an error, not the suggested skip, and worth saying why since the suggestion is the more natural-looking option. Both fallbacks are worse than the raise. Resolving ids against an absent catalog reports every id undefined, which hands out a work list whose remedy is to delete correct notes. Skipping the check when the catalog is missing makes it report clean over notes it never read, which is precisely the false clean this PR exists to remove, reintroduced one layer down.

The possessive

spec/audit.py:1175 - The issue preamble sentence ends with "AUDIT.md section 4's", which reads like a possessive and is grammatically unclear. Since this text is user-facing in generated issues, it would be clearer to say the verdict "lives in" (or "is in") AUDIT.md section 4.

Accepted. It now reads "the full letter and intent verdict lives in AUDIT.md section 4". The user-facing point is the right one: this string is rendered into a filed issue, not printed to a terminal.

Gates: --selftest PASS including the two new cases, validate.py clean, markdownlint 44 files 0 issues, prose_lint no new violations on spec/audit.py.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.