Skip to content

fix: make audit runs side-effect free - #2

Merged
FHoffarth merged 1 commit into
mainfrom
fix/audit-run-side-effects
Sep 6, 2026
Merged

FHoffarth merged 1 commit into
mainfrom
fix/audit-run-side-effects

Conversation

@FHoffarth

Copy link
Copy Markdown
Owner

An audit run used to rewrite a tracked evidence file. That is the same signal the
evidence chain uses to say something moved, so the suite must not produce it by itself.

What was actually wrong

Measured rather than assumed: a full audit run left eleven files showing as modified,
but git diff --numstat showed ten of them byte-identical. Exactly one line of
content had changed:

-bomb: refused after 21 ms: The text in this image unpacks to more than FilePass will read...
+bomb: refused after 19 ms: ...

Three separate causes, all removed:

  1. A timing in tracked evidence. Milliseconds are a fact about the machine, not
    about FilePass. The outcome (refused: <message>) stays in audit/sabotage-notes.txt;
    the timing goes to audit/run/, which is now ignored.

  2. A foreign append — the latent half. audit/resource.test.ts appended to a file
    that audit/sabotage.test.ts truncates at module load. What the file ended up holding
    depended on which of the two finished last. In the observed run the line simply
    vanished; under another interleaving it would have dirtied the tree instead. That
    append moves to the same ignored run log.

  3. Line endings. The remaining ten came from the tests writing LF while a Windows
    checkout under core.autocrlf=true holds CRLF. .gitattributes pins the generated
    evidence to eol=lf. Without it, "clean working tree" is permanently a false signal
    on Windows — worse for an evidence chain than the timing line was.

The rule this establishes

tracked evidence  = deterministic + stable + single owner
run measurements  = ephemeral + ignored (audit/run/)

Evidence

Run after the commit, on Windows with core.autocrlf=true:

Gate Result
Product tests 5 files / 43 tests passed
Audit tests 20 files / 290 tests passed
tsc --noEmit clean
vite build clean
git status --porcelain after a full audit run empty

The .gitattributes addition carries a renormalisation risk, so it was checked
explicitly rather than assumed: git ls-files --eol reports i/lf w/lf for the
evidence set, and git add --renormalize . stages nothing. The commit touches six
files and no evidence file was dragged in by EOL normalisation.

Merge gate

Full product and audit suites pass, and a complete audit run leaves the tracked
working tree clean on Windows with core.autocrlf=true.

Known and deliberately not fixed here

Two more instances of the same class, left alone because closing them changes the
layout of evidence files that earlier review rounds cite:

  • audit/gaps.test.ts appends to audit/evidence-notes.txt, which audit/evidence.test.ts writes.
  • audit/icc-check.test.ts appends to audit/review-attacks.txt, which audit/review-attacks.test.ts writes.

Both contents are deterministic; the ordering across test files is not. Tracked as a
follow-up: Evidence-file ownership — one tracked evidence artifact = one writer.
Existing citable evidence stays stable; each test either owns exactly one tracked file
or writes volatile run data to audit/run/ only.

No tag, release or deploy is proposed by this PR.

🤖 Generated with Claude Code

A tracked note that changes on every run is the same signal the evidence chain
uses to say something moved, so the suite must not produce it by itself.

Two causes, both removed. The bomb notes carried the milliseconds the run took,
which is a fact about the machine rather than about FilePass; the outcome stays
in the note and the timing goes to audit/run/, which is not tracked. And
resource.test.ts appended to a file sabotage.test.ts truncates at load, so what
the file ended up holding depended on which of the two finished last - that
append moves to the same untracked run log.

.gitattributes pins the generated evidence to LF. The tests write LF, a Windows
checkout under core.autocrlf=true holds CRLF, and every run left the whole set
looking modified even where not a byte of content had changed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@FHoffarth
FHoffarth merged commit 228b9d0 into main Sep 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant