Skip to content

test: give every tracked evidence file exactly one writer - #4

Merged
FHoffarth merged 1 commit into
mainfrom
fix/evidence-file-ownership
Sep 6, 2026
Merged

FHoffarth merged 1 commit into
mainfrom
fix/evidence-file-ownership

Conversation

@FHoffarth

Copy link
Copy Markdown
Owner

Closes #3.

tracked evidence = deterministic + stable + single owner
run measurements = ephemeral + ignored (audit/run/)

The defect, reproduced rather than assumed

Two audit tests appended to a file another test owns and rewrites whole. Vitest runs
test files in parallel, so what the tracked file ended up holding depended on which of
the pair finished last. Forcing the adverse order with vitest's own file selection — no
test hack — destroyed recorded evidence, silently, with every test still green:

order run evidence lost
gaps.test.tsevidence.test.ts 4 lines
icc-check.test.tsreview-attacks.test.ts 2 lines

A full inventory of every writeFileSync / appendFileSync in audit/ and src/
confirmed these were the only two cases — the judgement is not based on filenames.

The fix

Both appenders record real evidence — findings, verdicts, canary survival — not timings,
so they get their own tracked file rather than the ignored run log.

evidence file writer before writer now
audit/evidence-notes.txt evidence.test.ts + gaps.test.ts evidence.test.ts
audit/review-attacks.txt review-attacks.test.ts + icc-check.test.ts review-attacks.test.ts
audit/gaps-notes.txt (new) gaps.test.ts
audit/icc-notes.txt (new) icc-check.test.ts

No citable evidence text changed. The five lines move across verbatim into the file of
the test that produced them. Nothing is reworded, reordered or dropped, and neither
long-standing file is restructured — the remaining diff lines are "no newline at end of
file" shifts.

Regression guard

EVIDENCE_OWNERS in the harness names the owner of each artifact;
audit/evidence-ownership.test.ts checks that table against what the sources actually
write. It counts every write call against the ones whose path it can read, so a path
assembled at runtime fails the check instead of slipping past it.

Verified by reintroducing both mistakes:

injected regression guard
a second writer for review-attacks.txt FAIL — "is written by more than one test file"
a write path built at runtime FAIL — "writes to a path this check cannot read"

An earlier version of the guard asserted that each declared artifact exists. That check was
itself order-dependent — it ran before the new files had been produced — which is precisely
the fault this PR removes. It was dropped.

Evidence

Windows, core.autocrlf=true:

Gate Result
Product tests 5 files / 43 tests passed
Audit tests 21 files / 296 tests passed (was 20 / 290)
tsc --noEmit clean
vite build 204 modules, 4.26s
npm audit --audit-level=high 0 vulnerabilities
git diff --check clean
git status --porcelain after a full audit run on the committed tree empty
repeated full audit runs clean both times
both previously destructive orderings, owner last clean

npm ci could not run in the working checkout: Windows held an EPERM lock on
@rolldown/binding-win32-x64-msvc. Rather than kill unattributable node processes, the
lockfile was verified by running npm ci against a copy of package.json and
package-lock.json in a scratch directory — clean, 0 vulnerabilities. CI runs the real
npm ci on a fresh runner.

Merge gate

Full product and audit suites pass, every tracked evidence file has exactly one writer, and
a complete audit run leaves the tracked working tree clean on Windows with
core.autocrlf=true — including under the orderings that previously destroyed evidence.

Not in this PR

package-lock.json records "license": "ISC" while package.json says "MIT", so any
npm install rewrites the lockfile. Same class as this sprint, but out of scope; the change
was reverted and is left for a separate decision.

No product, parser or verification logic is touched. No test removed or weakened. No tag,
release or deploy proposed.

🤖 Generated with Claude Code

Two audit tests appended to files another test owns and rewrites whole:
gaps.test.ts into evidence-notes.txt, icc-check.test.ts into review-attacks.txt.
Vitest runs test files in parallel, so what the tracked file ended up holding
depended on which of the pair finished last. Running the appender and then the
owner destroyed four recorded lines in the first case and two in the second,
silently, with every test still green.

Both appenders record real evidence - findings, verdicts, canary survival - not
timings, so they get their own tracked file rather than the ignored run log. The
lines move across verbatim; no existing evidence text is reworded, reordered or
dropped, and neither of the two long-standing files is restructured.

EVIDENCE_OWNERS in the harness names the owner of each artifact, and
evidence-ownership.test.ts checks that table against what the sources actually
write. It counts every write call against the ones whose path it can read, so a
path assembled at runtime fails the check instead of slipping past it. Verified
by reintroducing both mistakes and watching it go red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@FHoffarth
FHoffarth merged commit c2347b7 into main Sep 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evidence-file ownership: one tracked evidence artifact = one writer

1 participant