Skip to content

audit: state and enforce the scan ledger's ordering rule - #125

Closed
thedavidmeister wants to merge 4 commits into
mainfrom
2026-08-16-issue-89
Closed

audit: state and enforce the scan ledger's ordering rule#125
thedavidmeister wants to merge 4 commits into
mainfrom
2026-08-16-issue-89

Conversation

@thedavidmeister

@thedavidmeisterthedavidmeister commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Closes#89

audit/mutation-test-scans.json is a bare JSON array whose readers — the org
health check, and any human opening it — treat the newest record as the one that
describes current coverage. Nothing in this repo said that, so "which element is
newest" was a convention held in the adversarial-mutation-test skill and
nowhere else, and a reader had to diff commit against HEAD to find out
whether the file was current at all.

This PR states that rule where the file lives. It does not enforce it.

  • audit/README.md — the ledger is appended in run order, so its last element is
    the most recent scan; every element is a fact about the tree its commit
    names; whether the newest scan still describes the working tree is a fact about
    HEAD that no record can carry, so the README gives the one-liner that
    measures it.

audit/mutation-test-scans.json itself is untouched, byte for byte, and
foundry.toml's fs_permissions block is byte-identical to main's.

No test guards this, and it can regress silently

This PR previously shipped test/lib/LibAuditLedger.sol,
test/concrete/AuditLedgerExternal.sol and
test/audit/mutation-test-scans.t.sol — 320 lines asserting that the committed
ledger is non-empty, that every record carries a timestamp reading
YYYY-MM-DDTHH:MM:SSZ, and that each timestamp is strictly after the one before
it. Those three files are deleted here, along with the
{ access = "read", path = "audit/mutation-test-scans.json" } grant they needed.

So, stated plainly: the ordering rule is documented and nothing checks it. A
record appended out of run order, a record with a malformed timestamp, a record
with no timestamp at all, or an emptied ledger will all land green. That
regression is silent — forge test stays green, CI stays green, and the only
thing standing between the rule and its violation is a human reading
audit/README.md and then reading the JSON. audit/README.md says so in the
file rather than leaving a reader to find it out, which is the whole of what this
PR now delivers.

The check is not rehomed to rainlanguage/rainix#317. The record schema is
owned by rainlanguage/adversarial-mutation-test
— its README carries the field template and its SKILL.md states that campaigns
close by appending here — so a checker for that schema belongs with the schema,
not in rainix-static's generic lint set and not in one consumer of it.

Why they went: 12 of the 13 test functions fed synthetic JSON, built by the test
file's own helpers, into LibAuditLedger — a library added by the same PR. All
17 mutants in the matrix mutated that library; nothing under src/ was probed.
The one test whose subject was the real committed file,
testCommittedLedgerIsAppendOrdered, is vacuous: the ledger holds exactly one
record, so an append-order assertion has no pair to compare, and its timestamp
oracle restates TIMESTAMP_SHAPE's own layout rather than disagreeing with it.

Cause, for the record: the fixer brief driving this fleet said "every fix ships a
test, verify-by-hand is never acceptable", with no exemption for changes that have
no behaviour. That was wrong and is corrected. It produced hand-rolled static
analysis in Solidity across nine PRs, this one included.

The decision: no second scan record here

The issue proposes two halves. The second (state the ordering rule) is landed.
The first — re-run the campaign at HEAD and append a record — is deliberately
not, and this is the call and its ledger:

  • A scan record is a record of a run that happened. Writing one without running
    the campaign is fabricated evidence, not a fix.
  • Running it is the whole adversarial-mutation-test campaign over all three
    source libraries. That campaign closes by filing issues and opening coverage
    PRs, which this brief forbids; it is its own work order, not a line in a
    fix-one-finding PR.
  • Carrying cost if it were appended now: it is falsified within a day. A fleet of
    PRs is open against src/** and test/** right now, every one of which
    changes the tree the record would name. Cost of removing it later: it cannot be
    removed — records are historical facts about a tree and are never edited.
  • And it would not close the finding. The issue's own statement of the problem is
    that "a stale single entry is indistinguishable from a current one". A fresh
    record resets the clock; it does not make the next stale entry any more
    distinguishable. Stating the rule does, permanently.

The re-run is genuinely due, and it is due after the fleet's PRs land.

Where the proposed fix was changed

The issue offered either "a records object with the invariant documented" or
"a sibling audit/README.md line". The README route was taken, for two reasons
the issue could not have known:

  1. The bare array is the schema rainlanguage/adversarial-mutation-test defines
    for every repo's ledger — its README carries the field template, its
    SKILL.md states that campaigns close by appending to it and that "the org
    health check reads the newest entry". Wrapping this repo's copy in a records
    object forks that schema in one consumer and breaks the reader the rule exists
    to serve. The invariant is documented without changing the shape.
  2. audit/mutation-test-scans.json is also the file audit/mutation-test-scans.json testsAfter reads 102, a count that occurs at no commit in the range, and no field names the tree it describes #86 edits
    (summary.testsAfter 102 → 84, plus a new testsAfterCommit). Leaving it
    byte-identical here leaves audit/mutation-test-scans.json testsAfter reads 102, a count that occurs at no commit in the range, and no field names the tree it describes #86 a clean landing.

The README also picked up a one-line section for audit/protofire/, because a
README for audit/ that documents one of the directory's two entries is
misleading about the other.

QA

Run as nix develop -c … from this branch, after merging origin/main
(b422d97) in — merged, never rebased.

  • forge test at the branch tip — Ran 18 test suites … 142 tests passed, 0 failed, 0 skipped (142 total tests).
  • forge test at the merge commit ac297bf, immediately before the strip,
    measured in a clean worktree — Ran 19 test suites … 155 tests passed. The
    count drops by exactly the 13 the deleted file contributed, and the suite count
    by the one file.
  • forge test at origin/main (b422d97), measured in a clean worktree — Ran 18 test suites … 142 tests passed. Identical to the branch tip, which is the
    point: this branch now adds no test.
  • forge fmt --check — clean, exit 0.
  • git diff origin/main...HEAD --stat is one file: audit/README.md, +31.

Left to siblings

`audit/README.md` states that `mutation-test-scans.json` is appended in run
order, so its last element is the most recent scan, and that currency against
the working tree is measured by comparing that element's `commit` to `HEAD`.
`test/audit/mutation-test-scans.t.sol` holds the committed ledger to that rule
through `test/lib/LibAuditLedger.sol`: non-empty, every record carrying a
`YYYY-MM-DDTHH:MM:SSZ` timestamp, each strictly after the one before it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thedavidmeisterthedavidmeister self-assigned this Aug 16, 2026
@coderabbitai

coderabbitaiBot commented Aug 16, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@thedavidmeister, you've reached your PR review limit, so we couldn't start this review.

Next review available in:57 minutes

Limit details: You’ve used all 1 included review currently available under your plan.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c30c45d6-21f2-4741-abff-327e6dbe10c6

📥 Commits

Reviewing files that changed from the base of the PR and between b422d97 and 6176813.

📒 Files selected for processing (1)
  • audit/README.md

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

thedavidmeisterand others added 3 commits August 16, 2026 18:55
`iLedger` is an immutable built in the constructor, matching how `LibFs.t.sol`
holds `iExternal`, and `requireStrictlyAfter` states the shape precondition its
byte comparison relies on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Delete test/audit/mutation-test-scans.t.sol, test/lib/LibAuditLedger.sol and
test/concrete/AuditLedgerExternal.sol, and revert the
audit/mutation-test-scans.json read grant in foundry.toml.
12 of the 13 test functions fed synthetic JSON built by the test file's own
helpers into a library added by the same PR; all 17 mutants in the matrix
mutated that library and nothing under src/ was probed. The one test whose
subject was the real committed ledger is vacuous, since the ledger holds a
single record and an append-order assertion has no pair to compare. The record
schema is owned by rainlanguage/adversarial-mutation-test, so a checker for it
belongs with the schema rather than in this consumer.
audit/README.md stays and now says plainly that nothing here enforces the
ordering rule it states.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thedavidmeister

Copy link
Copy Markdown
ContributorAuthor

@coderabbitai review

@coderabbitai

coderabbitaiBot commented Aug 16, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@thedavidmeister

Copy link
Copy Markdown
ContributorAuthor

Closing: the rule this PR states and enforces belongs to the producer, not to
this repo.

audit/mutation-test-scans.json is written by the adversarial-mutation-test
skill, and the missing ordering rule is a property of the format that skill
specifies. Stating the invariant in one consumer leaves every other repo running
a campaign to rediscover it, and leaves the next record written into this repo
free to ignore a rule that lives only here.

The issue this PR closed, #89, is closed for the same reason. The fix is filed
at the producer with the evidence measured here:
rainlanguage/adversarial-mutation-test#25

That covers the ordering and authority rule, the tie-break, a schema version
distinct from skillVersion, and the migration rule for the bare array this
repo already has committed — which is the part that decides what happens to this
repo's existing ledger.

Nothing in this PR is wrong. It is the right change in the wrong repo.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

audit/mutation-test-scans.json holds one record 35 commits and 5 published tags behind HEAD, with no stated ordering rule

1 participant

@thedavidmeister