From 2ca0280cd852b253cd9068272ecdfb21f5da89b5 Mon Sep 17 00:00:00 2001 From: GenericJam Date: Sat, 5 Sep 2026 23:22:09 -0600 Subject: [PATCH] docs(agents): add a verification fidelity ladder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tuned for a repo whose failures are mostly failures to notice that nothing happened: rung 3 is running the task against a real project, rung 4 is proving the deployed app answers rather than trusting the deploy tool, and rung 6 is compiling from the packed tarball because Hex omits root dotfiles. Insider (GenericJam/insider) gates agent work on a named fidelity ladder and a rule that a lower rung may not stand in for a higher one. The principle was already here in pieces — "verify effects, not exit codes" in the agentic guide, the fast/slow split in .githooks/pre-push — but there was no named ladder an agent could point at, and so no vocabulary for "I verified at rung 3, rung 4 was unavailable." Every rung cites something that actually got through the rung below it, so the list reads as history rather than process. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index a5996b5..8358cef 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,6 +31,47 @@ mix test # all tests mix test --exclude integration # skip the device-dependent ones ``` +## Verification fidelity ladder + +This repo builds and deploys other people's apps, so its failures are mostly +failures to notice that nothing happened. Run every applicable lower rung, plus +the highest rung the change actually reaches, and say which rung you stopped at. + +1. **Static.** `mix format --check-formatted`, `mix credo --strict` (ex_slop + included), `mix compile --warnings-as-errors`. +2. **Host unit.** `mix test`. Proves the task's logic against fixtures. Proves + nothing about a real toolchain. +3. **Task run against a real project.** Actually invoke the task on a generated + app and read the output. A task that unit-tests green can still skip its real + work — a missing toolchain or a mismatched device id once produced a clean + exit-0 from a deploy that never built anything, and cost several hours of + debugging failures that had not happened. +4. **Deployed, and the app answers.** `mix mob.deploy` to a simulator or + emulator, then attach and confirm the BEAM is up and a screen renders. + Deploy success is the tool's opinion; a reachable node is evidence. +5. **Physical device, and the release variant.** Release changes linkage and + packaging: iOS release links plugin NIFs by a separate path + (`decisions/2026-07-07-ios-release-links-plugin-nifs.md`), and release + `otp.zip` handling is variant-scoped + (`decisions/2026-07-24-release-otp-zip-variant-scoped-assets.md`). A release + also leaves an `assets/otp.zip` that crash-loops the next debug deploy. +6. **From the packed Hex tarball, not the working tree.** Hex omits + repository-root dotfiles, so code under `lib/` that reads one compiles here + and fails for everyone else. Two releases shipped broken this way. Build the + package and compile from it before publishing. + +Every rung above exists because something got through the one below it. + +Two rules that outrank the list: + +- **Never substitute a lower rung because a higher one is slow, broken, or + inconvenient.** Fix the harness, open an issue, or state plainly that the rung + was unavailable and why. An unavailable rung is a fine answer. A silently + skipped one is not. +- **Verify effects, not exit codes.** This repo is where that rule was learned + and it is the repo most able to break it: every task here should prove its + effect happened rather than reporting that it returned. + ## Things that bite specifically in mob_dev - **Compile-time regex literals are unsafe** on Elixir 1.19 / OTP 28.0. Use