fix(package): close the vacuous pass in the release gate (#98 landed only 1 of 4 commits) - #101
Conversation
…t not pass
LaneCerts found the sharper version of the bug I was fixing: at master,
`SKIP_REBUILD=1 SKIP_VICE=1 make package-verify` printed
2/2 checks passed
RELEASE ARTIFACTS VERIFIED
on a dist/ with half its PRGs missing and no disk images or manifest at
all. Reproduced here, and it is worse than reported — with the build
record present but dist/ emptied, the gate printed
0/0 checks passed
RELEASE ARTIFACTS VERIFIED exit 0
A gate that verified literally nothing reported success. My earlier
commit on this branch did not close it: it only caught variants that
FAILED to build, not artifacts that were never produced.
Cause: both disk checks iterated `DIST.glob("*.d64")`. No disks meant no
iterations, no records, nothing failed. Absence of evidence read as
evidence of correctness — which is the exact failure mode this repo's
own guidance warns about.
- expected_d64_images() derives the images that MUST exist from the
build record (one per built variant, one per backend). Both disk
checks iterate that, and record a FAIL for any expected image that
is absent, plus one for stray images no variant accounts for.
build_prgs.sh now records `backend=` per variant so this derivation
needs no second copy of the matrix.
- A run that records zero checks fails outright.
- "RELEASE ARTIFACTS VERIFIED" is now reserved for a run where every
section executed. Any SKIP_* yields "PARTIAL VERIFICATION — ... Not
a release gate", still exit 0 so SKIP_* stays usable for narrowing.
Measured, all four cases:
A built PRGs, disks deleted 0/4, four "expected but absent", exit 1
B make package at the broken pin partial artifacts + blocker, exit 1
C partial dist, disks present 6/6 present verify, 2 variants
missing, RELEASE INCOMPLETE, exit 1
D all-OK record, SKIP_* set 4/4, PARTIAL VERIFICATION, exit 0
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Adds tools/test_package_verify.py — 31 assertions over the gate's verdict logic and its coverage derivation. No VICE, no builds, milliseconds. summarize() is split out of main() to make it testable, and because the ordering inside it IS the contract: zero checks fails before the friendly skip branch can rescue it, failures outrank the missing-variant note, and "RELEASE ARTIFACTS VERIFIED" is reachable only from a complete run. The case guarded hardest is SKIP_* with everything passing (suggested by LaneCerts): it is the invocation people actually use while iterating, so it is where the pressure to just say VERIFIED will come back. It must stay exit 0 to remain usable while never producing the word VERIFIED. The subtlest test is that skipping every section on an empty dist/ must NOT downgrade to a cheerful PARTIAL — that is the original bug wearing a different hat. Also corrects an overstatement in this branch's PR description. On a genuinely clean tree master's gate does not go green: with no listener bundle it reports 0/1 exit 1, because check_listener asserts the bundle's presence explicitly rather than iterating. The vacuous pass needed the bundle to be present: no disks, no bundle SKIP_REBUILD SKIP_VICE 0/1 exit 1 no disks, bundle present SKIP_REBUILD SKIP_VICE 2/2 VERIFIED exit 0 no disks, bundle present + SKIP_LISTENER 0/0 VERIFIED exit 0 The real defect is that inconsistency — the disk checks were globs, the listener check was an assertion, and the gate's correctness rode on which style each check happened to use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A regression test nobody runs is not a guard. tools/test_package_verify.py now runs as a preflight to `make package-verify`. The ordering is the point: verify_release.py shipped with a bug where it reported RELEASE ARTIFACTS VERIFIED having checked nothing, so "the gate said yes" is only worth something if the gate's yes still means what it should. A broken verdict function now stops the run instead of blessing a release. Costs milliseconds; needs no VICE and no build. Not added to tools/run_all_tests.py: that runner allocates a VICE instance per suite and dispatches run_tests(transport, labels, seed), a shape a pure-logic test does not fit. The gate is the right home. Measured: preflight 31 passed, then the gate reports RELEASE INCOMPLETE against the current broken pin, exit 1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
JC-000
commented
Aug 14, 2026
Independent confirmation from the packaging lane, plus one thing to check at merge time. I arrived here from the other direction — pushed this branch, tried to open a PR, and found #101 already filed with exactly these three commits ( I made one factual correction to the description above: the commits were pushed, not stranded unpushed. Reproduced on merged master before applying anythingAt An all-OK build record, zero artifacts on disk, green gate, exit 0. Identical setup with this PR applied: Preflight: Please verify the merge carries all threeThis is the whole reason the PR exists, so it would be an unusually bad one to repeat: git log --oneline origin/master | head -4 # expect 061f0ab, 2b02390, 68c61c8
git cat-file -e origin/master:tools/test_package_verify.py &&echo present
git show origin/master:tools/package/verify_release.py | grep -c expected_d64_imagesIf the last two don't come back clean, the merge dropped commits again. Unrelated blocker, still open
|
Recovers three commits that #98 was supposed to carry.#98 merged with only its first commit (
c1bc429, partial-matrix handling); the actual fix, its tests, and the preflight wiring never reached master.(Correction to an earlier draft of this description: the three commits were pushed, not stranded unpushed —
origin/fix/package-partial-failurecarried all four at87f501b. They were pushed to the branch after the PR was opened and arrived after the merge at15:00:33Z, sogh pr view 98 --json commitsshows the merge saw onlyc1bc429. The distinction matters for the lesson: every individualgit pushreported success, which is exactly why "pushed" got mistaken for "merged" in the sign-off.)Verified on
origin/masterbefore filing:This is worse than a fix that plainly didn't land, because the half that did land is the reporting half.
make package-verifynow presents as more trustworthy than before while the hole underneath is untouched, and #98's title states the problem is fixed. The new BLOCKER section also masks the defect whenever the build record has FAILED rows, so a casual run looks healthy.Reproduced on master with the disk checks isolated — build record rewritten all-OK, no artifacts present,
SKIP_LISTENER=1:An all-OK record, zero artifacts, green gate.
The three commits
061f0ab— coverage derived from the build record rather thanglob("*.d64"), so absent images fail instead of contributing zero checks; any zero-check run fails; and the wordVERIFIEDis reserved for runs where every section actually executed.2b02390—tools/test_package_verify.py, 31 assertions pinning the verdict logic, no VICE and no builds required.summarize()was made pure so the verdict is testable at all.68c61c8— those tests wired as a preflight tomake package-verify, because a regression test nobody runs is the same defect class it guards against.Verified here
The underlying defect is worth stating plainly, since it is subtler than "the gate passes vacuously": the disk checks globbed while the listener check asserted, so whether an empty run read green depended on which style the one surviving check happened to use. A correctness property riding on an unrelated check's defensive style stays invisible until that check is skipped — precisely when someone is narrowing a failure and least likely to be suspicious.
Found by the certs lane, which noticed a sign-off saying "#98 open with four commits" disagreed with a query saying MERGED, and resolved the mismatch by measuring rather than assuming one side was stale.
🤖 Generated with Claude Code