fix(test): F3 skip-as-pass in test_x509.py + F7 negative KAT vectors

[JC-000]

Audit remediation, Lane C: F3 (a suite that reports success while running nothing) and F7 (a KAT oracle with no negative vector). Both VICE-only, no hardware. All runs use default_vice_config() (mandatory -reu).

Three commits: one per finding, plus a third for a second instance of F3 found in run_all_tests.py during review (see "F3, second door").

---

F3 — a missing label deleted a whole test group, and the suite still said "ALL 7 TESTS PASSED"

The defect, in one sentence: dropping the single symbol ecdsa_verify from build/labels.txt made tools/test_x509.py skip its entire ECDSA group, report ALL 7 TESTS PASSED, and exit 0 — the skipped assertions left the denominator instead of counting against it.

The realistic trigger is a libs/nistcurves bump renaming an export, which is exactly the change this suite exists to catch, and it is live risk now: the siblings are two minor versions ahead of the pin and a bump is being planned. While a skipped group can report success, every other guarantee in the suite is conditional on symbols a dependency bump can rename.

The fix

Every group the suite claims to run is declared in REQUIRED_LABEL_SETS and is required:

• main() fails closed in a preflight, before launching VICE — exit 1, and the summary line names the group and its missing labels.
• run_tests() counts each un-runnable group as a failure and names it. This matters independently of the preflight: tools/run_all_tests.py imports run_tests directly and only ever sees (passed, failed), so the return arity is deliberately unchanged and a vanished group now turns that runner red too.
• "No tests ran" is a failure rather than a shrug.
• CV_LABELS is deliberately not required — no group drives it, and a group that does not exist cannot be silently skipped.

Also fixed the one other skip-as-pass in the same file: a P-384 certificate that failed to parse reported SKIP: may not be supported yet and returned (0, 0), which would have hidden a P-384 DER regression completely. Now a failure. P-384 certificate parsing works today (group 2 measured green, 2/2); only P-384 ECDSA verify is stubbed at the TLS layer.

Acceptance — both directions, verbatim

Reproduced with the real script against a real VICE session, using a build/labels.txt with exactly the two .ecdsa_verify lines removed (ecdsa_verify_256 / _384 deliberately left intact — a rename of one export, not a demolition).

Direction A — ecdsa_verify absent from the labels view.

Before (baseline f0127a0, reproducing the audit finding):

 OK: 2/2 passed
SKIP: label 'ecdsa_verify' not found -- skipping test group
============================================================
RESULTS
============================================================
Passed: 7/7
Failed: 0/7
[+] X.509/ECDSA: ALL 7 TESTS PASSED
============================================================
EXIT_CODE=0

After:

 DER parser labels: OK
ECDSA verify labels: MISSING
CertificateVerify labels: OK (informational; no group uses these yet)
============================================================
RESULTS
============================================================
Passed: 0/0
Failed: 0/0
[-] X.509/ECDSA: ABORTED -- declared test group(s) cannot run: ECDSA P-256 verify (group 3) [missing: ecdsa_verify]
A missing label means the build stopped exporting a symbol
this suite depends on (e.g. a libs/nistcurves bump renamed
an export). Skipping the group would report success while
testing nothing, so this is a failure.
============================================================
EXIT_CODE=1

Direction B — labels intact.

 [3a] ECDSA verify: r=0 rejected (C=1, instant)
PASS: ecdsa_verify returned C=1 (r=0 rejected)
[3b] ECDSA verify: s=0 rejected (C=1, instant)
PASS: ecdsa_verify returned C=1 (s=0 rejected)
[3c] ECDSA verify: valid signature (C=0)
PASS: ecdsa_verify returned C=0 (valid) [60s]
[3d] ECDSA verify: tampered s (C=1)
PASS: ecdsa_verify returned C=1 (tampered rejected) [60s]
OK: 4/4 passed
============================================================
RESULTS
============================================================
Passed: 11/11
Failed: 0/11
[+] X.509/ECDSA: ALL 11 TESTS PASSED
============================================================
EXIT_CODE=0

The denominator moving 7 → 11 is the fix working: the ECDSA group now contributes its 4 assertions instead of evaporating.

The audit's ready-made probe at hi/e4_silent_skip.py (which calls run_tests() directly with a labels view that drops the symbol, rather than editing a file) was reused as a second regression check against the run_tests() backstop path.

---

F7 — the KAT oracle had no negative vector, so it could not fail

The defect, in one sentence:tools/test_ecdsa_kat_oracle.py ran three CAVP vectors, all Result = P, all expecting C=0 — against a verify forced to always report valid it reported 3/3 and exited 0.

An oracle with no negative case does not test verification; it tests that the routine returns.

The fix

Three genuine CAVP SigVer P-256/SHA-256 Result = F records, one per modification class:

subtest	CAVP record
[3h]	F (3 - S changed)
[3i]	F (2 - R changed)
[3j]	F (1 - Message changed)

Every Result = F record in the file has Q on the curve and r, s in [1, n-1], so none of these is rejectable by a cheap range or point-validity gate — each one forces the full verify math and the recovered-R.x comparison. (Confirmed by raw-int arithmetic, not by asking a library.)

Each vector now carries expect_carry, mirroring the .rsp Result column — never an observed C64 result. Two structural guards keep F7 from regressing: the run aborts if the vector set ever contains zero negatives, and fails if any declared vector does not produce a verdict.

Vector provenance — verified before adding, and again after transcription

A wrong vector, or one generated by running this implementation, would be a serious own goal. So:

1. Before adding anything, all 15 records in libs/nistcurves/tools/vectors/nist_p256_sigver.rsp were checked host-side against OpenSSL (via cryptography 48.0.0), plus independent raw-int on-curve and range checks. Zero disagreements with the .rsp Result column. This also re-confirmed the extraction convention: the 3 positives already in the file are records [3], [4], [14] and their hardcoded hashes equal SHA-256(Msg) exactly.
2. After transcription, each of the six vectors now in the file was matched byte-for-byte back to a .rsp record, its expect_carry checked against the Result column, and re-verified through OpenSSL as transcribed in the oracle (prehashed) — so a hand-transcription typo could not slip past. 0 problems.

No vector was produced by running this implementation.

Acceptance — both directions, verbatim

Pristine:

 [3h] ECDSA verify: CAVP SigVer P-256/SHA-256 invalid #1 (F: S changed) (expected C=1, INVALID)
PASS: ecdsa_verify returned C=1 (invalid) [60s]
[3i] ECDSA verify: CAVP SigVer P-256/SHA-256 invalid #2 (F: R changed) (expected C=1, INVALID)
PASS: ecdsa_verify returned C=1 (invalid) [60s]
[3j] ECDSA verify: CAVP SigVer P-256/SHA-256 invalid #3 (F: Message changed) (expected C=1, INVALID)
PASS: ecdsa_verify returned C=1 (invalid) [60s]
Passed: 6/6
Failed: 0/6
[+] ECDSA KAT oracle: ALL 6 VECTORS PASSED (3 valid accepted, 3 invalid rejected)
EXIT_CODE=0

Mutant — clc; rts at the top of ecdsa_verify (always report VALID, execute no crypto).

Per protocol, the mutant was proven present in the built binary, not merely in the source. ecdsa_verify was resolved from the built build/labels.txt to $6B23, mapped through the PRG load address $0801 to file offset 25378, and the bytes actually there decoded:

ecdsa_verify : $6B23 -> file offset 25378 ($6322)
bytes at label: 18 60 AD 73 B6 F0 03 4C EE 56 A9 74
decoded : CLC, RTS, LDA abs, $73
MUTANT CONFIRMED IN BINARY: ecdsa_verify begins CLC; RTS

 Passed: 3/6
Failed: 3/6
[-] ECDSA KAT oracle: 3 VECTOR(S) FAILED
EXIT_CODE=1

with, per failing vector:

 FAIL: ecdsa_verify returned C=0 (valid), expected C=1 (INVALID) [30s]
A CAVP Result=F vector was accepted. The verifier is reporting
signatures valid that NIST says are not.

The mutant's three passes are exactly the three vectors this file used to contain — the old oracle would have reported a clean 3/3 against a verifier that validates everything. That is F7 demonstrated directly rather than argued.

The mutation was reverted before commit; the rebuilt PRG hashes identical to the pristine build (db31111031e2f30c52c9116576d74f21e2cc6ddee025041a881345efa7ba8f60). No mutation is in this branch.

---

---

F3, second door — --skip-slow silently dropped the entire x509 suite

Found by Lane G while reviewing the F3 fix, and squarely the same defect one level up: tools/run_all_tests.py --skip-slow omitted x509 from the suite list by never adding it, so the aggregate printed a TOTAL and exited 0 with no trace that the whole X.509/ECDSA suite had not run.

The numbers make it concrete. A full aggregate reports TOTAL: 264/264 with x509 11/11; --skip-slow reported TOTAL: 253/253 — the same "everything passed" shape, 11 assertions lighter, and nothing saying which 11 or why. Given F3 is precisely about the x509 ECDSA group reporting success while running nothing, this was a second door into the same room.

Note on the absolute totals. The load-bearing quantity here is the 11-assertion gap (x509's contribution), which is invariant. The two totals it sits between are not: PR #81 adds 2 x25519 assertions, so on a merged tree the same demonstration reads 266/266 full and 255/255 with --skip-slow — same 11-assertion gap, same behaviour, different printed numbers. Lane G verified the merged tree directly (auto-merge clean, both lanes' changes present in one verdict). If you are reviewing after #81 lands and see 266/255 rather than 264/253, that is expected.

Skipped suites are now recorded as (name, reason) and surfaced three ways: named on the TOTAL line, given their own SKIP row, and followed by an explicit warning.

Acceptance (--skip-slow --workers 4, real aggregate runs both ways):

Before:

TOTAL: 253/253 passed, 0 failed
OK crypto 22/ 22 (0.5s)
...
OK x25519 71/ 71 (1.7s)
============================================================
EXIT_CODE=0

(no x509 row, no mention anywhere)

After:

TOTAL: 253/253 passed, 0 failed -- 1 suite(s) SKIPPED: x509
OK crypto 22/ 22 (0.5s)
...
OK x25519 71/ 71 (1.7s)
SKIP x509 --- did not run: --skip-slow (X.509 DER parsing + ECDSA P-256 verify)
WARNING: the suite(s) above did not run. This aggregate result does not
certify them, and their assertions are absent from the TOTAL.
============================================================
EXIT_CODE=0

Exit code deliberately stays 0. An operator passing --skip-slow made a choice; F3's missing label is an unrequested environmental failure. The rule the two cases share is the one that matters, and it is the rule this PR is really about:

An involuntary skip is a failure. An explicit skip is allowed, but must never be silent.

This matches the convention Lane G adopted for test_x25519.py's --fast gate in #81, so the two suites agree on the principle rather than diverging on the exit code.

---

Blast radius

Nothing turned red. Both fixed suites pass at HEAD (11/11 and 6/6), and the aggregate is unchanged at 253/253 with --skip-slow / 264/264 without (266/255 once #81 lands — see the note above).

run_tests() in test_x509.py keeps its 2-tuple arity deliberately, so the tools/run_all_tests.py:86 importer is unaffected. #81 changes test_x25519.run_tests to a 3-tuple and updates its own importer in the same file; the two PRs touch different hunks of run_all_tests.py (suite-list + summary block here, x25519 import branch there). Lane G has verified this rather than assumed it: git merge --no-commit --no-ff is clean, and the merged tree was then run both ways — TOTAL: 266/266 full and TOTAL: 255/255 passed, 0 failed -- 1 suite(s) SKIPPED: x509 with the flag, i.e. both lanes' changes present in a single verdict. Textual auto-merge and composing behaviour are separate claims; both were checked.

The only counter changes anywhere are denominators growing — 7→11 in test_x509.py, 3→6 in the KAT oracle — which is the intended effect: assertions that used to evaporate now show up in the denominator and get run.

The P-384-parse skip→fail conversion is the one place I converted a currently-passing skip into a potential failure. It is green today (group 2, 2/2, observed in three separate runs).

Audit of the other tools/test_*.py for the same shape

The check_label*(...) → print SKIP → return 0, 0 idiom is duplicated across the suite; 26 return 0, 0 skip sites total. Findings, with what I did about each:

file	sites	status
test_x509.py	5	fixed here
test_ecdsa_kat_oracle.py	1	fixed here (guarded by the main() fatal + the new total-verdict check)
test_tls_handshake.py	11	not mine — see F10 below
test_tls_record.py	6	Lane A's F1 file; left alone
test_tls_p384_negotiation.py	2	follow-up; P-384 is a known-broken area (all P-384 build targets fail at the v0.6.0 pin), so making it fail-closed would add noise, not signal
test_http.py	1	benign — a static len(response) > 256 on a fixed ~160-byte response; dead branch

These skips are dormant, not live: every label they guard resolves against the current build. That is precisely the F3 risk profile — one upstream rename away from silently deleting a group while the suite still reports success.

Two findings worth acting on separately

1. A second site of Lane A's dead-register-read defect (routed as F10). The coordinator asked me to check whether my files read the processor flags via a "P" key. They do not, and are immune by construction: test_x509.py and test_ecdsa_kat_oracle.py capture carry with a 6502 trampoline (LDA #$00 / ROL A / STA $0352) — the CPU materialises the carry into memory and the host reads it with read_bytes. No register dict is consulted anywhere in either file, and there is no weaker fallback: jsr_with_carry either returns a real 0/1 or raises, and every caller counts the exception as FAIL.

But tools/test_tls_handshake.py:625-627 has the identical defect:

carry=0ifregsand"P"inregs:
carry=regs["P"] &0x01

jsr() returns read_registers(), whose keys come from VICE's monitor register-name table where the status register is FL (c64-test-harness backends/vice_binary.py:1325 resolves FL/FLAGS/SR). "P" is never a key, so carry is pinned at 0 and the "parser returned C=1" branch is unreachable.

The interaction is the part that matters: latent behind that branch, at :654-657, is an F3-shaped silent skip (return 0, 0 when the routine is a clc; rts stub). It is unreachable only because the "P" bug pins carry to 0 — so fixing "P" → FL alone would trade a dead assertion for a vanished test group. The two must land together. Filed and routed to Lane A as F10 on a separate branch; not touched here, to keep the ownership boundary clean.

2. An entire suite that reports success without running, on the primary dev platform.tools/test_http_integration.py:91 gates on /sys/class/net/tap-c64 — a Linux sysfs path — and then sys.exit(0). Measured on macOS, which is this project's documented dev/CI platform:

$ python3 tools/test_http_integration.py
SKIP: tap-c64 interface not found
EXIT_CODE=0

A whole integration suite, permanently green in under a second. It looks superseded by the macOS feth rig (tools/rig-up-macos.sh), so the right fix is probably deletion or a platform-appropriate rig rather than a mechanical patch — listed as follow-up, not fixed here, since that is a judgement call about the suite's future rather than a small change.

3. Cross-referenced instance from Lane G (PR #81), same class, different mechanism.tools/test_x25519.py gates its only two end-to-end x25519_scalarmult vectors behind --slow; the default run prints RESULTS: 71/71 passed, 0/71 failed and exits 0, with the summary never mentioning the skip — all 71 assertions are field arithmetic. Vector 2 (U_2 ending 0x93, bit 255 set) is exactly the vector that catches the upstream c64-x25519 #64 MSB bug present in the pinned v0.6.0 — so the skipped test was the regression gate for the bump this audit is meant to de-risk. Lane G then timed it properly: 37.9 s with both vectors vs 4.8 s without, ~16.5 s per vector — the in-file "~100 min each" comment is off by a factor of ~360. With no real cost to trade against, they made the vectors run by default (--fast to skip) and named the skip in the verdict. Fixed in #81, not here, to avoid two PRs editing one file; noted because it shows the shape recurs across suites rather than being a test_x509 quirk.

Reproducing

A fresh worktree needs three steps, not two:

git submodule update --init --recursive
make ip65-libs # <-- mandatory on a fresh clone, and undocumented as such
make

Skip the middle step and make dies with ld65: Error: Input file '../ip65/ip65/ip65_tcp.lib' not found. The ip65-build/ip65-c64.bin blob is gitignored (.gitignore:6), so a fresh tree has none and must build one; the submodule ships ip65 sources, and make ip65-libs is what compiles them into the .lib archives the blob link consumes.

The ip65 artifact chain is reproducible from source — verified, not assumed. The blob circulating in existing clones (built 2026-05-06) and a fresh from-source rebuild against the pinned submodule 25a9c5aa are byte-identical:

copied blob 6,951 B sha256 cf1a5ff7809af4e4655e385b378b936054f41046ff2b7604828af3240c2d90dd
from-source build 6,951 B sha256 cf1a5ff7809af4e4655e385b378b936054f41046ff2b7604828af3240c2d90dd
cmp -> BYTE-IDENTICAL

Lane G reproduced the same hash independently in a separate worktree, so that is three agreeing builds. And the PRG built through the fully-from-source chain hashes db31111031e2f30c52c9116576d74f21e2cc6ddee025041a881345efa7ba8f60 at 47,105 B — identical to the binary every acceptance run in this PR was produced against, and matching the size recorded for the post-#68 refit.

Two CLAUDE.md Build-section errors surfaced while establishing the above. Routed to Lane E (their F8/F9 territory) rather than fixed here, with Lane G supplying the full reproduction:

• :29 calls this "the committed blob" — it is gitignored and untracked (git ls-files ip65-build/ returns only ip65.cfg and ip65_stub.s).
• :27 says make ip65-libs is "only needed if the ip65 submodule changes" — it is needed on every fresh clone.

Individually wrong; together worse, because they reinforce. One says no rebuild is needed, the other says the build step is conditional, and a fresh-clone reader following both is steered exactly past the only step that works. Two lanes rediscovered this by tripping over it. It is the same shape as F8, and a good example of the general point: this is an assertion about make's behaviour, stored somewhere make cannot reach.

🤖 Generated with Claude Code

[JC-000]

Independently verified by the supervising session.

F7 vectors — the thing most worth double-checking. I re-derived all six against OpenSSL from the branch's own KAT_VECTORS table, mapping expect_carry == 0 to "valid":

AGREE openssl=VALID expect_carry=0 CAVP SigVer P-256/SHA-256 valid #1 / #2 / #3
AGREE openssl=INVALID expect_carry=1 invalid #1 (F: S changed) / #2 (F: R changed) / #3 (F: Message changed)
6/6 agree with OpenSSL

Genuine CAVP records, correctly transcribed. (My first pass reported three disagreements — that was my harness reading a expect_valid key that does not exist in this table, not a defect here.)

F3 preflight, reproduced by deleting the .ecdsa_verify line from a copy of build/labels.txt:

ECDSA verify labels: MISSING
Passed: 0/0 Failed: 0/0
[-] X.509/ECDSA: ABORTED -- declared test group(s) cannot run: ECDSA P-256 verify (group 3) [missing: ecdsa_verify]
exit=1

Failing in a preflight before VICE launches is better than failing during the run — it costs nothing and the message names exactly what is missing.

Two design calls worth endorsing explicitly. Keeping the (passed, failed) arity because run_all_tests.py imports run_tests() directly — changing the signature would have left the aggregate runner green while the individual script went red, which is the same defect one level up. And the observation that the mutant's three surviving passes are exactly the three vectors the file used to contain: the old oracle would have reported a clean 3/3 against a verifier that accepts everything. That is the finding in one line.

On urgency, tempered by Lane F: the nistcurves v0.8.0 bump would NOT have fired the rename scenario — bare LIB_* exports are still emitted unless a consumer opts into LIB_NO_BARE_EXPORTS=1. This fix is prudent rather than urgent on current evidence, which is a better reason to merge it than the one I gave when I assigned it.

test_http_integration.py gating on a Linux-only path and sys.exit(0)-ing on macOS is a real find and correctly scoped out — deletion versus a new rig is a judgement call, not a patch.

Confirms audit findings F3 and F7.

[JC-000]

Cross-PR note before review — the totals cited in this PR body go stale once #81 lands.

#81 adds two x25519 assertions (the RFC 7748 vectors it un-gates), so on the merged tree:

this PR's body	merged with #81
TOTAL: 264/264 (full)	266/266
TOTAL: 253/253 (--skip-slow)	255/255

Behaviour is identical and the demonstration still holds — but this PR's argument asks the reader to notice an 11-assertion gap between two totals, and a reviewer running the merged tree would see neither figure. Worth updating the body so the numbers match what the reviewer actually gets, rather than leaving them to doubt the demonstration instead of the arithmetic. A slightly ironic failure mode for a PR about misleading output.

Verified on a merged probe: Automatic merge went well, and both lanes' changes surface in one verdict —

TOTAL: 255/255 passed, 0 failed -- 1 suite(s) SKIPPED: x509
OK x25519 73/ 73
SKIP x509 --- did not run: --skip-slow (X.509 DER parsing + ECDSA P-256 verify)
WARNING: the suite(s) above did not run...

So the run_all_tests.py hunks compose behaviourally, not just textually — which is the confirmation this PR asked whoever landed second to perform.

[JC-000]

The withdrawn "byte-identical to a rebuild" claim is now backed by evidence — settled by the supervising session using the rebuild Lane G completed.

Withdrawing it was right: the rebuild path fails in a fresh worktree without the make ip65-libs prerequisite, so the comparison genuinely could not be made from this lane. Lane G got past that step and produced the rebuilt artifact, which makes the comparison possible after the fact:

main clone ip65-build/ip65-c64.bin 6951 bytes
sha256 cf1a5ff7809af4e4655e385b378b936054f41046ff2b7604828af3240c2d90dd
rebuild from the pinned ip65 submodule 6951 bytes
sha256 cf1a5ff7... (identical)

So the claim was true, and is now checkable. The blob in circulation is reproducible from the pinned submodule — worth recording because the alternative would have been materially worse than a documentation bug: an ip65 artifact nobody could regenerate from the pin, which would have put every ip65 result in this audit in question.

The body can stay as it is — stating only what was verified is never wrong — but if you would rather restore the claim, the sha above is the evidence for it.