fix(test/uci): make the HTTPS e2e oracle fail when it should (audit F5 + F6) - #82
Conversation
…it F5) test_https_local.py passed the run whenever the 5 characters "HELLO" appeared anywhere in the 1000 bytes of screen RAM. That branch was reachable only after the full-body assertion had already failed, so the test substituted a weaker criterion at exactly the moment the strong one did not hold: a truncated body, a mis-decrypted body that kept its first word, or a stale HELLO left on screen by an earlier run all passed. The C64-side criteria now live in `_check_c64_result()` — one rule, that http_resp_buf holds the complete expected body. Screen RAM stays diagnostic: a HELLO on screen without the body is reported as one of the reasons the run failed, not as a pass. A/B (host-side, same four inputs through the pre-fix block and the new function): input before after full body in resp_buf, HELLO on screen PASS PASS body truncated, HELLO still on screen PASS FAIL resp_buf empty, stale HELLO on screen PASS FAIL nothing anywhere FAIL FAIL Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dit F6) Every run already wrote server_result.json — whether the listener completed the handshake, the decrypted request, any TLS error — and no pass criterion ever read a byte of it. All assertions were against C64-side memory and screen RAM, i.e. against state the client produces itself. The listener's record is the one piece of evidence in this test the client cannot fabricate, and it was archived and discarded. A run now passes only when both sides agree. `_check_server_result()` fails on: a recorded `error`, `listening` unset, `client_addr` absent (the listener never accepted a connection), and a `request` that is missing, `<timeout>`, or not the GET this script programmed into the C64 (request line + `Host:` line — both values this script chose). Deliberately not asserted, to avoid failing healthy runs: byte-exact request equality (trailing headers are src/http.s's business), and anything at all under EXTERNAL_LISTENER=1, where by contract there is no inline listener and server_result is empty — that skip is printed. `--check-artifact <run-dir>` re-runs the server-side criteria against an archived run's server_result.json with no hardware, so the criteria are testable and stored runs can be re-adjudicated. A/B over the five captured runs under the audit scratchpad, each composed with a healthy C64 side (http_status=200, the full body) so the "C64 memory looks right" direction is the one under test: server_result fixture before after 214751 clean (listening/addr/GET) PASS PASS 215648 clean (listening/addr/GET) PASS PASS 215331 SSLEOFError UNEXPECTED_EOF PASS FAIL 220559 SSLEOFError UNEXPECTED_EOF PASS FAIL 215221 listening only, never accepted PASS FAIL Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
JC-000
commented
Aug 13, 2026
Independently reproduced by the supervising session, using the
Both failing fixtures passed under the baseline criteria, because the C64-side memory looked healthy in each — which is precisely the gap F6 described. Three judgement calls here that I think are right: Asserting the request line and Deleting the screen-RAM fallback rather than strengthening it, and re-purposing Failing closed on a missing or corrupt The hardware confirmation matters as much as the negative cases: PASS in 133.5 s against a ~136 s baseline shows the added criteria cost nothing on a genuine run, so this does not trade false passes for false failures. Noted for follow-up, correctly left alone here: Confirms audit findings F5 and F6. |
Lane D of the audit remediation. Two findings, both in
tools/uci/test_https_local.py'spass criteria. One commit each; no production code touched — this changes only what the
e2e oracle is willing to call a pass.
F5 — the screen-RAM
HELLOfallbackDefect. The run passed whenever the five characters
HELLOappeared anywhere in the1000 bytes of screen RAM, and that branch was reachable only after the full-body
assertion had already failed — the test substituted a weaker criterion at exactly the
moment the strong one did not hold.
Fix. The fallback is deleted, not replaced (same standard Lane A applied in #78: an
assertion that cannot be evaluated is a failure, not a pass). C64-side criteria now live in
_check_c64_result()with one rule —http_resp_bufcontains the complete expected body.Screen RAM stays diagnostic: a
HELLOon screen without the body is now printed as one ofthe reasons the run failed.
Acceptance A/B — the four inputs run through the pre-fix block (transcribed verbatim
from
origin/docs/benchmark-refresh:tools/uci/test_https_local.py:1391-1408) and throughthe new function:
Both directions: the F5 case (body absent,
HELLOon screen) flips PASS → FAIL, and thegenuine-body case stays PASS.
F6 — the server's record was archived and discarded
Defect. Every run wrote
server_result.json— whether the listener completed thehandshake, the decrypted request, any TLS error — and no pass criterion ever read a byte of
it. Every assertion was against C64-side memory and screen RAM, i.e. against state the
client produces itself. The listener's record is the one piece of evidence in this test the
client cannot fabricate.
Fix. A run now passes only when both sides agree.
_check_server_result()is calledbefore the verdict;
--check-artifact <run-dir>re-runs the same criteria against anarchived run with no hardware, so the criteria are testable and stored runs can be
re-adjudicated.
Which fields are load-bearing, and why not more
Asserted:
errorSSLEOFError: UNEXPECTED_EOF_WHILE_READINGdid not complete a TLS session with this listener, whatever the C64's RAM says afterwards.listeningaccept(). Absent ⇒ there was no server for the C64 to have talked to.client_addrhttp_resp_bufis stale, not this run's evidence.requestDeliberately not asserted, because over-strict criteria that flake on healthy runs would be
worse than the gap:
GET / HTTP/1.) and theHost:line are checked. Both are values this script programs into the C64, so they cannot
drift accidentally, whereas the trailing
Connection: closeissrc/http.s's business —a future header change should not fail the e2e oracle. A short/partial
recvalso stillsatisfies a prefix check.
EXTERNAL_LISTENER=1. There is no inline listener in that mode,server_resultis empty by construction, and the documented contract is that passcriteria come from C64 state only. The skip is printed rather than silent.
request == b"<timeout>"is treated as a failure, not as an absent field: it means thesession never carried the GET.
Fail-closed.
listening,client_addrandrequestare all required, so an emptyrecord — thread never started, crashed before recording anything — fails
(
_check_server_result({})returns three problems). In the offline checker a missing,unreadable, or non-object
server_result.jsonexits non-zero with an explicit messagerather than being read as "no error recorded" — the same defect shape as the fallbacks
themselves, one layer up.
Acceptance A/B
Every captured run under the audit scratchpad, each composed with a healthy C64 side
(
http_status=200, full body) so that "C64-side memory looks right" is precisely thecondition under test. BEFORE = baseline criteria (C64-side only,
server_resultunread):The same verdicts through the shipped CLI, e.g.
Hardware confirmation — a genuine pass still passes
Verified, not assumed: full e2e on the U64E at 10.43.23.81 (UCI/REU profile, 48 MHz,
inline listener, this branch at 4d6907c). 133.5 s, exit 0, both criteria met:
run_info.txt:outcome = PASS,duration_s = 133.531,exit_code = 0,turbo_mhz = 48— in line with the ~136 s baseline runs, so the added criteria cost nothing. Feeding that
run's own artifact back through the new offline checker also passes, closing the loop:
Blast radius
tools/uci/test_https_local.pychanges; nosrc/code, no build inputs, so everyPRG and benchmark is byte-identical.
tools/uci/test_https_local_p384.pycarries the same fallback and the same unreadserver_result, but it is unrunnable at HEAD (the P-384 build is broken at the v0.6.0pin), so it is left alone rather than changed blind. Worth folding in when P-384
enablement resumes.
EXTERNAL_LISTENER=1runs — the packaging validation matrix in CLAUDE.md — keep theirprevious behaviour exactly: F5's tightening applies (they always asserted the full body
in practice), F6's server-side block is skipped by contract and prints that it was.
Convergence with Lane B's
tools/https_e2e/evil_listener.py: its result dict uses the samefour base keys (
listening,client_addr,request,error) with the same meanings, andadds handshake-stage detail (
client_hello_seen,server_flight_sent,client_accepted_finished,client_finished_valid,client_reaction,response_sent)._check_server_result()keys only on the shared subset and ignores unknown keys, so italready reads an evil-listener result correctly; the richer fields are the right basis for a
stricter checker in that lane's own test, without either lane having to move first.
Environment notes for a fresh worktree:
git submodule update --init --recursive, andtools/https_e2e/certs/server.{pem,key}are gitignored — auto-generated by the listener, orcopied from an existing clone. (This lane builds
BACKEND=uci, so the gitignoredip65-build/ip65-c64.binnever came into it.)