Skip to content

fix(test/uci): make the HTTPS e2e oracle fail when it should (audit F5 + F6) - #82

Merged
JC-000 merged 2 commits into
docs/benchmark-refreshfrom
fix/audit-f5-f6-e2e-oracle
Aug 13, 2026
Merged

fix(test/uci): make the HTTPS e2e oracle fail when it should (audit F5 + F6)#82
JC-000 merged 2 commits into
docs/benchmark-refreshfrom
fix/audit-f5-f6-e2e-oracle

Conversation

@JC-000

Copy link
Copy Markdown
Owner

Lane D of the audit remediation. Two findings, both in tools/uci/test_https_local.py's
pass criteria. One commit each; no production code touched — this changes only what the
e2e oracle is willing to call a pass.


F5 — the screen-RAM HELLO fallback

Defect. The run passed whenever the five characters HELLO appeared anywhere in the
1000 bytes of screen RAM, and that branch was reachable only after the full-body
assertion had already failed — the test substituted a weaker criterion at exactly the
moment the strong one did not hold.

Fix. The fallback is deleted, not replaced (same standard Lane A applied in #78: an
assertion that cannot be evaluated is a failure, not a pass). C64-side criteria now live in
_check_c64_result() with one rule — http_resp_buf contains the complete expected body.
Screen RAM stays diagnostic: a HELLO on screen without the body is now printed as one of
the reasons the run failed.

Acceptance A/B — the four inputs run through the pre-fix block (transcribed verbatim
from origin/docs/benchmark-refresh:tools/uci/test_https_local.py:1391-1408) and through
the new function:

=== BEFORE (baseline block) ===
PASS genuine pass: full body in resp_buf, HELLO on screen (spec wants PASS)
PASS F5 case: body truncated, HELLO still on screen (spec wants FAIL)
PASS F5 case: resp_buf empty, stale HELLO on screen (spec wants FAIL)
FAIL plain failure: nothing anywhere (spec wants FAIL)
=== AFTER (this branch) ===
[ok ] genuine pass: full body in resp_buf, HELLO on screen
want=PASS got=PASS
[ok ] F5 case: body truncated, HELLO still on screen
want=FAIL got=FAIL
- http_resp_buf does not contain the expected body 'HELLO FROM TLS SERVER' (got 'HTTP/1.0 200 OK\r\n\r\nHELLO FROM TLS')
- screen RAM contains 'HELLO' but that is not a pass — only the complete body in http_resp_buf counts (audit F5)
[ok ] F5 case: resp_buf empty, stale HELLO on screen
want=FAIL got=FAIL
- http_resp_buf does not contain the expected body 'HELLO FROM TLS SERVER' (got '')
- screen RAM contains 'HELLO' but that is not a pass — only the complete body in http_resp_buf counts (audit F5)
[ok ] plain failure: nothing anywhere
want=FAIL got=FAIL
- http_resp_buf does not contain the expected body 'HELLO FROM TLS SERVER' (got '')
F5 A/B: as specified

Both directions: the F5 case (body absent, HELLO on screen) flips PASS → FAIL, and the
genuine-body case stays PASS.


F6 — the server's record was archived and discarded

Defect. Every run wrote server_result.json — whether the listener completed the
handshake, the decrypted request, any TLS error — and no pass criterion ever read a byte of
it. Every assertion was against C64-side memory and screen RAM, i.e. against state the
client produces itself. The listener's record is the one piece of evidence in this test the
client cannot fabricate.

Fix. A run now passes only when both sides agree. _check_server_result() is called
before the verdict; --check-artifact <run-dir> re-runs the same criteria against an
archived run with no hardware, so the criteria are testable and stored runs can be
re-adjudicated.

Which fields are load-bearing, and why not more

Asserted:

fieldwhy it is evidence
errorDecisive. Set when the TLS handshake or socket failed server-side. A run that reached SSLEOFError: UNEXPECTED_EOF_WHILE_READING did not complete a TLS session with this listener, whatever the C64's RAM says afterwards.
listeningThe listener reached accept(). Absent ⇒ there was no server for the C64 to have talked to.
client_addrThe listener accepted a connection. Absent ⇒ the C64 never reached this listener, so anything in http_resp_buf is stale, not this run's evidence.
requestThe strongest link between the two sides: the server can only produce these plaintext bytes by having completed the handshake and derived the same application keys.

Deliberately not asserted, because over-strict criteria that flake on healthy runs would be
worse than the gap:

  • Byte-exact request equality. Only the request line (GET / HTTP/1.) and the Host:
    line are checked. Both are values this script programs into the C64, so they cannot
    drift accidentally, whereas the trailing Connection: close is src/http.s's business —
    a future header change should not fail the e2e oracle. A short/partial recv also still
    satisfies a prefix check.
  • Anything at all under EXTERNAL_LISTENER=1. There is no inline listener in that mode,
    server_result is empty by construction, and the documented contract is that pass
    criteria come from C64 state only. The skip is printed rather than silent.
  • request == b"<timeout>" is treated as a failure, not as an absent field: it means the
    session never carried the GET.

Fail-closed.listening, client_addr and request are all required, so an empty
record — thread never started, crashed before recording anything — fails
(_check_server_result({}) returns three problems). In the offline checker a missing,
unreadable, or non-object server_result.json exits non-zero with an explicit message
rather than being read as "no error recorded" — the same defect shape as the fallbacks
themselves, one layer up.

Acceptance A/B

Every captured run under the audit scratchpad, each composed with a healthy C64 side
(http_status=200, full body) so that "C64-side memory looks right" is precisely the
condition under test. BEFORE = baseline criteria (C64-side only, server_result unread):

[ok ] genuine pass (fixture 214751, real C64 body)
before=PASS after=PASS (spec wants after=PASS)
[ok ] genuine pass (fixture 215648, real C64 body)
before=PASS after=PASS (spec wants after=PASS)
[ok ] F6: server SSLEOFError, C64 memory looks right (215331 server side)
before=PASS after=FAIL (spec wants after=FAIL)
- listener recorded an error: TLS handshake failed: SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1032)
- listener recorded no decrypted request — the TLS session never carried the GET
[ok ] F6: server SSLEOFError, C64 memory looks right (220559 server side)
before=PASS after=FAIL (spec wants after=FAIL)
- listener recorded an error: TLS handshake failed: SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1032)
- listener recorded no decrypted request — the TLS session never carried the GET
[ok ] F6: listener never accepted, C64 memory looks right (215221)
before=PASS after=FAIL (spec wants after=FAIL)
- listener never accepted a connection (`client_addr` absent) — the C64 did not reach this listener
- listener recorded no decrypted request — the TLS session never carried the GET
F6 A/B: as specified

The same verdicts through the shipped CLI, e.g.

$ ./test_https_local.py --check-artifact <2026-08-12T21:53:31 run> --host 10.43.23.85
FAIL: server-side criteria not met:
- listener recorded an error: TLS handshake failed: SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1032)
- listener recorded no decrypted request — the TLS session never carried the GET
exit=1
$ ./test_https_local.py --check-artifact <2026-08-12T21:47:51 run> --host 10.43.23.85
request : b'GET / HTTP/1.1\r\nHost: 10.43.23.85\r\nConnection: close\r\n\r\n'
PASS: server-side criteria met
exit=0

Hardware confirmation — a genuine pass still passes

Verified, not assumed: full e2e on the U64E at 10.43.23.81 (UCI/REU profile, 48 MHz,
inline listener, this branch at 4d6907c). 133.5 s, exit 0, both criteria met:

http_get carry = 0 (0=success, 1=failure)
http_status = 200
http_resp_len = 21
http_resp_buf = b'HELLO FROM TLS SERVER'
--- server-side ---
listening : True
client_addr : ('10.43.23.81', 52434)
request : b'GET / HTTP/1.1\r\nHost: 10.43.23.85\r\nConnection: close\r\n\r\n'
error : <none>
Server-side check: listener completed the handshake and decrypted the expected GET
PASS: http_resp_buf contains 'HELLO FROM TLS SERVER'
exit=0

run_info.txt: outcome = PASS, duration_s = 133.531, exit_code = 0, turbo_mhz = 48
— in line with the ~136 s baseline runs, so the added criteria cost nothing. Feeding that
run's own artifact back through the new offline checker also passes, closing the loop:

$ ./test_https_local.py --check-artifact <this run> --host 10.43.23.85
PASS: server-side criteria met
exit=0

Blast radius

  • Only tools/uci/test_https_local.py changes; no src/ code, no build inputs, so every
    PRG and benchmark is byte-identical.
  • tools/uci/test_https_local_p384.py carries the same fallback and the same unread
    server_result, but it is unrunnable at HEAD (the P-384 build is broken at the v0.6.0
    pin), so it is left alone rather than changed blind. Worth folding in when P-384
    enablement resumes.
  • EXTERNAL_LISTENER=1 runs — the packaging validation matrix in CLAUDE.md — keep their
    previous behaviour exactly: F5's tightening applies (they always asserted the full body
    in practice), F6's server-side block is skipped by contract and prints that it was.

Convergence with Lane B's tools/https_e2e/evil_listener.py: its result dict uses the same
four base keys (listening, client_addr, request, error) with the same meanings, and
adds handshake-stage detail (client_hello_seen, server_flight_sent,
client_accepted_finished, client_finished_valid, client_reaction, response_sent).
_check_server_result() keys only on the shared subset and ignores unknown keys, so it
already reads an evil-listener result correctly; the richer fields are the right basis for a
stricter checker in that lane's own test, without either lane having to move first.

Environment notes for a fresh worktree: git submodule update --init --recursive, and
tools/https_e2e/certs/server.{pem,key} are gitignored — auto-generated by the listener, or
copied from an existing clone. (This lane builds BACKEND=uci, so the gitignored
ip65-build/ip65-c64.bin never came into it.)

JC-000and others added 2 commits August 13, 2026 06:10
…it F5)
test_https_local.py passed the run whenever the 5 characters "HELLO"
appeared anywhere in the 1000 bytes of screen RAM. That branch was
reachable only after the full-body assertion had already failed, so the
test substituted a weaker criterion at exactly the moment the strong one
did not hold: a truncated body, a mis-decrypted body that kept its first
word, or a stale HELLO left on screen by an earlier run all passed.
The C64-side criteria now live in `_check_c64_result()` — one rule, that
http_resp_buf holds the complete expected body. Screen RAM stays
diagnostic: a HELLO on screen without the body is reported as one of the
reasons the run failed, not as a pass.
A/B (host-side, same four inputs through the pre-fix block and the new
function):
input before after
full body in resp_buf, HELLO on screen PASS PASS
body truncated, HELLO still on screen PASS FAIL
resp_buf empty, stale HELLO on screen PASS FAIL
nothing anywhere FAIL FAIL
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dit F6)
Every run already wrote server_result.json — whether the listener
completed the handshake, the decrypted request, any TLS error — and no
pass criterion ever read a byte of it. All assertions were against
C64-side memory and screen RAM, i.e. against state the client produces
itself. The listener's record is the one piece of evidence in this test
the client cannot fabricate, and it was archived and discarded.
A run now passes only when both sides agree. `_check_server_result()`
fails on: a recorded `error`, `listening` unset, `client_addr` absent
(the listener never accepted a connection), and a `request` that is
missing, `<timeout>`, or not the GET this script programmed into the C64
(request line + `Host:` line — both values this script chose).
Deliberately not asserted, to avoid failing healthy runs: byte-exact
request equality (trailing headers are src/http.s's business), and
anything at all under EXTERNAL_LISTENER=1, where by contract there is no
inline listener and server_result is empty — that skip is printed.
`--check-artifact <run-dir>` re-runs the server-side criteria against an
archived run's server_result.json with no hardware, so the criteria are
testable and stored runs can be re-adjudicated.
A/B over the five captured runs under the audit scratchpad, each
composed with a healthy C64 side (http_status=200, the full body) so the
"C64 memory looks right" direction is the one under test:
server_result fixture before after
214751 clean (listening/addr/GET) PASS PASS
215648 clean (listening/addr/GET) PASS PASS
215331 SSLEOFError UNEXPECTED_EOF PASS FAIL
220559 SSLEOFError UNEXPECTED_EOF PASS FAIL
215221 listening only, never accepted PASS FAIL
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@JC-000

Copy link
Copy Markdown
OwnerAuthor

Independently reproduced by the supervising session, using the --check-artifact CLI this branch adds — which is itself the reason the claim was cheap to verify. Run against three captured artifact dirs from real hardware runs:

FixtureServer-side recordResult
20260812_214751decrypted GET / HTTP/1.1exit 0PASS: server-side criteria met
20260812_215331SSLEOFError: UNEXPECTED_EOF_WHILE_READINGexit 1FAIL: server-side criteria not met
20260812_215221listener never accepted a connectionexit 1FAIL: server-side criteria not met

Both failing fixtures passed under the baseline criteria, because the C64-side memory looked healthy in each — which is precisely the gap F6 described.

Three judgement calls here that I think are right:

Asserting the request line and Host: rather than byte-exact equality. Both values are chosen by this script, so the oracle tests what it programmed while staying immune to a future src/http.s header change. Strict enough to catch a wrong request, loose enough not to flake.

Deleting the screen-RAM fallback rather than strengthening it, and re-purposing HELLO-on-screen into a printed reason for failure. That keeps the diagnostic value without the pass path.

Failing closed on a missing or corrupt server_result.json. Treating an unreadable record as "no error recorded" would have reproduced the original defect one layer up.

The hardware confirmation matters as much as the negative cases: PASS in 133.5 s against a ~136 s baseline shows the added criteria cost nothing on a genuine run, so this does not trade false passes for false failures.

Noted for follow-up, correctly left alone here: tools/uci/test_https_local_p384.py carries the identical fallback and the identical unread server_result, but is unrunnable at HEAD because the P-384 build is broken at the v0.6.0 pin. Changing it blind would have meant shipping an unverifiable fix.

Confirms audit findings F5 and F6.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@JC-000