feat(e2e): hardware-free REU-less HTTPS e2e in VICE + fix(tls): post-ServerHello drain by JC-000 · Pull Request #71 · JC-000/c64-https

JC-000 · 2026-07-29T06:02:59Z

Summary

The REU-less stock-C64 configuration (BACKEND=ip65 USE_NISTCURVES_ONCHIP=1)
now completes HTTPS end-to-end in VICE with no REU attached — the first
full validation of the stock-machine story, entirely hardware-free
(patched-VICE pcap + macOS feth rig + local TLS 1.3 listener).

Deliverable run (ran twice, deterministic): boot→DHCP OK +38 s; then
TCP CONNECTED +3 s → CH +276 → SH +588 → CertificateVerify ≈1,175 s
(cert_buf/verify-scratch union exercised live) → FIN +1,781 →
TLS HANDSHAKE OK → HTTP 200, CONNECTION CLOSED +1,802 s (~31 min at
~1.2× emulation ⇒ ~36-37 min honest 1 MHz). REU-profile control also passes
(+977 s after 'G'), validating the fix on both profiles.

The bug this flushed out (`ac18513`) — backend-generic, wire-proven

ip65's SYN carries no MSS option → peers may segment small (macOS: 512 B),
and the client only ACKs when polled. The server flight's tail sat unACKed
through the ~5-min ECDHE stall; macOS hard-drops after 13 retransmits
(~54 s). The C64 then verified the entire buffered flight correctly
offline and died sending client Finished into a long-RST'd socket
(fingerprint: tls_state=$FF, tls_read_seq=4; identical REU/onchip).

Fix: bounded 8×250 net_poll drain after ServerHello parse (~15 s at
1 MHz) — zero unACKed data across all later crypto stalls; wire shows the
full flight ACKed 34 ms after landing. Why no rig ever saw it: Linux
retransmits 15-30 min (historical TAP rig passed); UCI firmware ACKs
autonomously (all Ultimate runs passed). Real internet servers sit between —
this drain is a prerequisite for any future 1 MHz real-server work. Note the
drain is unconditional (the bug is backend-generic); cost is ~15 s at
1 MHz ip65, unmeasurable on the REU control, and estimated ~1 s of fenced
polls on UCI turbo — worth re-validating on Ultimate hardware at next
opportunity. Follow-up documented in-code: flights larger than the TCP
window additionally need polling inside long crypto (real-server chains).

Also in here

HTTPS_PORT build knob (609c9aa): default builds byte-identical
(sha256-proven both backends); e2e uses 4433 so the listener runs
unprivileged.
tests/test_vice_https_macos.py: macOS rig preflight (Local-Network
permission self-probe, feth0 duplicate-MAC detection, dnsmasq liveness),
-minimized VICE (host keystrokes leaked into the emulated C64),
per-read transport.resume() (binary-monitor reads pause the emulated
CPU), autostart RUN fallback, per-phase timeline, failure-time TLS/net
state dump, memory-based response assertions (E2E_PROFILE,
E2E_NO_WARP knobs).
Listener: darwin TCP_RXT_CONNDROPTIME kept with a comment documenting
why it is insufficient alone (24abf3e assertion contract for the
boot.s demo path — see follow-up issue on its first-record-only copy).

Requires locally: the ethernet-capable VICE at ~/opt/vice-eth/bin/x64sc
and tools/rig-up-macos.sh (see c64-test-harness#144).

🤖 Generated with Claude Code

…p rig test, listener TCP patience - src/boot.s: HTTPS_PORT build knob (.ifndef default 443; default build byte-identical, verified ip65+uci PRG sha256 pre/post). Makefile plumbs an optional HTTPS_PORT=... to ca65 so the e2e listener can bind an unprivileged port (4433). - tests/test_vice_https_macos.py: macOS counterpart to test_phase3_https (Linux path untouched). Rig preflight (feth1 addr, dnsmasq pidfile with EPERM-means-alive, bpf perms, ethernet-VICE binary, feth0-conflict detection incl. the duplicate-MAC hazard), listener self-probe (catches the macOS Local Network permission block), autostart-RUN fallback, -minimized (host keystrokes must never reach the emulated C64), per-iteration transport.resume() (binary-monitor reads pause the CPU; without resume the machine idles at ~19% duty and 'CH' forever), VICE 3.10 speed model (no runtime warp; Speed=100000 measures ~1.2x on SDL2 — budget wall-clock for ~1x), SIGTERM runs teardown (an orphaned VICE stays on feth0 as a duplicate-MAC node), per-phase timeline table. - tools/https_e2e/https_listener.py: darwin-gated TCP_RXT_CONNDROPTIME (0x80) = 7200 s on listening + accepted sockets. macOS RSTs a local connection after ~30 s of unACKed retransmission; ip65 ACKs only when the C64 polls, and its 1 MHz crypto stalls run 4-25 min (observed: 5x rexmt over 33 s then RST mid-flight, C64 ACKed into the dead socket 4.5 min later -> TLS HANDSHAKE FAILED at EE decrypt). Linux default patience is minutes (why the historical TAP rig never hit it); UCI never hits it (Ultimate firmware TCP ACKs autonomously). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… stall First-ever REU-less ip65 HTTPS e2e (VICE, no -reu, onchip profile) PASSES with this: G->CONNECTION CLOSED in 1832 s at ~1.2x emulation, HTTP 200 + Content-Length-terminated body. Root cause chain (wire-proven on the macOS feth rig, runs 6-8): - ip65's SYN carries no MSS option -> macOS peers default to 512 B segments; the ~690 B server flight always splits 512+178. - The client ACKs segment 1 while awaiting SH, then disappears into the multi-minute ECDHE compute without draining segment 2. macOS hard-drops a connection after 13 retransmissions (~54 s at LAN RTO) — note TCP_RXT_CONNDROPTIME is a ceiling-shortener in xnu, it cannot extend past the 13-rexmt limit. - The C64 then processes the entire buffered flight OFFLINE — EE, Cert, CertificateVerify (signature verifies!), server Finished — and dies only when SENDING client Finished into the RST'd socket. Fingerprint: tls_state=$FF, tls_read_seq=4, last rec type $16 len $0035; identical under REU and onchip profiles (backend-generic, not REU-less-specific). Fix: after tls_parse_server_hello, run a bounded 8x250 net_poll drain (~10-20 s at 1 MHz) so everything already at the NIC is consumed and ACKed before compute starts. Zero unACKed data then exists across every later crypto stall; idle connections survive arbitrarily long verifies. Wire proof: client ACKs the full flight 34 ms after it lands. Why other rigs never saw this: Linux servers retransmit for 15-30 min (historical TAP rig passed); UCI firmware ACKs autonomously regardless of C64 polling (all Ultimate hardware runs passed). Real internet servers sit between those — this drain is also a prerequisite for the eventual 1 MHz real-server story, though flights larger than the 1460 B window will additionally need periodic polling inside long crypto (follow-up). Also: test asserts the response from C64 memory (http_status/ http_resp_buf via labels) instead of screen-scraping (body scrolls off); failure-time TLS/net state dump; listener keeps the darwin TCP_RXT_CONNDROPTIME attempt with a comment documenting why it is insufficient alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… body-in-buffer is informational boot.s's HTTPS demo path copies only the FIRST decrypted TLS record into http_resp_buf (158 B = status line + headers; the 22 B body arrives as record #2, delivered+ACKed on the wire but never copied). Pre-existing demo-path behavior, unrelated to the REU-less work — the full Content-Length body semantics live in the http_get/http_recv_response path. Follow-up: make the boot.s demo loop append subsequent records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

JC-000 · 2026-07-29T06:40:32Z

Honest-timing benchmark (no warp, true 1 MHz, no REU)

Final validation run on this branch: E2E_NO_WARP=1, BACKEND=ip65 USE_NISTCURVES_ONCHIP=1, VICE with no REU — PASS, 'G' → CONNECTION
CLOSED in 2,159.7 s ≈ 36.0 min (HTTP 200, Content-Length 22, clean close).

Per-phase (seconds after 'G'):

Phase	at	Δ	Note
TCP CONNECTED	3.0	3.0	DNS + 3-way handshake
CH (ClientHello)	329.2	326.2	X25519 keygen, onchip rows
SH processed	700.7	371.5	X25519 shared secret + ~15 s SH drain
PROC (flight decrypt)	718.9	18.2	EE + Cert under HS keys
FIN (verify + sFin)	2,135.5	1,416.7	ECDSA-P256 verify, REU-less
REQUEST SENT	2,153.6	18.1	cFin + app keys + GET
CONNECTION CLOSED	2,159.7	6.0	response + close

Model check: verify measured 1,417 s vs the v0.6.0 onchip fit's 1,397 s
(+1.4%); X25519 scalarmults 326/~356 s vs 324 s analytical (PR #69).
The stock-machine story is now a measured fact: a C64 with only an
RR-Net cartridge fetches an HTTPS page in ~36 minutes. For CLAUDE.md's
benchmark table alongside the 19-min REU-profile @1 MHz row.

🤖 Generated with Claude Code

…ndshake wall-clock (#74) * fix(tls): per-backend post-ServerHello drain budget — UCI 125.4 s -> expect ~56 s Fixes#73. Regression introduced by #71 (merged), measured on C64 Ultimate hardware: the shipped UCI onchip handshake+GET went 51.0 s -> 125.4 s at 48 MHz, a ~2.5x wall-clock regression. Correctness was never affected. The drain compensates for a property only ip65 has: it ACKs inbound TCP data only when the consumer pumps net_poll, so without draining, the server's post-SH flight tail sits unACKed through the multi-minute ECDHE/verify stalls and impatient peers drop the connection. UCI firmware ACKs autonomously — which is exactly why that bug was never observable on Ultimate hardware — so on UCI the drain has nothing to buy. Its cost, though, is anything but backend-neutral. An ip65 net_poll is a cheap NIC pump; a UCI net_poll is a full firmware command round-trip (SOCKET_READ: uci_wait_not_busy, uci_begin_cmd, 4x uci_put_byte, uci_push_wait, uci_check_err, header read, uci_drain_resp + uci_drain_status + uci_ack — ~25 fenced register accesses plus FPGA turnaround). Measured ~37 ms/poll at 48 MHz, of which only ~2.8 ms is fence time; the rest is clock-invariant firmware turnaround, so turbo does not amortize it. 2000 polls = ~70 s. Move the budget into a per-backend net_tuning.inc, resolved through the existing `-I src/net/$(BACKEND)` include path so tls13.s stays backend-agnostic: ip65: 8 x 250 = 2000 polls (UNCHANGED — the validated figure) uci: 1 x 16 = 16 polls (~0.6 s; a deliberate small hedge rather than 0, so anything already queued still lands in the ring before the long stalls without relying on firmware autonomy being absolute. Non-zero matters: the loop's dex/bne shape turns an INNER of 0 into 256 iterations.) Verified in the assembled listing: ip65 emits A0 08 / A2 FA, uci emits A0 01 / A2 10. All five profiles link with unchanged sizes (47,105 B ip65, 62,977 B uci). The ip65 images are byte-identical to master, so the ip65 e2e evidence from #71 carries over untouched; the UCI side needs a hardware re-measure (expected ~56 s = 51.0 s baseline + ~4.5 s for #69's on-chip X25519 rows at this clock). Process note for the record: I approved #71's unconditional drain with an "~1 s on UCI turbo" estimate that counted fence time only and ignored firmware turnaround — off by ~70x. An iteration-count budget calibrated on one backend's poll cost is precisely the failure mode that c64-lib-contract SPEC §13.4 (bounded waits must be wall-clock-based) exists to prevent. A TOD-bounded idle drain (poll until the ring stops growing) would be the principled backend-agnostic variant; it needs care around CIA1 TOD latch interaction with the UCI adapter's own TOD waits, so it is left as a documented refinement rather than bundled here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * tools/uci: skip redundant turbo write — unblocks C64U 64 MHz measurement The C64 Ultimate speed-switch quirk is sharper than CLAUDE.md records, and in a way that made 64 MHz unmeasurable: the bridge glitch is caused by the REST config WRITE itself, it SURVIVES the following reset, and it fires even when the written value equals the current one. Evidence (C64U 10.53.21.158, onchip UCI build, 2026-07-29): three 64 MHz attempts each wrote "64" while the device was ALREADY at 64 MHz, and each lost its first TCP_CONNECT — UCI_ERR_NO_SOCKET, net_tcp_state=CONNECT_FAIL, all TLS/HTTP state zero, ring head==tail==0, no SYN on the wire. The same PRG at the same 64 MHz setting passes under tools/uci/test_http_local.py, whose only material difference is that it performs no config write before its reset. At 48 MHz the identical pattern costs only the first attempt (fail, pass on retry), which is why this hid for so long. Fix: probe Turbo Control + CPU Speed first and skip the write entirely when they already match (the common case for repeat runs at one speed), and give a genuine change a 3.0 s settle instead of 0.5 s (TURBO_SETTLE overrides). The probe is best-effort — if it raises, we write as before. Both sides of the speed comparison are str()-normalised so a firmware type change cannot silently restore always-write. Not the wedge signature (that reaches ENC1 RX and stalls); this never opens a socket. Found by the hardware worker while validating #72/#73 — it blocked the 64 MHz headline number for PR #74. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tools/uci): turbo probe read the wrong JSON shape — skip was unreachable The redundant-write skip added in e044f51 never fired on the C64U: the probe read `get_config_item(...).get("value")`, but the REST config responses wrap items in a `<Category>` key and put each value directly under the ITEM name — there is no per-item "value" key. The probe therefore returned None/None on every device, `str(None) != "48"`, and the code always took the write path (log line: "Setting turbo to 48 MHz (from None/None)..."). Fix: mirror the harness's own get_reu_config — fetch the category, unwrap, index by item name — using public API only (no reliance on the private _unwrap). One request now covers both items instead of two. `.get(CAT, cat)` tolerates a response with or without the wrapper. Verified without hardware by parsing all three plausible shapes (wrapped, flat, int-typed CPU Speed): the skip decision comes out True for each, so a firmware shape or type change degrades to "write anyway" rather than to a silent wrong answer. Caught because the write path logs the probed values ("from None/None") — keeping the observed state in that message is what made an unreachable branch visible in a passing run. Worth remembering. Note the 3.0 s settle from e044f51 is doing real work independently: the first genuine speed change under it (64->48) connected on the first attempt, where every pre-fix genuine change cost a NO_SOCKET retry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

The benchmark tables predated three merged changes and one open fix, and several status claims had gone stale. Measured numbers only — no extrapolated figures. **e2e wall-clock.** The 2026-07-20 campaign rows are kept as history and labelled as such; a new "Post-#74 e2e numbers" block records HEAD: device profile clock pre-#71 post-#71 post-#74 C64U onchip 48 MHz 51.0 s 125.4 s 44.6 s C64U onchip 64 MHz 39.7 s (unmeas.) 33.7 s U64E REU 48 MHz 82.1 s 161.0 s 82.1 s Both onchip rows land BELOW their pre-regression baselines and the REU row lands exactly AT it — a REU build cannot contain #69 (its change is inside .ifdef USE_NISTCURVES_ONCHIP), so the pair is a clean control showing #69 is a SPEEDUP at turbo, not a cost. The doc now states that explicitly, including why the sign is easy to get wrong: the profile's 1 MHz penalty exists only because REU DMA is cheap relative to the CPU down there, and inverts above the crossover. (I got this backwards during the campaign; recording the reasoning so the next reader doesn't.) **New: ip65 / stock-C64 wall-clock**, the first ip65 e2e figures we have — 36.0 min honest 1 MHz REU-less, with the phase breakdown, plus the accelerated runs. Notes that the verify stretch came in 1.4% off the T(f)=D+C/f prediction three orders of magnitude from where that model was fit, and that ip65's drain budget is byte-identically unchanged by #74 so the numbers stand at HEAD. **Corrected stale claims:** - "ip65 is NOT packaged: does not link" — it links (#68). Explains the SCRATCH_UNION lifetime argument and its guards, notes packaging it is now a live option since a stock C64 + RR-Net has no shipped PRG today, and demotes c64-nist-curves#54 from blocker to optional headroom. - The CRYPTO_COLD_SHADOW "1,662 B overflow, cfg relief exhausted" entry, same fix. - The X25519-sibling entry claimed the old BSS overflow. Re-measured 2026-07-29: USE_X25519_SIBLING=1 under ip65 still fails, but on a DIFFERENT problem — X25519_RODATA over CRYPTO_OVERLAY by 2,048 B and LIB_NISTCURVES_P256_CODE over CRYPTO_RESIDENT by 103 B, i.e. code/rodata placement (ip65's overlay slot is 4,212 B vs UCI's 7.5 KB), not BSS. Better to state the measured failure than leave a fixed one on the page. **New design note** for the post-ServerHello drain: the ip65 property that motivates it (no MSS in SYN + ACK-only-when-polled), the offline-verify failure signature it prevents, why the budget must be per-backend (~40 ms per UCI net_poll vs a cheap ip65 pump — the #73 regression), current values, and the two open follow-ups (in-crypto polling for large flights; a wall-clock/idle bound instead of an iteration count, which is what the section's own rule actually demands). **New Smoke-tests subsection** for the hardware-free VICE ip65 rig, with the two prerequisites that are easiest to lose: the patched ethernet-capable VICE (stock macOS builds gate pcap on geteuid()==0) and the /dev/bpf permissions that reset every reboot. Also replaces the "blocked on an upstream ip65 bug (see lost memory note)" line with what is actually known now. Stacked on fix/drain-backend-budget: the post-#74 rows describe that PR's tree, not master's. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

VICE ip65 rig, onchip profile (REU-less stock-C64 config), master 2ceb5b1: PASS, http_status=200, resp_len=22, body match, 1,876.0 s accelerated against the 1,813.9 s reference (+3.4%). Phase shape unchanged. The reason this run was owed: the ip65 PRG is no longer byte-identical to the #71-era build (417c708594... vs db31111031e2...) because #75's span-input parser is real code. That change had only ever been exercised on UCI. This is also ip65's first e2e since July, so it is the first to cover #74, #75 and the ten audit PRs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

JC-000and others added 3 commits July 28, 2026 22:45

JC-000 mentioned this pull request Jul 29, 2026
boot.s HTTPS demo path copies only the first decrypted TLS record into http_resp_buf — body never displayed #72
Closed

JC-000 merged commit 897b110 into masterJul 29, 2026

This was referenced Aug 13, 2026
docs: benchmark refresh + ip65 status, LSP artifact cleanup, drain-budget fixes #77
Merged
docs: correct the x25519 sibling record; add pin-drift check; fix test_x25519 skip-as-pass #81
Merged

This was referenced Aug 14, 2026
docs: retract the ~10 s clock-invariant claim + complete the U64E sweep, ip65 re-validation #88
Merged
chore(contract): align to c64-lib-contract v0.10.0 — §6.6 footprint asserts + pin-adaptive ZP spelling #114
Merged

Navigation Menu

feat(e2e): hardware-free REU-less HTTPS e2e in VICE + fix(tls): post-ServerHello drain - #71

feat(e2e): hardware-free REU-less HTTPS e2e in VICE + fix(tls): post-ServerHello drain#71
JC-000 merged 3 commits into
masterfrom
feat/vice-macos-e2e

JC-000 commented Jul 29, 2026

Uh oh!

JC-000 commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

JC-000 commented Jul 29, 2026

Summary

The bug this flushed out (ac18513) — backend-generic, wire-proven

Also in here

Uh oh!

JC-000 commented Jul 29, 2026

Honest-timing benchmark (no warp, true 1 MHz, no REU)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

The bug this flushed out (`ac18513`) — backend-generic, wire-proven