diff --git a/CLAUDE.md b/CLAUDE.md index 1f400f4..3daf610 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -496,7 +496,7 @@ The scripts: `EXTERNAL_PORT`, default 4433) skips the inline listener + repo-cert load and points the C64 at an out-of-band server — e.g. the - packaged `dist/c64-https-listener.zip` + packaged `dist/c64-https-listener.py` listener; pass criteria then come from C64-side state only. Default OFF. Each run writes a timestamped artifact dir @@ -1350,50 +1350,74 @@ ld65 and ca65 edge cases; they are intentional and should stay: ## Packaging -`make package` builds the release artifacts into `dist/` (gitignored): - - - `c64-https-uci-reu.prg` — default REU profile (`make BACKEND=uci`). - Requires REU hardware/enabled; fastest - at stock 1 MHz (the REU profile is the - right default below ~7 MHz). - - `c64-https-uci-onchip.prg` — `USE_NISTCURVES_ONCHIP=1`. **No REU - required** — for stock machines without - an REU (~3.9x the verify CPU work; at - 1 MHz expect ~23 min for the ECDSA - verify alone). - - `c64-https.d64` — both PRGs on one 1541 image - (`HTTPS-REU`, `HTTPS-NOREU`), built - with VICE's `c1541`. - - `c64-https-listener.zip` — self-contained Python TLS 1.3 test - listener (source: `tools/package/ - listener/`): `run.sh` creates a venv, - installs `cryptography`, **generates - fresh P-256 certs** (`gen_certs.py`), - and serves the canonical response. - Requires an OpenSSL 1.1.1+/3.x python - (refuses LibreSSL, e.g. macOS system - python, with a clear error). - - `MANIFEST.txt` — sizes, git HEAD, sha256 checksums. - -Scripts live in `tools/package/` (`build_prgs.sh`, `build_d64.sh`, -`build_listener_zip.sh`); each variant build does `make clean` first -(flag changes are not tracked by make). Builds are deterministic — -`make package` reproduces the validated hashes at the same HEAD. - -**ip65 is not packaged yet, but it now LINKS.** The historical blocker -(`BSS overflows CRYPTO_COLD_SHADOW by 1406 bytes`) was closed by the -#68 refit — the overflow was exactly `LIB_NISTCURVES_P256_BSS`, which -now time-shares `cert_buf`'s RAM via the `SCRATCH_UNION` region (their -lifetimes are disjoint; see the cfg comment block and the lifetime -contract at `cert_buf` in `src/der_decode.s`). Both ip65 profiles -build, and the REU-less ip65+onchip image is validated end-to-end in -VICE (see "ip65 / stock-C64 wall-clock"). Adding it to `make package` -is a live option — it is the only artifact that serves a stock C64 + -RR-Net cartridge, which today has no shipped PRG at all. Note -c64-nist-curves#54 is CLOSED-as-completed and already inside our pin, -so it is not headroom in reserve. The comb profile stays deliberately excluded (REU -bank 2 residency + ~40 min boot precompute at 1 MHz make it wrong for -a general release). +`make package` builds the release artifacts into `dist/` (gitignored). +**All four backend x profile combinations ship**, `make clean` between +every one: + + - `c64-https-uci-reu.prg` `make BACKEND=uci` + - `c64-https-uci-onchip.prg` `+ USE_NISTCURVES_ONCHIP=1` + - `c64-https-ip65-reu.prg` `make BACKEND=ip65` + - `c64-https-ip65-onchip.prg` `+ USE_NISTCURVES_ONCHIP=1` + +ip65 is now packaged (it was previously excluded on a link failure that +the #68 refit closed) — it is the only artifact that serves a stock C64 ++ RR-Net cartridge, and `ip65-onchip` is the only image a bone-stock +machine with no REU can run at all. + +The REU-vs-onchip guidance in `MANIFEST.txt` is the measured **~18 MHz** +crossover, not the older ~7 MHz figure: the REU profile carries a +wall-clock floor (DMA anchored to the ~1 MHz bus) that turbo cannot +touch, the onchip profile has none, and on a U64E the sign flips between +the 16 and 20 MHz CPU-speed settings. See the ECDSA wall-clock section. + +Disk images: + + - `c64-https-.d64` x4 — one PRG each, `LOAD"*",8,1` + - `c64-https-uci.d64`, `c64-https-ip65.d64` — both of that backend's + profiles on one disk + +There is deliberately **no all-in-one image**: the four PRGs total 868 +blocks against a .d64's 664 free. Each backend's pair does fit (UCI 496, +ip65 372), which makes the per-backend disk the largest useful bundle. + +`c64-https-listener.py` is a **single self-extracting Python file** (was +a zip + `run.sh` + venv + pip). It has **no third-party dependency at +all**: `cryptography` was only ever used to mint the self-signed P-256 +cert, and `tools/package/listener/gen_certs.py` now does that in pure +Python (P-256 point arithmetic + minimal DER encoder + ECDSA-SHA256). +TLS was always stdlib `ssl`. What remains is a property of the +*interpreter*, not an installable package — an `ssl` with TLS 1.3 +(OpenSSL 1.1.1+); macOS's `/usr/bin/python3` is LibreSSL 2.8.3 and +cannot serve this client at any price. That is detected at startup and +reported in one line (never a traceback, `--debug` restores it), and it +is stated in `MANIFEST.txt` rather than left to be discovered. +`--selftest` proves the whole path with no C64: mint cert, serve on +loopback, drive it with a Python `ssl` client, then again with `openssl +s_client -ciphersuites TLS_CHACHA20_POLY1305_SHA256` — the C64's only +suite, which the stdlib client can never force because CPython exposes +no API to restrict TLS 1.3 suites. + +Scripts live in `tools/package/`: `_common.sh` (the variant matrix — one +line per shipped PRG, every other script derives from it), +`build_prgs.sh`, `build_d64.sh`, `build_listener.py`, `write_manifest.sh`. +Nothing is version-specific; re-running `make package` after a submodule +bump regenerates every artifact with zero edits. + +`make package-verify` is the acceptance gate (`tools/package/ +verify_release.py`): rebuilds every variant and compares **PRG** hashes +(object hashes are not evidence — ca65 stamps build time into every +`.o`), reads each PRG back out of its .d64 with `c1541` and +byte-compares, boots every image in VICE asserting the banner, and runs +the listener selftest. `SKIP_REBUILD` / `SKIP_VICE` / `SKIP_LISTENER` +narrow it. + +Booting a .d64 in VICE needs `-trapdevice8 +drive8truedrive`: under true +drive emulation the ~250-block load never completes inside any sane +budget, and the symptom is a screen stuck on `LOADING` that looks like a +bad image rather than a slow one. `verify_release.py` passes both flags. + +The comb profile stays deliberately excluded (REU bank 2 residency + +~40 min boot precompute at 1 MHz make it wrong for a general release). Validation record (2026-07-27, HEAD cb6eab4): - onchip PRG passes the 3-vector ECDSA KAT in VICE **without** REU diff --git a/Makefile b/Makefile index f425912..68d4c74 100644 --- a/Makefile +++ b/Makefile @@ -231,7 +231,7 @@ ALL_OBJS := $(TOP_OBJS) $(CRYPTO_OBJS) $(CRYPTO_SHARED_OBJS) $(NET_OBJS) PRG := build/c64-https.prg LABELS := build/labels.txt -.PHONY: all link run clean ip65-libs ip65-blob package +.PHONY: all link run clean ip65-libs ip65-blob package package-verify all: $(PRG) @@ -443,17 +443,28 @@ run: $(PRG) clean: rm -rf build -# Release packaging: build the PRG matrix, bundle both UCI PRGs onto a D64, -# and write dist/MANIFEST.txt (sizes, git HEAD, sha256 checksums). The scripts -# run `make clean` between flag combinations themselves, so `package` does not -# depend on any build artifact. build_listener_zip.sh is skipped gracefully -# when absent so a partial checkout can still package the PRGs. +# Release packaging: build all four PRG variants (both backends x both crypto +# profiles), put each on its own D64 plus a per-backend D64, generate the +# single-file test listener, and write dist/MANIFEST.txt. +# +# The scripts run `make clean` between flag combinations themselves, so +# `package` deliberately depends on no build artifact — and re-running it after +# a submodule bump regenerates everything with no edits anywhere. Order +# matters: build_prgs.sh writes dist/build-info.txt and build_d64.sh writes +# dist/d64-listings.txt, both of which write_manifest.sh consumes. +# +# PACKAGE_PYTHON must be an interpreter that can run the listener's own +# selftest — see `make package-verify`. +PACKAGE_PYTHON ?= python3 package: bash tools/package/build_prgs.sh - @if [ -x tools/package/build_listener_zip.sh ]; then \ - echo "[package] running tools/package/build_listener_zip.sh"; \ - bash tools/package/build_listener_zip.sh; \ - else \ - echo "[package] tools/package/build_listener_zip.sh absent — skipping listener zip"; \ - fi bash tools/package/build_d64.sh + $(PACKAGE_PYTHON) tools/package/build_listener.py + bash tools/package/write_manifest.sh + +# Acceptance gate for the release artifacts: rebuild every PRG a second time +# and compare PRG hashes, boot every D64 in VICE and assert the banner, and run +# the built listener end to end against a Python ssl client. Measures; does not +# assert. Run it after `make package`. +package-verify: + $(PACKAGE_PYTHON) tools/package/verify_release.py diff --git a/README.md b/README.md index a8e0946..e800c43 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,31 @@ An HTTPS client for the Commodore 64 in 6502 assembly. Implements TLS 1.3 over T **For demonstration and educational purposes only — not cryptographically secure.** +## I just want to run it + +Grab a release: every build is prebuilt, as a `.prg` and as a bootable `.d64`. +No assembler, no cc65, no Python packages, no build step. Two questions pick +your image, and `MANIFEST.txt` in the release walks through them: + +| | REU present | no REU | +|---|---|---| +| **Ultimate 64 / C64 Ultimate** | `c64-https-uci-reu` | `c64-https-uci-onchip` | +| **stock C64 + RR-Net** | `c64-https-ip65-reu` | `c64-https-ip65-onchip` | + +The `reu` images are faster below roughly 18 MHz — which is every real stock +C64 — because they offload the ECDSA verify to REU DMA. The `onchip` images +need no REU at all and win above that crossover, so they are the right pick +for Ultimate turbo modes. `ip65-onchip` is the only image a bone-stock +machine with no expansion RAM can run end to end. + +`c64-https-listener.py` in the same release is a single self-extracting file +that stands up the server side to point the C64 at: it mints its own +certificate and needs nothing installed, only a `python3` whose `ssl` has +TLS 1.3. Run `python3 c64-https-listener.py --selftest` to check that before +involving a C64. + +To build these yourself: `make package && make package-verify`. + ## Architecture ``` diff --git a/tools/package/_common.sh b/tools/package/_common.sh new file mode 100644 index 0000000..ef56ce1 --- /dev/null +++ b/tools/package/_common.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# ============================================================================= +# tools/package/_common.sh — shared definitions for the release packaging +# scripts. Sourced, never executed. +# +# The single source of truth for the release variant matrix lives here so that +# build_prgs.sh, build_d64.sh and write_manifest.sh cannot drift apart. +# ============================================================================= + +# --- Variant matrix ----------------------------------------------------------- +# One line per shipped PRG: +# |||<1541 filename>|| +# +# 1541 filenames are <=16 chars and lowercase here because c1541 uppercases +# into PETSCII on write. They are the same on the single-variant disk and on +# the per-backend disk, so a user only ever learns one name. Keep them stable +# across releases — people type them. +# +# Nothing here is version-specific: adding a profile or a backend is one line, +# and every downstream script picks it up with no further edits. +PACKAGE_VARIANTS=( + "uci-reu|c64-https-uci-reu.prg|BACKEND=uci|uci-reu|uci|Ultimate 64 / C64 Ultimate (UCI networking). Needs the REU enabled. Faster below ~18 MHz — the right pick at stock 1 MHz." + "uci-onchip|c64-https-uci-onchip.prg|BACKEND=uci USE_NISTCURVES_ONCHIP=1|uci-noreu|uci|Ultimate 64 / C64 Ultimate (UCI networking). No REU required. Faster above ~18 MHz — the right pick at 32/48/64 MHz turbo." + "ip65-reu|c64-https-ip65-reu.prg|BACKEND=ip65|ip65-reu|ip65|Stock C64 + RR-Net / cs8900a cartridge. Needs an REU. Faster below ~18 MHz, i.e. at any speed a real stock C64 runs at." + "ip65-onchip|c64-https-ip65-onchip.prg|BACKEND=ip65 USE_NISTCURVES_ONCHIP=1|ip65-noreu|ip65|Stock C64 + RR-Net / cs8900a cartridge, no REU at all. The only image a bone-stock C64 can run end to end; slowest (~36 min per handshake at 1 MHz)." +) + +# Backends, in matrix order, deduplicated. Used for the per-backend disks. +package_backends() { + local line + for line in "${PACKAGE_VARIANTS[@]}"; do + printf '%s\n' "$(variant_field "$line" 5)" + done | awk 'NF && !seen[$0]++' +} + +# Field accessors — `variant_field <1-based index>`. +variant_field() { printf '%s' "$1" | cut -d'|' -f"$2"; } + +# --- Paths -------------------------------------------------------------------- +# PROJECT_ROOT must be set by the caller before sourcing (it knows its own $0). +: "${PROJECT_ROOT:?_common.sh: PROJECT_ROOT must be set before sourcing}" + +DIST="$PROJECT_ROOT/dist" +BUILT_PRG="$PROJECT_ROOT/build/c64-https.prg" +BUILD_INFO="$DIST/build-info.txt" # machine-readable; write_manifest.sh reads it +MANIFEST="$DIST/MANIFEST.txt" + +# --- sha256, portably --------------------------------------------------------- +if command -v sha256sum >/dev/null 2>&1; then + sha256_of() { sha256sum "$1" | cut -d' ' -f1; } +else + sha256_of() { shasum -a 256 "$1" | cut -d' ' -f1; } +fi + +# --- Submodule pins (offline) ------------------------------------------------- +# Deliberately NOT `git submodule status`: it renders versions via `git +# describe` *without* `--tags`, so a lightweight tag (c64-x25519 v0.6.0 is one) +# is invisible and the pin reads as "5 commits past v0.5.0". Read the gitlink +# from the tree and resolve the tag inside the submodule with --tags. +# Prints " " per submodule. +submodule_pins() { + git -C "$PROJECT_ROOT" config --file .gitmodules \ + --get-regexp '^submodule\..*\.path$' 2>/dev/null \ + | awk '{print $2}' | sort | while read -r sub; do + local sha tag + sha="$(git -C "$PROJECT_ROOT" ls-tree HEAD "$sub" | awk '{print $3}')" + [ -n "$sha" ] || continue + tag="$(git -C "$PROJECT_ROOT/$sub" describe --tags --exact-match "$sha" \ + 2>/dev/null || true)" + [ -n "$tag" ] || tag="(untagged)" + printf '%s %s %s\n' "$sub" "$sha" "$tag" + done +} diff --git a/tools/package/build_d64.sh b/tools/package/build_d64.sh index e3ad905..17fab6e 100755 --- a/tools/package/build_d64.sh +++ b/tools/package/build_d64.sh @@ -1,97 +1,95 @@ #!/usr/bin/env bash # ============================================================================= -# tools/package/build_d64.sh - Assemble dist/c64-https.d64 and finalize the -# release manifest. +# tools/package/build_d64.sh — assemble the release 1541 disk images. # -# Bundles both UCI PRGs (produced by build_prgs.sh) onto a single 1541 disk -# image via VICE's `c1541`, verifies the directory reads back, then appends the -# D64 section and the final sha256 checksum section to dist/MANIFEST.txt. This -# script runs LAST in the package flow so its checksum section covers every -# artifact in dist/ (PRGs, the D64, and the optional listener zip). +# Produces, from the PRGs build_prgs.sh left in dist/: # -# Disk filenames are <=16 char PETSCII: -# HTTPS-REU <- c64-https-uci-reu.prg (default REU verify profile) -# HTTPS-NOREU <- c64-https-uci-onchip.prg (on-chip / no-REU verify path) +# c64-https-.d64 one per variant, one PRG each (4 images) +# c64-https-.d64 one per backend, both of that backend's +# profiles on one disk (2 images) # -# Usage: tools/package/build_d64.sh +# WHY NOT ONE DISK WITH ALL FOUR: it does not fit, and that is arithmetic, not +# preference. A .d64 holds 664 free blocks = 168,656 usable bytes; the four +# PRGs total ~220 KB (2 x 62,977 UCI + 2 x 47,105 ip65). Each backend's pair +# does fit (UCI ~496 blocks, ip65 ~371), so the per-backend disk is the largest +# useful bundle. The per-variant singles are the "I know what I want, give me +# one disk" case and are what the release notes point at. +# +# Every image is bootable with LOAD"*",8,1 (the wanted PRG is the first file on +# the single-variant disks). File names are shared between the single and the +# per-backend disk so a user only ever learns one name. +# +# Usage: tools/package/build_d64.sh [C1541=... to override the tool] # ============================================================================= -set -eo pipefail +set -euo pipefail -PROJECT_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" cd "$PROJECT_ROOT" +# shellcheck source=tools/package/_common.sh +. "$PROJECT_ROOT/tools/package/_common.sh" -DIST="$PROJECT_ROOT/dist" -MANIFEST="$DIST/MANIFEST.txt" -D64="$DIST/c64-https.d64" C1541="${C1541:-c1541}" - -REU_PRG="$DIST/c64-https-uci-reu.prg" -ONCHIP_PRG="$DIST/c64-https-uci-onchip.prg" +D64_LIST="$DIST/d64-listings.txt" # write_manifest.sh reads this if ! command -v "$C1541" >/dev/null 2>&1; then - echo "ERROR: c1541 not found in PATH (ships with VICE). Set C1541=... to override." >&2 + echo "ERROR: c1541 not found in PATH (it ships with VICE). Set C1541=... to override." >&2 exit 1 fi -for f in "$REU_PRG" "$ONCHIP_PRG"; do - if [ ! -f "$f" ]; then - echo "ERROR: missing $f — run tools/package/build_prgs.sh first." >&2 - exit 1 - fi -done -if [ ! -f "$MANIFEST" ]; then - echo "ERROR: missing $MANIFEST — run tools/package/build_prgs.sh first." >&2 - exit 1 -fi - -# --- Assemble the disk image --- -# c1541 -write takes a host path + a target 1541 filename; PRGs are written as -# PRG-type files. -format wipes/labels the image first. -rm -f "$D64" -echo "[package] formatting $D64" -"$C1541" -format "c64-https,01" d64 "$D64" >/dev/null -echo "[package] writing PRGs to disk image" -"$C1541" -attach "$D64" \ - -write "$REU_PRG" "https-reu,p" \ - -write "$ONCHIP_PRG" "https-noreu,p" >/dev/null -# --- Read back the directory to verify --- -# c1541 prints a harmless "OPENCBM: ... libopencbm.dylib failed!" line when the -# real-drive backend is absent (always, on a dev box); drop it so the recorded -# listing is just the disk directory. -echo "[package] directory listing:" -LISTING="$("$C1541" -attach "$D64" -list | grep -v '^OPENCBM:')" -echo "$LISTING" +# c1541 interleaves its own chatter with the directory: a harmless OPENCBM line +# when the real-drive backend is absent (i.e. always, on a dev box), plus +# attach/detach/recognised notices naming absolute host paths. Strip all of it +# so the recorded listing is the disk directory and nothing machine-specific — +# otherwise the manifest would differ between two builders' checkouts. +c1541_list() { + "$C1541" -attach "$1" -list \ + | grep -Ev '^(OPENCBM:|D64 disk image |Unit [0-9]+ drive )' +} -# --- Append D64 section to the manifest --- -{ - echo - echo "== d64 image ==" - echo "c64-https.d64 (1541 image, both UCI PRGs)" - echo "directory listing:" - echo "----------------------------------------------------------------" - echo "$LISTING" - echo "----------------------------------------------------------------" -} >> "$MANIFEST" +: > "$D64_LIST" -# --- Final section: sha256 of every artifact in dist/ (except the manifest) --- -{ - echo - echo "== sha256 checksums ==" -} >> "$MANIFEST" +# make_disk <1541-name> [...] +make_disk() { + local image="$1" label="$2" id="$3"; shift 3 + rm -f "$image" + "$C1541" -format "$label,$id" d64 "$image" >/dev/null + local -a writes=() + while [ "$#" -gt 0 ]; do + [ -f "$1" ] || { echo "ERROR: missing $1 — run build_prgs.sh first." >&2; exit 1; } + writes+=(-write "$1" "$2,p") + shift 2 + done + "$C1541" -attach "$image" "${writes[@]}" >/dev/null + local listing + listing="$(c1541_list "$image")" + echo "[package] $(basename "$image"):" + printf '%s\n' "$listing" | sed 's/^/[package] /' + { + echo "image=$(basename "$image")" + printf '%s\n' "$listing" + echo "---" + } >> "$D64_LIST" +} -# Prefer sha256sum; fall back to `shasum -a 256` on macOS. -if command -v sha256sum >/dev/null 2>&1; then - SHA_CMD=(sha256sum) -else - SHA_CMD=(shasum -a 256) -fi +# --- One image per variant ---------------------------------------------------- +for line in "${PACKAGE_VARIANTS[@]}"; do + key="$(variant_field "$line" 1)" + prg="$(variant_field "$line" 2)" + name="$(variant_field "$line" 4)" + # Disk label is the variant key: <=16 PETSCII chars, and it makes the + # directory header self-identifying when four near-identical disks are + # sitting in a downloads folder. + make_disk "$DIST/c64-https-$key.d64" "$key" "01" "$DIST/$prg" "$name" +done -# Stable, sorted list of artifacts, manifest excluded so the checksum section -# never has to hash the file it is being written into. BSD find (macOS) lacks -# -printf, so strip the leading ./ with sed. -( cd "$DIST" && find . -maxdepth 1 -type f ! -name 'MANIFEST.txt' | sed 's|^\./||' | sort ) \ -| while IFS= read -r f; do - ( cd "$DIST" && "${SHA_CMD[@]}" "$f" ) >> "$MANIFEST" +# --- One image per backend, carrying that backend's profiles ------------------ +for backend in $(package_backends); do + args=() + for line in "${PACKAGE_VARIANTS[@]}"; do + [ "$(variant_field "$line" 5)" = "$backend" ] || continue + args+=("$DIST/$(variant_field "$line" 2)" "$(variant_field "$line" 4)") + done + make_disk "$DIST/c64-https-$backend.d64" "c64-https $backend" "01" "${args[@]}" done -echo "[package] wrote $D64 and finalized $MANIFEST" +echo "[package] disk images complete." diff --git a/tools/package/build_listener.py b/tools/package/build_listener.py new file mode 100755 index 0000000..b92b8c3 --- /dev/null +++ b/tools/package/build_listener.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +"""Build dist/c64-https-listener.py — the single-file, self-extracting +TLS 1.3 test listener. + +Packs the sources under tools/package/listener/ into one runnable .py: a small +bootstrap header followed by a base64+zlib tar of the modules, which unpacks +into a temp dir and runs. One file, `python3 c64-https-listener.py`, done. + +WHAT HAPPENED TO THE DEPENDENCY +------------------------------- +The obvious blocker for a self-extracting Python bundle is that `cryptography` +is a compiled extension — wheels are per-platform and per-Python-version, so +it cannot be embedded in a source payload and stay portable. The options were: +embed the pure-Python parts and error out asking for `pip install +cryptography`; vendor a pure-Python TLS stack; or bootstrap a venv on first +run. + +None of those were taken, because the premise was wrong. `cryptography` was +never needed for TLS here — the server has always been stdlib `ssl`. It was +needed for exactly one thing: minting the self-signed P-256 certificate. So +gen_certs.py now does that in pure Python (~200 lines: P-256 point +arithmetic, a minimal DER encoder, ECDSA-SHA256), and the bundle has **no +third-party dependency at all**. Nothing to pip install, no venv, no network. + +What remains is a requirement on the *interpreter*, not on a package: an `ssl` +module with TLS 1.3, i.e. linked against OpenSSL 1.1.1+. macOS's system +python3 is LibreSSL 2.8.3 and cannot serve TLS 1.3 at any price. That is +detected up front and reported in one line, never as a traceback, and it is +stated in the manifest so nobody has to discover it by running the thing. + +DETERMINISM +----------- +Tar metadata is pinned (mtime 0, uid/gid 0, mode 0644, USTAR) and zlib level 9 +is deterministic, so the same sources produce a byte-identical bundle. That +lets the manifest's sha256 mean something. + +Usage: tools/package/build_listener.py [--out PATH] [--selftest] + --selftest additionally runs the built bundle's own --selftest. +""" +from __future__ import annotations + +import argparse +import base64 +import hashlib +import io +import subprocess +import sys +import tarfile +import zlib +from pathlib import Path + +HERE = Path(__file__).resolve().parent +SRC_DIR = HERE / "listener" +REPO_ROOT = HERE.parent.parent + +# Modules that go into the bundle. Nothing else from the tree leaks in. +PAYLOAD_FILES = ["listener.py", "gen_certs.py", "README.md"] +ENTRY = "listener.py" + +BOOTSTRAP = '''#!/usr/bin/env python3 +"""c64-https TLS 1.3 test listener — single file, no dependencies. + +Stands up the entire server side of the c64-https end-to-end test: mints a +fresh self-signed ECDSA P-256 certificate, serves TLS 1.3 only, and returns +the canonical response the Commodore 64 client expects. + + python3 c64-https-listener.py # port 443, falls back to 4433 + python3 c64-https-listener.py --port 4433 # unprivileged + python3 c64-https-listener.py --selftest # prove it works, no C64 needed + python3 c64-https-listener.py --extract DIR # unpack the sources and exit + python3 c64-https-listener.py --help + +REQUIREMENTS: Python 3.8+ whose ssl module has TLS 1.3 (OpenSSL 1.1.1+ or +3.x). There is nothing to pip install. macOS's /usr/bin/python3 is linked +against LibreSSL 2.8.3, has no TLS 1.3, and will refuse with a one-line +message — use python.org or Homebrew python3 there. + +This file is generated by tools/package/build_listener.py in the c64-https +repo; edit the sources there, not the payload below. + +The certificate it generates is a throwaway test fixture, not a trust anchor. +Do not deploy this anywhere real. +""" +import atexit +import base64 +import hashlib +import io +import os +import runpy +import shutil +import sys +import tarfile +import tempfile +import zlib + +_PAYLOAD_SHA256 = "@SHA256@" +_ENTRY = "@ENTRY@" +_PAYLOAD = """\\ +@PAYLOAD@""" + + +def _die(msg): + sys.stderr.write("ERROR: %s\\n" % msg) + raise SystemExit(1) + + +def _unpack(dest): + raw = base64.b64decode(_PAYLOAD) + got = hashlib.sha256(raw).hexdigest() + if got != _PAYLOAD_SHA256: + _die("this file's embedded payload is corrupt (sha256 %s, expected " + "%s) — re-download it" % (got[:16], _PAYLOAD_SHA256[:16])) + with tarfile.open(fileobj=io.BytesIO(zlib.decompress(raw)), mode="r:") as tar: + # Every member is a plain file written by build_listener.py; reject + # anything else rather than trusting the archive. + for member in tar.getmembers(): + if not member.isfile() or "/" in member.name or member.name.startswith("."): + _die("unexpected payload member %r" % member.name) + # filter="data" is the 3.12+ safe extraction mode and the default from + # 3.14; passing it unconditionally there and omitting it below keeps + # one file working across 3.8 -> 3.14 without a DeprecationWarning. + if sys.version_info >= (3, 12): + tar.extractall(dest, filter="data") + else: + tar.extractall(dest) + + +def _main(): + if sys.version_info < (3, 8): + _die("needs Python 3.8+, this is %d.%d" + % (sys.version_info[0], sys.version_info[1])) + + argv = sys.argv[1:] + if "--extract" in argv: + i = argv.index("--extract") + if i + 1 >= len(argv): + _die("--extract needs a directory") + dest = os.path.abspath(argv[i + 1]) + os.makedirs(dest, exist_ok=True) + _unpack(dest) + print("extracted %d files to %s" % (len(os.listdir(dest)), dest)) + return 0 + + tmp = tempfile.mkdtemp(prefix="c64-https-listener-") + atexit.register(shutil.rmtree, tmp, True) + _unpack(tmp) + sys.path.insert(0, tmp) + # run_path with run_name="__main__" makes the extracted module run its own + # __main__ block, which raises SystemExit with the real exit code. + runpy.run_path(os.path.join(tmp, _ENTRY), run_name="__main__") + return 0 + + +if __name__ == "__main__": + try: + sys.exit(_main()) + except SystemExit: + raise + except KeyboardInterrupt: + sys.exit(130) + except Exception as exc: + _die("%s: %s" % (type(exc).__name__, exc)) +''' + + +def build(out_path: Path) -> Path: + missing = [f for f in PAYLOAD_FILES if not (SRC_DIR / f).is_file()] + if missing: + raise SystemExit(f"ERROR: missing listener sources: {missing}") + + buf = io.BytesIO() + with tarfile.open(fileobj=buf, mode="w", + format=tarfile.USTAR_FORMAT) as tar: + for name in PAYLOAD_FILES: + data = (SRC_DIR / name).read_bytes() + info = tarfile.TarInfo(name) + info.size = len(data) + # Pinned metadata: without this the bundle's sha256 would change + # every time a source file's mtime did, and the manifest checksum + # would stop being reproducible. + info.mtime = 0 + info.mode = 0o644 + info.uid = info.gid = 0 + info.uname = info.gname = "" + tar.addfile(info, io.BytesIO(data)) + + compressed = zlib.compress(buf.getvalue(), 9) + digest = hashlib.sha256(compressed).hexdigest() + b64 = base64.b64encode(compressed).decode("ascii") + lines = "\n".join(b64[i:i + 76] for i in range(0, len(b64), 76)) + + text = (BOOTSTRAP + .replace("@SHA256@", digest) + .replace("@ENTRY@", ENTRY) + .replace("@PAYLOAD@", lines)) + out_path.parent.mkdir(parents=True, exist_ok=True) + out_path.write_text(text) + out_path.chmod(0o755) + print(f"[package] wrote {out_path}") + print(f"[package] {out_path.stat().st_size} bytes, " + f"{len(PAYLOAD_FILES)} embedded modules, payload sha256 {digest}") + return out_path + + +def main(argv=None) -> int: + p = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--out", default=str(REPO_ROOT / "dist" / "c64-https-listener.py"), + help="output path (default dist/c64-https-listener.py)") + p.add_argument("--selftest", action="store_true", + help="run the built bundle's own --selftest afterwards") + args = p.parse_args(argv) + + out = build(Path(args.out)) + if args.selftest: + print(f"[package] running {out.name} --selftest") + rc = subprocess.call([sys.executable, str(out), "--selftest"]) + if rc != 0: + print(f"[package] bundle selftest FAILED (exit {rc})", file=sys.stderr) + return rc + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/package/build_listener_zip.sh b/tools/package/build_listener_zip.sh deleted file mode 100755 index b4fc7a6..0000000 --- a/tools/package/build_listener_zip.sh +++ /dev/null @@ -1,56 +0,0 @@ -#!/usr/bin/env bash -# Build dist/c64-https-listener.zip — the self-contained TLS 1.3 test -# listener package. The zip lets someone stand up the entire server side -# of the c64-https end-to-end test on a fresh machine (including cert -# generation) with zero dependency on this repo's committed certs or on -# the c64-test-harness package. -# -# Standalone-runnable; a teammate wires this into `make package`. -set -euo pipefail - -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$HERE/../.." && pwd)" -SRC_DIR="$HERE/listener" -DIST_DIR="$REPO_ROOT/dist" -ZIP_PATH="$DIST_DIR/c64-https-listener.zip" - -# Files that make up the package (nothing else from the tree leaks in). -FILES=( - "run.sh" - "listener.py" - "gen_certs.py" - "requirements.txt" - "README.md" -) - -echo "Building c64-https listener zip" -echo " source : $SRC_DIR" -echo " output : $ZIP_PATH" - -# Verify every expected file exists before we build. -for f in "${FILES[@]}"; do - if [ ! -f "$SRC_DIR/$f" ]; then - echo "ERROR: missing $SRC_DIR/$f" >&2 - exit 1 - fi -done - -# Ensure the shell scripts are executable inside the zip. -chmod +x "$SRC_DIR/run.sh" - -mkdir -p "$DIST_DIR" -rm -f "$ZIP_PATH" - -# Stage into a top-level "listener/" directory so the unzip is tidy. -STAGE="$(mktemp -d)" -trap 'rm -rf "$STAGE"' EXIT -mkdir -p "$STAGE/listener" -for f in "${FILES[@]}"; do - cp "$SRC_DIR/$f" "$STAGE/listener/$f" -done -chmod +x "$STAGE/listener/run.sh" - -( cd "$STAGE" && zip -r -q "$ZIP_PATH" listener ) - -echo "Wrote $ZIP_PATH" -unzip -l "$ZIP_PATH" diff --git a/tools/package/build_prgs.sh b/tools/package/build_prgs.sh index 4ccf027..2f4725a 100755 --- a/tools/package/build_prgs.sh +++ b/tools/package/build_prgs.sh @@ -1,102 +1,120 @@ #!/usr/bin/env bash # ============================================================================= -# tools/package/build_prgs.sh - Build the release PRG matrix into dist/. +# tools/package/build_prgs.sh — build the release PRG matrix into dist/. # -# Produces the shippable PRG variants and starts a fresh dist/MANIFEST.txt -# build log (git HEAD, per-variant result + PRG size). The D64 assembly step -# (build_d64.sh) appends to the same manifest and writes the final checksum -# section, so this script MUST run first. +# Four variants, every combination of networking backend x crypto profile: # -# Variants: -# 1. c64-https-uci-reu.prg — make BACKEND=uci (default REU -# profile, nistcurves v0.6.0) -# 2. c64-https-uci-onchip.prg — make BACKEND=uci USE_NISTCURVES_ONCHIP=1 -# (no-REU on-chip verify path) -# 3. ip65/RR-Net — make BACKEND=ip65. Expected to FAIL to link -# at the current nistcurves pin -# (CRYPTO_COLD_SHADOW overflow, tracked as -# c64-nist-curves#54). The exact ld65 error -# is captured to dist/ip65-link-error.txt and -# summarized in the manifest. If it links -# anyway, dist/c64-https-ip65-reu.prg is kept. +# c64-https-uci-reu.prg make BACKEND=uci +# c64-https-uci-onchip.prg make BACKEND=uci USE_NISTCURVES_ONCHIP=1 +# c64-https-ip65-reu.prg make BACKEND=ip65 +# c64-https-ip65-onchip.prg make BACKEND=ip65 USE_NISTCURVES_ONCHIP=1 # -# A `make clean` runs between every flag combination: the build does NOT track -# CA65FLAGS changes, so stale .o files cause spurious unresolved externals. +# The matrix itself lives in _common.sh; this script has no per-variant +# knowledge and nothing version-specific, so it survives a library bump with +# zero edits. +# +# TWO TRAPS this script exists to avoid, both observed in this repo: +# +# 1. `make clean` runs before EVERY variant. `BACKEND=` is not a -D flag, it +# selects an include path, and make's dependency graph cannot see it. +# Skipping the clean yields either a mixed binary at exactly the right +# size, or (macOS GNU Make 3.81, 1-second mtime resolution) no relink at +# all, leaving the *other* variant's PRG in place. Exit 0 either way. +# 2. Every build is checked by PRG sha256, recorded in dist/build-info.txt. +# Object hashes are worthless as evidence — ca65 stamps wall-clock time +# into every .o header — but ld65 does not propagate it, so the PRG is +# deterministic and comparable. +# +# Also bootstraps the ip65 blob (`ip65-build/ip65-c64.bin`), which is a +# gitignored artifact a plain `make` will NOT build: a fresh clone dies in the +# ca65 `.incbin` before any link. +# +# Writes dist/build-info.txt (git HEAD, submodule pins, per-variant args/size/ +# sha256). write_manifest.sh consumes it. # # Usage: tools/package/build_prgs.sh # ============================================================================= -set -eo pipefail +set -euo pipefail -PROJECT_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" cd "$PROJECT_ROOT" - -DIST="$PROJECT_ROOT/dist" -BUILT_PRG="$PROJECT_ROOT/build/c64-https.prg" -MANIFEST="$DIST/MANIFEST.txt" -IP65_ERR="$DIST/ip65-link-error.txt" +# shellcheck source=tools/package/_common.sh +. "$PROJECT_ROOT/tools/package/_common.sh" mkdir -p "$DIST" GIT_HEAD="$(git rev-parse HEAD)" GIT_HEAD_SHORT="$(git rev-parse --short HEAD)" +GIT_DIRTY="" +git diff --quiet HEAD -- 2>/dev/null || GIT_DIRTY=" (working tree DIRTY)" -# Fresh manifest — this script owns the header + build-log section. +# --- ip65 blob bootstrap ------------------------------------------------------ +IP65_BIN="$PROJECT_ROOT/ip65-build/ip65-c64.bin" +if [ ! -f "$IP65_BIN" ]; then + echo "[package] ip65 blob absent — bootstrapping (make ip65-libs && make ip65-blob)" + if [ ! -e "$PROJECT_ROOT/ip65/Makefile" ]; then + echo "ERROR: ip65 submodule not checked out. Run:" >&2 + echo " git submodule update --init --recursive" >&2 + exit 1 + fi + make ip65-libs >/dev/null + make ip65-blob >/dev/null + echo "[package] ip65 blob built: $(wc -c < "$IP65_BIN" | tr -d ' ') bytes, $(sha256_of "$IP65_BIN")" +fi + +# --- Fresh build-info --------------------------------------------------------- { - echo "c64-https release manifest" - echo "generated: $(date -u +%Y-%m-%dT%H:%M:%SZ)" - echo "git HEAD: $GIT_HEAD" - echo - echo "== build variants ==" -} > "$MANIFEST" + echo "# c64-https release build info" + echo "# generated by tools/package/build_prgs.sh" + echo "generated=$(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "git_head=$GIT_HEAD" + echo "git_head_short=$GIT_HEAD_SHORT" + echo "git_dirty=$([ -n "$GIT_DIRTY" ] && echo yes || echo no)" + echo "ip65_blob_sha256=$(sha256_of "$IP65_BIN")" + echo "ip65_blob_bytes=$(wc -c < "$IP65_BIN" | tr -d ' ')" + submodule_pins | while read -r sub sha tag; do + echo "submodule=$sub $sha $tag" + done +} > "$BUILD_INFO" + +# --- Build every variant ------------------------------------------------------ +echo "[package] HEAD $GIT_HEAD_SHORT$GIT_DIRTY" +failed=0 +for line in "${PACKAGE_VARIANTS[@]}"; do + key="$(variant_field "$line" 1)" + prg="$(variant_field "$line" 2)" + args="$(variant_field "$line" 3)" -# build_variant -# Runs `make clean && make `, copies the resulting PRG to dist/ and -# records size + git sha in the manifest. -build_variant() { - local out="$1"; shift - echo "[package] make clean && make $*" + echo "[package] === $key ===" + echo "[package] make clean && make $args" make clean >/dev/null - make "$@" + log="$DIST/build-$key.log" + # Word-splitting $args is intentional — it is a make argument list. + # shellcheck disable=SC2086 + if ! make $args >"$log" 2>&1; then + echo "[package] BUILD FAILED for $key — see $log" >&2 + tail -n 15 "$log" >&2 + echo "variant=$key prg=$prg args=$args result=FAILED log=$(basename "$log")" \ + >> "$BUILD_INFO" + failed=1 + continue + fi if [ ! -f "$BUILT_PRG" ]; then - echo "ERROR: expected $BUILT_PRG after 'make $*', not found" >&2 - exit 1 + echo "[package] ERROR: make $args exited 0 but $BUILT_PRG is missing" >&2 + failed=1 + continue fi - cp "$BUILT_PRG" "$DIST/$out" - local bytes - bytes=$(wc -c < "$DIST/$out" | tr -d ' ') - printf '%-28s %8s bytes make %s (HEAD %s)\n' \ - "$out" "$bytes" "$*" "$GIT_HEAD_SHORT" >> "$MANIFEST" - echo "[package] wrote dist/$out ($bytes bytes)" -} + cp "$BUILT_PRG" "$DIST/$prg" + rm -f "$log" + bytes="$(wc -c < "$DIST/$prg" | tr -d ' ')" + sha="$(sha256_of "$DIST/$prg")" + echo "variant=$key prg=$prg args=$args result=OK bytes=$bytes sha256=$sha" \ + >> "$BUILD_INFO" + printf '[package] wrote dist/%s %s bytes %s\n' "$prg" "$bytes" "$sha" +done -# --- Variant 1: UCI, default REU profile --- -build_variant "c64-https-uci-reu.prg" BACKEND=uci - -# --- Variant 2: UCI, on-chip (no-REU) verify path --- -build_variant "c64-https-uci-onchip.prg" BACKEND=uci USE_NISTCURVES_ONCHIP=1 - -# --- Variant 3: ip65 / RR-Net (expected link failure at current pin) --- -echo "[package] make clean && make BACKEND=ip65 (expected to fail at current nistcurves pin)" -make clean >/dev/null -if make BACKEND=ip65 >"$IP65_ERR" 2>&1; then - # Surprise: it linked. Keep the artifact. - cp "$BUILT_PRG" "$DIST/c64-https-ip65-reu.prg" - bytes=$(wc -c < "$DIST/c64-https-ip65-reu.prg" | tr -d ' ') - printf '%-28s %8s bytes make BACKEND=ip65 (HEAD %s)\n' \ - "c64-https-ip65-reu.prg" "$bytes" "$GIT_HEAD_SHORT" >> "$MANIFEST" - echo "[package] ip65 UNEXPECTEDLY linked — wrote dist/c64-https-ip65-reu.prg ($bytes bytes)" - { echo; echo "== ip65 result =="; echo "ip65 linked successfully (unexpected — see prior campaign notes)."; } >> "$MANIFEST" -else - echo "[package] ip65 build failed as expected — error captured to dist/ip65-link-error.txt" - { - echo - echo "== ip65 result ==" - echo "ip65 FAILED to link (expected; c64-nist-curves#54). Last lines of ld65 output:" - echo "----------------------------------------------------------------" - tail -n 12 "$IP65_ERR" - echo "----------------------------------------------------------------" - echo "(full output: dist/ip65-link-error.txt)" - } >> "$MANIFEST" +if [ "$failed" -ne 0 ]; then + echo "[package] PRG matrix INCOMPLETE — at least one variant failed to build." >&2 + exit 1 fi - -echo "[package] PRG matrix complete." +echo "[package] PRG matrix complete (${#PACKAGE_VARIANTS[@]} variants)." diff --git a/tools/package/listener/README.md b/tools/package/listener/README.md index 74cf787..ce830bc 100644 --- a/tools/package/listener/README.md +++ b/tools/package/listener/README.md @@ -1,23 +1,39 @@ # c64-https TLS 1.3 test listener -A self-contained TLS 1.3 HTTPS listener that stands up the **entire server -side** of the c64-https end-to-end test on a fresh machine — including -minting its own certificate. No dependency on the c64-https repo or the -`c64-test-harness` package. +A TLS 1.3 HTTPS listener that stands up the **entire server side** of the +c64-https end-to-end test on a fresh machine — including minting its own +certificate. No dependency on the c64-https repo, on `c64-test-harness`, or on +any third-party Python package. It is a stand-alone clone of the inline listener in the c64-https repo -(`tools/uci/test_https_local.py`). The protocol behavior is copied -verbatim so the Commodore 64 client sees exactly what it expects. +(`tools/uci/test_https_local.py`). The protocol behavior is copied verbatim so +the Commodore 64 client sees exactly what it expects. + +## Dependencies: none. Requirement: a Python that can do TLS 1.3. + +There is nothing to `pip install` and no venv to create. + +The one thing this cannot supply for itself is a property of the interpreter: +an `ssl` module with TLS 1.3, i.e. one linked against **OpenSSL 1.1.1 or +newer**. macOS's `/usr/bin/python3` is linked against LibreSSL 2.8.3, has no +TLS 1.3 at all, and will refuse with a one-line message — use python.org or +Homebrew `python3` there. Most Linux distributions ship a suitable python3. + +Historically this package needed `cryptography` to generate its certificate. +It no longer does: `gen_certs.py` implements P-256 keygen, DER encoding and +ECDSA-SHA256 signing in pure Python. TLS itself was always the stdlib. That +removal is what lets the whole thing ship as one self-extracting `.py`. ## What it does -- Serves **TLS 1.3 only** (min = max pinned to TLS 1.3). The C64 advertises - a single cipher suite, `TLS_CHACHA20_POLY1305_SHA256` (0x1303), which the +- Serves **TLS 1.3 only** (min = max pinned to TLS 1.3). The C64 advertises a + single cipher suite, `TLS_CHACHA20_POLY1305_SHA256` (0x1303), which the stdlib server offers among its TLS 1.3 defaults and selects. - Presents a self-signed **ECDSA P-256** (`secp256r1`, `ecdsa-with-SHA256`) - leaf certificate. The C64 verifies the CertificateVerify signature against - this leaf key, so a freshly generated self-signed cert is sufficient — - it is *not* a trust anchor and must never be deployed anywhere real. + leaf certificate, freshly generated on first run into `./certs/`. The C64 + verifies the CertificateVerify signature against this leaf key, so a + self-signed cert is sufficient — it is *not* a trust anchor and **must never + be deployed anywhere real**. - Reads one request and replies with the fixed canonical response: ``` @@ -34,37 +50,45 @@ verbatim so the Commodore 64 client sees exactly what it expects. - Writes `server_result.json` (listening / client_addr / request / cipher / error), matching the schema the reference harness emits. +Certs, `server_result.json` and any other output land in the **current +directory**, not next to the script: the shipped single-file build extracts +itself into a throwaway temp dir, so anchoring on `__file__` would hide them. + ## Run it +As the shipped single file: + ```sh -./run.sh # default port 443, auto-fallback to 4433 -./run.sh --port 4433 # unprivileged port, no root needed -./run.sh --serve-forever # keep accepting connections +python3 c64-https-listener.py # port 443, auto-falls back to 4433 +python3 c64-https-listener.py --port 4433 # unprivileged +python3 c64-https-listener.py --serve-forever +python3 c64-https-listener.py --selftest # prove it works, no C64 needed +python3 c64-https-listener.py --extract ./src # unpack these sources +python3 c64-https-listener.py --help ``` -`run.sh` creates a local `.venv`, installs `cryptography` (the only -third-party dependency; the TLS server itself is pure stdlib `ssl`), -generates `certs/server.{pem,key}` if absent, and starts the listener. +Or straight from these sources: -Point the C64 client (or the c64-https test) at this machine's LAN IP on -the chosen port. Default port 443 needs root to bind; if that fails the -listener automatically falls back to 4433. +```sh +python3 listener.py --port 4433 +python3 listener.py --selftest +``` -## Files +## Proving it works without a C64 -| File | Purpose | -|--------------------|-----------------------------------------------------------| -| `run.sh` | One-shot: venv + deps + certs + start listener | -| `listener.py` | The TLS 1.3 listener (stdlib `ssl`) | -| `gen_certs.py` | Mint a fresh P-256 self-signed cert into `certs/` | -| `requirements.txt` | `cryptography` (cert generation only) | +`--selftest` mints a cert into a temp dir, serves on loopback, and drives +itself with a Python `ssl` client, then — where an `openssl s_client` +supporting `-ciphersuites` is available — with a client restricted to +`TLS_CHACHA20_POLY1305_SHA256`, the only suite the C64 offers. That second +round matters: CPython exposes no API to restrict TLS 1.3 suites, so the +stdlib client always picks AES-256-GCM and would never catch a server that +could not speak ChaCha20 to the C64. Where openssl is missing the round is +reported as SKIP, not as a failure. -## Regenerate the certificate +Exit code 0 means PASS. -```sh -python gen_certs.py --force # fresh P-256 cert/key -python gen_certs.py --cn example.test --san example.test -``` +## Errors -By default the CN is `www.foo.bar` with SANs `foo.bar` and `www.foo.bar`, -matching the c64-https test fixtures. +Every failure is one human-readable line, not a traceback: no TLS 1.3 in this +Python, port already in use, unreadable cert. Pass `--debug` to get the +traceback back if you are working on the listener itself. diff --git a/tools/package/listener/gen_certs.py b/tools/package/listener/gen_certs.py index 415798a..7888c92 100755 --- a/tools/package/listener/gen_certs.py +++ b/tools/package/listener/gen_certs.py @@ -1,38 +1,202 @@ #!/usr/bin/env python3 """Generate a fresh self-signed ECDSA P-256 cert for the c64-https listener. -This mirrors the cert the c64-https end-to-end test harness expects: +**Python standard library only.** No `cryptography`, no pip, no venv. + +That is the whole point of this module. The listener is shipped as a single +self-extracting .py file, and `cryptography` is a compiled extension: it +cannot be embedded in a portable source bundle. Rather than bootstrap a venv +on first run (network required, minutes of pip, and a new class of failure on +every machine that has a broken toolchain), the one thing that actually needed +`cryptography` — minting a self-signed P-256 certificate — is done here in +pure Python. TLS itself was always stdlib `ssl`. + +What that leaves as the listener's real requirement is a property of the +*interpreter*, not of any installable package: an `ssl` module with TLS 1.3 +(OpenSSL 1.1.1+). See listener.py, which checks for it and says so in one line. + +The certificate this produces is byte-shaped like the one the previous +`cryptography`-based generator emitted, deliberately: * key : ECDSA on NIST P-256 (secp256r1 / prime256v1) * sig : ecdsa-with-SHA256 * CN : www.foo.bar (overridable via --cn) * SAN : foo.bar, www.foo.bar (overridable via --san, repeatable) - * valid : 10 years from now + * valid : now-5min .. now+3650 days, UTCTime + * exts : subjectAltName ONLY, non-critical -The C64 client verifies the server's CertificateVerify signature against -the leaf public key, so any freshly generated P-256 self-signed cert works -— there is no trust-anchor requirement. Do NOT deploy these anywhere real. +That last line is a constraint, not an oversight. The C64 client parses this +certificate with a hand-written 6502 DER walker, and the extension set above +is the one it has been validated against end to end. Do not add +basicConstraints/keyUsage/EKU here "for correctness" without re-running the +hardware e2e — this cert is a test fixture, never a trust anchor, and must not +be deployed anywhere real. -Idempotent: by default it refuses to overwrite existing files (pass ---force to regenerate). Writes into ./certs/ next to this script unless ---out-dir is given. +Idempotent: refuses to overwrite existing files unless --force. Writes into +./certs/ relative to the current directory unless --out-dir is given. """ from __future__ import annotations import argparse import datetime +import hashlib +import secrets import sys from pathlib import Path -from cryptography import x509 -from cryptography.hazmat.primitives import hashes, serialization -from cryptography.hazmat.primitives.asymmetric import ec -from cryptography.x509.oid import NameOID +# --------------------------------------------------------------------------- +# NIST P-256 (secp256r1) domain parameters — SEC 2 / FIPS 186-4. +# --------------------------------------------------------------------------- +P = 0xFFFFFFFF00000001000000000000000000000000FFFFFFFFFFFFFFFFFFFFFFFF +A = P - 3 +B = 0x5AC635D8AA3A93E7B3EBBD55769886BC651D06B0CC53B0F63BCE3C3E27D2604B +N = 0xFFFFFFFF00000000FFFFFFFFFFFFFFFFBCE6FAADA7179E84F3B9CAC2FC632551 +GX = 0x6B17D1F2E12C4247F8BCE6E563A440F277037D812DEB33A0F4A13945D898C296 +GY = 0x4FE342E2FE1A7F9B8EE7EB4A7C0F9E162BCE33576B315ECECBB6406837BF51F5 + +# Affine point arithmetic. None is the point at infinity. This runs a handful +# of times per invocation (one keygen, one signature), so clarity beats speed. + + +def _inv(x: int, m: int) -> int: + return pow(x, m - 2, m) + + +def _add(p1, p2): + if p1 is None: + return p2 + if p2 is None: + return p1 + x1, y1 = p1 + x2, y2 = p2 + if x1 == x2: + if (y1 + y2) % P == 0: + return None + lam = (3 * x1 * x1 + A) * _inv(2 * y1, P) % P + else: + lam = (y2 - y1) * _inv(x2 - x1, P) % P + x3 = (lam * lam - x1 - x2) % P + return (x3, (lam * (x1 - x3) - y1) % P) + + +def _mul(k: int, point): + """Double-and-add. Not constant time — this is a test fixture minting a + throwaway key on the operator's own machine, not a production signer.""" + result = None + addend = point + while k: + if k & 1: + result = _add(result, addend) + addend = _add(addend, addend) + k >>= 1 + return result + + +# --------------------------------------------------------------------------- +# Minimal DER encoder. Every helper returns a complete TLV. +# --------------------------------------------------------------------------- + +def _der_len(n: int) -> bytes: + if n < 0x80: + return bytes([n]) + body = n.to_bytes((n.bit_length() + 7) // 8, "big") + return bytes([0x80 | len(body)]) + body + + +def _tlv(tag: int, body: bytes) -> bytes: + return bytes([tag]) + _der_len(len(body)) + body + + +def _int(value: int) -> bytes: + body = value.to_bytes((value.bit_length() + 8) // 8 or 1, "big") + # Positive INTEGERs need a leading 0x00 when the top bit would read as a + # sign bit; the +8 above already reserves that byte, so this is minimal. + return _tlv(0x02, body) + + +def _oid(dotted: str) -> bytes: + parts = [int(x) for x in dotted.split(".")] + body = bytes([40 * parts[0] + parts[1]]) + for arc in parts[2:]: + chunk = bytearray([arc & 0x7F]) + arc >>= 7 + while arc: + chunk.insert(0, (arc & 0x7F) | 0x80) + arc >>= 7 + body += bytes(chunk) + return _tlv(0x06, body) + + +def _seq(*items: bytes) -> bytes: + return _tlv(0x30, b"".join(items)) + + +def _set(*items: bytes) -> bytes: + return _tlv(0x31, b"".join(items)) + +def _bitstring(data: bytes, unused: int = 0) -> bytes: + return _tlv(0x03, bytes([unused]) + data) -def generate(cn: str, sans: list[str], out_dir: Path, - force: bool = False) -> tuple[Path, Path]: + +def _octetstring(data: bytes) -> bytes: + return _tlv(0x04, data) + + +def _utf8(text: str) -> bytes: + return _tlv(0x0C, text.encode("utf-8")) + + +def _ia5(text: str) -> bytes: + return _tlv(0x16, text.encode("ascii")) + + +def _utctime(when: datetime.datetime) -> bytes: + # UTCTime is YYMMDDHHMMSSZ and is only valid through 2049; the caller's + # 10-year validity keeps us well inside that. + if when.year >= 2050: + raise ValueError("date beyond UTCTime range; GeneralizedTime needed") + return _tlv(0x17, when.strftime("%y%m%d%H%M%SZ").encode("ascii")) + + +def _explicit(num: int, body: bytes) -> bytes: + return _tlv(0xA0 | num, body) + + +OID_EC_PUBLIC_KEY = "1.2.840.10045.2.1" +OID_PRIME256V1 = "1.2.840.10045.3.1.7" +OID_ECDSA_SHA256 = "1.2.840.10045.4.3.2" +OID_COMMON_NAME = "2.5.4.3" +OID_SUBJECT_ALT_NAME = "2.5.29.17" + + +def _pem(label: str, der: bytes) -> bytes: + import base64 + b64 = base64.b64encode(der).decode("ascii") + lines = [b64[i:i + 64] for i in range(0, len(b64), 64)] + return ("-----BEGIN %s-----\n%s\n-----END %s-----\n" + % (label, "\n".join(lines), label)).encode("ascii") + + +def _ecdsa_sign(digest: bytes, d: int) -> bytes: + """ECDSA-SHA256 over P-256. Returns the DER SEQUENCE{r,s}.""" + e = int.from_bytes(digest, "big") # SHA-256 and n are both 256 bits + while True: + k = secrets.randbelow(N - 1) + 1 + point = _mul(k, (GX, GY)) + r = point[0] % N + if r == 0: + continue + s = _inv(k, N) * (e + r * d) % N + if s == 0: + continue + return _seq(_int(r), _int(s)) + + +def generate(cn: str, sans: list, out_dir: Path, + force: bool = False): """Generate key + self-signed cert into out_dir. Returns (cert, key).""" + out_dir = Path(out_dir) out_dir.mkdir(parents=True, exist_ok=True) cert_path = out_dir / "server.pem" key_path = out_dir / "server.key" @@ -42,46 +206,71 @@ def generate(cn: str, sans: list[str], out_dir: Path, f"(use --force to regenerate)") return cert_path, key_path - # ECDSA P-256 private key (matches tools/https_e2e/certs/server.key). - key = ec.generate_private_key(ec.SECP256R1()) - - name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, cn)]) - san = x509.SubjectAlternativeName([x509.DNSName(h) for h in sans]) + # --- key --- + d = secrets.randbelow(N - 1) + 1 + qx, qy = _mul(d, (GX, GY)) + pub_point = b"\x04" + qx.to_bytes(32, "big") + qy.to_bytes(32, "big") + # --- names / validity --- + name = _seq(_set(_seq(_oid(OID_COMMON_NAME), _utf8(cn)))) now = datetime.datetime.now(datetime.timezone.utc) - cert = ( - x509.CertificateBuilder() - .subject_name(name) - .issuer_name(name) # self-signed - .public_key(key.public_key()) - .serial_number(x509.random_serial_number()) - .not_valid_before(now - datetime.timedelta(minutes=5)) - .not_valid_after(now + datetime.timedelta(days=3650)) - .add_extension(san, critical=False) - # Sign with SHA-256 -> ecdsa-with-SHA256, matching the P-256 profile. - .sign(key, hashes.SHA256()) + not_before = now - datetime.timedelta(minutes=5) + not_after = now + datetime.timedelta(days=3650) + + sig_alg = _seq(_oid(OID_ECDSA_SHA256)) + spki = _seq( + _seq(_oid(OID_EC_PUBLIC_KEY), _oid(OID_PRIME256V1)), + _bitstring(pub_point), ) + # GeneralNames: dNSName is [2] IMPLICIT IA5String, i.e. tag 0x82. + san_value = _seq(*[_tlv(0x82, h.encode("ascii")) for h in sans]) + extensions = _explicit(3, _seq( + _seq(_oid(OID_SUBJECT_ALT_NAME), _octetstring(san_value)), + )) - key_path.write_bytes( - key.private_bytes( - encoding=serialization.Encoding.PEM, - format=serialization.PrivateFormat.TraditionalOpenSSL, - encryption_algorithm=serialization.NoEncryption(), - ) + tbs = _seq( + _explicit(0, _int(2)), # version v3 + _int(secrets.randbits(159) | 1), # positive serial, <20 B + sig_alg, + name, # issuer == subject + _seq(_utctime(not_before), _utctime(not_after)), + name, + spki, + extensions, ) - cert_path.write_bytes(cert.public_bytes(serialization.Encoding.PEM)) + + signature = _ecdsa_sign(hashlib.sha256(tbs).digest(), d) + cert_der = _seq(tbs, sig_alg, _bitstring(signature)) + + # SEC1 / RFC 5915 ECPrivateKey — "EC PRIVATE KEY" PEM, which is what the + # previous generator's TraditionalOpenSSL format produced. + key_der = _seq( + _int(1), + _octetstring(d.to_bytes(32, "big")), + _explicit(0, _oid(OID_PRIME256V1)), + _explicit(1, _bitstring(pub_point)), + ) + + key_path.write_bytes(_pem("EC PRIVATE KEY", key_der)) + try: + key_path.chmod(0o600) + except OSError: + pass + cert_path.write_bytes(_pem("CERTIFICATE", cert_der)) print(f"wrote {cert_path}") print(f"wrote {key_path}") print(f" CN = {cn}") print(f" SAN = {', '.join(sans)}") - print(f" key = ECDSA P-256 (secp256r1), sig = ecdsa-with-SHA256") + print(" key = ECDSA P-256 (secp256r1), sig = ecdsa-with-SHA256") + print(" (generated with the Python stdlib only — no 'cryptography')") return cert_path, key_path -def main(argv: list[str] | None = None) -> int: - p = argparse.ArgumentParser(description=__doc__, - formatter_class=argparse.RawDescriptionHelpFormatter) +def main(argv=None) -> int: + p = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) p.add_argument("--cn", default="www.foo.bar", help="certificate Common Name (default: www.foo.bar)") p.add_argument("--san", action="append", default=None, metavar="DNS", @@ -89,14 +278,13 @@ def main(argv: list[str] | None = None) -> int: "default: foo.bar and www.foo.bar)") p.add_argument("--out-dir", default=None, help="output directory for server.pem/server.key " - "(default: ./certs next to this script)") + "(default: ./certs)") p.add_argument("--force", action="store_true", help="overwrite existing cert/key") args = p.parse_args(argv) sans = args.san if args.san else ["foo.bar", "www.foo.bar"] - out_dir = (Path(args.out_dir) if args.out_dir - else Path(__file__).resolve().parent / "certs") + out_dir = Path(args.out_dir) if args.out_dir else Path.cwd() / "certs" generate(args.cn, sans, out_dir, force=args.force) return 0 diff --git a/tools/package/listener/listener.py b/tools/package/listener/listener.py index e0bb33e..ffcaa46 100755 --- a/tools/package/listener/listener.py +++ b/tools/package/listener/listener.py @@ -29,6 +29,11 @@ * Writes server_result.json with the same schema the reference harness emits (listening / client_addr / request / error). +DEPENDENCIES: the Python standard library, and nothing else. Cert generation +is pure-Python P-256 (see gen_certs.py, which explains why). The one thing +this cannot supply for itself is a TLS 1.3-capable ``ssl`` module, which is a +property of the interpreter — checked below, reported in one line. + Environment / CLI: HTTPS_PORT env or --port listener port. Default 443, auto-falls back to 4433 if the privileged bind fails (clones the @@ -36,12 +41,19 @@ --bind bind address (default 0.0.0.0 so the C64 on the LAN can reach it; the reference binds the dev host's LAN IP). - --cert / --key cert + key paths (default ./certs/server.{pem,key}). + --cert / --key cert + key paths (default ./certs/server.{pem,key}, + relative to the CURRENT DIRECTORY — the single-file + build runs from a throwaway temp dir, so anchoring + on __file__ would hide the certs). --result server_result.json path (default ./server_result.json). --accept-timeout accept + per-connection timeout seconds (default 600). --serve-forever keep accepting connections instead of exiting after the first (convenience; off by default to match the reference one-shot behavior). + --selftest prove the whole thing works without a C64: mint + certs, serve on loopback, connect with a Python + ``ssl`` client, assert TLS 1.3 + the canonical + response. Exits 0 on PASS. """ from __future__ import annotations @@ -49,15 +61,16 @@ import base64 import json import os +import shutil import socket import ssl +import subprocess import sys +import threading import time import traceback from pathlib import Path -HERE = Path(__file__).resolve().parent - # --- Canonical response bytes (cloned verbatim from test_https_local.py) --- EXPECTED_BODY = "HELLO FROM TLS SERVER" HTTP_RESPONSE = ( @@ -76,7 +89,7 @@ def _ensure_certs(cert_path: Path, key_path: Path) -> None: if cert_path.is_file() and key_path.is_file(): return print(f"cert/key missing ({cert_path} / {key_path}); generating...") - # Import lazily so `listener.py --help` works without cryptography. + sys.path.insert(0, str(Path(__file__).resolve().parent)) import gen_certs gen_certs.generate( cn="www.foo.bar", @@ -211,6 +224,152 @@ def _handle_one(srv: socket.socket, ctx: ssl.SSLContext, pass +def selftest() -> int: + """Prove the listener works end to end with no C64 and no network. + + Mints a fresh cert into a temp dir, serves on loopback, and drives itself + with a Python ``ssl`` client — the same shape as + ``tools/https_e2e/evil_listener.py --selftest`` — then again with a client + pinned to the C64's single cipher suite. This is what makes the shipped + single-file artifact checkable by whoever cuts the release. + """ + import tempfile + print("== c64-https listener selftest ==") + with tempfile.TemporaryDirectory(prefix="c64-listener-selftest-") as tmp: + tmp_path = Path(tmp) + cert_path = tmp_path / "certs" / "server.pem" + key_path = tmp_path / "certs" / "server.key" + _ensure_certs(cert_path, key_path) + ctx = _make_ssl_context(cert_path, key_path) + + srv = _try_bind("127.0.0.1", 0) + if srv is None: + print("FAIL: could not bind 127.0.0.1:0") + return 1 + port = srv.getsockname()[1] + srv.listen(1) + print(f" server listening on 127.0.0.1:{port}") + + def one_round(label: str): + """Serve one connection, drive it with a client, return findings.""" + result: dict = {"listening": True, "port": port} + server_exc: list = [] + + def _serve(): + try: + _handle_one(srv, ctx, 30.0, result) + except BaseException as exc: # noqa: BLE001 - reported below + server_exc.append(exc) + + thread = threading.Thread(target=_serve, daemon=True) + thread.start() + + client_ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + client_ctx.check_hostname = False + client_ctx.verify_mode = ssl.CERT_REQUIRED + client_ctx.load_verify_locations(cafile=str(cert_path)) + got = b"" + version = cipher = None + try: + with socket.create_connection(("127.0.0.1", port), + timeout=30) as raw: + with client_ctx.wrap_socket(raw) as tls: + version = tls.version() + cipher = tls.cipher()[0] + print(f" [{label}] client handshake: " + f"{version} / {cipher}") + tls.sendall( + b"GET / HTTP/1.1\r\nHost: www.foo.bar\r\n\r\n") + while len(got) < len(HTTP_RESPONSE): + chunk = tls.recv(4096) + if not chunk: + break + got += chunk + except Exception as exc: # noqa: BLE001 + return {"error": f"client: {type(exc).__name__}: {exc}"} + thread.join(timeout=30) + if server_exc: + exc = server_exc[0] + return {"error": f"server: {type(exc).__name__}: {exc}"} + return {"version": version, "cipher": cipher, "body": got, + "request": result.get("request")} + + checks: list = [] + + # Round 1 — a stock Python client: does this thing serve at all. + out = one_round("stdlib client") + if "error" in out: + print(f" [FAIL] stdlib client: {out['error']}") + print("SELFTEST FAILED") + srv.close() + return 1 + checks += [ + ("stdlib client: TLS 1.3 negotiated", out["version"] == "TLSv1.3"), + ("stdlib client: server saw a request", + bool(out["request"]) and out["request"] != b""), + ("stdlib client: canonical response received", + out["body"] == HTTP_RESPONSE), + ] + + # Round 2 — force TLS_CHACHA20_POLY1305_SHA256, the ONLY suite the C64 + # offers (src/tls_handshake.s, 0x1303). Round 1 cannot cover this: the + # stdlib client prefers AES-256-GCM, and CPython exposes no API to + # restrict TLS 1.3 suites (set_ciphers() drives SSL_CTX_set_cipher_list, + # which TLS 1.3 ignores). So this round shells out to `openssl + # s_client -ciphersuites`, and SKIPs rather than fails where that is + # unavailable — a missing openssl says nothing about the listener. + openssl = shutil.which("openssl") + have_flag = False + if openssl: + probe = subprocess.run([openssl, "s_client", "-help"], + capture_output=True, text=True) + have_flag = "-ciphersuites" in (probe.stdout + probe.stderr) + if not have_flag: + print(" [SKIP] C64 suite TLS_CHACHA20_POLY1305_SHA256: needs an " + "`openssl s_client` supporting -ciphersuites") + else: + server_exc: list = [] + result2: dict = {} + + def _serve2(): + try: + _handle_one(srv, ctx, 30.0, result2) + except BaseException as exc: # noqa: BLE001 + server_exc.append(exc) + + t2 = threading.Thread(target=_serve2, daemon=True) + t2.start() + proc = subprocess.run( + [openssl, "s_client", "-connect", f"127.0.0.1:{port}", + "-tls1_3", "-ciphersuites", "TLS_CHACHA20_POLY1305_SHA256", + "-CAfile", str(cert_path), "-servername", "www.foo.bar", + "-quiet", "-ign_eof"], + input=b"GET / HTTP/1.1\r\nHost: www.foo.bar\r\n\r\n", + capture_output=True, timeout=60) + t2.join(timeout=30) + blob = proc.stdout + proc.stderr + if server_exc: + print(f" [FAIL] C64 suite round, server side: {server_exc[0]}") + checks.append(("C64 suite TLS_CHACHA20_POLY1305_SHA256", False)) + else: + checks.append( + ("C64 suite TLS_CHACHA20_POLY1305_SHA256 negotiates and " + "gets the canonical body", + EXPECTED_BODY.encode() in blob + and result2.get("cipher") == "TLS_CHACHA20_POLY1305_SHA256")) + srv.close() + + ok = True + for label, passed in checks: + print(f" [{'PASS' if passed else 'FAIL'}] {label}") + ok = ok and passed + if not ok: + print("SELFTEST FAILED") + return 1 + print(f"SELFTEST PASSED ({len(checks)} checks)") + return 0 + + def main(argv: list[str] | None = None) -> int: p = argparse.ArgumentParser( description="TLS 1.3 test listener for the c64-https client.") @@ -219,18 +378,31 @@ def main(argv: list[str] | None = None) -> int: f"auto-falls back to {FALLBACK_PORT})") p.add_argument("--bind", default="0.0.0.0", help="bind address (default 0.0.0.0)") - p.add_argument("--cert", default=str(HERE / "certs" / "server.pem"), + p.add_argument("--cert", default=None, help="server cert PEM (default ./certs/server.pem)") - p.add_argument("--key", default=str(HERE / "certs" / "server.key"), + p.add_argument("--key", default=None, help="server key PEM (default ./certs/server.key)") - p.add_argument("--result", default=str(HERE / "server_result.json"), + p.add_argument("--result", default=None, help="server_result.json path (default ./server_result.json)") p.add_argument("--accept-timeout", type=float, default=600.0, help="accept + per-connection timeout seconds (default 600)") p.add_argument("--serve-forever", action="store_true", help="keep serving after the first connection") + p.add_argument("--selftest", action="store_true", + help="run a loopback TLS 1.3 self-test and exit") args = p.parse_args(argv) + if args.selftest: + return selftest() + + # Paths default relative to the CURRENT DIRECTORY, not __file__: the + # shipped single-file build extracts itself into a throwaway temp dir, and + # certs written next to __file__ there would vanish on exit. + cwd = Path.cwd() + args.cert = args.cert or str(cwd / "certs" / "server.pem") + args.key = args.key or str(cwd / "certs" / "server.key") + args.result = args.result or str(cwd / "server_result.json") + cert_path = Path(args.cert) key_path = Path(args.key) result_path = Path(args.result) @@ -287,5 +459,39 @@ def main(argv: list[str] | None = None) -> int: return exit_code +def cli() -> int: + """Entry point that never shows the operator a traceback. + + A stack trace is the wrong failure mode for a single-file artifact handed + to someone who does not want to build anything: every plausible failure + here (no TLS 1.3 in this Python, port in use, unreadable cert) has a + one-line explanation, and the trace only buries it. ``--debug`` puts the + traceback back for anyone who is actually debugging this file. + """ + debug = "--debug" in sys.argv + argv = [a for a in sys.argv[1:] if a != "--debug"] + try: + return main(argv) + except SystemExit: + # argparse and the TLS 1.3 check exit this way, already having said + # something readable. Pass it through untouched. + raise + except KeyboardInterrupt: + print("\ninterrupted") + return 130 + except OSError as exc: + print(f"ERROR: {exc.strerror or exc} " + f"({getattr(exc, 'filename', None) or 'listener'})", + file=sys.stderr) + except Exception as exc: # noqa: BLE001 + print(f"ERROR: {type(exc).__name__}: {exc}", file=sys.stderr) + if debug: + traceback.print_exc() + else: + print(" (re-run with --debug for the full traceback)", + file=sys.stderr) + return 1 + + if __name__ == "__main__": - sys.exit(main()) + sys.exit(cli()) diff --git a/tools/package/listener/requirements.txt b/tools/package/listener/requirements.txt deleted file mode 100644 index 373b26d..0000000 --- a/tools/package/listener/requirements.txt +++ /dev/null @@ -1,5 +0,0 @@ -# c64-https test listener dependencies. -# The TLS server itself uses only the Python stdlib `ssl` module; the sole -# third-party requirement is `cryptography`, used by gen_certs.py to mint -# the self-signed ECDSA P-256 certificate. -cryptography>=41.0 diff --git a/tools/package/listener/run.sh b/tools/package/listener/run.sh deleted file mode 100755 index 77b9c99..0000000 --- a/tools/package/listener/run.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash -# One-shot bootstrap for the c64-https TLS 1.3 test listener. -# -# Creates a local virtualenv if needed, installs the (minimal) deps, -# generates a fresh self-signed ECDSA P-256 cert if one isn't present, -# and starts the listener. Works on a fresh machine with only a system -# `python3` (>=3.8) available. -# -# All arguments are passed straight through to listener.py, e.g.: -# ./run.sh --port 4433 -# ./run.sh --bind 0.0.0.0 --serve-forever -set -euo pipefail - -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -cd "$HERE" - -PYTHON="${PYTHON:-python3}" -VENV_DIR="${VENV_DIR:-$HERE/.venv}" - -if ! command -v "$PYTHON" >/dev/null 2>&1; then - echo "ERROR: '$PYTHON' not found on PATH. Install Python 3.8+ or set PYTHON=..." >&2 - exit 1 -fi - -if [ ! -d "$VENV_DIR" ]; then - echo "Creating virtualenv at $VENV_DIR ..." - "$PYTHON" -m venv "$VENV_DIR" -fi - -# shellcheck disable=SC1091 -source "$VENV_DIR/bin/activate" - -echo "Installing dependencies ..." -python -m pip install --upgrade pip >/dev/null -python -m pip install -r "$HERE/requirements.txt" - -echo "Ensuring certificate ..." -python "$HERE/gen_certs.py" - -echo "Starting listener ..." -exec python "$HERE/listener.py" "$@" diff --git a/tools/package/verify_release.py b/tools/package/verify_release.py new file mode 100755 index 0000000..9e64082 --- /dev/null +++ b/tools/package/verify_release.py @@ -0,0 +1,309 @@ +#!/usr/bin/env python3 +"""tools/package/verify_release.py — acceptance gate for the release artifacts. + +Run after `make package`. Measures three things and prints the evidence; it +does not assert anything it has not just observed. + + 1. REPRODUCIBILITY. Rebuilds every PRG variant a second time from clean and + compares **PRG** sha256 against dist/build-info.txt. Object hashes are + deliberately not compared: ca65 stamps wall-clock time into every .o + header, so nobody can reproduce their own object hash twice. ld65 does + not propagate that field, which is exactly what makes the PRG comparable. + + 2. DISK IMAGES. Extracts each .d64's contents with c1541 and byte-compares + them to the dist PRGs, then boots every image in VICE and asserts the + banner. ip65 images will print NETWORK INIT FAILED without a network — + that is expected and is NOT part of the pass criteria; the banner is. + + 3. LISTENER. Runs the built single-file listener's own --selftest from a + clean temp directory with no venv, which mints a cert and drives the + server with a Python ssl client. + +Environment: + SKIP_REBUILD=1 skip check 1 (it costs four full builds) + SKIP_VICE=1 skip the VICE boots (keeps the c1541 byte-compare) + SKIP_LISTENER=1 skip check 3 + VICE_BOOT_TIMEOUT seconds to wait for the menu (default 180) +""" +from __future__ import annotations + +import os +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent.parent +DIST = REPO_ROOT / "dist" +BUILD_INFO = DIST / "build-info.txt" +BUILT_PRG = REPO_ROOT / "build" / "c64-https.prg" + +sys.path.insert(0, str(REPO_ROOT / "tools")) + +COMMON_BANNER = "C64-HTTPS CLIENT V0.1" +MENU_MARKER = "Q=QUIT" +# src/net/ip65/net_banner.s and src/net/uci/net.s respectively. +BACKEND_BANNERS = {"uci": "UCI NETWORKING", "ip65": "RR-NET (CS8900A) ETHERNET"} + +results: list[tuple[str, bool, str]] = [] + + +def record(name: str, ok: bool, detail: str = "") -> bool: + results.append((name, ok, detail)) + print(f" [{'PASS' if ok else 'FAIL'}] {name}" + (f" — {detail}" if detail else "")) + return ok + + +def sha256_of(path: Path) -> str: + import hashlib + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def parse_build_info() -> list[dict]: + """Return the per-variant records build_prgs.sh left behind.""" + if not BUILD_INFO.is_file(): + sys.exit(f"ERROR: {BUILD_INFO} missing — run `make package` first.") + variants = [] + for line in BUILD_INFO.read_text().splitlines(): + if not line.startswith("variant="): + continue + rec: dict = {} + # args= holds spaces, so peel the fixed keys off either end. + head, _, rest = line.partition(" prg=") + rec["key"] = head[len("variant="):] + prg, _, rest = rest.partition(" args=") + rec["prg"] = prg + args, _, rest = rest.partition(" result=") + rec["args"] = args + # rest is " [bytes=N sha256=H]" — the result value is bare. + fields = rest.split() + rec["result"] = fields[0] if fields else "" + for kv in fields[1:]: + k, _, v = kv.partition("=") + rec[k] = v + variants.append(rec) + return variants + + +# --------------------------------------------------------------------------- +# 1. PRG reproducibility +# --------------------------------------------------------------------------- + +def check_reproducible(variants: list[dict]) -> None: + print("\n=== 1. PRG byte-reproducibility (second build from clean) ===") + for rec in variants: + if rec.get("result") != "OK": + record(f"{rec['key']} rebuild", False, "first build had already failed") + continue + subprocess.run(["make", "clean"], cwd=REPO_ROOT, check=True, + stdout=subprocess.DEVNULL) + proc = subprocess.run(["make"] + rec["args"].split(), cwd=REPO_ROOT, + capture_output=True, text=True) + if proc.returncode != 0: + record(f"{rec['key']} rebuild", False, + f"make failed: {proc.stderr.strip().splitlines()[-1:]}") + continue + again = sha256_of(BUILT_PRG) + record(f"{rec['key']} reproduces", + again == rec["sha256"], + f"{again[:16]}… vs {rec['sha256'][:16]}…") + + +# --------------------------------------------------------------------------- +# 2. Disk images +# --------------------------------------------------------------------------- + +def d64_images() -> list[Path]: + return sorted(DIST.glob("*.d64")) + + +def check_d64_contents(variants: list[dict]) -> None: + """Read each PRG back out of each disk and byte-compare it.""" + print("\n=== 2a. D64 contents (c1541 read-back, byte-compare) ===") + c1541 = os.environ.get("C1541", "c1541") + if not shutil.which(c1541): + record("c1541 available", False, "not on PATH") + return + by_prg = {r["prg"]: r for r in variants} + for image in d64_images(): + listing = subprocess.run([c1541, "-attach", str(image), "-list"], + capture_output=True, text=True).stdout + names = [ln.split('"')[1] for ln in listing.splitlines() + if ln.count('"') >= 2 and " prg " in ln.lower()] + if not names: + record(f"{image.name} has files", False, "no PRG entries in directory") + continue + with tempfile.TemporaryDirectory() as tmp: + ok = True + detail = [] + for name in names: + out = Path(tmp) / f"{name}.prg" + subprocess.run([c1541, "-attach", str(image), + "-read", f"{name},p", str(out)], + capture_output=True, text=True) + if not out.is_file(): + ok = False + detail.append(f"{name}: unreadable") + continue + # Match it against whichever dist PRG has the same bytes. + got = sha256_of(out) + match = [p for p, r in by_prg.items() + if r.get("sha256") == got] + if match: + detail.append(f"{name} == {match[0]}") + else: + ok = False + detail.append(f"{name}: {got[:12]}… matches no dist PRG") + record(f"{image.name} carries the built PRGs", ok, "; ".join(detail)) + + +def check_d64_boots() -> None: + """Autostart every disk image in VICE and assert the boot banner. + + Two flags are load-bearing, and both were found the hard way: + + -trapdevice8 +drive8truedrive — use the KERNAL load traps instead of + true drive emulation. Under TDE the ~250-block serial load of a 63 KB + PRG does not finish inside any budget worth waiting for, and the + symptom (a screen frozen on LOADING) reads as a corrupt image rather + than a slow one. The image's contents are byte-compared separately in + 2a, so nothing is lost by loading it fast. + + The pass criterion is the BANNER, not the menu. Two reasons, both + measured. The ip65 images print NETWORK INIT FAILED with no network + attached, which is expected and says nothing about the image. And the menu + is not reachable in a test-shaped budget: boot's table init runs at + emulated 1 MHz because VICE 3.10 has no usable warp, and a 900 s probe on + c64-https-uci-reu.d64 saw the banner at 6.0 s and never reached Q=QUIT. + Whether the menu appeared is reported as extra information, never as the + verdict. + """ + print("\n=== 2b. D64 boots to the banner in VICE ===") + try: + from c64_test_harness import ViceInstanceManager + from c64_test_harness.screen import ScreenGrid + from _vice_helpers import default_vice_config + except Exception as exc: # noqa: BLE001 + record("c64_test_harness importable", False, f"{type(exc).__name__}: {exc}") + return + timeout = float(os.environ.get("VICE_BOOT_TIMEOUT", "240")) + import time + for image in d64_images(): + # Backend is in the filename by construction (see _common.sh); the + # per-backend disks autostart their first file, which is that + # backend's REU profile. + backend = "uci" if "-uci" in image.name else "ip65" + expected = BACKEND_BANNERS[backend] + foreign_banners = {k: v for k, v in BACKEND_BANNERS.items() + if k != backend} + # -autostart on a .d64 loads and runs the first program on the disk. + config = default_vice_config(prg_path=str(image), warp=True, + ntsc=True, sound=False, + extra_args=["-trapdevice8", + "+drive8truedrive"]) + seen: dict[str, float] = {} + needles = [COMMON_BANNER, expected, MENU_MARKER] + list(foreign_banners.values()) + try: + with ViceInstanceManager(config=config) as mgr: + inst = mgr.acquire() + start = time.monotonic() + while time.monotonic() - start < timeout: + time.sleep(5.0) + try: + inst.transport.resume() + text = ScreenGrid.from_transport( + inst.transport).continuous_text().upper() + except Exception: # noqa: BLE001 + continue + for needle in needles: + if needle in text and needle not in seen: + seen[needle] = time.monotonic() - start + # Stop as soon as the verdict is decided. The banner is + # printed in one pass, so a foreign backend line cannot + # appear after the expected one; waiting on for the menu + # would add minutes of emulated table-init per image + # without changing the answer. + if COMMON_BANNER in seen and expected in seen: + break + if MENU_MARKER in seen: + break + mgr.release(inst) + except Exception as exc: # noqa: BLE001 + record(f"{image.name} boots", False, f"{type(exc).__name__}: {exc}") + continue + foreign = [v for v in foreign_banners.values() if v in seen] + ok = COMMON_BANNER in seen and expected in seen and not foreign + detail = ", ".join(f"{n!r} at {t:.0f}s" for n, t in sorted( + seen.items(), key=lambda kv: kv[1])) + if not detail: + detail = f"nothing recognisable on screen within {timeout:.0f}s" + if foreign: + detail += f"; UNEXPECTED foreign banner {foreign}" + record(f"{image.name} boots to the banner", ok, detail) + + +# --------------------------------------------------------------------------- +# 3. Listener +# --------------------------------------------------------------------------- + +def check_listener() -> None: + print("\n=== 3. Single-file listener selftest (clean temp dir, no venv) ===") + bundle = DIST / "c64-https-listener.py" + if not bundle.is_file(): + record("listener bundle present", False, f"{bundle} missing") + return + import ssl + if not getattr(ssl, "HAS_TLSv1_3", False): + record("listener selftest", False, + f"this interpreter has no TLS 1.3 ({ssl.OPENSSL_VERSION}); " + "re-run with PACKAGE_PYTHON=") + return + with tempfile.TemporaryDirectory(prefix="c64-listener-verify-") as tmp: + proc = subprocess.run([sys.executable, str(bundle), "--selftest"], + cwd=tmp, capture_output=True, text=True) + for ln in proc.stdout.splitlines(): + print(f" {ln}") + record("listener selftest", proc.returncode == 0, + f"exit {proc.returncode}") + # The bundle must not have left anything behind in the working dir. + leftovers = sorted(p.name for p in Path(tmp).iterdir()) + record("selftest leaves no droppings", not leftovers, + f"found {leftovers}" if leftovers else "clean") + + +def main() -> int: + variants = parse_build_info() + print(f"Verifying {len(variants)} PRG variants and " + f"{len(d64_images())} disk images in {DIST}") + + if os.environ.get("SKIP_REBUILD") != "1": + check_reproducible(variants) + else: + print("\n=== 1. PRG reproducibility SKIPPED (SKIP_REBUILD=1) ===") + + check_d64_contents(variants) + if os.environ.get("SKIP_VICE") != "1": + check_d64_boots() + else: + print("\n=== 2b. VICE boots SKIPPED (SKIP_VICE=1) ===") + + if os.environ.get("SKIP_LISTENER") != "1": + check_listener() + else: + print("\n=== 3. Listener SKIPPED (SKIP_LISTENER=1) ===") + + failed = [n for n, ok, _ in results if not ok] + print(f"\n{'=' * 60}") + print(f"{len(results) - len(failed)}/{len(results)} checks passed") + if failed: + print("FAILED:") + for name in failed: + print(f" - {name}") + return 1 + print("RELEASE ARTIFACTS VERIFIED") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/package/write_manifest.sh b/tools/package/write_manifest.sh new file mode 100755 index 0000000..b5f54de --- /dev/null +++ b/tools/package/write_manifest.sh @@ -0,0 +1,168 @@ +#!/usr/bin/env bash +# ============================================================================= +# tools/package/write_manifest.sh — compose dist/MANIFEST.txt. +# +# Runs LAST. Reads dist/build-info.txt (from build_prgs.sh) and +# dist/d64-listings.txt (from build_d64.sh), then hashes every artifact +# actually present in dist/. Nothing here is version-specific: sizes, hashes, +# git HEAD and submodule pins are all read at run time, and the variant +# guidance comes from the matrix in _common.sh. +# +# Usage: tools/package/write_manifest.sh +# ============================================================================= +set -euo pipefail + +PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +cd "$PROJECT_ROOT" +# shellcheck source=tools/package/_common.sh +. "$PROJECT_ROOT/tools/package/_common.sh" + +D64_LIST="$DIST/d64-listings.txt" + +[ -f "$BUILD_INFO" ] || { echo "ERROR: missing $BUILD_INFO — run build_prgs.sh first." >&2; exit 1; } + +info() { grep "^$1=" "$BUILD_INFO" | head -1 | cut -d= -f2-; } + +{ +echo "==============================================================================" +echo " c64-https — prebuilt release artifacts" +echo "==============================================================================" +echo +echo "TLS 1.3 / HTTPS client for the Commodore 64. Everything here is prebuilt:" +echo "you need no assembler, no cc65, no Python packages and no build step." +echo +echo "generated : $(info generated)" +echo "git HEAD : $(info git_head)" +if [ "$(info git_dirty)" = "yes" ]; then +echo " WARNING: built from a DIRTY working tree, not a clean checkout" +fi +echo "ip65 blob : $(info ip65_blob_bytes) bytes, sha256 $(info ip65_blob_sha256)" +echo +echo "submodule pins:" +grep '^submodule=' "$BUILD_INFO" | cut -d= -f2- | while read -r sub sha tag; do + printf ' %-18s %s %s\n' "$sub" "$tag" "$sha" +done +echo +echo "------------------------------------------------------------------------------" +echo " WHICH ONE DO I WANT?" +echo "------------------------------------------------------------------------------" +echo +echo "Two questions decide it." +echo +echo "1. How is your C64 on the network?" +echo " Ultimate 64 / Ultimate 64 Elite / C64 Ultimate -> the 'uci' images" +echo " a stock C64 with an RR-Net / cs8900a cartridge -> the 'ip65' images" +echo +echo "2. Do you have a RAM Expansion Unit (REU), and how fast is the CPU?" +echo " 'reu' images use the REU to accelerate the ECDSA verify. They need" +echo " one, and they are FASTER below about 18 MHz — which includes every" +echo " real stock C64 at 1 MHz." +echo " 'onchip' images need NO REU at all and do the same work on the CPU." +echo " They are FASTER above about 18 MHz, so they are the right pick for" +echo " Ultimate turbo modes (32/48/64 MHz)." +echo " The ~18 MHz crossover is measured, not estimated: the REU's DMA rate is" +echo " anchored to the ~1 MHz bus, so the REU profile carries a wall-clock" +echo " floor no amount of turbo removes, while the on-chip profile scales with" +echo " the clock. On a U64E the sign flips between the 16 and 20 MHz settings." +echo +for line in "${PACKAGE_VARIANTS[@]}"; do + key="$(variant_field "$line" 1)" + prg="$(variant_field "$line" 2)" + note="$(variant_field "$line" 6)" + echo " $prg" + echo " $note" + echo " disk: c64-https-$key.d64 (also on c64-https-$(variant_field "$line" 5).d64)" + echo +done +echo "------------------------------------------------------------------------------" +echo " PRG VARIANTS" +echo "------------------------------------------------------------------------------" +echo +printf ' %-28s %8s %s\n' "file" "bytes" "built with" +grep '^variant=' "$BUILD_INFO" | while read -r rec; do + # shellcheck disable=SC2086 + set -- $rec + prg=""; bytes=""; args="" + for kv in "$@"; do + case "$kv" in + prg=*) prg="${kv#prg=}" ;; + bytes=*) bytes="${kv#bytes=}" ;; + esac + done + args="$(printf '%s' "$rec" | sed -n 's/.*args=\(.*\) result=.*/\1/p')" + printf ' %-28s %8s make %s\n' "$prg" "$bytes" "$args" +done +echo +echo " Every variant is built after a 'make clean' — BACKEND= selects an include" +echo " path that make's dependency graph cannot see, so an incremental build can" +echo " silently produce a mixed image at exactly the right size." +echo +echo "------------------------------------------------------------------------------" +echo " DISK IMAGES (.d64)" +echo "------------------------------------------------------------------------------" +echo +echo " Each variant ships as its own single-PRG 1541 image, plus one image per" +echo " networking backend carrying both of that backend's profiles. Load with:" +echo +echo " LOAD\"*\",8,1 (single-variant disks — one file, first on disk)" +echo " LOAD\"UCI-REU\",8,1 (or whichever name the directory shows)" +echo " RUN" +echo +echo " There is deliberately no all-in-one image: the four PRGs total 868 blocks" +echo " and a .d64 has 664 free. Each backend's pair does fit (UCI 496, ip65 372)," +echo " so the per-backend disk is the largest bundle a real 1541 can hold." +echo +if [ -f "$D64_LIST" ]; then + while IFS= read -r ln; do + case "$ln" in + image=*) echo " ${ln#image=}" ;; + ---) echo ;; + *) echo " $ln" ;; + esac + done < "$D64_LIST" +fi +echo "------------------------------------------------------------------------------" +echo " TEST LISTENER — c64-https-listener.py" +echo "------------------------------------------------------------------------------" +echo +echo " A single self-extracting Python file that stands up the whole server side" +echo " of the end-to-end test: it mints a fresh self-signed ECDSA P-256" +echo " certificate, serves TLS 1.3 only, and returns the canonical response the" +echo " C64 client expects." +echo +echo " python3 c64-https-listener.py --port 4433 # serve" +echo " python3 c64-https-listener.py --selftest # prove it works, no C64" +echo " python3 c64-https-listener.py --extract ./src # unpack its sources" +echo +echo " DEPENDENCIES: none. Nothing to pip install, no venv, no network access." +echo " Certificate generation is pure Python (P-256 + DER + ECDSA-SHA256); TLS" +echo " is the standard library's ssl module." +echo +echo " LIMITATION, stated here so you do not discover it by running it: the" +echo " listener needs an interpreter whose ssl module supports TLS 1.3, i.e. one" +echo " linked against OpenSSL 1.1.1 or newer. macOS's /usr/bin/python3 is linked" +echo " against LibreSSL 2.8.3, has no TLS 1.3, and cannot serve this client at" +echo " any price — install python3 from python.org or Homebrew there. The" +echo " listener detects this at startup and says so in one line. Most Linux" +echo " distributions ship a suitable python3." +echo +echo " The certificate it generates is a throwaway test fixture and not a trust" +echo " anchor. Do not deploy it anywhere real." +echo +echo "------------------------------------------------------------------------------" +echo " SHA256 CHECKSUMS" +echo "------------------------------------------------------------------------------" +echo +} > "$MANIFEST" + +# Hash every artifact in dist/, excluding the manifest itself and the two +# intermediate files the packaging scripts pass between each other. +( cd "$DIST" && find . -maxdepth 1 -type f \ + ! -name 'MANIFEST.txt' ! -name 'build-info.txt' ! -name 'd64-listings.txt' \ + ! -name '*.log' \ + | sed 's|^\./||' | sort ) \ +| while IFS= read -r f; do + printf ' %s %s\n' "$(sha256_of "$DIST/$f")" "$f" >> "$MANIFEST" +done + +echo "[package] wrote $MANIFEST" diff --git a/tools/uci/test_https_local.py b/tools/uci/test_https_local.py index d1817cf..16e6d94 100644 --- a/tools/uci/test_https_local.py +++ b/tools/uci/test_https_local.py @@ -203,7 +203,7 @@ def _keep_cycle(word: int) -> bool: # --- External-listener mode ---------------------------------------------- # When EXTERNAL_LISTENER=1 the script does NOT stand up its own inline TLS # listener and does NOT load the repo cert/key: the server side is provided -# out-of-band (e.g. the packaged dist/c64-https-listener.zip listener). The +# out-of-band (e.g. the packaged dist/c64-https-listener.py listener). The # C64 client is pointed at EXTERNAL_HOST:EXTERNAL_PORT and the pass criteria # come purely from C64-side state (http_resp_buf / screen RAM). Default OFF — # behavior is unchanged unless the var is set.