fix(tests): generate the gitignored test certs on demand (#93)

README.md

@@ -256,7 +256,7 @@ python3 tools/uci/test_http_local.py # HTTP GET against local listener
 python3 tools/uci/test_https_local.py # HTTPS GET (TLS 1.3 + ECDSA-P256)
 ```
 
-`test_https_local.py` is the end-to-end HTTPS demo (UCI backend only): it boots the U64E at 48 MHz turbo, connects to a local Python TLS listener using the test cert under `tools/https_e2e/certs/`, and confirms a full TLS 1.3 handshake + HTTP GET. With `DEBUG_CAPTURE=1`, each run writes a timestamped artifact directory under `$UCI_DEBUG_DIR` (default `/tmp/uci_https_debug/`) with raw 6510 bus trace, TLS state snapshot, and listener result.
+`test_https_local.py` is the end-to-end HTTPS demo (UCI backend only): it boots the U64E at 48 MHz turbo, connects to a local Python TLS listener using the test cert under `tools/https_e2e/certs/`, and confirms a full TLS 1.3 handshake + HTTP GET. That cert is gitignored throwaway material — the directory is empty in a fresh clone and the pair is generated on first use, with no dependency beyond the standard library (`python3 tools/https_e2e/ensure_certs.py` mints it by hand). With `DEBUG_CAPTURE=1`, each run writes a timestamped artifact directory under `$UCI_DEBUG_DIR` (default `/tmp/uci_https_debug/`) with raw 6510 bus trace, TLS state snapshot, and listener result.
 
 Environment variables honored by `test_https_local.py`:
 

tools/https_e2e/certs/README

@@ -6,6 +6,11 @@ These are self-signed certs used by the local TLS 1.3 listener
 against `www.foo.bar`. They are NOT trust-anchors for anything; do not
 deploy them anywhere real.
 
+**This directory is empty in a fresh clone.** The cert files are
+gitignored (see the repo `.gitignore`) because they are throwaway
+material that should be minted locally, not shipped. Nothing is wrong
+if `ls` shows only this README.
+
 Two cert profiles are supported:
 
  server.pem / server.key -- P-256 (secp256r1 / prime256v1),
@@ -16,23 +21,48 @@ Two cert profiles are supported:
 CN is `www.foo.bar` and SAN covers `foo.bar` + `www.foo.bar` for both.
 Validity is 10 years from generation.
 
-The cert files themselves are gitignored (see the repo `.gitignore`);
-they are generated on demand by the listener.
-
 Auto-generation (default path)
 ------------------------------
 
-Both pairs are auto-generated lazily by `https_listener._ensure_certs_*()`
-the first time the listener is started under each profile, using the
-Python `cryptography` package. To force regeneration, delete the
-files and start the listener once with the matching `cert_profile`.
+Each pair is generated the first time something needs it, and reused
+thereafter. Every in-tree consumer goes through
+`tools/https_e2e/ensure_certs.py`, which delegates to
+`tools/package/listener/gen_certs.py` so the in-tree tests and the
+packaged listener mint identical material:
+
+ - `https_listener.py` (via `_ensure_certs(cert_profile)`), and so
+ everything built on it -- `tests/test_vice_https_macos.py`,
+ `evil_listener.py`, `tools/uci/test_https_bad_finished.py`
+ - `tools/uci/test_https_local.py`, which inlines its own listener
+ (this one used to fail with `ERROR: cert/key not found` instead --
+ issue #93)
+ - `tools/uci/test_https_local_p384.py`, same, P-384 profile
+
+Generation needs nothing beyond the Python standard library: PR #96
+reimplemented `gen_certs.py` in pure Python (curve arithmetic, a
+minimal DER encoder, ECDSA) so the packaged listener has no
+third-party dependency, and the P-384 profile rides on the same code.
+No pip, no venv.
+
+Under `EXTERNAL_LISTENER=1` no repo cert is loaded or generated at
+all, by design: the server is out of band.
+
+Minting a pair by hand
+----------------------
+
+ python3 tools/https_e2e/ensure_certs.py # P-256
+ python3 tools/https_e2e/ensure_certs.py --profile p384 # P-384
+ python3 tools/https_e2e/ensure_certs.py --force # regenerate
+
+To regenerate, either pass `--force` or delete the files and let the
+next run recreate them.
 
 Manual regeneration with openssl
 --------------------------------
 
-If you need to (re)create the P-384 pair without invoking the
-listener (e.g. for debugging with `openssl s_client` directly), the
-following openssl invocation produces the same cert:
+If you need to (re)create a pair without Python (e.g. for debugging
+with `openssl s_client` directly), this produces an equivalent P-384
+cert:
 
  cat > /tmp/p384_san.cnf <<'EOF'
  [req]
@@ -56,8 +86,8 @@ following openssl invocation produces the same cert:
  -days 3650 -sha384 -config /tmp/p384_san.cnf
 
 The P-256 pair can be (re)created the same way with
-`-name prime256v1` and `-sha256` — but the listener's auto-generator
-is the canonical source.
+`-name prime256v1` and `-sha256` -- but `ensure_certs.py` is the
+canonical source.
 
 Selecting which cert the listener presents
 ------------------------------------------

tools/https_e2e/ensure_certs.py

@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+"""Ensure the local test certs exist, generating them on demand.
+
+The certs under ``tools/https_e2e/certs/`` are gitignored — they are
+throwaway self-signed test material, not something to ship in a repo. That
+means a fresh clone has none, and every script that starts a local TLS
+listener needs them.
+
+``https_listener.py`` has always generated its own pair, but
+``tools/uci/test_https_local.py`` inlines its own listener and so never
+crossed that path: it just printed ``ERROR: cert/key not found`` and exited
+2, leaving the reader to discover that a generator existed somewhere else
+entirely (issue #93). This module is the single entry point both now use.
+
+Generation is delegated to ``tools/package/listener/gen_certs.py`` rather
+than duplicated, so the packaged listener and the in-tree tests produce
+identical material: self-signed ECDSA, CN ``www.foo.bar``, SAN covering
+``foo.bar`` and ``www.foo.bar``, 10 year validity.
+
+That generator is **pure Python stdlib** (PR #96): no ``cryptography``, no
+pip, no venv on either path. P-384 was the one profile that still needed
+the package, and #96's implementation extends to it — same algorithm,
+different curve — so the in-tree tests now have no third-party dependency
+for certs either.
+
+Two profiles, matching the listener's ``cert_profile`` selector:
+
+ p256 (default) -> certs/server.pem + certs/server.key
+ p384 -> certs/server-p384.pem + certs/server-p384.key
+
+Usable directly, too:
+
+ python3 tools/https_e2e/ensure_certs.py [--profile p256|p384] [--force]
+"""
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+CERTS_DIR = REPO_ROOT / "tools" / "https_e2e" / "certs"
+GEN_DIR = REPO_ROOT / "tools" / "package" / "listener"
+
+CN = "www.foo.bar"
+SANS = ["foo.bar", "www.foo.bar"]
+PROFILES = ("p256", "p384")
+
+# Filenames per profile — must stay in step with gen_certs.CURVE_PROFILES.
+_FILENAMES = {
+ "p256": ("server.pem", "server.key"),
+ "p384": ("server-p384.pem", "server-p384.key"),
+}
+
+
+def cert_paths(profile: str = "p256",
+ certs_dir: Path | None = None) -> tuple[Path, Path]:
+ """Return (cert_path, key_path) for *profile*. Does not generate."""
+ if profile not in _FILENAMES:
+ raise ValueError(f"unknown cert profile {profile!r}; "
+ f"expected one of {PROFILES}")
+ base = Path(certs_dir) if certs_dir is not None else CERTS_DIR
+ cert_name, key_name = _FILENAMES[profile]
+ return base / cert_name, base / key_name
+
+
+def ensure_certs(profile: str = "p256",
+ certs_dir: Path | None = None,
+ force: bool = False,
+ quiet: bool = False) -> tuple[Path, Path]:
+ """Return (cert_path, key_path), generating them if absent.
+
+ Idempotent: an existing pair is returned untouched unless *force*.
+ Raises SystemExit with a one-line actionable message if generation is
+ impossible — never a bare traceback, since the usual cause is a missing
+ dependency rather than a bug.
+ """
+ cert_path, key_path = cert_paths(profile, certs_dir)
+
+ if cert_path.is_file() and key_path.is_file() and not force:
+ return cert_path, key_path
+
+ if not quiet:
+ pretty = f"P-{profile[1:]}" # p256 -> P-256
+ print(f"test certs not found in {cert_path.parent} — generating "
+ f"(self-signed {pretty}, CN={CN}); "
+ f"they are gitignored by design")
+
+ if str(GEN_DIR) not in sys.path:
+ sys.path.insert(0, str(GEN_DIR))
+ try:
+ from gen_certs import generate # noqa: PLC0415
+ except ImportError as exc:
+ # The generator is stdlib-only, so this is a missing/broken file
+ # rather than a missing package. Say which file, in one line.
+ raise SystemExit(
+ f"cannot generate test certs: {exc} "
+ f"(expected the stdlib-only generator at "
+ f"{GEN_DIR / 'gen_certs.py'})") from exc
+
+ try:
+ return generate(CN, SANS, cert_path.parent, force=force,
+ curve=profile)
+ except SystemExit:
+ raise
+ except Exception as exc: # noqa: BLE001 - surface as one readable line
+ raise SystemExit(
+ f"test cert generation failed: {type(exc).__name__}: {exc}") from exc
+
+
+# Backwards-compatible alias for the P-256-only spelling.
+def ensure_p256_certs(certs_dir: Path | None = None, force: bool = False,
+ quiet: bool = False) -> tuple[Path, Path]:
+ return ensure_certs("p256", certs_dir, force=force, quiet=quiet)
+
+
+def main(argv: list[str] | None = None) -> int:
+ import argparse # noqa: PLC0415 - keep import cost off the library path
+
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--profile", default="p256", choices=list(PROFILES),
+ help="cert profile to ensure (default: p256)")
+ p.add_argument("--force", action="store_true",
+ help="regenerate even if the pair already exists")
+ args = p.parse_args(argv)
+
+ cert, key = ensure_certs(args.profile, force=args.force)
+ print(f"cert: {cert}\nkey: {key}")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())

tools/https_e2e/https_listener.py

@@ -7,10 +7,11 @@
 The TLS layer uses a self-signed ECDSA certificate. Two cert profiles
 are available:
 
- "p256" (default) -- P-256 / ecdsa-with-SHA256, generated lazily by
- this module the first time the listener starts.
- "p384" -- P-384 / ecdsa-with-SHA384, generated out-of-band
- with openssl (see tools/https_e2e/certs/README).
+ "p256" (default) -- P-256 / ecdsa-with-SHA256
+ "p384" -- P-384 / ecdsa-with-SHA384
+
+Both are generated lazily on first use (see ensure_certs.py) into the
+gitignored tools/https_e2e/certs/ directory, and reused thereafter.
 
 TLS 1.3 is required; older versions are rejected.
 
@@ -42,11 +43,9 @@
 # Certificate generation
 # ---------------------------------------------------------------------------
 
-_CERTS_DIR = os.path.join(os.path.dirname(__file__), "certs")
-_CERT_PATH = os.path.join(_CERTS_DIR, "server.pem")
-_KEY_PATH = os.path.join(_CERTS_DIR, "server.key")
-_CERT_PATH_P384 = os.path.join(_CERTS_DIR, "server-p384.pem")
-_KEY_PATH_P384 = os.path.join(_CERTS_DIR, "server-p384.key")
+# Cert/key filenames per profile live in ensure_certs.py — the module that
+# also generates them — so the names are stated once.
+_HERE = os.path.dirname(os.path.abspath(__file__))
 
 _CERT_PROFILE_ENV = "HTTPS_LISTENER_CERT_PROFILE"
 _DEFAULT_CERT_PROFILE = "p256"
@@ -68,123 +67,35 @@ def _resolve_cert_profile(cert_profile: str | None) -> str:
  return cert_profile
 
 
-def _ensure_certs_p256() -> tuple[str, str]:
- """Return (cert_path, key_path) for the P-256 profile.
+def _ensure_certs(cert_profile: str) -> tuple[str, str]:
+ """Return (cert_path, key_path) for *cert_profile*, generating if absent.
 
- Generates the cert pair on first use; subsequent calls reuse the
- cached files in the certs/ directory.
+ Both profiles are generated by the shared helper in ensure_certs.py,
+ which delegates to tools/package/listener/gen_certs.py — so the in-tree
+ tests and the packaged listener mint identical material, and there is
+ one place to fix if the profile ever changes. This module used to
+ inline two near-identical copies of the cert builder.
  """
- if os.path.isfile(_CERT_PATH) and os.path.isfile(_KEY_PATH):
- return _CERT_PATH, _KEY_PATH
-
- from cryptography import x509
- from cryptography.hazmat.primitives import hashes, serialization
- from cryptography.hazmat.primitives.asymmetric import ec
- from cryptography.x509.oid import NameOID
- import datetime
-
- key = ec.generate_private_key(ec.SECP256R1())
-
- subject = issuer = x509.Name([
- x509.NameAttribute(NameOID.COMMON_NAME, "www.foo.bar"),
- ])
-
- cert = (
- x509.CertificateBuilder()
- .subject_name(subject)
- .issuer_name(issuer)
- .public_key(key.public_key())
- .serial_number(x509.random_serial_number())
- .not_valid_before(datetime.datetime.utcnow())
- .not_valid_after(datetime.datetime.utcnow() + datetime.timedelta(days=3650))
- .add_extension(
- x509.SubjectAlternativeName([
- x509.DNSName("foo.bar"),
- x509.DNSName("www.foo.bar"),
- ]),
- critical=False,
- )
- .sign(key, hashes.SHA256())
- )
+ if cert_profile not in _CERT_PROFILES:
+ # Should be unreachable thanks to _resolve_cert_profile().
+ raise ValueError(f"unknown cert_profile {cert_profile!r}")
 
- os.makedirs(_CERTS_DIR, exist_ok=True)
+ if _HERE not in sys.path:
+ sys.path.insert(0, _HERE)
+ from ensure_certs import ensure_certs # noqa: PLC0415
 
- with open(_KEY_PATH, "wb") as f:
- f.write(key.private_bytes(
- serialization.Encoding.PEM,
- serialization.PrivateFormat.TraditionalOpenSSL,
- serialization.NoEncryption(),
- ))
+ cert_path, key_path = ensure_certs(cert_profile)
+ return str(cert_path), str(key_path)
 
- with open(_CERT_PATH, "wb") as f:
- f.write(cert.public_bytes(serialization.Encoding.PEM))
 
- return _CERT_PATH, _KEY_PATH
+def _ensure_certs_p256() -> tuple[str, str]:
+ """Return (cert_path, key_path) for the P-256 profile, generating if absent."""
+ return _ensure_certs("p256")
 
 
 def _ensure_certs_p384() -> tuple[str, str]:
- """Return (cert_path, key_path) for the P-384 profile.
-
- Generates the cert pair on first use; subsequent calls reuse the
- cached files in the certs/ directory. Mirrors _ensure_certs_p256()
- but uses SECP384R1 + SHA-384.
- """
- if os.path.isfile(_CERT_PATH_P384) and os.path.isfile(_KEY_PATH_P384):
- return _CERT_PATH_P384, _KEY_PATH_P384
-
- from cryptography import x509
- from cryptography.hazmat.primitives import hashes, serialization
- from cryptography.hazmat.primitives.asymmetric import ec
- from cryptography.x509.oid import NameOID
- import datetime
-
- key = ec.generate_private_key(ec.SECP384R1())
-
- subject = issuer = x509.Name([
- x509.NameAttribute(NameOID.COMMON_NAME, "www.foo.bar"),
- ])
-
- cert = (
- x509.CertificateBuilder()
- .subject_name(subject)
- .issuer_name(issuer)
- .public_key(key.public_key())
- .serial_number(x509.random_serial_number())
- .not_valid_before(datetime.datetime.utcnow())
- .not_valid_after(datetime.datetime.utcnow() + datetime.timedelta(days=3650))
- .add_extension(
- x509.SubjectAlternativeName([
- x509.DNSName("foo.bar"),
- x509.DNSName("www.foo.bar"),
- ]),
- critical=False,
- )
- .sign(key, hashes.SHA384())
- )
-
- os.makedirs(_CERTS_DIR, exist_ok=True)
-
- with open(_KEY_PATH_P384, "wb") as f:
- f.write(key.private_bytes(
- serialization.Encoding.PEM,
- serialization.PrivateFormat.TraditionalOpenSSL,
- serialization.NoEncryption(),
- ))
-
- with open(_CERT_PATH_P384, "wb") as f:
- f.write(cert.public_bytes(serialization.Encoding.PEM))
-
- return _CERT_PATH_P384, _KEY_PATH_P384
-
-
-def _ensure_certs(cert_profile: str) -> tuple[str, str]:
- """Dispatch to the per-profile cert loader."""
- if cert_profile == "p256":
- return _ensure_certs_p256()
- if cert_profile == "p384":
- return _ensure_certs_p384()
- # Should be unreachable thanks to _resolve_cert_profile().
- raise ValueError(f"unknown cert_profile {cert_profile!r}")
+ """Return (cert_path, key_path) for the P-384 profile, generating if absent."""
+ return _ensure_certs("p384")
 
 
 # ---------------------------------------------------------------------------