fix(tests): pin the pytest collection boundary so a root pytest cannot imply coverage it does not have

[JC-000]

Closes#109.

What was wrong — and it is worse than the report

The report is right that tests/ holds five test_*.py files with no def test_ in them, so pytest walks a directory named exactly the way its discovery convention expects, collects zero, and says nothing about it.

But the reported "green 154 passed" does not reproduce. pytest is available on this bench (/opt/homebrew/bin/pytest 9.0.3, on an interpreter that can import c64_test_harness and cryptography), so this was measured rather than inferred. On a clean origin/master snapshot, run from the repo root:

$ pytest -> 25 passed, 1 warning, 75 errors in 0.14s exit 1
$ pytest tests/ -> no tests ran in 0.04s exit 5

The 75 errors are all fixture 'transport' not found. tools/ is not a pytest suite either. Its test_* functions take positional (transport, labels) supplied by tools/run_all_tests.py, which allocates a VICE instance per suite; there is no conftest.py anywhere in the repo, no plugin provides those fixtures, and the repo declares no pytest dependency at all.

So the real defect is broader than "tests/ contributes zero": nothing in this repo is meaningfully pytest-shaped, and what a bare pytest prints depends entirely on which interpreter and plugins happen to be around. The reporter got a green; this bench gets a red. Both are noise.

Collection breakdown on master (100, not 154 — the libs/ submodules are unpopulated in this worktree, which accounts for the other 54):

Path	Collected	Outcome
tools/test_net_test_env.py	14	14 pass (real unittest.TestCase suite)
tools/test_package_verify.py	12	11 pass, 1 error
12 other tools/test_*.py	74	74 fixture errors
tests/	0	silent
tools/uci/test_*.py	0	silent (same shape as tests/)

What shipped

Option 1 + 3 from the issue. Both were needed, and neither is sufficient alone:

1. tests/test_*.py -> tests/rig_*.py. This part had to be a rename. testpaths only takes effect when pytest is invoked from the rootdir, so config alone leaves the directory collectable from anywhere else. A rename holds unconditionally.
2. pytest.ini pins testpaths to the three modules pytest can genuinely run, and norecursedirs keeps collection out of libs/, ip65/, tests/ and tools/uci/ — every one of which is test_*.py-named and contributes zero.
3. conftest.py prints the scope of the run in the header and again immediately above the summary line (the header scrolls away on a long run; the summary line is what people read). It imports nothing and defines no fixtures, so it stays inert without pytest — the repo's zero-dependency posture is unchanged.
4. tools/test_package_verify.py::test_parse_build_info_records took a tmp_lines argument its own main() passed in, which is why an otherwise pure-logic module had one erroring test. Defaulted to a module constant; both runners still work (python3 tools/test_package_verify.py -> 31 passed, unchanged, and make package-verify invokes exactly that).
5. tools/test_pytest_boundary.py — a pure-AST guard (no VICE, no build, milliseconds) so this cannot silently regress. It asserts tests/ holds no test_*.py, every testpaths entry exists, and testpaths is exactly the set of tools/test_*.py modules pytest can run — both directions, so a new pure-logic suite cannot become invisible to a bare pytest either. It runs under pytest and standalone.
6. tests/README.md — what each rig script needs, how to run it, and why it is not collectable.

Before / after, measured

Run via subprocess with an explicit cwd (the "before" column is a git archive of origin/master extracted to a scratch dir, so it is a genuinely pristine tree):

Invocation	Before	After
pytest (repo root)	25 passed, 75 errors, exit 1	30 passed, exit 0, with the scope note
pytest tests/	no tests ran, exit 5, unexplained	no tests ran, exit 5, explained

After, at the repo root:

============================= test session starts ==============================
c64-https: pytest runs ONLY the pure-logic host-side modules pinned in
pytest.ini `testpaths`. It does NOT run the C64 suites (those need VICE:
`python3 tools/run_all_tests.py`) and it does NOT run the live-rig
scripts in tests/ (manual, sudo + network rig: see tests/README.md).
A green run here says nothing about either.
configfile: pytest.ini
testpaths: tools/test_net_test_env.py, tools/test_package_verify.py, tools/test_pytest_boundary.py
collected 30 items
tools/test_net_test_env.py .............. [ 46%]
tools/test_package_verify.py ............ [ 86%]
tools/test_pytest_boundary.py .... [100%]
============================== scope of this run ===============================
c64-https: pytest runs ONLY the pure-logic host-side modules pinned in
[...]
============================== 30 passed in 0.14s ==============================

pytest tests/ now ends with:

pytest collected nothing from the paths you gave it.
If that was `pytest tests/`: tests/ holds manual live-rig scripts
(tests/rig_*.py, main() programs needing sudo and a network rig), not
pytest tests. See tests/README.md for how to run them.
============================ no tests ran in 0.00s =============================

One honest limitation, measured rather than glossed:testpaths applies only at the rootdir, so pytest from a subdirectory collects that subdirectory instead — from tools/ it is 30 passed + 74 fixture errors, exit 1. That is left as-is deliberately. It is loud, and it is correct: those modules really cannot run under pytest. Suppressing it with collect_ignore_glob would trade a true red for a manufactured green. README and CLAUDE.md both state this rather than claiming determinism the config does not provide.

Guard verified by probe, not by assertion

$ printf 'x = 1\n' > tests/test_bogus_guard_probe.py; python3 tools/test_pytest_boundary.py
FAIL test_tests_dir_holds_no_pytest_named_files
tests/ contains pytest-named files ['test_bogus_guard_probe.py'] ...
$ printf 'def test_probe() -> None:\n assert True\n' > tools/test_bogus_guard_probe.py
FAIL test_testpaths_lists_every_runnable_tools_module
these tools/ modules are pytest-runnable but absent from pytest.ini
testpaths: ['tools/test_bogus_guard_probe.py'] ...

Both probes removed afterwards. The guard also correctly classifies tools/test_net_test_env.py as runnable (it is a unittest.TestCase module whose extra method arguments come from @mock.patch, not from fixtures) — an earlier, cruder version of the rule got that wrong and the probe caught it.

Why not option 2 (pytest wrappers + pytest.skip())

Two reasons, and the second is the load-bearing one:

• It would introduce a pytest dependency the repo does not declare. requirements.txt is cryptography plus a comment; the documented runner is python3 tools/run_all_tests.py. Wrapper modules that import pytest would make a currently-optional tool mandatory for a test tree that pytest cannot run anyway.
• A rig test reported as skipped is exactly the signal this repo has been removing. fix(package): close the vacuous pass in the release gate (#98 landed only 1 of 4 commits) #101 fixed a release gate that said RELEASE ARTIFACTS VERIFIED having checked nothing. "0 network tests ran" and "5 network tests skipped for a reason nobody read" are the same fact with different amounts of reassurance attached. The rename plus an unmissable scope banner says it once, in plain language, on every single run.

Reference sweep

grep -rn 'tests/test_' is clean. Updated: README.md (x5 plus a new "pytest is not the runner here" subsection), CLAUDE.md (x4 plus a boundary subsection under Smoke tests, and the C64_SKIP_BUILD grep recipe now globs tests/rig_*.py), docs/library-ingestion-architecture.md, tools/https_e2e/__init__.py, tools/https_e2e/certs/README, and 6 docstring/comment references inside the rig scripts themselves.

No Makefile target, no CI (the repo has no .github/), and no reference anywhere in the sibling c64-test-harness repo (grepped the whole tree; zero hits). tests/rig_phase1_dhcp.py was executed post-rename and still resolves its own paths correctly, skipping loudly: SKIP: missing prerequisites: ip not on PATH; iptables not on PATH; sudo requires a password.

Scope

Python and docs only — no .s, no .cfg, no Makefile, so the build-evidence bar does not apply and no PRG changes. .gitignore gains .pytest_cache/.

Adjacent, deliberately not fixed here:tools/uci/test_*.py have the identical shape (rig scripts named test_*.py collecting zero). norecursedirs closes the misleading path for the default invocation, but the honest fix is the same rig_ rename, and that blast radius (CLAUDE.md documents those scripts extensively) belongs in its own PR.

🤖 Generated with Claude Code