tests/ contains no pytest-collectable tests — 'pytest' at repo root reports 154 passed while the five rig tests contribute zero

[JC-000]

Noticed from the c64-test-harness side. This is a discovery/layout issue rather than a broken-test issue — the rig tests themselves are fine and do real verification when run the documented way.

The layout inverts the usual convention

Directory	Contents	Collected by pytest
tools/ (+ libs/*/tools/)	the actual pytest suite	154 tests
tests/	five script-style live rig tests	0 tests

All five files in tests/ are main() scripts behind if __name__ == "__main__": sys.exit(main()), with no def test_ functions:

test_phase1_dhcp.py 0 test fns 0 collected
test_phase2_http.py 0 test fns 0 collected
test_phase3_https_1mhz.py 0 test fns 0 collected
test_phase3_https.py 0 test fns 0 collected
test_vice_https_macos.py 0 test fns 0 collected

That is deliberate — README lines 333-334 document running them directly:

sudo PYTHONPATH=tools python3 tests/test_phase1_dhcp.py
sudo PYTHONPATH=tools python3 tests/test_phase2_http.py

and they do verify properly that way. test_vice_https_macos.py hard-fails on FAIL: DHCP not acquired after 3 attempts and only prints PASS after a completed TLS handshake + GET, so it is not a vacuous test.

Why it still bites

The files carry test_*.py names in a directory called tests/, which is exactly pytest's discovery convention — but they are the one part of the tree pytest cannot collect. Depending on where you invoke from you get opposite signals:

$ pytest tests/ # rc=5, "no tests collected" -- loud, fine
$ pytest # rc=0, "154 tests collected" -- from tools/, tests/ silently contributes 0

The root-level invocation is the problem: it succeeds, reports a healthy-looking 154 passed, and gives no indication that the five network tests in tests/ never ran. Anyone reasonably assumes a green pytest covered them.

There is also no [tool.pytest.ini_options] in pyproject.toml — no testpaths, no norecursedirs — so which behaviour you get depends entirely on the working directory.

Flagging it because #101 (fix/package-verify-vacuous-pass) suggests this class of "looked green, verified nothing" is one you care about. This is a milder cousin: the tests are real, but a green root-level pytest overstates what was covered.

Options

Any one of these closes the gap:

1. Move the rig scripts out of pytest's namespace — e.g. tools/rig/ or scripts/, or rename to rig_phase1_dhcp.py. Cheapest, and honest about them being manual sudo/rig-dependent scripts.
2. Add thin pytest wrappers that shell out to main() and pytest.skip() when the rig is absent (no feth0, no root). Then a root-level pytest reports them as skipped rather than omitting them silently.
3. Pin the boundary in config — set testpaths in pyproject.toml so the invocation is unambiguous, and note in tests/README that the directory is manual-only.

Option 2 is the most informative if you want a single command to reflect true coverage; option 1 is the least work.

Not an issue

To be explicit about what I am not reporting: the rig tests are not broken, they are not passing vacuously when run as documented, and pytest tests/ does fail loudly (rc=5) rather than pretending. The only misleading path is a bare pytest at the repo root.

🤖 Generated with Claude Code