scripts/run_plugin_tests.py silently executes nothing for 61% of plugin tests — script-style files are found, then collected as 0 by pytest

[ChuckBuilds]

Summary

scripts/run_plugin_tests.py in this repo discovers plugin test files and hands them to pytest. Most plugin tests are written as standalone scripts (module-level main() plus if __name__ == "__main__"), which pytest collects zero items from. The runner prints how many files it found, then reports "no tests ran" — so it looks like the plugin simply has no tests, when in fact it has tests that were never executed.

151 of 248 plugin test files (61%) across 29 plugins are in that category on this rig.

Reproduction

$ cd ~/LEDMatrix
$ python3 scripts/run_plugin_tests.py -p countdown -d ~/LEDMatrix/plugin-repos
Found 1 test file(s)
 - /home/hdpi/LEDMatrix/plugin-repos/countdown/test_countdown_row_save.py

============================= test session starts ==============================
collected 0 items

============================ no tests ran in 0.34s =============================

The same test, run the way its own docstring documents:

$ PYTHONPATH=$PWD LEDMATRIX_CORE=$PWD python3 plugin-repos/countdown/test_countdown_row_save.py
 PASS populated sections validate
 PASS a number is not a layout
 PASS layout(None) yields a mapping
 PASS style(None) yields a mapping
 PASS layout('') yields a mapping
 PASS style('') yields a mapping
 PASS layout(17) yields a mapping
 PASS style(17) yields a mapping
All checks passed (rc=0)

Spot-checked two more, both real and both passing when executed directly:

geochron/test_per_size_map_cache.py rc=0 "All checks passed"
calendar/test_display_does_not_blank_the_panel.py rc=0 "7 checks, 0 failed"
ledmatrix-stocks/test_symbol_formats.py rc=0 "All checks passed"

Scale

plugin	test files	never collected
football-scoreboard	41	28
hockey-scoreboard	22	16
baseball-scoreboard	30	15
basketball-scoreboard	20	13
lacrosse-scoreboard	18	13
soccer-scoreboard	24	13
afl-scoreboard	15	12
nrl-scoreboard	15	12
ledmatrix-flights	18	7
ufc-scoreboard	7	6
ledmatrix-weather	5	4
… 18 more
total	248	151 (61%)

Four plugins have no executable coverage at all through this runner, despite shipping working tests: calendar, countdown, geochron, ledmatrix-stocks (1/1 each).

This is not a CI gap

Worth stating clearly, because the filenames collide: ledmatrix-plugins has its own scripts/run_plugin_tests.py, and that one is purpose-built for these scripts — it spawns each with subprocess and uses a 0 pass / 2 skip / 1 fail exit convention. .github/workflows/test-plugins.yml calls that one, so plugin CI is fine.

The problem is confined to the copy in this repo, which is not referenced by any workflow here. Its docstring says "Supports both unittest and pytest", so script-style was never claimed — but it still counts those files as found before running none of them, and someone working in the core who reaches for the runner sitting next to check_plugin.py gets a clean-looking result that covered nothing.

The exit code is pytest's 5 ("no tests collected"), which is at least non-zero — but "no tests ran" reads as "this plugin has no tests" rather than "this runner cannot execute these tests".

Suggested fix

Cheapest option that removes the trap: detect script-style files and either delegate to the plugins-repo convention (spawn with subprocess, honour 0/2/1) or refuse them explicitly:

Found 1 test file(s)
 - .../countdown/test_countdown_row_save.py
 ! 1 file is script-style (module main() + __main__ guard) and cannot be
 collected by pytest. Run it directly, or use ledmatrix-plugins'
 scripts/run_plugin_tests.py, which handles this convention.

Alternatively, drop this script and point at the plugins-repo runner, since that one already solved the problem and the duplicate filename doing something different is its own source of confusion.

Environment

LEDMatrix v3.3.0-4-g0730d952, 256x64 rig, 44 first-party plugins installed under plugin-repos/, pytest 9.1.1.