Skip to content

test_bootstrap.py runs in no CI job, so the loader invariant and the spec-to-installer coverage are unchecked #754

Description

@ptr727

scripts/tests/test_bootstrap.py defines its checks as module-level test_* functions with its own check() collector and main() runner, and no unittest.TestCase. The only invocation in CI is validate-task.yml:117:

uvx coverage@latest run --source=scripts,spec,host-setup -m unittest discover -s scripts/tests

unittest discover loads TestCase subclasses from each module it imports. A module-level function named test_something is not one, so the file is imported (an import-time error would still surface) and none of its ten checks run. Nothing else invokes it either: the file is named in prose by host-setup/README.md, scripts/host_gate.py:220, and spec/host-tools.json's own note, and by no workflow, script, or runbook.

OPERATIONS.md "Run the gates the way CI runs them" lists the unittest discover line and no separate python3 scripts/tests/test_bootstrap.py, so a local verification run reproduces the gap rather than catching it.

What is not running

Confirmed by running the file directly, which passes:

$ python3 scripts/tests/test_bootstrap.py
[ OK ] bootstrap loader invariant and spec coverage
$ python3 -m unittest discover -s scripts/tests -v 2>&1 | grep -c bootstrap
1

That one line is a docstring from test_skills_install.py, not a collected bootstrap test.

The two properties the file's own docstring names as failing silently if nobody checks them are the ones nobody is checking:

  • The loader invariant, that bootstrap.sh and bootstrap.ps1 each read exactly one path into the fetched tree and no payload directory. The file states this is asserted rather than promised in prose precisely because it is a property of each file.
  • Spec-to-installer coverage, that every tool spec/host-tools.json requires is one the platform installers can provide, plus the totality of the remedy mapping. spec/host-tools.json's note names this file as what keeps that mapping total, and scripts/host_gate.py:220 names it as what holds the fleet-wide remedy contract that host_gate.py deliberately does not hold itself.

So a tool could be declared required with nothing here able to install it, or a remedy could name a tool an installer does not manage, and the pull request that did it would pass.

Shapes that would close it

Not a recommendation, just what seems available:

  1. Wrap the existing checks in a TestCase so unittest discover collects them, keeping main() for the direct-run path. Smallest change, and the two idioms then coexist in one file.
  2. Convert the file to unittest the way the other six files under scripts/tests/ are written, and drop the check()/failures collector for subTest or plain assertions. Consistent with its siblings, larger diff.
  3. Invoke it as its own CI step, beside the spec/audit.py --selftest and host-setup/agent-safety/test_install.py lines that already run outside unittest discover, and add it to OPERATIONS.md's list. Leaves the file's idiom alone.

Found while fixing #750 and #751, looking for a live home for a new test.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions