tests: add a unit test suite and measure coverage

[insatomcat]

The repository had no test and no CI. The two OpenSSF gold coverage criteria (test_statement_coverage90 and test_branch_coverage80) could therefore not be evaluated at all: they ask for a measured figure, not for a test suite that merely exists.

What this brings

A pytest suite of 219 tests over the five modules, running in about 0.2 s. Everything that touches the system is mocked (subprocess, sysfs, /proc, the network stack), so the suite needs neither root, nor OVS, nor a cluster, and runs on any machine.

Measured result:

Criterion	Before	After
test_statement_coverage90	no test	99.09 % (435/439)
test_branch_coverage80	no test	98.65 % (292/296)
build_reproducible	unknown	reproducible, with SOURCE_DATE_EPOCH

Contents

• test suite in tests/, a test extra, and branch coverage enabled in pyproject.toml
• CI workflow: tests on Python 3.9 to 3.13, a check that the wheel builds reproducibly, and the coverage report sent to SonarCloud
• sonar-project.properties so the report is imported
• requirements-ci.txt, a pinned toolchain used as a pip constraints file, so a CI run does not silently pick up a new release of pytest or build between two runs. Every pin supports Python 3.9 to 3.13. Actions are pinned to full commit SHAs for the same reason, with the major version kept in a trailing comment.
• PyYAML declared in the project dependencies. setup_ovs.py imports yaml to read .yaml and .yml configurations, but nothing required it, so an installed package crashed on those files.

Reproducible build

Measured both ways rather than assumed. Two builds with SOURCE_DATE_EPOCH pinned produce byte-for-byte identical wheels. Without it they differ: setuptools stamps the archive with the source file mtimes, which change on every checkout. The reproducible-build job therefore builds the wheel twice and compares the SHA-256 of the two archives.

Bugs found while writing the tests, to be fixed in a follow-up

This pull request changes no production code. git diff main..HEAD -- setup_ovs/ is empty. The five defects below were not introduced here: git log -L puts every one of them in 6c83aa0, "init repo from meta-seapath folder", 2022-05-21. They have been there since the repository was created, over four years ago.

Writing the tests did not create them, it executed those code paths for the first time. That is also why the coverage is not 100 %: the four uncovered statements in check.py are literally the dead code these defects produce, and no input can reach them.

They are deliberately left unfixed here. Correcting them means changing the behaviour of validation code that has been running on SEAPATH clusters for four years, which deserves its own pull request and its own discussion rather than being buried in a test suite. Mixing the two would also make this diff impossible to review: right now it adds tests and touches nothing else.

None of them is encoded as expected behaviour either. Each is marked xfail(strict=True) with its reason, so the day one is fixed the suite goes red and the marker has to be removed. The cleanup cannot be forgotten. They are listed in tests/test_helpers.py:180 and tests/test_check.py lines 324, 382, 427 and 520, and pytest -rx prints them with their reason on every run.

1. helpers.run_command does nothing when the caller passes check= (helpers.py:60). The return subprocess.run(...) sits inside the if "check" not in kargs branch. No caller passes check today, so it is latent, but it is a trap for the next one.
2. The VLAN tag range is never validated (check.py:283). The guard is if "vlan" in port while the attribute actually consumed by ovs._create_bridges is tag.
3. The "must be set if type is vxlan" check is unreachable (check.py:270): it sits inside if attribute in port and then tests attribute not in port. A vxlan port with neither key nor remote_ip passes the check, then raises KeyError further down in ovs.py:239.
4. The IANA VXLAN port 4789 is rejected (check.py:111). _attribute_is_a_port is documented as a TCP/UDP port check but enforces the VLAN tag range, 0 to 4095. A TCP/UDP port goes up to 65535.
5. The duplicate-NIC guard can never fire (check.py:165). dpdk_interfaces and system_interfaces are locals of _check_port_configuration, which runs once per port, so they are always empty.

SonarCloud analysis

The project now runs CI-based analysis. Automatic Analysis has been turned off
and SONAR_TOKEN is configured, so the scanner step actually runs and the
coverage report is imported. Automatic Analysis could never have done it: it
runs neither the build nor the tests, and the Sonar documentation states
plainly that "Code coverage information is not supported".

The first real analysis confirms the figures measured locally:

Sonar metric	Value
line_coverage	99.1 %
branch_coverage	98.6 %
coverage (combined)	98.9 %

The quality gate passes on its five conditions. new_coverage is not
evaluated, since new_lines_to_cover is empty: this branch adds no line to
sonar.sources.

Worth knowing for later: the analysis token expires. When it does, the scanner
will fail with a 403 and nothing will warn beforehand. That is exactly the
state vm_manager is in today, with a token issued in 2023.

Why the CI matrix does not start at Python 3.6

pyproject.toml declares requires-python = ">=3.6". That declaration is factually wrong: the package cannot run on 3.6.

subprocess.run() is called with capture_output=True in two places, setup_ovs/check.py:18 and setup_ovs/ovs.py:76. That argument was added in Python 3.7. On 3.6 both calls raise TypeError, which means system_check() and clear_ovs() fail immediately. So the real floor is at least 3.7, whatever the metadata says. Python 3.6 itself has been end of life since December 2021.

The matrix therefore runs 3.9 to 3.13, the versions available on current GitHub runners. 3.7 and 3.8 are also end of life and are no longer provided there, so testing the declared floor is not possible even if we wanted to.

requires-python is left untouched here, and so is the rest of the packaging metadata. Fixing the declared floor means deciding what the package is supported on, which depends on the SEAPATH deployment targets rather than on what the code happens to tolerate, and it belongs with a second metadata fix: the build warns that license = { text = "Apache-2.0" } should become a plain SPDX string, which requires raising setuptools to 77 in build-system and could break building on distributions that ship an older one. That warning is not fatal before 2027-02-18. Both go together in a dedicated packaging pull request, so this one keeps adding tests and touching nothing else.

[insatomcat]

Done, all seven switched to tags: checkout v6, setup-python v7, upload-artifact v7, download-artifact v8. I left the SHA on SonarSource/sonarqube-scan-action, since a third-party tag can be moved by its publisher and that job holds SONAR_TOKEN.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud