refactor test directories by mazunki · Pull Request #2326 · includeos/IncludeOS

mazunki · 2025-10-20T16:30:06Z

This PR reorganizes the ./test/ subdirectory and reunites a few files which are specific to tests only into this subdirectory. ./test.sh is now under here too.

There is now a clear separation between what constitutes as a unit test, and what constitutes as an integration test at the top level. Before, this was done under each target. This will make it easier to later refactor ./test.sh.

This comes with quite many atomic commits, which might seem scary at first sight; but doing it this way makes it easier to understand each change. I hope.

I am pretty confident we're running all the same tests, but I would appreciate someone double checking it. All 8 (previously 7) test sets are passing on my end.

I've noticed several tests are never actually being run: util/{tar,tar_gz}/*, posix/*, plugin/*, mod/*, hw/*, fs/*, unless I'm missing something. This remains the same as it was.

I also cannot sea any use of /userspace nor /test/userspace within any of the scripts. I haven't touched these, but I have to say it's a bit confusing: what does it do, and why does it exist? Is it legacy code?

In a future PR I would like to move unittests.nix into here too, but (while still possible) would be better done after refactoring some of the other *.nix files to reduce maintenance burden.

mazunki · 2025-10-20T16:34:52Z

Useful tip: tree -d test/ is useful to navigate tests. find -empty is also worth looking at:

$ find -empty
./.git/objects/info
./.git/refs/tags
./api/net/http/parse.hpp
./src/plugins/system_log.cpp
./test/tests-integration/hw/serial/README.md
./test/tests-integration/performance/.keep
./test/tests-integration/fs/memdisk/sector0.disk
./test/tests-unit/performance/.keep
./test/tests-unit/stl/.keep

Also there's several references to conan:

$ grep -rn conan
grep: .git/objects/pack/pack-f1464c7e43d57642af99b907de2ae8a1459fb08b.pack: binary file matches
CHANGELOG.md:13: - 3rd party dependencies are now built and managed in Jenkins. All recipes can be found [here](https://github.com/includeos/conan)
CHANGELOG.md:17: - A repo to install Conan configs for IncludeOS: [conan_config](https://github.com/includeos/conan_config)
cmake/includeos.cmake:54:#TODO improve this to get the platform from conan somehow
cmake/os.cmake:282: #if in conan expect it to be in order ?
deps/uzlib/default.nix:17: # same version as listed in ./conanfile.py
deps/botan/WIP.md:2:The current conanfile builds botan 3.4, which is the latest at the time of writing.
deps/s2n/default.nix:8: # ./conanfile.py lists 0.8, but there are not tags in the repo with version < 0.9.0
lib/LiveUpdate/README.md:3:# Put Liveupdate in conan editable mode
lib/LiveUpdate/README.md:6:1. Remove the version variable from the `conanfile.py`.
lib/LiveUpdate/README.md:8:awk '!/version = conan_tools/' lib/LiveUpdate/conanfile.py > temp && mv temp lib/LiveUpdate/conanfile.py
lib/LiveUpdate/README.md:12:conan editable add lib/LiveUpdate liveupdate/9.9.9@includeos/latest --layout lib/LiveUpdate/layout.txt
lib/LiveUpdate/README.md:16:conan install -if lib/LiveUpdate/build lib/LiveUpdate
lib/LiveUpdate/README.md:20:conan build -sf . -bf lib/LiveUpdate/build lib/LiveUpdate
src/net/CMakeLists.txt:12: #TODO get from conan
src/plugins/CMakeLists.txt:43:#handle build targets for conan editable
test/misc/solo5-hvt/test.sh:18:conan install .. -pr clang-6.0-linux-x86_64
vmbuild/CMakeLists.txt:20:#TODO pull vmbuild conanfile.py inn when not building with conan to get deps
.gitignore:35:build_conan/

I just mention this since it's probably worth cleaning that up at some point.

alfreb

Conditional approval: I prefer ./test/integration over ./test/tests-integration - besides that this is all good.

There is a case to be made for keeping tests testing the same thing next to one another, since most humans will probably find it easier to understand which tests we have for a certain set of features if they're bundled together. So you're prioritizing reorganizing this to make test.sh maintainance slightly easier - which was not very hard to begin with. Still, I don't object - it's no big deal. Just change the name.

Adding a separate category for memory tests is a big improvement - thanks for that!

alfreb · 2025-10-22T18:27:55Z

From the perspective of maintaining test.sh this makes sense. But from the perspective of working on a specific feature, or understanding what coverage we have for it, it's a bit worse. But either way is fine, so this is ok.

I intend to make that sort of overview rather simple after refactoring test.sh (again). If we requested to test some module (say net/tcp), it will report that there were no unit tests, integration tests or performance/stres found for that unit. Not an error, but an informative message.
Personally I use find and tree a lot for this form of generic overview, but I do see your point.

alfreb · 2025-10-22T18:42:27Z

Oh well, ok.

You prefer to clutter the top level directory? I was hoping to move unittests.nix into here eventually too. I can revert it if you think that's better.

alfreb · 2025-10-22T18:42:53Z

 # to make sure core functionality is not broken by missing locks etc. when waking up more cores.
 nix-shell --pure --arg smp true $CCACHE_FLAG --argstr unikernel ${INTEGRATION_TESTS}/net/udp --run ./test.py
- nix-shell --pure --arg smp true $CCACHE_FLAG --argstr unikernel ${INTEGRATION_TESTS}/kernel/paging --run ./test.py
+ nix-shell --pure --arg smp true $CCACHE_FLAG --argstr unikernel ${INTEGRATION_TESTS}/memory/paging --run ./test.py


Yep, this is better

Related: I don't have a good argument, but I feel ilke kernel is a very broad term for anything test-related. Everything is a kernel test, is it not? The files under it probably should all be moved to more semantic locations.

alfreb · 2025-10-22T18:43:35Z

 ${UNIT_TESTS}/util/base64.cpp
 ${UNIT_TESTS}/util/bitops.cpp
- ${UNIT_TESTS}/util/buddy_alloc_test.cpp
+ ${UNIT_TESTS}/memory/alloc/buddy_alloc_test.cpp


Yep, much better

Downside to this change is that it no longer matches the structure of src/. Oh well. (Maybe that should be moved too)

alfreb · 2025-10-22T18:45:00Z

+#
+exclusions=(
+)
+run_testsuite "${INTEGRATION_TESTS}/memory" "${exclusions[@]}"


Did you catch the other missing ones? I feel they should be added too, but probably best on its own PR.

mazunki · 2025-10-22T22:28:06Z

Removed the prefix and reverted performance to stress, all 8 tests are passing ok.

mazunki · 2025-10-27T15:02:17Z

Rebased to merge with #2316. Tests are still happy.

MagnusS · 2025-10-28T14:06:04Z

@mazunki looks like #2309 introduced new conflicts...

mazunki · 2025-10-28T14:24:28Z

Weird. It rebased cleanly on my end. Let's see if this helps. Running tests now.

mazunki · 2025-10-28T14:38:27Z

Yep, they're all fine. Maybe Github just wanted confirmation on the path changes?

MagnusS

Generally lgtm, but README also needs an update to refer to the new test.sh location

mazunki · 2025-10-29T09:53:09Z

I have updated the README. One can now also actually use the unikernel.nix as a shell. Tests are still passing on my end.

alfreb approved these changes Oct 22, 2025
View reviewed changes

mazunki force-pushed the ref-dirs branch 2 times, most recently from 9f3a85d to 382b435Compare October 27, 2025 15:01

mazunki added 17 commits October 28, 2025 15:23

move integration tests to shared directory
77391af

move unit tests to shared directory
3f609bb

delete unused directories
b450250

convert stress test into targeted subtest
b48c305

move testing scripts into testing dir
82fa2b5

inline traps for readability
b1c748d

rename for readbility
1d2c5be

move into subdirectory
dc5929b

unite image with its unit test
2abe4ad

align tests to new paths
e1c87a3

update wrong path
1bde56a

move test script into subdir
636546d

rename confusing name
656d6df

give memory tests their own testing scope
298df4a

give memory unit tests their own scope
6d3d3f0

enable memory tests
ffed7ed

remove tests- prefix
86c6f06

mazunki force-pushed the ref-dirs branch from 382b435 to 86c6f06Compare October 28, 2025 14:24

MagnusS requested changes Oct 29, 2025
View reviewed changes

mazunki added 2 commits October 29, 2025 10:42

update README to match paths and unikernel.nix changes
5cb809c

permit entering the shell when not testing
d61052b

MagnusS approved these changes Oct 29, 2025
View reviewed changes

MagnusS merged commit ddd0585 into includeos:mainOct 29, 2025

Uh oh!

Conversation

mazunki commented Oct 20, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mazunki commented Oct 20, 2025

Uh oh!

alfreb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mazunki commented Oct 22, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mazunki commented Oct 27, 2025

Uh oh!

MagnusS commented Oct 28, 2025

Uh oh!

mazunki commented Oct 28, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mazunki commented Oct 28, 2025

Uh oh!

MagnusS left a comment

Choose a reason for hiding this comment

Uh oh!

mazunki commented Oct 29, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mazunki commented Oct 20, 2025•
edited
Loading

mazunki commented Oct 22, 2025•
edited
Loading

mazunki commented Oct 28, 2025•
edited
Loading

mazunki commented Oct 29, 2025•
edited
Loading