Uh oh!
There was an error while loading. Please reload this page.
Run the standard library's own tests - #1243
Merged
Merged
Conversation
A bump of the pinned version was only checked for still compiling, which says little about a change whose point is behaviour and less about the parts of the library which degrade quietly: the multibyte detection in String.wurst decides the engine has no multibyte characters when it cannot find what it probes for, so a version where it silently gave up looked exactly like one where it worked. All 460 of its test functions run now. They were invisible because a library compiles in only what is imported, so a program importing nothing runs none of them - and the first version of this test passed while doing exactly that. The imports are collected from the checkout rather than written down, so a later bump brings its new test files with it. That near miss is why executeTests reports how many ran and expectAtLeastTests fails when too few do: every test passing and there being no tests were the same green, in a harness whose whole job is to tell those apart.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A bump of the pinned standard library was only ever checked for still compiling. That is a weak signal for a change whose point is behaviour, and a bad one for the parts of the library which degrade quietly rather than failing: the multibyte detection in
String.wurstconcludes the engine has no multibyte characters when it cannot find what it probes for, so a version where it silently gave up would look exactly like one where it worked.All 460 of the library's test functions run now, on the interpreter.
Why they were invisible
A library compiles in only what is imported, so a program importing nothing pulls in no test functions at all — and
executeTestsfails only whenpassed < total, which0 < 0is not.The first version of this test therefore passed while running nothing. It looked exactly like the version that works. I caught it by printing the count rather than trusting the colour, and that near miss shaped the rest of the change.
What it does
StdLibOwnTestscollects every package in the checkout whose name ends inTests, reading the package name out of the file rather than assuming it matches the path, and imports them all. A test file added by a later bump is picked up by being there rather than by someone remembering to list it.executeTestsreturns how many tests ran, andexpectAtLeastTests(n)fails when fewer thanndid. The floor is deliberately far below the real count — it exists to catch a program holding none, not to be updated on every bump.The guard is checked rather than assumed: raising the floor to 100000 produces
expected at least 100000 Wurst tests to run, but 460 did, which is also where the 460 comes from.Scope
The interpreter half only.
executeTestsruns the tests throughRunTestson the intermediate language, so running them on Lua needs the harness to execute a Wurst test function on that target rather than aninitblock — new machinery rather than a flag, and left as backlog item 22. It is worth doing: every disagreement between the targets found so far was found by running the same program on both.Full suite green, 8m41s.