diff --git a/BACKLOG.md b/BACKLOG.md index 86165e337..379cbc813 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -17,27 +17,15 @@ The container these bounds were added for now compiles and runs against the stan Jass, and compiles on Lua (#1239). What is left is generality around it rather than the feature itself, and one gap in what the suite can see. -21. **Nothing executes a standard library program on Lua.** Every test which puts the library on - that target compiles only, because the runtime shim cannot initialise the library's own - packages — `GameTimer` fails first, and the generated fallback for an undefined native raises - rather than returning. So the one thing a user actually does, running library code on Lua, is - the one thing never run here. - - This is why `fastHashMapAgainstTheStandardLibraryLua` asserts on emitted shape instead of on a - result. It is also how the empty-allocation bug in #1239 survived as long as it did: the paths - which would have caught it are compiled and never executed. - - Worth finding out how far it is. If it is a handful of missing natives in `wc3shim.lua`, the - payoff is every existing library test on that target becoming a real one. Start by capturing - what `GameTimer` actually fails on rather than the message Wurst wraps it in. - -22. **A bump of the pinned library is only checked for compiling.** Nothing runs the library's own - test functions, so a behaviour change in it is invisible here. `StdLibStringTests` (#1240) - covers the string handling one bump turned on, which is a start rather than a solution: the - library's multibyte detection degrades quietly to an ascii-only path when it cannot find what - it probes for, so a version where it silently gave up would otherwise look exactly like one - where it worked. Running the library's own tests generalises this, and its Lua half depends on - item 21. +22. **The library's own tests do not run on Lua.** They run on the interpreter now — all 460 of + them, collected by importing every package in the checkout whose name ends in `Tests`. That + half is done; this is the other one. + + `executeTests` runs them through `RunTests` on the intermediate language, so it covers the + interpreter only. Running them on Lua needs the harness to execute a Wurst test function on + that target rather than an `init` block, which is new machinery rather than a flag. Worth it: + the two targets have disagreed before, and every disagreement found so far was found by running + the same program on both. 6. **Lua dispatch inside the constructor** of a bounded generic class. Works on Jass; there is now a repro for both targets, `TypeClassTests.dispatchInsideConstructor` and @@ -184,6 +172,19 @@ itself, and one gap in what the suite can see. ## Done +- 21. A standard library program executes on Lua (#1242). Three packages could not initialise, each + on one native the shim did not define — `StringHash` for Colors, `Location` for Vectors, + `TimerStart` for GameTimer — and `StringCase` made a fourth once the program itself ran. The + larger half was the harness: success is read off stdout and the library's own `testSuccess` is + empty, so such a test could not have passed whatever it did. + +- 22 (interpreter half). The library's 460 test functions run. They were invisible because a library + compiles in only what is imported, so a program importing nothing runs none of them and passes; + the imports are collected from the checkout so a bump brings its new tests with it. `executeTests` + now reports how many ran and `expectAtLeastTests` fails when too few do, because "every test + passed" and "there were no tests" were the same green — which is how the first version of this + test looked right while running nothing. + - 5, 11, 14. The container these bounds were added for works. `FastHashMapTests` runs a whole hash map — two type parameters with only the first bounded, static arrays of a bounded parameter, a bound reached from a private method, linear probing calling both requirements, tombstoned removal, diff --git a/de.peeeq.wurstscript/src/test/java/tests/wurstscript/tests/StdLibOwnTests.java b/de.peeeq.wurstscript/src/test/java/tests/wurstscript/tests/StdLibOwnTests.java new file mode 100644 index 000000000..b0e56cc76 --- /dev/null +++ b/de.peeeq.wurstscript/src/test/java/tests/wurstscript/tests/StdLibOwnTests.java @@ -0,0 +1,72 @@ +package tests.wurstscript.tests; + +import org.testng.annotations.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Stream; + +import static org.testng.Assert.assertTrue; + +/** + * Runs the standard library's own {@code @Test} functions. + *

+ * A bump of the pinned version was otherwise only checked for still compiling, which is a weak + * signal for a change whose point is behaviour, and a bad one for the parts of the library which + * degrade quietly rather than failing — the multibyte detection in {@code String.wurst} concludes + * the engine has no multibyte characters when it cannot find what it probes for, so a version where + * it silently gave up would look exactly like one where it worked. + *

+ * The library is a library: only imported packages are compiled in, so a program importing nothing + * runs none of its tests and passes. The imports are therefore collected from the checkout rather + * than written down, which also means a test file added by a later bump is picked up by being there + * rather than by someone remembering. + */ +public class StdLibOwnTests extends WurstScriptTest { + + /** + * Every package in the library whose name ends in {@code Tests}. Read from the file rather than + * assumed from the path, because a package need not be named after the file holding it. + */ + private static List testPackages() throws IOException { + Path root = new File(StdLib.getLib()).toPath(); + List packages = new ArrayList<>(); + try (Stream files = Files.walk(root)) { + for (Path file : (Iterable) files.filter(p -> p.toString().endsWith("Tests.wurst"))::iterator) { + for (String line : Files.readAllLines(file, StandardCharsets.UTF_8)) { + String trimmed = line.trim(); + if (trimmed.startsWith("package ")) { + packages.add(trimmed.substring("package ".length()).trim()); + break; + } + } + } + } + packages.sort(String::compareTo); + return packages; + } + + @Test + public void standardLibraryTestsPass() throws IOException { + List packages = testPackages(); + assertTrue(packages.size() > 20, + "expected the library to carry its test packages, found " + packages.size()); + + List program = new ArrayList<>(); + program.add("package test"); + for (String p : packages) { + program.add("import " + p); + } + program.add("init"); + program.add(" skip"); + + // One per test function in those packages. The floor is deliberately far below the real + // count: it is there to catch the program holding none, not to be updated on every bump. + test().withStdLib().expectAtLeastTests(100).lines(program.toArray(new String[0])); + } +} diff --git a/de.peeeq.wurstscript/src/test/java/tests/wurstscript/tests/WurstScriptTest.java b/de.peeeq.wurstscript/src/test/java/tests/wurstscript/tests/WurstScriptTest.java index 8ed7d2e71..ebffa759a 100644 --- a/de.peeeq.wurstscript/src/test/java/tests/wurstscript/tests/WurstScriptTest.java +++ b/de.peeeq.wurstscript/src/test/java/tests/wurstscript/tests/WurstScriptTest.java @@ -78,9 +78,18 @@ protected boolean printDebugScripts() { return false; } + /** + * How many Wurst tests the last run executed. Tests are stripped by the time the second pass + * happens, so this keeps the largest either saw. Zero means the program held none, which a + * caller expecting some needs to hear about: "every test passed" and "there were no tests" are + * otherwise the same green. + */ + private int testsRun; + @BeforeMethod(alwaysRun = true) public void _clearBefore() { GlobalCaches.clearAll(); + testsRun = 0; } @AfterMethod(alwaysRun = true) @@ -95,6 +104,7 @@ class TestConfig { private boolean withStdLib; private boolean executeProg; private boolean executeTests; + private int minimumTestsExpected; private boolean executeProgOnlyAfterTransforms; private String expectedError; private String expectedWarning; @@ -135,6 +145,13 @@ public TestConfig executeTests() { return this; } + /** Fails when fewer than this many Wurst tests ran, so a program holding none is not green. */ + public TestConfig expectAtLeastTests(int minimum) { + this.executeTests = true; + this.minimumTestsExpected = minimum; + return this; + } + public TestConfig executeTests(boolean b) { this.executeTests = b; return this; @@ -197,6 +214,11 @@ CompilationResult compilationUnits(CU... units) { CompilationResult run() { try { CompilationResult res = testScript(); + if (minimumTestsExpected > 0 && testsRun < minimumTestsExpected) { + fail("expected at least " + minimumTestsExpected + " Wurst tests to run, but " + + testsRun + " did. A program which holds no tests passes every one of them," + + " so this would otherwise be green while checking nothing."); + } if (expectedError != null) { if (res.getGui().getErrorCount() == 0) { fail("No errors were discovered"); @@ -1002,7 +1024,7 @@ private void translateAndTest(String name, boolean executeProg, if (!executeProgOnlyAfterTransforms) { // we want to test that the interpreter works correctly before transforming the program in the translation step if (executeTests) { - executeTests(gui, compiler.getImTranslator(), imProg); + testsRun = Math.max(testsRun, executeTests(gui, compiler.getImTranslator(), imProg)); } if (executeProg) { WLogger.info("Executing imProg before jass transformation"); @@ -1021,7 +1043,7 @@ private void translateAndTest(String name, boolean executeProg, } if (executeTests) { - executeTests(gui, compiler.getImTranslator(), imProg); + testsRun = Math.max(testsRun, executeTests(gui, compiler.getImTranslator(), imProg)); } if (executeProg) { WLogger.info("Executing imProg after jass transformation"); @@ -1132,13 +1154,15 @@ private void executeJassProg(JassProg prog) throw new Error(currentTestEnv + ": Succeed function not called"); } - private void executeTests(WurstGui gui, ImTranslator translator, ImProg imProg) { + /** @return how many tests ran, so a caller can tell "all passed" from "there were none". */ + private int executeTests(WurstGui gui, ImTranslator translator, ImProg imProg) { RunTests runTests = new RunTests(Optional.empty(), 0, 0, Optional.empty()); RunTests.TestResult res = runTests.runTests(translator, imProg, Optional.empty(), Optional.empty()); if (res.getPassedTests() < res.getTotalTests()) { throw new Error("tests failed: " + res.getPassedTests() + " / " + res.getTotalTests() + "\n" + gui.getErrors()); } + return res.getTotalTests(); } /**