Run the standard library's own tests

BACKLOG.md

@@ -17,27 +17,15 @@ The container these bounds were added for now compiles and runs against the stan
 Jass, and compiles on Lua (#1239). What is left is generality around it rather than the feature
 itself, and one gap in what the suite can see.
 
-21. **Nothing executes a standard library program on Lua.** Every test which puts the library on
- that target compiles only, because the runtime shim cannot initialise the library's own
- packages — `GameTimer` fails first, and the generated fallback for an undefined native raises
- rather than returning. So the one thing a user actually does, running library code on Lua, is
- the one thing never run here.
-
- This is why `fastHashMapAgainstTheStandardLibraryLua` asserts on emitted shape instead of on a
- result. It is also how the empty-allocation bug in #1239 survived as long as it did: the paths
- which would have caught it are compiled and never executed.
-
- Worth finding out how far it is. If it is a handful of missing natives in `wc3shim.lua`, the
- payoff is every existing library test on that target becoming a real one. Start by capturing
- what `GameTimer` actually fails on rather than the message Wurst wraps it in.
-
-22. **A bump of the pinned library is only checked for compiling.** Nothing runs the library's own
- test functions, so a behaviour change in it is invisible here. `StdLibStringTests` (#1240)
- covers the string handling one bump turned on, which is a start rather than a solution: the
- library's multibyte detection degrades quietly to an ascii-only path when it cannot find what
- it probes for, so a version where it silently gave up would otherwise look exactly like one
- where it worked. Running the library's own tests generalises this, and its Lua half depends on
- item 21.
+22. **The library's own tests do not run on Lua.** They run on the interpreter now — all 460 of
+ them, collected by importing every package in the checkout whose name ends in `Tests`. That
+ half is done; this is the other one.
+
+ `executeTests` runs them through `RunTests` on the intermediate language, so it covers the
+ interpreter only. Running them on Lua needs the harness to execute a Wurst test function on
+ that target rather than an `init` block, which is new machinery rather than a flag. Worth it:
+ the two targets have disagreed before, and every disagreement found so far was found by running
+ the same program on both.
 
 6. **Lua dispatch inside the constructor** of a bounded generic class. Works on Jass; there is now
  a repro for both targets, `TypeClassTests.dispatchInsideConstructor` and
@@ -184,6 +172,19 @@ itself, and one gap in what the suite can see.
 
 ## Done
 
+- 21. A standard library program executes on Lua (#1242). Three packages could not initialise, each
+ on one native the shim did not define — `StringHash` for Colors, `Location` for Vectors,
+ `TimerStart` for GameTimer — and `StringCase` made a fourth once the program itself ran. The
+ larger half was the harness: success is read off stdout and the library's own `testSuccess` is
+ empty, so such a test could not have passed whatever it did.
+
+- 22 (interpreter half). The library's 460 test functions run. They were invisible because a library
+ compiles in only what is imported, so a program importing nothing runs none of them and passes;
+ the imports are collected from the checkout so a bump brings its new tests with it. `executeTests`
+ now reports how many ran and `expectAtLeastTests` fails when too few do, because "every test
+ passed" and "there were no tests" were the same green — which is how the first version of this
+ test looked right while running nothing.
+
 - 5, 11, 14. The container these bounds were added for works. `FastHashMapTests` runs a whole hash
  map — two type parameters with only the first bounded, static arrays of a bounded parameter, a
  bound reached from a private method, linear probing calling both requirements, tombstoned removal,

de.peeeq.wurstscript/src/test/java/tests/wurstscript/tests/StdLibOwnTests.java

@@ -0,0 +1,72 @@
+package tests.wurstscript.tests;
+
+import org.testng.annotations.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.testng.Assert.assertTrue;
+
+/**
+ * Runs the standard library's own {@code @Test} functions.
+ * <p>
+ * A bump of the pinned version was otherwise only checked for still compiling, which is a weak
+ * signal for a change whose point is behaviour, and a bad one for the parts of the library which
+ * degrade quietly rather than failing — the multibyte detection in {@code String.wurst} concludes
+ * the engine has no multibyte characters when it cannot find what it probes for, so a version where
+ * it silently gave up would look exactly like one where it worked.
+ * <p>
+ * The library is a library: only imported packages are compiled in, so a program importing nothing
+ * runs none of its tests and passes. The imports are therefore collected from the checkout rather
+ * than written down, which also means a test file added by a later bump is picked up by being there
+ * rather than by someone remembering.
+ */
+public class StdLibOwnTests extends WurstScriptTest {
+
+ /**
+ * Every package in the library whose name ends in {@code Tests}. Read from the file rather than
+ * assumed from the path, because a package need not be named after the file holding it.
+ */
+ private static List<String> testPackages() throws IOException {
+ Path root = new File(StdLib.getLib()).toPath();
+ List<String> packages = new ArrayList<>();
+ try (Stream<Path> files = Files.walk(root)) {
+ for (Path file : (Iterable<Path>) files.filter(p -> p.toString().endsWith("Tests.wurst"))::iterator) {
+ for (String line : Files.readAllLines(file, StandardCharsets.UTF_8)) {
+ String trimmed = line.trim();
+ if (trimmed.startsWith("package ")) {
+ packages.add(trimmed.substring("package ".length()).trim());
+ break;
+ }
+ }
+ }
+ }
+ packages.sort(String::compareTo);
+ return packages;
+ }
+
+ @Test
+ public void standardLibraryTestsPass() throws IOException {
+ List<String> packages = testPackages();
+ assertTrue(packages.size() > 20,
+ "expected the library to carry its test packages, found " + packages.size());
+
+ List<String> program = new ArrayList<>();
+ program.add("package test");
+ for (String p : packages) {
+ program.add("import " + p);
+ }
+ program.add("init");
+ program.add(" skip");
+
+ // One per test function in those packages. The floor is deliberately far below the real
+ // count: it is there to catch the program holding none, not to be updated on every bump.
+ test().withStdLib().expectAtLeastTests(100).lines(program.toArray(new String[0]));
+ }
+}

de.peeeq.wurstscript/src/test/java/tests/wurstscript/tests/WurstScriptTest.java

@@ -78,9 +78,18 @@ protected boolean printDebugScripts() {
  return false;
  }
 
+ /**
+ * How many Wurst tests the last run executed. Tests are stripped by the time the second pass
+ * happens, so this keeps the largest either saw. Zero means the program held none, which a
+ * caller expecting some needs to hear about: "every test passed" and "there were no tests" are
+ * otherwise the same green.
+ */
+ private int testsRun;
+
  @BeforeMethod(alwaysRun = true)
  public void _clearBefore() {
  GlobalCaches.clearAll();
+ testsRun = 0;
  }
 
  @AfterMethod(alwaysRun = true)
@@ -95,6 +104,7 @@ class TestConfig {
  private boolean withStdLib;
  private boolean executeProg;
  private boolean executeTests;
+ private int minimumTestsExpected;
  private boolean executeProgOnlyAfterTransforms;
  private String expectedError;
  private String expectedWarning;
@@ -135,6 +145,13 @@ public TestConfig executeTests() {
  return this;
  }
 
+ /** Fails when fewer than this many Wurst tests ran, so a program holding none is not green. */
+ public TestConfig expectAtLeastTests(int minimum) {
+ this.executeTests = true;
+ this.minimumTestsExpected = minimum;
+ return this;
+ }
+
  public TestConfig executeTests(boolean b) {
  this.executeTests = b;
  return this;
@@ -197,6 +214,11 @@ CompilationResult compilationUnits(CU... units) {
  CompilationResult run() {
  try {
  CompilationResult res = testScript();
+ if (minimumTestsExpected > 0 && testsRun < minimumTestsExpected) {
+ fail("expected at least " + minimumTestsExpected + " Wurst tests to run, but "
+ + testsRun + " did. A program which holds no tests passes every one of them,"
+ + " so this would otherwise be green while checking nothing.");
+ }
  if (expectedError != null) {
  if (res.getGui().getErrorCount() == 0) {
  fail("No errors were discovered");
@@ -1002,7 +1024,7 @@ private void translateAndTest(String name, boolean executeProg,
  if (!executeProgOnlyAfterTransforms) {
  // we want to test that the interpreter works correctly before transforming the program in the translation step
  if (executeTests) {
- executeTests(gui, compiler.getImTranslator(), imProg);
+testsRun = Math.max(testsRun, executeTests(gui, compiler.getImTranslator(), imProg));
  }
  if (executeProg) {
  WLogger.info("Executing imProg before jass transformation");
@@ -1021,7 +1043,7 @@ private void translateAndTest(String name, boolean executeProg,
  }
 
  if (executeTests) {
- executeTests(gui, compiler.getImTranslator(), imProg);
+testsRun = Math.max(testsRun, executeTests(gui, compiler.getImTranslator(), imProg));
  }
  if (executeProg) {
  WLogger.info("Executing imProg after jass transformation");
@@ -1132,13 +1154,15 @@ private void executeJassProg(JassProg prog)
  throw new Error(currentTestEnv + ": Succeed function not called");
  }
 
- private void executeTests(WurstGui gui, ImTranslator translator, ImProg imProg) {
+ /** @return how many tests ran, so a caller can tell "all passed" from "there were none". */
+ private int executeTests(WurstGui gui, ImTranslator translator, ImProg imProg) {
  RunTests runTests = new RunTests(Optional.empty(), 0, 0, Optional.empty());
  RunTests.TestResult res = runTests.runTests(translator, imProg, Optional.empty(), Optional.empty());
  if (res.getPassedTests() < res.getTotalTests()) {
  throw new Error("tests failed: " + res.getPassedTests() + " / " + res.getTotalTests() + "\n" +
  gui.getErrors());
  }
+ return res.getTotalTests();
  }
 
  /**