Differential testing

[Elias Castegren (EliasC)]

This pull request adds support for per-pass differential testing. It works by supplying the path to another implementation which acts as an oracle and generates input and expected output:

$ ./my_compiler test --diff /path/to/oracle
Testing x100, seed: 123456789
Using oracle command: ./path/to/oracle test -l None --oracle -c 100 -s 123456789 -d 10 -r 200 --gen_bound true <pass_name>
============
in oracle output:
(infix-ref
(infix-ident 2:$3))
------------
in resulting output:
(error
(errormsg 9:undefined)
(errorast
(infix-ident 2:$3)))
------------
<input tree>
------------
oracle output:
<tree>
------------
resulting output:
<tree>
============
Oracle mismatch for pass: my_pass

The oracle prints input/output pairs to stdout together with an indicator of well-formedness (OK or FAIL). The implementation under test reports whenever a tree does not match the expected output, presenting the first sub-tree that is different in the two trees.

This is opened as a draft to enable discussion. Below is a list of outstanding TODOs.

TODO

• Figure out how to deal with failures from the oracle
  • If the oracle has a wellformedness error but the implementation under test does not, it can be argued that we have fixed a bug and therefore shouldn't report it even though there is a difference in the output.
• Also support differential testing for sequences
  • The current design works pass by pass in order to avoid having to wait for the oracle to finish all its output, being able to abort early, etc. This is not compatible with the sequence testing which carries survivors between passes, and where it is not as clear how many trees there are to parse.
• Think about whether there is a more compact error reporting than printing all three trees, plus sub-trees.
  • The output of even a single error is currently quite noisy. One approach would be to print a single resulting tree where the diff is highlighted somehow.
• Get I/O working on Windows.