extend compiletest to support revisions, incremental tests

[nikomatsakis]

This PR extends compiletest to support test revisions and with a preliminary incremental testing harness. run-pass, compile-fail, and run-fail tests may be tagged with

// revisions: a b c d

This will cause the test to be re-run four times with --cfg {a,b,c,d} in turn. This means you can write very closely related things using cfg. You can also configure the headers/expected-errors by writing //[foo] header: value or //[foo]~ ERROR bar, where foo is the name of your revision. See the changes to coherence-cow.rs as a proof of concept.

The main point of this work is to support the incremental testing harness. This PR contains an initial, unused version. The code that uses it will land later. The incremental testing harness compiles each revision in turn, and requires that the revisions have particular names (e.g., rpass2, cfail3), which tell it whether a particular revision is expected to compile or not.

Two questions:

• Is there compiletest documentation anywhere I can update?
• Should I hold off on landing the incremental testing harness until I have the code to exercise it? (That will come in a separate PR, still fixing a few details)

r? @alexcrichton
cc @rust-lang/compiler <-- new testing capabilities

[eddyb]

This is great! LLVM's FileCheck has a similar feature, where you can include several test commands in a single file and a prefix for each one such that you check each of their outputs independently.

I wonder if we could extend this to work for MIR or even LLVM IR tests, to avoid depending on FileCheck.

[nikomatsakis]

PS, it'd be nice if we could write automated tests for compiletest itself, but I don't know of any way to do that right now... but I have manually tested this ;)

[eddyb]

@nikomatsakis Would a marker for "this test must fail (according to compiletest)" suffice?

[nikomatsakis]

@eddyb I could see two things:

• add some "metatests" into the makefile that are expected to fail
• or add a header switch to "invert" expectations

The latter requires us still to trust compiletest, but maybe a bit less (and of course the former requires trusting makefiles, which is probably scarier).

[nikomatsakis]

yeah, thinking about I like the idea of a // should-fail header. I think the meta runner of compiletest could detect and handle such a header with relative ease. I'll investigate.

[apasel422]

@nikomatsakis There's some compiletest documentation in COMPILER_TESTS.md.

[alexcrichton]

Shouldn't this just be fs::remove_dir_all?

[alexcrichton]

Looks good to me, nice idea @nikomatsakis!

Implementation looks all good to me, although I didn't really audit the new incremental tests that closely. I also basically just skimmed the changes as "yep, looks like a refactor". Feel free to r=me whenever this is ready.

[nikomatsakis]

Decided I might as well land the incremental runner separately. I may well need to tweak it some more, so it seems clearer to put it in the same PR as the code which uses it.

[nikomatsakis]

@bors r=alexcrichton

[bors]

📌 Commit fff9c2b has been approved by alexcrichton

[bors]

⌛ Testing commit fff9c2b with merge 356759a...

[bors]

💔 Test failed - auto-linux-64-opt

[nikomatsakis]

@bors r=alexcrichton

[bors]

📌 Commit 9601e6f has been approved by alexcrichton

[bors]

⌛ Testing commit 9601e6f with merge 71669bb...

[bors]

💔 Test failed - auto-linux-64-opt

[nikomatsakis]

@bors r=alexcrichton

[bors]

📌 Commit fc4d0ec has been approved by alexcrichton

[nikomatsakis]

ok, this should work now. I made the executive call that should-fail will be ignored for pretty printing tests, since usually they are checking something different than the thing which should-fail was intending to test. Another option would have to been to add // ignore-pretty to meta tests, but this seemed to be more likely to be what we want. We can add a should-fail-pretty header if we ever want that.

[bors]

⌛ Testing commit fc4d0ec with merge 493d999...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-mac-ios-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-opt