add UI testing framework

[nikomatsakis]

This adds a framework for capturing and tracking the precise output of rustc, which allows us to check all manner of minor details with the output. It's pretty strict right now -- the output must match almost exactly -- and hence maybe a bit too strict. But I figure we can add wildcards or whatever later. There is also a script intended to make updating the references easy, though the script could make things a bit easier (in particular, it'd be nice if it would find the build directory for you automatically).

One thing I was wondering about is the best way to test colors. Since windows doesn't embed those in the output stream, this test framework can't test colors on windows -- so I figure we can just write tests that are ignored on windows and which pass --color=always or whatever to rustc.

cc @jonathandturner
r? @alexcrichton

[alexcrichton]

May be worth using String::from_utf8_lossy instead of "not uf8" on invalid utf-8

[retep998]

It is possible to create a console screen buffer, redirect rustc to that, and then read the output including all the color attributes. A bit complicated, but nonetheless very much possible.

[alexcrichton]

Nice! Some thoughts:

• Could you add this to rustbuild as well? If you search for "rpass" it should just be a similar set of modifications to that.
• Cargo has a similar method for diffing output, some other normalizations it applies are removing all carriage returns and making tabs more readable as <tab> to match in the output. Neither of those may apply too much here.
• Cargo's method calculates differences between lines of input and output to print out out only the lines that mismatch. The only clever thing it does here is to use [..]as a placeholder for "any text"

I'd personally want to get rustbuild running these tests as well as printing the actual/expected on failure, but other than that the remaining pieces seem like they're nice-to-have and we can add later. Having a script to blanket overwrite all the previous output may also be a bit heavy-handed? In theory these shouldn't be too hard to run by hand, right? I'd just be worried that having these scripts checked in makes it "too easy" to blanket change the rustc output without thinking much.

[nikomatsakis]

@alexcrichton

Cargo's method calculates differences between lines of input and output to print out out only the lines that mismatch. The only clever thing it does here is to use [..] as a placeholder for "any text"

I may try to at least get diffs of the output working.

I'd personally want to get rustbuild running these tests as well as printing the actual/expected on failure, but other than that the remaining pieces seem like they're nice-to-have and we can add later.

I agree those are necessary.

Having a script to blanket overwrite all the previous output may also be a bit heavy-handed?...I'd just be worried that having these scripts checked in makes it "too easy" to blanket change the rustc output without thinking much.

This was sort of the goal =) I figured we'd make it easy to update, then use review + git diff to check the results.

@retep998

It is possible to create a console screen buffer, redirect rustc to that, and then read the output including all the color attributes. A bit complicated, but nonetheless very much possible.

Cool. Nonetheless, after discussing with @jonathandturner and @alexcrichton, we were thinking we'd just move all color checking to operate at a more semantic level rather than using this framework (i.e., if we get more "presentation" info in the JSON as discussed here, we can check that this is as expected, and then we'll just trust that the translation to ANSI/Windows goes swimmingly).

[alexcrichton]

@nikomatsakis all sounds good to me!

[nikomatsakis]

@alexcrichton ok, I did the following:

• normalize \r\n to \n (I have tested this on windows and this normalization wasn't needed, but it seems like a good idea)
• normalize tabs to \r
• ensured that we always dump the (normalized) actual and expected output
• print a simple diff using cargo's code (maybe worth pulling in code from one of these projects instead, but that can happen later)
• added to rust-build (but that...is not quite working yet, still investigating)

I chose not to:

• keep the [..] wildcard:
  • I hope we can get away with normalization: otherwise, the compiler can't automatically produce the reference for you to copy in.
  • but it'd be simple enough to add back. maybe I should just add it proactively.

[alexcrichton]

Could you be sure to add these to the blanket Check target below?

[alexcrichton]

Looks good to me! Just one minor nit and otherwise r=me

[nikomatsakis]

@bors r=alexcrichton

[bors]

📌 Commit 3ff521b has been approved by alexcrichton

[bors]

⌛ Testing commit 3ff521b with merge 9c87f71...

[bors]

⛄ The build was interrupted to prioritize another pull request.

[bors]

⌛ Testing commit 3ff521b with merge 9bbfe4d...

[bors]

💔 Test failed - auto-linux-64-opt-rustbuild

[sanxiyn]

This failed tidy.

[nikomatsakis]

@bors r=alexcrichton

[bors]

📌 Commit 474eb38 has been approved by alexcrichton

[nikomatsakis]

@bors r=acrichto

[bors]

📌 Commit 24cfa1e has been approved by acrichto

[bors]

⌛ Testing commit 24cfa1e with merge cd6a400...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-32cross-opt, auto-linux-64-cargotest, auto-linux-64-cross-armhf, auto-linux-64-cross-armsf, auto-linux-64-cross-freebsd, auto-linux-64-cross-netbsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-mir, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-cross-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-mir, auto-win-msvc-64-opt-rustbuild