Convert `run-make/coverage-reports` tests to use a custom compiletest mode by Zalathar · Pull Request #112300 · rust-lang/rust

Zalathar · 2023-06-05T03:19:05Z

I was frustrated by the fact that most of the coverage tests are glued together with makefiles and shell scripts, so I tried my hand at converting most of them over to a newly-implemented run-coverage mode/suite in compiletest.

This ~~mostly~~resolves#85009, ~~though I've left a small number of the existing tests as-is because they would require more work to fix/support~~.

I had time to go back and add support for the more troublesome tests that I had initially skipped over, so this PR now manages to completely get rid of run-make/coverage-reports.

The patches are arranged as follows:

Declare the new mode/suite in bootstrap
Small changes to compiletest that will be used by the new mode
Implement the new mode in compiletest
Migrate most of the tests over
Add more code to bootstrap and compiletest to support the remaining tests
Migrate the remaining tests (with some temporary hacks to avoid re-blessing them)
Remove the temporary hacks and re-bless the migrated tests
Remove the unused remnants of run-make/coverage-reports

rustbot · 2023-06-05T03:19:13Z

r? @wesleywiser

(rustbot has picked a reviewer for you, use r? to override)

Zalathar · 2023-06-05T03:23:56Z

The part I’m least confident about is the implied needs-profiler-support check for each test, since the existing needs-checking code doesn't really anticipate that sort of use-case.

Zalathar · 2023-06-05T03:29:34Z

@rustbot label +A-code-coverage

wesleywiser · 2023-06-06T15:31:19Z

The part I’m least confident about is the implied needs-profiler-support check for each test, since the existing needs-checking code doesn't really anticipate that sort of use-case.

This part looks ok to me but if @jyn514 wants to check as well, that would be awesome!

ignore-windows-gnu

If I recall correctly, code coverage isn't supported on windows-gnu which is why the tests ignore it.

Zalathar · 2023-06-07T00:45:27Z

Rebased and added some small tweaks based on feedback:

Adjusted comments and control flow in the bootstrap code that supplies --llvm-bin-dir
profiler_support is made pub(super) in the patch that actually needs it
Added a comment to the profiler_support check

I haven't looked into the ignore-windows-gnu situation.

Zalathar · 2023-06-07T02:51:12Z

In light of still needing to support ignore-windows-gnu, what I might do is add a way to inject “extra” directives into iter_header, so that they are processed as though they appeared directly in the file. That seems like the least intrusive way to support arbitrary mode-implied directives.

Zalathar · 2023-06-07T03:23:57Z

OK, I’ve fundamentally changed how the mode-implied directives work.

iter_header_extra now takes a list of caller-supplied directives that should be processed as though they appeared on “line 0” of the file being inspected. There's a hard-coded check that adds three extra directives for Mode::RunCoverage tests, and does nothing for all other modes.

This is still a little bit gross, but I think it’s the best compromise for now.

Zalathar · 2023-06-07T03:26:45Z

(A side-effect of the above is that it’s no longer necessary to move profiler_support into the needs cache struct, but I’ve left that change intact because I think it’s a reasonable little improvement in its own right.)

Zalathar · 2023-06-12T03:05:15Z

I've now migrated all of the run-make/coverage-reports tests, so some code has been changed/added since the last review.

Zalathar · 2023-06-12T12:29:51Z

Looks like this is all working again now.

Zalathar · 2023-06-12T22:43:36Z

Dealing with doctest.rs was tricky, since it’s inherently a lot more complicated than the other tests.

I was tempted to leave it as its own run-make test, but that would ultimately leave us with make/shell/Python code duplicating much of run-coverage.
I thought about including it in the main run-coverage suite, but that would force the whole suite to require rustdoc (which is a real cost from a clean build), and require a new directive to actually trigger running the doctests (or running them unconditionally for everything).

So while it feels a bit weird to introduce a whole extra suite for one test, I think it ends up being the more natural solution.

wesleywiser

This is looking pretty good to me! I noticed there used to be a mechanism in the coverage makefile to bless the tests, does --bless still work?

wesleywiser · 2023-06-13T14:31:55Z

Thanks for including these FIXMEs here, that will make it easier to understand in the future when they can be removed!

Zalathar · 2023-06-13T22:18:01Z

This is looking pretty good to me! I noticed there used to be a mechanism in the coverage makefile to bless the tests, does --bless still work?

Yes, --bless is now handled automatically by the compiletest methods for comparing output. I’ve used it on these migrated tests to manually verify that it still works (e.g. in the patch that re-blesses some tests).

…iaskrgr Rollup of 7 pull requests Successful merges: - rust-lang#111322 (Support for native WASM exceptions) - rust-lang#112086 (resolve: Remove artificial import ambiguity errors) - rust-lang#112234 (refactor `tool_doc!`) - rust-lang#112300 (Convert `run-make/coverage-reports` tests to use a custom compiletest mode) - rust-lang#112795 (Migrate some rustc_builtin_macros to SessionDiagnostic) - rust-lang#113144 (Make the `Elaboratable` trait take clauses) - rust-lang#113161 (Fix type privacy lints error message) r? `@ghost` `@rustbot` modify labels: rollup

Re-enable some coverage tests on Linux These tests were originally disabled (on all platforms) in rust-lang#110393, because those changes had made them start failing on Linux for unclear reasons. I tried to re-enable them unconditionally in rust-lang#111179, since they worked locally on my Mac, but I found that they were still failing on Linux, so I gave up at that time. Later while working on rust-lang#112300 I was able to re-enable them on Windows and Mac, since those changes made it possible to add specific `ignore-` directives to individual tests. I noticed at the time that the tests actually seemed to be working again on Linux, but by that point I didn't want to risk more CI failures, so I left them disabled on Linux with an intention to re-enable them later. Now I'm going back to re-enable them on Linux too, since they seem to work fine. --- Because `run-coverage` tests are sensitive to line numbers, and `x test tidy` doesn't like leading blank lines, I've replaced the old comment/ignore with an informative comment that occupies the same number of lines.

Prior to rust-lang#114875, these tests were very sensitive to lines being added/removed, so the migration to `run-coverage` in rust-lang#112300 tried hard to avoid disturbing the existing line numbers. That resulted in some awkward reshuffling when certain comments/directives needed to be added or moved. Now that we don't have to worry about preserving line numbers, we can rearrange those comments into a more conventional layout.

The demangler was only needed by coverage tests, but those tests were migrated into their own custom test mode in rust-lang#112300. This avoids having to build the demangler just for run-make tests. It will still be built as needed by run-coverage tests or for other purposes.

Avoid unnecessary builds/rebuilds of `rust-demangler` This is a combination of two loosely-related changes: - Don't build `rust-demangler` as a dependency of `tests/run-make`, because after rust-lang#112300 none of the remaining run-make tests actually use it. (If future run-make tests ever do need the demangler, it'll be easy to add it back.) - For `tests/run-coverage`, build the demangler with the stage 0 compiler instead of the current-stage compiler. This avoids having to uselessly rebuild the demangler after modifying and rebuilding the compiler itself.

When these extra directives were ported over as part of rust-lang#112300, it made sense to introduce `iter_header_extra` and pass them in as an extra argument. But now that rust-lang#120881 has added a `mode` parameter to `iter_header` for its own purposes, it's slightly simpler to move the coverage special-case code directly into `iter_header` as well. This lets us get rid of `iter_header_extra`.

Move the extra directives for `Mode::CoverageRun` into `iter_header` When these extra directives were ported over as part of rust-lang#112300, it made sense to introduce `iter_header_extra` and pass them in as an extra argument. But now that rust-lang#120881 has added a `mode` parameter to `iter_header` for its own purposes, it's slightly simpler to move the coverage special-case code directly into `iter_header` as well. This lets us get rid of `iter_header_extra`.

rustbot assigned wesleywiser Jun 5, 2023

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jun 5, 2023

This comment was marked as outdated.
Sign in to view

rustbot added the A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) label Jun 5, 2023

wesleywiser reviewed Jun 6, 2023
View reviewed changes

Comment threadsrc/bootstrap/test.rs Outdated
Comment threadsrc/tools/compiletest/src/header/needs.rs Outdated

Zalathar force-pushed the run-coverage branch from 0a730d8 to 551e74fCompare June 7, 2023 00:43

Zalathar force-pushed the run-coverage branch from 551e74f to b1910ebCompare June 7, 2023 03:20

Zalathar force-pushed the run-coverage branch from b1910eb to d96cc6eCompare June 12, 2023 02:59

Zalathar changed the title ~~Convert most run-make/coverage-reports tests to use a custom compiletest mode~~Convert run-make/coverage-reports tests to use a custom compiletest modeJun 12, 2023

This comment has been minimized.
Sign in to view

Zalathar force-pushed the run-coverage branch from d96cc6e to 610d9baCompare June 12, 2023 03:07

This comment was marked as outdated.
Sign in to view

Zalathar mentioned this pull request Jun 12, 2023
Dummy PR to test coverage test changes in CI #112514
Closed

Zalathar force-pushed the run-coverage branch from 610d9ba to 87107a3Compare June 12, 2023 08:07

jyn514 self-assigned this Jun 12, 2023

wesleywiser reviewed Jun 13, 2023
View reviewed changes

bors merged commit f00db43 into rust-lang:masterJun 30, 2023

rustbot added this to the 1.72.0 milestone Jun 30, 2023

Zalathar deleted the run-coverage branch June 30, 2023 06:05

Zalathar mentioned this pull request Jul 6, 2023
Re-enable some coverage tests on Linux #113389
Merged

Zalathar mentioned this pull request Oct 26, 2023
Avoid unnecessary builds/rebuilds of rust-demangler#117197
Merged

Zalathar mentioned this pull request Feb 17, 2024
Move the extra directives for Mode::CoverageRun into iter_header#121233
Merged

Zalathar mentioned this pull request May 30, 2024
compiletest: Unify cmd2procres with run_command_to_procres#125753
Merged

Uh oh!

Uh oh!

Conversation

Zalathar commented Jun 5, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Jun 5, 2023

Uh oh!

Zalathar commented Jun 5, 2023

Uh oh!

This comment was marked as outdated.

Zalathar commented Jun 5, 2023

Uh oh!

Uh oh!

Uh oh!

wesleywiser commented Jun 6, 2023

Uh oh!

Zalathar commented Jun 7, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Zalathar commented Jun 7, 2023

Uh oh!

Zalathar commented Jun 7, 2023

Uh oh!

Zalathar commented Jun 7, 2023

Uh oh!

Zalathar commented Jun 12, 2023

Uh oh!

This comment has been minimized.

This comment was marked as outdated.

Zalathar commented Jun 12, 2023

Uh oh!

Zalathar commented Jun 12, 2023

Uh oh!

wesleywiser left a comment

Choose a reason for hiding this comment

Uh oh!

wesleywiserJun 13, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Zalathar commented Jun 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Zalathar commented Jun 5, 2023•
edited
Loading

Zalathar commented Jun 7, 2023•
edited
Loading