Add `--pass $mode` to compiletest through `./x.py` by Centril · Pull Request #61755 · rust-lang/rust

Centril · 2019-06-11T23:44:22Z

Adds a flag --pass $mode to compiletest, which is exposed through ./x.py.

When --pass $mode is passed, {check,build,compile,run}-pass tests will be forced to run under the given $mode unless the directive // ignore-pass exists in the test file.

The modes are explained in #61778:

check has the same effect as cargo check
build or compile have the same effect as cargo build
run has the same effect as cargo run

On my machine, ./x.py -i test src/test/run-pass --stage 1 --pass check takes 38 seconds whereas it takes 2 min 7 seconds without --pass check.

cc #61712

r? @petrochenkov

petrochenkov · 2019-06-12T09:39:56Z

It's an open question what exactly "no codegen" option should be used for // check-pass and --force check.
In case of --emit metadata I'm not sure whether

metadata files hit the "file name is too long" cases (Skip codegen for one UI test with long file path #60648) when emitted.
UPDATE: Turns out // skip-codegen already used --emit metadata, so it doesn't have issues with long file paths.
we probably don't really need the metadata files emitted for the check mode (unlike cargo), but on the other hand it would be nice to specify check/build/run as doing exactly the same thing as cargo.
Need to measure how expensive the metadata emission is.

Centril · 2019-06-12T17:33:59Z

@petrochenkov Fixed the comments :)

petrochenkov · 2019-06-12T18:43:56Z

This should probably wait for #61778.
Otherwise I'll have to revert the compiletest part of this PR to rebase #61778.
(Or let's squash and land, then reverting will be easy.)

Centril · 2019-06-12T19:01:46Z

OK Let's wait for #61778 to land first.

Centril · 2019-06-12T20:57:19Z

Squashed the commits a bit.

compiletest: Introduce `// {check,build,run}-pass` pass modes Pass UI tests now have three modes ``` // check-pass // build-pass // run-pass ``` mirroring equivalent well-known `cargo` commands. `// check-pass` will compile the test skipping codegen (which is expensive and isn't supposed to fail in most cases). `// build-pass` will compile and link the test without running it. `// run-pass` will compile, link and run the test. Tests without a "pass" annotation are still considered "fail" tests. Most UI tests would probably want to switch to `check-pass`. Tests validating codegen would probably want to run the generated code as well and use `run-pass`. `build-pass` should probably be rare (linking tests?). rust-lang#61755 will provide a way to run the tests with any mode, e.g. bump `check-pass` tests to `run-pass` to satisfy especially suspicious people, and be able to make sure that codegen doesn't breaks in some entirely unexpected way. Tests marked with any mode are expected to pass with any other mode, if that's not the case for some legitimate reason, then the test should be made a "fail" test rather than a "pass" test. Perhaps some secondary CI can verify this invariant, but that's not super urgent. `// compile-pass` still works and is equivalent to `build-pass`. Why is `// compile-pass` bad - 1) it gives an impression that the test is only compiled, but not linked, 2) it doesn't mirror a cargo command. It can be removed some time in the future in a separate PR. cc rust-lang#61712

compiletest: Introduce `// {check,build,run}-pass` pass modes Pass UI tests now have three modes ``` // check-pass // build-pass // run-pass ``` mirroring equivalent well-known `cargo` commands. `// check-pass` will compile the test skipping codegen (which is expensive and isn't supposed to fail in most cases). `// build-pass` will compile and link the test without running it. `// run-pass` will compile, link and run the test. Tests without a "pass" annotation are still considered "fail" tests. Most UI tests would probably want to switch to `check-pass`. Tests validating codegen would probably want to run the generated code as well and use `run-pass`. `build-pass` should probably be rare (linking tests?). #61755 will provide a way to run the tests with any mode, e.g. bump `check-pass` tests to `run-pass` to satisfy especially suspicious people, and be able to make sure that codegen doesn't breaks in some entirely unexpected way. Tests marked with any mode are expected to pass with any other mode, if that's not the case for some legitimate reason, then the test should be made a "fail" test rather than a "pass" test. Perhaps some secondary CI can verify this invariant, but that's not super urgent. `// compile-pass` still works and is equivalent to `build-pass`. Why is `// compile-pass` bad - 1) it gives an impression that the test is only compiled, but not linked, 2) it doesn't mirror a cargo command. It can be removed some time in the future in a separate PR. cc #61712

@petrochenkov

…petrochenkov Add `--pass $mode` to compiletest through `./x.py` Adds a flag `--pass $mode` to compiletest, which is exposed through `./x.py`. When `--pass $mode` is passed, `{check,build,compile,run}-pass` tests will be forced to run under the given `$mode` unless the directive `// ignore-pass` exists in the test file. The modes are explained in rust-lang#61778: - `check` has the same effect as `cargo check` - `build` or `compile` have the same effect as `cargo build` - `run` has the same effect as `cargo run` On my machine, `./x.py -i test src/test/run-pass --stage 1 --pass check` takes 38 seconds whereas it takes 2 min 7 seconds without `--pass check`. cc rust-lang#61712 r? @petrochenkov

RalfJung · 2019-06-29T09:26:35Z

This does not, however, give me the option to "filter" by mode, right? Say I wanted to only run non-pass tests, or only run *-pass tests?

Centril · 2019-06-29T09:28:50Z

@RalfJung That would be #61719.

petrochenkov · 2019-06-29T10:03:27Z

@RalfJung
Filtering was my first idea too, but after some thinking about testing scenarios (why I run tests and what I want from it) --pass check appeared to be a better approach.
With pass modes available I personally expect to never use filtering.

Centril · 2019-06-29T10:07:43Z

I do agree --pass check is what you want for regular development; but from a language team perspective I find the ability to do e.g. --filter-mode pass --test-args async to be useful if I want to extract all the relevant pass tests for async await (that assumes good organization for the async/await test, which is probably a great assumption but it's serviceable as a blunt instrument). That's not something I'd do every day, but it's still useful on occasion.

@ghost

Rollup of 7 pull requests Successful merges: - #61199 (Revert "Set test flag when rustdoc is running with --test option" ) - #61755 (Add `--pass $mode` to compiletest through `./x.py`) - #61818 (Issue #60709 test) - #62023 (publish_toolstate: don't use 'new' from inside the loop) - #62104 (Inform the query system about properties of queries at compile time) - #62163 (Avoid mem::uninitialized() in std::sys::unix) - #62204 (doc(libcore) Fix CS) Failed merges: r? @ghost

tests: Require run-pass tests without annotations to run successfully again Fixes#62775 (regression from #61755)

@Centril

Move run-pass tests to ui This is the second attempt at doing #53994 (which was previously reverted in #54530). The issue with inability to run the test suite in a faster way (#54047) that motivated the revert was recently addressed by #61755. r? @Centril

rust-highfive assigned petrochenkov Jun 11, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 11, 2019

Mark-Simulacrum reviewed Jun 12, 2019
View reviewed changes

Comment threadsrc/bootstrap/flags.rs Outdated
Comment threadsrc/bootstrap/flags.rs Outdated
Comment threadsrc/test/ui/print_type_sizes/packed.rs Outdated

petrochenkov mentioned this pull request Jun 12, 2019
Add --filter-mode=run-pass,compile-pass to compiletest #61719
Closed

petrochenkov reviewed Jun 12, 2019
View reviewed changes

Comment threadsrc/test/ui/consts/const-eval/promoted_errors.rs Outdated

petrochenkov reviewed Jun 12, 2019
View reviewed changes

Comment threadsrc/test/ui/lint/lint-type-overflow2.rs Outdated

petrochenkov reviewed Jun 12, 2019
View reviewed changes

Comment threadsrc/test/ui/rmeta-lib-pass.rs Outdated

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 12, 2019

petrochenkov mentioned this pull request Jun 12, 2019
compiletest: Introduce // {check,build,run}-pass pass modes #61778
Merged

Centril added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 12, 2019

Centril changed the title ~~Add --force check to compiletest through ./x.py~~Add --pass check to compiletest through ./x.pyJun 12, 2019

This comment has been minimized.
Sign in to view

Centril added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 12, 2019

Centril force-pushed the compiletest-force-check branch from 22c4a9e to 57b6a20Compare June 12, 2019 20:56

This comment has been minimized.
Sign in to view

Centril added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels Jun 24, 2019

Centril added 3 commits June 24, 2019 07:58

bootstrap: pass '--pass' on to compiletest.
a56a6d7

compiletest: support '--pass check' and '// ignore-pass'.
54337fa

add '// ignore-pass' where applicable.
e364eb4

petrochenkov mentioned this pull request Jun 29, 2019
Don't store locals that have been moved from in generators #61922
Merged

Centril mentioned this pull request Jun 29, 2019
Rollup of 7 pull requests #62226
Merged

bors merged commit 93077f3 into rust-lang:masterJun 29, 2019

Centril deleted the compiletest-force-check branch June 29, 2019 15:32

petrochenkov mentioned this pull request Jun 29, 2019
More systematic compile/run markup for "pass" UI tests #61712
Closed

Centril mentioned this pull request Jul 4, 2019
Document --pass $moderust-lang/rustc-dev-guide#381
Closed

petrochenkov mentioned this pull request Jul 19, 2019
tests: Require run-pass tests without annotations to run successfully again #62797
Merged

bors added a commit that referenced this pull request Jul 21, 2019
Auto merge of#62797- petrochenkov:run-pass-hole, r=Mark-Simulacrum
529828e
tests: Require run-pass tests without annotations to run successfully again Fixes#62775 (regression from #61755)

bors added a commit that referenced this pull request Jul 21, 2019
Auto merge of#62797- petrochenkov:run-pass-hole, r=Mark-Simulacrum
787acc6
tests: Require run-pass tests without annotations to run successfully again Fixes#62775 (regression from #61755)

bors added a commit that referenced this pull request Jul 21, 2019
Auto merge of#62797- petrochenkov:run-pass-hole, r=Mark-Simulacrum
25035ca
tests: Require run-pass tests without annotations to run successfully again Fixes#62775 (regression from #61755)

bors added a commit that referenced this pull request Jul 22, 2019
Auto merge of#62797- petrochenkov:run-pass-hole, r=Mark-Simulacrum
4bc1ce7
tests: Require run-pass tests without annotations to run successfully again Fixes#62775 (regression from #61755)

petrochenkov mentioned this pull request Jul 27, 2019
Move run-pass tests to ui #63029
Merged

Zalathar mentioned this pull request Oct 21, 2024
compiletest: what does --pass (pass mode) actually intend to do? #131990
Closed

Uh oh!

Uh oh!

Conversation

Centril commented Jun 11, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

petrochenkov commented Jun 12, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Centril commented Jun 12, 2019

Uh oh!

This comment has been minimized.

petrochenkov commented Jun 12, 2019

Uh oh!

Centril commented Jun 12, 2019

Uh oh!

Centril commented Jun 12, 2019

Uh oh!

This comment has been minimized.

RalfJung commented Jun 29, 2019

Uh oh!

Centril commented Jun 29, 2019

Uh oh!

petrochenkov commented Jun 29, 2019

Uh oh!

Centril commented Jun 29, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Centril commented Jun 11, 2019•
edited
Loading

petrochenkov commented Jun 12, 2019•
edited
Loading

Centril commented Jun 29, 2019•
edited
Loading