Add x.py check --stage 1

[jyn514]

This is useful for several cases:

• Checking changes to the standard library with cfg(not(bootstrap)) (Warn when checking library/std when profile = "compiler" #78466)
• Checking platform-specific changes to the compiler. In particular,
  checking with stage 1 doesn't require a cross-compiling C native
  toolchain, unlike building. (Add x.py check --stage 1 #46955)
• Testing newly added internal lints without having to rebuild
  rustc_middle twice (this is what finally made me decide to work on it: Don't make tools responsible for checking unknown and renamed lints #80524)

It is also very easy to support.

Closes#46955.

[rust-highfive]

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

[jyn514]

cc @Diggsey - is this what you expected for #46955 ?

[Diggsey]

Heh, that's an old issue 😄 - yeah it looks like this solves it.

[jyn514]

Yeah, I was triaging bootstrap issues a while ago and it popped up as something I'd want myself too 😆

[jyn514]

I don't know why I thought this would be easy 🤦

error: internal compiler error: compiler/rustc_mir/src/monomorphize/collector.rs:826:9: cannot create local mono-item for DefId(1:1638 ~ std[338b]::env::var)
query stack during panic:
#0 [collect_and_partition_mono_items] collect_and_partition_mono_items
end of query stack

[rust-log-analyzer]

The job mingw-check failed! Check out the build log: (web)(plain)

Click to see the possible cause of the failure (guessed by this bot)

configure: rust.channel := nightly
configure: rust.debug-assertions := True
configure: llvm.assertions := True
configure: dist.missing-tools := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: configure: run `python /checkout/x.py --help`
configure: ---
skip untracked path cpu-usage.csv during rustfmt invocations
skip untracked path src/doc/book/ during rustfmt invocations
skip untracked path src/doc/rust-by-example/ during rustfmt invocations
skip untracked path src/llvm-project/ during rustfmt invocations
Diff in /checkout/src/bootstrap/check.rs at line 84:
Running `"/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustfmt" "--config-path" "/checkout" "--edition" "2018" "--unstable-features" "--skip-children" "--check" "/checkout/src/bootstrap/check.rs"` failed.
If you're running `tidy`, try again with `--bless`. Or, if you just want to format code, run `./x.py fmt` instead.
);
std_cargo(builder, target, compiler.stage, &mut cargo);
- builder.info(&format!("Checking stage{} std artifacts ({} -> {})", builder.top_stage, &compiler.host, target));
+ builder.info(&format!(
+ "Checking stage{} std artifacts ({} -> {})",
+ builder.top_stage, &compiler.host, target
+ ));
run_cargo(
cargo,
Diff in /checkout/src/bootstrap/check.rs at line 187:
Diff in /checkout/src/bootstrap/check.rs at line 187:
cargo.arg("-p").arg(krate.name);
- builder.info(&format!("Checking stage{} compiler artifacts ({} -> {})", builder.top_stage, &compiler.host, target));
+ builder.info(&format!(
+ "Checking stage{} compiler artifacts ({} -> {})",
+ builder.top_stage, &compiler.host, target
+ ));
run_cargo(
cargo,
failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --stage 2 src/tools/tidy
Build completed unsuccessfully in 0:00:18

[jyn514]

Reporting on my progress so far, because I won't get it done tonight and I will definitely forget by tomorrow:

• The ICE comes from

  rust/compiler/rustc_mir/src/monomorphize/collector.rs

  Line 825 in 058a710

  if !tcx.is_mir_available(def_id){

  . It happens because there's no MIR built for the standard library, only check metadata (because x.py check didn't build it). In particular, this only happens for build scripts and proc-macros, because they need to actually run.
• The way stage 0 solves this is with a 'snapshot sysroot':

  rust/src/bootstrap/builder.rs

  Lines 1173 to 1194 in 150d1fe

  	// Almost all of the crates that we compile as part of the bootstrap may
  	// have a build script, including the standard library. To compile a
  	// build script, however, it itself needs a standard library! This
  	// introduces a bit of a pickle when we're compiling the standard
  	// library itself.
  	//
  	// To work around this we actually end up using the snapshot compiler
  	// (stage0) for compiling build scripts of the standard library itself.
  	// The stage0 compiler is guaranteed to have a libstd available for use.
  	//
  	// For other crates, however, we know that we've already got a standard
  	// library up and running, so we can use the normal compiler to compile
  	// build scripts in that situation.
  	if mode == Mode::Std{
  	cargo
  	.env("RUSTC_SNAPSHOT",&self.initial_rustc)
  	.env("RUSTC_SNAPSHOT_LIBDIR",self.rustc_snapshot_libdir());
  	}else{
  	cargo
  	.env("RUSTC_SNAPSHOT",self.rustc(compiler))
  	.env("RUSTC_SNAPSHOT_LIBDIR",self.rustc_libdir(compiler));
  	}

  . This works because the metadata generated by x.py check, stage0-std, is separate from the beta libstd, in stage0. This doesn't work for stage 1 because the beta libstd was built by beta, and the build scripts/proc macros are being compiled by stage1. It's not possible to compile the proc macros with stage 0 because the ABI is different and they need to link to the standard library with the stage1 compiler.
• A possible fix I thought of is to unconditionally build a standard library for the host with stage1. It will only be used for building build scripts/proc macros and nothing else; all other crates should use the library of the target instead.

I tried implementing that just now and it went badly. I'm planning to wait to hear from @Mark-Simulacrum about whether this is a good idea first and whether there's a good way to do this without rewriting half of bootstrap.

[Mark-Simulacrum]

• Checking changes to the standard library with cfg(not(bootstrap)) (Warn when checking library/std when profile = "compiler" #78466)

This sort of makes sense, but I think I'd prefer folks use --keep-stage-std 0 for this case. They're going to need to build a compiler anyway, and the difference between checking and building std is relatively minimal.

• Checking platform-specific changes to the compiler. In particular,
  checking with stage 1 doesn't require a cross-compiling C native
  toolchain, unlike building. (Add x.py check --stage 1 #46955)

I admit I don't fully follow this use case. I think there's very little platform-specific code in the cargo check phase of the compiler (indeed, other than differences between 32/64 bit, I can't obviously think of some). Maybe would be good to elaborate on this a bit.

• Testing newly added internal lints without having to rebuild
  rustc_middle twice (this is what finally made me decide to work on it: Don't make tools responsible for checking unknown and renamed lints #80524)

This, on the other hand, is where the value is much more likely to be. I've filed #81064 to replace this PR, and set you as the reviewer. It looks like there's some configuration etc changes here which would be good to propagate, so feel free to push to my branch (or cherry-pick my commit, whatever).

[jyn514]

I admit I don't fully follow this use case. I think there's very little platform-specific code in the cargo check phase of the compiler (indeed, other than differences between 32/64 bit, I can't obviously think of some). Maybe would be good to elaborate on this a bit.

I think the point is not that the compiler is platform specific but that the code being compiled is platform specific. @Diggsey probably knows more than I do, but it seems useful to check libstd for various different targets without having to have a cross-compiling C toolchain installed.

[jyn514]

Anyway, closing this in favor of #81064, which I think is a better approach.

[Mark-Simulacrum]

Well, the intersection of cfg not bootstrap and target specific code is probably really low, so probably regular check should work ok for that.