Skip stage 0 libstd

[Zoxc]

r? @alexcrichton

[Zoxc]

This removes the compiler dependency on the libstd in the rustc repo and instead uses the libstd of the bootstrap compiler. This cuts about a minute of build time and changes to libstd no longer require rebuilding the compiler. We can also get rid of the usage of cfg(stage0). The downside is that the compiler can only libstd features from the bootstrap compiler, but this seems quite minor as you can see from the 2 line changes required to make the compiler work.

[alexcrichton]

Thanks for the PR!

I think there's both some pros and cons to a change like this. When working on the standard library it's quite nice that a stage0 build includes libstd, and when adding an unstable method to libstd which is destined for usage in librustc it's also quite convenient that you can add it and then use it immediately in librustc. Finally the amount of time taken to build one stage of libstd is quite small across the aggregate of all our CI builds, I think it accounts for maybe a total of an hour for the 7.5 days of cpu time we burn on each CI run (aka ~1 minute of a 3 hour build).

On the other hand, though, this is clearly a reduction in CI time (small though it may be) and it also means we'd probably never need to write #[cfg(stage0)] in libstd ever again. In any case I think this is a large enough procedural change that it should get eyes from more than just me. @Zoxc when this PR is ready I think we can cc a few temas to get their thoughts on it as well.

[Zoxc]

When working on the standard library it's quite nice that a stage0 build includes libstd, and when adding an unstable method to libstd which is destined for usage in librustc

We should not be adding thing to libstd which are only used by librustc. I consider it a pro that this PR would make that harder.

We don't need to wait for the PR to be ready to discuss things, so I'm just going to cc some teams already ;)
cc @rust-lang/compiler
cc @rust-lang/libs

[withoutboats]

Since I never work on the compiler but sometimes work on std, I'd be eager for changes that result in separating the build process for these two components.

[eddyb]

I think this is fine in the current climate of changes and releases, it's not as annoying anymore.

[rust-highfive]

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Downloading https://files.pythonhosted.org/packages/c9/6c/72524c483cd871e1d55fb3c3fa029605d17182c83de6e27ca143aed635a8/awscli-1.15.16-py2.py3-none-any.whl (1.3MB)
0% |▎ | 10kB 10.3MB/s eta 0:00:01
1% |▌ | 20kB 1.8MB/s eta 0:00:01
2% |▉ | 30kB 2.1MB/s eta 0:00:01
3% |█ | 40kB 2.0MB/s eta 0:00:01
---
[00:45:19] test builder::__test::dist_with_targets_and_hosts ... ok
[00:45:19] [00:45:19] failures:
[00:45:19] [00:45:19] ---- builder::__test::build_default stdout ----
[00:45:19] Checking std artifacts (A -> A)
[00:45:19] Checking test artifacts (A -> A)
[00:45:19] Checking std artifacts (A -> B)
[00:45:19] Checking test artifacts (A -> B)
[00:45:19] Checking std artifacts (A -> C)
[00:45:19] Checking test artifacts (A -> C)
[00:45:19] thread 'builder::__test::build_default' panicked at 'assertion failed: `(left == right)`
[00:45:19] [00:45:19] Diff < left / right > :
[00:45:19] Test {
[00:45:19] target: "A",
[00:45:19] compiler: Compiler {
[00:45:19] > stage: 0,
[00:45:19] > stage: 0,
[00:45:19] > host: "A"
[00:45:19] > }
[00:45:19] > },
[00:45:19] > Test {
[00:45:19] > target: "A",
[00:45:19] > compiler: Compiler {
[00:45:19] host: "A"
[00:45:19] }
[00:45:19] },
[00:45:19] Test {
---
[00:45:19] },
[00:45:19] Test {
[00:45:19] target: "B",
[00:45:19] compiler: Compiler {
[00:45:19] > stage: 0,
[00:45:19] > host: "A"
[00:45:19] > }
[00:45:19] > },
[00:45:19] > Test {
[00:45:19] > target: "B",
[00:45:19] > compiler: Compiler {
[00:45:19] host: "A"
[00:45:19] }
[00:45:19] },
[00:45:19] Test {
---
[00:45:19] [00:45:19] ', bootstrap/builder.rs:1284:9
[00:45:19] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:45:19] [00:45:19] ---- builder::__test::build_with_target_flag stdout ----
[00:45:19] Checking std artifacts (A -> A)
[00:45:19] Checking test artifacts (A -> A)
[00:45:19] Checking std artifacts (A -> B)
[00:45:19] Checking test artifacts (A -> B)
[00:45:19] Checking std artifacts (A -> C)
[00:45:19] Checking test artifacts (A -> C)
[00:45:19] thread 'builder::__test::build_with_target_flag' panicked at 'assertion failed: `(left == right)`
[00:45:19] [00:45:19] Diff < left / right > :
[00:45:19] Test {
[00:45:19] target: "A",
[00:45:19] compiler: Compiler {
[00:45:19] > stage: 0,
[00:45:19] > stage: 0,
[00:45:19] > host: "A"
[00:45:19] > }
[00:45:19] > },
[00:45:19] > Test {
[00:45:19] > target: "A",
[00:45:19] > compiler: Compiler {
[00:45:19] host: "A"
[00:45:19] }
[00:45:19] },
[00:45:19] Test {
---
[00:45:19] },
[00:45:19] Test {
[00:45:19] target: "B",
[00:45:19] compiler: Compiler {
[00:45:19] > stage: 0,
[00:45:19] > host: "A"
[00:45:19] > }
[00:45:19] > },
[00:45:19] > Test {
[00:45:19] > target: "B",
[00:45:19] > compiler: Compiler {
[00:45:19] host: "A"
[00:45:19] }
df007:start=1525805087257174004,finish=1525807807020821314,duration=2719763647310

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[bors]

☔ The latest upstream changes (presumably #50807) made this pull request unmergeable. Please resolve the merge conflicts.

[rust-highfive]

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:06:31] Compiling flate2 v1.0.1
[00:06:34] Compiling backtrace v0.3.6
[00:06:38] Compiling rustc-rayon v0.1.0
[00:06:46] Compiling rustc_data_structures v0.0.0 (file:///checkout/src/librustc_data_structures)
[00:06:47] error[E0658]: use of unstable library feature 'nonzero' (see issue #49137)
[00:06:47] --> librustc_data_structures/obligation_forest/node_index.rs:26:21
[00:06:47] |
[00:06:47] 26 | (self.index.get() - 1) as usize
[00:06:47] |
[00:06:47] |
[00:06:47] = help: add #![feature(nonzero)] to the crate attributes to enable
[00:06:47] error: aborting due to 4 previous errors
[00:06:47] [00:06:47] For more information about this error, try `rustc --explain E0658`.
[00:06:47] error: Could not compile `rustc_data_structures`.
[00:06:47] error: Could not compile `rustc_data_structures`.
[00:06:47] [00:06:47] Caused by:
[00:06:47] process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name rustc_data_structures librustc_data_structures/lib.rs --color always --error-format json --crate-type dylib --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=949c6ac74d89c4f0 -C extra-filename=-949c6ac74d89c4f0 --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps --extern parking_lot=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libparking_lot-f15de87dbbd7d9a3.rlib --extern serialize=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libserialize-a476bbed33563995.so --extern serialize=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libserialize-a476bbed33563995.rlib --extern rustc_rayon=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_rayon-1c77598eb0f8659a.rlib --extern parking_lot_core=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libparking_lot_core-f49dd5dd4b2cebc0.rlib --extern cfg_if=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libcfg_if-cc766300bf84fd17.rlib --extern stable_deref_trait=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libstable_deref_trait-f3aba095da831e72.rlib --extern ena=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libena-0217b26d5f94d5ee.rlib --extern rustc_cratesio_shim=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_cratesio_shim-e46c6bd33b6a136f.so --extern log=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/liblog-bde077c9625bcd15.rlib` (exit code: 101)
[00:06:47] command did not execute successfully: "/checkout/obj/build/base/x86_64-unknown-l2091292 .
1200960 ./obj/build
726436 ./src
549332 ./obj/build/base
549328 ./obj/build/base/x86_64-unknown-linux-gnu
---
150740 ./.git/modules/src
149112 ./src/llvm-emscripten/test
137216 ./obj/build/bootstrap/debug/incremental
122652 ./obj/build/bootstrap/debug/incremental/bootstrap-1r3bppl29tbrj
122648 ./obj/build/bootstrap/debug/incremental/bootstrap-1r3bppl29tbrj/s-f14qgk03o8-16db1qo-2lvcwg9v0hou0
103608 ./obj/build/base/x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends
77588 ./obj/build/x86_64-unknown-linux-gnu/stage0-rustc
71856 ./.git/modules/src/tools
70944 ./obj/build/x86_64-unknown-linux-gnu/native
70300 ./obj/build/x86_64-unknown-linux-gnu/native/jemalloc

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[Gankra]

Another datapoint: this would make const-ifying libstd way easier since it's generally blocked on teaching the compiler how to constify more stuff, leading to tons of stage0 cfgs.

[rust-highfive]

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:41:01] test builder::__test::dist_with_targets_and_hosts ... ok
[00:41:01] [00:41:01] failures:
[00:41:01] [00:41:01] ---- builder::__test::build_default stdout ----
[00:41:01] Checking std artifacts (A -> A)
[00:41:01] Checking test artifacts (A -> A)
[00:41:01] Checking std artifacts (A -> B)
[00:41:01] Checking test artifacts (A -> B)
[00:41:01] Checking std artifacts (A -> C)
[00:41:01] Checking test artifacts (A -> C)
[00:41:01] thread 'builder::__test::build_default' panicked at 'assertion failed: `(left == right)`
[00:41:01] [00:41:01] Diff < left / right > :
[00:41:01] Test {
[00:41:01] target: "A",
[00:41:01] compiler: Compiler {
[00:41:01] > stage: 0,
[00:41:01] > stage: 0,
[00:41:01] > host: "A"
[00:41:01] > }
[00:41:01] > },
[00:41:01] > Test {
[00:41:01] > target: "A",
[00:41:01] > compiler: Compiler {
[00:41:01] host: "A"
[00:41:01] }
[00:41:01] },
[00:41:01] Test {
---
[00:41:01] },
[00:41:01] Test {
[00:41:01] target: "B",
[00:41:01] compiler: Compiler {
[00:41:01] > stage: 0,
[00:41:01] > host: "A"
[00:41:01] > }
[00:41:01] > },
[00:41:01] > Test {
[00:41:01] > target: "B",
[00:41:01] > compiler: Compiler {
[00:41:01] host: "A"
[00:41:01] }
[00:41:01] },
[00:41:01] Test {
---
[00:41:01] [00:41:01] ', bootstrap/builder.rs:1324:9
[00:41:01] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:41:01] [00:41:01] ---- builder::__test::build_with_target_flag stdout ----
[00:41:01] Checking std artifacts (A -> A)
[00:41:01] Checking test artifacts (A -> A)
[00:41:01] Checking std artifacts (A -> B)
[00:41:01] Checking test artifacts (A -> B)
[00:41:01] Checking std artifacts (A -> C)
[00:41:01] Checking test artifacts (A -> C)
[00:41:01] thread 'builder::__test::build_with_target_flag' panicked at 'assertion failed: `(left == right)`
[00:41:01] [00:41:01] Diff < left / right > :
[00:41:01] Test {
[00:41:01] target: "A",
[00:41:01] compiler: Compiler {
[00:41:01] > stage: 0,
[00:41:01] > stage: 0,
[00:41:01] > host: "A"
[00:41:01] > }
[00:41:01] > },
[00:41:01] > Test {
[00:41:01] > target: "A",
[00:41:01] > compiler: Compiler {
[00:41:01] host: "A"
[00:41:01] }
[00:41:01] },
[00:41:01] Test {
---
[00:41:01] },
[00:41:01] Test {
[00:41:01] target: "B",
[00:41:01] compiler: Compiler {
[00:41:01] > stage: 0,
[00:41:01] > host: "A"
[00:41:01] > }
[00:41:01] > },
[00:41:01] > Test {
[00:41:01] > target: "B",
[00:41:01] > compiler: Compiler {
[00:41:01] host: "A"
[00:41:01] }
[00:41:01] },
[00:41:01] Test {
---
[00:41:01] [00:41:01] ---- builder::__test::dist_with_same_targets_and_hosts stdout ----
[00:41:01] thread 'builder::__test::dist_with_same_targets_and_hosts' panicked at 'assertion failed: `(left == right)`
[00:41:01] [00:41:01] Diff < left / right > :
[00:41:01] Test {
[00:41:01] target: "A",
[00:41:01] compiler: Compiler {
[00:41:01] > stage: 0,
[00:41:01] > stage: 0,
[00:41:01] > host: "A"
[00:41:01] > }
[00:41:01] > },
[00:41:01] > Test {
[00:41:01] > target: "A",
[00:41:01] > compiler: Compiler {
[00:41:01] host: "A"
[00:41:01] }
[00:41:01] },
[00:41:01] Test {
---
[00:41:01] [00:41:01] [00:41:01] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[00:41:01] Build completed unsuccessfully in 0:00:46
[00:41:01] make: *** [check] Error 1
[00:41:01] Makefile:58: recipe for target 'check' failed
56028 ./obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release
51360 ./obj/build/base/x86_64-unknown-linux-gnu/bin
50556 ./obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps
48444 ./obj/build/x86_64-unknown-linux-gnu/stage0-std

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[Zoxc]

@Mark-Simulacrum This breaks your tests. Test is on longer cross-compiled using the stage0 compiler. I didn't expect to see that as I assumed we only cross-compiled using the stage1 compiler. Std also stills shows as built by stage0 in the tests, even though that doesn't happen (I have an assertion in the run method to make sure). Do you how that can happen?

[Mark-Simulacrum]

I have an assertion in the run method to make sure

I'm not seeing this in the diff, but perhaps I'm missing it?

I believe the tests are failing for the case where if ./x.py build --build A --host B --target C is passed, we currently cross-compile in stage 0 from A to B, but not A to C/B to C. This might not be necessary/could be wrong, I'm not sure, but with the changes introduced we should (rightly, I think?) no longer be doing that.

I think the tests can just be updated to fix the failures. It looks like doing that would be the correct diff given this patch.

[rust-highfive]

The job mingw-check of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:07:16] For more information about this error, try `rustc --explain E0463`.
[00:07:16] error: Could not compile `rustdoc`.
[00:07:16] [00:07:16] Caused by:
[00:07:16] process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name rustdoc librustdoc/lib.rs --color always --error-format json --crate-type lib --emit=dep-info,metadata -C opt-level=3 -C metadata=aa2214b76f9c28a8 -C extra-filename=-aa2214b76f9c28a8 --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/release/deps --extern minifier=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libminifier-656476578ea2b275.rmeta --extern tempdir=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libtempdir-3e386995dc337d92.rmeta --extern pulldown_cmark=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libpulldown_cmark-12d2a48b41e80411.rmeta` (exit code: 101)
[00:07:16] command did not execute successfully: "/checkout/obj/build/base/x86_64-unknown-linux-gnu/bin/cargo" "check" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--color" "always" "--manifest-path" "/checkout/src/tools/rustdoc/Cargo.toml" "--message-format" "json"
[00:07:16] expected success, got: exit code: 101
[00:07:16] thread 'main' panicked at 'cargo must succeed', bootstrap/compile.rs:1135:9
[00:07:16] travis_fold:end:stage0-rustdoc
[00:07:16] travis_time:end:stage0-rustdoc:start=1526876045406422159,finish=1526876054762813601,duration=9356391442

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[bjorn3]

@TimNN the last @rust-highfive comment doesn't contain what the error was.

[emilyalbini]

Ping from triage @Zoxc! It's been a while since we heard from you, will you have time to work on this again?

[bors]

☔ The latest upstream changes (presumably #51138) made this pull request unmergeable. Please resolve the merge conflicts.

[TimNN]

@Zoxc, we haven't heard from you in two weeks, so we are closing this PR for now to keep the queue clean. Feel free to re-open in the future.