add preliminary support for incremental compilation to rustbuild.py

[nikomatsakis]

This implements the integration described in #37929. It requires the use of a local nightly as your bootstrap compiler. The setup is described in src/bootstrap/README.md.

This does NOT implement the "copy stage0 libs to stage1" optimization described in #37929, just because that seems orthogonal to me.

In local testing, I do not yet see any incremental re-use when building rustc. I'm not sure why that is, more investigation needed.

(For these reasons, this is not marked as fixing the relevant issue.)

r? @alexcrichton -- I included one random cleanup (Step::noop()) that turned out to not be especially relevant. Feel free to tell me you liked it better the old way.

[nikomatsakis]

Probably I should remove the code that dumps the rustc command line -- well, more specifically, I should make it a separate option, distinct from --verbose. This is because unless you use -j1 you get some pretty gibberish output.

[michaelwoerister]

This seems like a copy-and-paste artifact.

[nikomatsakis]

So, I take it back, I actually am getting reuse locally. I'm experimenting more now.

I also have some local fixes to make before this lands:

• I am moving the incremental dir to out/<host>/stageN-incremental
  • this also means that x.py clean deletes it, as (I imagine) would be expected
• I permit --stage 0 or --stage 1 when using --incremental

[nikomatsakis]

OK, this is... good to go I guess, though maybe it would be better to avoid printing the full verbose output unless the verbosity is at "level 2" or something like that. Anyway, using it locally, and running with --verbose, I do observe that we get a decent amount of reuse. Not as much as I would like: just touching some random file in libstd, for example, gets 100% reuse in libstd, but we gradually get less and less for other crates, due to us thinking that metadata has changed.

[nikomatsakis]

Fixed the verbosity level concern in the latest commit by adding --very-verbose

[alexcrichton]

Technically rustbuild already places all of it's output in a build directory, so this'll create build/build. Maybe we could leave the separate directory bits to a different section of the readme?

[nikomatsakis]

Yeah, I'll remove it. I just still find that it's better to have a build directory, since it winds up with quite a lot of files in it:

> ls
build config.mk config.stamp Makefile tmp

[alexcrichton]

I think that --local-rust-root and --enable-local-rust can both be elided here. At least rustbuild only takes a look at --enable-local-rust, so the other two configuration options are ignored by rustbuild.

[nikomatsakis]

--local-rust-root is certainly used, since that's how we pick which binary -- but I think if you don't supply it, we will search your path. I can add some comments about that.

[alexcrichton]

Normally this is interpreted as -vv, but I'm not sure if getopts has the ability to tell us about that?

[nikomatsakis]

Yeah, I think it doesn't? I should dig a bit more I guess.

[nikomatsakis]

oh I guess there is optflagmulti

[alexcrichton]

👍

much nicer

[alexcrichton]

Hm I didn't quite see all the pieces I expected to see here. So right now if you pass --incremental it'll incremental build the entire compiler. But I think a ./x.py build will continue to use that compiler to build a whole new compiler, right? That second compilation isn't incremental from what it looks like.

[nikomatsakis]

@alexcrichton

But I think a ./x.py build will continue to use that compiler to build a whole new compiler, right? That second compilation isn't incremental from what it looks like.

Right. This is what I meant by this from the PR summary: 'This does NOT implement the "copy stage0 libs to stage1" optimization described in #37929, just because that seems orthogonal to me.'

Note that --incremental also implies --stage 1, so you're not going to rebuild all of rustc.

[nikomatsakis]

@alexcrichton pushed one bug fix and tweaked README. I didn't address the problem of copying libraries over yet.

[nikomatsakis]

So @alexcrichton and I had some chats over IRC and we came to some conclusions:

1. If we are not copying stage0 libs into the stage1 directory, but rather recompiling, then there is no need to use a local rust. I mean, you should just because the beta support for incremental compilation is weak. But it's not required.
2. The only reason to rebuild all the libs for stage1 is because the metadata format may have changed (well, that and testing etc). So we should perhaps say "if you use --incremental and --enable-local-rebuild (which is only set if your local rust has same version number), then we copy the libs by default".

This would mean that building with --incremental and a recent nightly as your base compiler is by default fast:

• It would only build to stage1 (because --incremental)
• It would not rebuild the libs 2x (because --incremental and --enable-local-rebuild)
• It would build incrementally up to stage1 (because --incremental)

@alexcrichton do you want to hold off on landing until I do that? doesn't seem so hard.

[nikomatsakis]

On the other hand, I could just remove the check that forces --incremental to use a local Rust and land this now, and then add the lib copying as a separate thing. That would be my preference.

[bors]

☔ The latest upstream changes (presumably #37800) made this pull request unmergeable. Please resolve the merge conflicts.

[alexcrichton]

@nikomatsakis I'd be fine landing this ahead of time, seems like a fine optimization no matter what to unconditionally incrementally build stage0 with --incremental (and it's always possible). I'd also be fine adding an orthogonal feature to copy libs instead of build them.

So in that sense, likely r=me with a rebase, but I'll take one more look.

[nikomatsakis]

@alexcrichton rebased and adjusted

[alexcrichton]

Aha! I found out where we actually read this. Ideally we wouldn't need --local-rust-root here because --enable-local-rebuild should say "get from PATH by default".

Let's fix that bug later.

[nikomatsakis]

@alexcrichton actually, I think the configure script does fetch from path by default -- I just used --local-rust-root because I have a different rustc in my path that takes precedence over rustup

[nikomatsakis]

oh, but I see that bootstrap doesn't. ok.

[alexcrichton]

r=me with a squash

[nikomatsakis]

@bors r=alexcrichton

[bors]

📌 Commit 3bfdd48 has been approved by alexcrichton

[bors]

⌛ Testing commit 3bfdd48 with merge 5daf4b6...

[bors]

💔 Test failed - auto-linux-64-cargotest

[bors]

☔ The latest upstream changes (presumably #37817) made this pull request unmergeable. Please resolve the merge conflicts.

[nikomatsakis]

@bors r=acrichto

[bors]

📌 Commit 83453bc has been approved by acrichto

[bors]

⌛ Testing commit 83453bc with merge 94ae2a2...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-cargotest, auto-linux-64-cross-freebsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-rustbuild
Approved by: acrichto
Pushing 94ae2a2 to master...