If local-rust is the same as the current version, then force a local-rebuild

[infinity0]

In Debian, we would like the option to build/rebuild the current release from
either the current or previous stable release. So we use enable-local-rust
instead of enable-local-rebuild, and read the bootstrap key dynamically from
whatever is installed locally.

In general, it does not make much sense to allow enable-local-rust without also
setting the bootstrap key, since the build would fail otherwise.

(The way I detect "the bootstrap key of [the local] rustc installation" is a bit hacky, suggestions welcome.)

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @brson (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[TimNN]

I use export RUSTC_BOOTSTRAP_KEY=rustc --version | cut -f2 -d' ' | tr -d "\n" | md5sum | cut -c-8`` to get the bootstrap key for a specific version of rustc.

[eddyb]

In Debian, we would like the option to build/rebuild the current release from either the current or previous stable release.

How is that supported in any way? AFAIK the only supported versions are "previous release" and "exact same as the source being compiled".

cc @rust-lang/tools

[cuviper]

If you're setting the --release-channel correctly, so you don't get "-dev" in your version, then your debian builds should get the exact same bootstrap key as upstream rust, whether for the previous or current local rust. So I'm not sure why you need any guesswork/hack detection.

The local-rebuild option also affects how unstable features are used internally. If you don't use this, then the first pass will get --cfg stage0 which toggles code to use only the previous release's features, not anything new. With rebuild it invokes even the first pass with --cfg stage1, as if saying "everything is new!" You might be lucky if this doesn't matter right now, but any breaking transition in unstable features needs to get this right.

[brson]

I can't tell what this is doing. Can you add more comments?

[brson]

Bootstrapping from the current release has been a known requirement (#29556), but that issue was closed by this PR (by @cuviper). Why did that solution end up not working?

[alexcrichton]

Hm yeah I believe the intention of these two flags was:

• --enable-local-rust just means you arranged a stage0 download yourself, but the build system assumes that it's the exact same stage0 that it would have downloaded.
• --enable-local-rebuild means that the stage0 in use is the exact same as the compiler that's going to be produced, in which case the bootstrap key the same.

Along those lines the two situations you've described in the description I think are covered by these two flags? When you're bootstrapping from a previous release --enable-local-rust should be all you need, and when you bootstrap from the current release you'll also pass --enable-local-rebuild.

[infinity0]

@TimNN unfortunately that doesn't work for stable builds.

@alexcrichton Thanks, yes --enable-local-rust does work with the previous stable release. I just thought it would be simpler to enable this functionality under one option, rather than two. IMO it's a little bit pointless for every distributor to have to pass an extra --enable-local-rebuild plus some custom version detection code for the current local rust. However, I do understand the desire to restrict the support to exactly "the current or previous stable release".

Going forward I could:

a) revert this whole PR and instead improve the documentation to be a bit clearer about what distributors are supposed to do with those flags
b) build on top of this PR to make local-stage0.sh error in case the output is not exactly equal to one of the two supported values for the bootstrap key. Then distributors would need to write less code, but still have the version of the stage0 be restricted to two possible options.

[TimNN]

@infinity0 Then I probably misunderstood something, this script behaves as expected for me (tested with stable 1.9.0 & 1.10.0): the first rustc code.rs invocation fails, the second one succeeds.

[cuviper]

Again, the bootstrap key isn't the whole story. Using --cfg stage0 is a problem if the stage0 compiler is actually the current release, instead of the previous release. That's the other part "rebuild" fixes.

I think just improving the documentation is the better choice, if it's not clear what these are doing.

[cuviper]

FWIW, so far I'm not trying to get fancy detecting the local version for the Fedora spec. I just have a rebuild flag to manually toggle in the spec, and then BuildRequires: rust with appropriate version constraints.

[alexcrichton]

@infinity0 actually both of your suggestions sound great to me! I think we could definitely use some bolstered docs here, and I'd love to see the local_stage0.sh script beefed up.

I think the script should:

• If only --enable-local-rust is passed, verify the bootstrap key is the same as the stage0 bootstrap key. In other words, md5 the version and ensure the first 8 characters are the same.
• If --enable-local-rebulid is also passed, verify the bootstrap key is the same as the bootstrap key for the compiler that will be created.

[infinity0]

@cuviper Could you explain in more detail this comment: "Using --cfg stage0 is a problem if the stage0 compiler is actually the current release, instead of the previous release"? I can see how it might be more efficient to "assume stage1 features" if doing a rebuild with the current version, but not why not-assuming stage1 would cause a problem.

[cuviper]

@infinity0 Sometimes there are breaking changes in syntax that hasn't stabilized yet, and --cfg stage0 is used to alternate between using the old and new way in the compiler itself. Consider #32202 for example, particularly commit f0174fc. That "HACK" would do the wrong thing with stage0 if the local rust is really the same current version. So --enable-local-rebuild tricks it to act like stage1 instead, here.

[infinity0]

@TimNN whoops my bad, your snippet works fine. I was confused and expected rustc --version to output the hash as well, and "skimmed" the snippet execution in my head.

@cuviper OK thanks for the explanation. I wasn't expecting those sorts of behaviour changes, but that makes sense.

Based on everyone's comments, my new idea is to set enable-local-rebuild implicitly, if the local-rust version matches CFG_RELEASE. Then I can also revert most of what I wrote in local-stage0.sh. I'll also force-push to this PR because this new idea basically obsoletes most of the original commit, so there's no point leaving it in the history. Please let me know if you prefer otherwise.

Technical details: I have to do this in the makefiles because ./configure does not have access to CFG_RELEASE. But then, config.mk will contain a value for CFG_ENABLE_LOCAL_REBUILD which gets effectively overridden later on, which might be a bit counter-intuitive. To fix this I can refactor the build scripts so that CFG_RELEASE is defined in ./configure instead, but this is more invasive so I'll leave it out for the time being. But if anyone prefers that, please let me know and I'll do that as well.

Naming suggestion: I'd also suggest to rename enable-local-rebuild to enable-stage1-rebuild or something like that, to make it a bit more explicit. The documentation will then read: opt stage1-rebuild 0 "assume the stage0 compiler has stage1 features; implies enable-local-rust. furthermore, this option is implicitly forced on if the local-rust version is the same as the current version being built."

[alexcrichton]

I actually like the idea of inferring enable-local-rebuild, we could basically remove that option from the ./configure script!

Also yeah doing this in the makefiles is fine, but could you also be sure to add support in rustbuild (src/bootstrap) as well?

[infinity0]

@alexcrichton@brson Updated commits, rewriting the original ones. Now I'm comparing local-rustc --version with CFG_RELEASE, and using this to set local-rebuild even if not explicitly set by the user. This should handle the issues raised by @cuviper but also result in less extra code for distributors to write.

These new commits make it harder for distributors to say "I really only want to use the {previous,current} version, fail if local-rust is the {current,previous} version", but I assume that this won't be a common use-case. The build still fails for any other versions of local-rust, and one could always check the build logs for the presence of the "auto-detected local-rebuild" message to see what was actually done.

I've left the actual config options in for now, but could also remove those.

[brson]

Patch is looking better. Nice work.

[alexcrichton]

I believe there's a build_helper::output function which should help with running a command and getting stdout as a string

[alexcrichton]

Nice! Looking great to me as well

[cuviper]

OK, adding on to my note about --version, I see you are actually matching the "rustc" prefix at least. The optional git info would throw it off. I still think we should be a little more precise, because we might want vendor info to show in the basic --version string too, the same way gcc calls itself "gcc version 6.1.1 20160621 (Red Hat 6.1.1-3) (GCC)".

[infinity0]

Understood, it does look like --verbose is more specifically designed to be machine-parsed. I will go ahead and make that change.

[infinity0]

OK, I've made some fixes. I only forced-pushed some of the later commits, but it looks like Github thinks the whole set is "outdated" - they aren't, the 3 initial commits with all your comments are still part of the PR.

Anyway, let me know if you want me to do any commit squashing.

[cuviper]

Not a big deal, but the wildcard is unnecessary. Just s/^release: //p will work.

[cuviper]

Apart from my sed-golfing, it looks good to me, thanks!

[infinity0]

Ack, have fixed the sed expressions as well.

[alexcrichton]

Looks good to me, thanks! Want to squash the commits down as well?

[infinity0]

Sure, done:

old (reviewed) commit: bf4cfb413b5ec08344f738a42ef5e7324899148d
new (squashed) commit: bbff336
shared tree (git cat-file -p $commit): 6288ecb3546e8be7841ae67f4e54eae501d2328f

[alexcrichton]

@bors: r+ bbff336

Thanks!

[bors]

⌛ Testing commit bbff336 with merge 9f9b415...

[infinity0]

Argh, I forgot to update the first post of this PR, and now it's included in the merge commit's commit message.

[alexcrichton]

@bors: r+ bbff336 force

[bors]

💡 This pull request was already approved, no need to approve it again.

• This pull request is currently being tested. If there's no response from the continuous integration service, you may use retry to trigger a build again.

[bors]

⛄ The build was interrupted to prioritize another pull request.

[alexcrichton]

Oh hm, I may have just gotten confused...

[bors]

⌛ Testing commit bbff336 with merge dc8212f...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-cargotest, auto-linux-64-cross-freebsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-mir, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-mir, auto-win-msvc-64-opt-rustbuild
Approved by: alexcrichton
Pushing dc8212f to master...