Update to the Rusty rust-installer

[cuviper]

This updates the rust-installer submodule to the new version written in Rust (rust-lang/rust-installer#62), now moved to src/tools/rust-installer and invoked in dist.rs as a cargo-based tool command. All of the former shell-script invocations now invoke the tool, otherwise keeping the same arguments as before.

As a small bonus, rustc-src now also uses the same tarball generator, so it gains a smaller .tar.xz too.

Fixes#41569. r? @alexcrichton

[cuviper]

All of the generated dist tarballs look sane to me, and seem to install manually just fine.

However, this does fail one bootstrap test:

step: Step { name: "build-crate-std", stage: 0, host: "A", target: "A" }
step: Step { name: "build-crate-test", stage: 0, host: "A", target: "A" }
step: Step { name: "build-crate-rustc-main", stage: 0, host: "A", target: "A" }
step: Step { name: "build-crate-std", stage: 0, host: "A", target: "B" }
thread 'step::tests::dist_host_with_target_flag' panicked at 'assertion failed: step.target != "B"', step.rs:1582

I think that may just be a reflection of the fact that "dist" now depends on a rust-based tool, but I'd like a second opinion before I go clobbering that test into submission.

cc @brson This re-introduces the file-sorting optimization in the tarballs, so if there's a way to test rustup against this, we should. I'm now writing all directory names in the tarball before the sorted file list, so hopefully that will be enough to avoid a recurrence of rust-lang/rustup#1092.

[alexcrichton]

Oh here (and in other steps) I think you'll want to lock the Step here to build.config.build for both the host and the target triple, that's the architecture we want to build the tool for (as we're running it here, not packaging it)

[alexcrichton]

If you've got a checkout locally you may also want to run cargo vendor followed by ./x.py test src/tools/tidy, that will do a license check of crates in src/vendor to see whether it'll pass the distcheck bot (which'll do this automatically)

[bors]

☔ The latest upstream changes (presumably #41846) made this pull request unmergeable. Please resolve the merge conflicts.

[cuviper]

• Rebased for the Cargo.lock conflict
• Fixed step::tests::dist_host_with_target_flag by making the rust-installer dep build-only as suggested.
• Verified that tidy's vendor license check passes.

[alexcrichton]

@bors: r+

How exciting!

[bors]

📌 Commit 8eaff4d has been approved by alexcrichton

[frewsxcv]

It looks like this fails on WIndows

@bors r-

https://ci.appveyor.com/project/rust-lang/rust/build/1.0.3199/job/21w6ovbl6nycrmcv

[cuviper]

Hmm, OK, I'll have to take a closer look at what sanitize_sh is doing. It might actually be harmful now.

[cuviper]

OK, I removed most of the sanitize_sh calls, where they don't actually reach shell scripts anymore. I don't have a working Windows environment to be sure this suffices though...

[alexcrichton]

@bors: r+

[bors]

📌 Commit 2729b71 has been approved by alexcrichton

[frewsxcv]

@bors r-

https://ci.appveyor.com/project/rust-lang/rust/build/1.0.3211/job/xorw4xry0k4k88rk

[frewsxcv]

@bors r-

https://ci.appveyor.com/project/rust-lang/rust/build/1.0.3211/job/xorw4xry0k4k88rk

[cuviper]

Hmm, it's a link error? I'm confused by this, as it built fine in the previous appveyor run, and I didn't change rust-installer itself since then. Has something else changed in the Windows build system?

 = note: LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other libs; use /NODEFAULTLIB:library
liblzma_sys-f02a9e47b09e132e.rlib(common.obj) : warning LNK4217: locally defined symbol calloc imported in function lzma_alloc_zero
liblzma_sys-f02a9e47b09e132e.rlib(common.obj) : warning LNK4217: locally defined symbol free imported in function lzma_end
liblzma_sys-f02a9e47b09e132e.rlib(common.obj) : warning LNK4217: locally defined symbol malloc imported in function lzma_alloc
liblzma_sys-f02a9e47b09e132e.rlib(lz_encoder.obj) : warning LNK4217: locally defined symbol memmove imported in function fill_window
liblzma_sys-f02a9e47b09e132e.rlib(lzma_decoder.obj) : warning LNK4049: locally defined symbol memmove imported
liblzma_sys-f02a9e47b09e132e.rlib(simple_coder.obj) : warning LNK4049: locally defined symbol memmove imported
liblzma_sys-f02a9e47b09e132e.rlib(stream_encoder_mt.obj) : error LNK2019: unresolved external symbol __imp__beginthreadex referenced in function initialize_new_thread
C:\projects\rust\build\x86_64-pc-windows-msvc\stage0-tools\x86_64-pc-windows-msvc\release\deps\rust_installer-32c88478981b650b.exe : fatal error LNK1120: 1 unresolved externals
error: aborting due to previous error
error: Could not compile `installer`.
To learn more, run the command again with --verbose.

[alexcrichton]

Oh I think that's a legitimate bug with the lzma-sys crate, it's right now apparently not compatible with -C target-feature=+crt-static, I'll look into fixing that.

[alexcrichton]

Looks like a fix was indeed needed, once that passes CI I'll publish to crates.io and you can update the dep here

[cuviper]

@alexcrichton I noticed that alexcrichton/xz2-rs#2 changed the VS version, which hasn't reached crates.io yet. Will that be a problem here?

[Mark-Simulacrum]

I suspect this failure in the rollup was caused by this PR.

@bors r-

[01:24:22] Dist extended stage2 (x86_64-pc-windows-msvc)
[01:27:05] thread 'main' panicked at 'fs::read_dir(src) failed with The system cannot find the path specified. (os error 3)', src\bootstrap\util.rs:61
[01:27:05] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:27:05] Build completed unsuccessfully in 1:15:33

[cuviper]

Do you have the full appveyor log? or a link to that build?
(but I probably won't be able to work on this over the weekend...)

[Mark-Simulacrum]

https://ci.appveyor.com/project/rust-lang/rust/build/1.0.3229

[cuviper]

Oh, I see. This is due to an off-hand optimization I made in rewriting combine-installer.sh. After extracting the input tarballs, instead of copying the components to the combined location, I just moved them. But here, both darwin and windows are expecting to still find those extracted components in the work-dir so they can also copy them into their OS-specific installers.

I don't love that sort of implicit expectation, but I'll go revert that part in rust-installer with a comment about this behavior. The alternative is to just extract them again explicitly -- which is also tempting.

[cuviper]

Hmm, or they could get components from the combined location. I think I like that better, if it works...

[cuviper]

OK, updated rust-installer to stop moving component dirs.

cc @Keruspe the submodule update now includes your changes too.

[bors]

☔ The latest upstream changes (presumably #41987) made this pull request unmergeable. Please resolve the merge conflicts.

[Mark-Simulacrum]

@cuviper Looks like you'll need to rebase to fix the merge conflict on Cargo.lock.

[ranma42]

Also let me warn you in advance about alexcrichton/xz2-rs#4
On FreeBSD, lzma-sys 0.1.2 does not build, you need 0.1.3+

[cuviper]

OK, rebased, and I squashed the trivial dependency fixups. I have included lzma-sys 0.1.3 as suggested.

[alexcrichton]

@bors: r+

[bors]

📌 Commit 1e709fc has been approved by alexcrichton

[bors]

⌛ Testing commit 1e709fc with merge abe1183...

[bors]

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing abe1183 to master...