unify bootstrap and shim binaries

[fee1-dead]

they are now dispatched based on the binary name it is called with.
target/debug/rust{c, doc} are now hard linked to bootstrap.

This fixes#95141, cc @jyn514.

[rust-highfive]

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

[jyn514]

This is awesome ❤️

[bjorn3]

Maybe use Build::copy instead? It hardlinks and falls back to copying if it fails. Some systems may not support hardlinks.

[jyn514]

Build::copy defeats the point, it means the new files won't be updated on changes.

I don't know a good solution here :/ maybe we could do test somewhere of whether the fs supports file links and copy unconditionally if so?

[fee1-dead]

it means the new files won't be updated on changes

Not sure what you meant. The current implementation is as follows:

/// Copies a file from `src` to `dst`pubfncopy(&self,src:&Path,dst:&Path){ifself.config.dry_run{return;}self.verbose_than(1,&format!("Copy {:?} to {:?}", src, dst));if src == dst {return;}let _ = fs::remove_file(&dst);let metadata = t!(src.symlink_metadata());if metadata.file_type().is_symlink(){let link = t!(fs::read_link(src));t!(symlink_file(link, dst));}elseifletOk(()) = fs::hard_link(src, dst){// Attempt to "easy copy" by creating a hard link// (symlinks don't work on windows), but if that fails// just fall back to a slow `copy` operation.}else{ifletErr(e) = fs::copy(src, dst){panic!("failed to copy `{}` to `{}`: {}", src.display(), dst.display(), e)}t!(fs::set_permissions(dst, metadata.permissions()));let atime = FileTime::from_last_access_time(&metadata);let mtime = FileTime::from_last_modification_time(&metadata);t!(filetime::set_file_times(dst, atime, mtime));}}

Which is akin to an unconditional hard_link.

[jyn514]

No, because if you hit the fs::copy path, then the next time bootstrap is executed "rustc".exists() will return true and it will never recopy the binary. Which defeats the whole point of this change.

[fee1-dead]

That is true, I will just remove the check for rustc.exists() because it was triggering a filesystem error because the dest file already existed.

[jyn514]

Ah, I suppose that works too. The hard link should be very cheap compared to everything else bootstrap does 👍

[Mark-Simulacrum]

Can we name the "primary" (i.e., the one produced by rustc) binary something less difficult to refer to? Maybe rustbuild-binary-dispatch-shim?

Please also squash the commits as you make changes, no need for "review changes" commits.

[bors]

☔ The latest upstream changes (presumably #95253) made this pull request unmergeable. Please resolve the merge conflicts.

[fee1-dead]

Umm, I took a look and it looks like it already tries to remove the destination before copying: https://cs.github.com/rust-lang/rust/blob/fee75fbe11b1fad5d93c723234178b2a329a3c03/src/bootstrap/lib.rs#L1418

[bjorn3]

Windows doesn't allow deleting files that are currently open by default. I believe FILE_DISPOSITION_POSIX_SEMANTICS allows deleting open files though.

[fee1-dead]

So I tried to use posix_delete on windows with the latest commit. Should we see whether that will work?

[Mark-Simulacrum]

I'm not sure that function is publicly exposed, but we can try. @bors r+

[bors]

📌 Commit 64a8af9 has been approved by Mark-Simulacrum

[bors]

⌛ Testing commit 64a8af9 with merge c0220834cf72ccea19c0454ad82db04e18ea67a7...

[bors]

💔 Test failed - checks-actions

[rust-log-analyzer]

The job x86_64-msvc-tools failed! Check out the build log: (web)(plain)

Click to see the possible cause of the failure (guessed by this bot)

 Compiling xz2 v0.1.6
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:36
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:67
--> src\bootstrap\lib.rs:1430:67
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0624]: associated function `posix_delete` is private
--> src\bootstrap\lib.rs:1431:30
|
1431 | let _ = file.posix_delete();
Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:36
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:67
--> src\bootstrap\lib.rs:1430:67
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0624]: associated function `posix_delete` is private
--> src\bootstrap\lib.rs:1431:30
|
1431 | let _ = file.posix_delete();
Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:36
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:67
--> src\bootstrap\lib.rs:1430:67
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0624]: associated function `posix_delete` is private
--> src\bootstrap\lib.rs:1431:30
|
1431 | let _ = file.posix_delete();
Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:36
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:67
--> src\bootstrap\lib.rs:1430:67
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0624]: associated function `posix_delete` is private
--> src\bootstrap\lib.rs:1431:30
|
1431 | let _ = file.posix_delete();
Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:36
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:67
--> src\bootstrap\lib.rs:1430:67
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0624]: associated function `posix_delete` is private
--> src\bootstrap\lib.rs:1431:30
|
1431 | let _ = file.posix_delete();
Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors

[bors]

⌛ Testing commit 64a8af9 with merge 83698375db8b68e4b56f5a650443f96accc44471...

[bors]

💔 Test failed - checks-actions

[rust-log-analyzer]

The job i686-msvc-1 failed! Check out the build log: (web)(plain)

Click to see the possible cause of the failure (guessed by this bot)

 Compiling xz2 v0.1.6
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:36
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:67
--> src\bootstrap\lib.rs:1430:67
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0624]: associated function `posix_delete` is private
--> src\bootstrap\lib.rs:1431:30
|
1431 | let _ = file.posix_delete();
Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:36
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:67
--> src\bootstrap\lib.rs:1430:67
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0624]: associated function `posix_delete` is private
--> src\bootstrap\lib.rs:1431:30
|
1431 | let _ = file.posix_delete();
Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:36
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:67
--> src\bootstrap\lib.rs:1430:67
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0624]: associated function `posix_delete` is private
--> src\bootstrap\lib.rs:1431:30
|
1431 | let _ = file.posix_delete();
Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:36
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:67
--> src\bootstrap\lib.rs:1430:67
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0624]: associated function `posix_delete` is private
--> src\bootstrap\lib.rs:1431:30
|
1431 | let _ = file.posix_delete();
Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:36
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0603]: module `sys` is private
--> src\bootstrap\lib.rs:1430:67
--> src\bootstrap\lib.rs:1430:67
|
1430 | if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
| ^^^ private module
note: the module `sys` is defined here
error[E0624]: associated function `posix_delete` is private
--> src\bootstrap\lib.rs:1431:30
|
1431 | let _ = file.posix_delete();
Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors

[jyn514]

This should no longer be necessary with the new plan, since no one will run cargo run locally (they'll either download it from CI or build all binaries from source through x.py): #95141 (comment)

Sorry to spend so much of your time on this.

[fee1-dead]

I did take a look at rustup, and it looks like they retry the deletion with timeout instead of invoking the functions once. Maybe that will resolve the issue (perhaps due to concurrency?) But if this is not needed anymore I'm happy to leave it.