Remove rustc_bitflags; use the bitflags crate

[tamird]

r? @alexcrichton

[Mark-Simulacrum]

Why is this adding a dependency on the github bitflags? Are there some unreleased changes we need?

[tamird]

@Mark-Simulacrumbitflags/bitflags#24 is unreleased.

[cuviper]

In the past, github dependencies caused problems for vendored builds, #42719.

[Mark-Simulacrum]

I'm going to suggest we hold off on merging this until we release bitflags. I don't see a reason to rush; and I'd expect that the release would happen fairly soon (though I guess it would be a breaking change).

[tamird]

@cuviper@Mark-Simulacrum holding off seems good to me.

In the meantime, there's a compilation issue here; perhaps you guys could offer some guidance (cc @alexcrichton): the individual crates touched by this change don't compile in the presence of the pre-existing #[no_link] annotations on extern crate bitflags; with errors like:

error[E0433]: failed to resolve. Could not find `_core` in `$crate`
--> src/librustc_llvm/ffi.rs:455:5
|
455 | / bitflags! {
456 | | #[repr(C)]
457 | | #[derive(Default)]
458 | | pub struct DIFlags: ::libc::uint32_t {
... |
477 | | }
478 | | }
| |_____^ Could not find `_core` in `$crate`
|
= note: this error originates in a macro outside of the current crate
error[E0433]: failed to resolve. Could not find `_core` in `$crate`
--> src/librustc_llvm/ffi.rs:455:5
|
455 | / bitflags! {
456 | | #[repr(C)]
457 | | #[derive(Default)]
458 | | pub struct DIFlags: ::libc::uint32_t {
... |
477 | | }
478 | | }
| |_____^ Could not find `_core` in `$crate`
|
= note: this error originates in a macro outside of the current crate
error[E0433]: failed to resolve. Could not find `_core` in `$crate`

However, removing all the #[no_link] annotations results in the linking(?) failure:

$ ./x.py build --stage 1 --incremental src/libcore
Updating submodules
Finished dev [unoptimized] target(s) in 0.0 secs
warning: Package `toml v0.4.5` does not have feature `serde`. It has a required dependency with that name, but only optional dependencies can be used as features. This is currently a warning to ease the transition, but it will become an error in the future.
warning: Package `toml v0.4.5` does not have feature `serde`. It has a required dependency with that name, but only optional dependencies can be used as features. This is currently a warning to ease the transition, but it will become an error in the future.
warning: Package `toml v0.4.5` does not have feature `serde`. It has a required dependency with that name, but only optional dependencies can be used as features. This is currently a warning to ease the transition, but it will become an error in the future.
Building stage0 std artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
Finished release [optimized] target(s) in 0.0 secs
Copying stage0 std from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building stage0 test artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
Finished release [optimized] target(s) in 0.0 secs
Copying stage0 test from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building stage0 compiler artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
Compiling rustc_trans v0.0.0 (file:///Users/tamird/src/rust/src/librustc_trans)
error: cannot satisfy dependencies so `bitflags` only shows up once
|
= help: having upstream crates all available in one format will likely make this go away
error: aborting due to previous error
error: Could not compile `rustc_trans`.
Caused by:
process didn't exit successfully: `/Users/tamird/src/rust/build/bootstrap/debug/rustc --crate-name rustc_trans src/librustc_trans/lib.rs --error-format json --crate-type dylib --emit=dep-info,link -C prefer-dynamic -C opt-level=2 -C metadata=2d33ef16b7a40565 -C extra-filename=-2d33ef16b7a40565 --out-dir /Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -L dependency=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps -L dependency=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/release/deps --extern log=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/liblog-cc7339a977b3dc68.rlib --extern rustc_const_math=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/librustc_const_math-51531a5a4f8c3d3b.dylib --extern jobserver=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/libjobserver-36d133162346d51f.rlib --extern serialize=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/libserialize-4a223d9ec894bec3.dylib --extern serialize=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/libserialize-4a223d9ec894bec3.rlib --extern rustc_incremental=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/librustc_incremental-b587bc6d7163f9f1.dylib --extern rustc_allocator=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/librustc_allocator-aa6dff806e40f32f.dylib --extern rustc_llvm=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/librustc_llvm-09b2cbdb4102bea8.dylib --extern rustc=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/librustc-936f28e604c3cd79.dylib --extern rustc_demangle=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/librustc_demangle-f36982605a35165c.rlib --extern rustc_errors=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/librustc_errors-0699d9bc11181c8d.dylib --extern rustc_back=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/librustc_back-da04b523dd699813.dylib --extern rustc_data_structures=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/librustc_data_structures-6bfac4716e737892.dylib --extern rustc_platform_intrinsics=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/librustc_platform_intrinsics-e239fdddc066f0d3.dylib --extern bitflags=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/libbitflags-263c0bb619d4a987.rlib --extern num_cpus=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/libnum_cpus-24b32baefe3bef43.rlib --extern syntax_pos=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/libsyntax_pos-00853d8a91369014.dylib --extern owning_ref=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/libowning_ref-c705c1e357343128.rlib --extern rustc_trans_utils=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/librustc_trans_utils-0cd15bab7e76f52f.dylib --extern syntax=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/libsyntax-b7ec86d4dc418fb0.dylib --extern flate2=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps/libflate2-3a651fa91d7bd23e.rlib -L native=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/build/miniz-sys-3abc8335811851be/out -L native=/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/build/rustc_llvm-a52c45e3d31f31f9/out -L native=/Users/tamird/src/rust/build/x86_64-apple-darwin/llvm/lib` (exit code: 101)
thread 'main' panicked at 'command did not execute successfully: "/Users/tamird/src/rust/build/x86_64-apple-darwin/stage0/bin/cargo" "build" "-j" "4" "--target" "x86_64-apple-darwin" "--release" "--features" " jemalloc llvm" "--manifest-path" "/Users/tamird/src/rust/src/rustc/Cargo.toml" "--message-format" "json"
expected success, got: exit code: 101', src/bootstrap/compile.rs:883:8
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failed to run: /Users/tamird/src/rust/build/bootstrap/debug/bootstrap build --stage 1 --incremental src/libcore
Build completed unsuccessfully in 0:00:40

[alexcrichton]

I've just published bitflags 1.0.0

[tamird]

@alexcrichton I think I've figured out the linking issue here - it's a vestige of the experimental support for no_std in bitflags. I'm going to open a PR - I think we'll need a 1.0.1.

[alexcrichton]

@tamird any luck with the investigation here?

[tamird]

I haven't had a chance to come back to this, but given recent experience in #44515 I'm surprised we don't see the same failure here.

In that PR, we saw a failure to locate libcore (when trying to use libc from crates.io), while this change presents with a failure that indicates multiple libcores. Why the difference? I'm clueless at the moment, but may dig more in the next few days. Any help would be more than welcome!

[alexcrichton]

This crate isn't depended on before the standard library, so it doesn't need the same treatment as compiler-builtins and liblibc, as to why there's any error at all though I'm not sure.

[tamird]

It appears this has something to do with bits of rustc being built as dylibs vs the bits that are built as rlibs and linked statically. Empirically, I observe this same failure much earlier (during the compilation of rustc_const_math rather than rustc_trans) if I change rustc_apfloat to crate-type = ["dylib"]. Also, looking at the code that produces this error suggests that to be the case. That code was added way back in a82f921.

Well, with that, I'm not exactly sure how to proceed.

[tamird]

Ah, I think bitflags is being statically linked into several dylibs, which is against the rules. Seems like a bug in the selection algorithm in librustc/middle/selection_algorithm.rs, since we should be able to build bitflags as a dylib, but we aren't.

[tamird]

Looks like this is exactly #34909 (comment).

[alexcrichton]

To fix this what you want to do is:

• Add a new crate, librustc_cratesio_shim
• Make this crate compile as a dylib
• Move all rustc crates.io dependencies to this crate
• Make sure all rustc crates transitively depend on this crate.

For now you can probably just do that with bitflags rather than all crates.

[tamird]

I've done what you suggested*, but I still get the same error. * I couldn't "move" bitflags to this crate; each crate that includes `extern crate bitflags` still needs to depend on bitflags explicitly. * I haven't made all rustc crates depend on it, only those that depend on bitflags (for now).

…

On Sat, Sep 16, 2017 at 10:13 AM, Alex Crichton ***@***.***> wrote: To fix this what you want to do is: - Add a new crate, librustc_cratesio_shim - Make this crate compile as a dylib - Move all rustc crates.io dependencies to this crate - Make sure all rustc crates transitively depend on this crate. For now you can probably just do that with bitflags rather than all crates. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44441 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABdsPPwh3fDnS-jpVkpnoyEji0et4ASBks5si9d8gaJpZM4PRx53> .

[alexcrichton]

Er what I mean is that there's a crate rustc_cratesio_shim which links to bitflags from crates.io and also has extern crate bitflags. All other crates that want to use bitflags depend on bitflags from crates.io and have extern crate bitflags. If a crate has extern crate bitflags it must also somehow link to rustc_cratesio_shim, either transitively or directly. Frequently that means you'll link to two crates, bitflags and rustc_cratesio_shim

[tamird]

Ah, I forgot to link to link to rustc_cratesio_shim; done now and seems to work!

I understand that this workaround is meant to get rustc to use bitflags as a dylib, but why is it necessary? all the consumers of bitflags here are already crate-type=dylib, so what does this shim do beyond that?

I would have expected the shim dylib to include bitflags statically, which would have produced the same problem we're seeing without the shim. What's more, why doesn't this problem manifest with other crates.io dependencies like log?

[alexcrichton]

The reasoning here is... complicated. The tl;dr; is that bitflags iteslf isn't a dylib, it's just an rlib (that's the problem). The compiler only wants one copy of a library in all final outputs, so the only way to do that is to get it into a dylib and then have everything else use it through that dylib.

[bors]

☔ The latest upstream changes (presumably #44634) made this pull request unmergeable. Please resolve the merge conflicts.

[tamird]

Thanks for your help! I think this is good to go.

[alexcrichton]

There's not really a bug here, the manifests say they want to get built as rlibs.

[tamird]

I didn't say it was a bug, but also I'm pretty sure bitflags' manifest doesn't specify one way or the other. https://github.com/rust-lang-nursery/bitflags/blob/master/Cargo.toml

[alexcrichton]

Can you add a comment here pointing to the Cargo.toml?

[tamird]

Done.

[alexcrichton]

Can you add a comment here pointing to the Cargo.toml of rustc_cratesio_shim?

[tamird]

Done.

[alexcrichton]

Can you add a comment here pointing to the Cargo.toml of rustc_cratesio_shim?

[tamird]

Done.

[alexcrichton]

Can you add a comment here pointing to the Cargo.toml of rustc_cratesio_shim?

[tamird]

Done.

[alexcrichton]

Can you add a comment here pointing to the Cargo.toml of rustc_cratesio_shim?

[tamird]

Done.

[alexcrichton]

Can you add a comment here pointing to the Cargo.toml of rustc_cratesio_shim?

[tamird]

Done.

[alexcrichton]

Since this (and a number of other crates) transitively depend on syntax you don't need to link this crate in

[tamird]

True, but that feels fragile to me. What's the downside to always including this?

[alexcrichton]

Let's go ahead and remove this b/c otherwise it's just slightly confusing why it's here.

[tamird]

Seems more confusing to me to omit it; as a naive reader I'd expect this to be present anywhere that bitflags is used.

[alexcrichton]

Let's omit it. This is just a "hack" and should be as contained as possible.

[tamird]

OK. Done.

[alexcrichton]

I think because this depends on libsyntax this also isn't needed

[tamird]

(copied from the other comment) True, but that feels fragile to me. What's the downside to always including this?

[alexcrichton]

This depends on libsyntax, so this isn't needed

[tamird]

(copied from the other comment) True, but that feels fragile to me. What's the downside to always including this?

[tamird]

@alexcrichton I think this is ready to go.

[alexcrichton]

@bors: r+

[bors]

📌 Commit 231d9e7 has been approved by alexcrichton

[bors]

⌛ Testing commit 231d9e7 with merge 8696486bcad2a94eeb25271b361a264acff0ac56...

[bors]

💔 Test failed - status-travis

[kennytm]

@bors retry #38618

Spurious musl error.

Details

[01:11:45] �[m�[m�[32m�[1m Running�[m build/x86_64-unknown-linux-gnu/stage2-std/i686-unknown-linux-musl/release/deps/coretests-fe8a2762af9e798e
[01:11:45] [01:11:45] running 676 tests
...
[01:11:45] �[m�[m�[31m�[1merror:�[m An unknown error occurred
[01:11:45] [01:11:45] To learn more, run the command again with --verbose.
[01:11:45] [01:11:45] [01:11:45] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "i686-unknown-linux-musl" "-j" "4" "--release" "--locked" "--color" "always" "--features" "panic-unwind jemalloc backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "-p" "std:0.0.0" "-p" "panic_abort:0.0.0" "-p" "alloc:0.0.0" "-p" "libc:0.0.0" "-p" "rand:0.0.0" "-p" "unwind:0.0.0" "-p" "collections:0.0.0" "-p" "compiler_builtins:0.0.0" "-p" "core:0.0.0" "-p" "std_unicode:0.0.0" "-p" "alloc_system:0.0.0" "--"
[01:11:45] expected success, got: exit code: 101
[01:11:45] [01:11:45] [01:11:45] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --target i686-unknown-linux-musl --target i586-unknown-linux-gnu
[01:11:45] Build completed unsuccessfully in 1:09:49

[bors]

⌛ Testing commit 231d9e7 with merge 3a7b960...

[bors]

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 3a7b960 to master...