Allow writing metadata without llvm by bjorn3 · Pull Request #44085 · rust-lang/rust

bjorn3 · 2017-08-25T17:34:10Z

Todo:

Rebase
Fix eventual errors
~~Find some crate to write elf files~~ (will do it later)

Cc #43842

rust-highfive · 2017-08-25T17:34:15Z

r? @arielb1

(rust_highfive has picked a reviewer for you, use r? to override)

Mark-Simulacrum

I'm happy to see this as a possibility! Left a few minor comments about bootstrap.

Mark-Simulacrum · 2017-08-25T23:41:07Z

+ builder.cargo(compiler, Mode::Libstd, target, "build")
+ }else{
+ builder.cargo(compiler, Mode::Libstd, target, "check")
+ };


This seems wrong/unrelated? If not, reasoning would be appreciated.

Should make this no llvm only. When stage 1 rustc doesnt have llvm support it cant build regular rlibs, but only metadata only rlibs.

Ah. I suppose that makes sense.

Mark-Simulacrum · 2017-08-25T23:41:28Z

 // missing
 // We also only build the runtimes when --enable-sanitizers (or its
 // config.toml equivalent) is used
+ //cargo.env("RUST_FLAGS", "-Zno-trans");


This will need to go before we merge.

bjorn3 · 2017-08-26T08:16:09Z

+ ::rustc_trans_utils::find_exported_symbols(tcx, &analysis.reachable);
+ let (metadata, _hashes) =
+ cstore.encode_metadata(tcx, &link_meta, &exported_symbols);
+ let mut builder = Builder::new(File::create("abc.rlib").unwrap());


Forgot to make this write to the file asked to, instead of abc.rlib.

bjorn3 · 2017-08-26T15:42:09Z

$ ./x.py build --stage 1
[...]
Building stage1 compiler artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
Compiling rustc_back v0.0.0 (file:///Users/bjorn/Documents/rust_fork/src/librustc_back)
Compiling syntax v0.0.0 (file:///Users/bjorn/Documents/rust_fork/src/libsyntax)
Compiling log v0.3.8
[...]
Compiling rustc_save_analysis v0.0.0 (file:///Users/bjorn/Documents/rust_fork/src/librustc_save_analysis)
warning: LLVM not supported, so non rlib output type dylib will be treated like rlib libraries
warning: LLVM not supported, so non rlib output type dylib will be treated like rlib libraries
Compiling rustc-main v0.0.0 (file:///Users/bjorn/Documents/rust_fork/src/rustc)
error: LLVM is not supported by this rustc
error: Could not compile `rustc-main`.
Caused by:
process didn't exit successfully: `/Users/bjorn/Documents/rust_fork/no_llvm_build/build/bootstrap/debug/rustc --crate-name rustc /Users/bjorn/Documents/rust_fork/src/rustc/rustc.rs --error-format json --crate-type bin --emit=dep-info,link -C opt-level=2 --cfg feature="jemalloc" --cfg feature="rustc_back" -C metadata=9c436ad388a4a0ca -C extra-filename=-9c436ad388a4a0ca --out-dir /Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -L dependency=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps -L dependency=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/release/deps --extern rustc_back=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps/librustc_back-853a8b36deced665.dylib --extern rustc_driver=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps/librustc_driver-cc8fef39b95e564b.dylib -L native=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/build/miniz-sys-e7bda6e0ddcea6f1/out` (exit code: 101)
thread 'main' panicked at 'command did not execute successfully: "/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage0/bin/cargo" "build" "-j" "8" "--target" "x86_64-apple-darwin" "--release" "--features" " jemalloc" "--manifest-path" "/Users/bjorn/Documents/rust_fork/src/rustc/Cargo.toml" "--message-format" "json"
expected success, got: exit code: 101', /Users/bjorn/Documents/rust_fork/src/bootstrap/compile.rs:883:8
[...]

bjorn3 · 2017-08-28T17:06:03Z

What is this error message of the "Fix tidy errors" commit:

[00:16:47] error: linking with `cc` failed: exit code: 1
[00:16:47] |
[00:16:47] = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/build/rustc_driver-78c26d64350eb204/build_script_build-78c26d64350eb204.0.o" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/build/rustc_driver-78c26d64350eb204/build_script_build-78c26d64350eb204" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/build/rustc_driver-78c26d64350eb204/build_script_build-78c26d64350eb204.crate.allocator.o" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-ea2e08566ed2ca9b.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librand-d011a0d28d72d28b.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-117698016bc6c938.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_system-aa6c11e69908f4f9.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-d94cd37ff18e6e1c.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-388107ef329739bd.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-c45b3574292a7ea1.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-c791e7351d8a272a.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_unicode-54feb014ef663465.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-81b84a407143b5e1.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-ba9896bad78dc777.rlib" "-Wl,-Bdynamic" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util"
[00:16:47] = note: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librand-d011a0d28d72d28b.rlib: error adding symbols: Archive has no index; run ranlib to add one
[00:16:47] collect2: error: ld returned 1 exit status
[00:16:47] [00:16:47] [00:16:47] error: aborting due to previous error
[00:16:47] [00:16:47] error: Could not compile `syntax`.

arielb1 · 2017-08-29T18:51:45Z

[00:02:07] ======================================================================
[00:02:07] FAIL: program_config (bootstrap.RustBuild)
[00:02:07] Doctest: bootstrap.RustBuild.program_config
[00:02:07] ----------------------------------------------------------------------
[00:02:07] Traceback (most recent call last):
[00:02:07] File "/usr/lib/python2.7/doctest.py", line 2226, in runTest
[00:02:07] raise self.failureException(self.format_failure(new.getvalue()))
[00:02:07] AssertionError: Failed doctest test for bootstrap.RustBuild.program_config
[00:02:07] File "/checkout/src/bootstrap/bootstrap.py", line 519, in program_config
[00:02:07] [00:02:07] ----------------------------------------------------------------------
[00:02:07] File "/checkout/src/bootstrap/bootstrap.py", line 527, in bootstrap.RustBuild.program_config
[00:02:07] Failed example:
[00:02:07] cargo_path.rstrip(".exe") == os.path.join("/tmp/rust",
[00:02:07] "bin", "cargo")
[00:02:07] Expected:
[00:02:07] True
[00:02:07] Got:
[00:02:07] False

arielb1 · 2017-08-29T18:52:30Z

Didn't you mean !self.build.config.llvm_enabled here?

This doesnt have self.build.config but i dont think its compiled with the feature, so i think i will have to use an env var.

Yeah, an env var would be fine. I think there's already precedent for that (i.e. grep for a env var with LLVM in it) so there may not be a need to add a new one.

arielb1 · 2017-09-05T18:10:31Z

[00:35:42] error[E0308]: mismatched types
[00:35:42] --> /checkout/src/librustdoc/test.rs:260:31
[00:35:42] |
[00:35:42] 260 | driver::compile_input(&sess, &cstore, &input, &out, &None, None, &control)
[00:35:42] | ^^^^^ types differ in mutability
[00:35:42] |
[00:35:42] = note: expected type `&mut rustc::session::Session`
[00:35:42] found type `&rustc::session::Session`

bors · 2017-09-08T14:43:21Z

☔ The latest upstream changes (presumably #44142) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2017-09-16T19:57:02Z

☔ The latest upstream changes (presumably #43017) made this pull request unmergeable. Please resolve the merge conflicts.

bjorn3 · 2017-09-17T08:33:09Z

I added a generic TransCrate trait, which is to be implemented by compiler backends.

arielb1 · 2017-09-17T11:06:52Z

Also modify the comment?

arielb1 · 2017-09-17T11:08:04Z

Consider moving this outside of the main driver function?

Should remove the #[cfg]

arielb1 · 2017-09-17T11:10:47Z

I'm also not sure about the idea of creating dylibs that are actually rlibs.

Without llvm we cant create real dylibs, but many rustc crates are compiled to dylibs, so by emitting rlibs instead we can compile all rustc libs.

arielb1 · 2017-09-17T11:12:10Z

Move this to an out-of-line module?

arielb1 · 2017-09-17T11:12:31Z

Move this to an out-of-line module?

arielb1 · 2017-09-17T11:17:16Z

Maybe also move the "metadata-only" TransCrate implementation ~~here~~ to rustc_trans_utils?

arielb1 · 2017-09-17T11:21:15Z

Why does this exist as a separate crate from rustc_trans_utils? They are both depended upon by exactly rustc_trans and rustc_driver, and they semantically perform a very similar task.

I was thinking about utils to be used by backends only, while traits contains the glue between backends and rustc_driver. I can merge them though.

arielb1 · 2017-09-17T11:23:43Z

Can you move all of these items to the TransCrate trait?

No, because most of them are llvm specific, eg they would still use llvm even if trans is non llvm. (enable_llvm_debug, print_passes)

bjorn3 · 2017-09-23T13:48:28Z

@arielb1 rebased and fixed most nits

arielb1 · 2017-09-24T09:38:24Z

@bors r+

bors · 2017-09-24T09:38:25Z

📌 Commit 3c32c6a has been approved by arielb1

bors · 2017-09-25T00:19:55Z

⌛ Testing commit 3c32c6a with merge 0e7f2f843dce90acc9e8ec1f29665ba69ac1f844...

bors · 2017-09-25T00:38:07Z

💔 Test failed - status-travis

bjorn3 · 2017-09-25T06:19:34Z

Error should be fixed now.

arielb1 · 2017-09-25T09:03:12Z

@bors r+

bors · 2017-09-25T09:03:12Z

📌 Commit 843cd5b has been approved by arielb1

bors · 2017-09-25T18:05:28Z

⌛ Testing commit 843cd5b with merge 3df1f7b...

Allow writing metadata without llvm # Todo: * [x] Rebase * [x] Fix eventual errors * [x] <strike>Find some crate to write elf files</strike> (will do it later) Cc #43842

bors · 2017-09-25T20:36:57Z

☀️ Test successful - status-appveyor, status-travis
Approved by: arielb1
Pushing 3df1f7b to master...

rust-highfive assigned arielb1 Aug 25, 2017

bjorn3 force-pushed the no_llvm_write_metadata branch from 2a63467 to dd253c1Compare August 25, 2017 17:51

shepmaster added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Aug 25, 2017

Mark-Simulacrum requested changes Aug 25, 2017
View reviewed changes

bjorn3 commented Aug 26, 2017
View reviewed changes

arielb1 reviewed Aug 29, 2017
View reviewed changes

bjorn3 force-pushed the no_llvm_write_metadata branch 4 times, most recently from bec1f52 to ea497a2Compare September 16, 2017 16:51

bjorn3 force-pushed the no_llvm_write_metadata branch 3 times, most recently from b88fcf7 to 243d64fCompare September 17, 2017 08:26

bjorn3 changed the title ~~[WIP] Allow writing metadata without llvm~~Allow writing metadata without llvmSep 17, 2017

arielb1 reviewed Sep 17, 2017
View reviewed changes

Comment threadsrc/bootstrap/builder.rs Outdated
arielb1Sep 17, 2017
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also modify the comment?

arielb1 reviewed Sep 17, 2017
View reviewed changes

Comment threadsrc/librustc_driver/lib.rs Outdated
arielb1Sep 17, 2017
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this to an out-of-line module?

arielb1 reviewed Sep 17, 2017
View reviewed changes

Comment threadsrc/librustc_driver/lib.rs Outdated
arielb1Sep 17, 2017
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this to an out-of-line module?

arielb1 reviewed Sep 17, 2017
View reviewed changes

Fix for upstream changes
e130ccc

bjorn3 force-pushed the no_llvm_write_metadata branch from 06155cf to e130cccCompare September 23, 2017 12:48

bjorn3 added 5 commits September 23, 2017 15:00

Fix some nits
43bfd4c

Remove build_diagnostic_array hack
793c02d

Fix tidy errors
70e5bae

Remove leftover
95fb1d0

Remove now unused dependency on flate2 from rustc_driver
1800520

Fix error
56c4670

bjorn3 force-pushed the no_llvm_write_metadata branch from 42e12da to 56c4670Compare September 23, 2017 16:08

Fix some tests with no llvm build
3c32c6a

Fix error
843cd5b

bors merged commit 843cd5b into rust-lang:masterSep 25, 2017

This was referenced Sep 25, 2017
rustc: Implement ThinLTO #44841
Merged
rustc: Enable LTO and multiple codegen units #44783
Closed

bjorn3 deleted the no_llvm_write_metadata branch September 26, 2017 06:17

eddyb mentioned this pull request Oct 14, 2017
Remove librustc_codegen_utils. #45276
Closed

bjorn3 mentioned this pull request Jan 8, 2019
MetadataOnlyCodegenBackend: run the collector only once #57418
Merged

bjorn3 mentioned this pull request Mar 14, 2020
Refactorings to get rid of rustc_codegen_utils #69965
Merged

Uh oh!

Uh oh!

Conversation

bjorn3 commented Aug 25, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Todo:

Uh oh!

rust-highfive commented Aug 25, 2017

Uh oh!

Mark-Simulacrum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bjorn3 commented Aug 26, 2017

Uh oh!

bjorn3 commented Aug 28, 2017

Uh oh!

arielb1 commented Aug 29, 2017

Uh oh!

arielb1Aug 29, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arielb1 commented Sep 5, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bors commented Sep 8, 2017

Uh oh!

bors commented Sep 16, 2017

Uh oh!

bjorn3 commented Sep 17, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arielb1Sep 17, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bjorn3 commented Sep 23, 2017

Uh oh!

arielb1 commented Sep 24, 2017

Uh oh!

bors commented Sep 24, 2017

Uh oh!

bors commented Sep 25, 2017

bjorn3 commented Aug 25, 2017•
edited
Loading

arielb1Aug 29, 2017•
edited
Loading

arielb1 commented Sep 5, 2017•
edited
Loading

arielb1Sep 17, 2017•
edited
Loading