Make incremental compilation thread-safe by Zoxc · Pull Request #49732 · rust-lang/rust

Zoxc · 2018-04-06T14:41:09Z

TimNN · 2018-04-06T14:50:50Z

Your PR failed on Travis. Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:00:46] configure: rust.quiet-tests := True
---
[00:07:31] error[E0433]: failed to resolve. Use of undeclared type or module `Lock`
[00:07:31] --> librustc/ty/context.rs:1815:24
[00:07:31] |
[00:07:31] 1815 | task: &Lock::new(OpenTask::Ignore),
[00:07:31] | ^^^^ Use of undeclared type or module `Lock`
[00:07:31]
[00:07:31] error[E0412]: cannot find type `Lock` in this scope
[00:07:31] --> librustc/ty/context.rs:1728:23
[00:07:31] |
[00:07:31] 1728 | pub task: &'a Lock<OpenTask>,
[00:07:31] | ^^^^ not found in this scope
[00:07:31] help: possible candidates are found in other modules, you can import them into scope
---
[00:08:10] process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name rustc librustc/lib.rs --color always --error-format json --crate-type dylib --emit=dep-info,link -C prefer-dynamic -C opt-level=2 -C metadata=b061046bb39a474d -C extra-filename=-b061046bb39a474d --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps --extern tempdir=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libtempdir-728fcbc86125f341.rlib --extern graphviz=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libgraphviz-01fc77d929e8bdd6.so --extern backtrace=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libbacktrace-a7efadae36f54a3c.rlib --extern jobserver=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libjobserver-3c385cb05f9c08fa.rlib --extern flate2=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libflate2-0249ed74490015f9.rlib --extern byteorder=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libbyteorder-f04ca02424a68c4e.rlib --extern rustc_apfloat=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_apfloat-b23276248c684a7d.rlib --extern rustc_errors=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_errors-d823127e323fa243.so --extern syntax=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax-67023beb7edf290b.so --extern rustc_back=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_back-c188df8980be19c3.so --extern fmt_macros=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libfmt_macros-5106459147843225.so --extern syntax_pos=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax_pos-069d1433ad059ff1.so --extern lazy_static=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/liblazy_static-3c1d922a28d91411.rlib --extern arena=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libarena-2e4ce590106ede04.so --extern bitflags=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libbitflags-e01ce88b04783514.rlib --extern rustc_const_math=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_const_math-2cbdd37d611532dc.so --extern proc_macro=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libproc_macro-414d7e12269f768c.so --extern log=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/liblog-7eebd272275b441b.rlib --extern rustc_data_structures=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_data_structures-1e725c22de2888b8.so --extern serialize=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libserialize-f27fded04dd31022.so --extern serialize=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libserialize-f27fded04dd31022.rlib -L native=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/build/backtrace-sys-751752dec0960570/out/.libs -L native=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/build/miniz-sys-07c608814ba8d6ba/out` (exit code: 101)
[00:08:10] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "--release" "--locked" "--color" "always" "--features" " jemalloc" "--manifest-path" "/checkout/src/rustc/Cargo.toml" "--message-format" "json"
[00:08:10] expected success, got: exit code: 101
[00:08:10] thread 'main' panicked at 'cargo must succeed', bootstrap/compile.rs:1085:9
[00:08:10] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:08:10] travis_fold:end:stage0-rustc
[00:08:10] travis_time:end:stage0-rustc:start=1523026037209707648,finish=1523026246774016214,duration=209564308566
[00:08:10] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap build
[00:08:10] Build completed unsuccessfully in 0:03:44
[00:08:10] Makefile:28: recipe for target 'all' failed
[00:08:10] make: *** [all] Error 1
---
$ ls -lat $HOME/Library/Logs/DiagnosticReports/
ls: cannot access /home/travis/Library/Logs/DiagnosticReports/: No such file or directory
travis_time:end:390c1fc4:start=1523026247263406863,finish=1523026247271580522,duration=8173659
travis_fold:end:after_failure.2
travis_fold:start:after_failure.3
travis_time:start:0163f860
$ find $HOME/Library/Logs/DiagnosticReports -type f -name '*.crash' -not -name '*.stage2-*.crash' -not -name 'com.apple.CoreSimulator.CoreSimulatorService-*.crash' -exec printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" {} \; -exec head -750 {} \; -exec echo travis_fold":"end:crashlog \; || true
find: `/home/travis/Library/Logs/DiagnosticReports': No such file or directory
travis_time:end:0163f860:start=1523026247279544820,finish=1523026247286696899,duration=7152079
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:04ed6ee0
$ dmesg | grep -i kill
[ 10.726120] init: failsafe main process (1094) killed by TERM signal

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN.

bors · 2018-04-10T11:55:12Z

☔ The latest upstream changes (presumably #49390) made this pull request unmergeable. Please resolve the merge conflicts.

Zoxc · 2018-04-11T19:04:11Z

+ if did_allocation {
+ // Only the thread which did the allocation emits the error messages
+
+ // FIXME: Ensure that these are printed before returning for all threads.


I left this FIXME as future work and added it to #48685

Zoxc · 2018-04-11T19:05:12Z

I've cleaned this up a bit now.

Zoxc · 2018-04-12T10:23:01Z

@bors try

bors · 2018-04-12T10:23:14Z

⌛ Trying commit 5bf7762 with merge 23547a1...

@michaelwoerister

Make incremental compilation thread-safe r? @michaelwoerister

bors · 2018-04-12T12:15:32Z

☀️ Test successful - status-travis
State: approved= try=True

Zoxc · 2018-04-12T12:17:19Z

@Mark-Simulacrum Can I get a perf run?

Mark-Simulacrum · 2018-04-12T12:35:02Z

Started.

emilyalbini · 2018-04-16T11:01:28Z

Ping from triage @Mark-Simulacrum! What's the status of the perf run?

Mark-Simulacrum · 2018-04-16T14:51:15Z

We are a little behind on collecting master commits due to an error on the collector, but should get to this soon-ish. Please ping me if I haven't said anything in a day or two...

emilyalbini · 2018-04-18T14:52:20Z

@Mark-Simulacrum well, ping :)

Mark-Simulacrum · 2018-04-18T15:49:08Z

Hm, looks like we're still backlogged. I've made the commit we need to compare against have a higher priority.

Zoxc · 2018-04-19T16:06:54Z

The results have some strange numbers for nll check: http://perf.rust-lang.org/compare.html?start=252a459d373f40512ed9137d59e4b6bea5d6aaee&end=23547a1db3bacc9e92f777b72c9cb3e34cf6a8db&stat=instructions%3Au

@Mark-Simulacrum Do you know what that is about?

Mark-Simulacrum · 2018-04-19T17:05:31Z

ignore nll-check for now, I think it was semi-broken because we removed the -Znll flag. I've since fixed, that, we can rerun try to be certain...

michaelwoerister · 2018-04-20T14:51:05Z

In theory all reads should always come from the same thread. I wonder if we could get rid of this lock.

That's not true. Queries may use multiple threads internally and use queries there.

A particular query invocation should only ever write to one vec and sub-invocations must get their own vec, so the same vec should never be shared among two invocations. Otherwise we would have a logical bug.

A particular query invocation can write to that one vec using multiple threads.

Oh, you mean when the query would use parallelism internally?

As a minor optimization, the Lock could be moved into OpenTask maybe. Then there'd at least be no locking for Ignore and EvalAlways.

michaelwoerister · 2018-04-24T12:57:28Z

We should refactor this to not be mutable at all.

michaelwoerister · 2018-04-24T13:06:42Z

This could be moved out of the dep-graph completely.

Moved where?

If I didn't overlook something then this list could be threaded through as a return value: save_trans_partition() -> copy_module_artifacts_into_incr_comp_cache() -> OngoingCrateTranslation::join().
Doesn't need to happen in this PR though.

michaelwoerister · 2018-04-24T13:20:46Z

Could you rename this to create_task?

michaelwoerister · 2018-04-24T13:22:49Z

And this to finish_task_and_alloc_depnode? (I take full responsibility for this name :))

Or finish_task_and_alloc_depnode, which would be more consistent with the other method names in this module.

michaelwoerister · 2018-04-24T13:25:43Z

Now that raii.rs is gone this can probably be private.

rust-highfive · 2018-04-25T00:35:16Z

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:01:36] warning: spurious network error (1 tries remaining): failed to get 200 response from `https://crates.io/api/v1/crates/getopts/0.2.17/download`, got 503
[00:01:36] error: unable to get packages from source
[00:01:36] [00:01:36] Caused by:
[00:01:36] failed to get 200 response from `https://crates.io/api/v1/crates/getopts/0.2.17/download`, got 503
[00:01:36] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:01:36] Build completed unsuccessfully in 0:00:47
[00:01:36] make: *** [prepare] Error 1
[00:01:36] Makefile:81: recipe for target 'prepare' failed
[00:01:36] Downloading serde_json v1.0.15
[00:01:36] warning: spurious network error (2 tries remaining): failed to get 200 response from `https://crates.io/api/v1/crates/serde_json/1.0.15/download`, got 503
[00:01:37] warning: spurious network error (1 tries remaining): failed to get 200 response from `https://crates.io/api/v1/crates/serde_json/1.0.15/download`, got 503
[00:01:37] error: unable to get packages from source
[00:01:37] error: unable to get packages from source
[00:01:37] [00:01:37] Caused by:
[00:01:37] failed to get 200 response from `https://crates.io/api/v1/crates/serde_json/1.0.15/download`, got 503
[00:01:37] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:01:37] Build completed unsuccessfully in 0:00:00
[00:01:37] Makefile:81: recipe for target 'prepare' failed
[00:01:37] make: *** [prepare] Error 1
[00:01:37] Downloading num_cpus v1.8.0
[00:01:37] Downloading num_cpus v1.8.0
[00:01:37] warning: spurious network error (2 tries remaining): failed to get 200 response from `https://crates.io/api/v1/crates/num_cpus/1.8.0/download`, got 503
[00:01:37] warning: spurious network error (1 tries remaining): failed to get 200 response from `https://crates.io/api/v1/crates/num_cpus/1.8.0/download`, got 503
[00:01:38] error: unable to get packages from source
[00:01:38] Caused by:
[00:01:38] Caused by:
[00:01:38] failed to get 200 response from `https://crates.io/api/v1/crates/num_cpus/1.8.0/download`, got 503
[00:01:38] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:01:38] Build completed unsuccessfully in 0:00:00
[00:01:38] Makefile:81: recipe for target 'prepare' failed
[00:01:38] make: *** [prepare] Error 1
[00:01:38] Downloading serde_derive v1.0.40
[00:01:38] warning: spurious network error (2 tries remaining): failed to get 200 response from `https://crates.io/api/v1/crates/serde_derive/1.0.40/download`, got 503
[00:01:38] warning: spurious network error (1 tries remaining): failed to get 200 response from `https://crates.io/api/v1/crates/serde_derive/1.0.40/download`, got 503
[00:01:38] error: unable to get packages from source
[00:01:38] error: unable to get packages from source
[00:01:38] [00:01:38] Caused by:
[00:01:38] failed to get 200 response from `https://crates.io/api/v1/crates/serde_derive/1.0.40/download`, got 503
[00:01:38] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:01:38] Build completed unsuccessfully in 0:00:00
[00:01:38] make: *** [prepare] Error 1
[00:01:38] Makefile:81: recipe for target 'prepare' failed
[00:01:39] Downloading lazy_static v0.2.11
[00:01:39] warning: spurious network error (2 tries remaining): failed to get 200 response from `https://crates.io/api/v1/crates/lazy_static/0.2.11/download`, got 503
[00:01:39] warning: spurious network error (1 tries remaining): failed to get 200 response from `https://crates.io/api/v1/crates/lazy_static/0.2.11/download`, got 503
[00:01:39] error: unable to get packages from source
[00:01:39] error: unable to get packages from source
[00:01:39] [00:01:39] Caused by:
[00:01:39] failed to get 200 response from `https://crates.io/api/v1/crates/lazy_static/0.2.11/download`, got 503
[00:01:39] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:01:39] Build completed unsuccessfully in 0:00:00
[00:01:39] make: *** [prepare] Error 1
[00:01:39] Makefile:81: recipe for target 'prepare' failed
[00:01:39] The command has failed after 5 attempts.
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 1.
travis_time:start:2b0b4cb4
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
---
travis_fold:start:after_failure.4
travis_time:start:145e86e8
$ dmesg | grep -i kill
[ 10.436234] init: failsafe main process (1094) killed by TERM signal
[ 41.766948] init: plymouth-upstart-bridge main process (510) killed by TERM signal
travis_fold:end:after_failure.4
Done. Your build exited with 1.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Zoxc · 2018-04-25T00:44:28Z

I added refactoring previous_work_products and `work_products to #48685, since they didn't seem trivial to do. I just ended up looking at how to refactor the HIR map...

michaelwoerister · 2018-04-25T08:56:38Z

I added refactoring previous_work_products and `work_products to #48685, since they didn't seem trivial to do. I just ended up looking at how to refactor the HIR map...

👍

michaelwoerister · 2018-04-25T09:34:06Z

OK, I think this looks great! @Zoxc, I'll let you decide if you want to move the Lock into OpenTask or keep things as they are now. r=me when you are done.

Zoxc · 2018-04-25T18:37:46Z

@bors r=michaelwoerister

bors · 2018-04-25T18:37:47Z

📌 Commit 3f802ee has been approved by michaelwoerister

bors · 2018-04-26T06:38:22Z

⌛ Testing commit 3f802ee with merge 1515cde...

@michaelwoerister

Make incremental compilation thread-safe r? @michaelwoerister

bors · 2018-04-26T08:51:34Z

☀️ Test successful - status-appveyor, status-travis
Approved by: michaelwoerister
Pushing 1515cde to master...

rust-highfive assigned michaelwoerister Apr 6, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 6, 2018

Zoxc force-pushed the sync-dep-graph branch from 0651447 to 05a4feaCompare April 6, 2018 15:16

Zoxc force-pushed the sync-dep-graph branch 2 times, most recently from 86d1309 to 5bf7762Compare April 11, 2018 19:02

Zoxc changed the title ~~[WIP] Make incremental compilation thread-safe~~Make incremental compilation thread-safeApr 11, 2018

Zoxc commented Apr 11, 2018
View reviewed changes

bors added a commit that referenced this pull request Apr 12, 2018
Auto merge of#49732- Zoxc:sync-dep-graph, r=<try>
23547a1
Make incremental compilation thread-safe r? @michaelwoerister

Zoxc force-pushed the sync-dep-graph branch from 5bf7762 to 9f41d4eCompare April 17, 2018 14:01

michaelwoerister reviewed Apr 20, 2018
View reviewed changes

michaelwoerister reviewed Apr 24, 2018
View reviewed changes

Comment threadsrc/librustc/dep_graph/graph.rs Outdated
michaelwoeristerApr 24, 2018
Copy link
Copy Markdown
Member
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should refactor this to not be mutable at all.

michaelwoerister reviewed Apr 24, 2018
View reviewed changes

Zoxc force-pushed the sync-dep-graph branch from 9f41d4e to e76c824Compare April 25, 2018 00:30

Zoxc added 2 commits April 25, 2018 11:25

Make DepGraph thread-safe
64b5d40

Address comments
8ec629b

Move the Lock into OpenTask
3f802ee

Zoxc force-pushed the sync-dep-graph branch from e76c824 to 3f802eeCompare April 25, 2018 09:55

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 25, 2018

bors added a commit that referenced this pull request Apr 26, 2018
Auto merge of#49732- Zoxc:sync-dep-graph, r=michaelwoerister
1515cde
Make incremental compilation thread-safe r? @michaelwoerister

bors merged commit 3f802ee into rust-lang:masterApr 26, 2018

bors mentioned this pull request Apr 26, 2018
[DO NOT MERGE] Checking for #[thread_local] crash #50210
Closed

Zoxc deleted the sync-dep-graph branch April 26, 2018 09:27

nikomatsakis mentioned this pull request May 7, 2018
Query Parallelization Tracking Issue #48685
Closed
31 tasks

aturon mentioned this pull request Aug 16, 2019
Audit sources of shared state across rustc #63643
Closed
37 tasks

Uh oh!

Uh oh!

Conversation

Zoxc commented Apr 6, 2018

Uh oh!

TimNN commented Apr 6, 2018

Uh oh!

bors commented Apr 10, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Zoxc commented Apr 11, 2018

Uh oh!

Zoxc commented Apr 12, 2018

Uh oh!

bors commented Apr 12, 2018

Uh oh!

bors commented Apr 12, 2018

Uh oh!

Zoxc commented Apr 12, 2018

Uh oh!

Mark-Simulacrum commented Apr 12, 2018

Uh oh!

emilyalbini commented Apr 16, 2018

Uh oh!

Mark-Simulacrum commented Apr 16, 2018

Uh oh!

emilyalbini commented Apr 18, 2018

Uh oh!

Mark-Simulacrum commented Apr 18, 2018

Uh oh!

Zoxc commented Apr 19, 2018

Uh oh!

Mark-Simulacrum commented Apr 19, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rust-highfive commented Apr 25, 2018

Uh oh!

Zoxc commented Apr 25, 2018

Uh oh!

michaelwoerister commented Apr 25, 2018

Uh oh!

michaelwoerister commented Apr 25, 2018