Compile some CGUs in parallel at the start of codegen by Zoxc · Pull Request #67889 · rust-lang/rust

Zoxc · 2020-01-05T04:54:56Z

This brings the compilation time for syntex_syntax from 11.542s to 10.453s with 6 threads in non-incremental debug mode. Just compiling n CGUs in parallel at the beginning of codegen seems sufficient to get rid of the staircase effect, at least for syntex_syntax.

Based on #67777.

r? @michaelwoerister
cc @alexcrichton @Mark-Simulacrum

michaelwoerister

Looks good to me with nits addressed.

I hope we really can get rid of OnGoingCodegen entirely at some point.

michaelwoerister · 2020-01-06T09:39:20Z

Heads up, @bjorn3: This PR changes the codegen backend protocol.

bjorn3 · 2020-01-06T12:17:25Z

Thanks for the ping @michaelwoerister! This is in a part which I don't yet use though. Because cranelift-module compiles a function as soon as you define it, isn't possible for me to use multiple threads until there are parallel rustc builds for every nightly.

bors · 2020-01-06T16:09:54Z

☔ The latest upstream changes (presumably #67886) made this pull request unmergeable. Please resolve the merge conflicts.

alexcrichton · 2020-01-07T17:23:16Z

This looks like a nice small improvement!

In the limit I'm looking forward to when we can fully reap the benefits here, including:

Removal of the coordinator thread for LLVM CGU management
Removal of cost approximation to know when to CGU and when to optimize
Straight-line code to express "codegen and optimize everything" all in one function that's much easier to read than the coordinator thread

I don't think we can get here until we remove the single-threaded compiler completely though. While this is adding complexity to an already complex backend it's not too too bad and has some modest wins. I think the biggest wins though are going to come when we're able to remove the coordinator thread entirely.

michaelwoerister · 2020-01-09T10:44:36Z

Thanks, @Zoxc.

@bors r+

I'm r+ing under the assumption that the job server questions is resolved to @Mark-Simulacrum's and @alexcrichton's satisfaction. I don't know the details there.

bors · 2020-01-09T10:44:38Z

📌 Commit e3a36c0d78c69ea59a6bde4bae0729dc3545a7b4 has been approved by michaelwoerister

bors · 2020-01-09T15:02:44Z

☔ The latest upstream changes (presumably #67988) made this pull request unmergeable. Please resolve the merge conflicts.

Zoxc · 2020-01-09T21:44:45Z

@bors r=michaelwoerister

bors · 2020-01-09T21:44:46Z

📌 Commit 69bacd0 has been approved by michaelwoerister

@michaelwoerister

Compile some CGUs in parallel at the start of codegen This brings the compilation time for `syntex_syntax` from 11.542s to 10.453s with 6 threads in non-incremental debug mode. Just compiling `n` CGUs in parallel at the beginning of codegen seems sufficient to get rid of the staircase effect, at least for `syntex_syntax`. Based on rust-lang#67777. r? @michaelwoerister cc @alexcrichton @Mark-Simulacrum

@michaelwoerister

Compile some CGUs in parallel at the start of codegen This brings the compilation time for `syntex_syntax` from 11.542s to 10.453s with 6 threads in non-incremental debug mode. Just compiling `n` CGUs in parallel at the beginning of codegen seems sufficient to get rid of the staircase effect, at least for `syntex_syntax`. Based on rust-lang#67777. r? @michaelwoerister cc @alexcrichton @Mark-Simulacrum

@ghost

Rollup of 9 pull requests Successful merges: - #67000 (Promote references to constants instead of statics) - #67756 (Collector tweaks) - #67889 (Compile some CGUs in parallel at the start of codegen) - #67930 (Rename Result::as_deref_ok to as_deref) - #68018 (feature_gate: Remove `GateStrength`) - #68070 (clean up E0185 explanation) - #68072 (Fix ICE #68058) - #68114 (Don't require `allow_internal_unstable` unless `staged_api` is enabled.) - #68120 (Ban `...X` pats, harden tests, and improve diagnostics) Failed merges: r? @ghost

@michaelwoerister

Compile some CGUs in parallel at the start of codegen This brings the compilation time for `syntex_syntax` from 11.542s to 10.453s with 6 threads in non-incremental debug mode. Just compiling `n` CGUs in parallel at the beginning of codegen seems sufficient to get rid of the staircase effect, at least for `syntex_syntax`. Based on rust-lang#67777. r? @michaelwoerister cc @alexcrichton @Mark-Simulacrum

@ghost

Rollup of 8 pull requests Successful merges: - #67756 (Collector tweaks) - #67889 (Compile some CGUs in parallel at the start of codegen) - #67930 (Rename Result::as_deref_ok to as_deref) - #68018 (feature_gate: Remove `GateStrength`) - #68070 (clean up E0185 explanation) - #68072 (Fix ICE #68058) - #68114 (Don't require `allow_internal_unstable` unless `staged_api` is enabled.) - #68120 (Ban `...X` pats, harden tests, and improve diagnostics) Failed merges: r? @ghost

rust-highfive assigned michaelwoerister Jan 5, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 5, 2020

Zoxc force-pushed the parallel-cgus branch from b288a4e to 82b43feCompare January 5, 2020 15:50

michaelwoerister reviewed Jan 6, 2020
View reviewed changes

Comment threadsrc/librustc_codegen_ssa/base.rs Outdated
Comment threadsrc/librustc_codegen_ssa/traits/backend.rs Outdated
Comment threadsrc/librustc_codegen_ssa/base.rs Outdated

Zoxc force-pushed the parallel-cgus branch from 82b43fe to e3a36c0Compare January 8, 2020 05:06

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 9, 2020

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 9, 2020

Zoxc added 2 commits January 9, 2020 22:36

Compile some CGUs in parallel at the start of codegen
f45758c

Precompile CGUs while the main thread has the implicit job server token
69bacd0

Zoxc force-pushed the parallel-cgus branch from e3a36c0 to 69bacd0Compare January 9, 2020 21:38

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 9, 2020

Centril mentioned this pull request Jan 11, 2020
Rollup of 6 pull requests #68121
Closed

Centril mentioned this pull request Jan 11, 2020
Rollup of 9 pull requests #68124
Closed

Centril mentioned this pull request Jan 11, 2020
Rollup of 8 pull requests #68126
Merged

bors merged commit 69bacd0 into rust-lang:masterJan 11, 2020

Zoxc deleted the parallel-cgus branch January 11, 2020 19:11

Navigation Menu

Uh oh!

Uh oh!

Compile some CGUs in parallel at the start of codegen - #67889

Compile some CGUs in parallel at the start of codegen#67889
bors merged 2 commits into
rust-lang:masterfrom
Zoxc:parallel-cgus

Zoxc commented Jan 5, 2020

Uh oh!

michaelwoerister left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelwoerister commented Jan 6, 2020

Uh oh!

bjorn3 commented Jan 6, 2020

Uh oh!

bors commented Jan 6, 2020

Uh oh!

alexcrichton commented Jan 7, 2020

Uh oh!

michaelwoerister commented Jan 9, 2020

Uh oh!

bors commented Jan 9, 2020

Uh oh!

bors commented Jan 9, 2020

Uh oh!

Zoxc commented Jan 9, 2020

Uh oh!

bors commented Jan 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Uh oh!

Conversation

Zoxc commented Jan 5, 2020

Uh oh!

michaelwoerister left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelwoerister commented Jan 6, 2020

Uh oh!

bjorn3 commented Jan 6, 2020

Uh oh!

bors commented Jan 6, 2020

Uh oh!

alexcrichton commented Jan 7, 2020

Uh oh!

michaelwoerister commented Jan 9, 2020

Uh oh!

bors commented Jan 9, 2020

Uh oh!

bors commented Jan 9, 2020

Uh oh!

Zoxc commented Jan 9, 2020

Uh oh!

bors commented Jan 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants