Use sharded maps for interning by Zoxc · Pull Request #61779 · rust-lang/rust

Zoxc · 2019-06-12T16:13:41Z

Cuts down runtime from 5.5s to 3.8s for non-incremental syntex_syntax check builds with 16 threads / 8 cores.

bors · 2019-06-12T16:44:26Z

☔ The latest upstream changes (presumably #61722) made this pull request unmergeable. Please resolve the merge conflicts.

eddyb · 2019-06-12T17:11:08Z

LGTM, modulo the concurrent data structure, which I don't know who should be reviewing that (@gankro?)

I should mention that my uncertainties around "parallel rustc" in the past were mostly around using locks instead of concurrent data structures, and this is the kind of approach I was hoping to see.
(There are, of course, more drastic things one can do, such as replacing pointer equality with hash equality, that would allow threads to cooperate despite duplication, but those are less clearly overall wins)

I'm now excited about being able to turn on "parallel rustc" by default this year!

cc @rust-lang/compiler

Gankra

just minor style nits on the map, it's not exactly subtle or interesting, afaict (literally just an array of Mutex<Map>s where one is "randomly" selected for each value).

Gankra · 2019-06-12T17:56:07Z

Absent intent to actually use this type as a map, you might as well just make this ShardedHashSet?

It doesn't have set operations, so I'd find that misleading.

Gankra · 2019-06-12T18:00:01Z

Worth putting a comment here explaining that we can't use the map's hasher because we need the hash to find the map?
Also you could arguably do something silly like make a HashMap::default here just so this code is easier to change but... meh?

The hash map's hasher is never used, so there isn't really a reason to access it.

Gankra · 2019-06-12T18:03:57Z

it's a bit off to call this v when it's a key (same for other function)

It's also the interned value we return =P

bors · 2019-06-14T19:12:39Z

☔ The latest upstream changes (presumably #61817) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2019-06-18T22:01:00Z

☔ The latest upstream changes (presumably #61891) made this pull request unmergeable. Please resolve the merge conflicts.

[WIP] Make dep node indices persistent between sessions This makes marking dep nodes green faster (and lock free in the case with no diagnostics). This change is split out from #60035. Unlike #60035 this makes loading the dep graph slower because it loads 2 copies of the dep graph, one immutable and one mutable. Based on #61845, #61779 and #61923.

Zoxc · 2019-06-22T20:46:42Z

Is anything blocking this now?

bors · 2019-06-23T03:20:06Z

☔ The latest upstream changes (presumably #62069) made this pull request unmergeable. Please resolve the merge conflicts.

Zoxc · 2019-06-23T07:37:30Z

@bors try

bors · 2019-06-23T07:37:42Z

⌛ Trying commit 30239ab615197a5f7d6f388f41f952661dcbd1db with merge 62b9c2cb74e80279d8090fafc17fb5bae1fd6fab...

bors · 2019-06-23T09:58:27Z

☀️ Try build successful - checks-travis
Build commit: 62b9c2cb74e80279d8090fafc17fb5bae1fd6fab

Zoxc · 2019-06-23T10:18:02Z

@rust-timer build 62b9c2cb74e80279d8090fafc17fb5bae1fd6fab

rust-timer · 2019-06-23T10:18:03Z

Success: Queued 62b9c2cb74e80279d8090fafc17fb5bae1fd6fab with parent a96ba96, comparison URL.

rust-timer · 2019-06-23T14:40:57Z

Finished benchmarking try commit 62b9c2cb74e80279d8090fafc17fb5bae1fd6fab, comparison URL.

oli-obk · 2019-07-20T12:50:41Z

@bors r+

bors · 2019-07-20T12:50:42Z

📌 Commit 0e73386 has been approved by oli-obk

bors · 2019-07-20T18:03:21Z

⌛ Testing commit 0e73386 with merge 3fe8285...

@eddyb

Use sharded maps for interning Cuts down runtime from 5.5s to 3.8s for non-incremental `syntex_syntax` check builds with 16 threads / 8 cores. r? @eddyb

bors · 2019-07-20T22:03:21Z

💥 Test timed out

oli-obk · 2019-07-22T06:01:42Z

@bors retry

@eddyb

Use sharded maps for interning Cuts down runtime from 5.5s to 3.8s for non-incremental `syntex_syntax` check builds with 16 threads / 8 cores. r? @eddyb

bors · 2019-07-23T09:58:56Z

⌛ Testing commit 0e73386 with merge 3ebca72...

@eddyb

Use sharded maps for interning Cuts down runtime from 5.5s to 3.8s for non-incremental `syntex_syntax` check builds with 16 threads / 8 cores. r? @eddyb

bors · 2019-07-23T13:31:59Z

☀️ Test successful - checks-azure
Approved by: oli-obk
Pushing 3ebca72 to master...

Use sharded maps for queries Based on rust-lang#61779. r? @gankro

rust-highfive assigned eddyb Jun 12, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 12, 2019

Zoxc force-pushed the sharded branch from 1abf914 to c82b67aCompare June 12, 2019 16:24

Gankra reviewed Jun 12, 2019
View reviewed changes

Comment threadsrc/librustc_data_structures/sharded.rs Outdated

Zoxc force-pushed the sharded branch from c82b67a to 3101cccCompare June 12, 2019 17:31

Gankra reviewed Jun 12, 2019
View reviewed changes

Zoxc force-pushed the sharded branch 2 times, most recently from dbefa63 to 6a88dbeCompare June 13, 2019 18:42

Zoxc mentioned this pull request Jun 14, 2019
[WIP] Use a sharded dep node to dep node index map #61845
Closed

Zoxc force-pushed the sharded branch from 6a88dbe to 537c1d1Compare June 14, 2019 19:22

Zoxc mentioned this pull request Jun 21, 2019
Make dep node indices persistent between sessions #62038
Closed

Zoxc force-pushed the sharded branch from 537c1d1 to 3132775Compare June 22, 2019 20:38

Zoxc force-pushed the sharded branch from 3132775 to 30239abCompare June 23, 2019 03:38

Zoxc changed the title ~~Use sharded maps for interning and queries~~[WIP] Use sharded maps for interning and queriesJun 24, 2019

Use sharded maps for interning
0e73386

Zoxc force-pushed the sharded branch from c5e7d05 to 0e73386Compare July 19, 2019 21:38

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 20, 2019

bors added a commit that referenced this pull request Jul 20, 2019
Auto merge of#61779- Zoxc:sharded, r=oli-obk
3fe8285
Use sharded maps for interning Cuts down runtime from 5.5s to 3.8s for non-incremental `syntex_syntax` check builds with 16 threads / 8 cores. r? @eddyb

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 20, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 22, 2019

Centril mentioned this pull request Jul 22, 2019
Rollup of 12 pull requests #62882
Closed

bors added a commit that referenced this pull request Jul 23, 2019
Auto merge of#61779- Zoxc:sharded, r=oli-obk
3ebca72
Use sharded maps for interning Cuts down runtime from 5.5s to 3.8s for non-incremental `syntex_syntax` check builds with 16 threads / 8 cores. r? @eddyb

bors mentioned this pull request Jul 23, 2019
Update RLS and Rustfmt #62805
Merged

bors added the merged-by-bors This PR was explicitly merged by bors. label Jul 23, 2019

bors merged commit 0e73386 into rust-lang:masterJul 23, 2019

Zoxc deleted the sharded branch July 24, 2019 02:05

Mark-Simulacrum added a commit to Mark-Simulacrum/rust that referenced this pull request Aug 11, 2019
Rollup merge ofrust-lang#62108- Zoxc:sharded-queries, r=oli-obk
6381937
Use sharded maps for queries Based on rust-lang#61779. r? @gankro

Centril added a commit to Centril/rust that referenced this pull request Aug 12, 2019
Rollup merge ofrust-lang#62108- Zoxc:sharded-queries, r=oli-obk
4f1cc83
Use sharded maps for queries Based on rust-lang#61779. r? @gankro

Centril added a commit to Centril/rust that referenced this pull request Aug 12, 2019
Rollup merge ofrust-lang#62108- Zoxc:sharded-queries, r=oli-obk
0b4823d
Use sharded maps for queries Based on rust-lang#61779. r? @gankro

Mark-Simulacrum mentioned this pull request Feb 18, 2022
Avoid query cache sharding code in single-threaded mode #94084
Merged

Uh oh!

Uh oh!

Conversation

Zoxc commented Jun 12, 2019

Uh oh!

bors commented Jun 12, 2019

Uh oh!

eddyb commented Jun 12, 2019

Uh oh!

Uh oh!

Gankra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

GankraJun 12, 2019

Choose a reason for hiding this comment

Uh oh!

ZoxcJun 12, 2019

Choose a reason for hiding this comment

Uh oh!

GankraJun 12, 2019

Choose a reason for hiding this comment

Uh oh!

ZoxcJun 12, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

GankraJun 12, 2019

Choose a reason for hiding this comment

Uh oh!

ZoxcJun 12, 2019

Choose a reason for hiding this comment

Uh oh!

bors commented Jun 14, 2019

Uh oh!

bors commented Jun 18, 2019

Uh oh!

Zoxc commented Jun 22, 2019

Uh oh!

bors commented Jun 23, 2019

Uh oh!

Zoxc commented Jun 23, 2019

Uh oh!

bors commented Jun 23, 2019

Uh oh!

bors commented Jun 23, 2019

Uh oh!

Zoxc commented Jun 23, 2019

Uh oh!

rust-timer commented Jun 23, 2019

Uh oh!

rust-timer commented Jun 23, 2019

Uh oh!

oli-obk commented Jul 20, 2019

Uh oh!

bors commented Jul 20, 2019

Uh oh!

bors commented Jul 20, 2019

Uh oh!

bors commented Jul 20, 2019

Uh oh!

oli-obk commented Jul 22, 2019

Uh oh!

bors commented Jul 23, 2019

Uh oh!

bors commented Jul 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants