macros: improve expansion performance by jseyfried · Pull Request #37569 · rust-lang/rust

jseyfried · 2016-11-03T22:46:36Z

#34908 regressed expansion performance on recursive, tt-heavy workloads.
This PR fixes that regression, further improves performance on recursive, tt-heavy workloads, and makes a variety of other improvements to parsing and expansion performance.

Expansion performance improvements:

Test case	Run-time	Memory usage
libsyntax	8%	10%
librustc	15%	6%
librustc_trans	30%	6%
#37074	20%	15%
#34630	40%	8%

r? @eddyb

jseyfried · 2016-11-03T22:47:22Z

The ast::ExprKind commit is the AST version of #37445 (h/t @nnethercote).
cc @durka @nrc

eddyb · 2016-11-03T23:00:36Z

@Manishearth r=me

Manishearth · 2016-11-03T23:07:40Z

@bors r=eddyb

cc @dtolnay

bors · 2016-11-03T23:07:41Z

📌 Commit e652060 has been approved by eddyb

`Token::Interpolated(Nonterminal)` -> `Token::Interpolated(Rc<Nonterminal>)`.

This reverts commit 41745f3.

bluss · 2016-11-04T03:34:36Z

Since not needing ast coercion is a feature, it should have tests so that it is not broken by this improvement. Are there any tests? I didn't find any in the tree.

jseyfried · 2016-11-04T03:40:30Z

By "not needing ast coercion", do you mean #22819? That is tested here.

bluss · 2016-11-04T08:44:42Z

great! I overlooked compile-fail.

jseyfried · 2016-11-04T22:50:11Z

@bors r=eddyb

bors · 2016-11-04T22:50:13Z

📌 Commit 51104e5 has been approved by eddyb

@eddyb

…=eddyb macros: improve expansion performance This PR fixes that regression, further improves performance on recursive, `tt`-heavy workloads, and makes a variety of other improvements to parsing and expansion performance. Expansion performance improvements: | Test case | Run-time | Memory usage | | -------------- | -------- | ------------ | | libsyntax | 8% | 10% | | librustc | 15% | 6% | | librustc_trans | 30% | 6% | | rust-lang#37074 | 20% | 15% | | rust-lang#34630 | 40% | 8% | r? @eddyb

Rollup of 24 pull requests - Successful merges: #37255, #37317, #37408, #37410, #37422, #37427, #37470, #37501, #37537, #37556, #37557, #37564, #37565, #37566, #37569, #37574, #37577, #37579, #37583, #37585, #37586, #37587, #37589, #37596 - Failed merges: #37521, #37547

@eddyb

…=eddyb macros: improve expansion performance This PR fixes that regression, further improves performance on recursive, `tt`-heavy workloads, and makes a variety of other improvements to parsing and expansion performance. Expansion performance improvements: | Test case | Run-time | Memory usage | | -------------- | -------- | ------------ | | libsyntax | 8% | 10% | | librustc | 15% | 6% | | librustc_trans | 30% | 6% | | rust-lang#37074 | 20% | 15% | | rust-lang#34630 | 40% | 8% | r? @eddyb

Rollup of 24 pull requests - Successful merges: #37255, #37317, #37408, #37410, #37422, #37427, #37470, #37501, #37537, #37556, #37557, #37564, #37565, #37566, #37569, #37574, #37577, #37579, #37583, #37585, #37586, #37587, #37589, #37596 - Failed merges: #37521, #37547

nnethercote · 2016-11-07T01:28:30Z

AFAICT this PR regressed the speed of html5ever compilation in rustc-benchmarks. perf.rust-lang.org is showing a 6% regression for html5ever this week while other benchmarks are basically unchanged. This PR is one of the PRs that show up in the regressing measurement, and I have profiling data that corroborates.

One cause of this is the span_to_filename call in Parser::new_with_doc_flag, which causes 800k extra allocations. Another cause is an extra 800k allocations in macro_parser::parse. I think this is because of the Box::new(rdr) at the top, but I'm not certain. (I found these by comparing DHAT output for before and after.)

I think these are not the only parts of the change that lead to regressions, but it's less obvious from my profiling what else might be a factor, other than vec and slice operations are showing up higher in the profile. (I got that from the Cachegrind output.)

jseyfried · 2016-11-07T01:33:45Z

@nnethercote Thanks, I'll fix the regression ASAP.

@jseyfried

…fried Macro parser performance improvements and refactoring This PR locally increased performance of #37074 by ~6.6 minutes. Follow up to #37569, but doesn't focus explicitly on expansion performance. History is relatively clean, but I can/will do some more polishing if this is deemed mergeable. Partially posting this now so I can get Travis to run tests for me. r? @jseyfried

rust-highfive assigned eddyb Nov 3, 2016

jseyfried added 7 commits November 3, 2016 23:48

Clean up parser.parse_token_tree().
5f280a5

Reduce the size of Token and make it cheaper to clone by refactoring
eb3ac29
`Token::Interpolated(Nonterminal)` -> `Token::Interpolated(Rc<Nonterminal>)`.

Revert "macros: Improve tt fragments"
3b71646
This reverts commit 41745f3.

Reimplement "macros: Improve tt fragments" with better performance.
6e9bf12

Make ast::ExprKind smaller.
b60bcba

Avoid recontructing the Parser in macro_parser.rs.
e2b3fec

Move doc comment desugaring into the parser.
7ae0833

jseyfried force-pushed the improve_expansion_perf branch from e652060 to 91c778aCompare November 3, 2016 23:48

jseyfried added 2 commits November 4, 2016 02:38

Improve tt-heavy expansion performance.
23ad6fd

Remove field TtReader::next_tok.
b7eed53

jseyfried force-pushed the improve_expansion_perf branch from 91c778a to b7eed53Compare November 4, 2016 02:40

Fix fallout in tests.
51104e5

alexcrichton mentioned this pull request Nov 4, 2016
Rollup of 24 pull requests #37597
Merged

jseyfried deleted the improve_expansion_perf branch November 6, 2016 09:27

bors merged commit 51104e5 into rust-lang:masterNov 6, 2016

Mark-Simulacrum mentioned this pull request Nov 11, 2016
Macro parser performance improvements and refactoring #37701
Merged

jseyfried mentioned this pull request Dec 6, 2016
possible regression around paths and macro-generated modules #38190
Closed

Mark-Simulacrum mentioned this pull request May 16, 2017
Compiling rust-uinput takes almost 20 minutes #37074
Closed

Navigation Menu

Uh oh!

Uh oh!

macros: improve expansion performance - #37569

macros: improve expansion performance#37569
bors merged 10 commits into
rust-lang:masterfrom
jseyfried:improve_expansion_perf

jseyfried commented Nov 3, 2016•
edited
Loading

Uh oh!

jseyfried commented Nov 3, 2016•
edited
Loading

Uh oh!

eddyb commented Nov 3, 2016

Uh oh!

Manishearth commented Nov 3, 2016

Uh oh!

bors commented Nov 3, 2016

Uh oh!

bluss commented Nov 4, 2016

Uh oh!

jseyfried commented Nov 4, 2016•
edited
Loading

Uh oh!

bluss commented Nov 4, 2016

Uh oh!

jseyfried commented Nov 4, 2016

Uh oh!

bors commented Nov 4, 2016

Uh oh!

nnethercote commented Nov 7, 2016

Uh oh!

jseyfried commented Nov 7, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Uh oh!

Conversation

jseyfried commented Nov 3, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jseyfried commented Nov 3, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eddyb commented Nov 3, 2016

Uh oh!

Manishearth commented Nov 3, 2016

Uh oh!

bors commented Nov 3, 2016

Uh oh!

bluss commented Nov 4, 2016

Uh oh!

jseyfried commented Nov 4, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bluss commented Nov 4, 2016

Uh oh!

jseyfried commented Nov 4, 2016

Uh oh!

bors commented Nov 4, 2016

Uh oh!

nnethercote commented Nov 7, 2016

Uh oh!

jseyfried commented Nov 7, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jseyfried commented Nov 3, 2016•
edited
Loading

jseyfried commented Nov 3, 2016•
edited
Loading

jseyfried commented Nov 4, 2016•
edited
Loading