Skip to content

Store tokens inside ast::Expr - #72287

Merged
bors merged 1 commit into
rust-lang:masterfrom
Aaron1011:feature/min-token-collect
May 25, 2020
Merged

Store tokens inside ast::Expr#72287
bors merged 1 commit into
rust-lang:masterfrom
Aaron1011:feature/min-token-collect

Conversation

@Aaron1011

@Aaron1011Aaron1011 commented May 17, 2020•
edited
Loading

Copy link
Copy Markdown
Contributor

This is a smaller version of #70091.

We now store captured tokens inside ast::Expr, which allows us to avoid some reparsing in nt_to_tokenstream. To try to mitigate the performance impact, we only collect tokens when we've seen an outer attribute.

This makes progress towards solving #43081. There are still many things left to do:

  • Collect tokens for other AST items.
  • Come up with a way to handle inner attributes (we need to be collecting tokens by the time we encounter them)
  • Avoid re-parsing when a #[cfg] attr is used.

However, this is enough to fix spans for a simple example, which I've included as a test case.

@rust-highfive

This comment has been minimized.

Sign in to view
@rust-highfiverust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 17, 2020
@Aaron1011

Aaron1011 commented May 17, 2020

Copy link
Copy Markdown
ContributorAuthor

r? @petrochenkov

@Aaron1011

This comment has been minimized.

Sign in to view
@rust-timer

This comment has been minimized.

Sign in to view
@bors

This comment has been minimized.

Sign in to view
@rust-highfive

This comment has been minimized.

Sign in to view
@bors

This comment has been minimized.

Sign in to view
@rust-timer

This comment has been minimized.

Sign in to view
@rust-timer

rust-timer commented May 17, 2020

Copy link
Copy Markdown
Collaborator

Finished benchmarking try commit 2728275cf62b7f7bfb2bbc99674bd8476722477a, comparison URL.

petrochenkov
petrochenkov reviewed May 17, 2020
View reviewed changes
Comment threadsrc/test/ui/proc-macro/auxiliary/noop-collect.rs Outdated
Comment threadsrc/librustc_parse/parser/expr.rs Outdated
Comment threadsrc/librustc_parse/parser/expr.rs Outdated
@petrochenkov

petrochenkov commented May 17, 2020

Copy link
Copy Markdown
Contributor

Not sure why there's such slowdown, absolute majority of expressions doesn't have attributes.
Perhaps it happens due to #72287 (comment)?
(We know that simply increasing the ast::Expr size doesn't result in a slowdown, see #70200.)

@petrochenkovpetrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 17, 2020
@Aaron1011

This comment has been minimized.

Sign in to view
@rust-timer

This comment has been minimized.

Sign in to view
@bors

This comment has been minimized.

Sign in to view
@rust-highfive

This comment has been minimized.

Sign in to view
@bors

This comment has been minimized.

Sign in to view
@rust-timer

This comment has been minimized.

Sign in to view
@rust-timer

rust-timer commented May 18, 2020

Copy link
Copy Markdown
Collaborator

Finished benchmarking try commit 11e2f20befcdd9b10073fe96ba2e880ac822352d, comparison URL.

@petrochenkovpetrochenkov added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 18, 2020
@Aaron1011
Aaron1011force-pushed the feature/min-token-collect branch from 72b44ee to 4855363CompareMay 19, 2020 20:56
@Aaron1011

Aaron1011 commented May 19, 2020

Copy link
Copy Markdown
ContributorAuthor

Let's make sure that my new collect_tokens implementation isn't slower.

@bors try @rust-timer queue

@rust-timer

This comment has been minimized.

Sign in to view
@rust-timer

rust-timer commented May 20, 2020

Copy link
Copy Markdown
Collaborator

Finished benchmarking try commit 7bc7cfa40f58523836318ebc43f17436ce067963, comparison URL.

estebank
estebank reviewed May 20, 2020
View reviewed changes
Comment threadsrc/librustc_parse/parser/expr.rs Outdated
@estebank

estebank commented May 20, 2020

Copy link
Copy Markdown
Contributor

FWIW, the code looks mergeable and I'm really excited about the UX improvement.

@Aaron1011
Aaron1011force-pushed the feature/min-token-collect branch from 4855363 to d5a2dbeCompareMay 20, 2020 03:32
@petrochenkov

petrochenkov commented May 20, 2020

Copy link
Copy Markdown
Contributor

I'll review this in detail ~tomorrow, but from a cursory view the collecting mechanism change in librustc_parse/parser/mod.rs looks entirely orthogonal to collecting tokens for expressions specifically.
Could you move it to a separate PR?

@Aaron1011Aaron1011 mentioned this pull request May 20, 2020
Merged
@Aaron1011

Aaron1011 commented May 20, 2020•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@petrochenkov: I opened #72393 with just the collect_tokens changes

@petrochenkov

petrochenkov commented May 21, 2020

Copy link
Copy Markdown
Contributor

Blocked on #72393.

@petrochenkovpetrochenkov added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 21, 2020
RalfJung added a commit to RalfJung/rust that referenced this pull request May 23, 2020
@RalfJung
Rollup merge ofrust-lang#72393- Aaron1011:feature/rewrite-collect-t…
8e98d8d
…okens, r=petrochenkov
Rewrite `Parser::collect_tokens`
The previous implementation did not work when called on an opening
delimiter, or when called re-entrantly from the same `TokenCursor` stack
depth.
I'm not sure how to test this apart from rust-lang#72287
RalfJung added a commit to RalfJung/rust that referenced this pull request May 24, 2020
@RalfJung
Rollup merge ofrust-lang#72393- Aaron1011:feature/rewrite-collect-t…
8692c45
…okens, r=petrochenkov
Rewrite `Parser::collect_tokens`
The previous implementation did not work when called on an opening
delimiter, or when called re-entrantly from the same `TokenCursor` stack
depth.
I'm not sure how to test this apart from rust-lang#72287
@bors

This comment has been minimized.

Sign in to view
@petrochenkovpetrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels May 24, 2020
@Aaron1011
Aaron1011force-pushed the feature/min-token-collect branch from d5a2dbe to 14382c6CompareMay 24, 2020 20:11
@Aaron1011

Aaron1011 commented May 24, 2020

Copy link
Copy Markdown
ContributorAuthor

@petrochenkov: Rebased

@petrochenkov

petrochenkov commented May 24, 2020

Copy link
Copy Markdown
Contributor

@bors r+

@bors

bors commented May 24, 2020

Copy link
Copy Markdown
Collaborator

📌 Commit 14382c6 has been approved by petrochenkov

@borsbors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 24, 2020
@bors

bors commented May 24, 2020

Copy link
Copy Markdown
Collaborator

⌛ Testing commit 14382c6 with merge 62da38d...

@bors

bors commented May 25, 2020

Copy link
Copy Markdown
Collaborator

☀️ Test successful - checks-azure
Approved by: petrochenkov
Pushing 62da38d to master...

@borsbors added the merged-by-bors This PR was explicitly merged by bors. label May 25, 2020
@bors
bors merged commit 62da38d into rust-lang:masterMay 25, 2020
@Aaron1011Aaron1011 mentioned this pull request May 25, 2020
Closed
@taiki-etaiki-e mentioned this pull request Jun 25, 2020
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@estebankestebankestebank left review comments
@petrochenkovpetrochenkovpetrochenkov left review comments

Assignees

@petrochenkovpetrochenkov

Labels

merged-by-borsThis PR was explicitly merged by bors.S-waiting-on-borsStatus: Waiting on bors to run and complete tests. Bors will change the label on completion.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Aaron1011@rust-highfive@rust-timer@bors@petrochenkov@estebank