Skip to content

Remove trivia tokens - #76170

Merged
bors merged 3 commits into
rust-lang:masterfrom
matklad:notrivia
Sep 2, 2020
Merged

Remove trivia tokens#76170
bors merged 3 commits into
rust-lang:masterfrom
matklad:notrivia

Conversation

@matklad

@matkladmatklad commented Aug 31, 2020

Copy link
Copy Markdown
Contributor

@matklad

matklad commented Aug 31, 2020

Copy link
Copy Markdown
ContributorAuthor

Builds on the #76166

This doesn't actually remove associated enum variants just yet, but, surprisingly, just never constructiong those variants works without any additional tweaks anywhere.

cc @petrochenkov

@matklad

matklad commented Aug 31, 2020

Copy link
Copy Markdown
ContributorAuthor

I think the end result here would be something like

cratefntokenize_into_token_trees(sess:&'aParseSess,source_file:Lrc<rustc_span::SourceFile>,override_span:Option<Span>,) -> (PResult<'a,TokenStream>,Vec<UnmatchedBrace>)

Ie, StringReader and TokenTreesReader would become, at minimum, private, but may actually get completely dissolved

@petrochenkovpetrochenkov self-assigned this Aug 31, 2020
@petrochenkovpetrochenkov added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 31, 2020
petrochenkov
petrochenkov reviewed Aug 31, 2020
View reviewed changes
Comment threadcompiler/rustc_parse/src/lexer/mod.rs Outdated
@petrochenkovpetrochenkov added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 31, 2020
@jyn514jyn514 added T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. WG-parselib labels Sep 1, 2020
@petrochenkovpetrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels Sep 1, 2020
@matkladmatklad changed the title WIP: Remove trivia tokensRemove trivia tokensSep 1, 2020
@matkladmatklad added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 1, 2020
@matklad

matklad commented Sep 1, 2020

Copy link
Copy Markdown
ContributorAuthor

Ready for review @petrochenkov!

I've decided to leave overall code shape changes (fully privatising StringReader) to a follow up PR, to separate pure refactoring from changes to core data structures.

@matklad

matklad commented Sep 1, 2020

Copy link
Copy Markdown
ContributorAuthor

Wait, tests fail locally...

@matkladmatklad added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 1, 2020
@matklad
Simplify TokenTreesReader
fabd8a6
This `joint_to_prev` bit of state is no longer needed.
matklad
matklad commented Sep 1, 2020
View reviewed changes
Comment threadcompiler/rustc_parse/src/lexer/tokentrees.rs
Comment on lines +266 to +268
if !self.token.is_op() {
is_joint = NonJoint;
}

@matkladmatkladSep 1, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally forgot this condition, and that failed some proc-macro tests. This is a bit weird -- I would expect this jointness censoring to happen at the proc_macro_srv layer, and not here.

@petrochenkovpetrochenkovSep 1, 2020•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this check should be moved to proc macro server.
Also lexer should produce the jointness flag for the "delimited group" token trees as well (the match branches above). (Two of such flags, actually, for both the opening and the closing delimiter.)

@matkladmatkladSep 1, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I do this in this PR, or a separate one?

I've also rebased #75528 on top of this PR, seems to work

matklad@21bf6ce

@petrochenkovpetrochenkovSep 1, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I do this in this PR, or a separate one?

Whichever is more convenient.

@matkladmatklad added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 1, 2020
@matklad

matklad commented Sep 1, 2020

Copy link
Copy Markdown
ContributorAuthor

Ok, should be good to go now!

@petrochenkov

petrochenkov commented Sep 1, 2020

Copy link
Copy Markdown
Contributor

@bors r+

@bors

bors commented Sep 1, 2020

Copy link
Copy Markdown
Collaborator

📌 Commit fabd8a6 has been approved by petrochenkov

@borsbors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 1, 2020
@bors

bors commented Sep 2, 2020

Copy link
Copy Markdown
Collaborator

⌛ Testing commit fabd8a6 with merge b4acb11...

@bors

bors commented Sep 2, 2020

Copy link
Copy Markdown
Collaborator

☀️ Test successful - checks-actions, checks-azure
Approved by: petrochenkov
Pushing b4acb11 to master...

@borsbors added the merged-by-bors This PR was explicitly merged by bors. label Sep 2, 2020
@bors
bors merged commit b4acb11 into rust-lang:masterSep 2, 2020
@matklad
matklad deleted the notrivia branch September 2, 2020 07:32
@petrochenkovpetrochenkov mentioned this pull request Dec 11, 2020
Closed
@cuvipercuviper added this to the 1.48.0 milestone Nov 17, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@petrochenkovpetrochenkovpetrochenkov left review comments

Assignees

@petrochenkovpetrochenkov

Labels

merged-by-borsThis PR was explicitly merged by bors.S-waiting-on-borsStatus: Waiting on bors to run and complete tests. Bors will change the label on completion.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Projects

None yet

Milestone

1.48.0

Development

Successfully merging this pull request may close these issues.

5 participants

@matklad@petrochenkov@bors@cuviper@jyn514