Skip to content

Move doc comment parsing to rustc_lexer - #75642

Merged
bors merged 2 commits into
rust-lang:masterfrom
matklad:lexer-comments
Aug 21, 2020
Merged

Move doc comment parsing to rustc_lexer#75642
bors merged 2 commits into
rust-lang:masterfrom
matklad:lexer-comments

Conversation

@matklad

@matkladmatklad commented Aug 17, 2020•
edited
Loading

Copy link
Copy Markdown
Contributor

Plain comments are trivia, while doc comments are not, so it feels
like this belongs to the rustc_lexer.

The specific reason to do this is the desire to use rustc_lexer in
rustdoc for syntax highlighting, without duplicating "is this a doc
comment?" logic there.

r? @ghost

@matklad

matklad commented Aug 17, 2020

Copy link
Copy Markdown
ContributorAuthor

r? @petrochenkov

matklad
matklad commented Aug 17, 2020
View reviewed changes
Comment threadsrc/librustc_parse/lexer/mod.rs Outdated
@petrochenkovpetrochenkov added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 17, 2020
petrochenkov
petrochenkov reviewed Aug 18, 2020
View reviewed changes
Comment threadsrc/librustc_ast/util/comments.rs Outdated
Comment threadsrc/librustc_ast/util/comments.rs Outdated
Comment threadsrc/librustc_parse/lexer/mod.rs Outdated
Comment threadsrc/librustc_parse/lexer/mod.rs Outdated
@petrochenkov

petrochenkov commented Aug 18, 2020

Copy link
Copy Markdown
Contributor

r=me with nits addressed.

@petrochenkovpetrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 18, 2020
@matklad

matklad commented Aug 19, 2020•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

I've addressed nits except for qualified paths. I think, in this case, qualified paths would be better.

We have parallel hierarchies of types here:

rustc_lexer: Token TokenKind DocCommentKind CommentKind -----------
rustc_ast: Token TokenKind AttrStyle ----------- CommentKind

So, it seems better if we stick to not qualifying _ast and qualifing _lexer

@matkladmatklad added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 19, 2020
@petrochenkov

petrochenkov commented Aug 19, 2020

Copy link
Copy Markdown
Contributor
CommentKind -----------
----------- CommentKind

This looks pretty bad.
I think we need to come up with a different name for one of these.

@petrochenkov

petrochenkov commented Aug 19, 2020

Copy link
Copy Markdown
Contributor

Block vs Line looks like a "heavier" candidate for CommentKind to me.

The other can be CommentIsDoc::(Yes,No).

Also DocCommentKind -> DocCommentStyle (and change its variant order) to mirror AttrStyle closer.
I'd probably just put AttrStyle as is into the lexer, personally, but there are issues with (en,de)coding and hash impls.

@petrochenkovpetrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 19, 2020
@matklad
Move doc comment parsing to rustc_lexer
39197e6
Plain comments are trivial, while doc comments are not, so it feels
like this belongs to the rustc_lexer.
The specific reason to do this is the desire to use rustc_lexer in
rustdoc for syntax highlighting, without duplicating "is this a doc
comment?" logic there.
@matklad
Simplify search for bare \r in doc comments
ccbe94b
Outer `if` is the fast path -- it calls into hyperoptimized memchr.
The inner loop is just the simplest code possible -- it doesn't
generated the tightest code, but that shouldn't matter if we are going
to error anyhow.
@matklad

matklad commented Aug 19, 2020

Copy link
Copy Markdown
ContributorAuthor

How this version look? I've get rid of CommentKind and renamed DocCommentKind -> DocStyle

@matkladmatklad added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 19, 2020
@petrochenkov

petrochenkov commented Aug 19, 2020

Copy link
Copy Markdown
Contributor

LGTM.
@bors r+

@bors

bors commented Aug 19, 2020

Copy link
Copy Markdown
Collaborator

📌 Commit ccbe94b has been approved by petrochenkov

@borsbors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 19, 2020
@bors

bors commented Aug 21, 2020

Copy link
Copy Markdown
Collaborator

⌛ Testing commit ccbe94b with merge b51651a...

@bors

bors commented Aug 21, 2020

Copy link
Copy Markdown
Collaborator

☀️ Test successful - checks-actions, checks-azure
Approved by: petrochenkov
Pushing b51651a to master...

@borsbors added the merged-by-bors This PR was explicitly merged by bors. label Aug 21, 2020
@bors
bors merged commit b51651a into rust-lang:masterAug 21, 2020
@matklad
matklad deleted the lexer-comments branch August 21, 2020 08:01
@borsbors mentioned this pull request Aug 21, 2020
Closed
@camelidcamelid added the A-parser Area: The lexing & parsing of Rust source code to an AST label Oct 19, 2020
@cuvipercuviper added this to the 1.47.0 milestone May 2, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Mark-SimulacrumMark-SimulacrumMark-Simulacrum left review comments
@petrochenkovpetrochenkovpetrochenkov left review comments

Assignees

@petrochenkovpetrochenkov

Labels

A-parserArea: The lexing & parsing of Rust source code to an ASTmerged-by-borsThis PR was explicitly merged by bors.S-waiting-on-borsStatus: Waiting on bors to run and complete tests. Bors will change the label on completion.

Projects

None yet

Milestone

1.47.0

Development

Successfully merging this pull request may close these issues.

6 participants

@matklad@petrochenkov@bors@Mark-Simulacrum@cuviper@camelid