Skip to content

Clean rustc/parse/src/lexer to improve maintainability - #144239

Merged
bors merged 1 commit into
rust-lang:masterfrom
xizheyin:clean-lexer
Jul 23, 2025
Merged

Clean rustc/parse/src/lexer to improve maintainability#144239
bors merged 1 commit into
rust-lang:masterfrom
xizheyin:clean-lexer

Conversation

@xizheyin

@xizheyinxizheyin commented Jul 21, 2025

Copy link
Copy Markdown
Member

This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:

  • rename make_unclosed_delims_error to more appropriate make_mismatched_closing_delims_errors
  • changes return type from Option to Vec<Diag> to avoid lengthy vec processing at lex_token_trees
  • use splice instead of extend to make the logic clearer, since errs sounds more generic and better suited as a return value

In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.

In the third commit, I moves eof_err function below parsing logic for better code flow.

In the forth one, I extract calculate_spacing function to eliminate duplicate spacing logic between bump and bump_minimal functions.

r? compiler

@rustbotrustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jul 21, 2025
@xizheyinxizheyin changed the title Clean lexerClean rustc/parse/src/lexer to improve maintainabilityJul 21, 2025
fee1-dead
fee1-dead requested changes Jul 21, 2025
View reviewed changes
Comment threadcompiler/rustc_parse/src/lexer/mod.rs Outdated
Comment threadcompiler/rustc_parse/src/lexer/diagnostics.rs Outdated
Comment threadcompiler/rustc_parse/src/lexer/mod.rs Outdated
Comment threadcompiler/rustc_parse/src/lexer/tokentrees.rs Outdated
@rustbotrustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 21, 2025
@xizheyin

xizheyin commented Jul 22, 2025

Copy link
Copy Markdown
MemberAuthor

@rustbot ready

@rustbotrustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 22, 2025
@fee1-dead

fee1-dead commented Jul 22, 2025

Copy link
Copy Markdown
Member

Please squash the commits

@xizheyin

xizheyin commented Jul 22, 2025

Copy link
Copy Markdown
MemberAuthor

ready. :)

@fee1-dead

fee1-dead commented Jul 22, 2025

Copy link
Copy Markdown
Member

@bors r+ rollup

@bors

bors commented Jul 22, 2025

Copy link
Copy Markdown
Collaborator

📌 Commit 7a50fe7 has been approved by fee1-dead

It is now in the queue for this repository.

@borsbors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 22, 2025
@fee1-dead

fee1-dead commented Jul 22, 2025

Copy link
Copy Markdown
Member

@bors r-

@borsbors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 22, 2025
fee1-dead
fee1-dead requested changes Jul 22, 2025
View reviewed changes
Comment threadcompiler/rustc_parse/src/lexer/tokentrees.rs Outdated
@xizheyin
Clean code for rustc_parse/src/lexer
2832517
1. Rename `make_unclosed_delims_error` and return `Vec<Diag>`
2. change magic number `unclosed_delimiter_show_limit` to const
3. move `eof_err` below parsing logic
4. Add `calculate_spacing` for `bump` and `bump_minimal`
Signed-off-by: xizheyin <xizheyin@smail.nju.edu.cn>
@xizheyin

xizheyin commented Jul 22, 2025

Copy link
Copy Markdown
MemberAuthor

@rustbot ready

@rustbotrustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 22, 2025
@fee1-dead

fee1-dead commented Jul 23, 2025

Copy link
Copy Markdown
Member

@bors r+ rollup

@bors

bors commented Jul 23, 2025

Copy link
Copy Markdown
Collaborator

📌 Commit 2832517 has been approved by fee1-dead

It is now in the queue for this repository.

@borsbors removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 23, 2025
@borsbors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Jul 23, 2025
jhpratt added a commit to jhpratt/rust that referenced this pull request Jul 23, 2025
@jhpratt
Rollup merge ofrust-lang#144239- xizheyin:clean-lexer, r=fee1-dead
48caa60
Clean `rustc/parse/src/lexer` to improve maintainability
This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value
In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.
In the third commit, I moves `eof_err` function below parsing logic for better code flow.
In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.
r? compiler
@jhprattjhpratt mentioned this pull request Jul 23, 2025
Closed
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jul 23, 2025
@matthiaskrgr
Rollup merge ofrust-lang#144239- xizheyin:clean-lexer, r=fee1-dead
7599e0c
Clean `rustc/parse/src/lexer` to improve maintainability
This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value
In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.
In the third commit, I moves `eof_err` function below parsing logic for better code flow.
In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.
r? compiler
@matthiaskrgrmatthiaskrgr mentioned this pull request Jul 23, 2025
Closed
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jul 23, 2025
@matthiaskrgr
Rollup merge ofrust-lang#144239- xizheyin:clean-lexer, r=fee1-dead
a4e3745
Clean `rustc/parse/src/lexer` to improve maintainability
This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value
In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.
In the third commit, I moves `eof_err` function below parsing logic for better code flow.
In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.
r? compiler
@matthiaskrgrmatthiaskrgr mentioned this pull request Jul 23, 2025
Closed
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jul 23, 2025
@matthiaskrgr
Rollup merge ofrust-lang#144239- xizheyin:clean-lexer, r=fee1-dead
d49ba15
Clean `rustc/parse/src/lexer` to improve maintainability
This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value
In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.
In the third commit, I moves `eof_err` function below parsing logic for better code flow.
In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.
r? compiler
@matthiaskrgrmatthiaskrgr mentioned this pull request Jul 23, 2025
Closed
bors added a commit that referenced this pull request Jul 23, 2025
@bors
Auto merge of#144349- matthiaskrgr:rollup-0eegcyt, r=matthiaskrgr
6adc93b
Rollup of 10 pull requests
Successful merges:
- #144173 (Remove tidy checks for `tests/ui/issues/`)
- #144234 (Fix broken TLS destructors on 32-bit win7)
- #144239 (Clean `rustc/parse/src/lexer` to improve maintainability)
- #144247 (coretests/num: use ldexp instead of hard-coding a power of 2)
- #144256 (Don't ICE on non-TypeId metadata within TypeId)
- #144290 (update tests/ui/SUMMARY.md)
- #144292 (mbe: Use concrete type for `get_unused_rule`)
- #144298 (coverage: Enlarge empty spans during MIR instrumentation, not codegen)
- #144311 (Add powerpc64le-unknown-linux-musl to CI rustc targets)
- #144315 (bootstrap: add package.json and package-lock.json to dist tarball)
r? `@ghost`
`@rustbot` modify labels: rollup
GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this pull request Jul 23, 2025
@GuillaumeGomez
Rollup merge ofrust-lang#144239- xizheyin:clean-lexer, r=fee1-dead
a3e9041
Clean `rustc/parse/src/lexer` to improve maintainability
This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value
In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.
In the third commit, I moves `eof_err` function below parsing logic for better code flow.
In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.
r? compiler
@GuillaumeGomezGuillaumeGomez mentioned this pull request Jul 23, 2025
Closed
bors added a commit that referenced this pull request Jul 23, 2025
@bors
Auto merge of#144349- matthiaskrgr:rollup-0eegcyt, r=matthiaskrgr
69e29e8
Rollup of 10 pull requests
Successful merges:
- #144173 (Remove tidy checks for `tests/ui/issues/`)
- #144234 (Fix broken TLS destructors on 32-bit win7)
- #144239 (Clean `rustc/parse/src/lexer` to improve maintainability)
- #144247 (coretests/num: use ldexp instead of hard-coding a power of 2)
- #144256 (Don't ICE on non-TypeId metadata within TypeId)
- #144290 (update tests/ui/SUMMARY.md)
- #144292 (mbe: Use concrete type for `get_unused_rule`)
- #144298 (coverage: Enlarge empty spans during MIR instrumentation, not codegen)
- #144311 (Add powerpc64le-unknown-linux-musl to CI rustc targets)
- #144315 (bootstrap: add package.json and package-lock.json to dist tarball)
r? `@ghost`
`@rustbot` modify labels: rollup
try-job: x86_64-gnu-aux
@matthiaskrgrmatthiaskrgr mentioned this pull request Jul 23, 2025
Merged
bors added a commit that referenced this pull request Jul 23, 2025
@bors
Auto merge of#144360- matthiaskrgr:rollup-b6ej0mm, r=matthiaskrgr
29a5872
Rollup of 9 pull requests
Successful merges:
- #144173 (Remove tidy checks for `tests/ui/issues/`)
- #144234 (Fix broken TLS destructors on 32-bit win7)
- #144239 (Clean `rustc/parse/src/lexer` to improve maintainability)
- #144256 (Don't ICE on non-TypeId metadata within TypeId)
- #144290 (update tests/ui/SUMMARY.md)
- #144292 (mbe: Use concrete type for `get_unused_rule`)
- #144298 (coverage: Enlarge empty spans during MIR instrumentation, not codegen)
- #144311 (Add powerpc64le-unknown-linux-musl to CI rustc targets)
- #144315 (bootstrap: add package.json and package-lock.json to dist tarball)
r? `@ghost`
`@rustbot` modify labels: rollup
@bors
bors merged commit 8034261 into rust-lang:masterJul 23, 2025
11 checks passed
@rustbotrustbot added this to the 1.90.0 milestone Jul 23, 2025
rust-timer added a commit that referenced this pull request Jul 23, 2025
@rust-timer
Unrolled build for#144239
3170f83
Rollup merge of #144239 - xizheyin:clean-lexer, r=fee1-dead
Clean `rustc/parse/src/lexer` to improve maintainability
This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value
In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.
In the third commit, I moves `eof_err` function below parsing logic for better code flow.
In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.
r? compiler
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fee1-deadfee1-deadfee1-dead requested changes

Assignees

@fee1-deadfee1-dead

Labels

S-waiting-on-borsStatus: Waiting on bors to run and complete tests. Bors will change the label on completion.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Projects

None yet

Milestone

1.90.0

Development

Successfully merging this pull request may close these issues.

4 participants

@xizheyin@fee1-dead@bors@rustbot