Skip to content

Allow lexer to recover from some homoglyphs - #62963

Merged
bors merged 2 commits into
rust-lang:masterfrom
estebank:homoglyph-recovery
Jul 26, 2019
Merged

Allow lexer to recover from some homoglyphs#62963
bors merged 2 commits into
rust-lang:masterfrom
estebank:homoglyph-recovery

Conversation

@estebank

@estebankestebank commented Jul 24, 2019

Copy link
Copy Markdown
Contributor

No description provided.

@rust-highfive

rust-highfive commented Jul 24, 2019

Copy link
Copy Markdown
Contributor

r? @eddyb

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfiverust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 24, 2019
Centril
Centril reviewed Jul 25, 2019
View reviewed changes
Comment threadsrc/libsyntax/parse/lexer/unicode_chars.rs Outdated
Comment threadsrc/libsyntax/parse/lexer/unicode_chars.rs Outdated
@Centril

Centril commented Jul 25, 2019

Copy link
Copy Markdown
Contributor

r? @Centril

r=me rollup with nits fixed.

@rust-highfiverust-highfive assigned Centril and unassigned eddybJul 25, 2019
eddyb
eddyb reviewed Jul 25, 2019
View reviewed changes
Comment threadsrc/libsyntax/parse/lexer/unicode_chars.rs
eddyb
eddyb reviewed Jul 25, 2019
View reviewed changes
Comment threadsrc/test/ui/did_you_mean/issue-49746-unicode-confusable-in-float-literal-expt.stderr
| ^

error: aborting due to 2 previous errors
error[E0277]: cannot subtract `{integer}` from `{float}`

@eddybeddybJul 25, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I think we should do the substitution earlier, so it works in literals too, not just full tokens.

@petrochenkovpetrochenkovJul 25, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Unicode stuff is legal inside string literals, so I'm not sure there's a single earlier point where this can be done.

@estebankestebankJul 25, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's something I want to do (precisely for string literals) if and only if there is one unambiguous way of parsing it (like somebody copy-pasted code from a PDF and ends up with println!(‟Hello, world!”);, but doing so would require a more aggressive implementation with backtracking written in such a way that it doesn't affect the common case, and I don't see it would ever handling anything more complex than println!(‟It’s 💖”);. All of that I think should be done, but I'd want it to be part of a separate PR.

@Centril

Centril commented Jul 25, 2019

Copy link
Copy Markdown
Contributor

r? @eddyb

@rust-highfiverust-highfive assigned eddyb and unassigned CentrilJul 25, 2019
@eddyb

eddyb commented Jul 25, 2019

Copy link
Copy Markdown
Contributor

r? @petrochenkov

petrochenkov
petrochenkov reviewed Jul 25, 2019
View reviewed changes
Comment threadsrc/libsyntax/parse/lexer/mod.rs Outdated
petrochenkov
petrochenkov reviewed Jul 25, 2019
View reviewed changes
Comment threadsrc/libsyntax/parse/lexer/unicode_chars.rs Outdated
@petrochenkovpetrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 25, 2019
@rust-highfive

This comment has been minimized.

Sign in to view
@estebank
estebankforce-pushed the homoglyph-recovery branch from cc5f7ab to 6844976CompareJuly 25, 2019 19:37
@petrochenkov

petrochenkov commented Jul 25, 2019

Copy link
Copy Markdown
Contributor

@bors r+

@bors

bors commented Jul 25, 2019

Copy link
Copy Markdown
Collaborator

📌 Commit 6844976 has been approved by petrochenkov

@borsbors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 25, 2019
Centril added a commit to Centril/rust that referenced this pull request Jul 26, 2019
@Centril
Rollup merge ofrust-lang#62963- estebank:homoglyph-recovery, r=petr…
1aae896
…ochenkov
Allow lexer to recover from some homoglyphs
@CentrilCentril mentioned this pull request Jul 26, 2019
Closed
bors added a commit that referenced this pull request Jul 26, 2019
@bors
Auto merge of#63001- Centril:rollup-up9baiv, r=Centril
f907629
Rollup of 14 pull requests
Successful merges:
- #62084 (allow clippy::unreadable_literal in unicode tables)
- #62421 (Introduce `as_deref` to Option)
- #62692 (rustc: precompute the largest Niche and store it in LayoutDetails.)
- #62801 (Remove support for -Zlower-128bit-ops)
- #62828 (Remove vector fadd/fmul reduction workarounds)
- #62862 (code cleanup)
- #62897 (Attempt to fix backtrace tests on i686-msvc)
- #62904 (Disable d32 on armv6 hf targets)
- #62907 (Initialize the MSP430 AsmParser)
- #62956 (Implement slow-path for FirstSets::first)
- #62963 (Allow lexer to recover from some homoglyphs)
- #62970 (ci: gate toolstate repo pushes on the TOOLSTATE_PUBLISH envvar)
- #62983 (Remove needless indirection through Rc)
- #62985 (librustc_errors: Support ui-testing flag in annotate-snippet emitter)
Failed merges:
r? @ghost
Centril added a commit to Centril/rust that referenced this pull request Jul 26, 2019
@Centril
Rollup merge ofrust-lang#62963- estebank:homoglyph-recovery, r=petr…
b8dadb0
…ochenkov
Allow lexer to recover from some homoglyphs
@CentrilCentril mentioned this pull request Jul 26, 2019
Closed
Centril added a commit to Centril/rust that referenced this pull request Jul 26, 2019
@Centril
Rollup merge ofrust-lang#62963- estebank:homoglyph-recovery, r=petr…
1893ac6
…ochenkov
Allow lexer to recover from some homoglyphs
@CentrilCentril mentioned this pull request Jul 26, 2019
Merged
bors added a commit that referenced this pull request Jul 26, 2019
@bors
Auto merge of#63015- Centril:rollup-ydhpcas, r=Centril
c43753f
Rollup of 22 pull requests
Successful merges:
- #62084 (allow clippy::unreadable_literal in unicode tables)
- #62120 (Add missing type links in documentation)
- #62310 (Add missing doc links in boxed module)
- #62421 (Introduce `as_deref` to Option)
- #62583 (Implement Unpin for all raw pointers)
- #62692 (rustc: precompute the largest Niche and store it in LayoutDetails.)
- #62801 (Remove support for -Zlower-128bit-ops)
- #62828 (Remove vector fadd/fmul reduction workarounds)
- #62862 (code cleanup)
- #62904 (Disable d32 on armv6 hf targets)
- #62907 (Initialize the MSP430 AsmParser)
- #62956 (Implement slow-path for FirstSets::first)
- #62963 (Allow lexer to recover from some homoglyphs)
- #62964 (clarify and unify some type test names)
- #62970 (ci: gate toolstate repo pushes on the TOOLSTATE_PUBLISH envvar)
- #62980 (std: Add more accessors for `Metadata` on Windows)
- #62983 (Remove needless indirection through Rc)
- #62985 (librustc_errors: Support ui-testing flag in annotate-snippet emitter)
- #63002 (error_index_generator should output stdout/stderr when it panics.)
- #63004 (Add test for issue-54062)
- #63007 (ci: debug network failures while downloading awscli from PyPI)
- #63009 (Remove redundant `mut` from variable declaration.)
Failed merges:
r? @ghost
@bors
bors merged commit 6844976 into rust-lang:masterJul 26, 2019
@petrochenkovpetrochenkov mentioned this pull request Jul 28, 2019
Merged
@estebank
estebank deleted the homoglyph-recovery branch November 9, 2023 05:19
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@petrochenkovpetrochenkovpetrochenkov left review comments
+3 more reviewers
@eddybeddybeddyb left review comments
@CentrilCentrilCentril left review comments
@matkladmatkladmatklad left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@petrochenkovpetrochenkov

Labels

S-waiting-on-borsStatus: Waiting on bors to run and complete tests. Bors will change the label on completion.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@estebank@rust-highfive@Centril@eddyb@petrochenkov@bors@matklad