Uh oh!
There was an error while loading. Please reload this page.
Keep original literal tokens in AST - #60679
Merged
Merged
Conversation
petrochenkov
commented
May 9, 2019
ContributorAuthor
@bors try (for perf) |
bors
commented
May 9, 2019
Collaborator
bors added a commit
that referenced
this pull request
May 9, 2019
Keep original literal tokens in AST The original literal tokens (`token::Lit`) are kept in AST until lowering to HIR. The tokens are kept together with their lowered "semantic" representation (`ast::LitKind`), so the size of `ast::Lit` is increased (this also increases the size of meta-item structs used for processing built-in attributes). However, the size of `ast::Expr` stays the same. The intent is to remove the "semantic" representation from AST eventually and keep literals as tokens until lowering to HIR (at least), and I'm going to work on that, but it would be good to land this sooner to unblock progress on the [lexer refactoring](#59706). Fixes a part of #43081 (literal tokens that are passed to proc macros are always precise, including hexadecimal numbers, strings with their original escaping, etc) Fixes a part of #60495 (everything except for proc macro API doesn't need escaping anymore) This also allows to eliminate a certain hack from the lexer (https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/pretty-printing.20comments/near/165005357). cc @matklad
This was referenced May 9, 2019
matklad
reviewed
May 9, 2019
Uh oh!
There was an error while loading. Please reload this page.
matklad
reviewed
May 9, 2019
Uh oh!
There was an error while loading. Please reload this page.
bors
commented
May 9, 2019
Collaborator
☀️ Try build successful - checks-travis |
petrochenkov
commented
May 9, 2019
ContributorAuthor
@rust-timer build 11c353b |
rust-timer
commented
May 9, 2019
Collaborator
Success: Queued 11c353b with parent ef01f29, comparison URL. |
rust-timer
commented
May 10, 2019
Collaborator
Finished benchmarking try commit 11c353b |
petrochenkov
commented
May 10, 2019
ContributorAuthor
Perf looks acceptable. |
rust-highfive
commented
May 11, 2019
Contributor
The job Click to expand the log.I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
ContributorAuthor
Updated with two commits (one with refactoring and another moving code around). @bors delegate=matklad |
bors
commented
May 11, 2019
Collaborator
✌️ @matklad can now approve this pull request |
matklad
reviewed
May 11, 2019
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Remove some dead code
rust-highfive
commented
May 12, 2019
Contributor
The job Click to expand the log.I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
matklad
commented
May 12, 2019
Contributor
Failure seems spurious? |
petrochenkov
commented
May 12, 2019
ContributorAuthor
Yes, looks spurious. |
Merged
bors
commented
May 12, 2019
Collaborator
bors added a commit
that referenced
this pull request
May 12, 2019
Keep original literal tokens in AST The original literal tokens (`token::Lit`) are kept in AST until lowering to HIR. The tokens are kept together with their lowered "semantic" representation (`ast::LitKind`), so the size of `ast::Lit` is increased (this also increases the size of meta-item structs used for processing built-in attributes). However, the size of `ast::Expr` stays the same. The intent is to remove the "semantic" representation from AST eventually and keep literals as tokens until lowering to HIR (at least), and I'm going to work on that, but it would be good to land this sooner to unblock progress on the [lexer refactoring](#59706). Fixes a part of #43081 (literal tokens that are passed to proc macros are always precise, including hexadecimal numbers, strings with their original escaping, etc) Fixes a part of #60495 (everything except for proc macro API doesn't need escaping anymore) This also allows to eliminate a certain hack from the lexer (https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/pretty-printing.20comments/near/165005357). cc @matklad
bors
commented
May 12, 2019
Collaborator
☀️ Test successful - checks-travis, status-appveyor |
rust-highfive
commented
May 12, 2019
Contributor
📣 Toolstate changed by #60679! Tested on commit 1764b29. 💔 clippy-driver on windows: test-pass → build-fail (cc @Manishearth@llogiq@mcarton@oli-obk@phansch, @rust-lang/infra). |
rust-highfive added a commit
to rust-lang-nursery/rust-toolstate
that referenced
this pull request
May 12, 2019
Tested on commit rust-lang/rust@1764b29. Direct link to PR: <rust-lang/rust#60679> 💔 clippy-driver on windows: test-pass → build-fail (cc @Manishearth@llogiq@mcarton@oli-obk@phansch, @rust-lang/infra). 💔 clippy-driver on linux: test-pass → build-fail (cc @Manishearth@llogiq@mcarton@oli-obk@phansch, @rust-lang/infra). 💔 rls on windows: test-pass → build-fail (cc @Xanewok, @rust-lang/infra). 💔 rls on linux: test-pass → build-fail (cc @Xanewok, @rust-lang/infra).
bors added a commit
that referenced
this pull request
May 23, 2019
syntax: Continue refactoring literals A follow up to #60679. a2fd002: Similarly to `EscapeError`, literal parsing now produces a `LitError`. This way we can get rid of `diag: Option<(Span, &Handler)>` in interfaces while leaving attr/mod alone. d9516d1: Gathers all components of a literal token in a single struct.
Centril added a commit
to Centril/rust
that referenced
this pull request
Jun 7, 2019
syntax: Treat error literals in more principled way Free them from their character literal origins. I actually tried to remove `LitKind::Err` entirely (by converting it into `ExprKind::Err` immediately), and it caused no diagnostic regressions in the test suite. However, I'd still want to use error literals as general purpose error tokens some day, so I kept them. The downside of having `LitKind::Err` in addition to `ExprKind::Err` is that every time you want to do something with `ExprKind::Err` you need to make sure that `ExprKind::Lit(LitKind::Err)` is treated in the same way. Fortunately, this usually happens automatically because both literals and errors are "leaf" expressions, however this PR does fix a couple of inconsistencies between them. Addresses rust-lang#60679 (comment) in a way
Centril added a commit
to Centril/rust
that referenced
this pull request
Jun 7, 2019
syntax: Treat error literals in more principled way Free them from their character literal origins. I actually tried to remove `LitKind::Err` entirely (by converting it into `ExprKind::Err` immediately), and it caused no diagnostic regressions in the test suite. However, I'd still want to use error literals as general purpose error tokens some day, so I kept them. The downside of having `LitKind::Err` in addition to `ExprKind::Err` is that every time you want to do something with `ExprKind::Err` you need to make sure that `ExprKind::Lit(LitKind::Err)` is treated in the same way. Fortunately, this usually happens automatically because both literals and errors are "leaf" expressions, however this PR does fix a couple of inconsistencies between them. Addresses rust-lang#60679 (comment) in a way
Centril added a commit
to Centril/rust
that referenced
this pull request
Jun 7, 2019
syntax: Treat error literals in more principled way Free them from their character literal origins. I actually tried to remove `LitKind::Err` entirely (by converting it into `ExprKind::Err` immediately), and it caused no diagnostic regressions in the test suite. However, I'd still want to use error literals as general purpose error tokens some day, so I kept them. The downside of having `LitKind::Err` in addition to `ExprKind::Err` is that every time you want to do something with `ExprKind::Err` you need to make sure that `ExprKind::Lit(LitKind::Err)` is treated in the same way. Fortunately, this usually happens automatically because both literals and errors are "leaf" expressions, however this PR does fix a couple of inconsistencies between them. Addresses rust-lang#60679 (comment) in a way
Centril added a commit
to Centril/rust
that referenced
this pull request
Jun 8, 2019
syntax: Treat error literals in more principled way Free them from their character literal origins. I actually tried to remove `LitKind::Err` entirely (by converting it into `ExprKind::Err` immediately), and it caused no diagnostic regressions in the test suite. However, I'd still want to use error literals as general purpose error tokens some day, so I kept them. The downside of having `LitKind::Err` in addition to `ExprKind::Err` is that every time you want to do something with `ExprKind::Err` you need to make sure that `ExprKind::Lit(LitKind::Err)` is treated in the same way. Fortunately, this usually happens automatically because both literals and errors are "leaf" expressions, however this PR does fix a couple of inconsistencies between them. Addresses rust-lang#60679 (comment) in a way
Merged
Centril added a commit
to Centril/rust
that referenced
this pull request
Nov 17, 2019
syntax: Keep string literals in ABIs and `asm!` more precisely As a result we don't lose spans when `extern` functions or blocks are passed to proc macros, and also escape all string literals consistently. Continuation of rust-lang#60679, which did a similar thing with all literals besides those in ABIs and `asm!`. TODO: Add tests. Fixesrust-lang#60493Fixesrust-lang#64561 r? @Centril
JohnTitor added a commit
to JohnTitor/rust
that referenced
this pull request
Nov 17, 2019
syntax: Keep string literals in ABIs and `asm!` more precisely As a result we don't lose spans when `extern` functions or blocks are passed to proc macros, and also escape all string literals consistently. Continuation of rust-lang#60679, which did a similar thing with all literals besides those in ABIs and `asm!`. TODO: Add tests. Fixesrust-lang#60493Fixesrust-lang#64561 r? @Centril
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The original literal tokens (
token::Lit) are kept in AST until lowering to HIR.The tokens are kept together with their lowered "semantic" representation (
ast::LitKind), so the size ofast::Litis increased (this also increases the size of meta-item structs used for processing built-in attributes).However, the size of
ast::Exprstays the same.The intent is to remove the "semantic" representation from AST eventually and keep literals as tokens until lowering to HIR (at least), and I'm going to work on that, but it would be good to land this sooner to unblock progress on the lexer refactoring.
Fixes a part of #43081 (literal tokens that are passed to proc macros are always precise, including hexadecimal numbers, strings with their original escaping, etc)
Fixes a part of #60495 (everything except for proc macro API doesn't need escaping anymore)
This also allows to eliminate a certain hack from the lexer (https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/pretty-printing.20comments/near/165005357).
cc @matklad