Skip to content

[Parser][NFC] Remove Token from lexer interface - #6333

Merged
tlively merged 1 commit into
mainfrom
parser-no-token-peek
Feb 22, 2024
Merged

[Parser][NFC] Remove Token from lexer interface#6333
tlively merged 1 commit into
mainfrom
parser-no-token-peek

Conversation

@tlively

Copy link
Copy Markdown
Member

Replace the general peek method that returned a Token with specific peek
methods that look for (but do not consume) specific kinds of tokens. This change
is a prerequisite for simplifying the lexer implementation by removing Token
entirely.

@tlively

tlively commented Feb 22, 2024

Copy link
Copy Markdown
MemberAuthor

@kripkenkripken left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is a prerequisite for simplifying the lexer implementation by removing Token entirely.

Perhaps a Token-based interface will be useful eventually for something? But lgtm if you don't expect that.

@tlively

Copy link
Copy Markdown
MemberAuthor

I think the peekXXX and takeXXX methods should be enough for everything we would want to do. Being able to query positions and create a new lexer over arbitrary substrings of the input even without explicit tokens also gives us enough flexibility that I'm comfortable removing tokens.

@tlively

tlively commented Feb 22, 2024

Copy link
Copy Markdown
MemberAuthor

Merge activity

  • Feb 22, 12:09 PM EST: @tlively started a stack merge that includes this pull request via Graphite.
  • Feb 22, 12:10 PM EST: Graphite rebased this pull request as part of a merge.
  • Feb 22, 12:37 PM EST: @tlively merged this pull request with Graphite.

Base automatically changed from parser-remove-input to mainFebruary 22, 2024 17:09
Replace the general `peek` method that returned a `Token` with specific peek
methods that look for (but do not consume) specific kinds of tokens. This change
is a prerequisite for simplifying the lexer implementation by removing `Token`
entirely.
@tlively
tlivelyforce-pushed the parser-no-token-peek branch from 2c5b813 to 7a7258cCompareFebruary 22, 2024 17:09
@tlively
tlively merged commit f6bb943 into mainFeb 22, 2024
@tlively
tlively deleted the parser-no-token-peek branch February 22, 2024 17:37
radekdoulik pushed a commit to dotnet/binaryen that referenced this pull request Jul 12, 2024
Replace the general `peek` method that returned a `Token` with specific peek
methods that look for (but do not consume) specific kinds of tokens. This change
is a prerequisite for simplifying the lexer implementation by removing `Token`
entirely.
@gkdngkdn mentioned this pull request Aug 31, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@tlively@kripken