Optimize `scan` with a lookup table to reduce branching - #59106

Closed

Daniel Rosenwasser (DanielRosenwasser) wants to merge 13 commits into

mainfrom

scan-experiments

Commits on Jul 2, 2024

Create a lookup table on token info to avoid excessive branching.
Daniel Rosenwasser (DanielRosenwasser)
committed
e6de1d9
Use an array for common cases.
Daniel Rosenwasser (DanielRosenwasser)
committed
f438918
Handle digits.
Daniel Rosenwasser (DanielRosenwasser)
committed
e8769ca
Fixes.
Daniel Rosenwasser (DanielRosenwasser)
committed
a087c24
Moved everything outside of identifier handling to use a lookup.
Daniel Rosenwasser (DanielRosenwasser)
committed
811526d
Check for CR, LF, space, and tab before doing the lookup.
Daniel Rosenwasser (DanielRosenwasser)
committed
9830671
Flip condition for readabiity (avoids indentation).
Daniel Rosenwasser (DanielRosenwasser)
committed
af340b7
Use `Debug.fail` to avoid a call.
Daniel Rosenwasser (DanielRosenwasser)
committed
d5898b5
`TokenInfo` -> `TokenCategory`
Daniel Rosenwasser (DanielRosenwasser)
committed
11a24a7
More rename.
Daniel Rosenwasser (DanielRosenwasser)
committed
9e5e29f
Add missing `nextLine` entry to whitespace, remove unnecessary branches.
Daniel Rosenwasser (DanielRosenwasser)
committed
db10cae
Rename `None` to `IdentifierOrUnknown`
Daniel Rosenwasser (DanielRosenwasser)
committed
3bdfe84

Commits on Jul 3, 2024

Hit the table once for whitespace, then tight-loop.
Daniel Rosenwasser (DanielRosenwasser)
committed
33ae58d